2021-05-25T11:48:38.845 INFO:root:teuthology version: 1.1.0-d92ec140 2021-05-25T11:48:38.847 DEBUG:teuthology.run:Teuthology command: teuthology --description rados/monthrash/{ceph clusters/3-mons mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8} thrashers/many workloads/rados_api_tests} --archive /home/teuthworker/archive/kchai-2021-05-25_11:32:10-rados-master-distro-basic-smithi/6134024 --owner scheduled_kchai@teuthology --verbose --name kchai-2021-05-25_11:32:10-rados-master-distro-basic-smithi -- /home/teuthworker/archive/kchai-2021-05-25_11:32:10-rados-master-distro-basic-smithi/6134024/orig.config.yaml 2021-05-25T11:48:38.887 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2021-05-25T11:48:38.939 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/kchai-2021-05-25_11:32:10-rados-master-distro-basic-smithi/6134024 branch: master description: rados/monthrash/{ceph clusters/3-mons mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8} thrashers/many workloads/rados_api_tests} email: null first_in_suite: false job_id: '6134024' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: kchai-2021-05-25_11:32:10-rados-master-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: centos os_version: '8.3' overrides: admin_socket: branch: master ceph: conf: client: debug monc: 20 debug ms: 1 global: debug ms: 1 debug objecter: 20 debug rados: 20 mon client directed command retry: 5 mon election default strategy: 3 ms bind msgr2: false ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 25 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub inject crc mismatch: 0.01 mon scrub inject missing keys: 0.05 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 5 osd: bluestore block size: 96636764160 bluestore compression algorithm: zlib bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon client ping interval: 4 mon client ping timeout: 12 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - ScrubResult - scrub mismatch - overall HEALTH_ - \(MGR_DOWN\) - \(PG_AVAILABILITY\) - \(SLOW_OPS\) - slow request - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(MON_DOWN\) - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_FULL\) - \(SLOW_OPS\) - \(MON_DOWN\) - \(PG_ - \(POOL_APP_NOT_ENABLED\) - \(SMALLER_PGP_NUM\) - slow request log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) mon_bind_msgr2: false sha1: ece5ed1ac99f43a50411c04b4ca92c148baf0cd1 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: ece5ed1ac99f43a50411c04b4ca92c148baf0cd1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-kefu-testing-2021-05-25-1039 sha1: a16ab1caacd8b41a7fe74bc40a10ec74ee8f5802 owner: scheduled_kchai@teuthology priority: 60 repo: git://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - osd.0 - osd.1 - osd.2 - - mon.b - mgr.x - osd.3 - osd.4 - osd.5 - client.0 sha1: ece5ed1ac99f43a50411c04b4ca92c148baf0cd1 sleep_before_teardown: 0 suite: rados suite_branch: wip-kefu-testing-2021-05-25-1039 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_a16ab1caacd8b41a7fe74bc40a10ec74ee8f5802/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: a16ab1caacd8b41a7fe74bc40a10ec74ee8f5802 targets: smithi102.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDPYTl+tVRlf0cNuMvygA/Yga0FJP6usxKwc8vN1QFMnOQPq3Hsoo1dLLjJSi/E932f0QySRVdI3MFop8if8qvKTFPgBkabdxHhSuca6vtL5KQM3riUwm2KX88rvHIkTWaeJURSRDCWN9tZI9frx9NJmi685km7RCn2ZfaNzDXXyV32ZrHp2b24yFJr9Pt4kTWUDZY0CdahLVlg88ITssRlbybE/Ms/kcus2JobIHApn7ZXKdoF2snUpocKm3yjYozagGLiNoNzjLOhpEPd2q+5iJ+YznN5BLxBPK093EtDmOUvlF0Mhv/Py9rSsmEKPcSMNu8Z+FVnMRO5st6KU3Y2IzJmahfB2nodNlfC+AnOcbpMl92cL02WVLAaIsVF2tg4xieaUGJrn3kuNCTq78rLjwfqO+Du3ZHbgbLelV3xHzaWbBJUxGddoS/jJiqU9j49pbofTkantWkmh99dAkJ0YWL/8VCVkHXtfxvLa2cWHNrvc290A4BpuC2RPIMHQHU= smithi181.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDb98fBVldQtZu7X/AXNeYvvo5iK2gGpwuPGtcT6Zal8pQTgaF0XFDDfAbxA2tY42m3wJA56o2D5PAcNvLpLaAjUPH+uSsClydVoLbbYRJM2syXkop0jH28lKW7uCDf3cv25VLGodjKS8la5KHw4xRRoqeBfKy4rbUx0AsEi5AHv3Xut/akBhVyX7UiE3z2FuRcIyM9e1/BOH9m+3Oumu6vZ/1xU/MHdPzpdpJHovFMEr8tYMEc/H8eMsZE8wzJDMZREt401+N3T8c9k94Vdan7IqqWzWxahq/JiJO34LoIpUWvc1fkReFuG657d95rqfpLVzd+acyOCcY/QOX4uGO7bnxqG/M+wmn/iwN8Uat9cn00jj9sMutvsqQDp9bgs7yCqGFpEAGPKzblljp8IoAVZB9ixJumFiAFGlOQc2iFQgnW6peJZsDNbNEvnln9FmaBykybWqOtIm3zXp/sWGC8JZO1Jrq97V2kH8JHmYYoVzaMA1Zxqfucbs0Ye7N2Mu8= tasks: - install: null - ceph: null - mon_thrash: freeze_mon_duration: 20 freeze_mon_probability: 10 revive_delay: 20 thrash_delay: 1 thrash_many: true - workunit: clients: client.0: - rados/test.sh teuthology_branch: master teuthology_sha1: d92ec1407c30d60262a45a2844110a5c94d567cc timestamp: 2021-05-25_11:32:10 tube: smithi user: kchai verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.8274 2021-05-25T11:48:38.939 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_a16ab1caacd8b41a7fe74bc40a10ec74ee8f5802/qa; will attempt to use it 2021-05-25T11:48:38.940 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_a16ab1caacd8b41a7fe74bc40a10ec74ee8f5802/qa/tasks 2021-05-25T11:48:38.941 INFO:teuthology.run_tasks:Running task internal.check_packages... 2021-05-25T11:48:38.942 INFO:teuthology.task.internal:Checking packages... 2021-05-25T11:48:38.964 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'basic' and ceph hash 'ece5ed1ac99f43a50411c04b4ca92c148baf0cd1' 2021-05-25T11:48:38.965 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2021-05-25T11:48:38.965 INFO:teuthology.packaging:ref: None 2021-05-25T11:48:38.966 INFO:teuthology.packaging:tag: None 2021-05-25T11:48:38.966 INFO:teuthology.packaging:branch: master 2021-05-25T11:48:38.966 INFO:teuthology.packaging:sha1: ece5ed1ac99f43a50411c04b4ca92c148baf0cd1 2021-05-25T11:48:38.967 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=master 2021-05-25T11:48:39.231 INFO:teuthology.task.internal:Found packages for ceph version 17.0.0-4502.gece5ed1a 2021-05-25T11:48:39.231 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2021-05-25T11:48:39.242 INFO:teuthology.task.internal:no buildpackages task found 2021-05-25T11:48:39.242 INFO:teuthology.run_tasks:Running task internal.save_config... 2021-05-25T11:48:39.499 INFO:teuthology.task.internal:Saving configuration 2021-05-25T11:48:39.514 INFO:teuthology.run_tasks:Running task internal.check_lock... 2021-05-25T11:48:39.604 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-25T11:48:39.621 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-05-25 11:43:51.871512', 'description': '/home/teuthworker/archive/kchai-2021-05-25_11:32:10-rados-master-distro-basic-smithi/6134024', 'locked_by': 'scheduled_kchai@teuthology', 'up': True, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCJPXxjmwNiYOVxMbyNBMqgtjUeEgou0P019A4IGm7azodRauT6vYonCBFN+tgupHMme61dCsBEEQv+IFL+JJcU=', 'os_version': '8', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'centos', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi102.front.sepia.ceph.com'} 2021-05-25T11:48:39.636 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-05-25 11:43:51.871492', 'description': '/home/teuthworker/archive/kchai-2021-05-25_11:32:10-rados-master-distro-basic-smithi/6134024', 'locked_by': 'scheduled_kchai@teuthology', 'up': True, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ97Snkm0BgFkcpfsYTmUXlW/fC0wZIAILi7MU8/ZrE25xx8JZor24GCqt2Cl00XKuq5Py7XmcSEXTXSMR6JFPw=', 'os_version': '8', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'centos', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi181.front.sepia.ceph.com'} 2021-05-25T11:48:39.636 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2021-05-25T11:48:39.647 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'] 2021-05-25T11:48:39.647 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0'] 2021-05-25T11:48:39.648 INFO:teuthology.run_tasks:Running task console_log... 2021-05-25T11:48:39.717 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fb8e1fcbbf8>, signals=[15]) 2021-05-25T11:48:39.718 INFO:teuthology.run_tasks:Running task internal.connect... 2021-05-25T11:48:39.729 INFO:teuthology.task.internal:Opening connections... 2021-05-25T11:48:39.729 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi102.front.sepia.ceph.com 2021-05-25T11:48:39.731 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-05-25T11:48:39.800 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi181.front.sepia.ceph.com 2021-05-25T11:48:39.801 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-05-25T11:48:39.868 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2021-05-25T11:48:39.880 DEBUG:teuthology.orchestra.run.smithi102:> uname -m 2021-05-25T11:48:39.921 INFO:teuthology.orchestra.run.smithi102.stdout:x86_64 2021-05-25T11:48:39.921 DEBUG:teuthology.orchestra.run.smithi102:> cat /etc/os-release 2021-05-25T11:48:39.974 INFO:teuthology.orchestra.run.smithi102.stdout:NAME="CentOS Linux" 2021-05-25T11:48:39.974 INFO:teuthology.orchestra.run.smithi102.stdout:VERSION="8" 2021-05-25T11:48:39.975 INFO:teuthology.orchestra.run.smithi102.stdout:ID="centos" 2021-05-25T11:48:39.975 INFO:teuthology.orchestra.run.smithi102.stdout:ID_LIKE="rhel fedora" 2021-05-25T11:48:39.975 INFO:teuthology.orchestra.run.smithi102.stdout:VERSION_ID="8" 2021-05-25T11:48:39.975 INFO:teuthology.orchestra.run.smithi102.stdout:PLATFORM_ID="platform:el8" 2021-05-25T11:48:39.976 INFO:teuthology.orchestra.run.smithi102.stdout:PRETTY_NAME="CentOS Linux 8" 2021-05-25T11:48:39.976 INFO:teuthology.orchestra.run.smithi102.stdout:ANSI_COLOR="0;31" 2021-05-25T11:48:39.976 INFO:teuthology.orchestra.run.smithi102.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-05-25T11:48:39.976 INFO:teuthology.orchestra.run.smithi102.stdout:HOME_URL="https://centos.org/" 2021-05-25T11:48:39.976 INFO:teuthology.orchestra.run.smithi102.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-05-25T11:48:39.977 INFO:teuthology.orchestra.run.smithi102.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-05-25T11:48:39.977 INFO:teuthology.orchestra.run.smithi102.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-05-25T11:48:39.977 INFO:teuthology.lock.ops:Updating smithi102.front.sepia.ceph.com on lock server 2021-05-25T11:48:39.991 DEBUG:teuthology.orchestra.run.smithi181:> uname -m 2021-05-25T11:48:40.007 INFO:teuthology.orchestra.run.smithi181.stdout:x86_64 2021-05-25T11:48:40.007 DEBUG:teuthology.orchestra.run.smithi181:> cat /etc/os-release 2021-05-25T11:48:40.061 INFO:teuthology.orchestra.run.smithi181.stdout:NAME="CentOS Linux" 2021-05-25T11:48:40.061 INFO:teuthology.orchestra.run.smithi181.stdout:VERSION="8" 2021-05-25T11:48:40.061 INFO:teuthology.orchestra.run.smithi181.stdout:ID="centos" 2021-05-25T11:48:40.062 INFO:teuthology.orchestra.run.smithi181.stdout:ID_LIKE="rhel fedora" 2021-05-25T11:48:40.062 INFO:teuthology.orchestra.run.smithi181.stdout:VERSION_ID="8" 2021-05-25T11:48:40.062 INFO:teuthology.orchestra.run.smithi181.stdout:PLATFORM_ID="platform:el8" 2021-05-25T11:48:40.062 INFO:teuthology.orchestra.run.smithi181.stdout:PRETTY_NAME="CentOS Linux 8" 2021-05-25T11:48:40.063 INFO:teuthology.orchestra.run.smithi181.stdout:ANSI_COLOR="0;31" 2021-05-25T11:48:40.063 INFO:teuthology.orchestra.run.smithi181.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-05-25T11:48:40.063 INFO:teuthology.orchestra.run.smithi181.stdout:HOME_URL="https://centos.org/" 2021-05-25T11:48:40.063 INFO:teuthology.orchestra.run.smithi181.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-05-25T11:48:40.063 INFO:teuthology.orchestra.run.smithi181.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-05-25T11:48:40.064 INFO:teuthology.orchestra.run.smithi181.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-05-25T11:48:40.064 INFO:teuthology.lock.ops:Updating smithi181.front.sepia.ceph.com on lock server 2021-05-25T11:48:40.078 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2021-05-25T11:48:40.092 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2021-05-25T11:48:40.115 INFO:teuthology.task.internal:Checking for old test directory... 2021-05-25T11:48:40.116 DEBUG:teuthology.orchestra.run.smithi102:> test '!' -e /home/ubuntu/cephtest 2021-05-25T11:48:40.119 DEBUG:teuthology.orchestra.run.smithi181:> test '!' -e /home/ubuntu/cephtest 2021-05-25T11:48:40.132 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2021-05-25T11:48:40.143 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2021-05-25T11:48:40.144 DEBUG:teuthology.orchestra.run.smithi102:> test -z $(ls -A /var/lib/ceph) 2021-05-25T11:48:40.172 DEBUG:teuthology.orchestra.run.smithi181:> test -z $(ls -A /var/lib/ceph) 2021-05-25T11:48:40.195 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2021-05-25T11:48:40.237 INFO:teuthology.run_tasks:Running task kernel... 2021-05-25T11:48:40.252 DEBUG:teuthology.task.kernel:normalized config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.c': {'kdb': True, 'sha1': 'distro'}, 'osd.0': {'kdb': True, 'sha1': 'distro'}, 'osd.1': {'kdb': True, 'sha1': 'distro'}, 'osd.2': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'mgr.x': {'kdb': True, 'sha1': 'distro'}, 'osd.3': {'kdb': True, 'sha1': 'distro'}, 'osd.4': {'kdb': True, 'sha1': 'distro'}, 'osd.5': {'kdb': True, 'sha1': 'distro'}, 'client.0': {'kdb': True, 'sha1': 'distro'}} 2021-05-25T11:48:40.253 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2021-05-25T11:48:40.253 DEBUG:teuthology.orchestra.run.smithi102:> uname -r 2021-05-25T11:48:40.267 INFO:teuthology.orchestra.run.smithi102.stdout:4.18.0-240.10.1.el8_3.x86_64 2021-05-25T11:48:40.267 INFO:teuthology.task.kernel:Running kernel on smithi102: 4.18.0-240.10.1.el8_3.x86_64 2021-05-25T11:48:40.267 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum install -y kernel 2021-05-25T11:48:43.381 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS-8 - AppStream 34 MB/s | 6.3 MB 00:00 2021-05-25T11:48:44.479 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS-8 - Base 25 MB/s | 2.3 MB 00:00 2021-05-25T11:48:44.915 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS-8 - Extras 150 kB/s | 9.6 kB 00:00 2021-05-25T11:48:45.491 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Linux 8 - AppStream 12 MB/s | 6.3 MB 00:00 2021-05-25T11:48:48.799 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Linux 8 - BaseOS 1.0 MB/s | 2.3 MB 00:02 2021-05-25T11:48:49.429 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Linux 8 - Extras 38 kB/s | 9.6 kB 00:00 2021-05-25T11:48:50.138 INFO:teuthology.orchestra.run.smithi102.stdout:Extra Packages for Enterprise Linux 14 MB/s | 9.4 MB 00:00 2021-05-25T11:48:52.161 INFO:teuthology.orchestra.run.smithi102.stdout:lab-extras 4.4 MB/s | 24 kB 00:00 2021-05-25T11:48:53.674 INFO:teuthology.orchestra.run.smithi102.stdout:Package kernel-4.18.0-240.10.1.el8_3.x86_64 is already installed. 2021-05-25T11:48:53.674 INFO:teuthology.orchestra.run.smithi102.stdout:Package kernel-4.18.0-240.el8.x86_64 is already installed. 2021-05-25T11:48:53.904 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2021-05-25T11:48:53.919 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-05-25T11:48:53.919 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2021-05-25T11:48:53.919 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-05-25T11:48:53.919 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2021-05-25T11:48:53.920 INFO:teuthology.orchestra.run.smithi102.stdout: kernel x86_64 4.18.0-240.22.1.el8_3 CentOS-Base 4.4 M 2021-05-25T11:48:53.920 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2021-05-25T11:48:53.920 INFO:teuthology.orchestra.run.smithi102.stdout: kernel-core x86_64 4.18.0-240.22.1.el8_3 CentOS-Base 30 M 2021-05-25T11:48:53.920 INFO:teuthology.orchestra.run.smithi102.stdout: kernel-modules x86_64 4.18.0-240.22.1.el8_3 CentOS-Base 26 M 2021-05-25T11:48:53.920 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:48:53.921 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2021-05-25T11:48:53.921 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-05-25T11:48:53.921 INFO:teuthology.orchestra.run.smithi102.stdout:Install 3 Packages 2021-05-25T11:48:53.921 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:48:53.922 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 60 M 2021-05-25T11:48:53.922 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 83 M 2021-05-25T11:48:53.922 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2021-05-25T11:48:54.026 INFO:teuthology.orchestra.run.smithi102.stdout:(1/3): kernel-4.18.0-240.22.1.el8_3.x86_64.rpm 41 MB/s | 4.4 MB 00:00 2021-05-25T11:48:54.713 INFO:teuthology.orchestra.run.smithi102.stdout:(2/3): kernel-core-4.18.0-240.22.1.el8_3.x86_64 38 MB/s | 30 MB 00:00 2021-05-25T11:48:54.939 INFO:teuthology.orchestra.run.smithi102.stdout:(3/3): kernel-modules-4.18.0-240.22.1.el8_3.x86 25 MB/s | 26 MB 00:01 2021-05-25T11:48:54.942 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2021-05-25T11:48:54.943 INFO:teuthology.orchestra.run.smithi102.stdout:Total 59 MB/s | 60 MB 00:01 2021-05-25T11:48:55.027 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2021-05-25T11:48:55.355 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2021-05-25T11:48:55.356 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2021-05-25T11:48:56.474 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2021-05-25T11:48:56.492 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2021-05-25T11:48:58.805 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2021-05-25T11:48:59.295 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : kernel-core-4.18.0-240.22.1.el8_3.x86_64 1/3 2021-05-25T11:49:02.385 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: kernel-core-4.18.0-240.22.1.el8_3.x86_64 1/3 2021-05-25T11:49:02.719 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : kernel-modules-4.18.0-240.22.1.el8_3.x86_64 2/3 2021-05-25T11:49:05.591 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: kernel-modules-4.18.0-240.22.1.el8_3.x86_64 2/3 2021-05-25T11:49:05.813 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : kernel-4.18.0-240.22.1.el8_3.x86_64 3/3 2021-05-25T11:49:25.768 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: kernel-core-4.18.0-240.22.1.el8_3.x86_64 3/3 2021-05-25T11:49:26.031 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: kernel-4.18.0-240.22.1.el8_3.x86_64 3/3 2021-05-25T11:49:26.031 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : kernel-4.18.0-240.22.1.el8_3.x86_64 1/3 2021-05-25T11:49:26.032 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : kernel-core-4.18.0-240.22.1.el8_3.x86_64 2/3 2021-05-25T11:49:26.312 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : kernel-modules-4.18.0-240.22.1.el8_3.x86_64 3/3 2021-05-25T11:49:26.312 INFO:teuthology.orchestra.run.smithi102.stdout:Installed products updated. 2021-05-25T11:49:26.674 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:49:26.674 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2021-05-25T11:49:26.675 INFO:teuthology.orchestra.run.smithi102.stdout: kernel-4.18.0-240.22.1.el8_3.x86_64 2021-05-25T11:49:26.675 INFO:teuthology.orchestra.run.smithi102.stdout: kernel-core-4.18.0-240.22.1.el8_3.x86_64 2021-05-25T11:49:26.675 INFO:teuthology.orchestra.run.smithi102.stdout: kernel-modules-4.18.0-240.22.1.el8_3.x86_64 2021-05-25T11:49:26.675 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:49:26.676 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2021-05-25T11:49:26.744 DEBUG:teuthology.orchestra.run.smithi102:> rpm -q kernel --last | head -n 1 2021-05-25T11:49:26.874 INFO:teuthology.orchestra.run.smithi102.stdout:kernel-4.18.0-240.22.1.el8_3.x86_64 Tue 25 May 2021 11:49:05 AM UTC 2021-05-25T11:49:26.876 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-240.22.1.el8_3.x86_64 2021-05-25T11:49:26.876 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-240.10.1.el8_3.x86_64 Expected: 4.18.0-240.22.1.el8_3.x86_64 2021-05-25T11:49:26.877 DEBUG:teuthology.orchestra.run.smithi181:> uname -r 2021-05-25T11:49:26.894 INFO:teuthology.orchestra.run.smithi181.stdout:4.18.0-240.10.1.el8_3.x86_64 2021-05-25T11:49:26.894 INFO:teuthology.task.kernel:Running kernel on smithi181: 4.18.0-240.10.1.el8_3.x86_64 2021-05-25T11:49:26.894 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum install -y kernel 2021-05-25T11:49:29.860 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS-8 - AppStream 37 MB/s | 6.3 MB 00:00 2021-05-25T11:49:30.949 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS-8 - Base 26 MB/s | 2.3 MB 00:00 2021-05-25T11:49:31.390 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS-8 - Extras 143 kB/s | 9.6 kB 00:00 2021-05-25T11:49:32.106 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Linux 8 - AppStream 9.4 MB/s | 6.3 MB 00:00 2021-05-25T11:49:36.074 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Linux 8 - BaseOS 793 kB/s | 2.3 MB 00:02 2021-05-25T11:49:36.612 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Linux 8 - Extras 58 kB/s | 9.6 kB 00:00 2021-05-25T11:49:37.278 INFO:teuthology.orchestra.run.smithi181.stdout:Extra Packages for Enterprise Linux 15 MB/s | 9.4 MB 00:00 2021-05-25T11:49:39.240 INFO:teuthology.orchestra.run.smithi181.stdout:lab-extras 4.3 MB/s | 24 kB 00:00 2021-05-25T11:49:40.735 INFO:teuthology.orchestra.run.smithi181.stdout:Package kernel-4.18.0-240.10.1.el8_3.x86_64 is already installed. 2021-05-25T11:49:40.735 INFO:teuthology.orchestra.run.smithi181.stdout:Package kernel-4.18.0-240.el8.x86_64 is already installed. 2021-05-25T11:49:40.963 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2021-05-25T11:49:40.975 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2021-05-25T11:49:40.975 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2021-05-25T11:49:40.976 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2021-05-25T11:49:40.976 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2021-05-25T11:49:40.976 INFO:teuthology.orchestra.run.smithi181.stdout: kernel x86_64 4.18.0-240.22.1.el8_3 CentOS-Base 4.4 M 2021-05-25T11:49:40.976 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2021-05-25T11:49:40.976 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-core x86_64 4.18.0-240.22.1.el8_3 CentOS-Base 30 M 2021-05-25T11:49:40.977 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-modules x86_64 4.18.0-240.22.1.el8_3 CentOS-Base 26 M 2021-05-25T11:49:40.977 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:49:40.977 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2021-05-25T11:49:40.977 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2021-05-25T11:49:40.977 INFO:teuthology.orchestra.run.smithi181.stdout:Install 3 Packages 2021-05-25T11:49:40.978 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:49:40.978 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 60 M 2021-05-25T11:49:40.979 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 83 M 2021-05-25T11:49:40.979 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2021-05-25T11:49:41.072 INFO:teuthology.orchestra.run.smithi181.stdout:(1/3): kernel-4.18.0-240.22.1.el8_3.x86_64.rpm 46 MB/s | 4.4 MB 00:00 2021-05-25T11:49:41.659 INFO:teuthology.orchestra.run.smithi181.stdout:(2/3): kernel-core-4.18.0-240.22.1.el8_3.x86_64 45 MB/s | 30 MB 00:00 2021-05-25T11:49:41.945 INFO:teuthology.orchestra.run.smithi181.stdout:(3/3): kernel-modules-4.18.0-240.22.1.el8_3.x86 26 MB/s | 26 MB 00:00 2021-05-25T11:49:41.949 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2021-05-25T11:49:41.949 INFO:teuthology.orchestra.run.smithi181.stdout:Total 62 MB/s | 60 MB 00:00 2021-05-25T11:49:42.034 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2021-05-25T11:49:42.361 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2021-05-25T11:49:42.361 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2021-05-25T11:49:43.470 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2021-05-25T11:49:43.484 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2021-05-25T11:49:45.725 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2021-05-25T11:49:46.159 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : kernel-core-4.18.0-240.22.1.el8_3.x86_64 1/3 2021-05-25T11:49:49.188 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: kernel-core-4.18.0-240.22.1.el8_3.x86_64 1/3 2021-05-25T11:49:49.691 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : kernel-modules-4.18.0-240.22.1.el8_3.x86_64 2/3 2021-05-25T11:49:52.569 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: kernel-modules-4.18.0-240.22.1.el8_3.x86_64 2/3 2021-05-25T11:49:52.752 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : kernel-4.18.0-240.22.1.el8_3.x86_64 3/3 2021-05-25T11:50:12.626 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: kernel-core-4.18.0-240.22.1.el8_3.x86_64 3/3 2021-05-25T11:50:12.881 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: kernel-4.18.0-240.22.1.el8_3.x86_64 3/3 2021-05-25T11:50:12.881 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : kernel-4.18.0-240.22.1.el8_3.x86_64 1/3 2021-05-25T11:50:12.881 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : kernel-core-4.18.0-240.22.1.el8_3.x86_64 2/3 2021-05-25T11:50:13.144 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : kernel-modules-4.18.0-240.22.1.el8_3.x86_64 3/3 2021-05-25T11:50:13.144 INFO:teuthology.orchestra.run.smithi181.stdout:Installed products updated. 2021-05-25T11:50:13.504 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:50:13.504 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2021-05-25T11:50:13.504 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-4.18.0-240.22.1.el8_3.x86_64 2021-05-25T11:50:13.505 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-core-4.18.0-240.22.1.el8_3.x86_64 2021-05-25T11:50:13.505 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-modules-4.18.0-240.22.1.el8_3.x86_64 2021-05-25T11:50:13.505 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:50:13.505 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2021-05-25T11:50:13.590 DEBUG:teuthology.orchestra.run.smithi181:> rpm -q kernel --last | head -n 1 2021-05-25T11:50:13.724 INFO:teuthology.orchestra.run.smithi181.stdout:kernel-4.18.0-240.22.1.el8_3.x86_64 Tue 25 May 2021 11:49:52 AM UTC 2021-05-25T11:50:13.726 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-240.22.1.el8_3.x86_64 2021-05-25T11:50:13.726 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-240.10.1.el8_3.x86_64 Expected: 4.18.0-240.22.1.el8_3.x86_64 2021-05-25T11:50:13.726 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2021-05-25T11:50:13.727 DEBUG:teuthology.task.kernel:src is distro, skipping download 2021-05-25T11:50:13.727 DEBUG:teuthology.task.kernel:src is distro, skipping download 2021-05-25T11:50:13.727 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2021-05-25T11:50:13.727 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi102.front.sepia.ceph.com, path=None, version=distro) 2021-05-25T11:50:13.728 DEBUG:teuthology.orchestra.run.smithi102:> rpm -q kernel --last | head -n 1 2021-05-25T11:50:13.843 INFO:teuthology.orchestra.run.smithi102.stdout:kernel-4.18.0-240.22.1.el8_3.x86_64 Tue 25 May 2021 11:49:05 AM UTC 2021-05-25T11:50:13.845 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-240.22.1.el8_3.x86_64 2021-05-25T11:50:13.845 DEBUG:teuthology.orchestra.run.smithi102:> sudo rpm -qi grub2-tools 2021-05-25T11:50:13.877 INFO:teuthology.orchestra.run.smithi102.stdout:Name : grub2-tools 2021-05-25T11:50:13.877 INFO:teuthology.orchestra.run.smithi102.stdout:Epoch : 1 2021-05-25T11:50:13.877 INFO:teuthology.orchestra.run.smithi102.stdout:Version : 2.02 2021-05-25T11:50:13.878 INFO:teuthology.orchestra.run.smithi102.stdout:Release : 90.el8 2021-05-25T11:50:13.878 INFO:teuthology.orchestra.run.smithi102.stdout:Architecture: x86_64 2021-05-25T11:50:13.878 INFO:teuthology.orchestra.run.smithi102.stdout:Install Date: Mon 01 Mar 2021 06:17:56 PM UTC 2021-05-25T11:50:13.878 INFO:teuthology.orchestra.run.smithi102.stdout:Group : System Environment/Base 2021-05-25T11:50:13.879 INFO:teuthology.orchestra.run.smithi102.stdout:Size : 9197207 2021-05-25T11:50:13.879 INFO:teuthology.orchestra.run.smithi102.stdout:License : GPLv3+ 2021-05-25T11:50:13.879 INFO:teuthology.orchestra.run.smithi102.stdout:Signature : RSA/SHA256, Tue 08 Sep 2020 07:14:39 PM UTC, Key ID 05b555b38483c65d 2021-05-25T11:50:13.879 INFO:teuthology.orchestra.run.smithi102.stdout:Source RPM : grub2-2.02-90.el8.src.rpm 2021-05-25T11:50:13.880 INFO:teuthology.orchestra.run.smithi102.stdout:Build Date : Tue 08 Sep 2020 07:00:29 PM UTC 2021-05-25T11:50:13.880 INFO:teuthology.orchestra.run.smithi102.stdout:Build Host : kbuilder.bsys.centos.org 2021-05-25T11:50:13.880 INFO:teuthology.orchestra.run.smithi102.stdout:Relocations : (not relocatable) 2021-05-25T11:50:13.880 INFO:teuthology.orchestra.run.smithi102.stdout:Packager : CentOS BuildSystem 2021-05-25T11:50:13.880 INFO:teuthology.orchestra.run.smithi102.stdout:Vendor : CentOS 2021-05-25T11:50:13.881 INFO:teuthology.orchestra.run.smithi102.stdout:URL : http://www.gnu.org/software/grub/ 2021-05-25T11:50:13.881 INFO:teuthology.orchestra.run.smithi102.stdout:Summary : Support tools for GRUB. 2021-05-25T11:50:13.881 INFO:teuthology.orchestra.run.smithi102.stdout:Description : 2021-05-25T11:50:13.881 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:50:13.882 INFO:teuthology.orchestra.run.smithi102.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2021-05-25T11:50:13.882 INFO:teuthology.orchestra.run.smithi102.stdout:customizable bootloader with modular architecture. It supports a rich 2021-05-25T11:50:13.882 INFO:teuthology.orchestra.run.smithi102.stdout:variety of kernel formats, file systems, computer architectures and 2021-05-25T11:50:13.882 INFO:teuthology.orchestra.run.smithi102.stdout:hardware devices. 2021-05-25T11:50:13.882 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:50:13.883 INFO:teuthology.orchestra.run.smithi102.stdout:This subpackage provides tools for support of all platforms. 2021-05-25T11:50:13.884 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2021-05-25T11:50:13.884 INFO:teuthology.task.kernel:Updating grub on smithi102 to boot 4.18.0-240.22.1.el8_3.x86_64 2021-05-25T11:50:13.884 DEBUG:teuthology.orchestra.run.smithi102:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2021-05-25T11:50:14.674 INFO:teuthology.orchestra.run.smithi102.stderr:Generating grub configuration file ... 2021-05-25T11:50:16.781 INFO:teuthology.orchestra.run.smithi102.stderr:done 2021-05-25T11:50:16.783 DEBUG:teuthology.orchestra.run.smithi102:> mktemp 2021-05-25T11:50:16.874 INFO:teuthology.orchestra.run.smithi102.stdout:/tmp/tmp.ykMoK776L6 2021-05-25T11:50:16.875 DEBUG:teuthology.orchestra.run.smithi102:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.ykMoK776L6 2021-05-25T11:50:16.905 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod 0666 /tmp/tmp.ykMoK776L6 2021-05-25T11:50:17.060 DEBUG:teuthology.orchestra.remote:smithi102:/tmp/tmp.ykMoK776L6 is 6KB 2021-05-25T11:50:17.109 DEBUG:teuthology.orchestra.run.smithi102:> rm -fr /tmp/tmp.ykMoK776L6 2021-05-25T11:50:17.123 DEBUG:teuthology.orchestra.run.smithi102:> sudo /bin/ls /boot/loader/entries || true 2021-05-25T11:50:17.184 INFO:teuthology.orchestra.run.smithi102.stdout:631d84db90f7409fbdd390573ae89718-0-rescue.conf 2021-05-25T11:50:17.185 INFO:teuthology.orchestra.run.smithi102.stdout:631d84db90f7409fbdd390573ae89718-4.18.0-240.10.1.el8_3.x86_64.conf 2021-05-25T11:50:17.185 INFO:teuthology.orchestra.run.smithi102.stdout:631d84db90f7409fbdd390573ae89718-4.18.0-240.22.1.el8_3.x86_64.conf 2021-05-25T11:50:17.185 INFO:teuthology.orchestra.run.smithi102.stdout:631d84db90f7409fbdd390573ae89718-4.18.0-240.el8.x86_64.conf 2021-05-25T11:50:17.186 DEBUG:teuthology.orchestra.run.smithi102:> sudo grub2-set-default 631d84db90f7409fbdd390573ae89718-4.18.0-240.22.1.el8_3.x86_64 2021-05-25T11:50:17.332 DEBUG:teuthology.orchestra.run.smithi102:> sudo shutdown -r now 2021-05-25T11:50:17.335 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2021-05-25T11:50:17.335 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi181.front.sepia.ceph.com, path=None, version=distro) 2021-05-25T11:50:17.335 DEBUG:teuthology.orchestra.run.smithi181:> rpm -q kernel --last | head -n 1 2021-05-25T11:50:17.450 INFO:teuthology.orchestra.run.smithi181.stdout:kernel-4.18.0-240.22.1.el8_3.x86_64 Tue 25 May 2021 11:49:52 AM UTC 2021-05-25T11:50:17.451 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-240.22.1.el8_3.x86_64 2021-05-25T11:50:17.452 DEBUG:teuthology.orchestra.run.smithi181:> sudo rpm -qi grub2-tools 2021-05-25T11:50:17.482 INFO:teuthology.orchestra.run.smithi181.stdout:Name : grub2-tools 2021-05-25T11:50:17.482 INFO:teuthology.orchestra.run.smithi181.stdout:Epoch : 1 2021-05-25T11:50:17.483 INFO:teuthology.orchestra.run.smithi181.stdout:Version : 2.02 2021-05-25T11:50:17.483 INFO:teuthology.orchestra.run.smithi181.stdout:Release : 90.el8 2021-05-25T11:50:17.483 INFO:teuthology.orchestra.run.smithi181.stdout:Architecture: x86_64 2021-05-25T11:50:17.483 INFO:teuthology.orchestra.run.smithi181.stdout:Install Date: Mon 01 Mar 2021 06:17:56 PM UTC 2021-05-25T11:50:17.484 INFO:teuthology.orchestra.run.smithi181.stdout:Group : System Environment/Base 2021-05-25T11:50:17.484 INFO:teuthology.orchestra.run.smithi181.stdout:Size : 9197207 2021-05-25T11:50:17.484 INFO:teuthology.orchestra.run.smithi181.stdout:License : GPLv3+ 2021-05-25T11:50:17.484 INFO:teuthology.orchestra.run.smithi181.stdout:Signature : RSA/SHA256, Tue 08 Sep 2020 07:14:39 PM UTC, Key ID 05b555b38483c65d 2021-05-25T11:50:17.485 INFO:teuthology.orchestra.run.smithi181.stdout:Source RPM : grub2-2.02-90.el8.src.rpm 2021-05-25T11:50:17.485 INFO:teuthology.orchestra.run.smithi181.stdout:Build Date : Tue 08 Sep 2020 07:00:29 PM UTC 2021-05-25T11:50:17.485 INFO:teuthology.orchestra.run.smithi181.stdout:Build Host : kbuilder.bsys.centos.org 2021-05-25T11:50:17.485 INFO:teuthology.orchestra.run.smithi181.stdout:Relocations : (not relocatable) 2021-05-25T11:50:17.485 INFO:teuthology.orchestra.run.smithi181.stdout:Packager : CentOS BuildSystem 2021-05-25T11:50:17.486 INFO:teuthology.orchestra.run.smithi181.stdout:Vendor : CentOS 2021-05-25T11:50:17.486 INFO:teuthology.orchestra.run.smithi181.stdout:URL : http://www.gnu.org/software/grub/ 2021-05-25T11:50:17.486 INFO:teuthology.orchestra.run.smithi181.stdout:Summary : Support tools for GRUB. 2021-05-25T11:50:17.486 INFO:teuthology.orchestra.run.smithi181.stdout:Description : 2021-05-25T11:50:17.487 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:50:17.487 INFO:teuthology.orchestra.run.smithi181.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2021-05-25T11:50:17.487 INFO:teuthology.orchestra.run.smithi181.stdout:customizable bootloader with modular architecture. It supports a rich 2021-05-25T11:50:17.487 INFO:teuthology.orchestra.run.smithi181.stdout:variety of kernel formats, file systems, computer architectures and 2021-05-25T11:50:17.488 INFO:teuthology.orchestra.run.smithi181.stdout:hardware devices. 2021-05-25T11:50:17.488 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:50:17.488 INFO:teuthology.orchestra.run.smithi181.stdout:This subpackage provides tools for support of all platforms. 2021-05-25T11:50:17.489 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2021-05-25T11:50:17.490 INFO:teuthology.task.kernel:Updating grub on smithi181 to boot 4.18.0-240.22.1.el8_3.x86_64 2021-05-25T11:50:17.490 DEBUG:teuthology.orchestra.run.smithi181:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2021-05-25T11:50:18.789 INFO:teuthology.orchestra.run.smithi181.stderr:Generating grub configuration file ... 2021-05-25T11:50:20.863 INFO:teuthology.orchestra.run.smithi181.stderr:done 2021-05-25T11:50:20.865 DEBUG:teuthology.orchestra.run.smithi181:> mktemp 2021-05-25T11:50:20.911 INFO:teuthology.orchestra.run.smithi181.stdout:/tmp/tmp.pqVwgT7MqL 2021-05-25T11:50:20.911 DEBUG:teuthology.orchestra.run.smithi181:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.pqVwgT7MqL 2021-05-25T11:50:20.951 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod 0666 /tmp/tmp.pqVwgT7MqL 2021-05-25T11:50:21.114 DEBUG:teuthology.orchestra.remote:smithi181:/tmp/tmp.pqVwgT7MqL is 6KB 2021-05-25T11:50:21.162 DEBUG:teuthology.orchestra.run.smithi181:> rm -fr /tmp/tmp.pqVwgT7MqL 2021-05-25T11:50:21.176 DEBUG:teuthology.orchestra.run.smithi181:> sudo /bin/ls /boot/loader/entries || true 2021-05-25T11:50:21.237 INFO:teuthology.orchestra.run.smithi181.stdout:631d84db90f7409fbdd390573ae89718-0-rescue.conf 2021-05-25T11:50:21.237 INFO:teuthology.orchestra.run.smithi181.stdout:631d84db90f7409fbdd390573ae89718-4.18.0-240.10.1.el8_3.x86_64.conf 2021-05-25T11:50:21.238 INFO:teuthology.orchestra.run.smithi181.stdout:631d84db90f7409fbdd390573ae89718-4.18.0-240.22.1.el8_3.x86_64.conf 2021-05-25T11:50:21.238 INFO:teuthology.orchestra.run.smithi181.stdout:631d84db90f7409fbdd390573ae89718-4.18.0-240.el8.x86_64.conf 2021-05-25T11:50:21.239 DEBUG:teuthology.orchestra.run.smithi181:> sudo grub2-set-default 631d84db90f7409fbdd390573ae89718-4.18.0-240.22.1.el8_3.x86_64 2021-05-25T11:50:21.395 DEBUG:teuthology.orchestra.run.smithi181:> sudo shutdown -r now 2021-05-25T11:50:51.427 INFO:teuthology.misc:Re-opening connections... 2021-05-25T11:50:51.428 INFO:teuthology.misc:trying to connect to ubuntu@smithi102.front.sepia.ceph.com 2021-05-25T11:50:51.428 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-25T11:50:51.429 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-05-25T11:51:09.771 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.102 2021-05-25T11:51:09.772 INFO:teuthology.misc:trying to connect to ubuntu@smithi181.front.sepia.ceph.com 2021-05-25T11:51:09.772 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-25T11:51:09.773 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-05-25T11:51:44.331 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.181 2021-05-25T11:51:44.331 DEBUG:teuthology.misc:waited 52.903913497924805 2021-05-25T11:51:45.333 INFO:teuthology.misc:trying to connect to ubuntu@smithi102.front.sepia.ceph.com 2021-05-25T11:51:45.334 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-25T11:51:45.334 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-05-25T11:52:01.065 DEBUG:teuthology.orchestra.run.smithi102:> true 2021-05-25T11:52:01.404 DEBUG:teuthology.misc:waited 69.97635054588318 2021-05-25T11:52:02.405 INFO:teuthology.misc:trying to connect to ubuntu@smithi181.front.sepia.ceph.com 2021-05-25T11:52:02.406 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-25T11:52:02.406 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-05-25T11:52:02.709 DEBUG:teuthology.orchestra.run.smithi181:> true 2021-05-25T11:52:03.050 DEBUG:teuthology.misc:waited 71.62284278869629 2021-05-25T11:52:04.051 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2021-05-25T11:52:04.052 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-240.22.1.el8_3.x86_64"... 2021-05-25T11:52:04.053 DEBUG:teuthology.orchestra.run.smithi102:> uname -r 2021-05-25T11:52:04.069 INFO:teuthology.orchestra.run.smithi102.stdout:4.18.0-240.22.1.el8_3.x86_64 2021-05-25T11:52:04.070 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-240.22.1.el8_3.x86_64 vs 4.18.0-240.22.1.el8_3.x86_64 2021-05-25T11:52:04.071 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2021-05-25T11:52:04.071 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2021-05-25T11:52:04.072 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2021-05-25T11:52:04.072 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "4.18.0-240.22.1.el8_3.x86_64"... 2021-05-25T11:52:04.073 DEBUG:teuthology.orchestra.run.smithi181:> uname -r 2021-05-25T11:52:04.088 INFO:teuthology.orchestra.run.smithi181.stdout:4.18.0-240.22.1.el8_3.x86_64 2021-05-25T11:52:04.089 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-240.22.1.el8_3.x86_64 vs 4.18.0-240.22.1.el8_3.x86_64 2021-05-25T11:52:04.089 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2021-05-25T11:52:04.090 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2021-05-25T11:52:05.092 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2021-05-25T11:52:05.092 DEBUG:teuthology.orchestra.run.smithi102:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-05-25T11:52:05.315 INFO:teuthology.orchestra.run.smithi102.stdout:ttyS1 2021-05-25T11:52:05.331 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2021-05-25T11:52:05.331 DEBUG:teuthology.orchestra.run.smithi181:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-05-25T11:52:05.447 INFO:teuthology.orchestra.run.smithi181.stdout:ttyS1 2021-05-25T11:52:05.463 INFO:teuthology.run_tasks:Running task internal.base... 2021-05-25T11:52:05.475 INFO:teuthology.task.internal:Creating test directory... 2021-05-25T11:52:05.475 DEBUG:teuthology.orchestra.run.smithi102:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-05-25T11:52:05.492 DEBUG:teuthology.orchestra.run.smithi181:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-05-25T11:52:05.509 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2021-05-25T11:52:05.521 INFO:teuthology.run_tasks:Running task internal.archive... 2021-05-25T11:52:05.561 INFO:teuthology.task.internal:Creating archive directory... 2021-05-25T11:52:05.562 DEBUG:teuthology.orchestra.run.smithi102:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-05-25T11:52:05.584 DEBUG:teuthology.orchestra.run.smithi181:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-05-25T11:52:05.650 INFO:teuthology.run_tasks:Running task internal.coredump... 2021-05-25T11:52:05.662 INFO:teuthology.task.internal:Enabling coredump saving... 2021-05-25T11:52:05.663 DEBUG:teuthology.orchestra.run.smithi102:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-05-25T11:52:05.706 DEBUG:teuthology.orchestra.run.smithi181:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-05-25T11:52:05.733 INFO:teuthology.orchestra.run.smithi181.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-25T11:52:05.743 INFO:teuthology.orchestra.run.smithi181.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-25T11:52:05.768 INFO:teuthology.orchestra.run.smithi102.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-25T11:52:05.784 INFO:teuthology.orchestra.run.smithi102.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-25T11:52:05.786 INFO:teuthology.run_tasks:Running task internal.sudo... 2021-05-25T11:52:05.798 INFO:teuthology.task.internal:Configuring sudo... 2021-05-25T11:52:05.798 DEBUG:teuthology.orchestra.run.smithi102:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-05-25T11:52:05.805 DEBUG:teuthology.orchestra.run.smithi181:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-05-25T11:52:05.881 INFO:teuthology.run_tasks:Running task internal.syslog... 2021-05-25T11:52:05.895 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2021-05-25T11:52:05.895 DEBUG:teuthology.orchestra.run.smithi102:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-05-25T11:52:05.919 DEBUG:teuthology.orchestra.run.smithi181:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-05-25T11:52:06.024 DEBUG:teuthology.orchestra.run.smithi102:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-05-25T11:52:06.072 DEBUG:teuthology.orchestra.run.smithi102:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2021-05-25T11:52:06.300 DEBUG:teuthology.orchestra.run.smithi102:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-05-25T11:52:06.373 DEBUG:teuthology.orchestra.run.smithi102:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2021-05-25T11:52:06.627 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2021-05-25T11:52:06.627 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-05-25T11:52:06.847 DEBUG:teuthology.orchestra.run.smithi181:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-05-25T11:52:06.962 DEBUG:teuthology.orchestra.run.smithi181:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2021-05-25T11:52:07.070 DEBUG:teuthology.orchestra.run.smithi181:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-05-25T11:52:07.142 DEBUG:teuthology.orchestra.run.smithi181:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2021-05-25T11:52:07.398 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2021-05-25T11:52:07.399 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-05-25T11:52:07.466 DEBUG:teuthology.orchestra.run.smithi102:> sudo service rsyslog restart 2021-05-25T11:52:07.468 DEBUG:teuthology.orchestra.run.smithi181:> sudo service rsyslog restart 2021-05-25T11:52:07.661 INFO:teuthology.orchestra.run.smithi181.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2021-05-25T11:52:07.780 INFO:teuthology.orchestra.run.smithi102.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2021-05-25T11:52:07.798 INFO:teuthology.run_tasks:Running task internal.timer... 2021-05-25T11:52:07.809 INFO:teuthology.task.internal:Starting timer... 2021-05-25T11:52:07.810 INFO:teuthology.run_tasks:Running task pcp... 2021-05-25T11:52:07.872 INFO:teuthology.run_tasks:Running task selinux... 2021-05-25T11:52:07.920 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2021-05-25T11:52:07.921 DEBUG:teuthology.orchestra.run.smithi102:> sudo service auditd rotate 2021-05-25T11:52:08.099 INFO:teuthology.orchestra.run.smithi102.stdout:Rotating logs: [ OK ] 2021-05-25T11:52:08.101 DEBUG:teuthology.orchestra.run.smithi181:> sudo service auditd rotate 2021-05-25T11:52:08.377 INFO:teuthology.orchestra.run.smithi181.stdout:Rotating logs: [ OK ] 2021-05-25T11:52:08.379 DEBUG:teuthology.task.selinux:Getting current SELinux state 2021-05-25T11:52:08.379 DEBUG:teuthology.orchestra.run.smithi102:> /usr/sbin/getenforce 2021-05-25T11:52:08.488 INFO:teuthology.orchestra.run.smithi102.stdout:Permissive 2021-05-25T11:52:08.488 DEBUG:teuthology.orchestra.run.smithi181:> /usr/sbin/getenforce 2021-05-25T11:52:08.629 INFO:teuthology.orchestra.run.smithi181.stdout:Permissive 2021-05-25T11:52:08.630 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi102.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi181.front.sepia.ceph.com': 'permissive'} 2021-05-25T11:52:08.631 DEBUG:teuthology.orchestra.run.smithi102:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|scontext=system_u:system_r:logrotate_t:s0\)' 2021-05-25T11:52:08.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-05-25T11:52:08.658 DEBUG:teuthology.orchestra.run.smithi181:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|scontext=system_u:system_r:logrotate_t:s0\)' 2021-05-25T11:52:08.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-05-25T11:52:08.693 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2021-05-25T11:52:08.694 DEBUG:teuthology.orchestra.run.smithi102:> sudo /usr/sbin/setenforce permissive 2021-05-25T11:52:08.722 DEBUG:teuthology.orchestra.run.smithi181:> sudo /usr/sbin/setenforce permissive 2021-05-25T11:52:08.784 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2021-05-25T11:52:08.796 DEBUG:teuthology.repo_utils:Setting repo remote to git://git.ceph.com/git/ceph-cm-ansible.git 2021-05-25T11:52:08.803 INFO:teuthology.repo_utils:Fetching master from origin 2021-05-25T11:52:08.834 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to origin/master 2021-05-25T11:52:08.850 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2021-05-25T11:52:08.852 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi102.front.sepia.ceph.com,smithi181.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2021-05-25T11:52:09.182 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2021-05-25T11:52:09.952 INFO:teuthology.task.ansible.out:[WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2021-05-25T11:52:12.668 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2021-05-25T11:52:12.689 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:52:12 +0000 (0:00:00.037) 0:00:00.037 *********** 2021-05-25T11:52:12.700 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:52:12 +0000 (0:00:00.010) 0:00:00.047 *********** 2021-05-25T11:52:14.131 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* 2021-05-25T11:52:14.133 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] 2021-05-25T11:52:14.161 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] 2021-05-25T11:52:14.188 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:52:14 +0000 (0:00:01.487) 0:00:01.536 *********** 2021-05-25T11:52:14.227 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:52:14 +0000 (0:00:00.039) 0:00:01.575 *********** 2021-05-25T11:52:14.606 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** 2021-05-25T11:52:14.608 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} ok: [smithi181.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-05-25T11:52:14.628 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:52:14 +0000 (0:00:00.400) 0:00:01.976 *********** 2021-05-25T11:52:14.646 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:52:14 +0000 (0:00:00.018) 0:00:01.994 *********** 2021-05-25T11:52:15.222 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2021-05-25T11:52:15.223 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} ok: [smithi181.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2021-05-25T11:52:15.245 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:52:15 +0000 (0:00:00.599) 0:00:02.594 *********** 2021-05-25T11:52:15.264 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:52:15 +0000 (0:00:00.017) 0:00:02.612 *********** 2021-05-25T11:52:15.311 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2021-05-25T11:52:15.313 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:52:15.333 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:52:15 +0000 (0:00:00.070) 0:00:02.682 *********** 2021-05-25T11:52:15.340 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:52:15.363 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:52:15 +0000 (0:00:00.029) 0:00:02.711 *********** 2021-05-25T11:52:15.777 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** 2021-05-25T11:52:15.779 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} ok: [smithi181.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-05-25T11:52:15.800 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:52:15 +0000 (0:00:00.436) 0:00:03.148 *********** 2021-05-25T11:52:15.819 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:52:15 +0000 (0:00:00.019) 0:00:03.167 *********** 2021-05-25T11:52:16.892 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** 2021-05-25T11:52:16.894 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1621943535.8415625-29838-158860481000550/source", "state": "file", "uid": 0} changed: [smithi181.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1621943535.8599603-29841-85464444012052/source", "state": "file", "uid": 0} 2021-05-25T11:52:16.914 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:52:16 +0000 (0:00:01.095) 0:00:04.262 *********** 2021-05-25T11:52:16.933 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:52:16 +0000 (0:00:00.018) 0:00:04.281 *********** 2021-05-25T11:52:17.480 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* 2021-05-25T11:52:17.481 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2021-05-25T11:52:17.483 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail 2021-05-25T11:52:17.485 INFO:teuthology.task.ansible.out:.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@ 2021-05-25T11:52:17.486 INFO:teuthology.task.ansible.out:aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2021-05-25T11:52:17.564 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** skipping: no hosts matched 2021-05-25T11:52:17.566 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2021-05-25T11:52:17.673 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* Tuesday 25 May 2021 11:52:17 +0000 (0:00:00.740) 0:00:05.021 *********** 2021-05-25T11:52:18.234 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] 2021-05-25T11:52:19.255 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] 2021-05-25T11:52:19.272 INFO:teuthology.task.ansible.out: TASK [set_fact] **************************************************************** Tuesday 25 May 2021 11:52:19 +0000 (0:00:01.598) 0:00:06.620 *********** 2021-05-25T11:52:19.297 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"ansible_facts": {"ran_from_cephlab_playbook": true}, "changed": false} 2021-05-25T11:52:19.308 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => {"ansible_facts": {"ran_from_cephlab_playbook": true}, "changed": false} 2021-05-25T11:52:19.312 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2021-05-25T11:52:19.466 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:52:19 +0000 (0:00:00.193) 0:00:06.814 *********** 2021-05-25T11:52:19.485 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:52:19 +0000 (0:00:00.018) 0:00:06.833 *********** 2021-05-25T11:52:20.103 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi181.front.sepia.ceph.com] 2021-05-25T11:52:20.156 INFO:teuthology.task.ansible.out:[WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. 2021-05-25T11:52:20.158 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:52:20 +0000 (0:00:00.671) 0:00:07.504 *********** 2021-05-25T11:52:20.661 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** 2021-05-25T11:52:20.663 INFO:teuthology.task.ansible.out:changed: [smithi181.front.sepia.ceph.com] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.156803", "end": "2021-05-25 11:52:20.638369", "rc": 0, "start": "2021-05-25 11:52:20.481566", "stderr": "", "stderr_lines": [], "stdout": "b2a91f5f527dcc382f8ed49697328a42af627ee8", "stdout_lines": ["b2a91f5f527dcc382f8ed49697328a42af627ee8"]} 2021-05-25T11:52:20.683 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:52:20 +0000 (0:00:00.527) 0:00:08.031 *********** 2021-05-25T11:52:20.875 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2021-05-25T11:52:20.876 INFO:teuthology.task.ansible.out:changed: [smithi181.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.010628", "end": "2021-05-25 11:52:20.850838", "failed_when_result": false, "rc": 0, "start": "2021-05-25 11:52:20.840210", "stderr": "", "stderr_lines": [], "stdout": "df8946f08e870e2d2565b79099289ca48bd825f3", "stdout_lines": ["df8946f08e870e2d2565b79099289ca48bd825f3"]} 2021-05-25T11:52:20.897 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:52:20 +0000 (0:00:00.213) 0:00:08.245 *********** 2021-05-25T11:52:20.965 INFO:teuthology.task.ansible.out: TASK [Determine if we can skip users and pubkeys updates] ********************** 2021-05-25T11:52:20.967 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:52:20.987 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:52:20 +0000 (0:00:00.090) 0:00:08.335 *********** 2021-05-25T11:52:21.055 INFO:teuthology.task.ansible.out: TASK [Merge extra_admin_users into managed_admin_users] ************************ 2021-05-25T11:52:21.057 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:52:21.077 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:52:21 +0000 (0:00:00.089) 0:00:08.425 *********** 2021-05-25T11:52:21.088 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi102.front.sepia.ceph.com] 2021-05-25T11:52:21.140 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:52:21 +0000 (0:00:00.062) 0:00:08.488 *********** 2021-05-25T11:52:21.146 INFO:teuthology.task.ansible.out: TASK [Remove managed_admin_users from managed_users] *************************** 2021-05-25T11:52:21.147 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:52:21.170 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:52:21 +0000 (0:00:00.030) 0:00:08.518 *********** 2021-05-25T11:52:21.250 INFO:teuthology.task.ansible.out: TASK [Filter the managed_users list] ******************************************* 2021-05-25T11:52:21.252 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:52:21.273 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:52:21 +0000 (0:00:00.102) 0:00:08.621 *********** 2021-05-25T11:52:21.341 INFO:teuthology.task.ansible.out: TASK [Filter the managed_admin_users list] ************************************* 2021-05-25T11:52:21.343 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:52:21.366 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:52:21 +0000 (0:00:00.092) 0:00:08.713 *********** 2021-05-25T11:52:21.527 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** 2021-05-25T11:52:21.529 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.160501", "end": "2021-05-25 11:52:21.505503", "rc": 0, "start": "2021-05-25 11:52:21.345002", "stderr": "", "stderr_lines": [], "stdout": "b2a91f5f527dcc382f8ed49697328a42af627ee8", "stdout_lines": ["b2a91f5f527dcc382f8ed49697328a42af627ee8"]} 2021-05-25T11:52:21.551 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:52:21 +0000 (0:00:00.185) 0:00:08.899 *********** 2021-05-25T11:52:21.600 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2021-05-25T11:52:21.602 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-05-25T11:52:21.625 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:52:21 +0000 (0:00:00.073) 0:00:08.973 *********** 2021-05-25T11:52:21.747 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2021-05-25T11:52:21.749 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.010074", "end": "2021-05-25 11:52:21.723164", "failed_when_result": false, "rc": 0, "start": "2021-05-25 11:52:21.713090", "stderr": "", "stderr_lines": [], "stdout": "df8946f08e870e2d2565b79099289ca48bd825f3", "stdout_lines": ["df8946f08e870e2d2565b79099289ca48bd825f3"]} 2021-05-25T11:52:21.770 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:52:21 +0000 (0:00:00.145) 0:00:09.118 *********** 2021-05-25T11:52:21.839 INFO:teuthology.task.ansible.out: TASK [Determine if we can skip users and pubkeys updates] ********************** 2021-05-25T11:52:21.840 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:52:21.861 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:52:21 +0000 (0:00:00.090) 0:00:09.209 *********** 2021-05-25T11:52:21.908 INFO:teuthology.task.ansible.out: TASK [Create all admin users with sudo access.] ******************************** 2021-05-25T11:52:21.910 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2021-05-25T11:52:21.920 INFO:teuthology.task.ansible.out: TASK [Merge extra_admin_users into managed_admin_users] ************************ 2021-05-25T11:52:21.922 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:52:21.942 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:52:21 +0000 (0:00:00.081) 0:00:09.290 *********** 2021-05-25T11:52:22.032 INFO:teuthology.task.ansible.out: TASK [Remove managed_admin_users from managed_users] *************************** 2021-05-25T11:52:22.034 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:52:22.054 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:52:22 +0000 (0:00:00.111) 0:00:09.402 *********** 2021-05-25T11:52:22.101 INFO:teuthology.task.ansible.out: TASK [Create all admin users with sudo access.] ******************************** 2021-05-25T11:52:22.103 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2021-05-25T11:52:22.124 INFO:teuthology.task.ansible.out: TASK [Filter the managed_users list] ******************************************* 2021-05-25T11:52:22.125 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:52:22.146 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:52:22 +0000 (0:00:00.091) 0:00:09.494 *********** 2021-05-25T11:52:22.214 INFO:teuthology.task.ansible.out: TASK [Filter the managed_admin_users list] ************************************* 2021-05-25T11:52:22.217 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:52:22.236 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:52:22 +0000 (0:00:00.090) 0:00:09.584 *********** 2021-05-25T11:52:22.294 INFO:teuthology.task.ansible.out: TASK [Create all admin users with sudo access.] ******************************** 2021-05-25T11:52:22.296 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2021-05-25T11:52:22.461 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2021-05-25T11:52:22.462 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-05-25T11:52:22.486 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:52:22 +0000 (0:00:00.249) 0:00:09.834 *********** 2021-05-25T11:52:22.493 INFO:teuthology.task.ansible.out: TASK [Create all admin users with sudo access.] ******************************** 2021-05-25T11:52:22.495 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2021-05-25T11:52:22.674 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2021-05-25T11:52:22.758 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2021-05-25T11:52:22.760 INFO:teuthology.task.ansible.out: 2021-05-25T11:52:22.864 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2021-05-25T11:52:22.959 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2021-05-25T11:52:23.063 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2021-05-25T11:52:23.147 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2021-05-25T11:52:23.251 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2021-05-25T11:52:23.339 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2021-05-25T11:52:23.444 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2021-05-25T11:52:23.538 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2021-05-25T11:52:23.642 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2021-05-25T11:52:23.727 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2021-05-25T11:52:23.831 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2021-05-25T11:52:23.925 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2021-05-25T11:52:24.020 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2021-05-25T11:52:24.115 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2021-05-25T11:52:24.209 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2021-05-25T11:52:24.304 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2021-05-25T11:52:24.399 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2021-05-25T11:52:24.493 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2021-05-25T11:52:24.597 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2021-05-25T11:52:24.691 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2021-05-25T11:52:24.786 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2021-05-25T11:52:24.881 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2021-05-25T11:52:24.976 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2021-05-25T11:52:25.081 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2021-05-25T11:52:25.175 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1118} 2021-05-25T11:52:25.270 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2021-05-25T11:52:25.364 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2021-05-25T11:52:25.459 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2021-05-25T11:52:25.555 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2021-05-25T11:52:25.653 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2021-05-25T11:52:25.748 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1121} 2021-05-25T11:52:25.853 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2021-05-25T11:52:25.937 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1122} 2021-05-25T11:52:26.042 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1118} 2021-05-25T11:52:26.136 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1123} 2021-05-25T11:52:26.230 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2021-05-25T11:52:26.346 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1124} 2021-05-25T11:52:26.430 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2021-05-25T11:52:26.535 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1125} 2021-05-25T11:52:26.620 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1121} 2021-05-25T11:52:26.734 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1126} 2021-05-25T11:52:26.808 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1122} 2021-05-25T11:52:26.924 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1127} 2021-05-25T11:52:26.998 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1123} 2021-05-25T11:52:27.112 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1128} 2021-05-25T11:52:27.218 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1124} 2021-05-25T11:52:27.312 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1129} 2021-05-25T11:52:27.406 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1125} 2021-05-25T11:52:27.500 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1130} 2021-05-25T11:52:27.595 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1126} 2021-05-25T11:52:27.700 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1131} 2021-05-25T11:52:27.786 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1127} 2021-05-25T11:52:27.890 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1132} 2021-05-25T11:52:27.974 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1128} 2021-05-25T11:52:28.079 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1133} 2021-05-25T11:52:28.163 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1129} 2021-05-25T11:52:28.278 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1134} 2021-05-25T11:52:28.352 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1130} 2021-05-25T11:52:28.468 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1135} 2021-05-25T11:52:28.544 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1131} 2021-05-25T11:52:28.659 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1136} 2021-05-25T11:52:28.743 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1132} 2021-05-25T11:52:28.862 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-05-25T11:52:28.937 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1133} 2021-05-25T11:52:29.052 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-05-25T11:52:29.125 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1134} 2021-05-25T11:52:29.251 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-05-25T11:52:29.325 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1135} 2021-05-25T11:52:29.450 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1141} 2021-05-25T11:52:29.514 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1136} 2021-05-25T11:52:29.640 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1142} 2021-05-25T11:52:29.703 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-05-25T11:52:29.841 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1143} 2021-05-25T11:52:29.905 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-05-25T11:52:30.040 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1144} 2021-05-25T11:52:30.104 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-05-25T11:52:30.229 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1145} 2021-05-25T11:52:30.293 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1141} 2021-05-25T11:52:30.428 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-05-25T11:52:30.492 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1142} 2021-05-25T11:52:30.618 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-05-25T11:52:30.681 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1143} 2021-05-25T11:52:30.807 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-05-25T11:52:30.871 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1144} 2021-05-25T11:52:31.007 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1149} 2021-05-25T11:52:31.071 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1145} 2021-05-25T11:52:31.197 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1150} 2021-05-25T11:52:31.260 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-05-25T11:52:31.395 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1151} 2021-05-25T11:52:31.458 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-05-25T11:52:31.583 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1152} 2021-05-25T11:52:31.646 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-05-25T11:52:31.773 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1153} 2021-05-25T11:52:31.846 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1149} 2021-05-25T11:52:31.960 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1154} 2021-05-25T11:52:32.034 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1150} 2021-05-25T11:52:32.160 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1155} 2021-05-25T11:52:32.223 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1151} 2021-05-25T11:52:32.349 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1156} 2021-05-25T11:52:32.422 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1152} 2021-05-25T11:52:32.538 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1157} 2021-05-25T11:52:32.611 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1153} 2021-05-25T11:52:32.726 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1158} 2021-05-25T11:52:32.800 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1154} 2021-05-25T11:52:32.925 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1159} 2021-05-25T11:52:32.998 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1155} 2021-05-25T11:52:33.114 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1160} 2021-05-25T11:52:33.188 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1156} 2021-05-25T11:52:33.313 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1161} 2021-05-25T11:52:33.376 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1157} 2021-05-25T11:52:33.502 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1162} 2021-05-25T11:52:33.576 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1158} 2021-05-25T11:52:33.691 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1163} 2021-05-25T11:52:33.765 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1159} 2021-05-25T11:52:33.890 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-05-25T11:52:33.953 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1160} 2021-05-25T11:52:34.079 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1164} 2021-05-25T11:52:34.153 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1161} 2021-05-25T11:52:34.268 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1165} 2021-05-25T11:52:34.343 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1162} 2021-05-25T11:52:34.468 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1166} 2021-05-25T11:52:34.532 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1163} 2021-05-25T11:52:34.657 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1167} 2021-05-25T11:52:34.722 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-05-25T11:52:34.849 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1168} 2021-05-25T11:52:34.912 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1164} 2021-05-25T11:52:35.050 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1169} 2021-05-25T11:52:35.114 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1165} 2021-05-25T11:52:35.240 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1170} 2021-05-25T11:52:35.303 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1166} 2021-05-25T11:52:35.429 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1171} 2021-05-25T11:52:35.492 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1167} 2021-05-25T11:52:35.628 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1172} 2021-05-25T11:52:35.681 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1168} 2021-05-25T11:52:35.817 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1173} 2021-05-25T11:52:35.880 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1169} 2021-05-25T11:52:36.006 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1174} 2021-05-25T11:52:36.070 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1170} 2021-05-25T11:52:36.196 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1175} 2021-05-25T11:52:36.259 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1171} 2021-05-25T11:52:36.395 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1176} 2021-05-25T11:52:36.447 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1172} 2021-05-25T11:52:36.582 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1177} 2021-05-25T11:52:36.646 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1173} 2021-05-25T11:52:36.772 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1178} 2021-05-25T11:52:36.835 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1174} 2021-05-25T11:52:36.961 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1179} 2021-05-25T11:52:37.035 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1175} 2021-05-25T11:52:37.161 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1180} 2021-05-25T11:52:37.224 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1176} 2021-05-25T11:52:37.349 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1181} 2021-05-25T11:52:37.412 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1177} 2021-05-25T11:52:37.538 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1182} 2021-05-25T11:52:37.611 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1178} 2021-05-25T11:52:37.736 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1183} 2021-05-25T11:52:37.800 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1179} 2021-05-25T11:52:37.925 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1184} 2021-05-25T11:52:37.998 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1180} 2021-05-25T11:52:38.123 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1185} 2021-05-25T11:52:38.188 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1181} 2021-05-25T11:52:38.313 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1186} 2021-05-25T11:52:38.376 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1182} 2021-05-25T11:52:38.511 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1187} 2021-05-25T11:52:38.574 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1183} 2021-05-25T11:52:38.710 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1188} 2021-05-25T11:52:38.763 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1184} 2021-05-25T11:52:38.902 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1189} 2021-05-25T11:52:38.965 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1185} 2021-05-25T11:52:39.091 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1190} 2021-05-25T11:52:39.155 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1186} 2021-05-25T11:52:39.291 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1191} 2021-05-25T11:52:39.355 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1187} 2021-05-25T11:52:39.480 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1192} 2021-05-25T11:52:39.545 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1188} 2021-05-25T11:52:39.670 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1193} 2021-05-25T11:52:39.744 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1189} 2021-05-25T11:52:39.872 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1194} 2021-05-25T11:52:39.935 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1190} 2021-05-25T11:52:40.060 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1195} 2021-05-25T11:52:40.133 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1191} 2021-05-25T11:52:40.248 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1196} 2021-05-25T11:52:40.323 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1192} 2021-05-25T11:52:40.449 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1197} 2021-05-25T11:52:40.522 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1193} 2021-05-25T11:52:40.637 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1198} 2021-05-25T11:52:40.710 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1194} 2021-05-25T11:52:40.825 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1199} 2021-05-25T11:52:40.909 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1195} 2021-05-25T11:52:41.014 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1200} 2021-05-25T11:52:41.098 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1196} 2021-05-25T11:52:41.212 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-05-25T11:52:41.286 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1197} 2021-05-25T11:52:41.401 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1201} 2021-05-25T11:52:41.484 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1198} 2021-05-25T11:52:41.589 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1202} 2021-05-25T11:52:41.673 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1199} 2021-05-25T11:52:41.791 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1203} 2021-05-25T11:52:41.866 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1200} 2021-05-25T11:52:41.981 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1204} 2021-05-25T11:52:42.065 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-05-25T11:52:42.170 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1205} 2021-05-25T11:52:42.254 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1201} 2021-05-25T11:52:42.358 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1206} 2021-05-25T11:52:42.442 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1202} 2021-05-25T11:52:42.558 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1207} 2021-05-25T11:52:42.642 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1203} 2021-05-25T11:52:42.748 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1208} 2021-05-25T11:52:42.832 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1204} 2021-05-25T11:52:42.947 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1209} 2021-05-25T11:52:43.022 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1205} 2021-05-25T11:52:43.137 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1210} 2021-05-25T11:52:43.220 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1206} 2021-05-25T11:52:43.325 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1211} 2021-05-25T11:52:43.408 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1207} 2021-05-25T11:52:43.523 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1212} 2021-05-25T11:52:43.596 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1208} 2021-05-25T11:52:43.712 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1213} 2021-05-25T11:52:43.796 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1209} 2021-05-25T11:52:43.900 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1214} 2021-05-25T11:52:43.984 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1210} 2021-05-25T11:52:44.099 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1215} 2021-05-25T11:52:44.183 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1211} 2021-05-25T11:52:44.287 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1216} 2021-05-25T11:52:44.371 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1212} 2021-05-25T11:52:44.475 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1217} 2021-05-25T11:52:44.569 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1213} 2021-05-25T11:52:44.674 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1218} 2021-05-25T11:52:44.758 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1214} 2021-05-25T11:52:44.873 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1219} 2021-05-25T11:52:44.946 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1215} 2021-05-25T11:52:45.061 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1220} 2021-05-25T11:52:45.134 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1216} 2021-05-25T11:52:45.249 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1221} 2021-05-25T11:52:45.322 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1217} 2021-05-25T11:52:45.437 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1222} 2021-05-25T11:52:45.521 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1218} 2021-05-25T11:52:45.636 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1223} 2021-05-25T11:52:45.710 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1219} 2021-05-25T11:52:45.825 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1224} 2021-05-25T11:52:45.903 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1220} 2021-05-25T11:52:46.017 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1225} 2021-05-25T11:52:46.091 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1221} 2021-05-25T11:52:46.206 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1226} 2021-05-25T11:52:46.279 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1222} 2021-05-25T11:52:46.406 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1227} 2021-05-25T11:52:46.479 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1223} 2021-05-25T11:52:46.596 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1228} 2021-05-25T11:52:46.670 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1224} 2021-05-25T11:52:46.786 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/jsalomon", "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "move_home": false, "name": "jsalomon", "shell": "/bin/bash", "state": "present", "uid": 1229} 2021-05-25T11:52:46.859 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1225} 2021-05-25T11:52:46.975 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1230, "groups": "sudo", "home": "/home/jolmomar", "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "move_home": false, "name": "jolmomar", "shell": "/bin/bash", "state": "present", "uid": 1230} 2021-05-25T11:52:47.049 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1226} 2021-05-25T11:52:47.174 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1231, "groups": "sudo", "home": "/home/hmunjulu", "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "move_home": false, "name": "hmunjulu", "shell": "/bin/bash", "state": "present", "uid": 1231} 2021-05-25T11:52:47.238 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1227} 2021-05-25T11:52:47.362 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1232, "groups": "sudo", "home": "/home/julpark", "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "move_home": false, "name": "julpark", "shell": "/bin/bash", "state": "present", "uid": 1232} 2021-05-25T11:52:47.436 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1228} 2021-05-25T11:52:47.551 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1233, "groups": "sudo", "home": "/home/pdhange", "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "move_home": false, "name": "pdhange", "shell": "/bin/bash", "state": "present", "uid": 1233} 2021-05-25T11:52:47.634 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/jsalomon", "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "move_home": false, "name": "jsalomon", "shell": "/bin/bash", "state": "present", "uid": 1229} 2021-05-25T11:52:47.749 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1234, "groups": "sudo", "home": "/home/dpivonka", "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "move_home": false, "name": "dpivonka", "shell": "/bin/bash", "state": "present", "uid": 1234} 2021-05-25T11:52:47.832 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1230, "groups": "sudo", "home": "/home/jolmomar", "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "move_home": false, "name": "jolmomar", "shell": "/bin/bash", "state": "present", "uid": 1230} 2021-05-25T11:52:47.937 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1235, "groups": "sudo", "home": "/home/adking", "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "move_home": false, "name": "adking", "shell": "/bin/bash", "state": "present", "uid": 1235} 2021-05-25T11:52:48.020 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1231, "groups": "sudo", "home": "/home/hmunjulu", "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "move_home": false, "name": "hmunjulu", "shell": "/bin/bash", "state": "present", "uid": 1231} 2021-05-25T11:52:48.135 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1236, "groups": "sudo", "home": "/home/amathuri", "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "move_home": false, "name": "amathuri", "shell": "/bin/bash", "state": "present", "uid": 1236} 2021-05-25T11:52:48.209 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1232, "groups": "sudo", "home": "/home/julpark", "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "move_home": false, "name": "julpark", "shell": "/bin/bash", "state": "present", "uid": 1232} 2021-05-25T11:52:48.323 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1237, "groups": "sudo", "home": "/home/nizamial09", "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "move_home": false, "name": "nizamial09", "shell": "/bin/bash", "state": "present", "uid": 1237} 2021-05-25T11:52:48.408 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1233, "groups": "sudo", "home": "/home/pdhange", "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "move_home": false, "name": "pdhange", "shell": "/bin/bash", "state": "present", "uid": 1233} 2021-05-25T11:52:48.595 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1234, "groups": "sudo", "home": "/home/dpivonka", "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "move_home": false, "name": "dpivonka", "shell": "/bin/bash", "state": "present", "uid": 1234} 2021-05-25T11:52:48.796 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1235, "groups": "sudo", "home": "/home/adking", "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "move_home": false, "name": "adking", "shell": "/bin/bash", "state": "present", "uid": 1235} 2021-05-25T11:52:48.984 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1236, "groups": "sudo", "home": "/home/amathuri", "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "move_home": false, "name": "amathuri", "shell": "/bin/bash", "state": "present", "uid": 1236} 2021-05-25T11:52:49.181 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1237, "groups": "sudo", "home": "/home/nizamial09", "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "move_home": false, "name": "nizamial09", "shell": "/bin/bash", "state": "present", "uid": 1237} 2021-05-25T11:52:49.945 INFO:teuthology.task.ansible.out:changed: [smithi181.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10102, "groups": "sudo", "home": "/home/dachary", "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "name": "dachary", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2021-05-25T11:52:50.691 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10102, "groups": "sudo", "home": "/home/dachary", "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "name": "dachary", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2021-05-25T11:52:51.612 INFO:teuthology.task.ansible.out:changed: [smithi181.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10103, "groups": "sudo", "home": "/home/amctagga", "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "name": "amctagga", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10103} 2021-05-25T11:52:52.326 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10103, "groups": "sudo", "home": "/home/amctagga", "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "name": "amctagga", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10103} 2021-05-25T11:52:53.165 INFO:teuthology.task.ansible.out:changed: [smithi181.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10104, "groups": "sudo", "home": "/home/kevinz", "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "name": "kevinz", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10104} 2021-05-25T11:52:53.880 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10104, "groups": "sudo", "home": "/home/kevinz", "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "name": "kevinz", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10104} 2021-05-25T11:52:54.617 INFO:teuthology.task.ansible.out:changed: [smithi181.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10105, "groups": "sudo", "home": "/home/liu-chunmei", "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "name": "liu-chunmei", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10105} 2021-05-25T11:52:55.351 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10105, "groups": "sudo", "home": "/home/liu-chunmei", "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "name": "liu-chunmei", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10105} 2021-05-25T11:52:56.136 INFO:teuthology.task.ansible.out:changed: [smithi181.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10106, "groups": "sudo", "home": "/home/cyx1231st", "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "name": "cyx1231st", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10106} 2021-05-25T11:52:56.263 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:52:56 +0000 (0:00:33.776) 0:00:43.611 *********** 2021-05-25T11:52:56.333 INFO:teuthology.task.ansible.out: TASK [Create all users without sudo access.] *********************************** 2021-05-25T11:52:56.353 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:52:56 +0000 (0:00:00.090) 0:00:43.701 *********** 2021-05-25T11:52:56.434 INFO:teuthology.task.ansible.out: TASK [Merge managed_users and managed_admin users] ***************************** 2021-05-25T11:52:56.435 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratza@fedora 2021-05-25T11:52:56.437 INFO:teuthology.task.ansible.out:rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265 2021-05-25T11:52:56.439 INFO:teuthology.task.ansible.out:c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop K 2021-05-25T11:52:56.440 INFO:teuthology.task.ansible.out:D0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9 2021-05-25T11:52:56.442 INFO:teuthology.task.ansible.out:df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRah 2021-05-25T11:52:56.443 INFO:teuthology.task.ansible.out:w fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de 2021-05-25T11:52:56.444 INFO:teuthology.task.ansible.out:53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d 2021-05-25T11:52:56.446 INFO:teuthology.task.ansible.out:7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumra 2021-05-25T11:52:56.447 INFO:teuthology.task.ansible.out:o", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b76 2021-05-25T11:52:56.449 INFO:teuthology.task.ansible.out:8c2ad90b59a12a5279ceaa18b47"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70f 2021-05-25T11:52:56.451 INFO:teuthology.task.ansible.out:f00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41 2021-05-25T11:52:56.452 INFO:teuthology.task.ansible.out:e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5 2021-05-25T11:52:56.454 INFO:teuthology.task.ansible.out:dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b6 2021-05-25T11:52:56.454 INFO:teuthology.task.ansible.out:90da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b 2021-05-25T11:52:56.456 INFO:teuthology.task.ansible.out:1ae7b0"}, {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}]}, "changed": false} 2021-05-25T11:52:56.480 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:52:56 +0000 (0:00:00.126) 0:00:43.828 *********** 2021-05-25T11:52:56.900 INFO:teuthology.task.ansible.out: TASK [Create all admin users with sudo access.] ******************************** 2021-05-25T11:52:56.901 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10106, "groups": "sudo", "home": "/home/cyx1231st", "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "name": "cyx1231st", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10106} 2021-05-25T11:52:57.023 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:52:57 +0000 (0:00:00.542) 0:00:44.371 *********** 2021-05-25T11:52:57.091 INFO:teuthology.task.ansible.out: TASK [Create all users without sudo access.] *********************************** 2021-05-25T11:52:57.113 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:52:57 +0000 (0:00:00.090) 0:00:44.461 *********** 2021-05-25T11:52:57.204 INFO:teuthology.task.ansible.out: TASK [Merge managed_users and managed_admin users] ***************************** 2021-05-25T11:52:57.206 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratza@fedora 2021-05-25T11:52:57.207 INFO:teuthology.task.ansible.out:rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265 2021-05-25T11:52:57.209 INFO:teuthology.task.ansible.out:c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop K 2021-05-25T11:52:57.210 INFO:teuthology.task.ansible.out:D0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9 2021-05-25T11:52:57.212 INFO:teuthology.task.ansible.out:df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de 2021-05-25T11:52:57.214 INFO:teuthology.task.ansible.out:53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d 2021-05-25T11:52:57.215 INFO:teuthology.task.ansible.out:7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumra 2021-05-25T11:52:57.217 INFO:teuthology.task.ansible.out:o", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b76 2021-05-25T11:52:57.218 INFO:teuthology.task.ansible.out:8c2ad90b59a12a5279ceaa18b47"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70f 2021-05-25T11:52:57.220 INFO:teuthology.task.ansible.out:f00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41 2021-05-25T11:52:57.222 INFO:teuthology.task.ansible.out:e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5 2021-05-25T11:52:57.224 INFO:teuthology.task.ansible.out:dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b6 2021-05-25T11:52:57.224 INFO:teuthology.task.ansible.out:90da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b 2021-05-25T11:52:57.226 INFO:teuthology.task.ansible.out:1ae7b0"}, {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}]}, "changed": false} 2021-05-25T11:52:57.251 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:52:57 +0000 (0:00:00.137) 0:00:44.599 *********** 2021-05-25T11:52:57.371 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2021-05-25T11:52:57.372 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => {"after": "b2a91f5f527dcc382f8ed49697328a42af627ee8", "attempts": 1, "before": "b2a91f5f527dcc382f8ed49697328a42af627ee8", "changed": false, "remote_url_changed": false} 2021-05-25T11:52:57.394 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:52:57 +0000 (0:00:00.143) 0:00:44.742 *********** 2021-05-25T11:52:57.720 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-05-25T11:52:57.722 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2021-05-25T11:52:57.919 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2021-05-25T11:52:57.982 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* ok: [smithi102.front.sepia.ceph.com] => {"after": "b2a91f5f527dcc382f8ed49697328a42af627ee8", "attempts": 1, "before": "b2a91f5f527dcc382f8ed49697328a42af627ee8", "changed": false, "remote_url_changed": false} 2021-05-25T11:52:58.005 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:52:57 +0000 (0:00:00.610) 0:00:45.353 *********** 2021-05-25T11:52:58.136 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-05-25T11:52:58.138 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2021-05-25T11:52:58.324 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2021-05-25T11:52:58.346 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2021-05-25T11:52:58.348 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2021-05-25T11:52:58.533 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2021-05-25T11:52:58.545 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2021-05-25T11:52:58.742 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2021-05-25T11:52:58.753 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2021-05-25T11:52:58.755 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2021-05-25T11:52:58.954 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2021-05-25T11:52:58.956 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2021-05-25T11:52:59.152 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2021-05-25T11:52:59.153 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgaz5AsLB+hPwnHjlf6vktlUQ8iWsKA+hViok+j+uCm0U/iho7O7Qd8E/3tp9yBx10Wkvuz1mmf6v2AO7Qa8+SE1QQPmA+4xLp4xXCqlUGFjnzm/7uQcQZtlWdfBHzQ9jVyPAgUpnGVUtW8PhW0txe8MEtecOO2C3Snx0yuTOx6O11jNvT938QjOeRQWw3X2YFUtMIOiwEhKtqstBIGXlMdaQKfvdlbA9MjyYy6VfK1x8lVcu8umUf/z56YmMkC2iO117/obhdu1XsU9SmjM51wHMOUCXSVqH8E3BWu0yohy4QmZDkdFihyYa+hoS9dOqGC91dFaQc46xrdmV4wqg1 jcaratza@MacBook-Pro\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAV78mqEy/VkB2BthIE/m7iE5sbTjTRS0KuH7AB0KPvZBsDIw5cL8jKyxiaPC85+aOkBa4LrP0TUFBzN87Snt+PLMz2QHBOvlSTfUuRkHVo3vWJFzVuUSzzddkYds3hAxUriAHKv6CRSy1yFBxPQQHTSATUHKWqihMElbUCZrrrfH8A7J9vwixzCSLg4KbINjfa77rxefcZwugUqiYiVUWUSOb023f+QEaDm7OP+/uzz3SjF1Y0LMwa75yctttIfkRBKjY7uFCuZdKc3l2A/fdPxbFwlZmIox3LLngdTzHMVXGbVHqq9Ixo8FSAXiI1Buy2LTMoXMJwQYSvzsv7xNwujKVlUjoE9cOOIdzCC5AziwSiJB4+tNh73c5tRUzMMC98HLOhPJriSPvOt1pMEdWDX3/evcyxvRgJUKgYnmQG7n8v8O/CLyG0sAJDmGC3U9S/3fUDUjZIKWSsc85Q3xRaJwZJIS2VZwH0zG1Gmynih8PPq7VeDx7tKuNm4uTYT8= bigjust@desktop-jsho3l9.lan", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2021-05-25T11:52:59.352 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2021-05-25T11:52:59.353 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2021-05-25T11:52:59.364 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:52:59.562 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2021-05-25T11:52:59.563 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2021-05-25T11:52:59.584 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2021-05-25T11:52:59.760 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemer 2021-05-25T11:52:59.762 INFO:teuthology.task.ansible.out:son", "validate_certs": true} 2021-05-25T11:52:59.771 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:52:59.773 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:52:59.783 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgaz5AsLB+hPwnHjlf6vktlUQ8iWsKA+hViok+j+uCm0U/iho7O7Qd8E/3tp9yBx10Wkvuz1mmf6v2AO7Qa8+SE1QQPmA+4xLp4xXCqlUGFjnzm/7uQcQZtlWdfBHzQ9jVyPAgUpnGVUtW8PhW0txe8MEtecOO2C3Snx0yuTOx6O11jNvT938QjOeRQWw3X2YFUtMIOiwEhKtqstBIGXlMdaQKfvdlbA9MjyYy6VfK1x8lVcu8umUf/z56YmMkC2iO117/obhdu1XsU9SmjM51wHMOUCXSVqH8E3BWu0yohy4QmZDkdFihyYa+hoS9dOqGC91dFaQc46xrdmV4wqg1 jcaratza@MacBook-Pro\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAV78mqEy/VkB2BthIE/m7iE5sbTjTRS0KuH7AB0KPvZBsDIw5cL8jKyxiaPC85+aOkBa4LrP0TUFBzN87Snt+PLMz2QHBOvlSTfUuRkHVo3vWJFzVuUSzzddkYds3hAxUriAHKv6CRSy1yFBxPQQHTSATUHKWqihMElbUCZrrrfH8A7J9vwixzCSLg4KbINjfa77rxefcZwugUqiYiVUWUSOb023f+QEaDm7OP+/uzz3SjF1Y0LMwa75yctttIfkRBKjY7uFCuZdKc3l2A/fdPxbFwlZmIox3LLngdTzHMVXGbVHqq9Ixo8FSAXiI1Buy2LTMoXMJwQYSvzsv7xNwujKVlUjoE9cOOIdzCC5AziwSiJB4+tNh73c5tRUzMMC98HLOhPJriSPvOt1pMEdWDX3/evcyxvRgJUKgYnmQG7n8v8O/CLyG0sAJDmGC3U9S/3fUDUjZIKWSsc85Q3xRaJwZJIS2VZwH0zG1Gmynih8PPq7VeDx7tKuNm4uTYT8= bigjust@desktop-jsho3l9.lan", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2021-05-25T11:52:59.991 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2021-05-25T11:52:59.993 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2021-05-25T11:53:00.002 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:00.199 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2021-05-25T11:53:00.200 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2021-05-25T11:53:00.210 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2021-05-25T11:53:00.211 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2021-05-25T11:53:00.396 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2021-05-25T11:53:00.418 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemer 2021-05-25T11:53:00.419 INFO:teuthology.task.ansible.out:son", "validate_certs": true} skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:00.429 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:00.594 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": 2021-05-25T11:53:00.596 INFO:teuthology.task.ansible.out:null, "state": "present", "user": "davidz", "validate_certs": true} 2021-05-25T11:53:00.626 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2021-05-25T11:53:00.813 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2021-05-25T11:53:00.824 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2021-05-25T11:53:00.826 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2021-05-25T11:53:01.011 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2021-05-25T11:53:01.032 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2021-05-25T11:53:01.218 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2021-05-25T11:53:01.229 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": 2021-05-25T11:53:01.231 INFO:teuthology.task.ansible.out:null, "state": "present", "user": "davidz", "validate_certs": true} 2021-05-25T11:53:01.416 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2021-05-25T11:53:01.418 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2021-05-25T11:53:01.427 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2021-05-25T11:53:01.624 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2021-05-25T11:53:01.625 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2021-05-25T11:53:01.821 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2021-05-25T11:53:01.832 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2021-05-25T11:53:02.030 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2021-05-25T11:53:02.032 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2021-05-25T11:53:02.227 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2021-05-25T11:53:02.228 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4 2021-05-25T11:53:02.230 INFO:teuthology.task.ansible.out:YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2021-05-25T11:53:02.425 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2021-05-25T11:53:02.426 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2021-05-25T11:53:02.633 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2021-05-25T11:53:02.633 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2021-05-25T11:53:02.831 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46 2021-05-25T11:53:02.833 INFO:teuthology.task.ansible.out:TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2021-05-25T11:53:02.842 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2021-05-25T11:53:03.029 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2021-05-25T11:53:03.040 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2021-05-25T11:53:03.238 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2021-05-25T11:53:03.249 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2021-05-25T11:53:03.436 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2021-05-25T11:53:03.438 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2021-05-25T11:53:03.448 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2021-05-25T11:53:03.645 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2021-05-25T11:53:03.656 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2021-05-25T11:53:03.854 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2021-05-25T11:53:03.865 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2021-05-25T11:53:04.062 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2021-05-25T11:53:04.063 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2021-05-25T11:53:04.065 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2021-05-25T11:53:04.260 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2021-05-25T11:53:04.261 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2021-05-25T11:53:04.468 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2021-05-25T11:53:04.470 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2021-05-25T11:53:04.665 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2021-05-25T11:53:04.687 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-05-25T11:53:04.874 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2021-05-25T11:53:04.886 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2021-05-25T11:53:05.083 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2021-05-25T11:53:05.094 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2021-05-25T11:53:05.280 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-05-25T11:53:05.292 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2021-05-25T11:53:05.293 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2021-05-25T11:53:05.489 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2021-05-25T11:53:05.500 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2021-05-25T11:53:05.501 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2021-05-25T11:53:05.687 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2021-05-25T11:53:05.698 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2021-05-25T11:53:05.906 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2021-05-25T11:53:05.908 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2021-05-25T11:53:06.104 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2021-05-25T11:53:06.106 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2021-05-25T11:53:06.312 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2021-05-25T11:53:06.314 INFO:teuthology.task.ansible.out:changed: [smithi181.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtV 2021-05-25T11:53:06.316 INFO:teuthology.task.ansible.out:TFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDISEXvGI2oJLTRR15ePZLkmdYjaQJxt9V/yBwKdGqYvnPzmLOpdmVpdQJo7ZOWwdp5tJsCy4q5uX5vIUv6Gyi3uVSGmZnrKUbsqoDNwAX5vx/gFBINIs/ZXXyD4fpUT8aoZeq1XmXdIaWG0nImsrtRnRxKP9NGZEcW8z0WPwgF0Ck+RH0qZUix6Bfbgy7vMXnivuDv+NdjEwyszcrPWvf6Krr0am9Q0+9nj/mjvk6CZLl+CH3HNdkhZuNs+qMG4XW8KhLvQB2p790XVlTHuVU4jWrFmjymggZxkhoCY/ukHH6wi2Pbx6yjjKOj8iIw1xKabJzJ7S8UKT23A41d918DUCUuEG66pN6Z4dnv+fZJxj5iEo1GdycSWIc4SpLTK3z4zDZypVKIzjI4o8VwQaN48HkVjUdTICTvGQfytFi3jX1I5nxIZiiPcxMQLR5hQggP8pn2XtvXpomn0aBc8tVuuZJLVq4zCVnhlVB6IACmb71evjFGOsoSuEsaTufUr48= prsrivas@t480.lab.blr.redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-05-25T11:53:06.510 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2021-05-25T11:53:06.521 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2021-05-25T11:53:06.523 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2021-05-25T11:53:06.524 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2021-05-25T11:53:06.718 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2021-05-25T11:53:06.720 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2021-05-25T11:53:06.917 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2021-05-25T11:53:06.928 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtV 2021-05-25T11:53:06.929 INFO:teuthology.task.ansible.out:TFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDISEXvGI2oJLTRR15ePZLkmdYjaQJxt9V/yBwKdGqYvnPzmLOpdmVpdQJo7ZOWwdp5tJsCy4q5uX5vIUv6Gyi3uVSGmZnrKUbsqoDNwAX5vx/gFBINIs/ZXXyD4fpUT8aoZeq1XmXdIaWG0nImsrtRnRxKP9NGZEcW8z0WPwgF0Ck+RH0qZUix6Bfbgy7vMXnivuDv+NdjEwyszcrPWvf6Krr0am9Q0+9nj/mjvk6CZLl+CH3HNdkhZuNs+qMG4XW8KhLvQB2p790XVlTHuVU4jWrFmjymggZxkhoCY/ukHH6wi2Pbx6yjjKOj8iIw1xKabJzJ7S8UKT23A41d918DUCUuEG66pN6Z4dnv+fZJxj5iEo1GdycSWIc4SpLTK3z4zDZypVKIzjI4o8VwQaN48HkVjUdTICTvGQfytFi3jX1I5nxIZiiPcxMQLR5hQggP8pn2XtvXpomn0aBc8tVuuZJLVq4zCVnhlVB6IACmb71evjFGOsoSuEsaTufUr48= prsrivas@t480.lab.blr.redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-05-25T11:53:07.125 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2021-05-25T11:53:07.136 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2021-05-25T11:53:07.137 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2021-05-25T11:53:07.139 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2021-05-25T11:53:07.344 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2021-05-25T11:53:07.345 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2021-05-25T11:53:07.541 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2021-05-25T11:53:07.543 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2021-05-25T11:53:07.739 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2021-05-25T11:53:07.741 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2021-05-25T11:53:07.750 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2021-05-25T11:53:07.947 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2021-05-25T11:53:07.959 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2021-05-25T11:53:08.146 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2021-05-25T11:53:08.167 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2021-05-25T11:53:08.354 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2021-05-25T11:53:08.356 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} 2021-05-25T11:53:08.376 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2021-05-25T11:53:08.377 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2021-05-25T11:53:08.552 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2021-05-25T11:53:08.584 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2021-05-25T11:53:08.750 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2021-05-25T11:53:08.792 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2021-05-25T11:53:08.958 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2021-05-25T11:53:08.960 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2021-05-25T11:53:09.000 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2021-05-25T11:53:09.001 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} 2021-05-25T11:53:09.156 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2021-05-25T11:53:09.198 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2021-05-25T11:53:09.364 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2021-05-25T11:53:09.396 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2021-05-25T11:53:09.561 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2021-05-25T11:53:09.603 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2021-05-25T11:53:09.605 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2021-05-25T11:53:09.769 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-05-25T11:53:09.801 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2021-05-25T11:53:09.968 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2021-05-25T11:53:10.000 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2021-05-25T11:53:10.177 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2021-05-25T11:53:10.219 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2021-05-25T11:53:10.395 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2021-05-25T11:53:10.417 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-05-25T11:53:10.593 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2021-05-25T11:53:10.614 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2021-05-25T11:53:10.791 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2021-05-25T11:53:10.813 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2021-05-25T11:53:10.989 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2021-05-25T11:53:11.010 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2021-05-25T11:53:11.196 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2021-05-25T11:53:11.218 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2021-05-25T11:53:11.395 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2021-05-25T11:53:11.416 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2021-05-25T11:53:11.603 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2021-05-25T11:53:11.615 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2021-05-25T11:53:11.801 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2021-05-25T11:53:11.812 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2021-05-25T11:53:12.009 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2021-05-25T11:53:12.021 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2021-05-25T11:53:12.208 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2021-05-25T11:53:12.220 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2021-05-25T11:53:12.407 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2021-05-25T11:53:12.418 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2021-05-25T11:53:12.604 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQu/CuhCIFVGCLbA0cjn+qqcwX12cRcFA1sHyJtkj6Ltfmge8FnfgAsex/LaPJwbtbBiVl9xRcazHNfBaDw5DxMRSJF+e7WTydZC2Jm4qIj4D3Lr+imGPrJTK+kjN0QXRXknGsNn36oRF/I41cGqYRGwawFGSPZBHuav+ehDk1doTrbDlll4zMUWm2iXTcBIcJo4NR1LoE2c9/MPfbH7jKm5ke7aS0ets+EANEplkeClfCVAashtIF+Bz6a9qk4F2n0baLT/s3IQkXCwwO2fxxY03x8gWiB1n4E7sousEB+J1Skl/z0bwmaiY0C+raPoV/FyZxQuh0ocvdU7fnQbKuNoGE/Usfm9Y3g8eH81b/k4fMJL2lqXHkPft6zUV9bJsp3v1iGEkIf2+ER/QLczteNz6kw/uTPQctMEDXM17GEnBXISVrQcPSW3i93eFwWacs8iT3G4bjEQvpxSBQrhDdCCoxy8bVWqkVeDWw5u2q/3WL9Ov1NrY+G4/DYb2qONNQds0vTHCGvemU5GNjtaeCbojkZ7Z3KOeAY8080kKrsO7fve+ZMACCIFD+se8kF7NN4ZJASSqq9lJpBWvtbc0O8jCcVGNuN7exXjrcbiN1L/9HkARV+KEJU3jbXObAhAnfAaeRWBwmTfZodCu+fDd/Jor6l36tjD4kC4/CPrDtyQ== alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2021-05-25T11:53:12.615 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2021-05-25T11:53:12.813 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2021-05-25T11:53:12.824 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2021-05-25T11:53:13.010 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2021-05-25T11:53:13.021 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2021-05-25T11:53:13.208 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1 2021-05-25T11:53:13.242 INFO:teuthology.task.ansible.out:lXTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQu/CuhCIFVGCLbA0cjn+qqcwX12cRcFA1sHyJtkj6Ltfmge8FnfgAsex/LaPJwbtbBiVl9xRcazHNfBaDw5DxMRSJF+e7WTydZC2Jm4qIj4D3Lr+imGPrJTK+kjN0QXRXknGsNn36oRF/I41cGqYRGwawFGSPZBHuav+ehDk1doTrbDlll4zMUWm2iXTcBIcJo4NR1LoE2c9/MPfbH7jKm5ke7aS0ets+EANEplkeClfCVAashtIF+Bz6a9qk4F2n0baLT/s3IQkXCwwO2fxxY03x8gWiB1n4E7sousEB+J1Skl/z0bwmaiY0C+raPoV/FyZxQuh0ocvdU7fnQbKuNoGE/Usfm9Y3g8eH81b/k4fMJL2lqXHkPft6zUV9bJsp3v1iGEkIf2+ER/QLczteNz6kw/uTPQctMEDXM17GEnBXISVrQcPSW3i93eFwWacs8iT3G4bjEQvpxSBQrhDdCCoxy8bVWqkVeDWw5u2q/3WL9Ov1NrY+G4/DYb2qONNQds0vTHCGvemU5GNjtaeCbojkZ7Z3KOeAY8080kKrsO7fve+ZMACCIFD+se8kF7NN4ZJASSqq9lJpBWvtbc0O8jCcVGNuN7exXjrcbiN1L/9HkARV+KEJU3jbXObAhAnfAaeRWBwmTfZodCu+fDd/Jor6l36tjD4kC4/CPrDtyQ== alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2021-05-25T11:53:13.416 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2021-05-25T11:53:13.427 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2021-05-25T11:53:13.613 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2021-05-25T11:53:13.625 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2021-05-25T11:53:13.812 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2021-05-25T11:53:13.823 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1 2021-05-25T11:53:13.825 INFO:teuthology.task.ansible.out:lXTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2021-05-25T11:53:14.020 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2021-05-25T11:53:14.032 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2021-05-25T11:53:14.218 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.locald 2021-05-25T11:53:14.219 INFO:teuthology.task.ansible.out:omain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2021-05-25T11:53:14.229 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2021-05-25T11:53:14.425 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2021-05-25T11:53:14.427 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2021-05-25T11:53:14.622 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2021-05-25T11:53:14.634 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2021-05-25T11:53:14.831 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2021-05-25T11:53:14.833 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.locald 2021-05-25T11:53:14.834 INFO:teuthology.task.ansible.out:omain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2021-05-25T11:53:15.028 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrGYBGPDQqn+2jnxKhDurSrAJvTT8dZMYGVCt6NbjpFhDFYtyvrF8Jv5NKBQynG18LwscN7a57L0Oq0e4jOEPRz//CDcAfP+L6p45L4BdUzTysN4+rQ7waMnIiYp7VjSswhGH/BnYIVKwmDqzM2fo1w/oLIT6mgwUYqAdceeswAkXBt7jgagL8ox3E4oHBqER7f60AHdTviuW953t5luVQiMjyF2fOwZuo2ycufIzBee0CwS+xyRuJQwdgD3HL4Qs2vtBCX+H8n3wC6A9JrMA5tzCDr5VZKBJZURIniwvaXfJodjO4SvBL5jAfXoMqriv8ZWHbap2fk82YE8/uPqNvSISjixJEBgQinklZ+iAOYm5tfPbhPM/8ErpN/yJ0EoGRn+nj6X6nAjXAIk3AENopeqtM7mLZ26RFIuEV5ld7eqUvAfiy6Hk2LH9Q1Z4CXaxD0gDGzUxgdAaaAgLAR4btWi2n7Tb7ZNDdOfctg2ya9ftzZBOHvCaVXSpJCry/GMU= skoduri@localhost.localdomain", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2021-05-25T11:53:15.029 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2021-05-25T11:53:15.030 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2021-05-25T11:53:15.226 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2021-05-25T11:53:15.237 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2021-05-25T11:53:15.434 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2021-05-25T11:53:15.445 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2021-05-25T11:53:15.633 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2021-05-25T11:53:15.644 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrGYBGPDQqn+2jnxKhDurSrAJvTT8dZMYGVCt6NbjpFhDFYtyvrF8Jv5NKBQynG18LwscN7a57L0Oq0e4jOEPRz//CDcAfP+L6p45L4BdUzTysN4+rQ7waMnIiYp7VjSswhGH/BnYIVKwmDqzM2fo1w/oLIT6mgwUYqAdceeswAkXBt7jgagL8ox3E4oHBqER7f60AHdTviuW953t5luVQiMjyF2fOwZuo2ycufIzBee0CwS+xyRuJQwdgD3HL4Qs2vtBCX+H8n3wC6A9JrMA5tzCDr5VZKBJZURIniwvaXfJodjO4SvBL5jAfXoMqriv8ZWHbap2fk82YE8/uPqNvSISjixJEBgQinklZ+iAOYm5tfPbhPM/8ErpN/yJ0EoGRn+nj6X6nAjXAIk3AENopeqtM7mLZ26RFIuEV5ld7eqUvAfiy6Hk2LH9Q1Z4CXaxD0gDGzUxgdAaaAgLAR4btWi2n7Tb7ZNDdOfctg2ya9ftzZBOHvCaVXSpJCry/GMU= skoduri@localhost.localdomain", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2021-05-25T11:53:15.840 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2021-05-25T11:53:15.851 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2021-05-25T11:53:16.039 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2021-05-25T11:53:16.050 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.loc 2021-05-25T11:53:16.051 INFO:teuthology.task.ansible.out:al", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2021-05-25T11:53:16.246 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2021-05-25T11:53:16.248 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2021-05-25T11:53:16.444 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2021-05-25T11:53:16.456 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2021-05-25T11:53:16.653 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2021-05-25T11:53:16.654 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.loc 2021-05-25T11:53:16.655 INFO:teuthology.task.ansible.out:al", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2021-05-25T11:53:16.851 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2021-05-25T11:53:16.862 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2021-05-25T11:53:17.048 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2021-05-25T11:53:17.060 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2021-05-25T11:53:17.257 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2021-05-25T11:53:17.258 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2021-05-25T11:53:17.465 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDISEXvGI2oJLTRR15ePZLkmdYjaQJxt9V/ 2021-05-25T11:53:17.467 INFO:teuthology.task.ansible.out:yBwKdGqYvnPzmLOpdmVpdQJo7ZOWwdp5tJsCy4q5uX5vIUv6Gyi3uVSGmZnrKUbsqoDNwAX5vx/gFBINIs/ZXXyD4fpUT8aoZeq1XmXdIaWG0nImsrtRnRxKP9NGZEcW8z0WPwgF0Ck+RH0qZUix6Bfbgy7vMXnivuDv+NdjEwyszcrPWvf6Krr0am9Q0+9nj/mjvk6CZLl+CH3HNdkhZuNs+qMG4XW8KhLvQB2p790XVlTHuVU4jWrFmjymggZxkhoCY/ukHH6wi2Pbx6yjjKOj8iIw1xKabJzJ7S8UKT23A41d918DUCUuEG66pN6Z4dnv+fZJxj5iEo1GdycSWIc4SpLTK3z4zDZypVKIzjI4o8VwQaN48HkVjUdTICTvGQfytFi3jX1I5nxIZiiPcxMQLR5hQggP8pn2XtvXpomn0aBc8tVuuZJLVq4zCVnhlVB6IACmb71evjFGOsoSuEsaTufUr48= prsrivas@t480.lab.blr.redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2021-05-25T11:53:17.664 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2021-05-25T11:53:17.666 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2021-05-25T11:53:17.872 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2021-05-25T11:53:17.874 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2021-05-25T11:53:18.070 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDISEXvGI2oJLTRR15ePZLkmdYjaQJxt9V/ 2021-05-25T11:53:18.072 INFO:teuthology.task.ansible.out:yBwKdGqYvnPzmLOpdmVpdQJo7ZOWwdp5tJsCy4q5uX5vIUv6Gyi3uVSGmZnrKUbsqoDNwAX5vx/gFBINIs/ZXXyD4fpUT8aoZeq1XmXdIaWG0nImsrtRnRxKP9NGZEcW8z0WPwgF0Ck+RH0qZUix6Bfbgy7vMXnivuDv+NdjEwyszcrPWvf6Krr0am9Q0+9nj/mjvk6CZLl+CH3HNdkhZuNs+qMG4XW8KhLvQB2p790XVlTHuVU4jWrFmjymggZxkhoCY/ukHH6wi2Pbx6yjjKOj8iIw1xKabJzJ7S8UKT23A41d918DUCUuEG66pN6Z4dnv+fZJxj5iEo1GdycSWIc4SpLTK3z4zDZypVKIzjI4o8VwQaN48HkVjUdTICTvGQfytFi3jX1I5nxIZiiPcxMQLR5hQggP8pn2XtvXpomn0aBc8tVuuZJLVq4zCVnhlVB6IACmb71evjFGOsoSuEsaTufUr48= prsrivas@t480.lab.blr.redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2021-05-25T11:53:18.278 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2021-05-25T11:53:18.279 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2021-05-25T11:53:18.477 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2021-05-25T11:53:18.478 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2021-05-25T11:53:18.675 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2021-05-25T11:53:18.677 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2021-05-25T11:53:18.875 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2021-05-25T11:53:18.876 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2021-05-25T11:53:19.083 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2021-05-25T11:53:19.084 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2021-05-25T11:53:19.280 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2021-05-25T11:53:19.281 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2021-05-25T11:53:19.488 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2021-05-25T11:53:19.499 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2021-05-25T11:53:19.686 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2021-05-25T11:53:19.697 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2021-05-25T11:53:19.883 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2021-05-25T11:53:19.894 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2021-05-25T11:53:20.081 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2021-05-25T11:53:20.103 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2021-05-25T11:53:20.289 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2021-05-25T11:53:20.301 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2021-05-25T11:53:20.487 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2021-05-25T11:53:20.498 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2021-05-25T11:53:20.695 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2021-05-25T11:53:20.706 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2021-05-25T11:53:20.892 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2021-05-25T11:53:20.903 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2021-05-25T11:53:21.090 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2021-05-25T11:53:21.101 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2021-05-25T11:53:21.297 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2021-05-25T11:53:21.309 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjL 2021-05-25T11:53:21.310 INFO:teuthology.task.ansible.out:d+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} 2021-05-25T11:53:21.495 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2021-05-25T11:53:21.506 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2021-05-25T11:53:21.704 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2021-05-25T11:53:21.706 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2021-05-25T11:53:21.902 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjL 2021-05-25T11:53:21.904 INFO:teuthology.task.ansible.out:d+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2021-05-25T11:53:22.100 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2021-05-25T11:53:22.111 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2021-05-25T11:53:22.308 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2021-05-25T11:53:22.319 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2021-05-25T11:53:22.527 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} 2021-05-25T11:53:22.672 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2021-05-25T11:53:22.725 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} 2021-05-25T11:53:22.870 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2021-05-25T11:53:22.943 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} 2021-05-25T11:53:23.069 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2021-05-25T11:53:23.142 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EkUbnaeC7Dqtq2N4/dX9v+WGlrgRS2ygHhlbZy4KHbd6+Q9T5NAWtqnFwqcbFpYYUBwMvz8zLo6HYN6gNskRN0qTiFKPQdtnydF0Tx1KHrdGGcX3G2lxyGH5P8nrqReXAPPvRiVc2/Mj7APCTmY2JftkSS1qYcTleeueivVyho6z6uD1AyC7OXANhtn7kGeIY/NvVKBhDr+3TuPmpPwPCQMcJUyZWCHYyp47v2vzNBy2Etp7WjGwOnfZ5ULiJGjH6PCXx0YfpmTUOPVkNbV6Om+0YJ3fcLlmQ/2nsdsWfOKvoAr+1uzO81h7ZBX9LVEitj8YU3WeSCV2vqS2VvOpq5NyHXWGq+ZkOCfhy4Z9MvMG+6hqp4p4z1JMpKsZ191kP7eAQua41GkTSQPlEfF+um1UV6ZjNzhB68utOXyGDoDcxct9oOdnhhjwu5W4/0t9iBHuMn02iHRYVrUWOH2FXVysR0p5x/L2xdvNPiTgFAZ2eN+EFWmdan4e9mAAw3M= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} 2021-05-25T11:53:23.277 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} 2021-05-25T11:53:23.340 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBYtdHKBXQcXQFkEW8fSwlFK/xZM16VcFY288ABI/DPfejTStHhxiRvmlsWzMy0Ar5WadZ8FU1ptjA3MKBEY1Do1r1Ul369k3zHCvxXyg9QGGMNPYfHmnNeoF4GEgoJpuounPs0mkLmrPTR+cR+/XuaZZt38nsad8M/zJetnolvqCWpcV87D7+Rt8F32J6kGDknC4ikEe8I0n7F+goOmJhlgkqegbKEeJ/EMnOItwVZHFwaC+sxOq4IlYp2f1aZMKGAtr5+ICwYH1SMCcvnp3qWL8O//t2tvguZgQxzIs6Nf5GaxwdQkh4CvjbWiCkIbGFJ57YICS6k++KMXMbEnwh jsalomon@Josh-laptop.tlv.redhat.com", "key_options": null, "keyfile": "/home/jsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsalomon", "validate_certs": true} 2021-05-25T11:53:23.476 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} 2021-05-25T11:53:23.682 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} 2021-05-25T11:53:23.683 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9ltUNitnn0FNwMeiGkBnozpBN5ziVR2MreaqkfqTQY2TgI24559xuILyy+TnAKumhemZ8i8nIQ876Iso85OzIXbUbRM45hpL/bMQX7pWyMIRNeFKrCDwf18OT0DPc8aeDOAFQ6YoN0j4awUENjdZuuAW80cfcjoQO3m7ORF8wBF0IvMWwPDIQQrz+Li31ECgXg6VC7K3e+BNHlfNQ/2/ahS0+V4J6JYFsylZL0r2tydhDMd6rR8QA9c9BgeBz7+KWobGtiZVe7KReTzJ0O38PxNd8br3aS50j5BiN63gch8eFxPAROaUMnuiavu6OTDy4/aeL3Zc5WpUy3VyyumWf jolmomar@localhost.localdomain", "key_options": null, "keyfile": "/home/jolmomar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jolmomar", "validate_certs": true} 2021-05-25T11:53:23.881 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQKiUAlB8dxudAY60M7jnIJvs9yb2pNaT+BVv1Vk1kMs0XjPjcw9cyT3DtQ1xGLF367phRi/cfQbXOYbHG+uyBzQ4EEnBFvziktWRx4mfeWYxKjkjbfkVfpKPsxztqCbwm6+NYzMxGK2cExN7jUHnghHAjfA0IUpVVg9peBD3kUveLwS+UxuJAnuj+G4480LS6GDyLrhFEXU7uOIIlW4S9oLnU6vJL6GAzwyLE4ijTA8Ek03SJ01Ey/gpU9AqnQNp9u12l/7sJely6sI9pAuIzmS8Ht324hexxSWN5sFjXgLvLfFK83ziDBdrFBr0WXnWsFdRpHs2CwSq1JTkByqbN hmunjulu@hmunjulu.remote.csb", "key_options": null, "keyfile": "/home/hmunjulu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hmunjulu", "validate_certs": true} 2021-05-25T11:53:23.883 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EkUbnaeC7Dqtq2N4/dX9v+WGlrgRS2ygHhlbZy4KHbd6+Q9T5NAWtqnFwqcbFpYYUBwMvz8zLo6HYN6gNskRN0qTiFKPQdtnydF0Tx1KHrdGGcX3G2lxyGH5P8nrqReXAPPvRiVc2/Mj7APCTmY2JftkSS1qYcTleeueivVyho6z6uD1AyC7OXANhtn7kGeIY/NvVKBhDr+3TuPmpPwPCQMcJUyZWCHYyp47v2vzNBy2Etp7WjGwOnfZ5ULiJGjH6PCXx0YfpmTUOPVkNbV6Om+0YJ3fcLlmQ/2nsdsWfOKvoAr+1uzO81h7ZBX9LVEitj8YU3WeSCV2vqS2VvOpq5NyHXWGq+ZkOCfhy4Z9MvMG+6hqp4p4z1JMpKsZ191kP7eAQua41GkTSQPlEfF+um1UV6ZjNzhB68utOXyGDoDcxct9oOdnhhjwu5W4/0t9iBHuMn02iHRYVrUWOH2FXVysR0p5x/L2xdvNPiTgFAZ2eN+EFWmdan4e9mAAw3M= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} 2021-05-25T11:53:24.089 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoefxOfeArToGucfWG7y1tyLF2I1J/IkBSlmxxEJyHyDUB6ElXjYIJReoiPfmB0xLEmINpmt8QA4XN/8UbR1r9LFOexlZZbX1LSbihi11LXCvfIJWqkHVwIyKHRTu/duMrntBFVENJ0LVf0GSWVTtllXaDA+fMkp6/ra/hlRHQbx19UXy/9+Pogj8Xl+SntnMGiHlX8FFq2A2tUqhdQSf/1GW1HfUIa5WtwdEqIpKpAgMM7sOW9i3QAmLAMo8HiKrArSiy7jrRUODCqsTJsfifxEbmnOS0ZtmDh0eeWL/Dyqbz99Rg4WbFNeK8oyZYr7D6bEAhd2az2A+wycPx4Nq7 julpark@julpark.remote.csb", "key_options": null, "keyfile": "/home/julpark/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "julpark", "validate_certs": true} 2021-05-25T11:53:24.090 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBYtdHKBXQcXQFkEW8fSwlFK/xZM16VcFY288ABI/DPfejTStHhxiRvmlsWzMy0Ar5WadZ8FU1ptjA3MKBEY1Do1r1Ul369k3zHCvxXyg9QGGMNPYfHmnNeoF4GEgoJpuounPs0mkLmrPTR+cR+/XuaZZt38nsad8M/zJetnolvqCWpcV87D7+Rt8F32J6kGDknC4ikEe8I0n7F+goOmJhlgkqegbKEeJ/EMnOItwVZHFwaC+sxOq4IlYp2f1aZMKGAtr5+ICwYH1SMCcvnp3qWL8O//t2tvguZgQxzIs6Nf5GaxwdQkh4CvjbWiCkIbGFJ57YICS6k++KMXMbEnwh jsalomon@Josh-laptop.tlv.redhat.com", "key_options": null, "keyfile": "/home/jsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsalomon", "validate_certs": true} 2021-05-25T11:53:24.289 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD03otezqmbR7v9dq62mWH5Kkg692jdmDZk6Ve4vux0A0YhxBuXDCyBWLE1Z4lLNxQahvK0pGsgokp+Uk+Q6aU+ZnF/E2jLp6YWerFXIHY3M7byObJx9+QuVBfNI0bZBkV3+vSjhSCGOYU+TrTHsKZqhsTcejZjmL3gFLmWfU6dTh0sX05qz97cdJl3CMPhCIR3KjtkiPygj417/N7wet/Efv1t79TvyS7dtK1oz+zRLEpKi/fZjvpXb8xec7as72Y0XexsWMDyZJ/N8PijTQ5L6EfLMV7ae/bWGzNAKsoC1dHVY7q/Q4pzktC6zU0Rc9sDRMtrgFg+q+04BfveSLfxTb9tqf7SbSvh7IAG1eKiEuDT/uy3T6Uiw+CiZ0eIqYzZscO/aDjUArGbS8eyaycybdYCBK9Jz38i0mYKkXg855yoVuIExr6MXieYwyVoVW8gbx1bAslaafrzra2dkAFByOJ0sbosPFRgR2+boUSxD6IaE+ALgno4T/0CI2RMiH0= prashant@cephdev", "key_options": null, "keyfile": "/home/pdhange/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdhange", "validate_certs": true} 2021-05-25T11:53:24.290 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9ltUNitnn0FNwMeiGkBnozpBN5ziVR2MreaqkfqTQY2TgI24559xuILyy+TnAKumhemZ8i8nIQ876Iso85OzIXbUbRM45hpL/bMQX7pWyMIRNeFKrCDwf18OT0DPc8aeDOAFQ6YoN0j4awUENjdZuuAW80cfcjoQO3m7ORF8wBF0IvMWwPDIQQrz+Li31ECgXg6VC7K3e+BNHlfNQ/2/ahS0+V4J6JYFsylZL0r2tydhDMd6rR8QA9c9BgeBz7+KWobGtiZVe7KReTzJ0O38PxNd8br3aS50j5BiN63gch8eFxPAROaUMnuiavu6OTDy4/aeL3Zc5WpUy3VyyumWf jolmomar@localhost.localdomain", "key_options": null, "keyfile": "/home/jolmomar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jolmomar", "validate_certs": true} 2021-05-25T11:53:24.488 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg7xW5+Mk7pG5MjLxOc7GTvDTCPBJj2L3MEQZIapmmo6t8uPODwDewXncSzrKDH6zFVE4TTTD9fJdu+BhyT+jCFgXrEOMMabj7vJsg0wO0SWqr+4NECFPhD9y4ewfYiewBeXisclA5Bto91v93AtYpZBCdmPYSi0BEcPVc6S3iBUD0PzIrL62I/kBAtm0f/bScRiT7XvQJXvkbEObW2yS2XhUv2qgq1VlFvO1jl3B85KMTHVb8+LrKpQWOKfRfxJ74c51y6tnEjZLFO6Xn0JevqOhd8gPQIXueFZOM0g0pNp1mbtGaaBlEIibbrzo2p9kzsdvJeHMq1XuMIi1/Qyj9 dpivonka@dhcp-41-144.bos.redhat.com", "key_options": null, "keyfile": "/home/dpivonka/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dpivonka", "validate_certs": true} 2021-05-25T11:53:24.489 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQKiUAlB8dxudAY60M7jnIJvs9yb2pNaT+BVv1Vk1kMs0XjPjcw9cyT3DtQ1xGLF367phRi/cfQbXOYbHG+uyBzQ4EEnBFvziktWRx4mfeWYxKjkjbfkVfpKPsxztqCbwm6+NYzMxGK2cExN7jUHnghHAjfA0IUpVVg9peBD3kUveLwS+UxuJAnuj+G4480LS6GDyLrhFEXU7uOIIlW4S9oLnU6vJL6GAzwyLE4ijTA8Ek03SJ01Ey/gpU9AqnQNp9u12l/7sJely6sI9pAuIzmS8Ht324hexxSWN5sFjXgLvLfFK83ziDBdrFBr0WXnWsFdRpHs2CwSq1JTkByqbN hmunjulu@hmunjulu.remote.csb", "key_options": null, "keyfile": "/home/hmunjulu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hmunjulu", "validate_certs": true} 2021-05-25T11:53:24.686 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoefxOfeArToGucfWG7y1tyLF2I1J/IkBSlmxxEJyHyDUB6ElXjYIJReoiPfmB0xLEmINpmt8QA4XN/8UbR1r9LFOexlZZbX1LSbihi11LXCvfIJWqkHVwIyKHRTu/duMrntBFVENJ0LVf0GSWVTtllXaDA+fMkp6/ra/hlRHQbx19UXy/9+Pogj8Xl+SntnMGiHlX8FFq2A2tUqhdQSf/1GW1HfUIa5WtwdEqIpKpAgMM7sOW9i3QAmLAMo8HiKrArSiy7jrRUODCqsTJsfifxEbmnOS0ZtmDh0eeWL/Dyqbz99Rg4WbFNeK8oyZYr7D6bEAhd2az2A+wycPx4Nq7 julpark@julpark.remote.csb", "key_options": null, "keyfile": "/home/julpark/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "julpark", "validate_certs": true} 2021-05-25T11:53:24.697 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0m+8wna4RnrOcuCUj971EBe4cFiTXVkEWkIsUX3yeCUwMaS/ZjaoOjcdYX7cYpCp1ilo4FSO5BsochU0RiMbQEwLGcE0zB2q1ac8+yYKCuWWyknig2t4bOReu5OrXvl6mj/9R+p7clFlYhwIoNG5ZZ/2eTbo3nH7gWgaOKaPbTtvEGg/5R/qjQKg8OTC8oix6kLsaNTisQJ78Zd7b4ukg9bO0jlq/2UhQOBM072sU1/kJpWTkcA0xxm3nsC4rWPwapgz7UwBmMdyx116GZgWdfQecTLKO7b8pyJw21npk2ntUmeMi7WPZGX6VI3IpaAhBjW3KiATAb8JbAQh8F9bmejkw6x7zrP3hYKQOg3A+U8DKRRo/Ad+f0zz9gjC42DUvMEOr09RBdxZcEAPY7H/YRWEqbA2zPX3Gfth4b7/letzM3m9ToUvVfSt1CdjaZD+gIia9pCgDMb+j1j2w9reG4F7Us+W/1f5pYCyFyrm4XO3AdoYM9/FOAOR3vffTIQU= adking@localhost.localdomain", "key_options": null, "keyfile": "/home/adking/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adking", "validate_certs": true} 2021-05-25T11:53:24.894 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD03otezqmbR7v9dq62mWH5Kkg692jdmDZk6Ve4vux0A0YhxBuXDCyBWLE1Z4lLNxQahvK0pGsgokp+Uk+Q6aU+ZnF/E2jLp6YWerFXIHY3M7byObJx9+QuVBfNI0bZBkV3+vSjhSCGOYU+TrTHsKZqhsTcejZjmL3gFLmWfU6dTh0sX05qz97cdJl3CMPhCIR3KjtkiPygj417/N7wet/Efv1t79TvyS7dtK1oz+zRLEpKi/fZjvpXb8xec7as72Y0XexsWMDyZJ/N8PijTQ5L6EfLMV7ae/bWGzNAKsoC1dHVY7q/Q4pzktC6zU0Rc9sDRMtrgFg+q+04BfveSLfxTb9tqf7SbSvh7IAG1eKiEuDT/uy3T6Uiw+CiZ0eIqYzZscO/aDjUArGbS8eyaycybdYCBK9Jz38i0mYKkXg855yoVuIExr6MXieYwyVoVW8gbx1bAslaafrzra2dkAFByOJ0sbosPFRgR2+boUSxD6IaE+ALgno4T/0CI2RMiH0= prashant@cephdev", "key_options": null, "keyfile": "/home/pdhange/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdhange", "validate_certs": true} 2021-05-25T11:53:24.895 INFO:teuthology.task.ansible.out:changed: [smithi181.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCs6q2b5NcUsNXl+w1Mrc7idNvGdwWkL0XRX5zTw22UNpQyqPkSrRTFKi4NxoY8ucv8JoLnaJL7hYSRHKTh7SqH78uPLZX0FLtoHvpYLgUgJzzJ5AOW5uekIhCMCm+3egmFnEBwFcNxtGnU2es02ByXNEXRxt4/LQ0f8K6lMqxVRSKYcltvWzlA7WFFdTnYMunbCYOdpw88RXm2pYjGIHRBnie7+pFiFSS6K/mkzQcfLiJmahgLSaYslI0ho1vQd8xfNLmZ700C5F4VTUlT/mJK/dI2LF3OfaPTsyqKNzOchrjLFx5b6dhZ4WZyqu7VKsHNm+xHZwT8VP3LPGSVVBdmEsiCsmV4g4QTI5ssZLi7siZDpaFp72nVAIO1i6kBGuiF3or+J4LmIh8ZRKzTc0DRDOw7S+IkWu1fv2UhYg83z6wrz4HeczI+bnolFFWW6HgGV3MFaUeRTRvrbxw2DPoEGQ+cjP0wsjyEhxlScwdh8W8IbVpNl+7mBe45YGUH2hg83Lst5QQxD4PHelriu40moWRHFKCnKfNesb4L1uqu29o8f+/PgpLdrWE/P5zIKe2ReWJyScYAI7jTMlFw+xgf+0TKryk44NMy90l3ei0umlx4qsFbbEv+lhzAALrUJ/da7fsQw7eIVykmcscBeYr99/YSYk4MGdk4jVCiHsQkdw== amathuri@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJ3ch+pU88/tX5TVVFl/1pPief8drYLRkK4BtfJk+xVNxX9qkDtOtiLAU+OugYF1/xk9hE9zTbm3UBL23unrzhlR80t8f2FiF1YFhHVzeZ/p/OunmkU0RFQzkMVuwKVOUNulw9FwpEZ/0Bjfxvaims0SBWaCzBgE5g0R+Ofdq/bVF82K8vU5Laa3KZ/cCdGzFgW00cBvPAKIpQ93g8TRqsCsFvR37nY9v5SU1hJ9araEp6LRA86tUt3EFtxD2tRcmqKGtCb0kOr+rRXWVGiaUBiedHxFudIy0+TNj4fNrJwpoC0hcBO9ek4Bh6hM7Zqzum9HpkPGlz84jkm+r+xMThaZCh8iFkgxyK+bw8B8smaVJD04ob9Wb4RYoFYfQ7TEkPkO9gr+UsSesbP9fWfucemUqwK0htOOtpc1YRwdOPcdPJWAfx9J5+YVL8BTWYBaTdmHslOKeUGM5fCob6QxC6Er9xweys9jh7MXttSpe013bY7P4ZDRF8uWvygpsytPE= amathuri@redhat.com", " 2021-05-25T11:53:24.896 INFO:teuthology.task.ansible.out:key_options": null, "keyfile": "/home/amathuri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amathuri", "validate_certs": true} 2021-05-25T11:53:25.101 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDr6H7hmtGCUrO3GppgMHwbmA4XT/Vipz/uLG/N8h6ZxtaLQKpCefRpdKcFlOdDwxkTmT5q9vmXQ8nl95lTGq+eeMxl4ho104ZD9e9AIYmgYqFy7/qNP8gkqduIPlypLL9m5rElTcGd7zqfMqgljMvAmnXLtZNJlIuFtszVU5WPI1QCwfQ3t5aHF6j66bpVoN9uWItVGB1LApQz9TtViWsQ9Hl8X4YauJToO+TkuSX+H/OGH7ZC6CPOzNZb3b3vF/jLMnTzXl86PJs1se4DjKkRGLVCVbr2WhLwQeZ8TEn3Hg1Gn5X8zNM5cTo6U8SnXBAe9ZYbuyfvghrNLdlZpmN+8l0/l5zC3z+N+sEzklXwHw4MQLF+KSloVFd9NN/kyenufTZAyp8VxllZ9TgdC+CPORIWyzmu1elOtYo7LNuE6CIh/Zm9B54SO6hVHhVoOCqe/dIyjeHJA0WKIZblK1XZEvWuu0E2T3jhxfe1GICyTukeB8WoBQH9DZ1yzMPt59E= nia@redhat.com", "key_options": null, "keyfile": "/home/nizamial09/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nizamial09", "validate_certs": true} 2021-05-25T11:53:25.103 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg7xW5+Mk7pG5MjLxOc7GTvDTCPBJj2L3MEQZIapmmo6t8uPODwDewXncSzrKDH6zFVE4TTTD9fJdu+BhyT+jCFgXrEOMMabj7vJsg0wO0SWqr+4NECFPhD9y4ewfYiewBeXisclA5Bto91v93AtYpZBCdmPYSi0BEcPVc6S3iBUD0PzIrL62I/kBAtm0f/bScRiT7XvQJXvkbEObW2yS2XhUv2qgq1VlFvO1jl3B85KMTHVb8+LrKpQWOKfRfxJ74c51y6tnEjZLFO6Xn0JevqOhd8gPQIXueFZOM0g0pNp1mbtGaaBlEIibbrzo2p9kzsdvJeHMq1XuMIi1/Qyj9 dpivonka@dhcp-41-144.bos.redhat.com", "key_options": null, "keyfile": "/home/dpivonka/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dpivonka", "validate_certs": true} 2021-05-25T11:53:25.299 INFO:teuthology.task.ansible.out:changed: [smithi181.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC//ZCaEeVN/J09FtsvuZ4eEs4rTBtYoQyG+ZpuWkJLL90VyJeYrZzRsgvveKW2W9oFxCFcMy7IBhLes0TZ/X/4Ox9eUmP/eyqK8fBqT3K1F1W/uOmKLFhlQhkxxj+p9BOMBD5trfES/QJQ6UFSYIS2WhWaitDgONjoLMR6wXZ9r7yZeQBxbmG4wGI0Cwx4Wgg+I59UbE/5iUfnX++gN3rXrPnv9V0yFeciS4HvXRFXdrIzvQFHcZMmpOOmGruHXN/ME2dZCG+kaZhZdI6kICSlOwpU7TU1lv99zjBCeEzGfFkjv041ykKdNVR3qE0k5u0AOMqxJBORL/woSCdGvrBXqju2G/2ohl3B4zD6TAve+JLfWOV5C20NFRPPbTd53JaNpV9OQqwOfVniowzSYYBdedjm9sq6YwhJt8iBXze3OA8ZO0GPXo7PC2MyrrUtu6uu3DZqU4x0cl8iJxqOlx5pH0Gydj8nuUG0+X6wHZzPXZRe5M7/RzmvXch9koG6FQMiikNjUkK8Guz4NMBdAYUpxYkdPGBhiuELoaAcgIVzSGoZTuICpNYwAxBHk6q18G8Qd8vaeUdGdim4YvwycsSOwFE47wg/XmpCHrNJUZzzB9UD7Kn84TG82LAmCuAzpzcp0uPjsBckbXzj9Zt0hRgF80yDgUazsQkbgtd2p8KxIQ== loic", "key_options": null, "keyfile": "/home/dachary/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dachary", "validate_certs": true} 2021-05-25T11:53:25.300 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0m+8wna4RnrOcuCUj971EBe4cFiTXVkEWkIsUX3yeCUwMaS/ZjaoOjcdYX7cYpCp1ilo4FSO5BsochU0RiMbQEwLGcE0zB2q1ac8+yYKCuWWyknig2t4bOReu5OrXvl6mj/9R+p7clFlYhwIoNG5ZZ/2eTbo3nH7gWgaOKaPbTtvEGg/5R/qjQKg8OTC8oix6kLsaNTisQJ78Zd7b4ukg9bO0jlq/2UhQOBM072sU1/kJpWTkcA0xxm3nsC4rWPwapgz7UwBmMdyx116GZgWdfQecTLKO7b8pyJw21npk2ntUmeMi7WPZGX6VI3IpaAhBjW3KiATAb8JbAQh8F9bmejkw6x7zrP3hYKQOg3A+U8DKRRo/Ad+f0zz9gjC42DUvMEOr09RBdxZcEAPY7H/YRWEqbA2zPX3Gfth4b7/letzM3m9ToUvVfSt1CdjaZD+gIia9pCgDMb+j1j2w9reG4F7Us+W/1f5pYCyFyrm4XO3AdoYM9/FOAOR3vffTIQU= adking@localhost.localdomain", "key_options": null, "keyfile": "/home/adking/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adking", "validate_certs": true} 2021-05-25T11:53:25.497 INFO:teuthology.task.ansible.out:changed: [smithi181.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDwVh9KqjV0b+65aBxQV66IUwNo3D5zTbN8U6Kh6eE4QtaIVe5Mmvi2SBbGMSWofE0zIIw2JmsI5LzdOoCyZl/3gNDPHQd6D9o0xRHQv94giZ2a5K+mW79foAxsuppTjV/Fbg8EZ/19W91+BqMWotlMR66ka0RPT/6U8V5mzXoydSminViO/B6q7ZYzIIgGZqaTsSxCMvz5BAzzeEaGfSlGxw5MOye9sF/tDppuRKG4rMUZULEWGuxgXAtabtXCduocAArqAY3BRJEgj7JetPdPNjipGrULyYQmT7GeknvP/waC0sPm5o8t7taJca7Vu9qKKMKPAqiK4KhlrW/yVBFliT2CjAdVuYvGj87LCcEn6NMeF+4ONwDz++YwjrAWQT2zHUJPvLnYOZPiD1LUm3F34cEHD1y4Uf+oeiYgriUMauskfFn+ShbTmWWooFRFyNobzweg9aMBZSkA+kFFifqOV1FwHTOYg342NumRGB8svK0vvTUID7AU1Ev8YXYqCEGbb0UHth/1pEbAF103enLTtPZ6qKua74rE0qJ89OF9dZXv4JY9+8Juyps7bi2XEyHNm1nHZOY5tqdY0eKFmSGJMoWBDzGgdMjmXlWp4jnY2fC619nZqxMrCQpeT70CqnBhtBemUuu6jHYUbbrPjnXIkH3FZ7/qvZt0jF0IMP9FGw== amctagga@redhat.com", "key_options": null, "keyfile": "/home/amctagga/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amctagga", "validate_certs": true} 2021-05-25T11:53:25.509 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCs6q2b5NcUsNXl+w1Mrc7idNvGdwWkL0XRX5zTw22UNpQyqPkSrRTFKi4NxoY8ucv8JoLnaJL7hYSRHKTh7SqH78uPLZX0FLtoHvpYLgUgJzzJ5AOW5uekIhCMCm+3egmFnEBwFcNxtGnU2es02ByXNEXRxt4/LQ0f8K6lMqxVRSKYcltvWzlA7WFFdTnYMunbCYOdpw88RXm2pYjGIHRBnie7+pFiFSS6K/mkzQcfLiJmahgLSaYslI0ho1vQd8xfNLmZ700C5F4VTUlT/mJK/dI2LF3OfaPTsyqKNzOchrjLFx5b6dhZ4WZyqu7VKsHNm+xHZwT8VP3LPGSVVBdmEsiCsmV4g4QTI5ssZLi7siZDpaFp72nVAIO1i6kBGuiF3or+J4LmIh8ZRKzTc0DRDOw7S+IkWu1fv2UhYg83z6wrz4HeczI+bnolFFWW6HgGV3MFaUeRTRvrbxw2DPoEGQ+cjP0wsjyEhxlScwdh8W8IbVpNl+7mBe45YGUH2hg83Lst5QQxD4PHelriu40moWRHFKCnKfNesb4L1uqu29o8f+/PgpLdrWE/P5zIKe2ReWJyScYAI7jTMlFw+xgf+0TKryk44NMy90l3ei0umlx4qsFbbEv+lhzAALrUJ/da7fsQw7eIVykmcscBeYr99/YSYk4MGdk4jVCiHsQkdw== amathuri@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJ3ch+pU88/tX5TVVFl/1pPief8drYLRkK4BtfJk+xVNxX9qkDtOtiLAU+OugYF1/xk9hE9zTbm3UBL23unrzhlR80t8f2FiF1YFhHVzeZ/p/OunmkU0RFQzkMVuwKVOUNulw9FwpEZ/0Bjfxvaims0SBWaCzBgE5g0R+Ofdq/bVF82K8vU5Laa3KZ/cCdGzFgW00cBvPAKIpQ93g8TRqsCsFvR37nY9v5SU1hJ9araEp6LRA86tUt3EFtxD2tRcmqKGtCb0kOr+rRXWVGiaUBiedHxFudIy0+TNj4fNrJwpoC0hcBO9ek4Bh6hM7Zqzum9HpkPGlz84jkm+r+xMThaZCh8iFkgxyK+bw8B8smaVJD04ob9Wb4RYoFYfQ7TEkPkO9gr+UsSesbP9fWfucemUqwK0htOOtpc1YRwdOPcdPJWAfx9J5+YVL8BTWYBaTdmHslOKeUGM5fCob6QxC6Er9xweys9jh7MXttSpe013bY7P4ZDRF8uWvygpsytPE= amathuri@redhat.com", " 2021-05-25T11:53:25.510 INFO:teuthology.task.ansible.out:key_options": null, "keyfile": "/home/amathuri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amathuri", "validate_certs": true} 2021-05-25T11:53:25.695 INFO:teuthology.task.ansible.out:changed: [smithi181.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDtiHTDuQk8EDH5R+JfDMucI4ZpOzKN5SLD+j35QLLySgGXholoDxtqjHnmRjM1rpj5SIaXjHBZdJi7iax+1MRywHYBi5BfTtzYPgA+NH8mt+XkcdkEuFPcYxGh6fMQDQWb2g9Euze/6qi0W9mbYXvvb1/Q3bhVCXg1UgKD42BndWEpbMYFfzJLZWPtOD/lg81pwUmf/4BpPFv52sy87lQVfkkvcUliLObR4S0TI5hPLYxBS+RXmHYKgvPMB2QJ9wum62lJ+YM1MxPygS0OWbIB6UQGXmvxRXA4osXkxJ8ksPPCupOgFNiiYro91cQh3mmi6iHc9kMz5R1aNGGOfDdlnLx9DRSIiNcacjXEVNFaoM0x441f4GeU2ZR8/7uq2dfk7QjPlh1eJGxG9pzRiQ8JCDF1car0eYwXdMTsQoQqpIi5imqcELdcqGGp9FhzFElgUwpf15JYlkouVhS6hq4LDjhLNk7dDT8fmCEnHOPcbxaTgN7J3idnJK6zeIniuuk= root@iZj6ce071s2zz3reioxn93Z", "key_options": null, "keyfile": "/home/kevinz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kevinz", "validate_certs": true} 2021-05-25T11:53:25.706 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDr6H7hmtGCUrO3GppgMHwbmA4XT/Vipz/uLG/N8h6ZxtaLQKpCefRpdKcFlOdDwxkTmT5q9vmXQ8nl95lTGq+eeMxl4ho104ZD9e9AIYmgYqFy7/qNP8gkqduIPlypLL9m5rElTcGd7zqfMqgljMvAmnXLtZNJlIuFtszVU5WPI1QCwfQ3t5aHF6j66bpVoN9uWItVGB1LApQz9TtViWsQ9Hl8X4YauJToO+TkuSX+H/OGH7ZC6CPOzNZb3b3vF/jLMnTzXl86PJs1se4DjKkRGLVCVbr2WhLwQeZ8TEn3Hg1Gn5X8zNM5cTo6U8SnXBAe9ZYbuyfvghrNLdlZpmN+8l0/l5zC3z+N+sEzklXwHw4MQLF+KSloVFd9NN/kyenufTZAyp8VxllZ9TgdC+CPORIWyzmu1elOtYo7LNuE6CIh/Zm9B54SO6hVHhVoOCqe/dIyjeHJA0WKIZblK1XZEvWuu0E2T3jhxfe1GICyTukeB8WoBQH9DZ1yzMPt59E= nia@redhat.com", "key_options": null, "keyfile": "/home/nizamial09/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nizamial09", "validate_certs": true} 2021-05-25T11:53:25.904 INFO:teuthology.task.ansible.out:changed: [smithi181.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1YAaxAvlRgu72huAggHpoDKSgtF/b00ACfUX4Wk5NyLFbYI3PAQapHNcNL4mIsH8pc6YkdjcJ5N/F12aEupJGQ0Q06OS324xapnaBoUy1hQl9IvhtPqTLEwR0X5xLIAhNBLIEJDNVg13MKwr0u5YVUXdrQRu7Kj9RXAPBahNZFo/Ed+cnR3LICQWliHeHFrk/DU2DUHol3a5wAPxrowAG0AxAecP6/7rASMmitvLQwdQcfIJAhWYgfyXzudg+gOmAEgv3O3U4QOikgmkm/EabUXLJnlTj6yuhtuuctdJr73UT3wLGAGmphLKCkug27kb1nTNoDa+PW/+grP63zhU1 chunmei@otccldstore05", "key_options": null, "keyfile": "/home/liu-chunmei/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "liu-chunmei", "validate_certs": true} 2021-05-25T11:53:25.915 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC//ZCaEeVN/J09FtsvuZ4eEs4rTBtYoQyG+ZpuWkJLL90VyJeYrZzRsgvveKW2W9oFxCFcMy7IBhLes0TZ/X/4Ox9eUmP/eyqK8fBqT3K1F1W/uOmKLFhlQhkxxj+p9BOMBD5trfES/QJQ6UFSYIS2WhWaitDgONjoLMR6wXZ9r7yZeQBxbmG4wGI0Cwx4Wgg+I59UbE/5iUfnX++gN3rXrPnv9V0yFeciS4HvXRFXdrIzvQFHcZMmpOOmGruHXN/ME2dZCG+kaZhZdI6kICSlOwpU7TU1lv99zjBCeEzGfFkjv041ykKdNVR3qE0k5u0AOMqxJBORL/woSCdGvrBXqju2G/2ohl3B4zD6TAve+JLfWOV5C20NFRPPbTd53JaNpV9OQqwOfVniowzSYYBdedjm9sq6YwhJt8iBXze3OA8ZO0GPXo7PC2MyrrUtu6uu3DZqU4x0cl8iJxqOlx5pH0Gydj8nuUG0+X6wHZzPXZRe5M7/RzmvXch9koG6FQMiikNjUkK8Guz4NMBdAYUpxYkdPGBhiuELoaAcgIVzSGoZTuICpNYwAxBHk6q18G8Qd8vaeUdGdim4YvwycsSOwFE47wg/XmpCHrNJUZzzB9UD7Kn84TG82LAmCuAzpzcp0uPjsBckbXzj9Zt0hRgF80yDgUazsQkbgtd2p8KxIQ== loic", "key_options": null, "keyfile": "/home/dachary/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dachary", "validate_certs": true} 2021-05-25T11:53:26.103 INFO:teuthology.task.ansible.out:changed: [smithi181.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDLbPdDb+XHOD2fo8g3RacATj0cG07uELwUvqQP35y+JmhNRSb/ttaysFLA5ss0y77TiXx22saP3ECEm5SnuPg5IERkPQmXiw84ATTNfLnzx2G5H223SLaTQiGfSgniBCJHVPCJfNahJiurlZoOAlwDz6SKXTUiivVrFJ22iTBDCQUO6HFQKjE7aM3RAo5mZXKVlTXzM6WOd8vlzk3HMzV0/91JR6H86Lt9MEzXT4dOzqmtxd8gS6U+vA+50zOIspdK+orTRJ0PO2Fn8c1RSYM8g7uqQtpDcOEHderW8TRsqBbDxpyslK6hRX5urUNBX506XE6JNoViLNvCofgseCGj yingxin@yingxin-server", "key_options": null, "keyfile": "/home/cyx1231st/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cyx1231st", "validate_certs": true} 2021-05-25T11:53:26.115 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDwVh9KqjV0b+65aBxQV66IUwNo3D5zTbN8U6Kh6eE4QtaIVe5Mmvi2SBbGMSWofE0zIIw2JmsI5LzdOoCyZl/3gNDPHQd6D9o0xRHQv94giZ2a5K+mW79foAxsuppTjV/Fbg8EZ/19W91+BqMWotlMR66ka0RPT/6U8V5mzXoydSminViO/B6q7ZYzIIgGZqaTsSxCMvz5BAzzeEaGfSlGxw5MOye9sF/tDppuRKG4rMUZULEWGuxgXAtabtXCduocAArqAY3BRJEgj7JetPdPNjipGrULyYQmT7GeknvP/waC0sPm5o8t7taJca7Vu9qKKMKPAqiK4KhlrW/yVBFliT2CjAdVuYvGj87LCcEn6NMeF+4ONwDz++YwjrAWQT2zHUJPvLnYOZPiD1LUm3F34cEHD1y4Uf+oeiYgriUMauskfFn+ShbTmWWooFRFyNobzweg9aMBZSkA+kFFifqOV1FwHTOYg342NumRGB8svK0vvTUID7AU1Ev8YXYqCEGbb0UHth/1pEbAF103enLTtPZ6qKua74rE0qJ89OF9dZXv4JY9+8Juyps7bi2XEyHNm1nHZOY5tqdY0eKFmSGJMoWBDzGgdMjmXlWp4jnY2fC619nZqxMrCQpeT70CqnBhtBemUuu6jHYUbbrPjnXIkH3FZ7/qvZt0jF0IMP9FGw== amctagga@redhat.com", "key_options": null, "keyfile": "/home/amctagga/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amctagga", "validate_certs": true} 2021-05-25T11:53:26.200 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:26 +0000 (0:00:28.195) 0:01:13.548 *********** 2021-05-25T11:53:26.280 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-05-25T11:53:26.282 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:26.291 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:26.293 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:26.303 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:26.304 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:26.314 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:26.325 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-05-25T11:53:26.327 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDtiHTDuQk8EDH5R+JfDMucI4ZpOzKN5SLD+j35QLLySgGXholoDxtqjHnmRjM1rpj5SIaXjHBZdJi7iax+1MRywHYBi5BfTtzYPgA+NH8mt+XkcdkEuFPcYxGh6fMQDQWb2g9Euze/6qi0W9mbYXvvb1/Q3bhVCXg1UgKD42BndWEpbMYFfzJLZWPtOD/lg81pwUmf/4BpPFv52sy87lQVfkkvcUliLObR4S0TI5hPLYxBS+RXmHYKgvPMB2QJ9wum62lJ+YM1MxPygS0OWbIB6UQGXmvxRXA4osXkxJ8ksPPCupOgFNiiYro91cQh3mmi6iHc9kMz5R1aNGGOfDdlnLx9DRSIiNcacjXEVNFaoM0x441f4GeU2ZR8/7uq2dfk7QjPlh1eJGxG9pzRiQ8JCDF1car0eYwXdMTsQoQqpIi5imqcELdcqGGp9FhzFElgUwpf15JYlkouVhS6hq4LDjhLNk7dDT8fmCEnHOPcbxaTgN7J3idnJK6zeIniuuk= root@iZj6ce071s2zz3reioxn93Z", "key_options": null, "keyfile": "/home/kevinz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kevinz", "validate_certs": true} 2021-05-25T11:53:26.452 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-05-25T11:53:26.453 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:26.463 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:26.464 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:26.526 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-05-25T11:53:26.528 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1YAaxAvlRgu72huAggHpoDKSgtF/b00ACfUX4Wk5NyLFbYI3PAQapHNcNL4mIsH8pc6YkdjcJ5N/F12aEupJGQ0Q06OS324xapnaBoUy1hQl9IvhtPqTLEwR0X5xLIAhNBLIEJDNVg13MKwr0u5YVUXdrQRu7Kj9RXAPBahNZFo/Ed+cnR3LICQWliHeHFrk/DU2DUHol3a5wAPxrowAG0AxAecP6/7rASMmitvLQwdQcfIJAhWYgfyXzudg+gOmAEgv3O3U4QOikgmkm/EabUXLJnlTj6yuhtuuctdJr73UT3wLGAGmphLKCkug27kb1nTNoDa+PW/+grP63zhU1 chunmei@otccldstore05", "key_options": null, "keyfile": "/home/liu-chunmei/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "liu-chunmei", "validate_certs": true} 2021-05-25T11:53:26.735 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDLbPdDb+XHOD2fo8g3RacATj0cG07uELwUvqQP35y+JmhNRSb/ttaysFLA5ss0y77TiXx22saP3ECEm5SnuPg5IERkPQmXiw84ATTNfLnzx2G5H223SLaTQiGfSgniBCJHVPCJfNahJiurlZoOAlwDz6SKXTUiivVrFJ22iTBDCQUO6HFQKjE7aM3RAo5mZXKVlTXzM6WOd8vlzk3HMzV0/91JR6H86Lt9MEzXT4dOzqmtxd8gS6U+vA+50zOIspdK+orTRJ0PO2Fn8c1RSYM8g7uqQtpDcOEHderW8TRsqBbDxpyslK6hRX5urUNBX506XE6JNoViLNvCofgseCGj yingxin@yingxin-server", "key_options": null, "keyfile": "/home/cyx1231st/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cyx1231st", "validate_certs": true} 2021-05-25T11:53:26.860 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2021-05-25T11:53:26.887 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-05-25T11:53:26.946 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-05-25T11:53:26.948 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:26.969 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:26 +0000 (0:00:00.768) 0:01:14.317 *********** 2021-05-25T11:53:26.977 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2021-05-25T11:53:27.053 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.054 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.064 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.066 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.075 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.086 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.088 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.097 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.099 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.161 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2021-05-25T11:53:27.163 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.172 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.183 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.185 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.194 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.196 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.206 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.207 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.217 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.228 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.230 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.240 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.241 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.251 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.262 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.263 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.274 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.276 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.286 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.287 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.297 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.299 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.309 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2021-05-25T11:53:27.311 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.321 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.323 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.333 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.334 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.344 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.346 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.356 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.366 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.368 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.378 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.379 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.389 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.391 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.400 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.411 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.413 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.423 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.424 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.434 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.436 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.445 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.456 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.458 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.467 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.469 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.479 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.480 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.490 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.501 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.502 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.512 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.514 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.524 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2021-05-25T11:53:27.524 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.535 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.537 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.547 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.548 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.558 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.560 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.569 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.580 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.582 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.591 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.592 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.603 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.605 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.615 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.616 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.626 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.637 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.638 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.648 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.650 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.659 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.661 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.671 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.682 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.683 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.693 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.694 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.704 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.706 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.715 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.716 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2021-05-25T11:53:27.718 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.728 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.729 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee 2021-05-25T11:53:27.731 INFO:teuthology.task.ansible.out:6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.740 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.742 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.752 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.754 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.764 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.764 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.765 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.765 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.776 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.778 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.788 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.790 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.800 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.801 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.812 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.813 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.813 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.815 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.824 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.826 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.837 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.838 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.848 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.850 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.860 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.862 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.862 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.873 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.874 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.885 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.886 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.897 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.898 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.909 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.911 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.922 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.923 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.934 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.936 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.946 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.948 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.959 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.960 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.971 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.973 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "skip_reason": "Conditional result was False"} skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.983 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.985 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "skip_reason": "Conditional result was False"} skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.995 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.996 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.996 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:27.996 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.007 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.027 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.029 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.029 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.051 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:28 +0000 (0:00:01.081) 0:01:15.399 *********** 2021-05-25T11:53:28.058 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.059 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.072 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.073 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.073 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.085 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.085 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.096 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.107 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.109 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.119 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.120 INFO:teuthology.task.ansible.out: TASK [Filter the revoked_users list] ******************************************* 2021-05-25T11:53:28.121 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.122 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.154 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:28 +0000 (0:00:00.103) 0:01:15.502 *********** 2021-05-25T11:53:28.163 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.165 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.166 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.167 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.168 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.169 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.171 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.171 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.183 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.185 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.195 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.196 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.206 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.217 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.219 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.229 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.231 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.240 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.251 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.253 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.263 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.264 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.274 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.276 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.286 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.287 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.297 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.308 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.310 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.320 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.321 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.331 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.333 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.342 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.354 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.355 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.365 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.367 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.376 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.378 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.388 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.390 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.399 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.400 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.411 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.422 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.424 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.434 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.434 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-05-25T11:53:28.435 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2021-05-25T11:53:28.435 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-05-25T11:53:28.436 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.446 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.447 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.458 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.469 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.470 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.480 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.482 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.492 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.494 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.503 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.514 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.516 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.526 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.527 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.538 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.540 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.550 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.561 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.606 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:28 +0000 (0:00:00.451) 0:01:15.954 *********** 2021-05-25T11:53:28.624 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-05-25T11:53:28.626 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2021-05-25T11:53:28.677 INFO:teuthology.task.ansible.out: TASK [Filter the revoked_users list] ******************************************* 2021-05-25T11:53:28.679 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:28.698 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:28 +0000 (0:00:00.092) 0:01:16.046 *********** 2021-05-25T11:53:28.820 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-05-25T11:53:28.822 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2021-05-25T11:53:28.967 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2021-05-25T11:53:29.009 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2021-05-25T11:53:29.155 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2021-05-25T11:53:29.198 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2021-05-25T11:53:29.343 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2021-05-25T11:53:29.385 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2021-05-25T11:53:29.532 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2021-05-25T11:53:29.574 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2021-05-25T11:53:29.730 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2021-05-25T11:53:29.762 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2021-05-25T11:53:29.920 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2021-05-25T11:53:29.952 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} 2021-05-25T11:53:30.108 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2021-05-25T11:53:30.140 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2021-05-25T11:53:30.307 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2021-05-25T11:53:30.328 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2021-05-25T11:53:30.495 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} 2021-05-25T11:53:30.517 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2021-05-25T11:53:30.683 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2021-05-25T11:53:30.705 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2021-05-25T11:53:30.872 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2021-05-25T11:53:30.894 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2021-05-25T11:53:31.060 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2021-05-25T11:53:31.082 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2021-05-25T11:53:31.249 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2021-05-25T11:53:31.271 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2021-05-25T11:53:31.438 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2021-05-25T11:53:31.460 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2021-05-25T11:53:31.618 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2021-05-25T11:53:31.650 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2021-05-25T11:53:31.807 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2021-05-25T11:53:31.840 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2021-05-25T11:53:32.006 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2021-05-25T11:53:32.018 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2021-05-25T11:53:32.195 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2021-05-25T11:53:32.206 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2021-05-25T11:53:32.384 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2021-05-25T11:53:32.395 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2021-05-25T11:53:32.573 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2021-05-25T11:53:32.595 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2021-05-25T11:53:32.761 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2021-05-25T11:53:32.783 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2021-05-25T11:53:32.950 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2021-05-25T11:53:32.972 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2021-05-25T11:53:33.139 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2021-05-25T11:53:33.161 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2021-05-25T11:53:33.328 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2021-05-25T11:53:33.349 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2021-05-25T11:53:33.525 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2021-05-25T11:53:33.536 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2021-05-25T11:53:33.712 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2021-05-25T11:53:33.723 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2021-05-25T11:53:33.900 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2021-05-25T11:53:33.911 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2021-05-25T11:53:34.088 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2021-05-25T11:53:34.100 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2021-05-25T11:53:34.277 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2021-05-25T11:53:34.288 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2021-05-25T11:53:34.476 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2021-05-25T11:53:34.477 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2021-05-25T11:53:34.664 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2021-05-25T11:53:34.665 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2021-05-25T11:53:34.852 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2021-05-25T11:53:34.853 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2021-05-25T11:53:35.041 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2021-05-25T11:53:35.042 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2021-05-25T11:53:35.220 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2021-05-25T11:53:35.231 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": false, "item": "dorinda", "name": "dorinda", "state": "absent"} 2021-05-25T11:53:35.409 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2021-05-25T11:53:35.421 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item=zyan) => {"ansible_loop_var": "item", "changed": false, "item": "zyan", "name": "zyan", "state": "absent"} 2021-05-25T11:53:35.599 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2021-05-25T11:53:35.786 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": false, "item": "dorinda", "name": "dorinda", "state": "absent"} 2021-05-25T11:53:35.973 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=zyan) => {"ansible_loop_var": "item", "changed": false, "item": "zyan", "name": "zyan", "state": "absent"} 2021-05-25T11:53:37.089 INFO:teuthology.task.ansible.out:changed: [smithi181.front.sepia.ceph.com] => (item=jdillaman) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "jdillaman", "name": "jdillaman", "remove": false, "state": "absent"} 2021-05-25T11:53:37.287 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2021-05-25T11:53:37.475 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2021-05-25T11:53:37.497 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item=jdillaman) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "jdillaman", "name": "jdillaman", "remove": false, "state": "absent"} 2021-05-25T11:53:37.668 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2021-05-25T11:53:37.701 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2021-05-25T11:53:37.857 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2021-05-25T11:53:37.889 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2021-05-25T11:53:38.045 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2021-05-25T11:53:38.077 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2021-05-25T11:53:38.244 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2021-05-25T11:53:38.265 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2021-05-25T11:53:38.433 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2021-05-25T11:53:38.455 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2021-05-25T11:53:38.614 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2021-05-25T11:53:38.648 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2021-05-25T11:53:38.804 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} 2021-05-25T11:53:38.842 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2021-05-25T11:53:38.994 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2021-05-25T11:53:39.027 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2021-05-25T11:53:39.050 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:39 +0000 (0:00:10.351) 0:01:26.398 *********** 2021-05-25T11:53:39.203 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} 2021-05-25T11:53:39.400 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2021-05-25T11:53:39.440 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:39 +0000 (0:00:00.389) 0:01:26.788 *********** 2021-05-25T11:53:39.570 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2021-05-25T11:53:39.572 INFO:teuthology.task.ansible.out:changed: [smithi181.front.sepia.ceph.com] => {"changed": true, "checksum": "09f175eb7e6987d91e85c0190334c5a90418c4b8", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "0e849644a39b9e2ba2a1abd6a16b889e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1621943619.1411104-1909-252674200740482/source", "state": "file", "uid": 0} 2021-05-25T11:53:39.592 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:39 +0000 (0:00:00.152) 0:01:26.940 *********** 2021-05-25T11:53:39.659 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2021-05-25T11:53:39.661 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => { "msg": "Host smithi181.front.sepia.ceph.com is running CentOS 8.3 (n/a)" } 2021-05-25T11:53:39.680 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:39 +0000 (0:00:00.088) 0:01:27.028 *********** 2021-05-25T11:53:39.748 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2021-05-25T11:53:39.749 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml) => {"ansible_facts": {"nrpe_selinux_packages": ["python3-libsemanage", "python3-policycoreutils"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"} 2021-05-25T11:53:39.769 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:39 +0000 (0:00:00.088) 0:01:27.117 *********** 2021-05-25T11:53:39.836 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2021-05-25T11:53:39.838 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:39.858 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:39 +0000 (0:00:00.088) 0:01:27.206 *********** 2021-05-25T11:53:39.966 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2021-05-25T11:53:39.967 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => {"changed": true, "checksum": "09f175eb7e6987d91e85c0190334c5a90418c4b8", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "0e849644a39b9e2ba2a1abd6a16b889e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1621943619.5283651-1918-172167483185405/source", "state": "file", "uid": 0} 2021-05-25T11:53:39.987 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:39 +0000 (0:00:00.129) 0:01:27.335 *********** 2021-05-25T11:53:40.065 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2021-05-25T11:53:40.067 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => { "msg": "Host smithi102.front.sepia.ceph.com is running CentOS 8.3 (n/a)" } 2021-05-25T11:53:40.088 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:40 +0000 (0:00:00.100) 0:01:27.436 *********** 2021-05-25T11:53:40.156 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2021-05-25T11:53:40.158 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml) => {"ansible_facts": {"nrpe_selinux_packages": ["python3-libsemanage", "python3-policycoreutils"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"} 2021-05-25T11:53:40.179 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:40 +0000 (0:00:00.090) 0:01:27.527 *********** 2021-05-25T11:53:40.246 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2021-05-25T11:53:40.248 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:40.267 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:40 +0000 (0:00:00.088) 0:01:27.615 *********** 2021-05-25T11:53:40.450 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2021-05-25T11:53:40.451 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.371289", "end": "2021-05-25 11:53:40.430695", "rc": 0, "start": "2021-05-25 11:53:40.059406", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2021-05-25T11:53:40.475 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:40 +0000 (0:00:00.207) 0:01:27.823 *********** 2021-05-25T11:53:40.543 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2021-05-25T11:53:40.545 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:40.566 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:40 +0000 (0:00:00.091) 0:01:27.914 *********** 2021-05-25T11:53:40.634 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2021-05-25T11:53:40.636 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:40.657 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:40 +0000 (0:00:00.090) 0:01:28.005 *********** 2021-05-25T11:53:40.726 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2021-05-25T11:53:40.728 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:40.748 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:40 +0000 (0:00:00.091) 0:01:28.096 *********** 2021-05-25T11:53:40.816 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2021-05-25T11:53:40.818 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:40.838 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:40 +0000 (0:00:00.089) 0:01:28.186 *********** 2021-05-25T11:53:40.844 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2021-05-25T11:53:40.846 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.338683", "end": "2021-05-25 11:53:40.807608", "rc": 0, "start": "2021-05-25 11:53:40.468925", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2021-05-25T11:53:40.868 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:40 +0000 (0:00:00.030) 0:01:28.216 *********** 2021-05-25T11:53:40.906 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2021-05-25T11:53:40.908 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:40.929 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:40 +0000 (0:00:00.060) 0:01:28.277 *********** 2021-05-25T11:53:40.946 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2021-05-25T11:53:40.947 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:40.969 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:40 +0000 (0:00:00.040) 0:01:28.317 *********** 2021-05-25T11:53:40.997 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2021-05-25T11:53:40.999 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:41.020 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:41 +0000 (0:00:00.050) 0:01:28.368 *********** 2021-05-25T11:53:41.038 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2021-05-25T11:53:41.040 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:41.060 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:41 +0000 (0:00:00.040) 0:01:28.408 *********** 2021-05-25T11:53:41.098 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2021-05-25T11:53:41.100 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi181.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-05-25T11:53:41.121 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:41 +0000 (0:00:00.060) 0:01:28.469 *********** 2021-05-25T11:53:41.137 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2021-05-25T11:53:41.139 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:41.160 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:41 +0000 (0:00:00.038) 0:01:28.508 *********** 2021-05-25T11:53:41.197 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2021-05-25T11:53:41.199 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:41.221 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:41 +0000 (0:00:00.060) 0:01:28.569 *********** 2021-05-25T11:53:41.238 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2021-05-25T11:53:41.239 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:41.260 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:41 +0000 (0:00:00.039) 0:01:28.609 *********** 2021-05-25T11:53:41.298 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2021-05-25T11:53:41.300 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:41.325 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:41 +0000 (0:00:00.064) 0:01:28.673 *********** 2021-05-25T11:53:41.342 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2021-05-25T11:53:41.344 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:41.366 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:41 +0000 (0:00:00.040) 0:01:28.714 *********** 2021-05-25T11:53:41.394 INFO:teuthology.task.ansible.out: TASK [common : Set the releasever] ********************************************* 2021-05-25T11:53:41.396 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:41.418 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:41 +0000 (0:00:00.051) 0:01:28.766 *********** 2021-05-25T11:53:41.435 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2021-05-25T11:53:41.437 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:41.458 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:41 +0000 (0:00:00.040) 0:01:28.806 *********** 2021-05-25T11:53:41.486 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2021-05-25T11:53:41.487 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-05-25T11:53:41.510 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:41 +0000 (0:00:00.051) 0:01:28.858 *********** 2021-05-25T11:53:41.537 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2021-05-25T11:53:41.539 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi102.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-05-25T11:53:41.562 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:41 +0000 (0:00:00.051) 0:01:28.909 *********** 2021-05-25T11:53:41.578 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2021-05-25T11:53:41.580 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:41.602 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:41 +0000 (0:00:00.040) 0:01:28.950 *********** 2021-05-25T11:53:41.630 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2021-05-25T11:53:41.631 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:41.653 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:41 +0000 (0:00:00.051) 0:01:29.001 *********** 2021-05-25T11:53:41.670 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2021-05-25T11:53:41.672 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-05-25T11:53:41.693 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:41 +0000 (0:00:00.039) 0:01:29.041 *********** 2021-05-25T11:53:41.720 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2021-05-25T11:53:41.722 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:41.743 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:41 +0000 (0:00:00.049) 0:01:29.091 *********** 2021-05-25T11:53:41.760 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2021-05-25T11:53:41.761 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:41.784 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:41 +0000 (0:00:00.040) 0:01:29.132 *********** 2021-05-25T11:53:41.811 INFO:teuthology.task.ansible.out: TASK [common : Set the releasever] ********************************************* 2021-05-25T11:53:41.813 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:41.834 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:41 +0000 (0:00:00.050) 0:01:29.182 *********** 2021-05-25T11:53:41.851 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2021-05-25T11:53:41.853 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:41.876 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:41 +0000 (0:00:00.041) 0:01:29.224 *********** 2021-05-25T11:53:41.903 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2021-05-25T11:53:41.905 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-05-25T11:53:41.927 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:41 +0000 (0:00:00.051) 0:01:29.275 *********** 2021-05-25T11:53:41.954 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2021-05-25T11:53:41.955 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:41.977 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:41 +0000 (0:00:00.050) 0:01:29.325 *********** 2021-05-25T11:53:41.994 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2021-05-25T11:53:41.996 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:42.018 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:42 +0000 (0:00:00.040) 0:01:29.366 *********** 2021-05-25T11:53:42.055 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2021-05-25T11:53:42.057 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:42.079 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:42 +0000 (0:00:00.061) 0:01:29.427 *********** 2021-05-25T11:53:42.096 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2021-05-25T11:53:42.098 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-05-25T11:53:42.119 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:42 +0000 (0:00:00.040) 0:01:29.467 *********** 2021-05-25T11:53:42.146 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2021-05-25T11:53:42.148 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-05-25T11:53:42.170 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:42 +0000 (0:00:00.050) 0:01:29.518 *********** 2021-05-25T11:53:42.197 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2021-05-25T11:53:42.199 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:42.220 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:42 +0000 (0:00:00.050) 0:01:29.568 *********** 2021-05-25T11:53:42.247 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-05-25T11:53:42.249 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:42.270 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:42 +0000 (0:00:00.050) 0:01:29.618 *********** 2021-05-25T11:53:42.298 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2021-05-25T11:53:42.299 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:42.325 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:42 +0000 (0:00:00.054) 0:01:29.673 *********** 2021-05-25T11:53:42.352 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2021-05-25T11:53:42.354 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:42.376 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:42 +0000 (0:00:00.051) 0:01:29.724 *********** 2021-05-25T11:53:42.413 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2021-05-25T11:53:42.415 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:42.439 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:42 +0000 (0:00:00.063) 0:01:29.787 *********** 2021-05-25T11:53:42.455 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2021-05-25T11:53:42.457 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:42.479 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:42 +0000 (0:00:00.039) 0:01:29.827 *********** 2021-05-25T11:53:42.505 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2021-05-25T11:53:42.507 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:42.529 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:42 +0000 (0:00:00.050) 0:01:29.877 *********** 2021-05-25T11:53:42.546 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2021-05-25T11:53:42.548 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:42.570 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:42 +0000 (0:00:00.040) 0:01:29.918 *********** 2021-05-25T11:53:42.597 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2021-05-25T11:53:42.598 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-05-25T11:53:42.621 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:42 +0000 (0:00:00.051) 0:01:29.969 *********** 2021-05-25T11:53:42.638 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2021-05-25T11:53:42.640 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:42.662 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:42 +0000 (0:00:00.041) 0:01:30.010 *********** 2021-05-25T11:53:42.689 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-05-25T11:53:42.691 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:42.713 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:42 +0000 (0:00:00.050) 0:01:30.061 *********** 2021-05-25T11:53:42.730 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2021-05-25T11:53:42.732 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:42.754 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:42 +0000 (0:00:00.040) 0:01:30.102 *********** 2021-05-25T11:53:42.780 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2021-05-25T11:53:42.782 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:42.804 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:42 +0000 (0:00:00.050) 0:01:30.152 *********** 2021-05-25T11:53:42.821 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2021-05-25T11:53:42.823 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-05-25T11:53:42.845 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:42 +0000 (0:00:00.040) 0:01:30.193 *********** 2021-05-25T11:53:42.871 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2021-05-25T11:53:42.873 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:42.895 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:42 +0000 (0:00:00.049) 0:01:30.243 *********** 2021-05-25T11:53:42.912 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2021-05-25T11:53:42.913 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:42.935 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:42 +0000 (0:00:00.040) 0:01:30.283 *********** 2021-05-25T11:53:42.962 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2021-05-25T11:53:42.963 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:42.985 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:42 +0000 (0:00:00.050) 0:01:30.333 *********** 2021-05-25T11:53:43.002 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2021-05-25T11:53:43.004 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:43.026 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:43 +0000 (0:00:00.040) 0:01:30.374 *********** 2021-05-25T11:53:43.052 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2021-05-25T11:53:43.054 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:43.076 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:43 +0000 (0:00:00.050) 0:01:30.424 *********** 2021-05-25T11:53:43.156 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2021-05-25T11:53:43.157 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:43.178 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:43 +0000 (0:00:00.101) 0:01:30.526 *********** 2021-05-25T11:53:43.246 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2021-05-25T11:53:43.248 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-05-25T11:53:43.268 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:43 +0000 (0:00:00.089) 0:01:30.616 *********** 2021-05-25T11:53:43.283 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2021-05-25T11:53:43.285 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-05-25T11:53:43.308 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:43 +0000 (0:00:00.039) 0:01:30.656 *********** 2021-05-25T11:53:43.335 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2021-05-25T11:53:43.337 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:43.358 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:43 +0000 (0:00:00.050) 0:01:30.706 *********** 2021-05-25T11:53:43.436 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2021-05-25T11:53:43.437 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:43.458 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:43 +0000 (0:00:00.100) 0:01:30.806 *********** 2021-05-25T11:53:43.718 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2021-05-25T11:53:43.719 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-05-25T11:53:43.739 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:43 +0000 (0:00:00.281) 0:01:31.088 *********** 2021-05-25T11:53:43.957 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2021-05-25T11:53:43.958 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "86c0d58bde7ad4cfaafe592d2fefd92c920c42ca", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 236, "state": "file", "uid": 0} 2021-05-25T11:53:44.258 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "86c0d58bde7ad4cfaafe592d2fefd92c920c42ca", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 236, "state": "file", "uid": 0} 2021-05-25T11:53:44.378 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a1bad5509ed35b895c2061841236b1cfe10619ae", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 262, "state": "file", "uid": 0} 2021-05-25T11:53:44.404 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:44 +0000 (0:00:00.663) 0:01:31.752 *********** 2021-05-25T11:53:44.679 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a1bad5509ed35b895c2061841236b1cfe10619ae", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 262, "state": "file", "uid": 0} 2021-05-25T11:53:44.704 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:44 +0000 (0:00:00.300) 0:01:32.052 *********** 2021-05-25T11:53:44.897 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2021-05-25T11:53:44.898 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "3c9ba7bacd1e30bed014cefd1fb4eab0f0f8fa10", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 484, "state": "file", "uid": 0} 2021-05-25T11:53:45.199 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "3c9ba7bacd1e30bed014cefd1fb4eab0f0f8fa10", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 484, "state": "file", "uid": 0} 2021-05-25T11:53:45.323 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "d542c3e2c30a0d3eb7d454775342c302c0ded15b", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 540, "state": "file", "uid": 0} 2021-05-25T11:53:45.348 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:45 +0000 (0:00:00.643) 0:01:32.696 *********** 2021-05-25T11:53:45.426 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-05-25T11:53:45.428 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:45.448 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:45 +0000 (0:00:00.100) 0:01:32.796 *********** 2021-05-25T11:53:45.516 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-05-25T11:53:45.517 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:45.538 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:45 +0000 (0:00:00.089) 0:01:32.886 *********** 2021-05-25T11:53:45.605 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2021-05-25T11:53:45.607 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:45.627 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:45 +0000 (0:00:00.089) 0:01:32.975 *********** 2021-05-25T11:53:45.633 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2021-05-25T11:53:45.633 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "d542c3e2c30a0d3eb7d454775342c302c0ded15b", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 540, "state": "file", "uid": 0} 2021-05-25T11:53:45.660 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:45 +0000 (0:00:00.032) 0:01:33.008 *********** 2021-05-25T11:53:45.696 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2021-05-25T11:53:45.698 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:45.719 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:45 +0000 (0:00:00.059) 0:01:33.067 *********** 2021-05-25T11:53:45.736 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-05-25T11:53:45.738 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:45.762 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:45 +0000 (0:00:00.042) 0:01:33.110 *********** 2021-05-25T11:53:45.789 INFO:teuthology.task.ansible.out: TASK [common : Mask sleep units] *********************************************** 2021-05-25T11:53:45.791 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item=sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "sleep.target", "skip_reason": "Conditional result was False"} 2021-05-25T11:53:45.800 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item=suspend.target) => {"ansible_loop_var": "item", "changed": false, "item": "suspend.target", "skip_reason": "Conditional result was False"} 2021-05-25T11:53:45.802 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item=hibernate.target) => {"ansible_loop_var": "item", "changed": false, "item": "hibernate.target", "skip_reason": "Conditional result was False"} skipping: [smithi181.front.sepia.ceph.com] => (item=hybrid-sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "hybrid-sleep.target", "skip_reason": "Conditional result was False"} 2021-05-25T11:53:45.825 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:45 +0000 (0:00:00.063) 0:01:33.173 *********** 2021-05-25T11:53:45.831 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-05-25T11:53:45.832 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:45.855 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:45 +0000 (0:00:00.029) 0:01:33.203 *********** 2021-05-25T11:53:45.882 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-05-25T11:53:45.884 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:45.906 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:45 +0000 (0:00:00.050) 0:01:33.254 *********** 2021-05-25T11:53:45.922 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2021-05-25T11:53:45.924 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:45.946 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:45 +0000 (0:00:00.040) 0:01:33.294 *********** 2021-05-25T11:53:45.973 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2021-05-25T11:53:45.975 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:45.997 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:45 +0000 (0:00:00.050) 0:01:33.345 *********** 2021-05-25T11:53:46.014 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2021-05-25T11:53:46.016 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:46.038 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:46 +0000 (0:00:00.040) 0:01:33.386 *********** 2021-05-25T11:53:46.065 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2021-05-25T11:53:46.067 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:46.088 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:46 +0000 (0:00:00.050) 0:01:33.436 *********** 2021-05-25T11:53:46.106 INFO:teuthology.task.ansible.out: TASK [common : Mask sleep units] *********************************************** skipping: [smithi102.front.sepia.ceph.com] => (item=sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "sleep.target", "skip_reason": "Conditional result was False"} 2021-05-25T11:53:46.116 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item=suspend.target) => {"ansible_loop_var": "item", "changed": false, "item": "suspend.target", "skip_reason": "Conditional result was False"} 2021-05-25T11:53:46.118 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item=hibernate.target) => {"ansible_loop_var": "item", "changed": false, "item": "hibernate.target", "skip_reason": "Conditional result was False"} 2021-05-25T11:53:46.128 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item=hybrid-sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "hybrid-sleep.target", "skip_reason": "Conditional result was False"} 2021-05-25T11:53:46.153 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:46 +0000 (0:00:00.064) 0:01:33.501 *********** 2021-05-25T11:53:46.159 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2021-05-25T11:53:46.160 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:46.184 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:46 +0000 (0:00:00.031) 0:01:33.532 *********** 2021-05-25T11:53:46.223 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:46.245 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:46 +0000 (0:00:00.061) 0:01:33.593 *********** 2021-05-25T11:53:46.263 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2021-05-25T11:53:46.264 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:46.287 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:46 +0000 (0:00:00.041) 0:01:33.635 *********** 2021-05-25T11:53:46.325 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2021-05-25T11:53:46.326 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:46.352 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:46 +0000 (0:00:00.065) 0:01:33.700 *********** 2021-05-25T11:53:46.421 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2021-05-25T11:53:46.423 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:46.444 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:46 +0000 (0:00:00.091) 0:01:33.792 *********** 2021-05-25T11:53:46.512 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2021-05-25T11:53:46.513 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:46.533 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:46 +0000 (0:00:00.089) 0:01:33.881 *********** 2021-05-25T11:53:46.540 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2021-05-25T11:53:46.542 INFO:teuthology.task.ansible.out:changed: [smithi181.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.048283", "end": "2021-05-25 11:53:46.500349", "rc": 0, "start": "2021-05-25 11:53:46.452066", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-05-25T11:53:46.565 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:46 +0000 (0:00:00.031) 0:01:33.913 *********** 2021-05-25T11:53:46.602 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2021-05-25T11:53:46.604 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:46.625 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:46 +0000 (0:00:00.060) 0:01:33.973 *********** 2021-05-25T11:53:47.522 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2021-05-25T11:53:47.523 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.712248", "end": "2021-05-25 11:53:47.500363", "rc": 0, "start": "2021-05-25 11:53:46.788115", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-05-25T11:53:47.544 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:47 +0000 (0:00:00.918) 0:01:34.892 *********** 2021-05-25T11:53:50.943 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2021-05-25T11:53:50.945 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-05-25T11:53:50.967 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:50 +0000 (0:00:03.423) 0:01:38.316 *********** 2021-05-25T11:53:51.036 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2021-05-25T11:53:51.038 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:51.060 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:51 +0000 (0:00:00.091) 0:01:38.408 *********** 2021-05-25T11:53:51.129 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2021-05-25T11:53:51.130 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:51.151 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:51 +0000 (0:00:00.091) 0:01:38.499 *********** 2021-05-25T11:53:51.219 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2021-05-25T11:53:51.221 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:51.242 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:51 +0000 (0:00:00.089) 0:01:38.589 *********** 2021-05-25T11:53:51.298 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2021-05-25T11:53:51.300 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:51.320 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:51 +0000 (0:00:00.079) 0:01:38.668 *********** 2021-05-25T11:53:51.575 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2021-05-25T11:53:51.576 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-05-25T11:53:51.598 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:51 +0000 (0:00:00.277) 0:01:38.946 *********** 2021-05-25T11:53:51.667 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2021-05-25T11:53:51.669 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:51.689 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:51 +0000 (0:00:00.090) 0:01:39.037 *********** 2021-05-25T11:53:51.757 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2021-05-25T11:53:51.759 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:51.779 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:51 +0000 (0:00:00.090) 0:01:39.127 *********** 2021-05-25T11:53:51.796 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2021-05-25T11:53:51.797 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2021-05-25T11:53:51.820 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:51 +0000 (0:00:00.040) 0:01:39.168 *********** 2021-05-25T11:53:51.847 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2021-05-25T11:53:51.849 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:51.870 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:51 +0000 (0:00:00.050) 0:01:39.219 *********** 2021-05-25T11:53:51.940 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2021-05-25T11:53:51.942 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:51.963 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:51 +0000 (0:00:00.092) 0:01:39.311 *********** 2021-05-25T11:53:52.454 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2021-05-25T11:53:52.456 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2021-05-25T11:53:52.477 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:52 +0000 (0:00:00.514) 0:01:39.825 *********** 2021-05-25T11:53:53.184 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2021-05-25T11:53:53.185 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-05-25T11:53:53.206 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:53 +0000 (0:00:00.729) 0:01:40.554 *********** 2021-05-25T11:53:53.722 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-05-25T11:53:53.724 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2021-05-25T11:53:53.844 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2021-05-25T11:53:53.846 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-05-25T11:53:53.867 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:53 +0000 (0:00:00.661) 0:01:41.215 *********** 2021-05-25T11:53:54.155 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-05-25T11:53:54.156 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2021-05-25T11:53:54.374 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2021-05-25T11:53:54.566 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2021-05-25T11:53:54.591 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:54 +0000 (0:00:00.723) 0:01:41.939 *********** 2021-05-25T11:53:54.773 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2021-05-25T11:53:54.775 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-05-25T11:53:54.796 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:54 +0000 (0:00:00.205) 0:01:42.144 *********** 2021-05-25T11:53:54.812 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-05-25T11:53:54.814 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2021-05-25T11:53:55.240 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2021-05-25T11:53:55.264 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:55 +0000 (0:00:00.468) 0:01:42.612 *********** 2021-05-25T11:53:55.272 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-05-25T11:53:55.273 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "secontext": "system_u:object_r:bin_t:s0", "size": 13355, "state": "file", "uid": 0} 2021-05-25T11:53:55.452 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2021-05-25T11:53:55.454 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-05-25T11:53:55.486 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:55 +0000 (0:00:00.221) 0:01:42.834 *********** 2021-05-25T11:53:55.669 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-05-25T11:53:55.670 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2021-05-25T11:53:55.971 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "secontext": "system_u:object_r:bin_t:s0", "size": 13355, "state": "file", "uid": 0} 2021-05-25T11:53:56.128 INFO:teuthology.task.ansible.out:changed: [smithi181.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": true, "checksum": "2fbc67b36d59f66fc2621f3ca2dcf60dea2167a5", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "md5sum": "fae049f2bb3feda9c19a2dc30bce838a", "mode": "0755", "owner": "root", "secontext": "system_u:object_r:bin_t:s0", "size": 3612, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1621943635.694424-3000-124267544598586/source", "state": "file", "uid": 0} 2021-05-25T11:53:56.153 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:56 +0000 (0:00:00.666) 0:01:43.501 *********** 2021-05-25T11:53:56.221 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2021-05-25T11:53:56.223 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:56.243 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:56 +0000 (0:00:00.090) 0:01:43.591 *********** 2021-05-25T11:53:56.320 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2021-05-25T11:53:56.322 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/yum_systems.yml"], "changed": false} 2021-05-25T11:53:56.343 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:56 +0000 (0:00:00.099) 0:01:43.691 *********** 2021-05-25T11:53:56.379 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-05-25T11:53:56.381 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2021-05-25T11:53:56.568 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* changed: [smithi181.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-05-25T11:53:56.590 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:56 +0000 (0:00:00.247) 0:01:43.938 *********** 2021-05-25T11:53:56.678 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2021-05-25T11:53:56.680 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-05-25T11:53:56.689 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-05-25T11:53:56.700 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-05-25T11:53:56.711 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-05-25T11:53:56.732 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:56 +0000 (0:00:00.142) 0:01:44.080 *********** 2021-05-25T11:53:56.842 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-05-25T11:53:56.843 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": true, "checksum": "2fbc67b36d59f66fc2621f3ca2dcf60dea2167a5", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "md5sum": "fae049f2bb3feda9c19a2dc30bce838a", "mode": "0755", "owner": "root", "secontext": "system_u:object_r:bin_t:s0", "size": 3612, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1621943636.398556-3083-171122675723040/source", "state": "file", "uid": 0} 2021-05-25T11:53:56.866 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:56 +0000 (0:00:00.133) 0:01:44.214 *********** 2021-05-25T11:53:56.934 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2021-05-25T11:53:56.936 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:53:56.955 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:56 +0000 (0:00:00.088) 0:01:44.303 *********** 2021-05-25T11:53:57.032 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2021-05-25T11:53:57.034 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/yum_systems.yml"], "changed": false} 2021-05-25T11:53:57.055 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:57 +0000 (0:00:00.099) 0:01:44.403 *********** 2021-05-25T11:53:57.289 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2021-05-25T11:53:57.290 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-05-25T11:53:57.310 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:57 +0000 (0:00:00.255) 0:01:44.659 *********** 2021-05-25T11:53:57.400 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2021-05-25T11:53:57.401 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-05-25T11:53:57.411 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-05-25T11:53:57.422 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-05-25T11:53:57.443 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-05-25T11:53:57.465 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:53:57 +0000 (0:00:00.154) 0:01:44.813 *********** 2021-05-25T11:54:02.579 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2021-05-25T11:54:02.581 INFO:teuthology.task.ansible.out:changed: [smithi181.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: nagios-plugins-load-2.3.3-5.el8.x86_64", "Installed: nagios-common-4.4.6-4.el8.x86_64", "Installed: nrpe-4.0.3-6.el8.x86_64", "Installed: nagios-plugins-2.3.3-5.el8.x86_64", "Removed: nagios-common-4.4.5-1.el8.x86_64", "Removed: nagios-plugins-2.3.3-4.el8.x86_64", "Removed: nagios-plugins-load-2.3.3-4.el8.x86_64", "Removed: nrpe-4.0.3-1.el8.x86_64"]} 2021-05-25T11:54:02.602 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:54:02 +0000 (0:00:05.136) 0:01:49.950 *********** 2021-05-25T11:54:02.670 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2021-05-25T11:54:02.672 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:54:02.693 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:54:02 +0000 (0:00:00.091) 0:01:50.041 *********** 2021-05-25T11:54:03.045 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2021-05-25T11:54:03.047 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: nagios-plugins-load-2.3.3-5.el8.x86_64", "Installed: nagios-common-4.4.6-4.el8.x86_64", "Installed: nrpe-4.0.3-6.el8.x86_64", "Installed: nagios-plugins-2.3.3-5.el8.x86_64", "Removed: nagios-common-4.4.5-1.el8.x86_64", "Removed: nagios-plugins-2.3.3-4.el8.x86_64", "Removed: nagios-plugins-load-2.3.3-4.el8.x86_64", "Removed: nrpe-4.0.3-1.el8.x86_64"]} 2021-05-25T11:54:03.068 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:54:03 +0000 (0:00:00.374) 0:01:50.416 *********** 2021-05-25T11:54:03.137 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2021-05-25T11:54:03.139 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:54:03.160 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:54:03 +0000 (0:00:00.091) 0:01:50.508 *********** 2021-05-25T11:54:03.186 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2021-05-25T11:54:03.188 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2021-05-25T11:54:03.210 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:54:03 +0000 (0:00:00.050) 0:01:50.558 *********** 2021-05-25T11:54:03.652 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2021-05-25T11:54:03.676 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:54:03 +0000 (0:00:00.466) 0:01:51.024 *********** 2021-05-25T11:54:03.696 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2021-05-25T11:54:03.698 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2021-05-25T11:54:03.721 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:54:03 +0000 (0:00:00.044) 0:01:51.069 *********** 2021-05-25T11:54:03.811 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2021-05-25T11:54:03.812 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "DAEMON_OPTS", "skip_reason": "Conditional result was False"} 2021-05-25T11:54:03.822 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "NRPE_OPTS", "skip_reason": "Conditional result was False"} 2021-05-25T11:54:03.845 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:54:03 +0000 (0:00:00.124) 0:01:51.193 *********** 2021-05-25T11:54:04.081 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2021-05-25T11:54:04.082 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-05-25T11:54:04.102 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:54:04 +0000 (0:00:00.256) 0:01:51.450 *********** 2021-05-25T11:54:04.161 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2021-05-25T11:54:04.162 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2021-05-25T11:54:04.183 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:54:04 +0000 (0:00:00.081) 0:01:51.531 *********** 2021-05-25T11:54:04.262 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2021-05-25T11:54:04.264 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "DAEMON_OPTS", "skip_reason": "Conditional result was False"} 2021-05-25T11:54:04.273 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "NRPE_OPTS", "skip_reason": "Conditional result was False"} 2021-05-25T11:54:04.298 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:54:04 +0000 (0:00:00.115) 0:01:51.647 *********** 2021-05-25T11:54:04.377 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2021-05-25T11:54:04.387 INFO:teuthology.task.ansible.out:fatal: [smithi181.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2021-05-25T11:54:04.401 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:54:04 +0000 (0:00:00.102) 0:01:51.749 *********** 2021-05-25T11:54:04.481 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2021-05-25T11:54:04.483 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:54:04.505 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:54:04 +0000 (0:00:00.103) 0:01:51.853 *********** 2021-05-25T11:54:04.532 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2021-05-25T11:54:04.534 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-05-25T11:54:04.558 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:54:04 +0000 (0:00:00.052) 0:01:51.905 *********** 2021-05-25T11:54:04.836 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2021-05-25T11:54:04.838 INFO:teuthology.task.ansible.out:fatal: [smithi102.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2021-05-25T11:54:04.859 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:54:04 +0000 (0:00:00.302) 0:01:52.207 *********** 2021-05-25T11:54:04.929 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2021-05-25T11:54:04.931 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:54:04.951 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:54:04 +0000 (0:00:00.091) 0:01:52.299 *********** 2021-05-25T11:54:04.999 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2021-05-25T11:54:05.001 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => {"changed": false, "checksum": "aa9150d4582f919d7a8504c01c8d86f05e3eeb5d", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1184, "state": "file", "uid": 0} 2021-05-25T11:54:05.024 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:54:05 +0000 (0:00:00.072) 0:01:52.372 *********** 2021-05-25T11:54:05.437 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "checksum": "aa9150d4582f919d7a8504c01c8d86f05e3eeb5d", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1184, "state": "file", "uid": 0} 2021-05-25T11:54:05.459 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:54:05 +0000 (0:00:00.435) 0:01:52.807 *********** 2021-05-25T11:54:05.777 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2021-05-25T11:54:05.778 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveEnterTimestamp": "Tue 2021-05-25 11:54:00 UTC", "ActiveEnterTimestampMonotonic": "140220966", "ActiveExitTimestamp": "Tue 2021-05-25 11:54:00 UTC", "ActiveExitTimestampMonotonic": "140198560", "ActiveState": "active", "After": "time-sync.target nss-lookup.target var-run.mount basic.target systemd-journald.socket tmp.mount sysinit.target -.mount local-fs.target network-online.target systemd-tmpfiles-setup.service system.slice", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Tue 2021-05-25 11:54:00 UTC", "AssertTimestampMonotonic": "140219955", "Before": "shutdown.target plymouth-quit.service xdm.service getty@tty1.service multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Tue 2021-05-25 11:54:00 UTC", "Co 2021-05-25T11:54:05.780 INFO:teuthology.task.ansible.out:nditionTimestampMonotonic": "140219955", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target nrpe.socket", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/nrpe (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "15250", "ExecMainStartTimestamp": "Tue 2021-05-25 11:54:00 UTC", "ExecMainStartTimestampMonotonic": "140220943", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_SSL_OPT ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nrpe.service", "FreezerState": "thawing", "GID": "984", "Group": "nrpe", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nrpe.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestamp": "Tue 2021-05-25 11:54:00 UTC", "InactiveEnterTimestampMonotonic": "140219386", "InactiveExitTimestamp": "Tue 2021-05-25 11:54:00 UTC", 2021-05-25T11:54:05.781 INFO:teuthology.task.ansible.out:"InactiveExitTimestampMonotonic": "140220966", "InvocationID": "da5610e2f6114296a79ebaf8b3c2f0c2", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127252", "LimitNPROCSoft": "127252", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127252", "LimitSIGPENDINGSoft": "127252", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "15250", "MemoryAccounting": "yes", "MemoryCurrent": "884736", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "nrpe.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/run/nrpe/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateT 2021-05-25T11:54:05.783 INFO:teuthology.task.ansible.out:mp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount sysinit.target system.slice", "RequiresMountsFor": "/run/nrpe /var/tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectory": "nrpe", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Tue 2021-05-25 11:54:00 UTC", "StateChangeTimestampMonotonic": "140220966", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203604", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "988", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "User": "nrpe", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Tue 2021-05-25 11:54:00 UTC", "WatchdogTimestampMonotonic": "140220965", "WatchdogUSec": "0"}} 2021-05-25T11:54:05.802 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:54:05 +0000 (0:00:00.343) 0:01:53.150 *********** 2021-05-25T11:54:05.810 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveEnterTimestamp": "Tue 2021-05-25 11:54:01 UTC", "ActiveEnterTimestampMonotonic": "141880783", "ActiveExitTimestamp": "Tue 2021-05-25 11:54:01 UTC", "ActiveExitTimestampMonotonic": "141849921", "ActiveState": "active", "After": "basic.target network-online.target nss-lookup.target systemd-tmpfiles-setup.service system.slice time-sync.target tmp.mount sysinit.target -.mount systemd-journald.socket var-run.mount local-fs.target", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Tue 2021-05-25 11:54:01 UTC", "AssertTimestampMonotonic": "141880025", "Before": "xdm.service plymouth-quit.service multi-user.target getty@tty1.service shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Tue 2021-05-25 11:54:01 UTC", "Co 2021-05-25T11:54:05.812 INFO:teuthology.task.ansible.out:nditionTimestampMonotonic": "141880025", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target nrpe.socket", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/nrpe (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "15269", "ExecMainStartTimestamp": "Tue 2021-05-25 11:54:01 UTC", "ExecMainStartTimestampMonotonic": "141880761", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_SSL_OPT ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nrpe.service", "FreezerState": "thawing", "GID": "984", "Group": "nrpe", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nrpe.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestamp": "Tue 2021-05-25 11:54:01 UTC", "InactiveEnterTimestampMonotonic": "141879372", "InactiveExitTimestamp": "Tue 2021-05-25 11:54:01 UTC", 2021-05-25T11:54:05.814 INFO:teuthology.task.ansible.out:"InactiveExitTimestampMonotonic": "141880783", "InvocationID": "4a46739ed383487784ce370096f450e9", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127253", "LimitNPROCSoft": "127253", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127253", "LimitSIGPENDINGSoft": "127253", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "15269", "MemoryAccounting": "yes", "MemoryCurrent": "876544", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "nrpe.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/run/nrpe/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateT 2021-05-25T11:54:05.815 INFO:teuthology.task.ansible.out:mp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice -.mount sysinit.target", "RequiresMountsFor": "/var/tmp /run/nrpe", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectory": "nrpe", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Tue 2021-05-25 11:54:01 UTC", "StateChangeTimestampMonotonic": "141880783", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203604", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "988", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "User": "nrpe", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Tue 2021-05-25 11:54:01 UTC", "WatchdogTimestampMonotonic": "141880782", "WatchdogUSec": "0"}} 2021-05-25T11:54:05.816 INFO:teuthology.task.ansible.out: 2021-05-25T11:54:05.840 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:54:05 +0000 (0:00:00.037) 0:01:53.188 *********** 2021-05-25T11:54:06.026 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2021-05-25T11:54:06.028 INFO:teuthology.task.ansible.out:changed: [smithi181.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.001690", "end": "2021-05-25 11:54:06.004491", "rc": 0, "start": "2021-05-25 11:54:06.002801", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2021-05-25T11:54:06.049 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:54:06 +0000 (0:00:00.209) 0:01:53.397 *********** 2021-05-25T11:54:06.066 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.001776", "end": "2021-05-25 11:54:06.047351", "rc": 0, "start": "2021-05-25 11:54:06.045575", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2021-05-25T11:54:06.090 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:54:06 +0000 (0:00:00.040) 0:01:53.438 *********** 2021-05-25T11:54:07.400 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2021-05-25T11:54:07.402 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-05-25T11:54:07.424 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:54:07 +0000 (0:00:01.334) 0:01:54.772 *********** 2021-05-25T11:54:07.462 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-05-25T11:54:07.485 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:54:07 +0000 (0:00:00.061) 0:01:54.833 *********** 2021-05-25T11:54:45.929 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* changed: [smithi102.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: selinux-policy-3.14.3-54.el8_3.4.noarch", "Installed: selinux-policy-targeted-3.14.3-54.el8_3.4.noarch", "Removed: selinux-policy-3.14.3-54.el8_3.2.noarch", "Removed: selinux-policy-targeted-3.14.3-54.el8_3.2.noarch"]} 2021-05-25T11:54:45.952 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:54:45 +0000 (0:00:38.466) 0:02:33.300 *********** 2021-05-25T11:54:46.052 INFO:teuthology.task.ansible.out:changed: [smithi181.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: selinux-policy-3.14.3-54.el8_3.4.noarch", "Installed: selinux-policy-targeted-3.14.3-54.el8_3.4.noarch", "Removed: selinux-policy-3.14.3-54.el8_3.2.noarch", "Removed: selinux-policy-targeted-3.14.3-54.el8_3.2.noarch"]} 2021-05-25T11:54:46.077 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:54:46 +0000 (0:00:00.125) 0:02:33.425 *********** 2021-05-25T11:54:46.698 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2021-05-25T11:54:46.700 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} 2021-05-25T11:54:46.731 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:54:46 +0000 (0:00:00.654) 0:02:34.079 *********** 2021-05-25T11:54:46.764 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:54:46 +0000 (0:00:00.032) 0:02:34.111 *********** 2021-05-25T11:55:03.207 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2021-05-25T11:55:03.208 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:16.242830", "end": "2021-05-25 11:55:03.183069", "failed_when_result": false, "rc": 0, "start": "2021-05-25 11:54:46.940239", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2021-05-25T11:55:03.231 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:03 +0000 (0:00:16.467) 0:02:50.579 *********** 2021-05-25T11:55:03.352 INFO:teuthology.task.ansible.out:changed: [smithi181.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:16.352939", "end": "2021-05-25 11:55:03.323028", "failed_when_result": false, "rc": 0, "start": "2021-05-25 11:54:46.970089", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2021-05-25T11:55:03.376 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:03 +0000 (0:00:00.145) 0:02:50.724 *********** 2021-05-25T11:55:03.725 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2021-05-25T11:55:03.727 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/tmp/nrpe.te", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "state": "file", "uid": 0} 2021-05-25T11:55:03.749 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:03 +0000 (0:00:00.373) 0:02:51.097 *********** 2021-05-25T11:55:03.874 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => {"changed": false, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/tmp/nrpe.te", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "state": "file", "uid": 0} 2021-05-25T11:55:03.899 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:03 +0000 (0:00:00.149) 0:02:51.247 *********** 2021-05-25T11:55:03.998 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2021-05-25T11:55:04.000 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.018549", "end": "2021-05-25 11:55:03.974017", "rc": 0, "start": "2021-05-25 11:55:03.955468", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-05-25T11:55:04.021 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:04 +0000 (0:00:00.121) 0:02:51.369 *********** 2021-05-25T11:55:04.141 INFO:teuthology.task.ansible.out:changed: [smithi181.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.018774", "end": "2021-05-25 11:55:04.122331", "rc": 0, "start": "2021-05-25 11:55:04.103557", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-05-25T11:55:04.164 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:04 +0000 (0:00:00.142) 0:02:51.512 *********** 2021-05-25T11:55:04.253 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2021-05-25T11:55:04.255 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.006347", "end": "2021-05-25 11:55:04.234093", "rc": 0, "start": "2021-05-25 11:55:04.227746", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-05-25T11:55:04.277 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:04 +0000 (0:00:00.113) 0:02:51.625 *********** 2021-05-25T11:55:04.398 INFO:teuthology.task.ansible.out:changed: [smithi181.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.007350", "end": "2021-05-25 11:55:04.375482", "rc": 0, "start": "2021-05-25 11:55:04.368132", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-05-25T11:55:04.421 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:04 +0000 (0:00:00.143) 0:02:51.769 *********** 2021-05-25T11:55:20.807 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2021-05-25T11:55:20.809 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:16.293715", "end": "2021-05-25 11:55:20.779538", "rc": 0, "start": "2021-05-25 11:55:04.485823", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-05-25T11:55:20.830 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:20 +0000 (0:00:16.409) 0:03:08.178 *********** 2021-05-25T11:55:20.995 INFO:teuthology.task.ansible.out:changed: [smithi181.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:16.343565", "end": "2021-05-25 11:55:20.975170", "rc": 0, "start": "2021-05-25 11:55:04.631605", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-05-25T11:55:21.018 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:21 +0000 (0:00:00.188) 0:03:08.366 *********** 2021-05-25T11:55:21.076 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2021-05-25T11:55:21.078 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2021-05-25T11:55:21.104 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:21 +0000 (0:00:00.085) 0:03:08.451 *********** 2021-05-25T11:55:21.172 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2021-05-25T11:55:21.174 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi"], "dash_stream": "", "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/dnf_systems.yml"], "changed": false} 2021-05-25T11:55:21.195 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:21 +0000 (0:00:00.091) 0:03:08.543 *********** 2021-05-25T11:55:21.253 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2021-05-25T11:55:21.255 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2021-05-25T11:55:21.276 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:21 +0000 (0:00:00.081) 0:03:08.624 *********** 2021-05-25T11:55:21.283 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2021-05-25T11:55:21.285 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-05-25T11:55:21.309 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:21 +0000 (0:00:00.032) 0:03:08.657 *********** 2021-05-25T11:55:21.346 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2021-05-25T11:55:21.348 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi"], "dash_stream": "", "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/dnf_systems.yml"], "changed": false} 2021-05-25T11:55:21.370 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:21 +0000 (0:00:00.061) 0:03:08.718 *********** 2021-05-25T11:55:21.387 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2021-05-25T11:55:21.388 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml) => {"ansible_facts": {"common_yum_repos": {"lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "epel_packages": [], "nfs_service": "nfs-server", "ntp_service_name": "chronyd", "packages": ["dnf-utils", "sysstat", "libedit", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse-libs", "openssl", "libuuid", "podman", "podman-docker", "attr", "ant", "lsof", "gettext", "bc", "xfsdump", "blktrace", "usbredir", "libev-devel", "valgrind", "nfs-utils", "ncurses-devel", "gcc", "git", "python3-nose", "python3-virtualenv", "genisoimage", "qemu-img", "qemu-kvm-core", "qemu-kvm-block-rbd", "libacl-devel", "dbench", "autoconf", "gdb", "iozone"], "packages_to_upgrade": ["libgcrypt"], "yum_mirrorlists": ["CentOS-AppStream-mirrorlist", "CentOS-Base-mirrorlist", "CentOS-Extras-mirrorlist"], "yum_repos": {"CentOS-AppStream": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}, "CentOS-Base": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}, "CentOS-Extras": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}}}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"} 2021-05-25T11:55:21.416 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:21 +0000 (0:00:00.046) 0:03:08.764 *********** 2021-05-25T11:55:21.454 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2021-05-25T11:55:21.456 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-05-25T11:55:21.478 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:21 +0000 (0:00:00.062) 0:03:08.826 *********** 2021-05-25T11:55:21.496 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2021-05-25T11:55:21.497 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-05-25T11:55:21.521 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:21 +0000 (0:00:00.042) 0:03:08.869 *********** 2021-05-25T11:55:21.559 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2021-05-25T11:55:21.561 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml) => {"ansible_facts": {"common_yum_repos": {"lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "epel_packages": [], "nfs_service": "nfs-server", "ntp_service_name": "chronyd", "packages": ["dnf-utils", "sysstat", "libedit", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse-libs", "openssl", "libuuid", "podman", "podman-docker", "attr", "ant", "lsof", "gettext", "bc", "xfsdump", "blktrace", "usbredir", "libev-devel", "valgrind", "nfs-utils", "ncurses-devel", "gcc", "git", "python3-nose", "python3-virtualenv", "genisoimage", "qemu-img", "qemu-kvm-core", "qemu-kvm-block-rbd", "libacl-devel", "dbench", "autoconf", "gdb", "iozone"], "packages_to_upgrade": ["libgcrypt"], "yum_mirrorlists": ["CentOS-AppStream-mirrorlist", "CentOS-Base-mirrorlist", "CentOS-Extras-mirrorlist"], "yum_repos": {"CentOS-AppStream": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}, "CentOS-Base": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}, "CentOS-Extras": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}}}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"} 2021-05-25T11:55:21.586 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:21 +0000 (0:00:00.064) 0:03:08.934 *********** 2021-05-25T11:55:21.665 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2021-05-25T11:55:21.667 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-05-25T11:55:21.698 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:21 +0000 (0:00:00.112) 0:03:09.046 *********** 2021-05-25T11:55:21.717 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2021-05-25T11:55:21.718 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-05-25T11:55:21.741 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:21 +0000 (0:00:00.042) 0:03:09.089 *********** 2021-05-25T11:55:21.881 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-05-25T11:55:21.903 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:21 +0000 (0:00:00.162) 0:03:09.252 *********** 2021-05-25T11:55:21.961 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2021-05-25T11:55:21.963 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "name": "ubuntu", "state": "present", "system": false} 2021-05-25T11:55:21.984 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:21 +0000 (0:00:00.080) 0:03:09.332 *********** 2021-05-25T11:55:22.062 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2021-05-25T11:55:22.063 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:22.087 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:22 +0000 (0:00:00.102) 0:03:09.435 *********** 2021-05-25T11:55:22.126 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2021-05-25T11:55:22.128 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "name": "ubuntu", "state": "present", "system": false} 2021-05-25T11:55:22.160 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:22 +0000 (0:00:00.073) 0:03:09.508 *********** 2021-05-25T11:55:22.168 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2021-05-25T11:55:22.169 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:22.202 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:22 +0000 (0:00:00.042) 0:03:09.550 *********** 2021-05-25T11:55:22.229 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2021-05-25T11:55:22.231 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:22.252 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:22 +0000 (0:00:00.050) 0:03:09.600 *********** 2021-05-25T11:55:22.331 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2021-05-25T11:55:22.333 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:22.353 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:22 +0000 (0:00:00.100) 0:03:09.701 *********** 2021-05-25T11:55:22.452 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2021-05-25T11:55:22.454 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2021-05-25T11:55:22.476 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:22 +0000 (0:00:00.122) 0:03:09.824 *********** 2021-05-25T11:55:22.628 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2021-05-25T11:55:22.653 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:22 +0000 (0:00:00.177) 0:03:10.002 *********** 2021-05-25T11:55:23.053 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2021-05-25T11:55:23.075 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:23 +0000 (0:00:00.421) 0:03:10.424 *********** 2021-05-25T11:55:23.144 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2021-05-25T11:55:23.146 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi102"}, "changed": false} 2021-05-25T11:55:23.175 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:23 +0000 (0:00:00.099) 0:03:10.523 *********** 2021-05-25T11:55:23.204 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2021-05-25T11:55:23.206 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2021-05-25T11:55:23.228 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:23 +0000 (0:00:00.052) 0:03:10.576 *********** 2021-05-25T11:55:23.306 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2021-05-25T11:55:23.308 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi181"}, "changed": false} 2021-05-25T11:55:23.329 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:23 +0000 (0:00:00.100) 0:03:10.677 *********** 2021-05-25T11:55:24.208 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2021-05-25T11:55:24.210 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi102.front.sepia.ceph.com", "ansible_hostname": "smithi102", "ansible_nodename": "smithi102"}, "changed": true, "name": "smithi102"} changed: [smithi181.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi181.front.sepia.ceph.com", "ansible_hostname": "smithi181", "ansible_nodename": "smithi181"}, "changed": true, "name": "smithi181"} 2021-05-25T11:55:24.230 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:24 +0000 (0:00:00.901) 0:03:11.578 *********** 2021-05-25T11:55:24.249 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:24 +0000 (0:00:00.019) 0:03:11.597 *********** 2021-05-25T11:55:24.702 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2021-05-25T11:55:24.703 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "checksum": "a5e550b2aa74f6355e926d70d73a50ad9ab4b4be", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1043, "state": "file", "uid": 0} 2021-05-25T11:55:24.725 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:24 +0000 (0:00:00.475) 0:03:12.073 *********** 2021-05-25T11:55:24.732 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => {"changed": false, "checksum": "a5e550b2aa74f6355e926d70d73a50ad9ab4b4be", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1043, "state": "file", "uid": 0} 2021-05-25T11:55:24.757 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:24 +0000 (0:00:00.031) 0:03:12.105 *********** 2021-05-25T11:55:25.211 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2021-05-25T11:55:25.213 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2021-05-25T11:55:25.234 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:25 +0000 (0:00:00.477) 0:03:12.582 *********** 2021-05-25T11:55:25.241 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2021-05-25T11:55:25.265 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:25 +0000 (0:00:00.030) 0:03:12.613 *********** 2021-05-25T11:55:25.842 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2021-05-25T11:55:25.844 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2021-05-25T11:55:25.865 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:25 +0000 (0:00:00.600) 0:03:13.213 *********** 2021-05-25T11:55:25.883 INFO:teuthology.task.ansible.out:changed: [smithi181.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2021-05-25T11:55:25.907 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:25 +0000 (0:00:00.042) 0:03:13.255 *********** 2021-05-25T11:55:26.360 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2021-05-25T11:55:26.362 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2021-05-25T11:55:26.383 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:26 +0000 (0:00:00.475) 0:03:13.731 *********** 2021-05-25T11:55:26.401 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2021-05-25T11:55:26.425 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:26 +0000 (0:00:00.041) 0:03:13.773 *********** 2021-05-25T11:55:26.620 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-05-25T11:55:26.622 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2021-05-25T11:55:26.645 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:26 +0000 (0:00:00.220) 0:03:13.993 *********** 2021-05-25T11:55:26.664 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2021-05-25T11:55:26.687 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:26 +0000 (0:00:00.041) 0:03:14.035 *********** 2021-05-25T11:55:26.880 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-05-25T11:55:26.882 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "gid": 36, "name": "kvm", "state": "present", "system": false} 2021-05-25T11:55:26.903 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:26 +0000 (0:00:00.215) 0:03:14.251 *********** 2021-05-25T11:55:26.920 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => {"changed": false, "gid": 36, "name": "kvm", "state": "present", "system": false} 2021-05-25T11:55:26.944 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:26 +0000 (0:00:00.040) 0:03:14.292 *********** 2021-05-25T11:55:27.167 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-05-25T11:55:27.169 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1238, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2021-05-25T11:55:27.190 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:27 +0000 (0:00:00.246) 0:03:14.538 *********** 2021-05-25T11:55:27.207 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1238, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2021-05-25T11:55:27.233 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:27 +0000 (0:00:00.042) 0:03:14.581 *********** 2021-05-25T11:55:27.685 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-05-25T11:55:27.686 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2021-05-25T11:55:27.708 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:27 +0000 (0:00:00.474) 0:03:15.056 *********** 2021-05-25T11:55:27.725 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2021-05-25T11:55:27.748 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:27 +0000 (0:00:00.040) 0:03:15.096 *********** 2021-05-25T11:55:28.198 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-05-25T11:55:28.200 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2021-05-25T11:55:28.224 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:28 +0000 (0:00:00.475) 0:03:15.572 *********** 2021-05-25T11:55:28.242 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2021-05-25T11:55:28.267 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:28 +0000 (0:00:00.043) 0:03:15.615 *********** 2021-05-25T11:55:30.765 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-05-25T11:55:30.767 INFO:teuthology.task.ansible.out:[WARNING]: Consider using the yum, dnf or zypper module rather than running 'rpm'. If you need to use command because yum, dnf or zypper is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi102.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:02.313566", "end": "2021-05-25 11:55:30.736509", "rc": 0, "start": "2021-05-25 11:55:28.422943", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-05-25T11:55:30.789 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:30 +0000 (0:00:02.522) 0:03:18.137 *********** 2021-05-25T11:55:30.797 INFO:teuthology.task.ansible.out:changed: [smithi181.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:02.287647", "end": "2021-05-25 11:55:30.767303", "rc": 0, "start": "2021-05-25 11:55:28.479656", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-05-25T11:55:30.821 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:30 +0000 (0:00:00.031) 0:03:18.169 *********** 2021-05-25T11:55:30.859 INFO:teuthology.task.ansible.out: TASK [testnode : Clean up Stream distro-provided repos] ************************ 2021-05-25T11:55:30.861 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:30.883 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:30 +0000 (0:00:00.061) 0:03:18.231 *********** 2021-05-25T11:55:30.901 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:30.925 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:30 +0000 (0:00:00.042) 0:03:18.273 *********** 2021-05-25T11:55:30.952 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-05-25T11:55:30.954 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:30.976 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:30 +0000 (0:00:00.051) 0:03:18.324 *********** 2021-05-25T11:55:30.996 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:31.021 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:31 +0000 (0:00:00.044) 0:03:18.368 *********** 2021-05-25T11:55:31.485 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2021-05-25T11:55:31.486 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=CentOS-AppStream-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "aa5ad73629677ef17172417e50d2cdadccb7a3f2", "dest": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-AppStream-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 397, "state": "file", "uid": 0} 2021-05-25T11:55:31.538 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item=CentOS-AppStream-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "aa5ad73629677ef17172417e50d2cdadccb7a3f2", "dest": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-AppStream-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 397, "state": "file", "uid": 0} 2021-05-25T11:55:31.912 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=CentOS-Base-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "4ed3c0272e6c4ec99169cd68f0b43d40bdd4b296", "dest": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Base-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 380, "state": "file", "uid": 0} 2021-05-25T11:55:31.954 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item=CentOS-Base-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "4ed3c0272e6c4ec99169cd68f0b43d40bdd4b296", "dest": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Base-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 380, "state": "file", "uid": 0} 2021-05-25T11:55:32.329 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=CentOS-Extras-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "a862f43d3b3c7b8b592aa7dce23a6d013ab20bc3", "dest": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Extras-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 382, "state": "file", "uid": 0} 2021-05-25T11:55:32.353 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:32 +0000 (0:00:01.332) 0:03:19.701 *********** 2021-05-25T11:55:32.370 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item=CentOS-Extras-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "a862f43d3b3c7b8b592aa7dce23a6d013ab20bc3", "dest": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Extras-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 382, "state": "file", "uid": 0} 2021-05-25T11:55:32.396 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:32 +0000 (0:00:00.042) 0:03:19.744 *********** 2021-05-25T11:55:32.848 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2021-05-25T11:55:32.849 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'key': 'lab-extras', 'value': {'name': 'lab-extras', 'baseurl': 'http://apt-mirror.front.sepia.ceph.com/lab-extras/8/', 'enabled': 1, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a0e2ac0628a264e14b96b1c57627a3aa344a6c52", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 203, "state": "file", "uid": 0} 2021-05-25T11:55:32.874 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:32 +0000 (0:00:00.478) 0:03:20.222 *********** 2021-05-25T11:55:32.891 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'key': 'lab-extras', 'value': {'name': 'lab-extras', 'baseurl': 'http://apt-mirror.front.sepia.ceph.com/lab-extras/8/', 'enabled': 1, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a0e2ac0628a264e14b96b1c57627a3aa344a6c52", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 203, "state": "file", "uid": 0} 2021-05-25T11:55:32.918 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:32 +0000 (0:00:00.043) 0:03:20.266 *********** 2021-05-25T11:55:33.372 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2021-05-25T11:55:33.374 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'key': 'CentOS-AppStream', 'value': {'name': 'CentOS-$releasever - AppStream', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "9b147c879e95cd23f3bbde027da102febcaad1a8", "dest": "/etc/yum.repos.d/CentOS-AppStream.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-AppStream", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 289, "state": "file", "uid": 0} 2021-05-25T11:55:33.416 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'key': 'CentOS-AppStream', 'value': {'name': 'CentOS-$releasever - AppStream', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "9b147c879e95cd23f3bbde027da102febcaad1a8", "dest": "/etc/yum.repos.d/CentOS-AppStream.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-AppStream", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 289, "state": "file", "uid": 0} 2021-05-25T11:55:33.800 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'key': 'CentOS-Base', 'value': {'name': 'CentOS-$releasever - Base', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-Base-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "77e4590ed0ecec056fa44f4e65af60e110822b85", "dest": "/etc/yum.repos.d/CentOS-Base.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Base", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 274, "state": "file", "uid": 0} 2021-05-25T11:55:33.832 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'key': 'CentOS-Base', 'value': {'name': 'CentOS-$releasever - Base', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-Base-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "77e4590ed0ecec056fa44f4e65af60e110822b85", "dest": "/etc/yum.repos.d/CentOS-Base.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Base", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 274, "state": "file", "uid": 0} 2021-05-25T11:55:34.217 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'key': 'CentOS-Extras', 'value': {'name': 'CentOS-$releasever - Extras', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-Extras-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "2856826614c7f1cb9409228f6c41319a023ca754", "dest": "/etc/yum.repos.d/CentOS-Extras.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Extras", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 280, "state": "file", "uid": 0} 2021-05-25T11:55:34.243 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:34 +0000 (0:00:01.324) 0:03:21.591 *********** 2021-05-25T11:55:34.260 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'key': 'CentOS-Extras', 'value': {'name': 'CentOS-$releasever - Extras', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-Extras-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "2856826614c7f1cb9409228f6c41319a023ca754", "dest": "/etc/yum.repos.d/CentOS-Extras.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Extras", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 280, "state": "file", "uid": 0} 2021-05-25T11:55:34.287 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:34 +0000 (0:00:00.043) 0:03:21.635 *********** 2021-05-25T11:55:34.303 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2021-05-25T11:55:34.327 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:34 +0000 (0:00:00.039) 0:03:21.675 *********** 2021-05-25T11:55:34.377 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:34 +0000 (0:00:00.050) 0:03:21.725 *********** 2021-05-25T11:55:34.393 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2021-05-25T11:55:34.395 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:34.416 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:34 +0000 (0:00:00.039) 0:03:21.764 *********** 2021-05-25T11:55:34.443 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:34.467 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:34 +0000 (0:00:00.050) 0:03:21.815 *********** 2021-05-25T11:55:34.515 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2021-05-25T11:55:34.517 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-05-25T11:55:34.526 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-05-25T11:55:34.550 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:34 +0000 (0:00:00.083) 0:03:21.898 *********** 2021-05-25T11:55:34.557 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-05-25T11:55:34.558 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-05-25T11:55:34.581 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:34 +0000 (0:00:00.031) 0:03:21.929 *********** 2021-05-25T11:55:34.618 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2021-05-25T11:55:34.620 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:34.641 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:34 +0000 (0:00:00.059) 0:03:21.989 *********** 2021-05-25T11:55:34.657 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:34.681 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:34 +0000 (0:00:00.039) 0:03:22.029 *********** 2021-05-25T11:55:34.710 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-05-25T11:55:34.712 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"ansible_facts": {"skip_packaging": false}, "changed": false} 2021-05-25T11:55:34.734 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:34 +0000 (0:00:00.053) 0:03:22.082 *********** 2021-05-25T11:55:34.762 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => {"ansible_facts": {"skip_packaging": false}, "changed": false} 2021-05-25T11:55:34.786 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:34 +0000 (0:00:00.052) 0:03:22.134 *********** 2021-05-25T11:55:34.803 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2021-05-25T11:55:34.805 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:34.829 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:34 +0000 (0:00:00.042) 0:03:22.177 *********** 2021-05-25T11:55:34.867 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:34.894 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:34 +0000 (0:00:00.065) 0:03:22.242 *********** 2021-05-25T11:55:34.901 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2021-05-25T11:55:34.903 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:34.925 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:34 +0000 (0:00:00.031) 0:03:22.273 *********** 2021-05-25T11:55:34.963 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:34.987 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:34 +0000 (0:00:00.061) 0:03:22.335 *********** 2021-05-25T11:55:35.160 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2021-05-25T11:55:35.162 INFO:teuthology.task.ansible.out:fatal: [smithi102.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.007025", "end": "2021-05-25 11:55:35.138117", "msg": "non-zero return code", "rc": 1, "start": "2021-05-25 11:55:35.131092", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2021-05-25T11:55:35.185 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:35 +0000 (0:00:00.198) 0:03:22.533 *********** 2021-05-25T11:55:35.237 INFO:teuthology.task.ansible.out:fatal: [smithi181.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.007186", "end": "2021-05-25 11:55:35.215508", "msg": "non-zero return code", "rc": 1, "start": "2021-05-25 11:55:35.208322", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2021-05-25T11:55:35.261 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:35 +0000 (0:00:00.075) 0:03:22.609 *********** 2021-05-25T11:55:35.267 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2021-05-25T11:55:35.269 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:35.293 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:35 +0000 (0:00:00.032) 0:03:22.641 *********** 2021-05-25T11:55:35.332 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:35.356 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:35 +0000 (0:00:00.063) 0:03:22.704 *********** 2021-05-25T11:55:38.187 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-05-25T11:55:38.189 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: librbd1-1:12.2.7-9.el8.x86_64", "Removed: qemu-kvm-block-rbd-15:4.2.0-34.module_el8.3.0+704+02c27f72.3.x86_64", "Removed: librados2-1:12.2.7-9.el8.x86_64"]} 2021-05-25T11:55:38.210 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:38 +0000 (0:00:02.853) 0:03:25.558 *********** 2021-05-25T11:55:38.249 INFO:teuthology.task.ansible.out:changed: [smithi181.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: librbd1-1:12.2.7-9.el8.x86_64", "Removed: qemu-kvm-block-rbd-15:4.2.0-34.module_el8.3.0+704+02c27f72.3.x86_64", "Removed: librados2-1:12.2.7-9.el8.x86_64"]} 2021-05-25T11:55:38.273 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:38 +0000 (0:00:00.062) 0:03:25.621 *********** 2021-05-25T11:55:40.901 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-05-25T11:55:40.903 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: boost-program-options-1.66.0-10.el8.x86_64", "Removed: boost-random-1.66.0-10.el8.x86_64"]} 2021-05-25T11:55:40.925 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:40 +0000 (0:00:02.652) 0:03:28.273 *********** 2021-05-25T11:55:40.954 INFO:teuthology.task.ansible.out:changed: [smithi181.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: boost-program-options-1.66.0-10.el8.x86_64", "Removed: boost-random-1.66.0-10.el8.x86_64"]} 2021-05-25T11:55:40.979 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:40 +0000 (0:00:00.054) 0:03:28.327 *********** 2021-05-25T11:55:48.177 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-05-25T11:55:48.179 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: qemu-kvm-block-rbd-15:4.2.0-34.module_el8.3.0+755+88436ea4.5.x86_64", "Installed: boost-program-options-1.66.0-10.el8.x86_64", "Installed: qemu-kvm-common-15:4.2.0-34.module_el8.3.0+755+88436ea4.5.x86_64", "Installed: boost-random-1.66.0-10.el8.x86_64", "Installed: qemu-kvm-core-15:4.2.0-34.module_el8.3.0+755+88436ea4.5.x86_64", "Installed: librados2-1:12.2.7-9.el8.x86_64", "Installed: qemu-img-15:4.2.0-34.module_el8.3.0+755+88436ea4.5.x86_64", "Installed: librbd1-1:12.2.7-9.el8.x86_64", "Removed: qemu-img-15:4.2.0-34.module_el8.3.0+704+02c27f72.3.x86_64", "Removed: qemu-kvm-common-15:4.2.0-34.module_el8.3.0+704+02c27f72.3.x86_64", "Removed: qemu-kvm-core-15:4.2.0-34.module_el8.3.0+704+02c27f72.3.x86_64"]} 2021-05-25T11:55:48.207 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:48 +0000 (0:00:07.227) 0:03:35.555 *********** 2021-05-25T11:55:48.276 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2021-05-25T11:55:48.278 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:48.298 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:48 +0000 (0:00:00.091) 0:03:35.646 *********** 2021-05-25T11:55:48.388 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-05-25T11:55:48.390 INFO:teuthology.task.ansible.out:changed: [smithi181.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: qemu-kvm-block-rbd-15:4.2.0-34.module_el8.3.0+755+88436ea4.5.x86_64", "Installed: boost-program-options-1.66.0-10.el8.x86_64", "Installed: qemu-kvm-common-15:4.2.0-34.module_el8.3.0+755+88436ea4.5.x86_64", "Installed: boost-random-1.66.0-10.el8.x86_64", "Installed: qemu-kvm-core-15:4.2.0-34.module_el8.3.0+755+88436ea4.5.x86_64", "Installed: librados2-1:12.2.7-9.el8.x86_64", "Installed: qemu-img-15:4.2.0-34.module_el8.3.0+755+88436ea4.5.x86_64", "Installed: librbd1-1:12.2.7-9.el8.x86_64", "Removed: qemu-img-15:4.2.0-34.module_el8.3.0+704+02c27f72.3.x86_64", "Removed: qemu-kvm-common-15:4.2.0-34.module_el8.3.0+704+02c27f72.3.x86_64", "Removed: qemu-kvm-core-15:4.2.0-34.module_el8.3.0+704+02c27f72.3.x86_64"]} 2021-05-25T11:55:48.418 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:48 +0000 (0:00:00.119) 0:03:35.766 *********** 2021-05-25T11:55:48.485 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2021-05-25T11:55:48.487 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:48.508 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:48 +0000 (0:00:00.090) 0:03:35.856 *********** 2021-05-25T11:55:49.741 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-05-25T11:55:49.743 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-05-25T11:55:49.767 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:49 +0000 (0:00:01.258) 0:03:37.115 *********** 2021-05-25T11:55:50.005 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-05-25T11:55:50.029 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:50 +0000 (0:00:00.261) 0:03:37.377 *********** 2021-05-25T11:55:51.170 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-05-25T11:55:51.172 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-05-25T11:55:51.194 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:51 +0000 (0:00:01.164) 0:03:38.542 *********** 2021-05-25T11:55:51.453 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-05-25T11:55:51.480 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:51 +0000 (0:00:00.285) 0:03:38.828 *********** 2021-05-25T11:55:51.539 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2021-05-25T11:55:51.540 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "sysinit.target dbus.socket polkit.service system.slice basic.target dbus.service", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "shutdown.target network-pre.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUAffinity": "", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "ebtables.service shutdown.target ip6tables.service iptables.service ipset.service nftables.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "firewalld - dynamic firewall daemon", "D 2021-05-25T11:55:51.542 INFO:teuthology.task.ansible.out:evicePolicy": "auto", "Documentation": "man:firewalld(1)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/firewalld.service", "FreezerState": "running", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "firewalld.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "mixed", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "Limi 2021-05-25T11:55:51.544 INFO:teuthology.task.ansible.out:tNOFILESoft": "1024", "LimitNPROC": "127253", "LimitNPROCSoft": "127253", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127253", "LimitSIGPENDINGSoft": "127253", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "yes", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "firewalld.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target system.slice dbus.socket", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "null", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "null", "S 2021-05-25T11:55:51.545 INFO:teuthology.task.ansible.out:tartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestampMonotonic": "0", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "203604", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "dbus", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "network-pre.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-05-25T11:55:51.565 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:51 +0000 (0:00:00.084) 0:03:38.913 *********** 2021-05-25T11:55:51.814 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "dbus.service polkit.service sysinit.target dbus.socket basic.target system.slice", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "shutdown.target network-pre.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUAffinity": "", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "iptables.service shutdown.target ebtables.service ipset.service ip6tables.service nftables.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "firewalld - dynamic firewall daemon", "D 2021-05-25T11:55:51.816 INFO:teuthology.task.ansible.out:evicePolicy": "auto", "Documentation": "man:firewalld(1)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/firewalld.service", "FreezerState": "running", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "firewalld.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "mixed", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "Limi 2021-05-25T11:55:51.817 INFO:teuthology.task.ansible.out:tNOFILESoft": "1024", "LimitNPROC": "127252", "LimitNPROCSoft": "127252", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127252", "LimitSIGPENDINGSoft": "127252", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "yes", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "firewalld.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target system.slice dbus.socket", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "null", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "null", "S 2021-05-25T11:55:51.819 INFO:teuthology.task.ansible.out:tartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestampMonotonic": "0", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "203604", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "dbus", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "network-pre.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-05-25T11:55:51.838 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:51 +0000 (0:00:00.273) 0:03:39.186 *********** 2021-05-25T11:55:51.886 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2021-05-25T11:55:51.888 INFO:teuthology.task.ansible.out:fatal: [smithi102.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2021-05-25T11:55:51.909 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:51 +0000 (0:00:00.071) 0:03:39.257 *********** 2021-05-25T11:55:52.157 INFO:teuthology.task.ansible.out:fatal: [smithi181.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2021-05-25T11:55:52.179 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:52 +0000 (0:00:00.270) 0:03:39.527 *********** 2021-05-25T11:55:52.663 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2021-05-25T11:55:52.665 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2021-05-25T11:55:52.686 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:52 +0000 (0:00:00.506) 0:03:40.034 *********** 2021-05-25T11:55:52.692 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2021-05-25T11:55:52.717 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:52 +0000 (0:00:00.030) 0:03:40.065 *********** 2021-05-25T11:55:52.754 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2021-05-25T11:55:52.756 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:52.778 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:52 +0000 (0:00:00.061) 0:03:40.126 *********** 2021-05-25T11:55:52.785 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:52.809 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:52 +0000 (0:00:00.031) 0:03:40.157 *********** 2021-05-25T11:55:52.847 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2021-05-25T11:55:52.848 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:52.870 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:52 +0000 (0:00:00.060) 0:03:40.218 *********** 2021-05-25T11:55:52.878 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:52.901 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:52 +0000 (0:00:00.031) 0:03:40.249 *********** 2021-05-25T11:55:52.939 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:52.962 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:52 +0000 (0:00:00.060) 0:03:40.310 *********** 2021-05-25T11:55:52.979 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:53.002 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:52 +0000 (0:00:00.039) 0:03:40.350 *********** 2021-05-25T11:55:53.029 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2021-05-25T11:55:53.030 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:53.052 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:53 +0000 (0:00:00.049) 0:03:40.400 *********** 2021-05-25T11:55:53.069 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:53.092 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:53 +0000 (0:00:00.040) 0:03:40.440 *********** 2021-05-25T11:55:53.119 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ 2021-05-25T11:55:53.121 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:53.142 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:53 +0000 (0:00:00.050) 0:03:40.490 *********** 2021-05-25T11:55:53.160 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:53.184 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:53 +0000 (0:00:00.041) 0:03:40.531 *********** 2021-05-25T11:55:53.210 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2021-05-25T11:55:53.212 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:53.234 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:53 +0000 (0:00:00.050) 0:03:40.582 *********** 2021-05-25T11:55:53.262 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:53.285 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:53 +0000 (0:00:00.051) 0:03:40.633 *********** 2021-05-25T11:55:53.323 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2021-05-25T11:55:53.348 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:53 +0000 (0:00:00.062) 0:03:40.696 *********** 2021-05-25T11:55:53.379 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:53 +0000 (0:00:00.030) 0:03:40.727 *********** 2021-05-25T11:55:53.417 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2021-05-25T11:55:53.418 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:53.440 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:53 +0000 (0:00:00.061) 0:03:40.788 *********** 2021-05-25T11:55:53.448 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:53.472 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:53 +0000 (0:00:00.031) 0:03:40.820 *********** 2021-05-25T11:55:53.499 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:53.523 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:53 +0000 (0:00:00.051) 0:03:40.871 *********** 2021-05-25T11:55:53.552 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:53.576 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:53 +0000 (0:00:00.052) 0:03:40.924 *********** 2021-05-25T11:55:53.592 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2021-05-25T11:55:53.593 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-05-25T11:55:53.593 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-05-25T11:55:53.618 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:53 +0000 (0:00:00.042) 0:03:40.966 *********** 2021-05-25T11:55:53.645 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-05-25T11:55:53.647 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-05-25T11:55:53.670 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:53 +0000 (0:00:00.052) 0:03:41.018 *********** 2021-05-25T11:55:53.686 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2021-05-25T11:55:53.688 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:53.710 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:53 +0000 (0:00:00.040) 0:03:41.059 *********** 2021-05-25T11:55:53.728 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:53.752 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:53 +0000 (0:00:00.041) 0:03:41.100 *********** 2021-05-25T11:55:53.769 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2021-05-25T11:55:53.793 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:53 +0000 (0:00:00.041) 0:03:41.141 *********** 2021-05-25T11:55:53.836 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:53 +0000 (0:00:00.042) 0:03:41.184 *********** 2021-05-25T11:55:53.864 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2021-05-25T11:55:53.866 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:53.888 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:53 +0000 (0:00:00.052) 0:03:41.236 *********** 2021-05-25T11:55:53.890 INFO:teuthology.task.ansible.out: 2021-05-25T11:55:53.917 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:53.942 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:53 +0000 (0:00:00.053) 0:03:41.290 *********** 2021-05-25T11:55:53.970 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2021-05-25T11:55:53.971 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:54.000 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:53 +0000 (0:00:00.057) 0:03:41.348 *********** 2021-05-25T11:55:54.018 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:54.046 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:54 +0000 (0:00:00.046) 0:03:41.394 *********** 2021-05-25T11:55:54.063 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2021-05-25T11:55:54.065 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:54.087 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:54 +0000 (0:00:00.041) 0:03:41.435 *********** 2021-05-25T11:55:54.136 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:54.161 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:54 +0000 (0:00:00.073) 0:03:41.509 *********** 2021-05-25T11:55:54.167 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-05-25T11:55:54.169 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:54.191 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:54 +0000 (0:00:00.030) 0:03:41.539 *********** 2021-05-25T11:55:54.229 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:54.254 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:54 +0000 (0:00:00.062) 0:03:41.602 *********** 2021-05-25T11:55:54.259 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2021-05-25T11:55:54.261 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:54.284 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:54 +0000 (0:00:00.030) 0:03:41.632 *********** 2021-05-25T11:55:54.322 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:54.345 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:54 +0000 (0:00:00.061) 0:03:41.693 *********** 2021-05-25T11:55:54.352 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-05-25T11:55:54.353 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:54.376 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:54 +0000 (0:00:00.030) 0:03:41.724 *********** 2021-05-25T11:55:54.414 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:54.441 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:54 +0000 (0:00:00.064) 0:03:41.789 *********** 2021-05-25T11:55:54.446 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2021-05-25T11:55:54.521 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Tuesday 25 May 2021 11:55:54 +0000 (0:00:00.029) 0:03:41.818 *********** skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:54.532 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:54 +0000 (0:00:00.061) 0:03:41.880 *********** 2021-05-25T11:55:54.581 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages with --no-install-recommends] **************** 2021-05-25T11:55:54.583 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:54.604 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:54 +0000 (0:00:00.071) 0:03:41.952 *********** 2021-05-25T11:55:54.610 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:54.633 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:54 +0000 (0:00:00.029) 0:03:41.981 *********** 2021-05-25T11:55:54.660 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2021-05-25T11:55:54.662 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:54.684 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:54 +0000 (0:00:00.050) 0:03:42.032 *********** 2021-05-25T11:55:54.722 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:54.745 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:54 +0000 (0:00:00.061) 0:03:42.093 *********** 2021-05-25T11:55:54.751 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2021-05-25T11:55:54.752 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:54.775 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:54 +0000 (0:00:00.029) 0:03:42.123 *********** 2021-05-25T11:55:54.812 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:54.836 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:54 +0000 (0:00:00.060) 0:03:42.184 *********** 2021-05-25T11:55:54.873 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2021-05-25T11:55:54.875 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:54.897 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:54 +0000 (0:00:00.061) 0:03:42.245 *********** 2021-05-25T11:55:54.904 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:54.932 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:54 +0000 (0:00:00.034) 0:03:42.279 *********** 2021-05-25T11:55:54.961 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2021-05-25T11:55:54.963 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:54.988 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:54 +0000 (0:00:00.056) 0:03:42.336 *********** 2021-05-25T11:55:55.007 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:55.033 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:55 +0000 (0:00:00.044) 0:03:42.381 *********** 2021-05-25T11:55:55.051 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2021-05-25T11:55:55.053 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:55.076 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:55 +0000 (0:00:00.043) 0:03:42.424 *********** 2021-05-25T11:55:55.104 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:55.127 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:55 +0000 (0:00:00.051) 0:03:42.475 *********** 2021-05-25T11:55:55.145 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2021-05-25T11:55:55.146 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:55.169 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:55 +0000 (0:00:00.041) 0:03:42.517 *********** 2021-05-25T11:55:55.198 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:55.221 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:55 +0000 (0:00:00.051) 0:03:42.569 *********** 2021-05-25T11:55:55.238 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2021-05-25T11:55:55.239 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:55.263 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:55 +0000 (0:00:00.041) 0:03:42.611 *********** 2021-05-25T11:55:55.292 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:55.314 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:55 +0000 (0:00:00.051) 0:03:42.662 *********** 2021-05-25T11:55:55.331 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-05-25T11:55:55.333 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:55.355 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:55 +0000 (0:00:00.040) 0:03:42.703 *********** 2021-05-25T11:55:55.383 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:55.406 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:55 +0000 (0:00:00.051) 0:03:42.754 *********** 2021-05-25T11:55:55.434 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-05-25T11:55:55.436 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:55.459 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:55 +0000 (0:00:00.052) 0:03:42.807 *********** 2021-05-25T11:55:55.476 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:55.499 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:55 +0000 (0:00:00.039) 0:03:42.847 *********** 2021-05-25T11:55:55.525 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-05-25T11:55:55.527 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:55.549 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:55 +0000 (0:00:00.050) 0:03:42.897 *********** 2021-05-25T11:55:55.566 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:55.590 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:55 +0000 (0:00:00.040) 0:03:42.938 *********** 2021-05-25T11:55:55.616 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-05-25T11:55:55.618 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:55.639 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:55 +0000 (0:00:00.049) 0:03:42.987 *********** 2021-05-25T11:55:55.656 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:55.678 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:55 +0000 (0:00:00.039) 0:03:43.026 *********** 2021-05-25T11:55:55.705 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-05-25T11:55:55.707 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:55.727 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:55 +0000 (0:00:00.049) 0:03:43.076 *********** 2021-05-25T11:55:55.744 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:55.767 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:55 +0000 (0:00:00.039) 0:03:43.115 *********** 2021-05-25T11:55:55.794 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-05-25T11:55:55.795 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:55.817 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:55 +0000 (0:00:00.049) 0:03:43.165 *********** 2021-05-25T11:55:55.834 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:55.857 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:55 +0000 (0:00:00.040) 0:03:43.205 *********** 2021-05-25T11:55:55.884 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-05-25T11:55:55.886 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:55.907 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:55 +0000 (0:00:00.050) 0:03:43.255 *********** 2021-05-25T11:55:55.924 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:55.948 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:55 +0000 (0:00:00.040) 0:03:43.296 *********** 2021-05-25T11:55:55.975 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-05-25T11:55:55.977 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:55.999 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:55 +0000 (0:00:00.051) 0:03:43.347 *********** 2021-05-25T11:55:56.016 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:56.039 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:56 +0000 (0:00:00.039) 0:03:43.387 *********** 2021-05-25T11:55:56.066 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-05-25T11:55:56.068 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:56.089 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:56 +0000 (0:00:00.049) 0:03:43.437 *********** 2021-05-25T11:55:56.106 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:56.129 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:56 +0000 (0:00:00.039) 0:03:43.477 *********** 2021-05-25T11:55:56.155 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-05-25T11:55:56.157 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:56.179 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:56 +0000 (0:00:00.050) 0:03:43.527 *********** 2021-05-25T11:55:56.196 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:56.218 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:56 +0000 (0:00:00.039) 0:03:43.566 *********** 2021-05-25T11:55:56.245 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-05-25T11:55:56.247 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:56.269 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:56 +0000 (0:00:00.050) 0:03:43.617 *********** 2021-05-25T11:55:56.286 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:56.309 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:56 +0000 (0:00:00.039) 0:03:43.657 *********** 2021-05-25T11:55:56.336 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-05-25T11:55:56.338 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:56.359 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:56 +0000 (0:00:00.050) 0:03:43.707 *********** 2021-05-25T11:55:56.377 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:56.400 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:56 +0000 (0:00:00.040) 0:03:43.748 *********** 2021-05-25T11:55:56.428 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-05-25T11:55:56.430 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:56.452 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:56 +0000 (0:00:00.051) 0:03:43.800 *********** 2021-05-25T11:55:56.470 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:56.493 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:56 +0000 (0:00:00.041) 0:03:43.841 *********** 2021-05-25T11:55:56.699 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-05-25T11:55:56.701 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/bin/lspci", "gid": 0, "group": "root", "mode": "0777", "owner": "root", "secontext": "unconfined_u:object_r:bin_t:s0", "size": 11, "src": "/sbin/lspci", "state": "link", "uid": 0} 2021-05-25T11:55:56.722 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:56 +0000 (0:00:00.228) 0:03:44.070 *********** 2021-05-25T11:55:56.739 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/bin/lspci", "gid": 0, "group": "root", "mode": "0777", "owner": "root", "secontext": "unconfined_u:object_r:bin_t:s0", "size": 11, "src": "/sbin/lspci", "state": "link", "uid": 0} 2021-05-25T11:55:56.767 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:56 +0000 (0:00:00.044) 0:03:44.115 *********** 2021-05-25T11:55:56.952 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-05-25T11:55:56.954 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/ltp/testcases/bin", "secontext": "unconfined_u:object_r:lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-05-25T11:55:56.975 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:56 +0000 (0:00:00.208) 0:03:44.323 *********** 2021-05-25T11:55:57.002 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/ltp/testcases/bin", "secontext": "unconfined_u:object_r:lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-05-25T11:55:57.026 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:57 +0000 (0:00:00.050) 0:03:44.374 *********** 2021-05-25T11:55:57.208 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-05-25T11:55:57.210 INFO:teuthology.task.ansible.out:[WARNING]: Cannot set fs attributes on a non-existent symlink target. follow should be set to False to avoid this. ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/lib/ltp/testcases/bin/fsstress", "src": "/usr/bin/fsstress"} 2021-05-25T11:55:57.232 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:57 +0000 (0:00:00.206) 0:03:44.580 *********** 2021-05-25T11:55:57.249 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/lib/ltp/testcases/bin/fsstress", "src": "/usr/bin/fsstress"} 2021-05-25T11:55:57.273 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:57 +0000 (0:00:00.041) 0:03:44.621 *********** 2021-05-25T11:55:57.290 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-05-25T11:55:57.292 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:57.313 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:57 +0000 (0:00:00.039) 0:03:44.661 *********** 2021-05-25T11:55:57.339 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:57.362 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:57 +0000 (0:00:00.048) 0:03:44.710 *********** 2021-05-25T11:55:57.378 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-05-25T11:55:57.381 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:57.405 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:57 +0000 (0:00:00.043) 0:03:44.753 *********** 2021-05-25T11:55:57.442 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:57.466 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:57 +0000 (0:00:00.061) 0:03:44.814 *********** 2021-05-25T11:55:57.473 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-05-25T11:55:57.475 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:57.498 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:57 +0000 (0:00:00.031) 0:03:44.846 *********** 2021-05-25T11:55:57.536 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:57.560 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:57 +0000 (0:00:00.062) 0:03:44.908 *********** 2021-05-25T11:55:57.577 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-05-25T11:55:57.579 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:57.601 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:57 +0000 (0:00:00.040) 0:03:44.949 *********** 2021-05-25T11:55:57.629 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:57.652 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:57 +0000 (0:00:00.051) 0:03:45.000 *********** 2021-05-25T11:55:57.669 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-05-25T11:55:57.671 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:57.694 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:57 +0000 (0:00:00.041) 0:03:45.042 *********** 2021-05-25T11:55:57.722 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:57.745 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:57 +0000 (0:00:00.051) 0:03:45.093 *********** 2021-05-25T11:55:57.762 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-05-25T11:55:57.764 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:57.786 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:57 +0000 (0:00:00.040) 0:03:45.133 *********** 2021-05-25T11:55:57.814 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:57.838 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:57 +0000 (0:00:00.052) 0:03:45.186 *********** 2021-05-25T11:55:57.861 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-05-25T11:55:57.863 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:57.884 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:57 +0000 (0:00:00.045) 0:03:45.232 *********** 2021-05-25T11:55:57.912 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:57.939 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:57 +0000 (0:00:00.055) 0:03:45.287 *********** 2021-05-25T11:55:57.957 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-05-25T11:55:57.958 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:57.983 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:57 +0000 (0:00:00.043) 0:03:45.331 *********** 2021-05-25T11:55:58.001 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:58.026 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:58 +0000 (0:00:00.042) 0:03:45.374 *********** 2021-05-25T11:55:58.045 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-05-25T11:55:58.046 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:58.068 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:58 +0000 (0:00:00.041) 0:03:45.416 *********** 2021-05-25T11:55:58.094 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:58.118 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:58 +0000 (0:00:00.049) 0:03:45.466 *********** 2021-05-25T11:55:58.135 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2021-05-25T11:55:58.136 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:58.158 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:58 +0000 (0:00:00.040) 0:03:45.506 *********** 2021-05-25T11:55:58.185 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:58.209 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:58 +0000 (0:00:00.050) 0:03:45.557 *********** 2021-05-25T11:55:58.226 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2021-05-25T11:55:58.227 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:58.249 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:58 +0000 (0:00:00.039) 0:03:45.597 *********** 2021-05-25T11:55:58.276 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:58.300 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:58 +0000 (0:00:00.050) 0:03:45.648 *********** 2021-05-25T11:55:58.317 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-05-25T11:55:58.318 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:58.340 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:58 +0000 (0:00:00.040) 0:03:45.688 *********** 2021-05-25T11:55:58.368 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:58.392 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:58 +0000 (0:00:00.051) 0:03:45.740 *********** 2021-05-25T11:55:58.409 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-05-25T11:55:58.410 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:58.432 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:58 +0000 (0:00:00.040) 0:03:45.780 *********** 2021-05-25T11:55:58.459 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:58.483 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:58 +0000 (0:00:00.051) 0:03:45.831 *********** 2021-05-25T11:55:58.521 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2021-05-25T11:55:58.522 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:58.544 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:58 +0000 (0:00:00.060) 0:03:45.892 *********** 2021-05-25T11:55:58.560 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:58.583 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:58 +0000 (0:00:00.039) 0:03:45.931 *********** 2021-05-25T11:55:58.620 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2021-05-25T11:55:58.622 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:58.644 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:58 +0000 (0:00:00.060) 0:03:45.992 *********** 2021-05-25T11:55:58.660 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:58.683 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:58 +0000 (0:00:00.039) 0:03:46.031 *********** 2021-05-25T11:55:58.710 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2021-05-25T11:55:58.712 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:58.733 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:58 +0000 (0:00:00.049) 0:03:46.081 *********** 2021-05-25T11:55:58.750 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:58.773 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:58 +0000 (0:00:00.039) 0:03:46.121 *********** 2021-05-25T11:55:58.800 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2021-05-25T11:55:58.802 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:58.823 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:58 +0000 (0:00:00.049) 0:03:46.171 *********** 2021-05-25T11:55:58.850 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:58.873 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:58 +0000 (0:00:00.049) 0:03:46.221 *********** 2021-05-25T11:55:58.889 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2021-05-25T11:55:58.891 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:58.913 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:58 +0000 (0:00:00.040) 0:03:46.261 *********** 2021-05-25T11:55:58.952 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:58.978 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:58 +0000 (0:00:00.064) 0:03:46.326 *********** 2021-05-25T11:55:58.994 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2021-05-25T11:55:58.996 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:59.017 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:59 +0000 (0:00:00.039) 0:03:46.365 *********** 2021-05-25T11:55:59.044 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:59.068 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:59 +0000 (0:00:00.051) 0:03:46.416 *********** 2021-05-25T11:55:59.107 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2021-05-25T11:55:59.109 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:59.141 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:59 +0000 (0:00:00.072) 0:03:46.489 *********** 2021-05-25T11:55:59.149 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:59.173 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:59 +0000 (0:00:00.031) 0:03:46.520 *********** 2021-05-25T11:55:59.210 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2021-05-25T11:55:59.212 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:59.235 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:59 +0000 (0:00:00.062) 0:03:46.583 *********** 2021-05-25T11:55:59.253 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:59.279 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:59 +0000 (0:00:00.044) 0:03:46.627 *********** 2021-05-25T11:55:59.298 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2021-05-25T11:55:59.300 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:59.323 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:59 +0000 (0:00:00.043) 0:03:46.671 *********** 2021-05-25T11:55:59.350 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:59.373 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:59 +0000 (0:00:00.050) 0:03:46.721 *********** 2021-05-25T11:55:59.390 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** ok: [smithi102.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2021-05-25T11:55:59.413 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:59 +0000 (0:00:00.040) 0:03:46.761 *********** 2021-05-25T11:55:59.451 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2021-05-25T11:55:59.475 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:59 +0000 (0:00:00.061) 0:03:46.823 *********** 2021-05-25T11:55:59.492 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2021-05-25T11:55:59.493 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2021-05-25T11:55:59.503 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2021-05-25T11:55:59.505 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2021-05-25T11:55:59.514 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2021-05-25T11:55:59.516 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2021-05-25T11:55:59.525 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2021-05-25T11:55:59.527 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2021-05-25T11:55:59.553 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:59 +0000 (0:00:00.077) 0:03:46.901 *********** 2021-05-25T11:55:59.560 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2021-05-25T11:55:59.561 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2021-05-25T11:55:59.572 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2021-05-25T11:55:59.583 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2021-05-25T11:55:59.584 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2021-05-25T11:55:59.594 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2021-05-25T11:55:59.595 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2021-05-25T11:55:59.621 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:59 +0000 (0:00:00.068) 0:03:46.969 *********** 2021-05-25T11:55:59.813 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2021-05-25T11:55:59.815 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003011", "end": "2021-05-25 11:55:59.787771", "rc": 0, "start": "2021-05-25 11:55:59.784760", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_d92ec1407c30d60262a45a2844110a5c94d567cc/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_d92ec1407c30d60262a45a2844110a5c94d567cc/virtualenv/bin/teuthology-lock"]} 2021-05-25T11:55:59.841 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:59 +0000 (0:00:00.219) 0:03:47.189 *********** 2021-05-25T11:55:59.858 INFO:teuthology.task.ansible.out:changed: [smithi181.front.sepia.ceph.com] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003480", "end": "2021-05-25 11:55:59.837311", "rc": 0, "start": "2021-05-25 11:55:59.833831", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_d92ec1407c30d60262a45a2844110a5c94d567cc/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_d92ec1407c30d60262a45a2844110a5c94d567cc/virtualenv/bin/teuthology-lock"]} 2021-05-25T11:55:59.885 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:59 +0000 (0:00:00.044) 0:03:47.233 *********** 2021-05-25T11:55:59.912 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2021-05-25T11:55:59.913 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:59.937 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:59 +0000 (0:00:00.052) 0:03:47.285 *********** 2021-05-25T11:55:59.954 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:55:59.985 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:55:59 +0000 (0:00:00.047) 0:03:47.333 *********** 2021-05-25T11:56:00.004 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2021-05-25T11:56:00.006 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:00.028 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:00 +0000 (0:00:00.042) 0:03:47.376 *********** 2021-05-25T11:56:00.065 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:00.091 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:00 +0000 (0:00:00.063) 0:03:47.439 *********** 2021-05-25T11:56:00.096 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2021-05-25T11:56:00.098 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:00.121 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:00 +0000 (0:00:00.029) 0:03:47.469 *********** 2021-05-25T11:56:00.158 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:00.182 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:00 +0000 (0:00:00.061) 0:03:47.530 *********** 2021-05-25T11:56:00.189 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2021-05-25T11:56:00.190 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:00.212 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:00 +0000 (0:00:00.030) 0:03:47.560 *********** 2021-05-25T11:56:00.239 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:00.263 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:00 +0000 (0:00:00.050) 0:03:47.611 *********** 2021-05-25T11:56:01.587 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2021-05-25T11:56:01.588 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-05-25T11:56:01.611 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:01 +0000 (0:00:01.347) 0:03:48.959 *********** 2021-05-25T11:56:01.628 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-05-25T11:56:01.659 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:01 +0000 (0:00:00.048) 0:03:49.007 *********** 2021-05-25T11:56:01.687 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-05-25T11:56:01.689 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930278215680, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227118705, 'block_used': 12976403, 'inode_total': 61054976, 'inode_available': 60985790, 'inode_used': 69186, 'uuid': 'a9bc468f-7d76-441a-b17b-cd059807719c'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 227118705, "block_size": 4096, "block_total": 240095108, "block_used": 12976403, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60985790, "inode_total": 61054976, "inode_used": 69186, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930278215680, "size_total": 983429562368, "uuid": "a9bc468f-7d76-441a-b17b-cd059807719c"}} 2021-05-25T11:56:01.724 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:01 +0000 (0:00:00.065) 0:03:49.072 *********** 2021-05-25T11:56:01.752 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930278215680, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227118705, 'block_used': 12976403, 'inode_total': 61054976, 'inode_available': 60985790, 'inode_used': 69186, 'uuid': 'a9bc468f-7d76-441a-b17b-cd059807719c'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 227118705, "block_size": 4096, "block_total": 240095108, "block_used": 12976403, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60985790, "inode_total": 61054976, "inode_used": 69186, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930278215680, "size_total": 983429562368, "uuid": "a9bc468f-7d76-441a-b17b-cd059807719c"}} 2021-05-25T11:56:01.778 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:01 +0000 (0:00:00.053) 0:03:49.126 *********** 2021-05-25T11:56:01.961 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2021-05-25T11:56:01.963 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.004931", "end": "2021-05-25 11:56:01.938481", "rc": 0, "start": "2021-05-25 11:56:01.933550", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-05-25T11:56:01.990 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:01 +0000 (0:00:00.211) 0:03:49.338 *********** 2021-05-25T11:56:02.018 INFO:teuthology.task.ansible.out:changed: [smithi181.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.005014", "end": "2021-05-25 11:56:01.988985", "rc": 0, "start": "2021-05-25 11:56:01.983971", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-05-25T11:56:02.042 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:02 +0000 (0:00:00.052) 0:03:49.390 *********** 2021-05-25T11:56:02.079 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2021-05-25T11:56:02.081 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930278215680, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227118705, 'block_used': 12976403, 'inode_total': 61054976, 'inode_available': 60985790, 'inode_used': 69186, 'uuid': 'a9bc468f-7d76-441a-b17b-cd059807719c'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227118705, "block_size": 4096, "block_total": 240095108, "block_used": 12976403, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60985790, "inode_total": 61054976, "inode_used": 69186, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930278215680, "size_total": 983429562368, "uuid": "a9bc468f-7d76-441a-b17b-cd059807719c"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:02.109 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:02 +0000 (0:00:00.066) 0:03:49.456 *********** 2021-05-25T11:56:02.137 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930278215680, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227118705, 'block_used': 12976403, 'inode_total': 61054976, 'inode_available': 60985790, 'inode_used': 69186, 'uuid': 'a9bc468f-7d76-441a-b17b-cd059807719c'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227118705, "block_size": 4096, "block_total": 240095108, "block_used": 12976403, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60985790, "inode_total": 61054976, "inode_used": 69186, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930278215680, "size_total": 983429562368, "uuid": "a9bc468f-7d76-441a-b17b-cd059807719c"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:02.161 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:02 +0000 (0:00:00.051) 0:03:49.509 *********** 2021-05-25T11:56:02.177 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2021-05-25T11:56:02.201 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:02 +0000 (0:00:00.040) 0:03:49.549 *********** 2021-05-25T11:56:02.252 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:02 +0000 (0:00:00.051) 0:03:49.600 *********** 2021-05-25T11:56:03.740 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-05-25T11:56:03.741 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-EK0ncn7E5SIwe47HJwnF6sJI6nY636I4VeAKTkv0gG8LWgdJarA0UydsAwenImVy'], 'uuids': ['b83d7151-8e03-4bcb-ae30-49037193015d'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.290739", "end": "2021-05-25 11:56:03.705633", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-EK0ncn7E5SIwe47HJwnF6sJI6nY636I4VeAKTkv0gG8LWgdJarA0UydsAwenImVy"], "labels": [], "masters": [], "uuids": ["b83d7151-8e03-4bcb-ae30-49037193015d"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-05-25 11:56:02.414894", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-05-25T11:56:03.817 INFO:teuthology.task.ansible.out:changed: [smithi181.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-406scBhL2gTJOm9YJzlfYQXtwS1Be0JhQDzd8PtqPWfuQcAgRjVUxEXKKgcU16yT'], 'uuids': ['b60de22c-0b23-40c2-8e6c-72c15d8266e0'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.301542", "end": "2021-05-25 11:56:03.782322", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-406scBhL2gTJOm9YJzlfYQXtwS1Be0JhQDzd8PtqPWfuQcAgRjVUxEXKKgcU16yT"], "labels": [], "masters": [], "uuids": ["b60de22c-0b23-40c2-8e6c-72c15d8266e0"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-05-25 11:56:02.480780", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-05-25T11:56:05.004 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-4YEYh3-Cc6n-9vwL-1gLd-RHTl-nLvG-Ee3xRX', 'nvme-INTEL_SSDPEDMD400G4_PHFT620400VK400BGN', 'nvme-nvme.8086-50484654363230343030564b34303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPEDMD400G4', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.074941", "end": "2021-05-25 11:56:04.971701", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-4YEYh3-Cc6n-9vwL-1gLd-RHTl-nLvG-Ee3xRX", "nvme-INTEL_SSDPEDMD400G4_PHFT620400VK400BGN", "nvme-nvme.8086-50484654363230343030564b34303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-05-25 11:56:03.896760", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! 2021-05-25T11:56:05.005 INFO:teuthology.task.ansible.out:You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-05-25T11:56:05.067 INFO:teuthology.task.ansible.out:changed: [smithi181.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-tL4kIH-GQcy-Rqt6-fsAH-Rg2b-4XQn-FfNoO0', 'nvme-INTEL_SSDPEDMD400G4_CVFT6246000E400BGN', 'nvme-nvme.8086-43564654363234363030304534303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPEDMD400G4', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.074214", "end": "2021-05-25 11:56:05.039676", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-tL4kIH-GQcy-Rqt6-fsAH-Rg2b-4XQn-FfNoO0", "nvme-INTEL_SSDPEDMD400G4_CVFT6246000E400BGN", "nvme-nvme.8086-43564654363234363030304534303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-05-25 11:56:03.965462", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! 2021-05-25T11:56:05.069 INFO:teuthology.task.ansible.out:You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-05-25T11:56:06.246 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-EK0ncn7E5SIwe47HJwnF6sJI6nY636I4yw97UpCkvSgeGLo6ab5JiNgZLkHniXjB'], 'uuids': ['55ab2e9a-45c1-42dc-beaf-649e710eccc0'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.061284", "end": "2021-05-25 11:56:06.214401", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-EK0ncn7E5SIwe47HJwnF6sJI6nY636I4yw97UpCkvSgeGLo6ab5JiNgZLkHniXjB"], "labels": [], "masters": [], "uuids": ["55ab2e9a-45c1-42dc-beaf-649e710eccc0"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-05-25 11:56:05.153117", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-05-25T11:56:06.320 INFO:teuthology.task.ansible.out:changed: [smithi181.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-406scBhL2gTJOm9YJzlfYQXtwS1Be0JhgEww0R8pG4c3Z4fxQhdfj0JTMhba9b78'], 'uuids': ['cec993cf-0fcd-4414-a30b-9c372dd6eaa4'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.068831", "end": "2021-05-25 11:56:06.289992", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-406scBhL2gTJOm9YJzlfYQXtwS1Be0JhgEww0R8pG4c3Z4fxQhdfj0JTMhba9b78"], "labels": [], "masters": [], "uuids": ["cec993cf-0fcd-4414-a30b-9c372dd6eaa4"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-05-25 11:56:05.221161", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-05-25T11:56:07.495 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-EK0ncn7E5SIwe47HJwnF6sJI6nY636I42LE1iymxFOQwouR9LER8MGR7GCzen0TH'], 'uuids': ['7adc5867-e939-46fa-ae79-b6131d7a1eb6'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.072431", "end": "2021-05-25 11:56:07.469151", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-EK0ncn7E5SIwe47HJwnF6sJI6nY636I42LE1iymxFOQwouR9LER8MGR7GCzen0TH"], "labels": [], "masters": [], "uuids": ["7adc5867-e939-46fa-ae79-b6131d7a1eb6"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-05-25 11:56:06.396720", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-05-25T11:56:07.559 INFO:teuthology.task.ansible.out:changed: [smithi181.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-406scBhL2gTJOm9YJzlfYQXtwS1Be0Jh6PPWU3mCchuKxUeg64rwTe8aUepMyyzr'], 'uuids': ['075a964e-317a-44ee-ba57-5c9f3f6457f4'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.060113", "end": "2021-05-25 11:56:07.531453", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-406scBhL2gTJOm9YJzlfYQXtwS1Be0Jh6PPWU3mCchuKxUeg64rwTe8aUepMyyzr"], "labels": [], "masters": [], "uuids": ["075a964e-317a-44ee-ba57-5c9f3f6457f4"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-05-25 11:56:06.471340", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-05-25T11:56:08.737 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-EK0ncn7E5SIwe47HJwnF6sJI6nY636I4PFzPN7oM0IeviO5Ak0dxMCB3B2FZpOQZ'], 'uuids': ['3d519848-ca5e-43f6-a410-4db79ab7a963'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.050882", "end": "2021-05-25 11:56:08.706086", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-EK0ncn7E5SIwe47HJwnF6sJI6nY636I4PFzPN7oM0IeviO5Ak0dxMCB3B2FZpOQZ"], "labels": [], "masters": [], "uuids": ["3d519848-ca5e-43f6-a410-4db79ab7a963"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-05-25 11:56:07.655204", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-05-25T11:56:08.738 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5QPPM', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5QPPM', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5QPPM', 'scsi-35000c50091e0bcf8', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5QPPM', 'wwn-0x5000c50091e0bcf8'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W5QPPM', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c50091e0bcf8', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5QPPM-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5QPPM-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5QPPM-part1', 'scsi-35000c50091e0bcf8-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5QPPM-part1', 'wwn-0x5000c50091e0bcf8-part1'], 'uuids': ['a9bc468f-7d76-441a-b17b-cd059807719c'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': 'a9bc468f-7d76-441a-b17b-cd059807719c', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QPPM", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5QPPM", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W5QPPM", "scsi-35000c50091e0bcf8", "scsi-SATA_ST1000NM0033-9ZM_Z1W5QPPM", "wwn-0x5000c50091e0bcf8"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QPPM-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5QPPM-part1", "scsi-1ATA_ST1000NM0033-9ZM173_ 2021-05-25T11:56:08.740 INFO:teuthology.task.ansible.out:Z1W5QPPM-part1", "scsi-35000c50091e0bcf8-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W5QPPM-part1", "wwn-0x5000c50091e0bcf8-part1"], "labels": [], "masters": [], "uuids": ["a9bc468f-7d76-441a-b17b-cd059807719c"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "a9bc468f-7d76-441a-b17b-cd059807719c"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5QPPM", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50091e0bcf8"}}, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:08.823 INFO:teuthology.task.ansible.out:changed: [smithi181.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-406scBhL2gTJOm9YJzlfYQXtwS1Be0Jhj8jpW6tU7bDew0yoTtjlKprH2aLL6YMr'], 'uuids': ['a76eea46-71d9-4006-b5b9-df448bcdd2bc'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.080704", "end": "2021-05-25 11:56:08.790836", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-406scBhL2gTJOm9YJzlfYQXtwS1Be0Jhj8jpW6tU7bDew0yoTtjlKprH2aLL6YMr"], "labels": [], "masters": [], "uuids": ["a76eea46-71d9-4006-b5b9-df448bcdd2bc"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-05-25 11:56:07.710132", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-05-25T11:56:08.825 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XTWX', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5XTWX', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XTWX', 'scsi-35000c5009293ee09', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5XTWX', 'wwn-0x5000c5009293ee09'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W5XTWX', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c5009293ee09', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XTWX-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5XTWX-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XTWX-part1', 'scsi-35000c5009293ee09-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5XTWX-part1', 'wwn-0x5000c5009293ee09-part1'], 'uuids': ['a9bc468f-7d76-441a-b17b-cd059807719c'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': 'a9bc468f-7d76-441a-b17b-cd059807719c', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XTWX", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5XTWX", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XTWX", "scsi-35000c5009293ee09", "scsi-SATA_ST1000NM0033-9ZM_Z1W5XTWX", "wwn-0x5000c5009293ee09"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XTWX-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5XTWX-part1", "scsi-1ATA_ST1000NM0033-9ZM173_ 2021-05-25T11:56:08.826 INFO:teuthology.task.ansible.out:Z1W5XTWX-part1", "scsi-35000c5009293ee09-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W5XTWX-part1", "wwn-0x5000c5009293ee09-part1"], "labels": [], "masters": [], "uuids": ["a9bc468f-7d76-441a-b17b-cd059807719c"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "a9bc468f-7d76-441a-b17b-cd059807719c"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XTWX", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c5009293ee09"}}, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:10.005 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-EK0ncn7E5SIwe47HJwnF6sJI6nY636I4xCC9hta6SBcC9vnXjRNbxv3G20r9vv6R'], 'uuids': ['be6af81b-2d55-4af9-9f4a-73e79e872026'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.087342", "end": "2021-05-25 11:56:09.979777", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-EK0ncn7E5SIwe47HJwnF6sJI6nY636I4xCC9hta6SBcC9vnXjRNbxv3G20r9vv6R"], "labels": [], "masters": [], "uuids": ["be6af81b-2d55-4af9-9f4a-73e79e872026"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-05-25 11:56:08.892435", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-05-25T11:56:10.045 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:10 +0000 (0:00:07.792) 0:03:57.393 *********** 2021-05-25T11:56:10.068 INFO:teuthology.task.ansible.out:changed: [smithi181.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-406scBhL2gTJOm9YJzlfYQXtwS1Be0JhVgb1bfytdhjoDvzovdyTHuhfS4WeSBW6'], 'uuids': ['112c0f34-abbd-4422-8727-e48c32f5b6a7'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.064285", "end": "2021-05-25 11:56:10.040554", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-406scBhL2gTJOm9YJzlfYQXtwS1Be0JhVgb1bfytdhjoDvzovdyTHuhfS4WeSBW6"], "labels": [], "masters": [], "uuids": ["112c0f34-abbd-4422-8727-e48c32f5b6a7"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-05-25 11:56:08.976269", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-05-25T11:56:10.109 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:10 +0000 (0:00:00.064) 0:03:57.457 *********** 2021-05-25T11:56:10.447 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-05-25T11:56:10.449 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-EK0ncn7E5SIwe47HJwnF6sJI6nY636I4VeAKTkv0gG8LWgdJarA0UydsAwenImVy'], 'uuids': ['b83d7151-8e03-4bcb-ae30-49037193015d'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.140091", "end": "2021-05-25 11:56:10.420658", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-EK0ncn7E5SIwe47HJwnF6sJI6nY636I4VeAKTkv0gG8LWgdJarA0UydsAwenImVy"], "labels": [], "masters": [], "uuids": ["b83d7151-8e03-4bcb-ae30-49037193015d"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-05-25 11:56:10.280567", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.137549 s, 839 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.137549 s, 839 MB/s"], "stdout": "", "stdout_lines": []} 2021-05-25T11:56:10.512 INFO:teuthology.task.ansible.out:changed: [smithi181.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-406scBhL2gTJOm9YJzlfYQXtwS1Be0JhQDzd8PtqPWfuQcAgRjVUxEXKKgcU16yT'], 'uuids': ['b60de22c-0b23-40c2-8e6c-72c15d8266e0'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.139770", "end": "2021-05-25 11:56:10.483496", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-406scBhL2gTJOm9YJzlfYQXtwS1Be0JhQDzd8PtqPWfuQcAgRjVUxEXKKgcU16yT"], "labels": [], "masters": [], "uuids": ["b60de22c-0b23-40c2-8e6c-72c15d8266e0"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-05-25 11:56:10.343726", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.137075 s, 841 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.137075 s, 841 MB/s"], "stdout": "", "stdout_lines": []} 2021-05-25T11:56:10.660 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-4YEYh3-Cc6n-9vwL-1gLd-RHTl-nLvG-Ee3xRX', 'nvme-INTEL_SSDPEDMD400G4_PHFT620400VK400BGN', 'nvme-nvme.8086-50484654363230343030564b34303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPEDMD400G4', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.037551", "end": "2021-05-25 11:56:10.634360", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-4YEYh3-Cc6n-9vwL-1gLd-RHTl-nLvG-Ee3xRX", "nvme-INTEL_SSDPEDMD400G4_PHFT620400VK400BGN", "nvme-nvme.8086-50484654363230343030564b34303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-05-25 11:56:10.596809", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0349196 s, 3.3 GB/s", "std 2021-05-25T11:56:10.662 INFO:teuthology.task.ansible.out:err_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0349196 s, 3.3 GB/s"], "stdout": "", "stdout_lines": []} 2021-05-25T11:56:10.724 INFO:teuthology.task.ansible.out:changed: [smithi181.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-tL4kIH-GQcy-Rqt6-fsAH-Rg2b-4XQn-FfNoO0', 'nvme-INTEL_SSDPEDMD400G4_CVFT6246000E400BGN', 'nvme-nvme.8086-43564654363234363030304534303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPEDMD400G4', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.037770", "end": "2021-05-25 11:56:10.692247", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-tL4kIH-GQcy-Rqt6-fsAH-Rg2b-4XQn-FfNoO0", "nvme-INTEL_SSDPEDMD400G4_CVFT6246000E400BGN", "nvme-nvme.8086-43564654363234363030304534303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-05-25 11:56:10.654477", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.035122 s, 3.3 GB/s", "stde 2021-05-25T11:56:10.726 INFO:teuthology.task.ansible.out:rr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.035122 s, 3.3 GB/s"], "stdout": "", "stdout_lines": []} 2021-05-25T11:56:10.975 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-EK0ncn7E5SIwe47HJwnF6sJI6nY636I4yw97UpCkvSgeGLo6ab5JiNgZLkHniXjB'], 'uuids': ['55ab2e9a-45c1-42dc-beaf-649e710eccc0'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.140356", "end": "2021-05-25 11:56:10.953291", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-EK0ncn7E5SIwe47HJwnF6sJI6nY636I4yw97UpCkvSgeGLo6ab5JiNgZLkHniXjB"], "labels": [], "masters": [], "uuids": ["55ab2e9a-45c1-42dc-beaf-649e710eccc0"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-05-25 11:56:10.812935", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.137727 s, 837 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.137727 s, 837 MB/s"], "stdout": "", "stdout_lines": []} 2021-05-25T11:56:11.038 INFO:teuthology.task.ansible.out:changed: [smithi181.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-406scBhL2gTJOm9YJzlfYQXtwS1Be0JhgEww0R8pG4c3Z4fxQhdfj0JTMhba9b78'], 'uuids': ['cec993cf-0fcd-4414-a30b-9c372dd6eaa4'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.142931", "end": "2021-05-25 11:56:11.012932", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-406scBhL2gTJOm9YJzlfYQXtwS1Be0JhgEww0R8pG4c3Z4fxQhdfj0JTMhba9b78"], "labels": [], "masters": [], "uuids": ["cec993cf-0fcd-4414-a30b-9c372dd6eaa4"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-05-25 11:56:10.870001", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.140306 s, 822 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.140306 s, 822 MB/s"], "stdout": "", "stdout_lines": []} 2021-05-25T11:56:11.287 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-EK0ncn7E5SIwe47HJwnF6sJI6nY636I42LE1iymxFOQwouR9LER8MGR7GCzen0TH'], 'uuids': ['7adc5867-e939-46fa-ae79-b6131d7a1eb6'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.139546", "end": "2021-05-25 11:56:11.262735", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-EK0ncn7E5SIwe47HJwnF6sJI6nY636I42LE1iymxFOQwouR9LER8MGR7GCzen0TH"], "labels": [], "masters": [], "uuids": ["7adc5867-e939-46fa-ae79-b6131d7a1eb6"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-05-25 11:56:11.123189", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.136977 s, 842 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.136977 s, 842 MB/s"], "stdout": "", "stdout_lines": []} 2021-05-25T11:56:11.351 INFO:teuthology.task.ansible.out:changed: [smithi181.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-406scBhL2gTJOm9YJzlfYQXtwS1Be0Jh6PPWU3mCchuKxUeg64rwTe8aUepMyyzr'], 'uuids': ['075a964e-317a-44ee-ba57-5c9f3f6457f4'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.140969", "end": "2021-05-25 11:56:11.322348", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-406scBhL2gTJOm9YJzlfYQXtwS1Be0Jh6PPWU3mCchuKxUeg64rwTe8aUepMyyzr"], "labels": [], "masters": [], "uuids": ["075a964e-317a-44ee-ba57-5c9f3f6457f4"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-05-25 11:56:11.181379", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.138407 s, 833 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.138407 s, 833 MB/s"], "stdout": "", "stdout_lines": []} 2021-05-25T11:56:11.600 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-EK0ncn7E5SIwe47HJwnF6sJI6nY636I4PFzPN7oM0IeviO5Ak0dxMCB3B2FZpOQZ'], 'uuids': ['3d519848-ca5e-43f6-a410-4db79ab7a963'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.143121", "end": "2021-05-25 11:56:11.576350", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-EK0ncn7E5SIwe47HJwnF6sJI6nY636I4PFzPN7oM0IeviO5Ak0dxMCB3B2FZpOQZ"], "labels": [], "masters": [], "uuids": ["3d519848-ca5e-43f6-a410-4db79ab7a963"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-05-25 11:56:11.433229", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.140529 s, 821 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.140529 s, 821 MB/s"], "stdout": "", "stdout_lines": []} 2021-05-25T11:56:11.612 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5QPPM', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5QPPM', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5QPPM', 'scsi-35000c50091e0bcf8', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5QPPM', 'wwn-0x5000c50091e0bcf8'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W5QPPM', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c50091e0bcf8', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5QPPM-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5QPPM-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5QPPM-part1', 'scsi-35000c50091e0bcf8-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5QPPM-part1', 'wwn-0x5000c50091e0bcf8-part1'], 'uuids': ['a9bc468f-7d76-441a-b17b-cd059807719c'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': 'a9bc468f-7d76-441a-b17b-cd059807719c', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QPPM", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5QPPM", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W5QPPM", "scsi-35000c50091e0bcf8", "scsi-SATA_ST1000NM0033-9ZM_Z1W5QPPM", "wwn-0x5000c50091e0bcf8"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QPPM-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5QPPM-part1", "scsi-1ATA_ST1000NM0033-9ZM173_ 2021-05-25T11:56:11.614 INFO:teuthology.task.ansible.out:Z1W5QPPM-part1", "scsi-35000c50091e0bcf8-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W5QPPM-part1", "wwn-0x5000c50091e0bcf8-part1"], "labels": [], "masters": [], "uuids": ["a9bc468f-7d76-441a-b17b-cd059807719c"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "a9bc468f-7d76-441a-b17b-cd059807719c"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5QPPM", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50091e0bcf8"}}, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:11.665 INFO:teuthology.task.ansible.out:changed: [smithi181.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-406scBhL2gTJOm9YJzlfYQXtwS1Be0Jhj8jpW6tU7bDew0yoTtjlKprH2aLL6YMr'], 'uuids': ['a76eea46-71d9-4006-b5b9-df448bcdd2bc'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.142527", "end": "2021-05-25 11:56:11.634878", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-406scBhL2gTJOm9YJzlfYQXtwS1Be0Jhj8jpW6tU7bDew0yoTtjlKprH2aLL6YMr"], "labels": [], "masters": [], "uuids": ["a76eea46-71d9-4006-b5b9-df448bcdd2bc"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-05-25 11:56:11.492351", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.139951 s, 824 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.139951 s, 824 MB/s"], "stdout": "", "stdout_lines": []} 2021-05-25T11:56:11.667 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XTWX', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5XTWX', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XTWX', 'scsi-35000c5009293ee09', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5XTWX', 'wwn-0x5000c5009293ee09'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W5XTWX', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c5009293ee09', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XTWX-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5XTWX-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XTWX-part1', 'scsi-35000c5009293ee09-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5XTWX-part1', 'wwn-0x5000c5009293ee09-part1'], 'uuids': ['a9bc468f-7d76-441a-b17b-cd059807719c'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': 'a9bc468f-7d76-441a-b17b-cd059807719c', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XTWX", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5XTWX", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XTWX", "scsi-35000c5009293ee09", "scsi-SATA_ST1000NM0033-9ZM_Z1W5XTWX", "wwn-0x5000c5009293ee09"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XTWX-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5XTWX-part1", "scsi-1ATA_ST1000NM0033-9ZM173_ 2021-05-25T11:56:11.669 INFO:teuthology.task.ansible.out:Z1W5XTWX-part1", "scsi-35000c5009293ee09-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W5XTWX-part1", "wwn-0x5000c5009293ee09-part1"], "labels": [], "masters": [], "uuids": ["a9bc468f-7d76-441a-b17b-cd059807719c"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "a9bc468f-7d76-441a-b17b-cd059807719c"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XTWX", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c5009293ee09"}}, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:11.916 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-EK0ncn7E5SIwe47HJwnF6sJI6nY636I4xCC9hta6SBcC9vnXjRNbxv3G20r9vv6R'], 'uuids': ['be6af81b-2d55-4af9-9f4a-73e79e872026'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.139794", "end": "2021-05-25 11:56:11.892244", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-EK0ncn7E5SIwe47HJwnF6sJI6nY636I4xCC9hta6SBcC9vnXjRNbxv3G20r9vv6R"], "labels": [], "masters": [], "uuids": ["be6af81b-2d55-4af9-9f4a-73e79e872026"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-05-25 11:56:11.752450", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.137298 s, 840 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.137298 s, 840 MB/s"], "stdout": "", "stdout_lines": []} 2021-05-25T11:56:11.947 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:11 +0000 (0:00:01.837) 0:03:59.295 *********** 2021-05-25T11:56:11.987 INFO:teuthology.task.ansible.out:changed: [smithi181.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-406scBhL2gTJOm9YJzlfYQXtwS1Be0JhVgb1bfytdhjoDvzovdyTHuhfS4WeSBW6'], 'uuids': ['112c0f34-abbd-4422-8727-e48c32f5b6a7'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.147605", "end": "2021-05-25 11:56:11.958614", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-406scBhL2gTJOm9YJzlfYQXtwS1Be0JhVgb1bfytdhjoDvzovdyTHuhfS4WeSBW6"], "labels": [], "masters": [], "uuids": ["112c0f34-abbd-4422-8727-e48c32f5b6a7"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-05-25 11:56:11.811009", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.144993 s, 796 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.144993 s, 796 MB/s"], "stdout": "", "stdout_lines": []} 2021-05-25T11:56:12.024 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:12 +0000 (0:00:00.076) 0:03:59.372 *********** 2021-05-25T11:56:12.218 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2021-05-25T11:56:12.220 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.047023", "end": "2021-05-25 11:56:12.150641", "rc": 0, "start": "2021-05-25 11:56:12.103618", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-05-25T11:56:12.253 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:12 +0000 (0:00:00.228) 0:03:59.601 *********** 2021-05-25T11:56:12.305 INFO:teuthology.task.ansible.out:changed: [smithi181.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.055490", "end": "2021-05-25 11:56:12.233549", "rc": 0, "start": "2021-05-25 11:56:12.178059", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-05-25T11:56:12.340 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:12 +0000 (0:00:00.087) 0:03:59.688 *********** 2021-05-25T11:56:12.473 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2021-05-25T11:56:12.475 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.043340", "end": "2021-05-25 11:56:12.452257", "rc": 0, "start": "2021-05-25 11:56:12.408917", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-05-25T11:56:12.497 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:12 +0000 (0:00:00.157) 0:03:59.845 *********** 2021-05-25T11:56:12.555 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2021-05-25T11:56:12.622 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:12 +0000 (0:00:00.124) 0:03:59.970 *********** 2021-05-25T11:56:12.629 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2021-05-25T11:56:12.631 INFO:teuthology.task.ansible.out:changed: [smithi181.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.045850", "end": "2021-05-25 11:56:12.541161", "rc": 0, "start": "2021-05-25 11:56:12.495311", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-05-25T11:56:12.654 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:12 +0000 (0:00:00.032) 0:04:00.002 *********** 2021-05-25T11:56:12.660 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2021-05-25T11:56:12.662 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:12.686 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:12 +0000 (0:00:00.031) 0:04:00.034 *********** 2021-05-25T11:56:12.692 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2021-05-25T11:56:12.717 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:12 +0000 (0:00:00.031) 0:04:00.065 *********** 2021-05-25T11:56:12.723 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2021-05-25T11:56:12.724 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:12.747 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:12 +0000 (0:00:00.029) 0:04:00.095 *********** 2021-05-25T11:56:12.752 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2021-05-25T11:56:12.754 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:12.777 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:12 +0000 (0:00:00.030) 0:04:00.125 *********** 2021-05-25T11:56:12.783 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-05-25T11:56:12.785 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:12.809 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:12 +0000 (0:00:00.031) 0:04:00.157 *********** 2021-05-25T11:56:12.816 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2021-05-25T11:56:12.818 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:12.839 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:12 +0000 (0:00:00.029) 0:04:00.187 *********** 2021-05-25T11:56:12.845 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2021-05-25T11:56:12.847 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:12.870 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:12 +0000 (0:00:00.031) 0:04:00.218 *********** 2021-05-25T11:56:12.876 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-05-25T11:56:12.878 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:12.905 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:12 +0000 (0:00:00.034) 0:04:00.252 *********** 2021-05-25T11:56:12.931 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2021-05-25T11:56:12.932 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:12.953 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:12 +0000 (0:00:00.048) 0:04:00.301 *********** 2021-05-25T11:56:13.051 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2021-05-25T11:56:13.053 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-05-25T11:56:13.075 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:13 +0000 (0:00:00.122) 0:04:00.423 *********** 2021-05-25T11:56:13.112 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-05-25T11:56:13.114 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930278215680, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227118705, 'block_used': 12976403, 'inode_total': 61054976, 'inode_available': 60985790, 'inode_used': 69186, 'uuid': 'a9bc468f-7d76-441a-b17b-cd059807719c'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227118705, "block_size": 4096, "block_total": 240095108, "block_used": 12976403, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60985790, "inode_total": 61054976, "inode_used": 69186, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930278215680, "size_total": 983429562368, "uuid": "a9bc468f-7d76-441a-b17b-cd059807719c"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:13.135 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:13 +0000 (0:00:00.060) 0:04:00.483 *********** 2021-05-25T11:56:13.142 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2021-05-25T11:56:13.144 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-05-25T11:56:13.166 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:13 +0000 (0:00:00.030) 0:04:00.514 *********** 2021-05-25T11:56:13.171 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2021-05-25T11:56:13.172 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:13.195 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:13 +0000 (0:00:00.029) 0:04:00.543 *********** 2021-05-25T11:56:13.202 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-05-25T11:56:13.204 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930278215680, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227118705, 'block_used': 12976403, 'inode_total': 61054976, 'inode_available': 60985790, 'inode_used': 69186, 'uuid': 'a9bc468f-7d76-441a-b17b-cd059807719c'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227118705, "block_size": 4096, "block_total": 240095108, "block_used": 12976403, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60985790, "inode_total": 61054976, "inode_used": 69186, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930278215680, "size_total": 983429562368, "uuid": "a9bc468f-7d76-441a-b17b-cd059807719c"}, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:13.226 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:13 +0000 (0:00:00.031) 0:04:00.574 *********** 2021-05-25T11:56:13.231 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-05-25T11:56:13.233 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:13.254 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:13 +0000 (0:00:00.028) 0:04:00.603 *********** 2021-05-25T11:56:13.260 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2021-05-25T11:56:13.262 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:13.284 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:13 +0000 (0:00:00.029) 0:04:00.632 *********** 2021-05-25T11:56:13.290 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-05-25T11:56:13.292 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:13.314 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:13 +0000 (0:00:00.029) 0:04:00.662 *********** 2021-05-25T11:56:13.319 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-05-25T11:56:13.321 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:13.346 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:13 +0000 (0:00:00.032) 0:04:00.694 *********** 2021-05-25T11:56:13.351 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2021-05-25T11:56:13.353 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:13.375 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:13 +0000 (0:00:00.028) 0:04:00.723 *********** 2021-05-25T11:56:13.380 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-05-25T11:56:13.382 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:13.560 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:13 +0000 (0:00:00.184) 0:04:00.907 *********** 2021-05-25T11:56:13.567 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2021-05-25T11:56:13.568 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:13.590 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:13 +0000 (0:00:00.030) 0:04:00.938 *********** 2021-05-25T11:56:13.596 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2021-05-25T11:56:13.597 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:13.619 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:13 +0000 (0:00:00.028) 0:04:00.967 *********** 2021-05-25T11:56:13.625 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2021-05-25T11:56:13.626 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:13.648 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:13 +0000 (0:00:00.028) 0:04:00.996 *********** 2021-05-25T11:56:13.654 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2021-05-25T11:56:13.655 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:13.678 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:13 +0000 (0:00:00.030) 0:04:01.026 *********** 2021-05-25T11:56:13.705 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2021-05-25T11:56:13.707 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:13.727 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:13 +0000 (0:00:00.049) 0:04:01.075 *********** 2021-05-25T11:56:14.223 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2021-05-25T11:56:14.225 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2021-05-25T11:56:14.247 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:14 +0000 (0:00:00.520) 0:04:01.595 *********** 2021-05-25T11:56:14.255 INFO:teuthology.task.ansible.out:changed: [smithi181.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2021-05-25T11:56:14.280 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:14 +0000 (0:00:00.032) 0:04:01.628 *********** 2021-05-25T11:56:14.734 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-05-25T11:56:14.735 INFO:teuthology.task.ansible.out:changed: [smithi181.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-05-25T11:56:14.756 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-05-25T11:56:15.056 INFO:teuthology.task.ansible.out:changed: [smithi181.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-05-25T11:56:15.078 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-05-25T11:56:15.400 INFO:teuthology.task.ansible.out:changed: [smithi181.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-05-25T11:56:15.401 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-05-25T11:56:15.711 INFO:teuthology.task.ansible.out:changed: [smithi181.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-05-25T11:56:15.744 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-05-25T11:56:16.024 INFO:teuthology.task.ansible.out:changed: [smithi181.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2021-05-25T11:56:16.048 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:16 +0000 (0:00:01.768) 0:04:03.396 *********** 2021-05-25T11:56:16.096 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2021-05-25T11:56:16.123 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:16 +0000 (0:00:00.075) 0:04:03.471 *********** 2021-05-25T11:56:16.245 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-05-25T11:56:16.246 INFO:teuthology.task.ansible.out:changed: [smithi181.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2021-05-25T11:56:16.269 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:16 +0000 (0:00:00.146) 0:04:03.617 *********** 2021-05-25T11:56:16.319 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2021-05-25T11:56:16.351 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:16 +0000 (0:00:00.081) 0:04:03.699 *********** 2021-05-25T11:56:16.479 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-05-25T11:56:16.479 INFO:teuthology.task.ansible.out:changed: [smithi181.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-05-25T11:56:16.564 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-05-25T11:56:16.648 INFO:teuthology.task.ansible.out:changed: [smithi181.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-05-25T11:56:16.743 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-05-25T11:56:16.817 INFO:teuthology.task.ansible.out:changed: [smithi181.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-05-25T11:56:16.913 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-05-25T11:56:16.988 INFO:teuthology.task.ansible.out:changed: [smithi181.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-05-25T11:56:17.010 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:17.035 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:17 +0000 (0:00:00.684) 0:04:04.383 *********** 2021-05-25T11:56:17.063 INFO:teuthology.task.ansible.out: TASK [testnode : Create filesystems] ******************************************* skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:17.085 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:17 +0000 (0:00:00.049) 0:04:04.433 *********** 2021-05-25T11:56:17.101 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-05-25T11:56:17.103 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-05-25T11:56:17.113 INFO:teuthology.task.ansible.out: TASK [testnode : Mount filesystems] ******************************************** skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:17.115 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** skipping: [smithi102.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:17.139 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:17 +0000 (0:00:00.053) 0:04:04.487 *********** 2021-05-25T11:56:17.159 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:17 +0000 (0:00:00.019) 0:04:04.506 *********** 2021-05-25T11:56:17.164 INFO:teuthology.task.ansible.out: TASK [testnode : Create filesystems] ******************************************* 2021-05-25T11:56:17.166 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:17.190 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:17 +0000 (0:00:00.030) 0:04:04.537 *********** 2021-05-25T11:56:17.216 INFO:teuthology.task.ansible.out: TASK [testnode : Mount filesystems] ******************************************** skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:17.239 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:17 +0000 (0:00:00.049) 0:04:04.587 *********** 2021-05-25T11:56:17.350 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2021-05-25T11:56:17.352 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-05-25T11:56:17.375 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:17 +0000 (0:00:00.135) 0:04:04.723 *********** 2021-05-25T11:56:17.412 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2021-05-25T11:56:17.414 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2021-05-25T11:56:17.437 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:17 +0000 (0:00:00.062) 0:04:04.785 *********** 2021-05-25T11:56:17.444 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2021-05-25T11:56:17.446 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-05-25T11:56:17.469 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:17 +0000 (0:00:00.031) 0:04:04.817 *********** 2021-05-25T11:56:17.507 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2021-05-25T11:56:17.508 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2021-05-25T11:56:17.529 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:17 +0000 (0:00:00.059) 0:04:04.877 *********** 2021-05-25T11:56:17.871 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2021-05-25T11:56:17.872 INFO:teuthology.task.ansible.out:changed: [smithi181.front.sepia.ceph.com] => {"changed": true} changed: [smithi102.front.sepia.ceph.com] => {"changed": true} 2021-05-25T11:56:17.895 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:17 +0000 (0:00:00.366) 0:04:05.243 *********** 2021-05-25T11:56:17.915 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:17 +0000 (0:00:00.019) 0:04:05.263 *********** 2021-05-25T11:56:18.327 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2021-05-25T11:56:18.329 INFO:teuthology.task.ansible.out:changed: [smithi181.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2021-05-25T11:56:18.349 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:18 +0000 (0:00:00.434) 0:04:05.698 *********** 2021-05-25T11:56:18.375 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd] ****************************************** 2021-05-25T11:56:18.377 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:18.399 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:18 +0000 (0:00:00.049) 0:04:05.747 *********** 2021-05-25T11:56:18.427 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd conf file] ******************************** 2021-05-25T11:56:18.429 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:18.449 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:18 +0000 (0:00:00.050) 0:04:05.797 *********** 2021-05-25T11:56:18.476 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-05-25T11:56:18.478 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:18.499 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:18 +0000 (0:00:00.049) 0:04:05.847 *********** 2021-05-25T11:56:18.525 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-05-25T11:56:18.527 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:18.548 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:18 +0000 (0:00:00.049) 0:04:05.896 *********** 2021-05-25T11:56:18.575 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2021-05-25T11:56:18.576 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:18.606 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:18 +0000 (0:00:00.057) 0:04:05.954 *********** 2021-05-25T11:56:18.635 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2021-05-25T11:56:18.637 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:18.658 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:18 +0000 (0:00:00.052) 0:04:06.006 *********** 2021-05-25T11:56:18.687 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2021-05-25T11:56:18.689 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:18.713 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:18 +0000 (0:00:00.054) 0:04:06.061 *********** 2021-05-25T11:56:18.741 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2021-05-25T11:56:18.742 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:18.765 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:18 +0000 (0:00:00.052) 0:04:06.113 *********** 2021-05-25T11:56:18.792 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2021-05-25T11:56:18.793 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:18.815 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:18 +0000 (0:00:00.050) 0:04:06.164 *********** 2021-05-25T11:56:18.842 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:18.869 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:18 +0000 (0:00:00.053) 0:04:06.217 *********** 2021-05-25T11:56:18.895 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2021-05-25T11:56:18.897 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:18.919 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:18 +0000 (0:00:00.049) 0:04:06.267 *********** 2021-05-25T11:56:18.947 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2021-05-25T11:56:18.949 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:18.971 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:18 +0000 (0:00:00.052) 0:04:06.319 *********** 2021-05-25T11:56:18.997 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:19.019 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:19 +0000 (0:00:00.048) 0:04:06.367 *********** 2021-05-25T11:56:19.046 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2021-05-25T11:56:19.048 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:19.068 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:19 +0000 (0:00:00.048) 0:04:06.416 *********** 2021-05-25T11:56:19.095 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2021-05-25T11:56:19.097 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:19.118 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:19 +0000 (0:00:00.049) 0:04:06.466 *********** 2021-05-25T11:56:19.321 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2021-05-25T11:56:19.323 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2021-05-25T11:56:19.350 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:19 +0000 (0:00:00.232) 0:04:06.698 *********** 2021-05-25T11:56:19.378 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd] ****************************************** 2021-05-25T11:56:19.380 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:19.405 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:19 +0000 (0:00:00.054) 0:04:06.753 *********** 2021-05-25T11:56:19.442 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd conf file] ******************************** 2021-05-25T11:56:19.444 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:19.465 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:19 +0000 (0:00:00.060) 0:04:06.813 *********** 2021-05-25T11:56:19.493 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-05-25T11:56:19.495 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:19.516 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:19 +0000 (0:00:00.050) 0:04:06.864 *********** 2021-05-25T11:56:19.543 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-05-25T11:56:19.545 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:19.570 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:19 +0000 (0:00:00.050) 0:04:06.915 *********** 2021-05-25T11:56:19.576 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2021-05-25T11:56:19.578 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => {"changed": false, "checksum": "a7ecef00073feab34f868065d34e9dac55134241", "dest": "/etc/chrony.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/chrony.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 215, "state": "file", "uid": 0} 2021-05-25T11:56:19.600 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:19 +0000 (0:00:00.032) 0:04:06.948 *********** 2021-05-25T11:56:19.606 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:19.631 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:19 +0000 (0:00:00.031) 0:04:06.979 *********** 2021-05-25T11:56:19.659 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2021-05-25T11:56:19.661 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:19.681 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:19 +0000 (0:00:00.050) 0:04:07.029 *********** 2021-05-25T11:56:19.719 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2021-05-25T11:56:19.721 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:19.750 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:19 +0000 (0:00:00.068) 0:04:07.098 *********** 2021-05-25T11:56:19.777 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2021-05-25T11:56:19.778 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:19.800 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:19 +0000 (0:00:00.049) 0:04:07.148 *********** 2021-05-25T11:56:19.826 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2021-05-25T11:56:19.828 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:19.858 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:19 +0000 (0:00:00.058) 0:04:07.206 *********** 2021-05-25T11:56:19.887 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2021-05-25T11:56:19.889 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:19.910 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:19 +0000 (0:00:00.051) 0:04:07.258 *********** 2021-05-25T11:56:19.920 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2021-05-25T11:56:19.922 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "chronyd", "state": "started", "status": {"ActiveEnterTimestamp": "Tue 2021-05-25 11:51:53 UTC", "ActiveEnterTimestampMonotonic": "13067244", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "ntpd.service sysinit.target -.mount ntpdate.service system.slice systemd-journald.socket systemd-tmpfiles-setup.service sntp.service tmp.mount basic.target", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Tue 2021-05-25 11:51:52 UTC", "AssertTimestampMonotonic": "11814641", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Tue 2021-05-25 11:51:52 UTC", "ConditionTimestampMonotonic": "11814609", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target ntpd.service systemd-timesyncd.service", "ControlGro 2021-05-25T11:56:19.924 INFO:teuthology.task.ansible.out:up": "/system.slice/chronyd.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "NTP client/server", "DevicePolicy": "auto", "Documentation": "man:chronyd(8) man:chrony.conf(5)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/chronyd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "878", "ExecMainStartTimestamp": "Tue 2021-05-25 11:51:53 UTC", "ExecMainStartTimestampMonotonic": "13003319", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/chronyd ; argv[]=/usr/sbin/chronyd $OPTIONS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPost": "{ path=/usr/libexec/chrony-helper ; argv[]=/usr/libexec/chrony-helper update-daemon ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/chronyd.service", "FreezerState": "running", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "chronyd.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Tue 2021-05-25 11:51:52 UTC", "InactiveExitTimestampMonotonic": "11829237", "InvocationID": "dded828c5a5e4497b62aa184a14651bc", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity" 2021-05-25T11:56:19.926 INFO:teuthology.task.ansible.out:, "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127252", "LimitNPROCSoft": "127252", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127252", "LimitSIGPENDINGSoft": "127252", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "878", "MemoryAccounting": "yes", "MemoryCurrent": "2879488", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "chronyd.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PIDFile": "/run/chrony/chronyd.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "yes", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "full", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice -.mount sysinit.target", "RequiresMountsFor": "/var/tmp", "Restart": "no 2021-05-25T11:56:19.928 INFO:teuthology.task.ansible.out:", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Tue 2021-05-25 11:51:53 UTC", "StateChangeTimestampMonotonic": "13067244", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203604", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Tue 2021-05-25 11:51:53 UTC", "WatchdogTimestampMonotonic": "13003335", "WatchdogUSec": "0"}} 2021-05-25T11:56:19.947 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:19 +0000 (0:00:00.037) 0:04:07.296 *********** 2021-05-25T11:56:19.953 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2021-05-25T11:56:19.955 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:19.977 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:19 +0000 (0:00:00.029) 0:04:07.325 *********** 2021-05-25T11:56:19.983 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-05-25T11:56:19.985 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2021-05-25T11:56:20.006 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:20 +0000 (0:00:00.029) 0:04:07.354 *********** 2021-05-25T11:56:20.012 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2021-05-25T11:56:20.014 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:20.037 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:20 +0000 (0:00:00.031) 0:04:07.385 *********** 2021-05-25T11:56:20.044 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-05-25T11:56:20.046 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => {"ansible_facts": {"ansible_python_interpreter": "/usr/bin/python3", "pip_executable": "pip3", "pip_version": "python3-pip"}, "changed": false} 2021-05-25T11:56:20.068 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:20 +0000 (0:00:00.031) 0:04:07.416 *********** 2021-05-25T11:56:20.076 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2021-05-25T11:56:20.078 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:20.101 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:20 +0000 (0:00:00.032) 0:04:07.449 *********** 2021-05-25T11:56:20.108 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2021-05-25T11:56:20.110 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:20.133 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:20 +0000 (0:00:00.032) 0:04:07.481 *********** 2021-05-25T11:56:20.140 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2021-05-25T11:56:20.142 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:20.165 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:20 +0000 (0:00:00.031) 0:04:07.513 *********** 2021-05-25T11:56:20.172 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2021-05-25T11:56:20.174 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:20.197 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:20 +0000 (0:00:00.032) 0:04:07.545 *********** 2021-05-25T11:56:20.204 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2021-05-25T11:56:20.206 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:20.228 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:20 +0000 (0:00:00.031) 0:04:07.576 *********** 2021-05-25T11:56:20.234 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2021-05-25T11:56:20.236 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:20.259 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:20 +0000 (0:00:00.031) 0:04:07.607 *********** 2021-05-25T11:56:20.515 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** ok: [smithi181.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2021-05-25T11:56:20.531 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:20 +0000 (0:00:00.272) 0:04:07.879 *********** 2021-05-25T11:56:20.715 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2021-05-25T11:56:20.717 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "checksum": "a7ecef00073feab34f868065d34e9dac55134241", "dest": "/etc/chrony.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/chrony.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 215, "state": "file", "uid": 0} 2021-05-25T11:56:20.739 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:20 +0000 (0:00:00.207) 0:04:08.087 *********** 2021-05-25T11:56:21.035 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2021-05-25T11:56:21.037 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2021-05-25T11:56:21.057 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:21 +0000 (0:00:00.318) 0:04:08.405 *********** 2021-05-25T11:56:21.065 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2021-05-25T11:56:21.067 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "chronyd", "state": "started", "status": {"ActiveEnterTimestamp": "Tue 2021-05-25 11:51:52 UTC", "ActiveEnterTimestampMonotonic": "12954175", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "ntpdate.service sntp.service sysinit.target system.slice tmp.mount basic.target ntpd.service systemd-journald.socket systemd-tmpfiles-setup.service -.mount", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Tue 2021-05-25 11:51:51 UTC", "AssertTimestampMonotonic": "11785022", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Tue 2021-05-25 11:51:51 UTC", "ConditionTimestampMonotonic": "11784990", "ConfigurationDirectoryMode": "0755", "Conflicts": "systemd-timesyncd.service ntpd.service shutdown.target", "ControlGro 2021-05-25T11:56:21.068 INFO:teuthology.task.ansible.out:up": "/system.slice/chronyd.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "NTP client/server", "DevicePolicy": "auto", "Documentation": "man:chronyd(8) man:chrony.conf(5)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/chronyd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "929", "ExecMainStartTimestamp": "Tue 2021-05-25 11:51:52 UTC", "ExecMainStartTimestampMonotonic": "12911227", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/chronyd ; argv[]=/usr/sbin/chronyd $OPTIONS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPost": "{ path=/usr/libexec/chrony-helper ; argv[]=/usr/libexec/chrony-helper update-daemon ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/chronyd.service", "FreezerState": "running", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "chronyd.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Tue 2021-05-25 11:51:51 UTC", "InactiveExitTimestampMonotonic": "12138190", "InvocationID": "8a4d539309ef4c4698f31b27e45c85ad", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity" 2021-05-25T11:56:21.070 INFO:teuthology.task.ansible.out:, "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127253", "LimitNPROCSoft": "127253", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127253", "LimitSIGPENDINGSoft": "127253", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "929", "MemoryAccounting": "yes", "MemoryCurrent": "3182592", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "chronyd.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PIDFile": "/run/chrony/chronyd.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "yes", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "full", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target -.mount system.slice", "RequiresMountsFor": "/var/tmp", "Restart": "no 2021-05-25T11:56:21.072 INFO:teuthology.task.ansible.out:", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Tue 2021-05-25 11:51:52 UTC", "StateChangeTimestampMonotonic": "12954175", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203604", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Tue 2021-05-25 11:51:52 UTC", "WatchdogTimestampMonotonic": "12911247", "WatchdogUSec": "0"}} 2021-05-25T11:56:21.092 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:21 +0000 (0:00:00.034) 0:04:08.440 *********** 2021-05-25T11:56:21.130 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** ok: [smithi102.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2021-05-25T11:56:21.152 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:21 +0000 (0:00:00.060) 0:04:08.500 *********** 2021-05-25T11:56:21.190 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-05-25T11:56:21.192 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"ansible_facts": {"ansible_python_interpreter": "/usr/bin/python3", "pip_executable": "pip3", "pip_version": "python3-pip"}, "changed": false} 2021-05-25T11:56:21.214 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:21 +0000 (0:00:00.061) 0:04:08.562 *********** 2021-05-25T11:56:21.242 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2021-05-25T11:56:21.244 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:21.265 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:21 +0000 (0:00:00.050) 0:04:08.613 *********** 2021-05-25T11:56:21.292 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2021-05-25T11:56:21.293 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:21.314 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:21 +0000 (0:00:00.049) 0:04:08.662 *********** 2021-05-25T11:56:21.352 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2021-05-25T11:56:21.353 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:21.374 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:21 +0000 (0:00:00.059) 0:04:08.722 *********** 2021-05-25T11:56:21.568 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2021-05-25T11:56:21.569 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2021-05-25T11:56:21.595 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:21 +0000 (0:00:00.220) 0:04:08.943 *********** 2021-05-25T11:56:22.071 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2021-05-25T11:56:22.074 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2021-05-25T11:56:22.099 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:22 +0000 (0:00:00.504) 0:04:09.447 *********** 2021-05-25T11:56:22.276 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2021-05-25T11:56:22.277 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages\nRequirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages", "Requirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2021-05-25T11:56:22.300 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:22 +0000 (0:00:00.200) 0:04:09.648 *********** 2021-05-25T11:56:22.327 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2021-05-25T11:56:22.329 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:22.349 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:22 +0000 (0:00:00.049) 0:04:09.697 *********** 2021-05-25T11:56:22.374 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2021-05-25T11:56:22.376 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2021-05-25T11:56:22.396 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:22 +0000 (0:00:00.047) 0:04:09.744 *********** 2021-05-25T11:56:22.578 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2021-05-25T11:56:22.579 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2021-05-25T11:56:22.608 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:22 +0000 (0:00:00.211) 0:04:09.956 *********** 2021-05-25T11:56:22.639 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2021-05-25T11:56:22.641 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:22.675 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:22 +0000 (0:00:00.067) 0:04:10.023 *********** 2021-05-25T11:56:22.715 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2021-05-25T11:56:22.716 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2021-05-25T11:56:22.737 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:22 +0000 (0:00:00.061) 0:04:10.085 *********** 2021-05-25T11:56:22.765 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2021-05-25T11:56:22.766 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:22.788 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:22 +0000 (0:00:00.050) 0:04:10.135 *********** 2021-05-25T11:56:22.980 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2021-05-25T11:56:22.982 INFO:teuthology.task.ansible.out:changed: [smithi181.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2021-05-25T11:56:23.004 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:23 +0000 (0:00:00.216) 0:04:10.352 *********** 2021-05-25T11:56:23.177 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2021-05-25T11:56:23.179 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages\nRequirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages", "Requirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2021-05-25T11:56:23.200 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:23 +0000 (0:00:00.195) 0:04:10.548 *********** 2021-05-25T11:56:23.206 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2021-05-25T11:56:23.208 INFO:teuthology.task.ansible.out:changed: [smithi181.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2021-05-25T11:56:23.230 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:23 +0000 (0:00:00.029) 0:04:10.578 *********** 2021-05-25T11:56:23.235 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2021-05-25T11:56:23.237 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:23.259 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:23 +0000 (0:00:00.029) 0:04:10.607 *********** 2021-05-25T11:56:23.266 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2021-05-25T11:56:23.267 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:23.289 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:23 +0000 (0:00:00.029) 0:04:10.637 *********** 2021-05-25T11:56:23.295 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2021-05-25T11:56:23.296 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2021-05-25T11:56:23.320 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:23 +0000 (0:00:00.031) 0:04:10.668 *********** 2021-05-25T11:56:23.328 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2021-05-25T11:56:23.330 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} skipping: [smithi181.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2021-05-25T11:56:23.340 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2021-05-25T11:56:23.342 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} skipping: [smithi181.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2021-05-25T11:56:23.365 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:23 +0000 (0:00:00.044) 0:04:10.713 *********** 2021-05-25T11:56:23.402 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2021-05-25T11:56:23.404 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi181.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:23.415 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:23.416 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:23.427 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:23.451 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:23 +0000 (0:00:00.086) 0:04:10.799 *********** 2021-05-25T11:56:23.479 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-05-25T11:56:23.481 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:23.513 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:23 +0000 (0:00:00.061) 0:04:10.861 *********** 2021-05-25T11:56:23.521 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2021-05-25T11:56:23.521 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2021-05-25T11:56:23.547 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:23 +0000 (0:00:00.034) 0:04:10.895 *********** 2021-05-25T11:56:23.553 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2021-05-25T11:56:23.554 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:23.577 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:23 +0000 (0:00:00.030) 0:04:10.925 *********** 2021-05-25T11:56:23.583 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2021-05-25T11:56:23.585 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:23.609 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:23 +0000 (0:00:00.031) 0:04:10.957 *********** 2021-05-25T11:56:23.615 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2021-05-25T11:56:23.617 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:23.642 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:23 +0000 (0:00:00.033) 0:04:10.990 *********** 2021-05-25T11:56:23.649 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2021-05-25T11:56:23.650 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2021-05-25T11:56:23.672 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:23 +0000 (0:00:00.030) 0:04:11.020 *********** 2021-05-25T11:56:23.678 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2021-05-25T11:56:23.680 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:23.702 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:23 +0000 (0:00:00.029) 0:04:11.050 *********** 2021-05-25T11:56:23.708 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2021-05-25T11:56:23.709 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:23.731 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:23 +0000 (0:00:00.028) 0:04:11.079 *********** 2021-05-25T11:56:23.737 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2021-05-25T11:56:23.738 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:23.771 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:23 +0000 (0:00:00.039) 0:04:11.119 *********** 2021-05-25T11:56:23.799 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-05-25T11:56:23.801 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:23.824 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:23 +0000 (0:00:00.053) 0:04:11.172 *********** 2021-05-25T11:56:23.861 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2021-05-25T11:56:23.863 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:23.887 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:23 +0000 (0:00:00.062) 0:04:11.234 *********** 2021-05-25T11:56:23.914 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-05-25T11:56:23.915 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:23.940 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:23 +0000 (0:00:00.053) 0:04:11.288 *********** 2021-05-25T11:56:23.946 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2021-05-25T11:56:23.946 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2021-05-25T11:56:23.972 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:23 +0000 (0:00:00.032) 0:04:11.320 *********** 2021-05-25T11:56:24.157 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-05-25T11:56:24.158 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.005094", "end": "2021-05-25 11:56:24.119724", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2021-05-25 11:56:24.114630", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* changed: [smithi102.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2021-05-25T11:56:24.184 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:24 +0000 (0:00:00.211) 0:04:11.532 *********** 2021-05-25T11:56:24.210 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2021-05-25T11:56:24.212 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:24.233 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:24 +0000 (0:00:00.049) 0:04:11.581 *********** 2021-05-25T11:56:24.261 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2021-05-25T11:56:24.263 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2021-05-25T11:56:24.272 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2021-05-25T11:56:24.273 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2021-05-25T11:56:24.273 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} 2021-05-25T11:56:24.284 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} 2021-05-25T11:56:24.308 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:24 +0000 (0:00:00.074) 0:04:11.656 *********** 2021-05-25T11:56:24.324 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-05-25T11:56:24.326 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.003128", "end": "2021-05-25 11:56:24.291209", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2021-05-25 11:56:24.288081", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-05-25T11:56:24.346 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2021-05-25T11:56:24.348 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi102.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:24.358 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:24.360 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi102.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:24.384 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:24 +0000 (0:00:00.076) 0:04:11.732 *********** 2021-05-25T11:56:24.411 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-05-25T11:56:24.414 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:24.436 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:24 +0000 (0:00:00.051) 0:04:11.784 *********** 2021-05-25T11:56:24.463 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2021-05-25T11:56:24.465 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:24.486 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:24 +0000 (0:00:00.049) 0:04:11.834 *********** 2021-05-25T11:56:24.493 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-05-25T11:56:24.494 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.002809", "end": "2021-05-25 11:56:24.457098", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2021-05-25 11:56:24.454289", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2021-05-25T11:56:24.516 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2021-05-25T11:56:24.517 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:24.540 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:24 +0000 (0:00:00.054) 0:04:11.888 *********** 2021-05-25T11:56:24.567 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2021-05-25T11:56:24.568 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:24.589 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:24 +0000 (0:00:00.049) 0:04:11.937 *********** 2021-05-25T11:56:24.616 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2021-05-25T11:56:24.618 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:24.639 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:24 +0000 (0:00:00.049) 0:04:11.987 *********** 2021-05-25T11:56:24.655 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-05-25T11:56:24.657 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.003224", "end": "2021-05-25 11:56:24.625782", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2021-05-25 11:56:24.622558", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-05-25T11:56:24.680 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:24 +0000 (0:00:00.040) 0:04:12.028 *********** 2021-05-25T11:56:24.687 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-05-25T11:56:24.688 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:24.712 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:24 +0000 (0:00:00.032) 0:04:12.060 *********** 2021-05-25T11:56:24.718 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2021-05-25T11:56:24.746 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:24 +0000 (0:00:00.033) 0:04:12.094 *********** 2021-05-25T11:56:24.753 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2021-05-25T11:56:24.754 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:24.779 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:24 +0000 (0:00:00.032) 0:04:12.127 *********** 2021-05-25T11:56:24.785 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2021-05-25T11:56:24.787 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:24.810 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:24.830 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:24 +0000 (0:00:00.051) 0:04:12.178 *********** 2021-05-25T11:56:25.027 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-05-25T11:56:25.029 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.004884", "end": "2021-05-25 11:56:24.998603", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2021-05-25 11:56:24.993719", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-05-25T11:56:25.203 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.003022", "end": "2021-05-25 11:56:25.170177", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2021-05-25 11:56:25.167155", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-05-25T11:56:25.370 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.002808", "end": "2021-05-25 11:56:25.335989", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2021-05-25 11:56:25.333181", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2021-05-25T11:56:25.547 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.003194", "end": "2021-05-25 11:56:25.516402", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2021-05-25 11:56:25.513208", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-05-25T11:56:25.571 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:25 +0000 (0:00:00.740) 0:04:12.919 *********** 2021-05-25T11:56:25.598 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2021-05-25T11:56:25.623 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:25 +0000 (0:00:00.051) 0:04:12.971 *********** 2021-05-25T11:56:25.650 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2021-05-25T11:56:25.652 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:25.717 INFO:teuthology.task.ansible.out: PLAY [testnodes,senta,vossi,jenkins_slaves] ************************************ 2021-05-25T11:56:25.734 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:25 +0000 (0:00:00.111) 0:04:13.082 *********** 2021-05-25T11:56:25.768 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:25 +0000 (0:00:00.033) 0:04:13.115 *********** 2021-05-25T11:56:26.357 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi102.front.sepia.ceph.com] 2021-05-25T11:56:26.406 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:26 +0000 (0:00:00.638) 0:04:13.754 *********** 2021-05-25T11:56:26.434 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-05-25T11:56:26.436 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:26.456 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:26 +0000 (0:00:00.050) 0:04:13.804 *********** 2021-05-25T11:56:26.492 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-05-25T11:56:26.494 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "yum"}, "changed": false} 2021-05-25T11:56:26.515 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:26 +0000 (0:00:00.059) 0:04:13.863 *********** 2021-05-25T11:56:26.552 INFO:teuthology.task.ansible.out: TASK [container-host : Including distro specific variables] ******************** 2021-05-25T11:56:26.554 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml) => {"ansible_facts": {"container_packages": ["podman", "podman-docker"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"} 2021-05-25T11:56:26.575 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:26 +0000 (0:00:00.059) 0:04:13.923 *********** 2021-05-25T11:56:27.410 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi181.front.sepia.ceph.com] 2021-05-25T11:56:27.459 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:27 +0000 (0:00:00.884) 0:04:14.807 *********** 2021-05-25T11:56:27.486 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:27.509 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:27 +0000 (0:00:00.049) 0:04:14.857 *********** 2021-05-25T11:56:27.556 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-05-25T11:56:27.558 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "yum"}, "changed": false} 2021-05-25T11:56:27.578 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:27 +0000 (0:00:00.069) 0:04:14.926 *********** 2021-05-25T11:56:27.614 INFO:teuthology.task.ansible.out: TASK [container-host : Including distro specific variables] ******************** 2021-05-25T11:56:27.616 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml) => {"ansible_facts": {"container_packages": ["podman", "podman-docker"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"} 2021-05-25T11:56:27.641 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:27 +0000 (0:00:00.062) 0:04:14.989 *********** 2021-05-25T11:56:27.925 INFO:teuthology.task.ansible.out: TASK [container-host : Install container packages] ***************************** 2021-05-25T11:56:27.927 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-05-25T11:56:27.949 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:27 +0000 (0:00:00.308) 0:04:15.297 *********** 2021-05-25T11:56:27.976 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-05-25T11:56:27.978 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/containers/registries.conf"}, "changed": false} 2021-05-25T11:56:27.998 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:27 +0000 (0:00:00.049) 0:04:15.346 *********** 2021-05-25T11:56:28.025 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-05-25T11:56:28.027 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:28.048 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:28 +0000 (0:00:00.049) 0:04:15.396 *********** 2021-05-25T11:56:28.260 INFO:teuthology.task.ansible.out: TASK [container-host : Create container_mirror_cert_paths] ********************* 2021-05-25T11:56:28.262 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-05-25T11:56:28.428 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-05-25T11:56:28.454 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:28 +0000 (0:00:00.405) 0:04:15.802 *********** 2021-05-25T11:56:28.945 INFO:teuthology.task.ansible.out: TASK [container-host : Copy docker-mirror.front.sepia.ceph.com:5000 self-signed cert] *** 2021-05-25T11:56:28.950 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:cert_t:s0", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1621943788.5182977-9950-44423741490867/source", "state": "file", "uid": 0} 2021-05-25T11:56:29.009 INFO:teuthology.task.ansible.out: TASK [container-host : Install container packages] ***************************** 2021-05-25T11:56:29.010 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-05-25T11:56:29.032 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:29 +0000 (0:00:00.578) 0:04:16.380 *********** 2021-05-25T11:56:29.058 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-05-25T11:56:29.060 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/containers/registries.conf"}, "changed": false} 2021-05-25T11:56:29.080 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:29 +0000 (0:00:00.048) 0:04:16.428 *********** 2021-05-25T11:56:29.106 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-05-25T11:56:29.108 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:29.129 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:29 +0000 (0:00:00.048) 0:04:16.477 *********** 2021-05-25T11:56:29.343 INFO:teuthology.task.ansible.out: TASK [container-host : Create container_mirror_cert_paths] ********************* 2021-05-25T11:56:29.345 INFO:teuthology.task.ansible.out:changed: [smithi181.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-05-25T11:56:29.417 INFO:teuthology.task.ansible.out: TASK [container-host : Copy docker-mirror.front.sepia.ceph.com:5000 self-signed cert] *** 2021-05-25T11:56:29.419 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1621943788.980751-9950-173196295999753/source", "state": "file", "uid": 0} 2021-05-25T11:56:29.442 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:29 +0000 (0:00:00.313) 0:04:16.790 *********** 2021-05-25T11:56:29.522 INFO:teuthology.task.ansible.out: TASK [container-host : Create container_mirror_cert_paths] ********************* 2021-05-25T11:56:29.524 INFO:teuthology.task.ansible.out:changed: [smithi181.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-05-25T11:56:29.548 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:29 +0000 (0:00:00.105) 0:04:16.896 *********** 2021-05-25T11:56:30.055 INFO:teuthology.task.ansible.out: TASK [container-host : Copy docker-mirror.front.sepia.ceph.com:5000 self-signed cert] *** 2021-05-25T11:56:30.057 INFO:teuthology.task.ansible.out:changed: [smithi181.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:cert_t:s0", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1621943789.6178207-10021-14747347191263/source", "state": "file", "uid": 0} 2021-05-25T11:56:30.515 INFO:teuthology.task.ansible.out:changed: [smithi181.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1621943790.0836685-10021-106793024928193/source", "state": "file", "uid": 0} 2021-05-25T11:56:30.538 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:30 +0000 (0:00:00.989) 0:04:17.886 *********** 2021-05-25T11:56:32.600 INFO:teuthology.task.ansible.out: TASK [container-host : Install registries-conf-ctl] **************************** 2021-05-25T11:56:32.602 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => {"changed": true, "cmd": ["/bin/pip3", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-xqqd1yjw-build\nCollecting toml (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl\nCollecting docopt (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz\nInstalling collected packages: toml, docopt, registries-conf-ctl\n Running setup.py install for docopt: started\n Running setup.py install for docopt: finished with status 'done'\n Running setup.py install for registries-conf-ctl: started\n Running setup.py install for registries-conf-ctl: finished with status 'done'\nSuccessfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2\n", "stdout_lines": ["Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-xqqd1yjw-build", "Collecting toml (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl", "Collecting docopt (from registries-conf-ctl==1.0)", " Downloading https://files. 2021-05-25T11:56:32.604 INFO:teuthology.task.ansible.out:pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz", "Installing collected packages: toml, docopt, registries-conf-ctl", " Running setup.py install for docopt: started", " Running setup.py install for docopt: finished with status 'done'", " Running setup.py install for registries-conf-ctl: started", " Running setup.py install for registries-conf-ctl: finished with status 'done'", "Successfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2"], "version": null, "virtualenv": null} 2021-05-25T11:56:32.625 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:32 +0000 (0:00:02.086) 0:04:19.973 *********** 2021-05-25T11:56:32.665 INFO:teuthology.task.ansible.out: TASK [container-host : Check for docker's daemon.json] ************************* 2021-05-25T11:56:32.667 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:32.689 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:32 +0000 (0:00:00.064) 0:04:20.037 *********** 2021-05-25T11:56:32.727 INFO:teuthology.task.ansible.out: TASK [container-host : Create /etc/containers/registries.conf if necessary] **** skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:32.750 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:32 +0000 (0:00:00.061) 0:04:20.098 *********** 2021-05-25T11:56:33.099 INFO:teuthology.task.ansible.out: TASK [container-host : Add local docker.io registry mirror] ******************** 2021-05-25T11:56:33.101 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.155599", "end": "2021-05-25 11:56:33.071771", "rc": 0, "start": "2021-05-25 11:56:32.916172", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-05-25T11:56:33.122 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:33 +0000 (0:00:00.371) 0:04:20.470 *********** 2021-05-25T11:56:33.150 INFO:teuthology.task.ansible.out: TASK [container-host : Restart docker service] ********************************* 2021-05-25T11:56:33.152 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:33.679 INFO:teuthology.task.ansible.out: TASK [container-host : Install registries-conf-ctl] **************************** 2021-05-25T11:56:33.681 INFO:teuthology.task.ansible.out:changed: [smithi181.front.sepia.ceph.com] => {"changed": true, "cmd": ["/bin/pip3", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-szvvpa4l-build\nCollecting toml (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl\nCollecting docopt (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz\nInstalling collected packages: toml, docopt, registries-conf-ctl\n Running setup.py install for docopt: started\n Running setup.py install for docopt: finished with status 'done'\n Running setup.py install for registries-conf-ctl: started\n Running setup.py install for registries-conf-ctl: finished with status 'done'\nSuccessfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2\n", "stdout_lines": ["Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-szvvpa4l-build", "Collecting toml (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl", "Collecting docopt (from registries-conf-ctl==1.0)", " Downloading https://files. 2021-05-25T11:56:33.683 INFO:teuthology.task.ansible.out:pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz", "Installing collected packages: toml, docopt, registries-conf-ctl", " Running setup.py install for docopt: started", " Running setup.py install for docopt: finished with status 'done'", " Running setup.py install for registries-conf-ctl: started", " Running setup.py install for registries-conf-ctl: finished with status 'done'", "Successfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2"], "version": null, "virtualenv": null} 2021-05-25T11:56:33.701 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:33 +0000 (0:00:00.579) 0:04:21.049 *********** 2021-05-25T11:56:33.728 INFO:teuthology.task.ansible.out: TASK [container-host : Check for docker's daemon.json] ************************* 2021-05-25T11:56:33.730 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:33.749 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:33 +0000 (0:00:00.048) 0:04:21.097 *********** 2021-05-25T11:56:33.776 INFO:teuthology.task.ansible.out: TASK [container-host : Create /etc/containers/registries.conf if necessary] **** 2021-05-25T11:56:33.778 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:33.799 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:33 +0000 (0:00:00.049) 0:04:21.147 *********** 2021-05-25T11:56:34.148 INFO:teuthology.task.ansible.out: TASK [container-host : Add local docker.io registry mirror] ******************** 2021-05-25T11:56:34.150 INFO:teuthology.task.ansible.out:changed: [smithi181.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.161129", "end": "2021-05-25 11:56:34.125899", "rc": 0, "start": "2021-05-25 11:56:33.964770", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-05-25T11:56:34.170 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:34 +0000 (0:00:00.370) 0:04:21.518 *********** 2021-05-25T11:56:34.196 INFO:teuthology.task.ansible.out: TASK [container-host : Restart docker service] ********************************* 2021-05-25T11:56:34.198 INFO:teuthology.task.ansible.out:skipping: [smithi181.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-25T11:56:34.264 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched 2021-05-25T11:56:34.265 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** skipping: no hosts matched 2021-05-25T11:56:34.266 INFO:teuthology.task.ansible.out: PLAY [pulpito] ***************************************************************** skipping: no hosts matched 2021-05-25T11:56:34.268 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2021-05-25T11:56:34.280 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* Tuesday 25 May 2021 11:56:34 +0000 (0:00:00.110) 0:04:21.628 *********** 2021-05-25T11:56:35.914 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] 2021-05-25T11:56:35.944 INFO:teuthology.task.ansible.out:ok: [smithi181.front.sepia.ceph.com] 2021-05-25T11:56:35.961 INFO:teuthology.task.ansible.out: TASK [Touch /ceph-qa-ready] **************************************************** 2021-05-25T11:56:35.963 INFO:teuthology.task.ansible.out:Tuesday 25 May 2021 11:56:35 +0000 (0:00:01.681) 0:04:23.310 *********** 2021-05-25T11:56:36.155 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2021-05-25T11:56:36.176 INFO:teuthology.task.ansible.out:changed: [smithi181.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2021-05-25T11:56:36.178 INFO:teuthology.task.ansible.out: PLAY RECAP ********************************************************************* 2021-05-25T11:56:36.180 INFO:teuthology.task.ansible.out:smithi102.front.sepia.ceph.com : ok=130 changed=43 unreachable=0 failed=0 skipped=188 rescued=0 ignored=3 smithi181.front.sepia.ceph.com : ok=130 changed=43 unreachable=0 failed=0 skipped=188 rescued=0 ignored=3 Tuesday 25 May 2021 11:56:36 +0000 (0:00:00.217) 0:04:23.527 *********** =============================================================================== common : nrpe - Ensure SELinux policy is up to date -------------------- 38.47s Create all admin users with sudo access. ------------------------------- 33.78s users : Update authorized_keys using the keys repo --------------------- 28.20s common : nrpe - Remove SELinux policy package -------------------------- 16.47s common : nrpe - Load SELinux policy package ---------------------------- 16.41s Remove revoked users --------------------------------------------------- 10.35s testnode : Zap all non-root disks --------------------------------------- 7.79s testnode : Install packages --------------------------------------------- 7.23s common : Install nrpe package and dependencies (RHEL/CentOS) ------------ 5.14s common : Install Kerberos Packages (RedHat) ----------------------------- 3.42s testnode : Ensure ceph packages are not present. ------------------------ 2.85s testnode : Ensure ceph dependency packages are not present. ------------- 2.65s testnode : Rebuild rpmdb ------------------------------------------------ 2.52s container-host : Install registries-conf-ctl ---------------------------- 2.09s testnode : Blow away lingering OSD data and FSIDs ----------------------- 1.84s testnode : Create logical volume(s) ------------------------------------- 1.77s Gathering Facts --------------------------------------------------------- 1.68s Gathering Facts --------------------------------------------------------- 1.60s Gathering Facts --------------------------------------------------------- 1.49s 2021-05-25T11:56:36.180 INFO:teuthology.task.ansible.out:testnode : Make sure rpm dependencies are install 2021-05-25T11:56:36.182 INFO:teuthology.task.ansible.out:ed --------------------- 1.35s 2021-05-25T11:56:36.514 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi102.front.sepia.ceph.com'), Remote(name='ubuntu@smithi181.front.sepia.ceph.com')] 2021-05-25T11:56:36.516 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-25T11:56:36.518 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-05-25T11:56:36.588 DEBUG:teuthology.orchestra.run.smithi102:> true 2021-05-25T11:56:36.678 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-25T11:56:36.679 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-05-25T11:56:36.751 DEBUG:teuthology.orchestra.run.smithi181:> true 2021-05-25T11:56:36.846 INFO:teuthology.run_tasks:Running task clock... 2021-05-25T11:56:36.857 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2021-05-25T11:56:36.858 INFO:teuthology.orchestra.run:Running command with timeout 360 2021-05-25T11:56:36.860 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-05-25T11:56:36.891 INFO:teuthology.orchestra.run.smithi102.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2021-05-25T11:56:36.905 INFO:teuthology.orchestra.run.smithi102.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2021-05-25T11:56:36.939 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: ntpd: command not found 2021-05-25T11:56:36.949 INFO:teuthology.orchestra.run.smithi102.stdout:506 Cannot talk to daemon 2021-05-25T11:56:36.963 INFO:teuthology.orchestra.run.smithi102.stderr:Failed to start ntp.service: Unit ntp.service not found. 2021-05-25T11:56:36.976 INFO:teuthology.orchestra.run.smithi102.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2021-05-25T11:56:37.032 INFO:teuthology.orchestra.run.smithi102.stderr:bash: ntpq: command not found 2021-05-25T11:56:37.035 INFO:teuthology.orchestra.run.smithi102.stdout:210 Number of sources = 4 2021-05-25T11:56:37.035 INFO:teuthology.orchestra.run.smithi102.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2021-05-25T11:56:37.035 INFO:teuthology.orchestra.run.smithi102.stdout:=============================================================================== 2021-05-25T11:56:37.036 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-05-25T11:56:37.036 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-05-25T11:56:37.036 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-05-25T11:56:37.036 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-05-25T11:56:37.037 INFO:teuthology.orchestra.run:Running command with timeout 360 2021-05-25T11:56:37.038 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-05-25T11:56:37.070 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2021-05-25T11:56:37.085 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2021-05-25T11:56:37.120 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: ntpd: command not found 2021-05-25T11:56:37.130 INFO:teuthology.orchestra.run.smithi181.stdout:506 Cannot talk to daemon 2021-05-25T11:56:37.144 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to start ntp.service: Unit ntp.service not found. 2021-05-25T11:56:37.158 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2021-05-25T11:56:37.221 INFO:teuthology.orchestra.run.smithi181.stderr:bash: ntpq: command not found 2021-05-25T11:56:37.223 INFO:teuthology.orchestra.run.smithi181.stdout:210 Number of sources = 4 2021-05-25T11:56:37.223 INFO:teuthology.orchestra.run.smithi181.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2021-05-25T11:56:37.224 INFO:teuthology.orchestra.run.smithi181.stdout:=============================================================================== 2021-05-25T11:56:37.224 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-05-25T11:56:37.224 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-05-25T11:56:37.224 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-05-25T11:56:37.225 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-05-25T11:56:37.226 INFO:teuthology.run_tasks:Running task install... 2021-05-25T11:56:37.235 DEBUG:teuthology.task.install:project ceph 2021-05-25T11:56:37.235 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': 'ece5ed1ac99f43a50411c04b4ca92c148baf0cd1'}} 2021-05-25T11:56:37.236 DEBUG:teuthology.task.install:config {'sha1': 'ece5ed1ac99f43a50411c04b4ca92c148baf0cd1'} 2021-05-25T11:56:37.236 INFO:teuthology.task.install:Using flavor: basic 2021-05-25T11:56:37.243 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2021-05-25T11:56:37.243 INFO:teuthology.task.install:extra packages: [] 2021-05-25T11:56:37.244 DEBUG:teuthology.orchestra.run.smithi102:> sudo lsb_release -is 2021-05-25T11:56:37.276 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS 2021-05-25T11:56:37.277 DEBUG:teuthology.misc:System to be installed: CentOS 2021-05-25T11:56:37.278 DEBUG:teuthology.orchestra.run.smithi181:> sudo lsb_release -is 2021-05-25T11:56:37.279 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'flavor': 'basic', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'ece5ed1ac99f43a50411c04b4ca92c148baf0cd1', 'tag': None, 'wait_for_package': False} 2021-05-25T11:56:37.279 DEBUG:teuthology.orchestra.run.smithi102:> sudo lsb_release -is 2021-05-25T11:56:37.311 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS 2021-05-25T11:56:37.312 DEBUG:teuthology.misc:System to be installed: CentOS 2021-05-25T11:56:37.313 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'flavor': 'basic', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'ece5ed1ac99f43a50411c04b4ca92c148baf0cd1', 'tag': None, 'wait_for_package': False} 2021-05-25T11:56:37.313 DEBUG:teuthology.orchestra.run.smithi181:> sudo lsb_release -is 2021-05-25T11:56:37.349 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS 2021-05-25T11:56:37.350 DEBUG:teuthology.misc:System to be installed: CentOS 2021-05-25T11:56:37.351 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=ece5ed1ac99f43a50411c04b4ca92c148baf0cd1 2021-05-25T11:56:37.388 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS 2021-05-25T11:56:37.388 DEBUG:teuthology.misc:System to be installed: CentOS 2021-05-25T11:56:37.388 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=ece5ed1ac99f43a50411c04b4ca92c148baf0cd1 2021-05-25T11:56:37.498 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/master/ece5ed1ac99f43a50411c04b4ca92c148baf0cd1/centos/8/flavors/default/ 2021-05-25T11:56:37.499 INFO:teuthology.task.install.rpm:Package version is 17.0.0-4502.gece5ed1a 2021-05-25T11:56:37.534 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/master/ece5ed1ac99f43a50411c04b4ca92c148baf0cd1/centos/8/flavors/default/ 2021-05-25T11:56:37.534 INFO:teuthology.task.install.rpm:Package version is 17.0.0-4502.gece5ed1a 2021-05-25T11:56:37.620 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/master/ece5ed1ac99f43a50411c04b4ca92c148baf0cd1/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/master/ece5ed1ac99f43a50411c04b4ca92c148baf0cd1/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/master/ece5ed1ac99f43a50411c04b4ca92c148baf0cd1/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2021-05-25T11:56:37.621 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2021-05-25T11:56:37.621 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/etc/yum.repos.d/ceph.repo 2021-05-25T11:56:37.646 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2021-05-25T11:56:37.646 DEBUG:teuthology.orchestra.run.smithi102:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/ece5ed1ac99f43a50411c04b4ca92c148baf0cd1/;g' /etc/yum.repos.d/ceph.repo ; fi 2021-05-25T11:56:37.649 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/master/ece5ed1ac99f43a50411c04b4ca92c148baf0cd1/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/master/ece5ed1ac99f43a50411c04b4ca92c148baf0cd1/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/master/ece5ed1ac99f43a50411c04b4ca92c148baf0cd1/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2021-05-25T11:56:37.649 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2021-05-25T11:56:37.650 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/yum.repos.d/ceph.repo 2021-05-25T11:56:37.674 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2021-05-25T11:56:37.674 DEBUG:teuthology.orchestra.run.smithi181:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/ece5ed1ac99f43a50411c04b4ca92c148baf0cd1/;g' /etc/yum.repos.d/ceph.repo ; fi 2021-05-25T11:56:37.711 DEBUG:teuthology.orchestra.run.smithi102:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2021-05-25T11:56:37.739 DEBUG:teuthology.orchestra.run.smithi181:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2021-05-25T11:56:37.784 DEBUG:teuthology.orchestra.run.smithi102:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2021-05-25T11:56:37.812 DEBUG:teuthology.orchestra.run.smithi181:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2021-05-25T11:56:37.847 INFO:teuthology.orchestra.run.smithi102.stdout:check_obsoletes = 1 2021-05-25T11:56:37.848 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum clean all 2021-05-25T11:56:37.876 INFO:teuthology.orchestra.run.smithi181.stdout:check_obsoletes = 1 2021-05-25T11:56:37.877 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum clean all 2021-05-25T11:56:38.222 INFO:teuthology.orchestra.run.smithi102.stdout:72 files removed 2021-05-25T11:56:38.256 INFO:teuthology.orchestra.run.smithi181.stdout:72 files removed 2021-05-25T11:56:38.260 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-radosgw 2021-05-25T11:56:38.292 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-radosgw 2021-05-25T11:56:38.908 INFO:teuthology.orchestra.run.smithi102.stdout:ceph packages for x86_64 317 kB/s | 77 kB 00:00 2021-05-25T11:56:38.938 INFO:teuthology.orchestra.run.smithi181.stdout:ceph packages for x86_64 318 kB/s | 77 kB 00:00 2021-05-25T11:56:39.137 INFO:teuthology.orchestra.run.smithi102.stdout:ceph noarch packages 78 kB/s | 16 kB 00:00 2021-05-25T11:56:39.168 INFO:teuthology.orchestra.run.smithi181.stdout:ceph noarch packages 78 kB/s | 16 kB 00:00 2021-05-25T11:56:39.359 INFO:teuthology.orchestra.run.smithi102.stdout:ceph source packages 7.2 kB/s | 1.5 kB 00:00 2021-05-25T11:56:39.389 INFO:teuthology.orchestra.run.smithi181.stdout:ceph source packages 7.2 kB/s | 1.5 kB 00:00 2021-05-25T11:56:39.526 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS-8 - AppStream 42 MB/s | 6.3 MB 00:00 2021-05-25T11:56:39.574 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS-8 - AppStream 37 MB/s | 6.3 MB 00:00 2021-05-25T11:56:40.604 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS-8 - Base 26 MB/s | 2.3 MB 00:00 2021-05-25T11:56:40.662 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS-8 - Base 26 MB/s | 2.3 MB 00:00 2021-05-25T11:56:41.040 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS-8 - Extras 147 kB/s | 9.6 kB 00:00 2021-05-25T11:56:41.104 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS-8 - Extras 151 kB/s | 9.6 kB 00:00 2021-05-25T11:56:41.632 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Linux 8 - AppStream 12 MB/s | 6.3 MB 00:00 2021-05-25T11:56:41.807 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Linux 8 - AppStream 9.6 MB/s | 6.3 MB 00:00 2021-05-25T11:56:46.271 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Linux 8 - BaseOS 677 kB/s | 2.3 MB 00:03 2021-05-25T11:56:46.831 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Linux 8 - BaseOS 553 kB/s | 2.3 MB 00:04 2021-05-25T11:56:46.967 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Linux 8 - Extras 30 kB/s | 9.6 kB 00:00 2021-05-25T11:56:47.507 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Linux 8 - Extras 32 kB/s | 9.6 kB 00:00 2021-05-25T11:56:47.726 INFO:teuthology.orchestra.run.smithi181.stdout:Extra Packages for Enterprise Linux 13 MB/s | 9.4 MB 00:00 2021-05-25T11:56:48.263 INFO:teuthology.orchestra.run.smithi102.stdout:Extra Packages for Enterprise Linux 13 MB/s | 9.4 MB 00:00 2021-05-25T11:56:49.688 INFO:teuthology.orchestra.run.smithi181.stdout:lab-extras 4.4 MB/s | 24 kB 00:00 2021-05-25T11:56:50.222 INFO:teuthology.orchestra.run.smithi102.stdout:lab-extras 4.6 MB/s | 24 kB 00:00 2021-05-25T11:56:51.421 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2021-05-25T11:56:51.442 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2021-05-25T11:56:51.442 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2021-05-25T11:56:51.442 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2021-05-25T11:56:51.443 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2021-05-25T11:56:51.443 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-radosgw x86_64 2:17.0.0-4502.gece5ed1a.el8 ceph 11 M 2021-05-25T11:56:51.443 INFO:teuthology.orchestra.run.smithi181.stdout:Upgrading: 2021-05-25T11:56:51.444 INFO:teuthology.orchestra.run.smithi181.stdout: librados2 x86_64 2:17.0.0-4502.gece5ed1a.el8 ceph 3.4 M 2021-05-25T11:56:51.444 INFO:teuthology.orchestra.run.smithi181.stdout: librbd1 x86_64 2:17.0.0-4502.gece5ed1a.el8 ceph 3.8 M 2021-05-25T11:56:51.445 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2021-05-25T11:56:51.445 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-base x86_64 2:17.0.0-4502.gece5ed1a.el8 ceph 5.4 M 2021-05-25T11:56:51.445 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-common x86_64 2:17.0.0-4502.gece5ed1a.el8 ceph 19 M 2021-05-25T11:56:51.445 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-selinux x86_64 2:17.0.0-4502.gece5ed1a.el8 ceph 23 k 2021-05-25T11:56:51.446 INFO:teuthology.orchestra.run.smithi181.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2021-05-25T11:56:51.446 INFO:teuthology.orchestra.run.smithi181.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2021-05-25T11:56:51.446 INFO:teuthology.orchestra.run.smithi181.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2021-05-25T11:56:51.446 INFO:teuthology.orchestra.run.smithi181.stdout: libcephfs2 x86_64 2:17.0.0-4502.gece5ed1a.el8 ceph 756 k 2021-05-25T11:56:51.447 INFO:teuthology.orchestra.run.smithi181.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2021-05-25T11:56:51.447 INFO:teuthology.orchestra.run.smithi181.stdout: librabbitmq x86_64 0.9.0-2.el8 CentOS-Base 47 k 2021-05-25T11:56:51.447 INFO:teuthology.orchestra.run.smithi181.stdout: libradosstriper1 x86_64 2:17.0.0-4502.gece5ed1a.el8 ceph 457 k 2021-05-25T11:56:51.447 INFO:teuthology.orchestra.run.smithi181.stdout: librdkafka x86_64 0.11.4-1.el8 CentOS-AppStream 353 k 2021-05-25T11:56:51.448 INFO:teuthology.orchestra.run.smithi181.stdout: librgw2 x86_64 2:17.0.0-4502.gece5ed1a.el8 ceph 3.9 M 2021-05-25T11:56:51.448 INFO:teuthology.orchestra.run.smithi181.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2021-05-25T11:56:51.448 INFO:teuthology.orchestra.run.smithi181.stdout: lttng-ust x86_64 2.8.1-11.el8 CentOS-AppStream 259 k 2021-05-25T11:56:51.448 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-argparse 2021-05-25T11:56:51.449 INFO:teuthology.orchestra.run.smithi181.stdout: x86_64 2:17.0.0-4502.gece5ed1a.el8 ceph 42 k 2021-05-25T11:56:51.449 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-common x86_64 2:17.0.0-4502.gece5ed1a.el8 ceph 77 k 2021-05-25T11:56:51.449 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cephfs x86_64 2:17.0.0-4502.gece5ed1a.el8 ceph 177 k 2021-05-25T11:56:51.449 INFO:teuthology.orchestra.run.smithi181.stdout: python3-prettytable noarch 0.7.2-14.el8 CentOS-AppStream 44 k 2021-05-25T11:56:51.449 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rados x86_64 2:17.0.0-4502.gece5ed1a.el8 ceph 364 k 2021-05-25T11:56:51.450 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rbd x86_64 2:17.0.0-4502.gece5ed1a.el8 ceph 345 k 2021-05-25T11:56:51.450 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rgw x86_64 2:17.0.0-4502.gece5ed1a.el8 ceph 109 k 2021-05-25T11:56:51.450 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:56:51.450 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2021-05-25T11:56:51.451 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2021-05-25T11:56:51.451 INFO:teuthology.orchestra.run.smithi181.stdout:Install 22 Packages 2021-05-25T11:56:51.451 INFO:teuthology.orchestra.run.smithi181.stdout:Upgrade 2 Packages 2021-05-25T11:56:51.451 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:56:51.452 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 50 M 2021-05-25T11:56:51.452 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2021-05-25T11:56:51.890 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2021-05-25T11:56:51.906 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-05-25T11:56:51.906 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2021-05-25T11:56:51.906 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-05-25T11:56:51.907 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2021-05-25T11:56:51.907 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-radosgw x86_64 2:17.0.0-4502.gece5ed1a.el8 ceph 11 M 2021-05-25T11:56:51.907 INFO:teuthology.orchestra.run.smithi102.stdout:Upgrading: 2021-05-25T11:56:51.907 INFO:teuthology.orchestra.run.smithi102.stdout: librados2 x86_64 2:17.0.0-4502.gece5ed1a.el8 ceph 3.4 M 2021-05-25T11:56:51.907 INFO:teuthology.orchestra.run.smithi102.stdout: librbd1 x86_64 2:17.0.0-4502.gece5ed1a.el8 ceph 3.8 M 2021-05-25T11:56:51.908 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2021-05-25T11:56:51.908 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-base x86_64 2:17.0.0-4502.gece5ed1a.el8 ceph 5.4 M 2021-05-25T11:56:51.908 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-common x86_64 2:17.0.0-4502.gece5ed1a.el8 ceph 19 M 2021-05-25T11:56:51.908 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-selinux x86_64 2:17.0.0-4502.gece5ed1a.el8 ceph 23 k 2021-05-25T11:56:51.909 INFO:teuthology.orchestra.run.smithi102.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2021-05-25T11:56:51.909 INFO:teuthology.orchestra.run.smithi102.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2021-05-25T11:56:51.909 INFO:teuthology.orchestra.run.smithi102.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2021-05-25T11:56:51.909 INFO:teuthology.orchestra.run.smithi102.stdout: libcephfs2 x86_64 2:17.0.0-4502.gece5ed1a.el8 ceph 756 k 2021-05-25T11:56:51.910 INFO:teuthology.orchestra.run.smithi102.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2021-05-25T11:56:51.910 INFO:teuthology.orchestra.run.smithi102.stdout: librabbitmq x86_64 0.9.0-2.el8 CentOS-Base 47 k 2021-05-25T11:56:51.910 INFO:teuthology.orchestra.run.smithi102.stdout: libradosstriper1 x86_64 2:17.0.0-4502.gece5ed1a.el8 ceph 457 k 2021-05-25T11:56:51.910 INFO:teuthology.orchestra.run.smithi102.stdout: librdkafka x86_64 0.11.4-1.el8 CentOS-AppStream 353 k 2021-05-25T11:56:51.910 INFO:teuthology.orchestra.run.smithi102.stdout: librgw2 x86_64 2:17.0.0-4502.gece5ed1a.el8 ceph 3.9 M 2021-05-25T11:56:51.911 INFO:teuthology.orchestra.run.smithi102.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2021-05-25T11:56:51.911 INFO:teuthology.orchestra.run.smithi102.stdout: lttng-ust x86_64 2.8.1-11.el8 CentOS-AppStream 259 k 2021-05-25T11:56:51.911 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ceph-argparse 2021-05-25T11:56:51.911 INFO:teuthology.orchestra.run.smithi102.stdout: x86_64 2:17.0.0-4502.gece5ed1a.el8 ceph 42 k 2021-05-25T11:56:51.912 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ceph-common x86_64 2:17.0.0-4502.gece5ed1a.el8 ceph 77 k 2021-05-25T11:56:51.912 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cephfs x86_64 2:17.0.0-4502.gece5ed1a.el8 ceph 177 k 2021-05-25T11:56:51.912 INFO:teuthology.orchestra.run.smithi102.stdout: python3-prettytable noarch 0.7.2-14.el8 CentOS-AppStream 44 k 2021-05-25T11:56:51.912 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rados x86_64 2:17.0.0-4502.gece5ed1a.el8 ceph 364 k 2021-05-25T11:56:51.913 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rbd x86_64 2:17.0.0-4502.gece5ed1a.el8 ceph 345 k 2021-05-25T11:56:51.913 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rgw x86_64 2:17.0.0-4502.gece5ed1a.el8 ceph 109 k 2021-05-25T11:56:51.913 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:56:51.913 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2021-05-25T11:56:51.913 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-05-25T11:56:51.914 INFO:teuthology.orchestra.run.smithi102.stdout:Install 22 Packages 2021-05-25T11:56:51.914 INFO:teuthology.orchestra.run.smithi102.stdout:Upgrade 2 Packages 2021-05-25T11:56:51.914 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:56:51.915 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 50 M 2021-05-25T11:56:51.915 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2021-05-25T11:56:52.111 INFO:teuthology.orchestra.run.smithi181.stdout:(1/24): ceph-base-17.0.0-4502.gece5ed1a.el8.x86 8.2 MB/s | 5.4 MB 00:00 2021-05-25T11:56:52.164 INFO:teuthology.orchestra.run.smithi181.stdout:(2/24): ceph-selinux-17.0.0-4502.gece5ed1a.el8. 448 kB/s | 23 kB 00:00 2021-05-25T11:56:52.245 INFO:teuthology.orchestra.run.smithi181.stdout:(3/24): libcephfs2-17.0.0-4502.gece5ed1a.el8.x8 9.1 MB/s | 756 kB 00:00 2021-05-25T11:56:52.320 INFO:teuthology.orchestra.run.smithi181.stdout:(4/24): libradosstriper1-17.0.0-4502.gece5ed1a. 6.0 MB/s | 457 kB 00:00 2021-05-25T11:56:52.784 INFO:teuthology.orchestra.run.smithi181.stdout:(5/24): librgw2-17.0.0-4502.gece5ed1a.el8.x86_6 8.3 MB/s | 3.9 MB 00:00 2021-05-25T11:56:52.889 INFO:teuthology.orchestra.run.smithi181.stdout:(6/24): ceph-radosgw-17.0.0-4502.gece5ed1a.el8. 7.4 MB/s | 11 MB 00:01 2021-05-25T11:56:52.889 INFO:teuthology.orchestra.run.smithi181.stdout:(7/24): python3-ceph-argparse-17.0.0-4502.gece5 402 kB/s | 42 kB 00:00 2021-05-25T11:56:52.923 INFO:teuthology.orchestra.run.smithi181.stdout:(8/24): python3-ceph-common-17.0.0-4502.gece5ed 2.2 MB/s | 77 kB 00:00 2021-05-25T11:56:52.929 INFO:teuthology.orchestra.run.smithi181.stdout:(9/24): python3-cephfs-17.0.0-4502.gece5ed1a.el 4.4 MB/s | 177 kB 00:00 2021-05-25T11:56:52.968 INFO:teuthology.orchestra.run.smithi181.stdout:(10/24): python3-rados-17.0.0-4502.gece5ed1a.el 7.9 MB/s | 364 kB 00:00 2021-05-25T11:56:52.981 INFO:teuthology.orchestra.run.smithi181.stdout:(11/24): python3-rbd-17.0.0-4502.gece5ed1a.el8. 6.5 MB/s | 345 kB 00:00 2021-05-25T11:56:53.011 INFO:teuthology.orchestra.run.smithi181.stdout:(12/24): python3-rgw-17.0.0-4502.gece5ed1a.el8. 2.5 MB/s | 109 kB 00:00 2021-05-25T11:56:53.021 INFO:teuthology.orchestra.run.smithi181.stdout:(13/24): librdkafka-0.11.4-1.el8.x86_64.rpm 8.7 MB/s | 353 kB 00:00 2021-05-25T11:56:53.023 INFO:teuthology.orchestra.run.smithi181.stdout:(14/24): python3-prettytable-0.7.2-14.el8.noarc 28 MB/s | 44 kB 00:00 2021-05-25T11:56:53.024 INFO:teuthology.orchestra.run.smithi181.stdout:(15/24): librabbitmq-0.9.0-2.el8.x86_64.rpm 32 MB/s | 47 kB 00:00 2021-05-25T11:56:53.043 INFO:teuthology.orchestra.run.smithi181.stdout:(16/24): lttng-ust-2.8.1-11.el8.x86_64.rpm 7.8 MB/s | 259 kB 00:00 2021-05-25T11:56:53.138 INFO:teuthology.orchestra.run.smithi181.stdout:(17/24): fmt-6.2.1-1.el8.x86_64.rpm 1.0 MB/s | 116 kB 00:00 2021-05-25T11:56:53.182 INFO:teuthology.orchestra.run.smithi181.stdout:(18/24): leveldb-1.22-1.el8.x86_64.rpm 4.0 MB/s | 181 kB 00:00 2021-05-25T11:56:53.204 INFO:teuthology.orchestra.run.smithi181.stdout:(19/24): gperftools-libs-2.7-9.el8.x86_64.rpm 1.9 MB/s | 306 kB 00:00 2021-05-25T11:56:53.205 INFO:teuthology.orchestra.run.smithi181.stdout:(20/24): liboath-2.6.2-3.el8.x86_64.rpm 2.5 MB/s | 59 kB 00:00 2021-05-25T11:56:53.232 INFO:teuthology.orchestra.run.smithi181.stdout:(21/24): libunwind-1.3.1-3.el8.x86_64.rpm 2.6 MB/s | 75 kB 00:00 2021-05-25T11:56:53.574 INFO:teuthology.orchestra.run.smithi181.stdout:(22/24): ceph-common-17.0.0-4502.gece5ed1a.el8. 9.1 MB/s | 19 MB 00:02 2021-05-25T11:56:53.648 INFO:teuthology.orchestra.run.smithi181.stdout:(23/24): librados2-17.0.0-4502.gece5ed1a.el8.x8 7.6 MB/s | 3.4 MB 00:00 2021-05-25T11:56:53.661 INFO:teuthology.orchestra.run.smithi181.stdout:(24/24): librbd1-17.0.0-4502.gece5ed1a.el8.x86_ 8.9 MB/s | 3.8 MB 00:00 2021-05-25T11:56:53.664 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2021-05-25T11:56:53.665 INFO:teuthology.orchestra.run.smithi181.stdout:Total 23 MB/s | 50 MB 00:02 2021-05-25T11:56:53.668 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2021-05-25T11:56:53.714 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2021-05-25T11:56:53.714 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2021-05-25T11:56:53.909 INFO:teuthology.orchestra.run.smithi102.stdout:(1/24): ceph-base-17.0.0-4502.gece5ed1a.el8.x86 2.7 MB/s | 5.4 MB 00:01 2021-05-25T11:56:53.937 INFO:teuthology.orchestra.run.smithi102.stdout:(2/24): ceph-selinux-17.0.0-4502.gece5ed1a.el8. 837 kB/s | 23 kB 00:00 2021-05-25T11:56:54.041 INFO:teuthology.orchestra.run.smithi102.stdout:(3/24): libcephfs2-17.0.0-4502.gece5ed1a.el8.x8 7.1 MB/s | 756 kB 00:00 2021-05-25T11:56:54.091 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2021-05-25T11:56:54.094 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2021-05-25T11:56:54.116 INFO:teuthology.orchestra.run.smithi102.stdout:(4/24): libradosstriper1-17.0.0-4502.gece5ed1a. 6.0 MB/s | 457 kB 00:00 2021-05-25T11:56:54.389 INFO:teuthology.orchestra.run.smithi102.stdout:(5/24): ceph-radosgw-17.0.0-4502.gece5ed1a.el8. 4.3 MB/s | 11 MB 00:02 2021-05-25T11:56:54.434 INFO:teuthology.orchestra.run.smithi102.stdout:(6/24): python3-ceph-argparse-17.0.0-4502.gece5 941 kB/s | 42 kB 00:00 2021-05-25T11:56:54.477 INFO:teuthology.orchestra.run.smithi102.stdout:(7/24): python3-ceph-common-17.0.0-4502.gece5ed 1.8 MB/s | 77 kB 00:00 2021-05-25T11:56:54.525 INFO:teuthology.orchestra.run.smithi102.stdout:(8/24): python3-cephfs-17.0.0-4502.gece5ed1a.el 3.6 MB/s | 177 kB 00:00 2021-05-25T11:56:54.537 INFO:teuthology.orchestra.run.smithi102.stdout:(9/24): librgw2-17.0.0-4502.gece5ed1a.el8.x86_6 9.2 MB/s | 3.9 MB 00:00 2021-05-25T11:56:54.564 INFO:teuthology.orchestra.run.smithi102.stdout:(10/24): python3-rados-17.0.0-4502.gece5ed1a.el 9.1 MB/s | 364 kB 00:00 2021-05-25T11:56:54.591 INFO:teuthology.orchestra.run.smithi102.stdout:(11/24): python3-rbd-17.0.0-4502.gece5ed1a.el8. 6.2 MB/s | 345 kB 00:00 2021-05-25T11:56:54.606 INFO:teuthology.orchestra.run.smithi102.stdout:(12/24): python3-rgw-17.0.0-4502.gece5ed1a.el8. 2.6 MB/s | 109 kB 00:00 2021-05-25T11:56:54.627 INFO:teuthology.orchestra.run.smithi102.stdout:(13/24): librdkafka-0.11.4-1.el8.x86_64.rpm 9.8 MB/s | 353 kB 00:00 2021-05-25T11:56:54.628 INFO:teuthology.orchestra.run.smithi102.stdout:(14/24): python3-prettytable-0.7.2-14.el8.noarc 32 MB/s | 44 kB 00:00 2021-05-25T11:56:54.630 INFO:teuthology.orchestra.run.smithi102.stdout:(15/24): librabbitmq-0.9.0-2.el8.x86_64.rpm 32 MB/s | 47 kB 00:00 2021-05-25T11:56:54.638 INFO:teuthology.orchestra.run.smithi102.stdout:(16/24): lttng-ust-2.8.1-11.el8.x86_64.rpm 7.8 MB/s | 259 kB 00:00 2021-05-25T11:56:54.885 INFO:teuthology.orchestra.run.smithi102.stdout:(17/24): ceph-common-17.0.0-4502.gece5ed1a.el8. 6.5 MB/s | 19 MB 00:02 2021-05-25T11:56:54.886 INFO:teuthology.orchestra.run.smithi102.stdout:(18/24): fmt-6.2.1-1.el8.x86_64.rpm 455 kB/s | 116 kB 00:00 2021-05-25T11:56:54.905 INFO:teuthology.orchestra.run.smithi102.stdout:(19/24): gperftools-libs-2.7-9.el8.x86_64.rpm 1.1 MB/s | 306 kB 00:00 2021-05-25T11:56:54.913 INFO:teuthology.orchestra.run.smithi102.stdout:(20/24): liboath-2.6.2-3.el8.x86_64.rpm 2.1 MB/s | 59 kB 00:00 2021-05-25T11:56:54.933 INFO:teuthology.orchestra.run.smithi102.stdout:(21/24): libunwind-1.3.1-3.el8.x86_64.rpm 2.7 MB/s | 75 kB 00:00 2021-05-25T11:56:54.995 INFO:teuthology.orchestra.run.smithi102.stdout:(22/24): leveldb-1.22-1.el8.x86_64.rpm 1.6 MB/s | 181 kB 00:00 2021-05-25T11:56:55.132 INFO:teuthology.orchestra.run.smithi102.stdout:(23/24): librbd1-17.0.0-4502.gece5ed1a.el8.x86_ 19 MB/s | 3.8 MB 00:00 2021-05-25T11:56:55.212 INFO:teuthology.orchestra.run.smithi102.stdout:(24/24): librados2-17.0.0-4502.gece5ed1a.el8.x8 11 MB/s | 3.4 MB 00:00 2021-05-25T11:56:55.215 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2021-05-25T11:56:55.216 INFO:teuthology.orchestra.run.smithi102.stdout:Total 15 MB/s | 50 MB 00:03 2021-05-25T11:56:55.219 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2021-05-25T11:56:55.264 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2021-05-25T11:56:55.264 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2021-05-25T11:56:55.278 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2021-05-25T11:56:55.451 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/26 2021-05-25T11:56:55.634 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2021-05-25T11:56:55.638 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2021-05-25T11:56:55.652 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/26 2021-05-25T11:56:55.811 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/26 2021-05-25T11:56:56.205 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/26 2021-05-25T11:56:56.436 INFO:teuthology.orchestra.run.smithi181.stdout: Upgrading : librados2-2:17.0.0-4502.gece5ed1a.el8.x86_64 4/26 2021-05-25T11:56:56.537 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librados2-2:17.0.0-4502.gece5ed1a.el8.x86_64 4/26 2021-05-25T11:56:56.769 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-rados-2:17.0.0-4502.gece5ed1a.el8.x86_64 5/26 2021-05-25T11:56:56.786 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2021-05-25T11:56:56.961 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libcephfs2-2:17.0.0-4502.gece5ed1a.el8.x86_64 6/26 2021-05-25T11:56:56.988 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/26 2021-05-25T11:56:57.214 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/26 2021-05-25T11:56:57.415 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/26 2021-05-25T11:56:57.430 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libcephfs2-2:17.0.0-4502.gece5ed1a.el8.x86_64 6/26 2021-05-25T11:56:57.636 INFO:teuthology.orchestra.run.smithi181.stdout: Upgrading : librbd1-2:17.0.0-4502.gece5ed1a.el8.x86_64 7/26 2021-05-25T11:56:57.710 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librbd1-2:17.0.0-4502.gece5ed1a.el8.x86_64 7/26 2021-05-25T11:56:57.806 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/26 2021-05-25T11:56:57.846 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : librabbitmq-0.9.0-2.el8.x86_64 8/26 2021-05-25T11:56:58.011 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 9/26 2021-05-25T11:56:58.090 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : librados2-2:17.0.0-4502.gece5ed1a.el8.x86_64 4/26 2021-05-25T11:56:58.188 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librados2-2:17.0.0-4502.gece5ed1a.el8.x86_64 4/26 2021-05-25T11:56:58.420 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 9/26 2021-05-25T11:56:58.456 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-rados-2:17.0.0-4502.gece5ed1a.el8.x86_64 5/26 2021-05-25T11:56:58.586 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : librgw2-2:17.0.0-4502.gece5ed1a.el8.x86_64 10/26 2021-05-25T11:56:58.650 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librgw2-2:17.0.0-4502.gece5ed1a.el8.x86_64 10/26 2021-05-25T11:56:58.657 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libcephfs2-2:17.0.0-4502.gece5ed1a.el8.x86_64 6/26 2021-05-25T11:56:58.783 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : leveldb-1.22-1.el8.x86_64 11/26 2021-05-25T11:56:58.896 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-ceph-argparse-2:17.0.0-4502.gece5ed1a.el8. 12/26 2021-05-25T11:56:59.106 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cephfs-2:17.0.0-4502.gece5ed1a.el8.x86_64 13/26 2021-05-25T11:56:59.128 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: libcephfs2-2:17.0.0-4502.gece5ed1a.el8.x86_64 6/26 2021-05-25T11:56:59.311 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-rgw-2:17.0.0-4502.gece5ed1a.el8.x86_64 14/26 2021-05-25T11:56:59.348 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : librbd1-2:17.0.0-4502.gece5ed1a.el8.x86_64 7/26 2021-05-25T11:56:59.449 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librbd1-2:17.0.0-4502.gece5ed1a.el8.x86_64 7/26 2021-05-25T11:56:59.520 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-rbd-2:17.0.0-4502.gece5ed1a.el8.x86_64 15/26 2021-05-25T11:56:59.640 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : librabbitmq-0.9.0-2.el8.x86_64 8/26 2021-05-25T11:56:59.702 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libradosstriper1-2:17.0.0-4502.gece5ed1a.el8.x86_6 16/26 2021-05-25T11:56:59.770 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libradosstriper1-2:17.0.0-4502.gece5ed1a.el8.x86_6 16/26 2021-05-25T11:56:59.831 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 9/26 2021-05-25T11:56:59.969 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 17/26 2021-05-25T11:57:00.144 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 18/26 2021-05-25T11:57:00.260 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 9/26 2021-05-25T11:57:00.322 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 19/26 2021-05-25T11:57:00.481 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : librgw2-2:17.0.0-4502.gece5ed1a.el8.x86_64 10/26 2021-05-25T11:57:00.511 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-ceph-common-2:17.0.0-4502.gece5ed1a.el8.x8 20/26 2021-05-25T11:57:00.559 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librgw2-2:17.0.0-4502.gece5ed1a.el8.x86_64 10/26 2021-05-25T11:57:00.695 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : leveldb-1.22-1.el8.x86_64 11/26 2021-05-25T11:57:00.820 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-ceph-argparse-2:17.0.0-4502.gece5ed1a.el8. 12/26 2021-05-25T11:57:01.002 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cephfs-2:17.0.0-4502.gece5ed1a.el8.x86_64 13/26 2021-05-25T11:57:01.181 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-rgw-2:17.0.0-4502.gece5ed1a.el8.x86_64 14/26 2021-05-25T11:57:01.375 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-rbd-2:17.0.0-4502.gece5ed1a.el8.x86_64 15/26 2021-05-25T11:57:01.582 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libradosstriper1-2:17.0.0-4502.gece5ed1a.el8.x86_6 16/26 2021-05-25T11:57:01.645 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: libradosstriper1-2:17.0.0-4502.gece5ed1a.el8.x86_6 16/26 2021-05-25T11:57:01.832 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 17/26 2021-05-25T11:57:02.007 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 18/26 2021-05-25T11:57:02.167 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 19/26 2021-05-25T11:57:02.348 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-ceph-common-2:17.0.0-4502.gece5ed1a.el8.x8 20/26 2021-05-25T11:57:04.600 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-common-2:17.0.0-4502.gece5ed1a.el8.x86_64 21/26 2021-05-25T11:57:04.836 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-common-2:17.0.0-4502.gece5ed1a.el8.x86_64 21/26 2021-05-25T11:57:05.576 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-common-2:17.0.0-4502.gece5ed1a.el8.x86_64 21/26 2021-05-25T11:57:05.855 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-base-2:17.0.0-4502.gece5ed1a.el8.x86_64 22/26 2021-05-25T11:57:05.958 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-base-2:17.0.0-4502.gece5ed1a.el8.x86_64 22/26 2021-05-25T11:57:06.060 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-selinux-2:17.0.0-4502.gece5ed1a.el8.x86_64 23/26 2021-05-25T11:57:06.437 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-common-2:17.0.0-4502.gece5ed1a.el8.x86_64 21/26 2021-05-25T11:57:06.665 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-common-2:17.0.0-4502.gece5ed1a.el8.x86_64 21/26 2021-05-25T11:57:07.357 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-common-2:17.0.0-4502.gece5ed1a.el8.x86_64 21/26 2021-05-25T11:57:07.629 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-base-2:17.0.0-4502.gece5ed1a.el8.x86_64 22/26 2021-05-25T11:57:07.725 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-base-2:17.0.0-4502.gece5ed1a.el8.x86_64 22/26 2021-05-25T11:57:07.831 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-selinux-2:17.0.0-4502.gece5ed1a.el8.x86_64 23/26 2021-05-25T11:57:23.060 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-selinux-2:17.0.0-4502.gece5ed1a.el8.x86_64 23/26 2021-05-25T11:57:23.061 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /sys 2021-05-25T11:57:23.061 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /proc 2021-05-25T11:57:23.061 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /dev 2021-05-25T11:57:23.062 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /run 2021-05-25T11:57:23.062 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /mnt 2021-05-25T11:57:23.062 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /var/tmp 2021-05-25T11:57:23.062 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /home 2021-05-25T11:57:23.063 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /tmp 2021-05-25T11:57:23.063 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /dev 2021-05-25T11:57:23.063 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:57:24.352 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-radosgw-2:17.0.0-4502.gece5ed1a.el8.x86_64 24/26 2021-05-25T11:57:24.444 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-radosgw-2:17.0.0-4502.gece5ed1a.el8.x86_64 24/26 2021-05-25T11:57:24.445 INFO:teuthology.orchestra.run.smithi181.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 25/26 2021-05-25T11:57:24.668 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 25/26 2021-05-25T11:57:24.668 INFO:teuthology.orchestra.run.smithi181.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 26/26 2021-05-25T11:57:24.960 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-selinux-2:17.0.0-4502.gece5ed1a.el8.x86_64 23/26 2021-05-25T11:57:24.961 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /sys 2021-05-25T11:57:24.961 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /proc 2021-05-25T11:57:24.961 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /dev 2021-05-25T11:57:24.962 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /run 2021-05-25T11:57:24.962 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /mnt 2021-05-25T11:57:24.962 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /var/tmp 2021-05-25T11:57:24.962 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /home 2021-05-25T11:57:24.962 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /tmp 2021-05-25T11:57:24.963 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /dev 2021-05-25T11:57:24.963 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:57:25.439 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 26/26 2021-05-25T11:57:25.439 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-base-2:17.0.0-4502.gece5ed1a.el8.x86_64 1/26 2021-05-25T11:57:25.440 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-common-2:17.0.0-4502.gece5ed1a.el8.x86_64 2/26 2021-05-25T11:57:25.440 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-radosgw-2:17.0.0-4502.gece5ed1a.el8.x86_64 3/26 2021-05-25T11:57:25.440 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-selinux-2:17.0.0-4502.gece5ed1a.el8.x86_64 4/26 2021-05-25T11:57:25.440 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libcephfs2-2:17.0.0-4502.gece5ed1a.el8.x86_64 5/26 2021-05-25T11:57:25.441 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libradosstriper1-2:17.0.0-4502.gece5ed1a.el8.x86_6 6/26 2021-05-25T11:57:25.441 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librgw2-2:17.0.0-4502.gece5ed1a.el8.x86_64 7/26 2021-05-25T11:57:25.441 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-ceph-argparse-2:17.0.0-4502.gece5ed1a.el8. 8/26 2021-05-25T11:57:25.441 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-ceph-common-2:17.0.0-4502.gece5ed1a.el8.x8 9/26 2021-05-25T11:57:25.441 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cephfs-2:17.0.0-4502.gece5ed1a.el8.x86_64 10/26 2021-05-25T11:57:25.442 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-rados-2:17.0.0-4502.gece5ed1a.el8.x86_64 11/26 2021-05-25T11:57:25.442 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-rbd-2:17.0.0-4502.gece5ed1a.el8.x86_64 12/26 2021-05-25T11:57:25.442 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-rgw-2:17.0.0-4502.gece5ed1a.el8.x86_64 13/26 2021-05-25T11:57:25.442 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 14/26 2021-05-25T11:57:25.443 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/26 2021-05-25T11:57:25.443 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/26 2021-05-25T11:57:25.443 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librabbitmq-0.9.0-2.el8.x86_64 17/26 2021-05-25T11:57:25.443 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 18/26 2021-05-25T11:57:25.443 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 19/26 2021-05-25T11:57:25.444 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : leveldb-1.22-1.el8.x86_64 20/26 2021-05-25T11:57:25.444 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 21/26 2021-05-25T11:57:25.444 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 22/26 2021-05-25T11:57:25.444 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librados2-2:17.0.0-4502.gece5ed1a.el8.x86_64 23/26 2021-05-25T11:57:25.445 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 24/26 2021-05-25T11:57:25.445 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librbd1-2:17.0.0-4502.gece5ed1a.el8.x86_64 25/26 2021-05-25T11:57:26.210 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-radosgw-2:17.0.0-4502.gece5ed1a.el8.x86_64 24/26 2021-05-25T11:57:26.278 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 26/26 2021-05-25T11:57:26.279 INFO:teuthology.orchestra.run.smithi181.stdout:Installed products updated. 2021-05-25T11:57:26.288 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-radosgw-2:17.0.0-4502.gece5ed1a.el8.x86_64 24/26 2021-05-25T11:57:26.288 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 25/26 2021-05-25T11:57:26.505 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 25/26 2021-05-25T11:57:26.506 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 26/26 2021-05-25T11:57:26.549 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:57:26.550 INFO:teuthology.orchestra.run.smithi181.stdout:Upgraded: 2021-05-25T11:57:26.550 INFO:teuthology.orchestra.run.smithi181.stdout: librados2-2:17.0.0-4502.gece5ed1a.el8.x86_64 2021-05-25T11:57:26.550 INFO:teuthology.orchestra.run.smithi181.stdout: librbd1-2:17.0.0-4502.gece5ed1a.el8.x86_64 2021-05-25T11:57:26.550 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:57:26.551 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2021-05-25T11:57:26.551 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-base-2:17.0.0-4502.gece5ed1a.el8.x86_64 2021-05-25T11:57:26.551 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-common-2:17.0.0-4502.gece5ed1a.el8.x86_64 2021-05-25T11:57:26.551 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-radosgw-2:17.0.0-4502.gece5ed1a.el8.x86_64 2021-05-25T11:57:26.551 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-selinux-2:17.0.0-4502.gece5ed1a.el8.x86_64 2021-05-25T11:57:26.552 INFO:teuthology.orchestra.run.smithi181.stdout: fmt-6.2.1-1.el8.x86_64 2021-05-25T11:57:26.552 INFO:teuthology.orchestra.run.smithi181.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2021-05-25T11:57:26.552 INFO:teuthology.orchestra.run.smithi181.stdout: leveldb-1.22-1.el8.x86_64 2021-05-25T11:57:26.552 INFO:teuthology.orchestra.run.smithi181.stdout: libcephfs2-2:17.0.0-4502.gece5ed1a.el8.x86_64 2021-05-25T11:57:26.553 INFO:teuthology.orchestra.run.smithi181.stdout: liboath-2.6.2-3.el8.x86_64 2021-05-25T11:57:26.553 INFO:teuthology.orchestra.run.smithi181.stdout: librabbitmq-0.9.0-2.el8.x86_64 2021-05-25T11:57:26.553 INFO:teuthology.orchestra.run.smithi181.stdout: libradosstriper1-2:17.0.0-4502.gece5ed1a.el8.x86_64 2021-05-25T11:57:26.553 INFO:teuthology.orchestra.run.smithi181.stdout: librdkafka-0.11.4-1.el8.x86_64 2021-05-25T11:57:26.553 INFO:teuthology.orchestra.run.smithi181.stdout: librgw2-2:17.0.0-4502.gece5ed1a.el8.x86_64 2021-05-25T11:57:26.554 INFO:teuthology.orchestra.run.smithi181.stdout: libunwind-1.3.1-3.el8.x86_64 2021-05-25T11:57:26.554 INFO:teuthology.orchestra.run.smithi181.stdout: lttng-ust-2.8.1-11.el8.x86_64 2021-05-25T11:57:26.554 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-argparse-2:17.0.0-4502.gece5ed1a.el8.x86_64 2021-05-25T11:57:26.554 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-common-2:17.0.0-4502.gece5ed1a.el8.x86_64 2021-05-25T11:57:26.554 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cephfs-2:17.0.0-4502.gece5ed1a.el8.x86_64 2021-05-25T11:57:26.555 INFO:teuthology.orchestra.run.smithi181.stdout: python3-prettytable-0.7.2-14.el8.noarch 2021-05-25T11:57:26.555 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rados-2:17.0.0-4502.gece5ed1a.el8.x86_64 2021-05-25T11:57:26.555 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rbd-2:17.0.0-4502.gece5ed1a.el8.x86_64 2021-05-25T11:57:26.555 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rgw-2:17.0.0-4502.gece5ed1a.el8.x86_64 2021-05-25T11:57:26.555 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:57:26.556 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2021-05-25T11:57:26.800 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-test 2021-05-25T11:57:27.288 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 26/26 2021-05-25T11:57:27.289 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-base-2:17.0.0-4502.gece5ed1a.el8.x86_64 1/26 2021-05-25T11:57:27.289 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-common-2:17.0.0-4502.gece5ed1a.el8.x86_64 2/26 2021-05-25T11:57:27.289 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-radosgw-2:17.0.0-4502.gece5ed1a.el8.x86_64 3/26 2021-05-25T11:57:27.289 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-selinux-2:17.0.0-4502.gece5ed1a.el8.x86_64 4/26 2021-05-25T11:57:27.290 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libcephfs2-2:17.0.0-4502.gece5ed1a.el8.x86_64 5/26 2021-05-25T11:57:27.290 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libradosstriper1-2:17.0.0-4502.gece5ed1a.el8.x86_6 6/26 2021-05-25T11:57:27.290 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librgw2-2:17.0.0-4502.gece5ed1a.el8.x86_64 7/26 2021-05-25T11:57:27.290 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-ceph-argparse-2:17.0.0-4502.gece5ed1a.el8. 8/26 2021-05-25T11:57:27.291 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-ceph-common-2:17.0.0-4502.gece5ed1a.el8.x8 9/26 2021-05-25T11:57:27.291 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cephfs-2:17.0.0-4502.gece5ed1a.el8.x86_64 10/26 2021-05-25T11:57:27.291 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-rados-2:17.0.0-4502.gece5ed1a.el8.x86_64 11/26 2021-05-25T11:57:27.291 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-rbd-2:17.0.0-4502.gece5ed1a.el8.x86_64 12/26 2021-05-25T11:57:27.292 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-rgw-2:17.0.0-4502.gece5ed1a.el8.x86_64 13/26 2021-05-25T11:57:27.292 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 14/26 2021-05-25T11:57:27.292 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/26 2021-05-25T11:57:27.292 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/26 2021-05-25T11:57:27.293 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librabbitmq-0.9.0-2.el8.x86_64 17/26 2021-05-25T11:57:27.293 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 18/26 2021-05-25T11:57:27.293 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 19/26 2021-05-25T11:57:27.293 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : leveldb-1.22-1.el8.x86_64 20/26 2021-05-25T11:57:27.294 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 21/26 2021-05-25T11:57:27.294 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 22/26 2021-05-25T11:57:27.294 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librados2-2:17.0.0-4502.gece5ed1a.el8.x86_64 23/26 2021-05-25T11:57:27.294 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 24/26 2021-05-25T11:57:27.295 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librbd1-2:17.0.0-4502.gece5ed1a.el8.x86_64 25/26 2021-05-25T11:57:27.329 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:00:38 ago on Tue 25 May 2021 11:56:49 AM UTC. 2021-05-25T11:57:27.992 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2021-05-25T11:57:27.993 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2021-05-25T11:57:27.993 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2021-05-25T11:57:27.994 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2021-05-25T11:57:27.994 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2021-05-25T11:57:27.994 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-test x86_64 2:17.0.0-4502.gece5ed1a.el8 ceph 43 M 2021-05-25T11:57:27.994 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2021-05-25T11:57:27.995 INFO:teuthology.orchestra.run.smithi181.stdout: jq x86_64 1.5-12.el8 CentOS-AppStream 161 k 2021-05-25T11:57:27.995 INFO:teuthology.orchestra.run.smithi181.stdout: libcephsqlite x86_64 2:17.0.0-4502.gece5ed1a.el8 ceph 159 k 2021-05-25T11:57:27.995 INFO:teuthology.orchestra.run.smithi181.stdout: libxslt x86_64 1.1.32-5.el8 CentOS-Base 250 k 2021-05-25T11:57:27.995 INFO:teuthology.orchestra.run.smithi181.stdout: oniguruma x86_64 6.8.2-2.el8 CentOS-AppStream 187 k 2021-05-25T11:57:27.995 INFO:teuthology.orchestra.run.smithi181.stdout: socat x86_64 1.7.3.3-2.el8 CentOS-AppStream 302 k 2021-05-25T11:57:27.996 INFO:teuthology.orchestra.run.smithi181.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2021-05-25T11:57:27.996 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:57:27.996 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2021-05-25T11:57:27.997 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2021-05-25T11:57:27.997 INFO:teuthology.orchestra.run.smithi181.stdout:Install 7 Packages 2021-05-25T11:57:27.997 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:57:27.998 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 44 M 2021-05-25T11:57:27.998 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 245 M 2021-05-25T11:57:27.998 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2021-05-25T11:57:28.029 INFO:teuthology.orchestra.run.smithi181.stdout:(1/7): jq-1.5-12.el8.x86_64.rpm 4.7 MB/s | 161 kB 00:00 2021-05-25T11:57:28.032 INFO:teuthology.orchestra.run.smithi181.stdout:(2/7): oniguruma-6.8.2-2.el8.x86_64.rpm 63 MB/s | 187 kB 00:00 2021-05-25T11:57:28.036 INFO:teuthology.orchestra.run.smithi181.stdout:(3/7): socat-1.7.3.3-2.el8.x86_64.rpm 72 MB/s | 302 kB 00:00 2021-05-25T11:57:28.051 INFO:teuthology.orchestra.run.smithi181.stdout:(4/7): libxslt-1.1.32-5.el8.x86_64.rpm 17 MB/s | 250 kB 00:00 2021-05-25T11:57:28.141 INFO:teuthology.orchestra.run.smithi181.stdout:(5/7): xmlstarlet-1.6.1-11.el8.x86_64.rpm 771 kB/s | 69 kB 00:00 2021-05-25T11:57:28.235 INFO:teuthology.orchestra.run.smithi181.stdout:(6/7): libcephsqlite-17.0.0-4502.gece5ed1a.el8. 664 kB/s | 159 kB 00:00 2021-05-25T11:57:28.520 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 26/26 2021-05-25T11:57:28.521 INFO:teuthology.orchestra.run.smithi102.stdout:Installed products updated. 2021-05-25T11:57:28.770 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:57:28.771 INFO:teuthology.orchestra.run.smithi102.stdout:Upgraded: 2021-05-25T11:57:28.771 INFO:teuthology.orchestra.run.smithi102.stdout: librados2-2:17.0.0-4502.gece5ed1a.el8.x86_64 2021-05-25T11:57:28.771 INFO:teuthology.orchestra.run.smithi102.stdout: librbd1-2:17.0.0-4502.gece5ed1a.el8.x86_64 2021-05-25T11:57:28.771 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:57:28.772 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2021-05-25T11:57:28.772 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-base-2:17.0.0-4502.gece5ed1a.el8.x86_64 2021-05-25T11:57:28.772 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-common-2:17.0.0-4502.gece5ed1a.el8.x86_64 2021-05-25T11:57:28.772 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-radosgw-2:17.0.0-4502.gece5ed1a.el8.x86_64 2021-05-25T11:57:28.773 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-selinux-2:17.0.0-4502.gece5ed1a.el8.x86_64 2021-05-25T11:57:28.773 INFO:teuthology.orchestra.run.smithi102.stdout: fmt-6.2.1-1.el8.x86_64 2021-05-25T11:57:28.773 INFO:teuthology.orchestra.run.smithi102.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2021-05-25T11:57:28.773 INFO:teuthology.orchestra.run.smithi102.stdout: leveldb-1.22-1.el8.x86_64 2021-05-25T11:57:28.773 INFO:teuthology.orchestra.run.smithi102.stdout: libcephfs2-2:17.0.0-4502.gece5ed1a.el8.x86_64 2021-05-25T11:57:28.774 INFO:teuthology.orchestra.run.smithi102.stdout: liboath-2.6.2-3.el8.x86_64 2021-05-25T11:57:28.774 INFO:teuthology.orchestra.run.smithi102.stdout: librabbitmq-0.9.0-2.el8.x86_64 2021-05-25T11:57:28.774 INFO:teuthology.orchestra.run.smithi102.stdout: libradosstriper1-2:17.0.0-4502.gece5ed1a.el8.x86_64 2021-05-25T11:57:28.774 INFO:teuthology.orchestra.run.smithi102.stdout: librdkafka-0.11.4-1.el8.x86_64 2021-05-25T11:57:28.775 INFO:teuthology.orchestra.run.smithi102.stdout: librgw2-2:17.0.0-4502.gece5ed1a.el8.x86_64 2021-05-25T11:57:28.775 INFO:teuthology.orchestra.run.smithi102.stdout: libunwind-1.3.1-3.el8.x86_64 2021-05-25T11:57:28.775 INFO:teuthology.orchestra.run.smithi102.stdout: lttng-ust-2.8.1-11.el8.x86_64 2021-05-25T11:57:28.775 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ceph-argparse-2:17.0.0-4502.gece5ed1a.el8.x86_64 2021-05-25T11:57:28.775 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ceph-common-2:17.0.0-4502.gece5ed1a.el8.x86_64 2021-05-25T11:57:28.776 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cephfs-2:17.0.0-4502.gece5ed1a.el8.x86_64 2021-05-25T11:57:28.776 INFO:teuthology.orchestra.run.smithi102.stdout: python3-prettytable-0.7.2-14.el8.noarch 2021-05-25T11:57:28.776 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rados-2:17.0.0-4502.gece5ed1a.el8.x86_64 2021-05-25T11:57:28.776 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rbd-2:17.0.0-4502.gece5ed1a.el8.x86_64 2021-05-25T11:57:28.777 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rgw-2:17.0.0-4502.gece5ed1a.el8.x86_64 2021-05-25T11:57:28.777 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:57:28.777 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2021-05-25T11:57:28.994 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-test 2021-05-25T11:57:29.517 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:00:39 ago on Tue 25 May 2021 11:56:50 AM UTC. 2021-05-25T11:57:30.061 INFO:teuthology.orchestra.run.smithi181.stdout:(7/7): ceph-test-17.0.0-4502.gece5ed1a.el8.x86_ 21 MB/s | 43 MB 00:02 2021-05-25T11:57:30.064 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2021-05-25T11:57:30.065 INFO:teuthology.orchestra.run.smithi181.stdout:Total 21 MB/s | 44 MB 00:02 2021-05-25T11:57:30.069 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2021-05-25T11:57:30.093 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2021-05-25T11:57:30.093 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2021-05-25T11:57:30.196 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2021-05-25T11:57:30.197 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-05-25T11:57:30.197 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2021-05-25T11:57:30.197 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-05-25T11:57:30.198 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2021-05-25T11:57:30.198 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-test x86_64 2:17.0.0-4502.gece5ed1a.el8 ceph 43 M 2021-05-25T11:57:30.198 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2021-05-25T11:57:30.198 INFO:teuthology.orchestra.run.smithi102.stdout: jq x86_64 1.5-12.el8 CentOS-AppStream 161 k 2021-05-25T11:57:30.199 INFO:teuthology.orchestra.run.smithi102.stdout: libcephsqlite x86_64 2:17.0.0-4502.gece5ed1a.el8 ceph 159 k 2021-05-25T11:57:30.199 INFO:teuthology.orchestra.run.smithi102.stdout: libxslt x86_64 1.1.32-5.el8 CentOS-Base 250 k 2021-05-25T11:57:30.199 INFO:teuthology.orchestra.run.smithi102.stdout: oniguruma x86_64 6.8.2-2.el8 CentOS-AppStream 187 k 2021-05-25T11:57:30.199 INFO:teuthology.orchestra.run.smithi102.stdout: socat x86_64 1.7.3.3-2.el8 CentOS-AppStream 302 k 2021-05-25T11:57:30.200 INFO:teuthology.orchestra.run.smithi102.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2021-05-25T11:57:30.200 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:57:30.200 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2021-05-25T11:57:30.200 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-05-25T11:57:30.200 INFO:teuthology.orchestra.run.smithi102.stdout:Install 7 Packages 2021-05-25T11:57:30.201 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:57:30.201 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 44 M 2021-05-25T11:57:30.201 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 245 M 2021-05-25T11:57:30.202 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2021-05-25T11:57:30.233 INFO:teuthology.orchestra.run.smithi102.stdout:(1/7): jq-1.5-12.el8.x86_64.rpm 4.7 MB/s | 161 kB 00:00 2021-05-25T11:57:30.236 INFO:teuthology.orchestra.run.smithi102.stdout:(2/7): oniguruma-6.8.2-2.el8.x86_64.rpm 66 MB/s | 187 kB 00:00 2021-05-25T11:57:30.240 INFO:teuthology.orchestra.run.smithi102.stdout:(3/7): socat-1.7.3.3-2.el8.x86_64.rpm 73 MB/s | 302 kB 00:00 2021-05-25T11:57:30.253 INFO:teuthology.orchestra.run.smithi102.stdout:(4/7): libxslt-1.1.32-5.el8.x86_64.rpm 68 MB/s | 250 kB 00:00 2021-05-25T11:57:30.361 INFO:teuthology.orchestra.run.smithi102.stdout:(5/7): xmlstarlet-1.6.1-11.el8.x86_64.rpm 642 kB/s | 69 kB 00:00 2021-05-25T11:57:30.368 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2021-05-25T11:57:30.371 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2021-05-25T11:57:30.387 INFO:teuthology.orchestra.run.smithi102.stdout:(6/7): libcephsqlite-17.0.0-4502.gece5ed1a.el8. 846 kB/s | 159 kB 00:00 2021-05-25T11:57:30.885 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2021-05-25T11:57:31.058 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libxslt-1.1.32-5.el8.x86_64 1/7 2021-05-25T11:57:31.259 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 2/7 2021-05-25T11:57:31.467 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 3/7 2021-05-25T11:57:31.639 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 4/7 2021-05-25T11:57:31.725 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 4/7 2021-05-25T11:57:31.913 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : jq-1.5-12.el8.x86_64 5/7 2021-05-25T11:57:32.089 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libcephsqlite-2:17.0.0-4502.gece5ed1a.el8.x86_64 6/7 2021-05-25T11:57:32.185 INFO:teuthology.orchestra.run.smithi102.stdout:(7/7): ceph-test-17.0.0-4502.gece5ed1a.el8.x86_ 22 MB/s | 43 MB 00:01 2021-05-25T11:57:32.188 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2021-05-25T11:57:32.189 INFO:teuthology.orchestra.run.smithi102.stdout:Total 22 MB/s | 44 MB 00:01 2021-05-25T11:57:32.193 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2021-05-25T11:57:32.217 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2021-05-25T11:57:32.217 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2021-05-25T11:57:32.489 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2021-05-25T11:57:32.492 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2021-05-25T11:57:32.993 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2021-05-25T11:57:33.149 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libxslt-1.1.32-5.el8.x86_64 1/7 2021-05-25T11:57:33.983 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 2/7 2021-05-25T11:57:34.173 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 3/7 2021-05-25T11:57:34.369 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 4/7 2021-05-25T11:57:34.453 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 4/7 2021-05-25T11:57:34.634 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : jq-1.5-12.el8.x86_64 5/7 2021-05-25T11:57:34.819 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libcephsqlite-2:17.0.0-4502.gece5ed1a.el8.x86_64 6/7 2021-05-25T11:57:36.633 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libcephsqlite-2:17.0.0-4502.gece5ed1a.el8.x86_64 6/7 2021-05-25T11:57:36.845 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-test-2:17.0.0-4502.gece5ed1a.el8.x86_64 7/7 2021-05-25T11:57:37.073 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-test-2:17.0.0-4502.gece5ed1a.el8.x86_64 7/7 2021-05-25T11:57:37.074 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-test-2:17.0.0-4502.gece5ed1a.el8.x86_64 1/7 2021-05-25T11:57:37.074 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libcephsqlite-2:17.0.0-4502.gece5ed1a.el8.x86_64 2/7 2021-05-25T11:57:37.074 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : jq-1.5-12.el8.x86_64 3/7 2021-05-25T11:57:37.075 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/7 2021-05-25T11:57:37.075 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 5/7 2021-05-25T11:57:37.075 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libxslt-1.1.32-5.el8.x86_64 6/7 2021-05-25T11:57:37.475 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 7/7 2021-05-25T11:57:37.475 INFO:teuthology.orchestra.run.smithi181.stdout:Installed products updated. 2021-05-25T11:57:37.744 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:57:37.744 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2021-05-25T11:57:37.744 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-test-2:17.0.0-4502.gece5ed1a.el8.x86_64 jq-1.5-12.el8.x86_64 2021-05-25T11:57:37.745 INFO:teuthology.orchestra.run.smithi181.stdout: libcephsqlite-2:17.0.0-4502.gece5ed1a.el8.x86_64 libxslt-1.1.32-5.el8.x86_64 2021-05-25T11:57:37.745 INFO:teuthology.orchestra.run.smithi181.stdout: oniguruma-6.8.2-2.el8.x86_64 socat-1.7.3.3-2.el8.x86_64 2021-05-25T11:57:37.745 INFO:teuthology.orchestra.run.smithi181.stdout: xmlstarlet-1.6.1-11.el8.x86_64 2021-05-25T11:57:37.745 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:57:37.746 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2021-05-25T11:57:37.953 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph 2021-05-25T11:57:38.475 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:00:49 ago on Tue 25 May 2021 11:56:49 AM UTC. 2021-05-25T11:57:39.501 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: libcephsqlite-2:17.0.0-4502.gece5ed1a.el8.x86_64 6/7 2021-05-25T11:57:39.535 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2021-05-25T11:57:39.539 INFO:teuthology.orchestra.run.smithi181.stdout:======================================================================================= 2021-05-25T11:57:39.539 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2021-05-25T11:57:39.540 INFO:teuthology.orchestra.run.smithi181.stdout:======================================================================================= 2021-05-25T11:57:39.540 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2021-05-25T11:57:39.540 INFO:teuthology.orchestra.run.smithi181.stdout: ceph x86_64 2:17.0.0-4502.gece5ed1a.el8 ceph 6.4 k 2021-05-25T11:57:39.540 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2021-05-25T11:57:39.540 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mds x86_64 2:17.0.0-4502.gece5ed1a.el8 ceph 2.2 M 2021-05-25T11:57:39.541 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr x86_64 2:17.0.0-4502.gece5ed1a.el8 ceph 1.5 M 2021-05-25T11:57:39.541 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-modules-core noarch 2:17.0.0-4502.gece5ed1a.el8 ceph-noarch 204 k 2021-05-25T11:57:39.541 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mon x86_64 2:17.0.0-4502.gece5ed1a.el8 ceph 3.8 M 2021-05-25T11:57:39.541 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-osd x86_64 2:17.0.0-4502.gece5ed1a.el8 ceph 16 M 2021-05-25T11:57:39.542 INFO:teuthology.orchestra.run.smithi181.stdout: python3-asn1crypto noarch 0.24.0-3.el8 CentOS-Base 181 k 2021-05-25T11:57:39.542 INFO:teuthology.orchestra.run.smithi181.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2021-05-25T11:57:39.542 INFO:teuthology.orchestra.run.smithi181.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2021-05-25T11:57:39.542 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cffi x86_64 1.11.5-5.el8 CentOS-Base 237 k 2021-05-25T11:57:39.543 INFO:teuthology.orchestra.run.smithi181.stdout: python3-chardet noarch 3.0.4-7.el8 CentOS-Base 195 k 2021-05-25T11:57:39.543 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2021-05-25T11:57:39.543 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2021-05-25T11:57:39.543 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cryptography x86_64 2.3-3.el8 CentOS-Base 510 k 2021-05-25T11:57:39.544 INFO:teuthology.orchestra.run.smithi181.stdout: python3-idna noarch 2.5-5.el8 CentOS-Base 97 k 2021-05-25T11:57:39.544 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2021-05-25T11:57:39.544 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2021-05-25T11:57:39.544 INFO:teuthology.orchestra.run.smithi181.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2021-05-25T11:57:39.545 INFO:teuthology.orchestra.run.smithi181.stdout: python3-lxml x86_64 4.2.3-1.el8 CentOS-AppStream 1.5 M 2021-05-25T11:57:39.545 INFO:teuthology.orchestra.run.smithi181.stdout: python3-mako noarch 1.0.6-13.el8 CentOS-AppStream 157 k 2021-05-25T11:57:39.545 INFO:teuthology.orchestra.run.smithi181.stdout: python3-markupsafe x86_64 0.23-19.el8 CentOS-AppStream 39 k 2021-05-25T11:57:39.545 INFO:teuthology.orchestra.run.smithi181.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2021-05-25T11:57:39.546 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2021-05-25T11:57:39.546 INFO:teuthology.orchestra.run.smithi181.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2021-05-25T11:57:39.546 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyOpenSSL noarch 18.0.0-1.el8 CentOS-AppStream 103 k 2021-05-25T11:57:39.546 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pycparser noarch 2.14-14.el8 CentOS-Base 109 k 2021-05-25T11:57:39.547 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pysocks noarch 1.6.8-3.el8 CentOS-Base 34 k 2021-05-25T11:57:39.547 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pytz noarch 2017.2-9.el8 CentOS-AppStream 54 k 2021-05-25T11:57:39.547 INFO:teuthology.orchestra.run.smithi181.stdout: python3-requests noarch 2.20.0-2.1.el8_1 CentOS-Base 123 k 2021-05-25T11:57:39.547 INFO:teuthology.orchestra.run.smithi181.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2021-05-25T11:57:39.548 INFO:teuthology.orchestra.run.smithi181.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2021-05-25T11:57:39.548 INFO:teuthology.orchestra.run.smithi181.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2021-05-25T11:57:39.548 INFO:teuthology.orchestra.run.smithi181.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2021-05-25T11:57:39.548 INFO:teuthology.orchestra.run.smithi181.stdout: python3-urllib3 noarch 1.24.2-4.el8 CentOS-Base 176 k 2021-05-25T11:57:39.549 INFO:teuthology.orchestra.run.smithi181.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2021-05-25T11:57:39.549 INFO:teuthology.orchestra.run.smithi181.stdout: python3-webencodings noarch 0.5.1-6.el8 CentOS-AppStream 27 k 2021-05-25T11:57:39.549 INFO:teuthology.orchestra.run.smithi181.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2021-05-25T11:57:39.549 INFO:teuthology.orchestra.run.smithi181.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2021-05-25T11:57:39.549 INFO:teuthology.orchestra.run.smithi181.stdout: python3-werkzeug noarch 0.12.2-4.el8 CentOS-AppStream 457 k 2021-05-25T11:57:39.550 INFO:teuthology.orchestra.run.smithi181.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2021-05-25T11:57:39.550 INFO:teuthology.orchestra.run.smithi181.stdout:Installing weak dependencies: 2021-05-25T11:57:39.550 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cssselect noarch 0.9.2-10.el8 epel 40 k 2021-05-25T11:57:39.550 INFO:teuthology.orchestra.run.smithi181.stdout: python3-html5lib noarch 1:0.999999999-6.el8 CentOS-AppStream 214 k 2021-05-25T11:57:39.551 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:57:39.551 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2021-05-25T11:57:39.551 INFO:teuthology.orchestra.run.smithi181.stdout:======================================================================================= 2021-05-25T11:57:39.551 INFO:teuthology.orchestra.run.smithi181.stdout:Install 42 Packages 2021-05-25T11:57:39.552 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:57:39.553 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 30 M 2021-05-25T11:57:39.553 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 126 M 2021-05-25T11:57:39.553 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2021-05-25T11:57:39.657 INFO:teuthology.orchestra.run.smithi181.stdout:(1/42): ceph-17.0.0-4502.gece5ed1a.el8.x86_64.r 58 kB/s | 6.4 kB 00:00 2021-05-25T11:57:39.834 INFO:teuthology.orchestra.run.smithi181.stdout:(2/42): ceph-mgr-17.0.0-4502.gece5ed1a.el8.x86_ 5.1 MB/s | 1.5 MB 00:00 2021-05-25T11:57:39.874 INFO:teuthology.orchestra.run.smithi181.stdout:(3/42): ceph-mds-17.0.0-4502.gece5ed1a.el8.x86_ 6.7 MB/s | 2.2 MB 00:00 2021-05-25T11:57:39.924 INFO:teuthology.orchestra.run.smithi181.stdout:(4/42): ceph-mgr-modules-core-17.0.0-4502.gece5 4.0 MB/s | 204 kB 00:00 2021-05-25T11:57:39.959 INFO:teuthology.orchestra.run.smithi181.stdout:(5/42): python3-html5lib-0.999999999-6.el8.noar 6.0 MB/s | 214 kB 00:00 2021-05-25T11:57:39.978 INFO:teuthology.orchestra.run.smithi181.stdout:(6/42): python3-lxml-4.2.3-1.el8.x86_64.rpm 82 MB/s | 1.5 MB 00:00 2021-05-25T11:57:39.981 INFO:teuthology.orchestra.run.smithi181.stdout:(7/42): python3-mako-1.0.6-13.el8.noarch.rpm 58 MB/s | 157 kB 00:00 2021-05-25T11:57:39.982 INFO:teuthology.orchestra.run.smithi181.stdout:(8/42): python3-markupsafe-0.23-19.el8.x86_64.r 30 MB/s | 39 kB 00:00 2021-05-25T11:57:39.984 INFO:teuthology.orchestra.run.smithi181.stdout:(9/42): python3-pyOpenSSL-18.0.0-1.el8.noarch.r 50 MB/s | 103 kB 00:00 2021-05-25T11:57:39.986 INFO:teuthology.orchestra.run.smithi181.stdout:(10/42): python3-pytz-2017.2-9.el8.noarch.rpm 37 MB/s | 54 kB 00:00 2021-05-25T11:57:39.987 INFO:teuthology.orchestra.run.smithi181.stdout:(11/42): python3-webencodings-0.5.1-6.el8.noarc 23 MB/s | 27 kB 00:00 2021-05-25T11:57:39.993 INFO:teuthology.orchestra.run.smithi181.stdout:(12/42): python3-werkzeug-0.12.2-4.el8.noarch.r 75 MB/s | 457 kB 00:00 2021-05-25T11:57:39.997 INFO:teuthology.orchestra.run.smithi181.stdout:(13/42): python3-asn1crypto-0.24.0-3.el8.noarch 49 MB/s | 181 kB 00:00 2021-05-25T11:57:40.001 INFO:teuthology.orchestra.run.smithi181.stdout:(14/42): python3-cffi-1.11.5-5.el8.x86_64.rpm 65 MB/s | 237 kB 00:00 2021-05-25T11:57:40.004 INFO:teuthology.orchestra.run.smithi181.stdout:(15/42): python3-chardet-3.0.4-7.el8.noarch.rpm 64 MB/s | 195 kB 00:00 2021-05-25T11:57:40.011 INFO:teuthology.orchestra.run.smithi181.stdout:(16/42): python3-cryptography-2.3-3.el8.x86_64. 77 MB/s | 510 kB 00:00 2021-05-25T11:57:40.013 INFO:teuthology.orchestra.run.smithi181.stdout:(17/42): python3-idna-2.5-5.el8.noarch.rpm 44 MB/s | 97 kB 00:00 2021-05-25T11:57:40.015 INFO:teuthology.orchestra.run.smithi181.stdout:(18/42): python3-pycparser-2.14-14.el8.noarch.r 51 MB/s | 109 kB 00:00 2021-05-25T11:57:40.017 INFO:teuthology.orchestra.run.smithi181.stdout:(19/42): python3-pysocks-1.6.8-3.el8.noarch.rpm 22 MB/s | 34 kB 00:00 2021-05-25T11:57:40.019 INFO:teuthology.orchestra.run.smithi181.stdout:(20/42): python3-requests-2.20.0-2.1.el8_1.noar 53 MB/s | 123 kB 00:00 2021-05-25T11:57:40.022 INFO:teuthology.orchestra.run.smithi181.stdout:(21/42): python3-urllib3-1.24.2-4.el8.noarch.rp 61 MB/s | 176 kB 00:00 2021-05-25T11:57:40.130 INFO:teuthology.orchestra.run.smithi181.stdout:(22/42): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 407 kB/s | 44 kB 00:00 2021-05-25T11:57:40.173 INFO:teuthology.orchestra.run.smithi181.stdout:(23/42): ceph-mon-17.0.0-4502.gece5ed1a.el8.x86 7.4 MB/s | 3.8 MB 00:00 2021-05-25T11:57:40.212 INFO:teuthology.orchestra.run.smithi181.stdout:(24/42): python3-beautifulsoup4-4.6.3-2.el8.1.n 2.3 MB/s | 185 kB 00:00 2021-05-25T11:57:40.265 INFO:teuthology.orchestra.run.smithi181.stdout:(25/42): python3-cherrypy-18.4.0-1.el8.noarch.r 6.8 MB/s | 384 kB 00:00 2021-05-25T11:57:40.283 INFO:teuthology.orchestra.run.smithi181.stdout:(26/42): python3-cheroot-8.5.2-1.el8.noarch.rpm 1.5 MB/s | 173 kB 00:00 2021-05-25T11:57:40.293 INFO:teuthology.orchestra.run.smithi181.stdout:(27/42): python3-cssselect-0.9.2-10.el8.noarch. 1.5 MB/s | 40 kB 00:00 2021-05-25T11:57:40.306 INFO:teuthology.orchestra.run.smithi181.stdout:(28/42): python3-jaraco-6.2-6.el8.noarch.rpm 495 kB/s | 11 kB 00:00 2021-05-25T11:57:40.319 INFO:teuthology.orchestra.run.smithi181.stdout:(29/42): python3-jaraco-functools-2.0-4.el8.noa 655 kB/s | 18 kB 00:00 2021-05-25T11:57:40.329 INFO:teuthology.orchestra.run.smithi181.stdout:(30/42): python3-logutils-0.3.5-11.el8.noarch.r 2.1 MB/s | 49 kB 00:00 2021-05-25T11:57:40.347 INFO:teuthology.orchestra.run.smithi181.stdout:(31/42): python3-more-itertools-7.2.0-3.el8.noa 2.1 MB/s | 59 kB 00:00 2021-05-25T11:57:40.374 INFO:teuthology.orchestra.run.smithi181.stdout:(32/42): python3-pecan-1.3.2-9.el8.noarch.rpm 6.2 MB/s | 283 kB 00:00 2021-05-25T11:57:40.375 INFO:teuthology.orchestra.run.smithi181.stdout:(33/42): python3-portend-2.6-1.el8.noarch.rpm 575 kB/s | 16 kB 00:00 2021-05-25T11:57:40.396 INFO:teuthology.orchestra.run.smithi181.stdout:(34/42): python3-simplegeneric-0.8.1-17.el8.noa 844 kB/s | 19 kB 00:00 2021-05-25T11:57:40.401 INFO:teuthology.orchestra.run.smithi181.stdout:(35/42): python3-singledispatch-3.4.0.3-18.el8. 896 kB/s | 24 kB 00:00 2021-05-25T11:57:40.419 INFO:teuthology.orchestra.run.smithi181.stdout:(36/42): python3-tempora-1.14.1-5.el8.noarch.rp 1.3 MB/s | 29 kB 00:00 2021-05-25T11:57:40.428 INFO:teuthology.orchestra.run.smithi181.stdout:(37/42): python3-trustme-0.6.0-4.el8.noarch.rpm 1.0 MB/s | 27 kB 00:00 2021-05-25T11:57:40.443 INFO:teuthology.orchestra.run.smithi181.stdout:(38/42): python3-waitress-1.2.1-2.el8.1.noarch. 9.7 MB/s | 241 kB 00:00 2021-05-25T11:57:40.466 INFO:teuthology.orchestra.run.smithi181.stdout:(39/42): python3-webtest-2.0.33-1.el8.noarch.rp 3.7 MB/s | 85 kB 00:00 2021-05-25T11:57:40.482 INFO:teuthology.orchestra.run.smithi181.stdout:(40/42): python3-webob-1.8.5-1.el8.1.noarch.rpm 4.6 MB/s | 251 kB 00:00 2021-05-25T11:57:40.489 INFO:teuthology.orchestra.run.smithi181.stdout:(41/42): python3-zc-lockfile-2.0-2.el8.noarch.r 1.0 MB/s | 23 kB 00:00 2021-05-25T11:57:40.649 INFO:teuthology.orchestra.run.smithi181.stdout:(42/42): ceph-osd-17.0.0-4502.gece5ed1a.el8.x86 20 MB/s | 16 MB 00:00 2021-05-25T11:57:40.653 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2021-05-25T11:57:40.653 INFO:teuthology.orchestra.run.smithi181.stdout:Total 27 MB/s | 30 MB 00:01 2021-05-25T11:57:40.667 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2021-05-25T11:57:40.708 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2021-05-25T11:57:40.709 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2021-05-25T11:57:41.040 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2021-05-25T11:57:41.043 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2021-05-25T11:57:41.050 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-test-2:17.0.0-4502.gece5ed1a.el8.x86_64 7/7 2021-05-25T11:57:41.477 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-test-2:17.0.0-4502.gece5ed1a.el8.x86_64 7/7 2021-05-25T11:57:41.478 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-test-2:17.0.0-4502.gece5ed1a.el8.x86_64 1/7 2021-05-25T11:57:41.478 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libcephsqlite-2:17.0.0-4502.gece5ed1a.el8.x86_64 2/7 2021-05-25T11:57:41.478 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : jq-1.5-12.el8.x86_64 3/7 2021-05-25T11:57:41.478 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/7 2021-05-25T11:57:41.479 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 5/7 2021-05-25T11:57:41.479 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libxslt-1.1.32-5.el8.x86_64 6/7 2021-05-25T11:57:41.848 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 7/7 2021-05-25T11:57:41.849 INFO:teuthology.orchestra.run.smithi102.stdout:Installed products updated. 2021-05-25T11:57:42.137 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:57:42.138 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2021-05-25T11:57:42.138 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-test-2:17.0.0-4502.gece5ed1a.el8.x86_64 jq-1.5-12.el8.x86_64 2021-05-25T11:57:42.138 INFO:teuthology.orchestra.run.smithi102.stdout: libcephsqlite-2:17.0.0-4502.gece5ed1a.el8.x86_64 libxslt-1.1.32-5.el8.x86_64 2021-05-25T11:57:42.138 INFO:teuthology.orchestra.run.smithi102.stdout: oniguruma-6.8.2-2.el8.x86_64 socat-1.7.3.3-2.el8.x86_64 2021-05-25T11:57:42.139 INFO:teuthology.orchestra.run.smithi102.stdout: xmlstarlet-1.6.1-11.el8.x86_64 2021-05-25T11:57:42.139 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:57:42.139 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2021-05-25T11:57:42.304 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph 2021-05-25T11:57:42.627 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2021-05-25T11:57:42.799 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/42 2021-05-25T11:57:42.828 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:00:52 ago on Tue 25 May 2021 11:56:50 AM UTC. 2021-05-25T11:57:43.013 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-idna-2.5-5.el8.noarch 2/42 2021-05-25T11:57:43.242 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 3/42 2021-05-25T11:57:43.461 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 4/42 2021-05-25T11:57:43.507 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2021-05-25T11:57:43.511 INFO:teuthology.orchestra.run.smithi102.stdout:======================================================================================= 2021-05-25T11:57:43.511 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2021-05-25T11:57:43.511 INFO:teuthology.orchestra.run.smithi102.stdout:======================================================================================= 2021-05-25T11:57:43.512 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2021-05-25T11:57:43.512 INFO:teuthology.orchestra.run.smithi102.stdout: ceph x86_64 2:17.0.0-4502.gece5ed1a.el8 ceph 6.4 k 2021-05-25T11:57:43.512 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2021-05-25T11:57:43.512 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mds x86_64 2:17.0.0-4502.gece5ed1a.el8 ceph 2.2 M 2021-05-25T11:57:43.513 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr x86_64 2:17.0.0-4502.gece5ed1a.el8 ceph 1.5 M 2021-05-25T11:57:43.513 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-modules-core noarch 2:17.0.0-4502.gece5ed1a.el8 ceph-noarch 204 k 2021-05-25T11:57:43.513 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mon x86_64 2:17.0.0-4502.gece5ed1a.el8 ceph 3.8 M 2021-05-25T11:57:43.513 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-osd x86_64 2:17.0.0-4502.gece5ed1a.el8 ceph 16 M 2021-05-25T11:57:43.513 INFO:teuthology.orchestra.run.smithi102.stdout: python3-asn1crypto noarch 0.24.0-3.el8 CentOS-Base 181 k 2021-05-25T11:57:43.514 INFO:teuthology.orchestra.run.smithi102.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2021-05-25T11:57:43.514 INFO:teuthology.orchestra.run.smithi102.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2021-05-25T11:57:43.514 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cffi x86_64 1.11.5-5.el8 CentOS-Base 237 k 2021-05-25T11:57:43.514 INFO:teuthology.orchestra.run.smithi102.stdout: python3-chardet noarch 3.0.4-7.el8 CentOS-Base 195 k 2021-05-25T11:57:43.515 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2021-05-25T11:57:43.515 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2021-05-25T11:57:43.515 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cryptography x86_64 2.3-3.el8 CentOS-Base 510 k 2021-05-25T11:57:43.515 INFO:teuthology.orchestra.run.smithi102.stdout: python3-idna noarch 2.5-5.el8 CentOS-Base 97 k 2021-05-25T11:57:43.515 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2021-05-25T11:57:43.516 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2021-05-25T11:57:43.516 INFO:teuthology.orchestra.run.smithi102.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2021-05-25T11:57:43.516 INFO:teuthology.orchestra.run.smithi102.stdout: python3-lxml x86_64 4.2.3-1.el8 CentOS-AppStream 1.5 M 2021-05-25T11:57:43.516 INFO:teuthology.orchestra.run.smithi102.stdout: python3-mako noarch 1.0.6-13.el8 CentOS-AppStream 157 k 2021-05-25T11:57:43.517 INFO:teuthology.orchestra.run.smithi102.stdout: python3-markupsafe x86_64 0.23-19.el8 CentOS-AppStream 39 k 2021-05-25T11:57:43.517 INFO:teuthology.orchestra.run.smithi102.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2021-05-25T11:57:43.517 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2021-05-25T11:57:43.517 INFO:teuthology.orchestra.run.smithi102.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2021-05-25T11:57:43.517 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyOpenSSL noarch 18.0.0-1.el8 CentOS-AppStream 103 k 2021-05-25T11:57:43.518 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pycparser noarch 2.14-14.el8 CentOS-Base 109 k 2021-05-25T11:57:43.518 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pysocks noarch 1.6.8-3.el8 CentOS-Base 34 k 2021-05-25T11:57:43.518 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pytz noarch 2017.2-9.el8 CentOS-AppStream 54 k 2021-05-25T11:57:43.518 INFO:teuthology.orchestra.run.smithi102.stdout: python3-requests noarch 2.20.0-2.1.el8_1 CentOS-Base 123 k 2021-05-25T11:57:43.519 INFO:teuthology.orchestra.run.smithi102.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2021-05-25T11:57:43.519 INFO:teuthology.orchestra.run.smithi102.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2021-05-25T11:57:43.519 INFO:teuthology.orchestra.run.smithi102.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2021-05-25T11:57:43.519 INFO:teuthology.orchestra.run.smithi102.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2021-05-25T11:57:43.519 INFO:teuthology.orchestra.run.smithi102.stdout: python3-urllib3 noarch 1.24.2-4.el8 CentOS-Base 176 k 2021-05-25T11:57:43.520 INFO:teuthology.orchestra.run.smithi102.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2021-05-25T11:57:43.520 INFO:teuthology.orchestra.run.smithi102.stdout: python3-webencodings noarch 0.5.1-6.el8 CentOS-AppStream 27 k 2021-05-25T11:57:43.520 INFO:teuthology.orchestra.run.smithi102.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2021-05-25T11:57:43.520 INFO:teuthology.orchestra.run.smithi102.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2021-05-25T11:57:43.521 INFO:teuthology.orchestra.run.smithi102.stdout: python3-werkzeug noarch 0.12.2-4.el8 CentOS-AppStream 457 k 2021-05-25T11:57:43.521 INFO:teuthology.orchestra.run.smithi102.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2021-05-25T11:57:43.521 INFO:teuthology.orchestra.run.smithi102.stdout:Installing weak dependencies: 2021-05-25T11:57:43.521 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cssselect noarch 0.9.2-10.el8 epel 40 k 2021-05-25T11:57:43.522 INFO:teuthology.orchestra.run.smithi102.stdout: python3-html5lib noarch 1:0.999999999-6.el8 CentOS-AppStream 214 k 2021-05-25T11:57:43.522 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:57:43.522 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2021-05-25T11:57:43.522 INFO:teuthology.orchestra.run.smithi102.stdout:======================================================================================= 2021-05-25T11:57:43.522 INFO:teuthology.orchestra.run.smithi102.stdout:Install 42 Packages 2021-05-25T11:57:43.523 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:57:43.523 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 30 M 2021-05-25T11:57:43.524 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 126 M 2021-05-25T11:57:43.524 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2021-05-25T11:57:43.616 INFO:teuthology.orchestra.run.smithi102.stdout:(1/42): ceph-17.0.0-4502.gece5ed1a.el8.x86_64.r 65 kB/s | 6.4 kB 00:00 2021-05-25T11:57:43.661 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 5/42 2021-05-25T11:57:43.808 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 6/42 2021-05-25T11:57:43.822 INFO:teuthology.orchestra.run.smithi102.stdout:(2/42): ceph-mgr-17.0.0-4502.gece5ed1a.el8.x86_ 4.8 MB/s | 1.5 MB 00:00 2021-05-25T11:57:43.868 INFO:teuthology.orchestra.run.smithi102.stdout:(3/42): ceph-mds-17.0.0-4502.gece5ed1a.el8.x86_ 6.3 MB/s | 2.2 MB 00:00 2021-05-25T11:57:43.929 INFO:teuthology.orchestra.run.smithi102.stdout:(4/42): ceph-mgr-modules-core-17.0.0-4502.gece5 3.3 MB/s | 204 kB 00:00 2021-05-25T11:57:43.962 INFO:teuthology.orchestra.run.smithi102.stdout:(5/42): python3-html5lib-0.999999999-6.el8.noar 6.4 MB/s | 214 kB 00:00 2021-05-25T11:57:43.981 INFO:teuthology.orchestra.run.smithi102.stdout:(6/42): python3-lxml-4.2.3-1.el8.x86_64.rpm 81 MB/s | 1.5 MB 00:00 2021-05-25T11:57:43.983 INFO:teuthology.orchestra.run.smithi102.stdout:(7/42): python3-mako-1.0.6-13.el8.noarch.rpm 62 MB/s | 157 kB 00:00 2021-05-25T11:57:43.985 INFO:teuthology.orchestra.run.smithi102.stdout:(8/42): python3-markupsafe-0.23-19.el8.x86_64.r 34 MB/s | 39 kB 00:00 2021-05-25T11:57:43.987 INFO:teuthology.orchestra.run.smithi102.stdout:(9/42): python3-pyOpenSSL-18.0.0-1.el8.noarch.r 52 MB/s | 103 kB 00:00 2021-05-25T11:57:43.988 INFO:teuthology.orchestra.run.smithi102.stdout:(10/42): python3-pytz-2017.2-9.el8.noarch.rpm 41 MB/s | 54 kB 00:00 2021-05-25T11:57:43.989 INFO:teuthology.orchestra.run.smithi102.stdout:(11/42): python3-webencodings-0.5.1-6.el8.noarc 26 MB/s | 27 kB 00:00 2021-05-25T11:57:43.995 INFO:teuthology.orchestra.run.smithi102.stdout:(12/42): python3-werkzeug-0.12.2-4.el8.noarch.r 77 MB/s | 457 kB 00:00 2021-05-25T11:57:43.998 INFO:teuthology.orchestra.run.smithi102.stdout:(13/42): python3-asn1crypto-0.24.0-3.el8.noarch 66 MB/s | 181 kB 00:00 2021-05-25T11:57:44.004 INFO:teuthology.orchestra.run.smithi102.stdout:(14/42): python3-cffi-1.11.5-5.el8.x86_64.rpm 68 MB/s | 237 kB 00:00 2021-05-25T11:57:44.005 INFO:teuthology.orchestra.run.smithi102.stdout:(15/42): python3-chardet-3.0.4-7.el8.noarch.rpm 66 MB/s | 195 kB 00:00 2021-05-25T11:57:44.007 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 7/42 2021-05-25T11:57:44.012 INFO:teuthology.orchestra.run.smithi102.stdout:(16/42): python3-cryptography-2.3-3.el8.x86_64. 77 MB/s | 510 kB 00:00 2021-05-25T11:57:44.014 INFO:teuthology.orchestra.run.smithi102.stdout:(17/42): python3-idna-2.5-5.el8.noarch.rpm 52 MB/s | 97 kB 00:00 2021-05-25T11:57:44.016 INFO:teuthology.orchestra.run.smithi102.stdout:(18/42): python3-pycparser-2.14-14.el8.noarch.r 56 MB/s | 109 kB 00:00 2021-05-25T11:57:44.017 INFO:teuthology.orchestra.run.smithi102.stdout:(19/42): python3-pysocks-1.6.8-3.el8.noarch.rpm 31 MB/s | 34 kB 00:00 2021-05-25T11:57:44.019 INFO:teuthology.orchestra.run.smithi102.stdout:(20/42): python3-requests-2.20.0-2.1.el8_1.noar 59 MB/s | 123 kB 00:00 2021-05-25T11:57:44.022 INFO:teuthology.orchestra.run.smithi102.stdout:(21/42): python3-urllib3-1.24.2-4.el8.noarch.rp 63 MB/s | 176 kB 00:00 2021-05-25T11:57:44.046 INFO:teuthology.orchestra.run.smithi102.stdout:(22/42): ceph-mon-17.0.0-4502.gece5ed1a.el8.x86 8.9 MB/s | 3.8 MB 00:00 2021-05-25T11:57:44.190 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 8/42 2021-05-25T11:57:44.256 INFO:teuthology.orchestra.run.smithi102.stdout:(23/42): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 187 kB/s | 44 kB 00:00 2021-05-25T11:57:44.316 INFO:teuthology.orchestra.run.smithi102.stdout:(24/42): python3-beautifulsoup4-4.6.3-2.el8.1.n 688 kB/s | 185 kB 00:00 2021-05-25T11:57:44.336 INFO:teuthology.orchestra.run.smithi102.stdout:(25/42): python3-cheroot-8.5.2-1.el8.noarch.rpm 2.1 MB/s | 173 kB 00:00 2021-05-25T11:57:44.352 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 9/42 2021-05-25T11:57:44.363 INFO:teuthology.orchestra.run.smithi102.stdout:(26/42): python3-cssselect-0.9.2-10.el8.noarch. 1.4 MB/s | 40 kB 00:00 2021-05-25T11:57:44.372 INFO:teuthology.orchestra.run.smithi102.stdout:(27/42): python3-cherrypy-18.4.0-1.el8.noarch.r 6.7 MB/s | 384 kB 00:00 2021-05-25T11:57:44.390 INFO:teuthology.orchestra.run.smithi102.stdout:(28/42): python3-jaraco-6.2-6.el8.noarch.rpm 416 kB/s | 11 kB 00:00 2021-05-25T11:57:44.399 INFO:teuthology.orchestra.run.smithi102.stdout:(29/42): python3-jaraco-functools-2.0-4.el8.noa 658 kB/s | 18 kB 00:00 2021-05-25T11:57:44.418 INFO:teuthology.orchestra.run.smithi102.stdout:(30/42): python3-logutils-0.3.5-11.el8.noarch.r 1.7 MB/s | 49 kB 00:00 2021-05-25T11:57:44.427 INFO:teuthology.orchestra.run.smithi102.stdout:(31/42): python3-more-itertools-7.2.0-3.el8.noa 2.1 MB/s | 59 kB 00:00 2021-05-25T11:57:44.454 INFO:teuthology.orchestra.run.smithi102.stdout:(32/42): python3-portend-2.6-1.el8.noarch.rpm 571 kB/s | 16 kB 00:00 2021-05-25T11:57:44.472 INFO:teuthology.orchestra.run.smithi102.stdout:(33/42): python3-pecan-1.3.2-9.el8.noarch.rpm 5.1 MB/s | 283 kB 00:00 2021-05-25T11:57:44.481 INFO:teuthology.orchestra.run.smithi102.stdout:(34/42): python3-simplegeneric-0.8.1-17.el8.noa 690 kB/s | 19 kB 00:00 2021-05-25T11:57:44.491 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 10/42 2021-05-25T11:57:44.499 INFO:teuthology.orchestra.run.smithi102.stdout:(35/42): python3-singledispatch-3.4.0.3-18.el8. 892 kB/s | 24 kB 00:00 2021-05-25T11:57:44.509 INFO:teuthology.orchestra.run.smithi102.stdout:(36/42): python3-tempora-1.14.1-5.el8.noarch.rp 1.0 MB/s | 29 kB 00:00 2021-05-25T11:57:44.613 INFO:teuthology.orchestra.run.smithi102.stdout:(37/42): ceph-osd-17.0.0-4502.gece5ed1a.el8.x86 20 MB/s | 16 MB 00:00 2021-05-25T11:57:44.613 INFO:teuthology.orchestra.run.smithi102.stdout:(38/42): python3-trustme-0.6.0-4.el8.noarch.rpm 241 kB/s | 27 kB 00:00 2021-05-25T11:57:44.615 INFO:teuthology.orchestra.run.smithi102.stdout:(39/42): python3-waitress-1.2.1-2.el8.1.noarch. 2.2 MB/s | 241 kB 00:00 2021-05-25T11:57:44.641 INFO:teuthology.orchestra.run.smithi102.stdout:(40/42): python3-webtest-2.0.33-1.el8.noarch.rp 3.0 MB/s | 85 kB 00:00 2021-05-25T11:57:44.642 INFO:teuthology.orchestra.run.smithi102.stdout:(41/42): python3-zc-lockfile-2.0-2.el8.noarch.r 854 kB/s | 23 kB 00:00 2021-05-25T11:57:44.662 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cssselect-0.9.2-10.el8.noarch 11/42 2021-05-25T11:57:44.746 INFO:teuthology.orchestra.run.smithi102.stdout:(42/42): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.8 MB/s | 251 kB 00:00 2021-05-25T11:57:44.749 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2021-05-25T11:57:44.750 INFO:teuthology.orchestra.run.smithi102.stdout:Total 24 MB/s | 30 MB 00:01 2021-05-25T11:57:44.763 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2021-05-25T11:57:44.805 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2021-05-25T11:57:44.806 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2021-05-25T11:57:44.906 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pysocks-1.6.8-3.el8.noarch 12/42 2021-05-25T11:57:45.097 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2021-05-25T11:57:45.100 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2021-05-25T11:57:45.165 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-urllib3-1.24.2-4.el8.noarch 13/42 2021-05-25T11:57:45.396 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 14/42 2021-05-25T11:57:45.624 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 15/42 2021-05-25T11:57:45.906 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 16/42 2021-05-25T11:57:46.190 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-chardet-3.0.4-7.el8.noarch 17/42 2021-05-25T11:57:46.437 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-requests-2.20.0-2.1.el8_1.noarch 18/42 2021-05-25T11:57:46.627 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2021-05-25T11:57:46.778 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-asn1crypto-0.24.0-3.el8.noarch 19/42 2021-05-25T11:57:46.804 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/42 2021-05-25T11:57:47.024 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cryptography-2.3-3.el8.x86_64 20/42 2021-05-25T11:57:47.051 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-idna-2.5-5.el8.noarch 2/42 2021-05-25T11:57:47.237 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pyOpenSSL-18.0.0-1.el8.noarch 21/42 2021-05-25T11:57:47.242 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 3/42 2021-05-25T11:57:47.447 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 22/42 2021-05-25T11:57:47.491 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 4/42 2021-05-25T11:57:47.707 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 5/42 2021-05-25T11:57:47.749 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 23/42 2021-05-25T11:57:47.864 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 6/42 2021-05-25T11:57:47.973 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 24/42 2021-05-25T11:57:48.086 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 7/42 2021-05-25T11:57:48.215 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-webencodings-0.5.1-6.el8.noarch 25/42 2021-05-25T11:57:48.262 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 8/42 2021-05-25T11:57:48.423 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 9/42 2021-05-25T11:57:48.458 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-html5lib-1:0.999999999-6.el8.noarch 26/42 2021-05-25T11:57:48.578 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 10/42 2021-05-25T11:57:48.758 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cssselect-0.9.2-10.el8.noarch 11/42 2021-05-25T11:57:48.813 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 27/42 2021-05-25T11:57:48.991 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pysocks-1.6.8-3.el8.noarch 12/42 2021-05-25T11:57:49.050 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-lxml-4.2.3-1.el8.x86_64 28/42 2021-05-25T11:57:49.259 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-urllib3-1.24.2-4.el8.noarch 13/42 2021-05-25T11:57:49.267 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 29/42 2021-05-25T11:57:49.456 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 30/42 2021-05-25T11:57:49.508 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 14/42 2021-05-25T11:57:49.675 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 31/42 2021-05-25T11:57:49.719 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 15/42 2021-05-25T11:57:49.956 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-portend-2.6-1.el8.noarch 32/42 2021-05-25T11:57:49.999 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 16/42 2021-05-25T11:57:50.232 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 33/42 2021-05-25T11:57:50.289 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-chardet-3.0.4-7.el8.noarch 17/42 2021-05-25T11:57:50.460 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 34/42 2021-05-25T11:57:50.506 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-requests-2.20.0-2.1.el8_1.noarch 18/42 2021-05-25T11:57:50.805 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 35/42 2021-05-25T11:57:50.835 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-asn1crypto-0.24.0-3.el8.noarch 19/42 2021-05-25T11:57:51.095 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cryptography-2.3-3.el8.x86_64 20/42 2021-05-25T11:57:51.160 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 36/42 2021-05-25T11:57:51.308 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pyOpenSSL-18.0.0-1.el8.noarch 21/42 2021-05-25T11:57:51.510 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 22/42 2021-05-25T11:57:51.559 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-modules-core-2:17.0.0-4502.gece5ed1a.el8. 37/42 2021-05-25T11:57:51.820 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 23/42 2021-05-25T11:57:51.840 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-2:17.0.0-4502.gece5ed1a.el8.x86_64 38/42 2021-05-25T11:57:52.070 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 24/42 2021-05-25T11:57:52.314 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-webencodings-0.5.1-6.el8.noarch 25/42 2021-05-25T11:57:52.554 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-html5lib-1:0.999999999-6.el8.noarch 26/42 2021-05-25T11:57:52.955 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 27/42 2021-05-25T11:57:53.199 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-lxml-4.2.3-1.el8.x86_64 28/42 2021-05-25T11:57:53.322 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-2:17.0.0-4502.gece5ed1a.el8.x86_64 38/42 2021-05-25T11:57:53.389 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 29/42 2021-05-25T11:57:53.586 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 30/42 2021-05-25T11:57:53.589 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-osd-2:17.0.0-4502.gece5ed1a.el8.x86_64 39/42 2021-05-25T11:57:53.822 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 31/42 2021-05-25T11:57:53.982 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-osd-2:17.0.0-4502.gece5ed1a.el8.x86_64 39/42 2021-05-25T11:57:54.102 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-portend-2.6-1.el8.noarch 32/42 2021-05-25T11:57:54.198 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mon-2:17.0.0-4502.gece5ed1a.el8.x86_64 40/42 2021-05-25T11:57:54.388 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 33/42 2021-05-25T11:57:54.418 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mon-2:17.0.0-4502.gece5ed1a.el8.x86_64 40/42 2021-05-25T11:57:54.627 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mds-2:17.0.0-4502.gece5ed1a.el8.x86_64 41/42 2021-05-25T11:57:54.639 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 34/42 2021-05-25T11:57:54.648 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mds-2:17.0.0-4502.gece5ed1a.el8.x86_64 41/42 2021-05-25T11:57:54.818 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-2:17.0.0-4502.gece5ed1a.el8.x86_64 42/42 2021-05-25T11:57:54.955 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 35/42 2021-05-25T11:57:55.202 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-2:17.0.0-4502.gece5ed1a.el8.x86_64 42/42 2021-05-25T11:57:55.203 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-2:17.0.0-4502.gece5ed1a.el8.x86_64 1/42 2021-05-25T11:57:55.203 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mds-2:17.0.0-4502.gece5ed1a.el8.x86_64 2/42 2021-05-25T11:57:55.203 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-2:17.0.0-4502.gece5ed1a.el8.x86_64 3/42 2021-05-25T11:57:55.203 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mon-2:17.0.0-4502.gece5ed1a.el8.x86_64 4/42 2021-05-25T11:57:55.204 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-osd-2:17.0.0-4502.gece5ed1a.el8.x86_64 5/42 2021-05-25T11:57:55.204 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-modules-core-2:17.0.0-4502.gece5ed1a.el8. 6/42 2021-05-25T11:57:55.204 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-html5lib-1:0.999999999-6.el8.noarch 7/42 2021-05-25T11:57:55.204 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-lxml-4.2.3-1.el8.x86_64 8/42 2021-05-25T11:57:55.205 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 9/42 2021-05-25T11:57:55.205 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 10/42 2021-05-25T11:57:55.205 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pyOpenSSL-18.0.0-1.el8.noarch 11/42 2021-05-25T11:57:55.205 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 12/42 2021-05-25T11:57:55.205 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-webencodings-0.5.1-6.el8.noarch 13/42 2021-05-25T11:57:55.206 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 14/42 2021-05-25T11:57:55.206 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-asn1crypto-0.24.0-3.el8.noarch 15/42 2021-05-25T11:57:55.206 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 16/42 2021-05-25T11:57:55.206 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-chardet-3.0.4-7.el8.noarch 17/42 2021-05-25T11:57:55.206 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cryptography-2.3-3.el8.x86_64 18/42 2021-05-25T11:57:55.207 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-idna-2.5-5.el8.noarch 19/42 2021-05-25T11:57:55.207 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 20/42 2021-05-25T11:57:55.207 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pysocks-1.6.8-3.el8.noarch 21/42 2021-05-25T11:57:55.207 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-requests-2.20.0-2.1.el8_1.noarch 22/42 2021-05-25T11:57:55.207 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-urllib3-1.24.2-4.el8.noarch 23/42 2021-05-25T11:57:55.208 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 24/42 2021-05-25T11:57:55.208 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 25/42 2021-05-25T11:57:55.210 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 26/42 2021-05-25T11:57:55.210 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 27/42 2021-05-25T11:57:55.210 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cssselect-0.9.2-10.el8.noarch 28/42 2021-05-25T11:57:55.211 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 29/42 2021-05-25T11:57:55.211 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 30/42 2021-05-25T11:57:55.211 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 31/42 2021-05-25T11:57:55.211 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 32/42 2021-05-25T11:57:55.211 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 33/42 2021-05-25T11:57:55.212 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-portend-2.6-1.el8.noarch 34/42 2021-05-25T11:57:55.212 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 35/42 2021-05-25T11:57:55.212 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 36/42 2021-05-25T11:57:55.212 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 37/42 2021-05-25T11:57:55.213 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 38/42 2021-05-25T11:57:55.213 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 39/42 2021-05-25T11:57:55.213 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 40/42 2021-05-25T11:57:55.213 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 41/42 2021-05-25T11:57:55.324 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 36/42 2021-05-25T11:57:55.741 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-modules-core-2:17.0.0-4502.gece5ed1a.el8. 37/42 2021-05-25T11:57:55.964 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-2:17.0.0-4502.gece5ed1a.el8.x86_64 38/42 2021-05-25T11:57:57.065 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 42/42 2021-05-25T11:57:57.065 INFO:teuthology.orchestra.run.smithi181.stdout:Installed products updated. 2021-05-25T11:57:57.289 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:57:57.290 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2021-05-25T11:57:57.290 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-2:17.0.0-4502.gece5ed1a.el8.x86_64 2021-05-25T11:57:57.290 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mds-2:17.0.0-4502.gece5ed1a.el8.x86_64 2021-05-25T11:57:57.291 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-2:17.0.0-4502.gece5ed1a.el8.x86_64 2021-05-25T11:57:57.291 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-modules-core-2:17.0.0-4502.gece5ed1a.el8.noarch 2021-05-25T11:57:57.291 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mon-2:17.0.0-4502.gece5ed1a.el8.x86_64 2021-05-25T11:57:57.291 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-osd-2:17.0.0-4502.gece5ed1a.el8.x86_64 2021-05-25T11:57:57.291 INFO:teuthology.orchestra.run.smithi181.stdout: python3-asn1crypto-0.24.0-3.el8.noarch 2021-05-25T11:57:57.292 INFO:teuthology.orchestra.run.smithi181.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2021-05-25T11:57:57.292 INFO:teuthology.orchestra.run.smithi181.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2021-05-25T11:57:57.292 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cffi-1.11.5-5.el8.x86_64 2021-05-25T11:57:57.292 INFO:teuthology.orchestra.run.smithi181.stdout: python3-chardet-3.0.4-7.el8.noarch 2021-05-25T11:57:57.292 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cheroot-8.5.2-1.el8.noarch 2021-05-25T11:57:57.293 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2021-05-25T11:57:57.293 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cryptography-2.3-3.el8.x86_64 2021-05-25T11:57:57.293 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cssselect-0.9.2-10.el8.noarch 2021-05-25T11:57:57.293 INFO:teuthology.orchestra.run.smithi181.stdout: python3-html5lib-1:0.999999999-6.el8.noarch 2021-05-25T11:57:57.293 INFO:teuthology.orchestra.run.smithi181.stdout: python3-idna-2.5-5.el8.noarch 2021-05-25T11:57:57.294 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-6.2-6.el8.noarch 2021-05-25T11:57:57.294 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2021-05-25T11:57:57.294 INFO:teuthology.orchestra.run.smithi181.stdout: python3-logutils-0.3.5-11.el8.noarch 2021-05-25T11:57:57.294 INFO:teuthology.orchestra.run.smithi181.stdout: python3-lxml-4.2.3-1.el8.x86_64 2021-05-25T11:57:57.295 INFO:teuthology.orchestra.run.smithi181.stdout: python3-mako-1.0.6-13.el8.noarch 2021-05-25T11:57:57.295 INFO:teuthology.orchestra.run.smithi181.stdout: python3-markupsafe-0.23-19.el8.x86_64 2021-05-25T11:57:57.295 INFO:teuthology.orchestra.run.smithi181.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2021-05-25T11:57:57.295 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pecan-1.3.2-9.el8.noarch 2021-05-25T11:57:57.295 INFO:teuthology.orchestra.run.smithi181.stdout: python3-portend-2.6-1.el8.noarch 2021-05-25T11:57:57.296 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyOpenSSL-18.0.0-1.el8.noarch 2021-05-25T11:57:57.296 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pycparser-2.14-14.el8.noarch 2021-05-25T11:57:57.296 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pysocks-1.6.8-3.el8.noarch 2021-05-25T11:57:57.296 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pytz-2017.2-9.el8.noarch 2021-05-25T11:57:57.296 INFO:teuthology.orchestra.run.smithi181.stdout: python3-requests-2.20.0-2.1.el8_1.noarch 2021-05-25T11:57:57.297 INFO:teuthology.orchestra.run.smithi181.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2021-05-25T11:57:57.297 INFO:teuthology.orchestra.run.smithi181.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2021-05-25T11:57:57.297 INFO:teuthology.orchestra.run.smithi181.stdout: python3-tempora-1.14.1-5.el8.noarch 2021-05-25T11:57:57.297 INFO:teuthology.orchestra.run.smithi181.stdout: python3-trustme-0.6.0-4.el8.noarch 2021-05-25T11:57:57.298 INFO:teuthology.orchestra.run.smithi181.stdout: python3-urllib3-1.24.2-4.el8.noarch 2021-05-25T11:57:57.298 INFO:teuthology.orchestra.run.smithi181.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2021-05-25T11:57:57.298 INFO:teuthology.orchestra.run.smithi181.stdout: python3-webencodings-0.5.1-6.el8.noarch 2021-05-25T11:57:57.298 INFO:teuthology.orchestra.run.smithi181.stdout: python3-webob-1.8.5-1.el8.1.noarch 2021-05-25T11:57:57.298 INFO:teuthology.orchestra.run.smithi181.stdout: python3-webtest-2.0.33-1.el8.noarch 2021-05-25T11:57:57.299 INFO:teuthology.orchestra.run.smithi181.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2021-05-25T11:57:57.299 INFO:teuthology.orchestra.run.smithi181.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2021-05-25T11:57:57.299 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:57:57.299 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2021-05-25T11:57:57.447 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mgr-2:17.0.0-4502.gece5ed1a.el8.x86_64 38/42 2021-05-25T11:57:57.523 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-base 2021-05-25T11:57:57.672 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-osd-2:17.0.0-4502.gece5ed1a.el8.x86_64 39/42 2021-05-25T11:57:58.059 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:09 ago on Tue 25 May 2021 11:56:49 AM UTC. 2021-05-25T11:57:58.061 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-osd-2:17.0.0-4502.gece5ed1a.el8.x86_64 39/42 2021-05-25T11:57:58.289 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mon-2:17.0.0-4502.gece5ed1a.el8.x86_64 40/42 2021-05-25T11:57:58.506 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mon-2:17.0.0-4502.gece5ed1a.el8.x86_64 40/42 2021-05-25T11:57:58.549 INFO:teuthology.orchestra.run.smithi181.stdout:Package ceph-base-2:17.0.0-4502.gece5ed1a.el8.x86_64 is already installed. 2021-05-25T11:57:58.683 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mds-2:17.0.0-4502.gece5ed1a.el8.x86_64 41/42 2021-05-25T11:57:58.700 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mds-2:17.0.0-4502.gece5ed1a.el8.x86_64 41/42 2021-05-25T11:57:58.716 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2021-05-25T11:57:58.717 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2021-05-25T11:57:58.717 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2021-05-25T11:57:58.777 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install cephadm 2021-05-25T11:57:58.823 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-2:17.0.0-4502.gece5ed1a.el8.x86_64 42/42 2021-05-25T11:57:59.213 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-2:17.0.0-4502.gece5ed1a.el8.x86_64 42/42 2021-05-25T11:57:59.213 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-2:17.0.0-4502.gece5ed1a.el8.x86_64 1/42 2021-05-25T11:57:59.213 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mds-2:17.0.0-4502.gece5ed1a.el8.x86_64 2/42 2021-05-25T11:57:59.213 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-2:17.0.0-4502.gece5ed1a.el8.x86_64 3/42 2021-05-25T11:57:59.214 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mon-2:17.0.0-4502.gece5ed1a.el8.x86_64 4/42 2021-05-25T11:57:59.214 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-osd-2:17.0.0-4502.gece5ed1a.el8.x86_64 5/42 2021-05-25T11:57:59.214 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-modules-core-2:17.0.0-4502.gece5ed1a.el8. 6/42 2021-05-25T11:57:59.214 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-html5lib-1:0.999999999-6.el8.noarch 7/42 2021-05-25T11:57:59.215 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-lxml-4.2.3-1.el8.x86_64 8/42 2021-05-25T11:57:59.215 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 9/42 2021-05-25T11:57:59.215 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 10/42 2021-05-25T11:57:59.215 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pyOpenSSL-18.0.0-1.el8.noarch 11/42 2021-05-25T11:57:59.216 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 12/42 2021-05-25T11:57:59.216 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-webencodings-0.5.1-6.el8.noarch 13/42 2021-05-25T11:57:59.216 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 14/42 2021-05-25T11:57:59.216 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-asn1crypto-0.24.0-3.el8.noarch 15/42 2021-05-25T11:57:59.217 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 16/42 2021-05-25T11:57:59.217 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-chardet-3.0.4-7.el8.noarch 17/42 2021-05-25T11:57:59.217 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cryptography-2.3-3.el8.x86_64 18/42 2021-05-25T11:57:59.217 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-idna-2.5-5.el8.noarch 19/42 2021-05-25T11:57:59.217 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 20/42 2021-05-25T11:57:59.218 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pysocks-1.6.8-3.el8.noarch 21/42 2021-05-25T11:57:59.218 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-requests-2.20.0-2.1.el8_1.noarch 22/42 2021-05-25T11:57:59.218 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-urllib3-1.24.2-4.el8.noarch 23/42 2021-05-25T11:57:59.218 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 24/42 2021-05-25T11:57:59.219 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 25/42 2021-05-25T11:57:59.219 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 26/42 2021-05-25T11:57:59.219 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 27/42 2021-05-25T11:57:59.219 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cssselect-0.9.2-10.el8.noarch 28/42 2021-05-25T11:57:59.220 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 29/42 2021-05-25T11:57:59.220 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 30/42 2021-05-25T11:57:59.220 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 31/42 2021-05-25T11:57:59.220 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 32/42 2021-05-25T11:57:59.220 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 33/42 2021-05-25T11:57:59.221 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-portend-2.6-1.el8.noarch 34/42 2021-05-25T11:57:59.221 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 35/42 2021-05-25T11:57:59.221 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 36/42 2021-05-25T11:57:59.221 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 37/42 2021-05-25T11:57:59.221 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 38/42 2021-05-25T11:57:59.222 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 39/42 2021-05-25T11:57:59.222 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 40/42 2021-05-25T11:57:59.222 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 41/42 2021-05-25T11:57:59.306 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:10 ago on Tue 25 May 2021 11:56:49 AM UTC. 2021-05-25T11:57:59.974 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2021-05-25T11:57:59.976 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2021-05-25T11:57:59.976 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2021-05-25T11:57:59.977 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2021-05-25T11:57:59.977 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2021-05-25T11:57:59.978 INFO:teuthology.orchestra.run.smithi181.stdout: cephadm noarch 2:17.0.0-4502.gece5ed1a.el8 ceph-noarch 67 k 2021-05-25T11:57:59.978 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:57:59.978 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2021-05-25T11:57:59.979 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2021-05-25T11:57:59.979 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2021-05-25T11:57:59.979 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:57:59.979 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 67 k 2021-05-25T11:57:59.979 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 297 k 2021-05-25T11:57:59.980 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2021-05-25T11:58:00.024 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 42/42 2021-05-25T11:58:00.025 INFO:teuthology.orchestra.run.smithi102.stdout:Installed products updated. 2021-05-25T11:58:00.122 INFO:teuthology.orchestra.run.smithi181.stdout:cephadm-17.0.0-4502.gece5ed1a.el8.noarch.rpm 458 kB/s | 67 kB 00:00 2021-05-25T11:58:00.123 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2021-05-25T11:58:00.124 INFO:teuthology.orchestra.run.smithi181.stdout:Total 454 kB/s | 67 kB 00:00 2021-05-25T11:58:00.124 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2021-05-25T11:58:00.131 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2021-05-25T11:58:00.131 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2021-05-25T11:58:00.133 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2021-05-25T11:58:00.136 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2021-05-25T11:58:00.193 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2021-05-25T11:58:00.262 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:58:00.263 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2021-05-25T11:58:00.263 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-2:17.0.0-4502.gece5ed1a.el8.x86_64 2021-05-25T11:58:00.263 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mds-2:17.0.0-4502.gece5ed1a.el8.x86_64 2021-05-25T11:58:00.263 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-2:17.0.0-4502.gece5ed1a.el8.x86_64 2021-05-25T11:58:00.264 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-modules-core-2:17.0.0-4502.gece5ed1a.el8.noarch 2021-05-25T11:58:00.264 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mon-2:17.0.0-4502.gece5ed1a.el8.x86_64 2021-05-25T11:58:00.264 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-osd-2:17.0.0-4502.gece5ed1a.el8.x86_64 2021-05-25T11:58:00.264 INFO:teuthology.orchestra.run.smithi102.stdout: python3-asn1crypto-0.24.0-3.el8.noarch 2021-05-25T11:58:00.264 INFO:teuthology.orchestra.run.smithi102.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2021-05-25T11:58:00.265 INFO:teuthology.orchestra.run.smithi102.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2021-05-25T11:58:00.265 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cffi-1.11.5-5.el8.x86_64 2021-05-25T11:58:00.265 INFO:teuthology.orchestra.run.smithi102.stdout: python3-chardet-3.0.4-7.el8.noarch 2021-05-25T11:58:00.265 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cheroot-8.5.2-1.el8.noarch 2021-05-25T11:58:00.266 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2021-05-25T11:58:00.266 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cryptography-2.3-3.el8.x86_64 2021-05-25T11:58:00.266 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cssselect-0.9.2-10.el8.noarch 2021-05-25T11:58:00.266 INFO:teuthology.orchestra.run.smithi102.stdout: python3-html5lib-1:0.999999999-6.el8.noarch 2021-05-25T11:58:00.267 INFO:teuthology.orchestra.run.smithi102.stdout: python3-idna-2.5-5.el8.noarch 2021-05-25T11:58:00.267 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-6.2-6.el8.noarch 2021-05-25T11:58:00.267 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2021-05-25T11:58:00.267 INFO:teuthology.orchestra.run.smithi102.stdout: python3-logutils-0.3.5-11.el8.noarch 2021-05-25T11:58:00.267 INFO:teuthology.orchestra.run.smithi102.stdout: python3-lxml-4.2.3-1.el8.x86_64 2021-05-25T11:58:00.268 INFO:teuthology.orchestra.run.smithi102.stdout: python3-mako-1.0.6-13.el8.noarch 2021-05-25T11:58:00.268 INFO:teuthology.orchestra.run.smithi102.stdout: python3-markupsafe-0.23-19.el8.x86_64 2021-05-25T11:58:00.268 INFO:teuthology.orchestra.run.smithi102.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2021-05-25T11:58:00.268 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pecan-1.3.2-9.el8.noarch 2021-05-25T11:58:00.268 INFO:teuthology.orchestra.run.smithi102.stdout: python3-portend-2.6-1.el8.noarch 2021-05-25T11:58:00.269 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyOpenSSL-18.0.0-1.el8.noarch 2021-05-25T11:58:00.269 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pycparser-2.14-14.el8.noarch 2021-05-25T11:58:00.269 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pysocks-1.6.8-3.el8.noarch 2021-05-25T11:58:00.269 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pytz-2017.2-9.el8.noarch 2021-05-25T11:58:00.270 INFO:teuthology.orchestra.run.smithi102.stdout: python3-requests-2.20.0-2.1.el8_1.noarch 2021-05-25T11:58:00.270 INFO:teuthology.orchestra.run.smithi102.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2021-05-25T11:58:00.270 INFO:teuthology.orchestra.run.smithi102.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2021-05-25T11:58:00.270 INFO:teuthology.orchestra.run.smithi102.stdout: python3-tempora-1.14.1-5.el8.noarch 2021-05-25T11:58:00.270 INFO:teuthology.orchestra.run.smithi102.stdout: python3-trustme-0.6.0-4.el8.noarch 2021-05-25T11:58:00.271 INFO:teuthology.orchestra.run.smithi102.stdout: python3-urllib3-1.24.2-4.el8.noarch 2021-05-25T11:58:00.271 INFO:teuthology.orchestra.run.smithi102.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2021-05-25T11:58:00.271 INFO:teuthology.orchestra.run.smithi102.stdout: python3-webencodings-0.5.1-6.el8.noarch 2021-05-25T11:58:00.271 INFO:teuthology.orchestra.run.smithi102.stdout: python3-webob-1.8.5-1.el8.1.noarch 2021-05-25T11:58:00.272 INFO:teuthology.orchestra.run.smithi102.stdout: python3-webtest-2.0.33-1.el8.noarch 2021-05-25T11:58:00.272 INFO:teuthology.orchestra.run.smithi102.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2021-05-25T11:58:00.272 INFO:teuthology.orchestra.run.smithi102.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2021-05-25T11:58:00.272 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:58:00.272 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2021-05-25T11:58:00.480 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-base 2021-05-25T11:58:01.010 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:11 ago on Tue 25 May 2021 11:56:50 AM UTC. 2021-05-25T11:58:01.504 INFO:teuthology.orchestra.run.smithi102.stdout:Package ceph-base-2:17.0.0-4502.gece5ed1a.el8.x86_64 is already installed. 2021-05-25T11:58:01.673 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2021-05-25T11:58:01.674 INFO:teuthology.orchestra.run.smithi102.stdout:Nothing to do. 2021-05-25T11:58:01.675 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2021-05-25T11:58:01.747 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install cephadm 2021-05-25T11:58:02.272 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:12 ago on Tue 25 May 2021 11:56:50 AM UTC. 2021-05-25T11:58:02.401 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: cephadm-2:17.0.0-4502.gece5ed1a.el8.noarch 1/1 2021-05-25T11:58:02.634 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : cephadm-2:17.0.0-4502.gece5ed1a.el8.noarch 1/1 2021-05-25T11:58:02.805 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: cephadm-2:17.0.0-4502.gece5ed1a.el8.noarch 1/1 2021-05-25T11:58:02.974 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2021-05-25T11:58:02.976 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-05-25T11:58:02.976 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2021-05-25T11:58:02.976 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-05-25T11:58:02.977 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2021-05-25T11:58:02.977 INFO:teuthology.orchestra.run.smithi102.stdout: cephadm noarch 2:17.0.0-4502.gece5ed1a.el8 ceph-noarch 67 k 2021-05-25T11:58:02.978 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:58:02.978 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2021-05-25T11:58:02.978 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-05-25T11:58:02.979 INFO:teuthology.orchestra.run.smithi102.stdout:Install 1 Package 2021-05-25T11:58:02.979 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:58:02.979 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 67 k 2021-05-25T11:58:02.980 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 297 k 2021-05-25T11:58:02.980 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2021-05-25T11:58:03.021 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : cephadm-2:17.0.0-4502.gece5ed1a.el8.noarch 1/1 2021-05-25T11:58:03.021 INFO:teuthology.orchestra.run.smithi181.stdout:Installed products updated. 2021-05-25T11:58:03.123 INFO:teuthology.orchestra.run.smithi102.stdout:cephadm-17.0.0-4502.gece5ed1a.el8.noarch.rpm 457 kB/s | 67 kB 00:00 2021-05-25T11:58:03.124 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2021-05-25T11:58:03.124 INFO:teuthology.orchestra.run.smithi102.stdout:Total 453 kB/s | 67 kB 00:00 2021-05-25T11:58:03.124 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2021-05-25T11:58:03.132 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2021-05-25T11:58:03.132 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2021-05-25T11:58:03.133 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2021-05-25T11:58:03.136 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2021-05-25T11:58:03.181 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2021-05-25T11:58:03.268 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:58:03.269 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2021-05-25T11:58:03.269 INFO:teuthology.orchestra.run.smithi181.stdout: cephadm-2:17.0.0-4502.gece5ed1a.el8.noarch 2021-05-25T11:58:03.270 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:58:03.270 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2021-05-25T11:58:03.470 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-immutable-object-cache 2021-05-25T11:58:04.017 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:15 ago on Tue 25 May 2021 11:56:49 AM UTC. 2021-05-25T11:58:04.679 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2021-05-25T11:58:04.680 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2021-05-25T11:58:04.680 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repo Size 2021-05-25T11:58:04.680 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2021-05-25T11:58:04.680 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2021-05-25T11:58:04.681 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-immutable-object-cache x86_64 2:17.0.0-4502.gece5ed1a.el8 ceph 167 k 2021-05-25T11:58:04.681 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:58:04.681 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2021-05-25T11:58:04.681 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2021-05-25T11:58:04.682 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2021-05-25T11:58:04.682 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:58:04.682 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 167 k 2021-05-25T11:58:04.682 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 633 k 2021-05-25T11:58:04.682 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2021-05-25T11:58:04.855 INFO:teuthology.orchestra.run.smithi181.stdout:ceph-immutable-object-cache-17.0.0-4502.gece5ed 955 kB/s | 167 kB 00:00 2021-05-25T11:58:04.856 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2021-05-25T11:58:04.856 INFO:teuthology.orchestra.run.smithi181.stdout:Total 948 kB/s | 167 kB 00:00 2021-05-25T11:58:04.857 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2021-05-25T11:58:04.869 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2021-05-25T11:58:04.869 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2021-05-25T11:58:04.916 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2021-05-25T11:58:04.920 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2021-05-25T11:58:05.033 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2021-05-25T11:58:05.205 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: cephadm-2:17.0.0-4502.gece5ed1a.el8.noarch 1/1 2021-05-25T11:58:05.293 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-immutable-object-cache-2:17.0.0-4502.gece5ed1a. 1/1 2021-05-25T11:58:05.468 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : cephadm-2:17.0.0-4502.gece5ed1a.el8.noarch 1/1 2021-05-25T11:58:05.635 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: cephadm-2:17.0.0-4502.gece5ed1a.el8.noarch 1/1 2021-05-25T11:58:05.640 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.0.0-4502.gece5ed1a. 1/1 2021-05-25T11:58:05.857 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : cephadm-2:17.0.0-4502.gece5ed1a.el8.noarch 1/1 2021-05-25T11:58:05.858 INFO:teuthology.orchestra.run.smithi102.stdout:Installed products updated. 2021-05-25T11:58:05.902 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-immutable-object-cache-2:17.0.0-4502.gece5ed1a. 1/1 2021-05-25T11:58:05.903 INFO:teuthology.orchestra.run.smithi181.stdout:Installed products updated. 2021-05-25T11:58:06.105 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:58:06.105 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2021-05-25T11:58:06.106 INFO:teuthology.orchestra.run.smithi102.stdout: cephadm-2:17.0.0-4502.gece5ed1a.el8.noarch 2021-05-25T11:58:06.106 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:58:06.106 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2021-05-25T11:58:06.167 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:58:06.168 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2021-05-25T11:58:06.168 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-immutable-object-cache-2:17.0.0-4502.gece5ed1a.el8.x86_64 2021-05-25T11:58:06.169 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:58:06.169 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2021-05-25T11:58:06.274 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-immutable-object-cache 2021-05-25T11:58:06.309 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-mgr 2021-05-25T11:58:06.819 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:16 ago on Tue 25 May 2021 11:56:50 AM UTC. 2021-05-25T11:58:06.848 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:17 ago on Tue 25 May 2021 11:56:49 AM UTC. 2021-05-25T11:58:07.486 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2021-05-25T11:58:07.487 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-05-25T11:58:07.487 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repo Size 2021-05-25T11:58:07.488 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-05-25T11:58:07.488 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2021-05-25T11:58:07.488 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-immutable-object-cache x86_64 2:17.0.0-4502.gece5ed1a.el8 ceph 167 k 2021-05-25T11:58:07.488 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:58:07.489 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2021-05-25T11:58:07.489 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-05-25T11:58:07.489 INFO:teuthology.orchestra.run.smithi102.stdout:Install 1 Package 2021-05-25T11:58:07.489 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:58:07.490 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 167 k 2021-05-25T11:58:07.490 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 633 k 2021-05-25T11:58:07.490 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2021-05-25T11:58:07.717 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-immutable-object-cache-17.0.0-4502.gece5ed 729 kB/s | 167 kB 00:00 2021-05-25T11:58:07.717 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2021-05-25T11:58:07.718 INFO:teuthology.orchestra.run.smithi102.stdout:Total 725 kB/s | 167 kB 00:00 2021-05-25T11:58:07.718 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2021-05-25T11:58:07.731 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2021-05-25T11:58:07.732 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2021-05-25T11:58:07.781 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2021-05-25T11:58:07.784 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2021-05-25T11:58:07.877 INFO:teuthology.orchestra.run.smithi181.stdout:Package ceph-mgr-2:17.0.0-4502.gece5ed1a.el8.x86_64 is already installed. 2021-05-25T11:58:08.045 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2021-05-25T11:58:08.047 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2021-05-25T11:58:08.047 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2021-05-25T11:58:08.280 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-mgr-dashboard 2021-05-25T11:58:08.634 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2021-05-25T11:58:08.816 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:19 ago on Tue 25 May 2021 11:56:49 AM UTC. 2021-05-25T11:58:09.491 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2021-05-25T11:58:09.491 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2021-05-25T11:58:09.492 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2021-05-25T11:58:09.492 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2021-05-25T11:58:09.493 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2021-05-25T11:58:09.493 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-dashboard noarch 2:17.0.0-4502.gece5ed1a.el8 ceph-noarch 2.3 M 2021-05-25T11:58:09.493 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2021-05-25T11:58:09.493 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-grafana-dashboards noarch 2:17.0.0-4502.gece5ed1a.el8 ceph-noarch 24 k 2021-05-25T11:58:09.494 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-prometheus-alerts noarch 2:17.0.0-4502.gece5ed1a.el8 ceph-noarch 8.9 k 2021-05-25T11:58:09.494 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jwt noarch 1.6.1-2.el8 CentOS-Base 43 k 2021-05-25T11:58:09.494 INFO:teuthology.orchestra.run.smithi181.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2021-05-25T11:58:09.494 INFO:teuthology.orchestra.run.smithi181.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2021-05-25T11:58:09.495 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:58:09.495 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2021-05-25T11:58:09.495 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2021-05-25T11:58:09.495 INFO:teuthology.orchestra.run.smithi181.stdout:Install 6 Packages 2021-05-25T11:58:09.496 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:58:09.496 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 2.6 M 2021-05-25T11:58:09.497 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 15 M 2021-05-25T11:58:09.497 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2021-05-25T11:58:09.602 INFO:teuthology.orchestra.run.smithi181.stdout:(1/6): ceph-prometheus-alerts-17.0.0-4502.gece5 82 kB/s | 8.9 kB 00:00 2021-05-25T11:58:09.617 INFO:teuthology.orchestra.run.smithi181.stdout:(2/6): ceph-grafana-dashboards-17.0.0-4502.gece 195 kB/s | 24 kB 00:00 2021-05-25T11:58:09.633 INFO:teuthology.orchestra.run.smithi181.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 1.4 MB/s | 43 kB 00:00 2021-05-25T11:58:09.685 INFO:teuthology.orchestra.run.smithi181.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 498 kB/s | 34 kB 00:00 2021-05-25T11:58:09.744 INFO:teuthology.orchestra.run.smithi181.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.7 MB/s | 196 kB 00:00 2021-05-25T11:58:09.782 INFO:teuthology.orchestra.run.smithi181.stdout:(6/6): ceph-mgr-dashboard-17.0.0-4502.gece5ed1a 7.9 MB/s | 2.3 MB 00:00 2021-05-25T11:58:09.785 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2021-05-25T11:58:09.785 INFO:teuthology.orchestra.run.smithi181.stdout:Total 8.8 MB/s | 2.6 MB 00:00 2021-05-25T11:58:09.786 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2021-05-25T11:58:09.795 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2021-05-25T11:58:09.795 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2021-05-25T11:58:09.823 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-immutable-object-cache-2:17.0.0-4502.gece5ed1a. 1/1 2021-05-25T11:58:09.934 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2021-05-25T11:58:09.937 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2021-05-25T11:58:10.170 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.0.0-4502.gece5ed1a. 1/1 2021-05-25T11:58:10.219 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2021-05-25T11:58:10.391 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-immutable-object-cache-2:17.0.0-4502.gece5ed1a. 1/1 2021-05-25T11:58:10.392 INFO:teuthology.orchestra.run.smithi102.stdout:Installed products updated. 2021-05-25T11:58:10.433 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2021-05-25T11:58:10.620 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:58:10.621 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2021-05-25T11:58:10.621 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-immutable-object-cache-2:17.0.0-4502.gece5ed1a.el8.x86_64 2021-05-25T11:58:10.622 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:58:10.622 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2021-05-25T11:58:10.674 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2021-05-25T11:58:10.796 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-mgr 2021-05-25T11:58:10.910 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2021-05-25T11:58:11.075 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-prometheus-alerts-2:17.0.0-4502.gece5ed1a.el8.n 4/6 2021-05-25T11:58:11.334 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:21 ago on Tue 25 May 2021 11:56:50 AM UTC. 2021-05-25T11:58:11.836 INFO:teuthology.orchestra.run.smithi102.stdout:Package ceph-mgr-2:17.0.0-4502.gece5ed1a.el8.x86_64 is already installed. 2021-05-25T11:58:12.003 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2021-05-25T11:58:12.004 INFO:teuthology.orchestra.run.smithi102.stdout:Nothing to do. 2021-05-25T11:58:12.004 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2021-05-25T11:58:12.057 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-mgr-dashboard 2021-05-25T11:58:12.129 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-grafana-dashboards-2:17.0.0-4502.gece5ed1a.el8. 5/6 2021-05-25T11:58:12.428 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-dashboard-2:17.0.0-4502.gece5ed1a.el8.noarc 6/6 2021-05-25T11:58:12.584 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:22 ago on Tue 25 May 2021 11:56:50 AM UTC. 2021-05-25T11:58:12.652 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.0.0-4502.gece5ed1a.el8.noarc 6/6 2021-05-25T11:58:12.653 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-grafana-dashboards-2:17.0.0-4502.gece5ed1a.el8. 1/6 2021-05-25T11:58:12.653 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-dashboard-2:17.0.0-4502.gece5ed1a.el8.noarc 2/6 2021-05-25T11:58:12.653 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-prometheus-alerts-2:17.0.0-4502.gece5ed1a.el8.n 3/6 2021-05-25T11:58:12.654 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2021-05-25T11:58:12.654 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2021-05-25T11:58:12.915 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2021-05-25T11:58:12.915 INFO:teuthology.orchestra.run.smithi181.stdout:Installed products updated. 2021-05-25T11:58:13.125 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:58:13.126 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2021-05-25T11:58:13.126 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-grafana-dashboards-2:17.0.0-4502.gece5ed1a.el8.noarch 2021-05-25T11:58:13.126 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-dashboard-2:17.0.0-4502.gece5ed1a.el8.noarch 2021-05-25T11:58:13.126 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-prometheus-alerts-2:17.0.0-4502.gece5ed1a.el8.noarch 2021-05-25T11:58:13.127 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jwt-1.6.1-2.el8.noarch 2021-05-25T11:58:13.127 INFO:teuthology.orchestra.run.smithi181.stdout: python3-repoze-lru-0.7-6.el8.noarch 2021-05-25T11:58:13.127 INFO:teuthology.orchestra.run.smithi181.stdout: python3-routes-2.4.1-12.el8.noarch 2021-05-25T11:58:13.127 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:58:13.127 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2021-05-25T11:58:13.213 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-mgr-diskprediction-local 2021-05-25T11:58:13.262 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2021-05-25T11:58:13.264 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-05-25T11:58:13.264 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2021-05-25T11:58:13.264 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-05-25T11:58:13.264 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2021-05-25T11:58:13.265 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-dashboard noarch 2:17.0.0-4502.gece5ed1a.el8 ceph-noarch 2.3 M 2021-05-25T11:58:13.265 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2021-05-25T11:58:13.265 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-grafana-dashboards noarch 2:17.0.0-4502.gece5ed1a.el8 ceph-noarch 24 k 2021-05-25T11:58:13.265 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-prometheus-alerts noarch 2:17.0.0-4502.gece5ed1a.el8 ceph-noarch 8.9 k 2021-05-25T11:58:13.265 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jwt noarch 1.6.1-2.el8 CentOS-Base 43 k 2021-05-25T11:58:13.266 INFO:teuthology.orchestra.run.smithi102.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2021-05-25T11:58:13.266 INFO:teuthology.orchestra.run.smithi102.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2021-05-25T11:58:13.266 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:58:13.266 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2021-05-25T11:58:13.267 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-05-25T11:58:13.267 INFO:teuthology.orchestra.run.smithi102.stdout:Install 6 Packages 2021-05-25T11:58:13.267 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:58:13.268 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 2.6 M 2021-05-25T11:58:13.268 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 15 M 2021-05-25T11:58:13.268 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2021-05-25T11:58:13.374 INFO:teuthology.orchestra.run.smithi102.stdout:(1/6): ceph-prometheus-alerts-17.0.0-4502.gece5 82 kB/s | 8.9 kB 00:00 2021-05-25T11:58:13.394 INFO:teuthology.orchestra.run.smithi102.stdout:(2/6): ceph-grafana-dashboards-17.0.0-4502.gece 188 kB/s | 24 kB 00:00 2021-05-25T11:58:13.405 INFO:teuthology.orchestra.run.smithi102.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 1.3 MB/s | 43 kB 00:00 2021-05-25T11:58:13.462 INFO:teuthology.orchestra.run.smithi102.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 492 kB/s | 34 kB 00:00 2021-05-25T11:58:13.589 INFO:teuthology.orchestra.run.smithi102.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.0 MB/s | 196 kB 00:00 2021-05-25T11:58:13.746 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:24 ago on Tue 25 May 2021 11:56:49 AM UTC. 2021-05-25T11:58:13.986 INFO:teuthology.orchestra.run.smithi102.stdout:(6/6): ceph-mgr-dashboard-17.0.0-4502.gece5ed1a 3.2 MB/s | 2.3 MB 00:00 2021-05-25T11:58:13.989 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2021-05-25T11:58:13.989 INFO:teuthology.orchestra.run.smithi102.stdout:Total 3.6 MB/s | 2.6 MB 00:00 2021-05-25T11:58:13.990 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2021-05-25T11:58:14.000 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2021-05-25T11:58:14.001 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2021-05-25T11:58:14.125 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2021-05-25T11:58:14.128 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2021-05-25T11:58:14.384 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2021-05-25T11:58:14.432 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2021-05-25T11:58:14.433 INFO:teuthology.orchestra.run.smithi181.stdout:===================================================================================================== 2021-05-25T11:58:14.433 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2021-05-25T11:58:14.434 INFO:teuthology.orchestra.run.smithi181.stdout:===================================================================================================== 2021-05-25T11:58:14.434 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2021-05-25T11:58:14.434 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-diskprediction-local noarch 2:17.0.0-4502.gece5ed1a.el8 ceph-noarch 7.4 M 2021-05-25T11:58:14.434 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2021-05-25T11:58:14.435 INFO:teuthology.orchestra.run.smithi181.stdout: libgfortran x86_64 8.3.1-5.1.el8 CentOS-Base 639 k 2021-05-25T11:58:14.435 INFO:teuthology.orchestra.run.smithi181.stdout: libquadmath x86_64 8.3.1-5.1.el8 CentOS-Base 169 k 2021-05-25T11:58:14.435 INFO:teuthology.orchestra.run.smithi181.stdout: openblas x86_64 0.3.3-5.el8 CentOS-AppStream 4.3 M 2021-05-25T11:58:14.435 INFO:teuthology.orchestra.run.smithi181.stdout: openblas-threads x86_64 0.3.3-5.el8 CentOS-AppStream 4.3 M 2021-05-25T11:58:14.435 INFO:teuthology.orchestra.run.smithi181.stdout: python3-numpy x86_64 1:1.14.3-9.el8 CentOS-AppStream 3.7 M 2021-05-25T11:58:14.436 INFO:teuthology.orchestra.run.smithi181.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 CentOS-AppStream 225 k 2021-05-25T11:58:14.436 INFO:teuthology.orchestra.run.smithi181.stdout: python3-scipy x86_64 1.0.0-20.module_el8.3.0+562+e162826a CentOS-AppStream 15 M 2021-05-25T11:58:14.436 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:58:14.436 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2021-05-25T11:58:14.437 INFO:teuthology.orchestra.run.smithi181.stdout:===================================================================================================== 2021-05-25T11:58:14.437 INFO:teuthology.orchestra.run.smithi181.stdout:Install 8 Packages 2021-05-25T11:58:14.437 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:58:14.438 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 35 M 2021-05-25T11:58:14.438 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 194 M 2021-05-25T11:58:14.438 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2021-05-25T11:58:14.527 INFO:teuthology.orchestra.run.smithi181.stdout:(1/8): openblas-0.3.3-5.el8.x86_64.rpm 46 MB/s | 4.3 MB 00:00 2021-05-25T11:58:14.584 INFO:teuthology.orchestra.run.smithi181.stdout:(2/8): python3-numpy-1.14.3-9.el8.x86_64.rpm 66 MB/s | 3.7 MB 00:00 2021-05-25T11:58:14.588 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2021-05-25T11:58:14.589 INFO:teuthology.orchestra.run.smithi181.stdout:(3/8): python3-numpy-f2py-1.14.3-9.el8.x86_64.r 42 MB/s | 225 kB 00:00 2021-05-25T11:58:14.630 INFO:teuthology.orchestra.run.smithi181.stdout:(4/8): openblas-threads-0.3.3-5.el8.x86_64.rpm 22 MB/s | 4.3 MB 00:00 2021-05-25T11:58:14.654 INFO:teuthology.orchestra.run.smithi181.stdout:(5/8): libgfortran-8.3.1-5.1.el8.x86_64.rpm 26 MB/s | 639 kB 00:00 2021-05-25T11:58:14.666 INFO:teuthology.orchestra.run.smithi181.stdout:(6/8): libquadmath-8.3.1-5.1.el8.x86_64.rpm 14 MB/s | 169 kB 00:00 2021-05-25T11:58:14.824 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2021-05-25T11:58:14.846 INFO:teuthology.orchestra.run.smithi181.stdout:(7/8): python3-scipy-1.0.0-20.module_el8.3.0+56 57 MB/s | 15 MB 00:00 2021-05-25T11:58:14.927 INFO:teuthology.orchestra.run.smithi181.stdout:(8/8): ceph-mgr-diskprediction-local-17.0.0-450 15 MB/s | 7.4 MB 00:00 2021-05-25T11:58:14.930 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2021-05-25T11:58:14.930 INFO:teuthology.orchestra.run.smithi181.stdout:Total 71 MB/s | 35 MB 00:00 2021-05-25T11:58:14.939 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2021-05-25T11:58:14.958 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2021-05-25T11:58:14.958 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2021-05-25T11:58:15.032 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2021-05-25T11:58:15.203 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-prometheus-alerts-2:17.0.0-4502.gece5ed1a.el8.n 4/6 2021-05-25T11:58:15.266 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2021-05-25T11:58:15.269 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2021-05-25T11:58:15.865 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2021-05-25T11:58:16.049 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libquadmath-8.3.1-5.1.el8.x86_64 1/8 2021-05-25T11:58:16.187 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libquadmath-8.3.1-5.1.el8.x86_64 1/8 2021-05-25T11:58:16.213 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-grafana-dashboards-2:17.0.0-4502.gece5ed1a.el8. 5/6 2021-05-25T11:58:16.324 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libgfortran-8.3.1-5.1.el8.x86_64 2/8 2021-05-25T11:58:16.498 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-dashboard-2:17.0.0-4502.gece5ed1a.el8.noarc 6/6 2021-05-25T11:58:16.727 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.0.0-4502.gece5ed1a.el8.noarc 6/6 2021-05-25T11:58:16.728 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-grafana-dashboards-2:17.0.0-4502.gece5ed1a.el8. 1/6 2021-05-25T11:58:16.728 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-dashboard-2:17.0.0-4502.gece5ed1a.el8.noarc 2/6 2021-05-25T11:58:16.728 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-prometheus-alerts-2:17.0.0-4502.gece5ed1a.el8.n 3/6 2021-05-25T11:58:16.728 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2021-05-25T11:58:16.729 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2021-05-25T11:58:16.857 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libgfortran-8.3.1-5.1.el8.x86_64 2/8 2021-05-25T11:58:16.984 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2021-05-25T11:58:16.985 INFO:teuthology.orchestra.run.smithi102.stdout:Installed products updated. 2021-05-25T11:58:17.074 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : openblas-0.3.3-5.el8.x86_64 3/8 2021-05-25T11:58:17.221 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:58:17.222 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2021-05-25T11:58:17.222 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-grafana-dashboards-2:17.0.0-4502.gece5ed1a.el8.noarch 2021-05-25T11:58:17.222 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-dashboard-2:17.0.0-4502.gece5ed1a.el8.noarch 2021-05-25T11:58:17.223 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-prometheus-alerts-2:17.0.0-4502.gece5ed1a.el8.noarch 2021-05-25T11:58:17.223 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jwt-1.6.1-2.el8.noarch 2021-05-25T11:58:17.223 INFO:teuthology.orchestra.run.smithi102.stdout: python3-repoze-lru-0.7-6.el8.noarch 2021-05-25T11:58:17.223 INFO:teuthology.orchestra.run.smithi102.stdout: python3-routes-2.4.1-12.el8.noarch 2021-05-25T11:58:17.223 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:58:17.224 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2021-05-25T11:58:17.297 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-mgr-diskprediction-local 2021-05-25T11:58:17.579 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: openblas-0.3.3-5.el8.x86_64 3/8 2021-05-25T11:58:17.748 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : openblas-threads-0.3.3-5.el8.x86_64 4/8 2021-05-25T11:58:17.825 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:27 ago on Tue 25 May 2021 11:56:50 AM UTC. 2021-05-25T11:58:18.483 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: openblas-threads-0.3.3-5.el8.x86_64 4/8 2021-05-25T11:58:18.495 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2021-05-25T11:58:18.496 INFO:teuthology.orchestra.run.smithi102.stdout:===================================================================================================== 2021-05-25T11:58:18.496 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2021-05-25T11:58:18.497 INFO:teuthology.orchestra.run.smithi102.stdout:===================================================================================================== 2021-05-25T11:58:18.497 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2021-05-25T11:58:18.497 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-diskprediction-local noarch 2:17.0.0-4502.gece5ed1a.el8 ceph-noarch 7.4 M 2021-05-25T11:58:18.497 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2021-05-25T11:58:18.498 INFO:teuthology.orchestra.run.smithi102.stdout: libgfortran x86_64 8.3.1-5.1.el8 CentOS-Base 639 k 2021-05-25T11:58:18.498 INFO:teuthology.orchestra.run.smithi102.stdout: libquadmath x86_64 8.3.1-5.1.el8 CentOS-Base 169 k 2021-05-25T11:58:18.498 INFO:teuthology.orchestra.run.smithi102.stdout: openblas x86_64 0.3.3-5.el8 CentOS-AppStream 4.3 M 2021-05-25T11:58:18.498 INFO:teuthology.orchestra.run.smithi102.stdout: openblas-threads x86_64 0.3.3-5.el8 CentOS-AppStream 4.3 M 2021-05-25T11:58:18.498 INFO:teuthology.orchestra.run.smithi102.stdout: python3-numpy x86_64 1:1.14.3-9.el8 CentOS-AppStream 3.7 M 2021-05-25T11:58:18.499 INFO:teuthology.orchestra.run.smithi102.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 CentOS-AppStream 225 k 2021-05-25T11:58:18.499 INFO:teuthology.orchestra.run.smithi102.stdout: python3-scipy x86_64 1.0.0-20.module_el8.3.0+562+e162826a CentOS-AppStream 15 M 2021-05-25T11:58:18.499 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:58:18.499 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2021-05-25T11:58:18.500 INFO:teuthology.orchestra.run.smithi102.stdout:===================================================================================================== 2021-05-25T11:58:18.500 INFO:teuthology.orchestra.run.smithi102.stdout:Install 8 Packages 2021-05-25T11:58:18.500 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:58:18.501 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 35 M 2021-05-25T11:58:18.501 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 194 M 2021-05-25T11:58:18.501 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2021-05-25T11:58:18.599 INFO:teuthology.orchestra.run.smithi102.stdout:(1/8): openblas-threads-0.3.3-5.el8.x86_64.rpm 43 MB/s | 4.3 MB 00:00 2021-05-25T11:58:18.659 INFO:teuthology.orchestra.run.smithi102.stdout:(2/8): python3-numpy-1.14.3-9.el8.x86_64.rpm 62 MB/s | 3.7 MB 00:00 2021-05-25T11:58:18.664 INFO:teuthology.orchestra.run.smithi102.stdout:(3/8): python3-numpy-f2py-1.14.3-9.el8.x86_64.r 41 MB/s | 225 kB 00:00 2021-05-25T11:58:18.678 INFO:teuthology.orchestra.run.smithi102.stdout:(4/8): openblas-0.3.3-5.el8.x86_64.rpm 24 MB/s | 4.3 MB 00:00 2021-05-25T11:58:18.703 INFO:teuthology.orchestra.run.smithi102.stdout:(5/8): libgfortran-8.3.1-5.1.el8.x86_64.rpm 26 MB/s | 639 kB 00:00 2021-05-25T11:58:18.713 INFO:teuthology.orchestra.run.smithi102.stdout:(6/8): libquadmath-8.3.1-5.1.el8.x86_64.rpm 16 MB/s | 169 kB 00:00 2021-05-25T11:58:18.813 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 5/8 2021-05-25T11:58:18.918 INFO:teuthology.orchestra.run.smithi102.stdout:(7/8): python3-scipy-1.0.0-20.module_el8.3.0+56 57 MB/s | 15 MB 00:00 2021-05-25T11:58:19.065 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-05-25T11:58:19.127 INFO:teuthology.orchestra.run.smithi102.stdout:(8/8): ceph-mgr-diskprediction-local-17.0.0-450 12 MB/s | 7.4 MB 00:00 2021-05-25T11:58:19.130 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2021-05-25T11:58:19.131 INFO:teuthology.orchestra.run.smithi102.stdout:Total 56 MB/s | 35 MB 00:00 2021-05-25T11:58:19.139 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2021-05-25T11:58:19.158 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2021-05-25T11:58:19.158 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2021-05-25T11:58:19.471 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2021-05-25T11:58:19.474 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2021-05-25T11:58:20.032 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2021-05-25T11:58:20.188 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libquadmath-8.3.1-5.1.el8.x86_64 1/8 2021-05-25T11:58:20.323 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: libquadmath-8.3.1-5.1.el8.x86_64 1/8 2021-05-25T11:58:20.588 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libgfortran-8.3.1-5.1.el8.x86_64 2/8 2021-05-25T11:58:21.048 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-05-25T11:58:21.084 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: libgfortran-8.3.1-5.1.el8.x86_64 2/8 2021-05-25T11:58:21.313 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : openblas-0.3.3-5.el8.x86_64 3/8 2021-05-25T11:58:21.821 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: openblas-0.3.3-5.el8.x86_64 3/8 2021-05-25T11:58:22.012 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : openblas-threads-0.3.3-5.el8.x86_64 4/8 2021-05-25T11:58:22.321 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-scipy-1.0.0-20.module_el8.3.0+562+e162826a.x 7/8 2021-05-25T11:58:22.523 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-diskprediction-local-2:17.0.0-4502.gece5ed1 8/8 2021-05-25T11:58:22.718 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: openblas-threads-0.3.3-5.el8.x86_64 4/8 2021-05-25T11:58:22.789 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.0.0-4502.gece5ed1 8/8 2021-05-25T11:58:22.789 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.0.0-4502.gece5ed1 1/8 2021-05-25T11:58:22.790 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : openblas-0.3.3-5.el8.x86_64 2/8 2021-05-25T11:58:22.790 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : openblas-threads-0.3.3-5.el8.x86_64 3/8 2021-05-25T11:58:22.790 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 4/8 2021-05-25T11:58:22.792 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 5/8 2021-05-25T11:58:22.792 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-scipy-1.0.0-20.module_el8.3.0+562+e162826a.x 6/8 2021-05-25T11:58:22.793 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libgfortran-8.3.1-5.1.el8.x86_64 7/8 2021-05-25T11:58:23.061 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 5/8 2021-05-25T11:58:23.067 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libquadmath-8.3.1-5.1.el8.x86_64 8/8 2021-05-25T11:58:23.068 INFO:teuthology.orchestra.run.smithi181.stdout:Installed products updated. 2021-05-25T11:58:23.282 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:58:23.283 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2021-05-25T11:58:23.283 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-diskprediction-local-2:17.0.0-4502.gece5ed1a.el8.noarch 2021-05-25T11:58:23.283 INFO:teuthology.orchestra.run.smithi181.stdout: libgfortran-8.3.1-5.1.el8.x86_64 2021-05-25T11:58:23.284 INFO:teuthology.orchestra.run.smithi181.stdout: libquadmath-8.3.1-5.1.el8.x86_64 2021-05-25T11:58:23.284 INFO:teuthology.orchestra.run.smithi181.stdout: openblas-0.3.3-5.el8.x86_64 2021-05-25T11:58:23.284 INFO:teuthology.orchestra.run.smithi181.stdout: openblas-threads-0.3.3-5.el8.x86_64 2021-05-25T11:58:23.284 INFO:teuthology.orchestra.run.smithi181.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2021-05-25T11:58:23.285 INFO:teuthology.orchestra.run.smithi181.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2021-05-25T11:58:23.285 INFO:teuthology.orchestra.run.smithi181.stdout: python3-scipy-1.0.0-20.module_el8.3.0+562+e162826a.x86_64 2021-05-25T11:58:23.285 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:58:23.285 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2021-05-25T11:58:23.304 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-05-25T11:58:23.390 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-mgr-rook 2021-05-25T11:58:23.927 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:34 ago on Tue 25 May 2021 11:56:49 AM UTC. 2021-05-25T11:58:24.615 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2021-05-25T11:58:24.616 INFO:teuthology.orchestra.run.smithi181.stdout:======================================================================================== 2021-05-25T11:58:24.617 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2021-05-25T11:58:24.617 INFO:teuthology.orchestra.run.smithi181.stdout:======================================================================================== 2021-05-25T11:58:24.617 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2021-05-25T11:58:24.617 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-rook noarch 2:17.0.0-4502.gece5ed1a.el8 ceph-noarch 26 k 2021-05-25T11:58:24.618 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2021-05-25T11:58:24.618 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2021-05-25T11:58:24.619 INFO:teuthology.orchestra.run.smithi181.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2021-05-25T11:58:24.619 INFO:teuthology.orchestra.run.smithi181.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2021-05-25T11:58:24.619 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jsonpatch noarch 1.21-2.el8 CentOS-AppStream 27 k 2021-05-25T11:58:24.619 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jsonpointer noarch 1.10-11.el8 CentOS-AppStream 20 k 2021-05-25T11:58:24.620 INFO:teuthology.orchestra.run.smithi181.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2021-05-25T11:58:24.620 INFO:teuthology.orchestra.run.smithi181.stdout: python3-oauthlib noarch 2.1.0-1.el8 CentOS-Base 155 k 2021-05-25T11:58:24.620 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyasn1 noarch 0.3.7-6.el8 CentOS-AppStream 126 k 2021-05-25T11:58:24.620 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 CentOS-AppStream 110 k 2021-05-25T11:58:24.620 INFO:teuthology.orchestra.run.smithi181.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 CentOS-Base 43 k 2021-05-25T11:58:24.621 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rsa noarch 3.4.2-16.el8 epel 73 k 2021-05-25T11:58:24.621 INFO:teuthology.orchestra.run.smithi181.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2021-05-25T11:58:24.621 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:58:24.621 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2021-05-25T11:58:24.622 INFO:teuthology.orchestra.run.smithi181.stdout:======================================================================================== 2021-05-25T11:58:24.622 INFO:teuthology.orchestra.run.smithi181.stdout:Install 13 Packages 2021-05-25T11:58:24.622 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:58:24.623 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 1.9 M 2021-05-25T11:58:24.623 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 24 M 2021-05-25T11:58:24.624 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2021-05-25T11:58:24.652 INFO:teuthology.orchestra.run.smithi181.stdout:(1/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 834 kB/s | 27 kB 00:00 2021-05-25T11:58:24.732 INFO:teuthology.orchestra.run.smithi181.stdout:(2/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 1.5 MB/s | 126 kB 00:00 2021-05-25T11:58:24.791 INFO:teuthology.orchestra.run.smithi181.stdout:(3/13): python3-jsonpointer-1.10-11.el8.noarch. 117 kB/s | 20 kB 00:00 2021-05-25T11:58:24.864 INFO:teuthology.orchestra.run.smithi181.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 867 kB/s | 110 kB 00:00 2021-05-25T11:58:24.923 INFO:teuthology.orchestra.run.smithi181.stdout:(5/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 1.1 MB/s | 155 kB 00:00 2021-05-25T11:58:24.969 INFO:teuthology.orchestra.run.smithi181.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 407 kB/s | 43 kB 00:00 2021-05-25T11:58:25.100 INFO:teuthology.orchestra.run.smithi181.stdout:(7/13): ceph-mgr-rook-17.0.0-4502.gece5ed1a.el8 54 kB/s | 26 kB 00:00 2021-05-25T11:58:25.101 INFO:teuthology.orchestra.run.smithi181.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 124 kB/s | 16 kB 00:00 2021-05-25T11:58:25.145 INFO:teuthology.orchestra.run.smithi181.stdout:(9/13): python3-cachetools-3.1.1-4.el8.noarch.r 149 kB/s | 33 kB 00:00 2021-05-25T11:58:25.154 INFO:teuthology.orchestra.run.smithi181.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 1.5 MB/s | 81 kB 00:00 2021-05-25T11:58:25.181 INFO:teuthology.orchestra.run.smithi181.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 2.2 MB/s | 61 kB 00:00 2021-05-25T11:58:25.233 INFO:teuthology.orchestra.run.smithi181.stdout:(12/13): python3-rsa-3.4.2-16.el8.noarch.rpm 831 kB/s | 73 kB 00:00 2021-05-25T11:58:25.278 INFO:teuthology.orchestra.run.smithi181.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 6.6 MB/s | 1.2 MB 00:00 2021-05-25T11:58:25.281 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2021-05-25T11:58:25.282 INFO:teuthology.orchestra.run.smithi181.stdout:Total 2.9 MB/s | 1.9 MB 00:00 2021-05-25T11:58:25.288 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2021-05-25T11:58:25.290 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-05-25T11:58:25.304 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2021-05-25T11:58:25.305 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2021-05-25T11:58:25.382 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2021-05-25T11:58:25.385 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2021-05-25T11:58:25.907 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2021-05-25T11:58:26.152 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2021-05-25T11:58:26.439 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2021-05-25T11:58:26.669 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-rsa-3.4.2-16.el8.noarch 3/13 2021-05-25T11:58:26.897 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2021-05-25T11:58:26.969 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-scipy-1.0.0-20.module_el8.3.0+562+e162826a.x 7/8 2021-05-25T11:58:27.090 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2021-05-25T11:58:27.162 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-diskprediction-local-2:17.0.0-4502.gece5ed1 8/8 2021-05-25T11:58:27.357 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2021-05-25T11:58:27.436 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.0.0-4502.gece5ed1 8/8 2021-05-25T11:58:27.437 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.0.0-4502.gece5ed1 1/8 2021-05-25T11:58:27.437 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : openblas-0.3.3-5.el8.x86_64 2/8 2021-05-25T11:58:27.437 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : openblas-threads-0.3.3-5.el8.x86_64 3/8 2021-05-25T11:58:27.438 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 4/8 2021-05-25T11:58:27.438 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 5/8 2021-05-25T11:58:27.439 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-scipy-1.0.0-20.module_el8.3.0+562+e162826a.x 6/8 2021-05-25T11:58:27.439 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libgfortran-8.3.1-5.1.el8.x86_64 7/8 2021-05-25T11:58:27.665 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2021-05-25T11:58:27.709 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libquadmath-8.3.1-5.1.el8.x86_64 8/8 2021-05-25T11:58:27.710 INFO:teuthology.orchestra.run.smithi102.stdout:Installed products updated. 2021-05-25T11:58:27.906 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2021-05-25T11:58:27.960 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:58:27.961 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2021-05-25T11:58:27.961 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-diskprediction-local-2:17.0.0-4502.gece5ed1a.el8.noarch 2021-05-25T11:58:27.961 INFO:teuthology.orchestra.run.smithi102.stdout: libgfortran-8.3.1-5.1.el8.x86_64 2021-05-25T11:58:27.961 INFO:teuthology.orchestra.run.smithi102.stdout: libquadmath-8.3.1-5.1.el8.x86_64 2021-05-25T11:58:27.962 INFO:teuthology.orchestra.run.smithi102.stdout: openblas-0.3.3-5.el8.x86_64 2021-05-25T11:58:27.962 INFO:teuthology.orchestra.run.smithi102.stdout: openblas-threads-0.3.3-5.el8.x86_64 2021-05-25T11:58:27.962 INFO:teuthology.orchestra.run.smithi102.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2021-05-25T11:58:27.962 INFO:teuthology.orchestra.run.smithi102.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2021-05-25T11:58:27.963 INFO:teuthology.orchestra.run.smithi102.stdout: python3-scipy-1.0.0-20.module_el8.3.0+562+e162826a.x86_64 2021-05-25T11:58:27.963 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:58:27.963 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2021-05-25T11:58:28.037 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-mgr-rook 2021-05-25T11:58:28.574 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:38 ago on Tue 25 May 2021 11:56:50 AM UTC. 2021-05-25T11:58:29.011 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2021-05-25T11:58:29.257 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2021-05-25T11:58:29.258 INFO:teuthology.orchestra.run.smithi102.stdout:======================================================================================== 2021-05-25T11:58:29.259 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2021-05-25T11:58:29.259 INFO:teuthology.orchestra.run.smithi102.stdout:======================================================================================== 2021-05-25T11:58:29.259 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2021-05-25T11:58:29.259 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-rook noarch 2:17.0.0-4502.gece5ed1a.el8 ceph-noarch 26 k 2021-05-25T11:58:29.259 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2021-05-25T11:58:29.260 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2021-05-25T11:58:29.260 INFO:teuthology.orchestra.run.smithi102.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2021-05-25T11:58:29.260 INFO:teuthology.orchestra.run.smithi102.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2021-05-25T11:58:29.260 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jsonpatch noarch 1.21-2.el8 CentOS-AppStream 27 k 2021-05-25T11:58:29.261 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jsonpointer noarch 1.10-11.el8 CentOS-AppStream 20 k 2021-05-25T11:58:29.261 INFO:teuthology.orchestra.run.smithi102.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2021-05-25T11:58:29.261 INFO:teuthology.orchestra.run.smithi102.stdout: python3-oauthlib noarch 2.1.0-1.el8 CentOS-Base 155 k 2021-05-25T11:58:29.261 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyasn1 noarch 0.3.7-6.el8 CentOS-AppStream 126 k 2021-05-25T11:58:29.261 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 CentOS-AppStream 110 k 2021-05-25T11:58:29.262 INFO:teuthology.orchestra.run.smithi102.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 CentOS-Base 43 k 2021-05-25T11:58:29.262 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rsa noarch 3.4.2-16.el8 epel 73 k 2021-05-25T11:58:29.262 INFO:teuthology.orchestra.run.smithi102.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2021-05-25T11:58:29.262 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:58:29.262 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2021-05-25T11:58:29.263 INFO:teuthology.orchestra.run.smithi102.stdout:======================================================================================== 2021-05-25T11:58:29.263 INFO:teuthology.orchestra.run.smithi102.stdout:Install 13 Packages 2021-05-25T11:58:29.263 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:58:29.264 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 1.9 M 2021-05-25T11:58:29.264 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 24 M 2021-05-25T11:58:29.264 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2021-05-25T11:58:29.270 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2021-05-25T11:58:29.293 INFO:teuthology.orchestra.run.smithi102.stdout:(1/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 851 kB/s | 27 kB 00:00 2021-05-25T11:58:29.298 INFO:teuthology.orchestra.run.smithi102.stdout:(2/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 25 MB/s | 126 kB 00:00 2021-05-25T11:58:29.298 INFO:teuthology.orchestra.run.smithi102.stdout:(3/13): python3-jsonpointer-1.10-11.el8.noarch. 544 kB/s | 20 kB 00:00 2021-05-25T11:58:29.301 INFO:teuthology.orchestra.run.smithi102.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 29 MB/s | 110 kB 00:00 2021-05-25T11:58:29.303 INFO:teuthology.orchestra.run.smithi102.stdout:(5/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 31 MB/s | 155 kB 00:00 2021-05-25T11:58:29.304 INFO:teuthology.orchestra.run.smithi102.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 16 MB/s | 43 kB 00:00 2021-05-25T11:58:29.416 INFO:teuthology.orchestra.run.smithi102.stdout:(7/13): python3-certifi-2018.10.15-7.el8.noarch 146 kB/s | 16 kB 00:00 2021-05-25T11:58:29.417 INFO:teuthology.orchestra.run.smithi102.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 292 kB/s | 33 kB 00:00 2021-05-25T11:58:29.461 INFO:teuthology.orchestra.run.smithi102.stdout:(9/13): python3-google-auth-1.1.1-10.el8.noarch 1.8 MB/s | 81 kB 00:00 2021-05-25T11:58:29.480 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2021-05-25T11:58:29.484 INFO:teuthology.orchestra.run.smithi102.stdout:(10/13): python3-rsa-3.4.2-16.el8.noarch.rpm 3.1 MB/s | 73 kB 00:00 2021-05-25T11:58:29.507 INFO:teuthology.orchestra.run.smithi102.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 2.6 MB/s | 61 kB 00:00 2021-05-25T11:58:29.545 INFO:teuthology.orchestra.run.smithi102.stdout:(12/13): ceph-mgr-rook-17.0.0-4502.gece5ed1a.el 92 kB/s | 26 kB 00:00 2021-05-25T11:58:29.594 INFO:teuthology.orchestra.run.smithi102.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 6.6 MB/s | 1.2 MB 00:00 2021-05-25T11:58:29.596 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2021-05-25T11:58:29.596 INFO:teuthology.orchestra.run.smithi102.stdout:Total 5.7 MB/s | 1.9 MB 00:00 2021-05-25T11:58:29.601 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2021-05-25T11:58:29.616 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2021-05-25T11:58:29.616 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2021-05-25T11:58:29.689 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2021-05-25T11:58:29.692 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2021-05-25T11:58:29.697 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2021-05-25T11:58:29.880 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-rook-2:17.0.0-4502.gece5ed1a.el8.noarch 13/13 2021-05-25T11:58:30.108 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-rook-2:17.0.0-4502.gece5ed1a.el8.noarch 13/13 2021-05-25T11:58:30.108 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-rook-2:17.0.0-4502.gece5ed1a.el8.noarch 1/13 2021-05-25T11:58:30.108 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2021-05-25T11:58:30.109 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2021-05-25T11:58:30.109 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2021-05-25T11:58:30.109 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2021-05-25T11:58:30.109 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2021-05-25T11:58:30.110 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2021-05-25T11:58:30.110 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2021-05-25T11:58:30.110 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2021-05-25T11:58:30.111 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2021-05-25T11:58:30.111 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2021-05-25T11:58:30.111 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-rsa-3.4.2-16.el8.noarch 12/13 2021-05-25T11:58:30.200 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2021-05-25T11:58:30.455 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2021-05-25T11:58:30.455 INFO:teuthology.orchestra.run.smithi181.stdout:Installed products updated. 2021-05-25T11:58:30.459 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2021-05-25T11:58:30.712 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2021-05-25T11:58:30.897 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:58:30.897 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2021-05-25T11:58:30.897 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-rook-2:17.0.0-4502.gece5ed1a.el8.noarch 2021-05-25T11:58:30.898 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cachetools-3.1.1-4.el8.noarch 2021-05-25T11:58:30.898 INFO:teuthology.orchestra.run.smithi181.stdout: python3-certifi-2018.10.15-7.el8.noarch 2021-05-25T11:58:30.898 INFO:teuthology.orchestra.run.smithi181.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2021-05-25T11:58:30.898 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jsonpatch-1.21-2.el8.noarch 2021-05-25T11:58:30.898 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jsonpointer-1.10-11.el8.noarch 2021-05-25T11:58:30.899 INFO:teuthology.orchestra.run.smithi181.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2021-05-25T11:58:30.899 INFO:teuthology.orchestra.run.smithi181.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2021-05-25T11:58:30.899 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2021-05-25T11:58:30.899 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2021-05-25T11:58:30.900 INFO:teuthology.orchestra.run.smithi181.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2021-05-25T11:58:30.900 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rsa-3.4.2-16.el8.noarch 2021-05-25T11:58:30.900 INFO:teuthology.orchestra.run.smithi181.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2021-05-25T11:58:30.900 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:58:30.900 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2021-05-25T11:58:30.975 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-rsa-3.4.2-16.el8.noarch 3/13 2021-05-25T11:58:31.183 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-mgr-cephadm 2021-05-25T11:58:31.728 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:42 ago on Tue 25 May 2021 11:56:49 AM UTC. 2021-05-25T11:58:32.015 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2021-05-25T11:58:32.390 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2021-05-25T11:58:32.391 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2021-05-25T11:58:32.391 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2021-05-25T11:58:32.391 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2021-05-25T11:58:32.392 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2021-05-25T11:58:32.392 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-cephadm noarch 2:17.0.0-4502.gece5ed1a.el8 ceph-noarch 85 k 2021-05-25T11:58:32.392 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2021-05-25T11:58:32.392 INFO:teuthology.orchestra.run.smithi181.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2021-05-25T11:58:32.393 INFO:teuthology.orchestra.run.smithi181.stdout: python3-babel noarch 2.5.1-5.el8 CentOS-AppStream 4.8 M 2021-05-25T11:58:32.393 INFO:teuthology.orchestra.run.smithi181.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2021-05-25T11:58:32.393 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 CentOS-AppStream 538 k 2021-05-25T11:58:32.393 INFO:teuthology.orchestra.run.smithi181.stdout: python3-remoto noarch 1.1.4-4.el8 epel 48 k 2021-05-25T11:58:32.393 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:58:32.394 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2021-05-25T11:58:32.394 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2021-05-25T11:58:32.394 INFO:teuthology.orchestra.run.smithi181.stdout:Install 6 Packages 2021-05-25T11:58:32.394 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:58:32.395 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 5.8 M 2021-05-25T11:58:32.395 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 24 M 2021-05-25T11:58:32.396 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2021-05-25T11:58:32.449 INFO:teuthology.orchestra.run.smithi181.stdout:(1/6): python3-jinja2-2.10.1-2.el8_0.noarch.rpm 9.5 MB/s | 538 kB 00:00 2021-05-25T11:58:32.489 INFO:teuthology.orchestra.run.smithi181.stdout:(2/6): python3-babel-2.5.1-5.el8.noarch.rpm 50 MB/s | 4.8 MB 00:00 2021-05-25T11:58:32.516 INFO:teuthology.orchestra.run.smithi181.stdout:(3/6): python3-apipkg-1.5-6.el8.noarch.rpm 315 kB/s | 21 kB 00:00 2021-05-25T11:58:32.537 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2021-05-25T11:58:32.560 INFO:teuthology.orchestra.run.smithi181.stdout:(4/6): python3-remoto-1.1.4-4.el8.noarch.rpm 1.1 MB/s | 48 kB 00:00 2021-05-25T11:58:32.571 INFO:teuthology.orchestra.run.smithi181.stdout:(5/6): ceph-mgr-cephadm-17.0.0-4502.gece5ed1a.e 479 kB/s | 85 kB 00:00 2021-05-25T11:58:32.684 INFO:teuthology.orchestra.run.smithi181.stdout:(6/6): python3-execnet-1.7.1-1.el8.noarch.rpm 1.8 MB/s | 355 kB 00:00 2021-05-25T11:58:32.686 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2021-05-25T11:58:32.686 INFO:teuthology.orchestra.run.smithi181.stdout:Total 20 MB/s | 5.8 MB 00:00 2021-05-25T11:58:32.689 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2021-05-25T11:58:32.698 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2021-05-25T11:58:32.698 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2021-05-25T11:58:32.778 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2021-05-25T11:58:32.781 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2021-05-25T11:58:32.797 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2021-05-25T11:58:33.073 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2021-05-25T11:58:33.100 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2021-05-25T11:58:33.311 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2021-05-25T11:58:33.326 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2021-05-25T11:58:33.576 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2021-05-25T11:58:34.435 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2021-05-25T11:58:34.581 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-remoto-1.1.4-4.el8.noarch 3/6 2021-05-25T11:58:34.710 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2021-05-25T11:58:34.952 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2021-05-25T11:58:34.998 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 4/6 2021-05-25T11:58:35.186 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2021-05-25T11:58:35.261 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 5/6 2021-05-25T11:58:35.361 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-rook-2:17.0.0-4502.gece5ed1a.el8.noarch 13/13 2021-05-25T11:58:35.504 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-cephadm-2:17.0.0-4502.gece5ed1a.el8.noarch 6/6 2021-05-25T11:58:35.607 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mgr-rook-2:17.0.0-4502.gece5ed1a.el8.noarch 13/13 2021-05-25T11:58:35.607 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-rook-2:17.0.0-4502.gece5ed1a.el8.noarch 1/13 2021-05-25T11:58:35.607 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2021-05-25T11:58:35.607 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2021-05-25T11:58:35.608 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2021-05-25T11:58:35.608 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2021-05-25T11:58:35.608 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2021-05-25T11:58:35.608 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2021-05-25T11:58:35.609 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2021-05-25T11:58:35.609 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2021-05-25T11:58:35.609 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2021-05-25T11:58:35.609 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2021-05-25T11:58:35.610 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-rsa-3.4.2-16.el8.noarch 12/13 2021-05-25T11:58:35.746 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.0.0-4502.gece5ed1a.el8.noarch 6/6 2021-05-25T11:58:35.746 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-cephadm-2:17.0.0-4502.gece5ed1a.el8.noarch 1/6 2021-05-25T11:58:35.747 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 2/6 2021-05-25T11:58:35.747 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 3/6 2021-05-25T11:58:35.747 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2021-05-25T11:58:35.747 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2021-05-25T11:58:35.932 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2021-05-25T11:58:35.933 INFO:teuthology.orchestra.run.smithi102.stdout:Installed products updated. 2021-05-25T11:58:36.017 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-remoto-1.1.4-4.el8.noarch 6/6 2021-05-25T11:58:36.018 INFO:teuthology.orchestra.run.smithi181.stdout:Installed products updated. 2021-05-25T11:58:36.146 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:58:36.146 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2021-05-25T11:58:36.146 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-rook-2:17.0.0-4502.gece5ed1a.el8.noarch 2021-05-25T11:58:36.147 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cachetools-3.1.1-4.el8.noarch 2021-05-25T11:58:36.147 INFO:teuthology.orchestra.run.smithi102.stdout: python3-certifi-2018.10.15-7.el8.noarch 2021-05-25T11:58:36.147 INFO:teuthology.orchestra.run.smithi102.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2021-05-25T11:58:36.148 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jsonpatch-1.21-2.el8.noarch 2021-05-25T11:58:36.148 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jsonpointer-1.10-11.el8.noarch 2021-05-25T11:58:36.148 INFO:teuthology.orchestra.run.smithi102.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2021-05-25T11:58:36.148 INFO:teuthology.orchestra.run.smithi102.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2021-05-25T11:58:36.149 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2021-05-25T11:58:36.149 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2021-05-25T11:58:36.149 INFO:teuthology.orchestra.run.smithi102.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2021-05-25T11:58:36.149 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rsa-3.4.2-16.el8.noarch 2021-05-25T11:58:36.150 INFO:teuthology.orchestra.run.smithi102.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2021-05-25T11:58:36.150 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:58:36.150 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2021-05-25T11:58:36.243 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:58:36.244 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2021-05-25T11:58:36.244 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-cephadm-2:17.0.0-4502.gece5ed1a.el8.noarch 2021-05-25T11:58:36.244 INFO:teuthology.orchestra.run.smithi181.stdout: python3-apipkg-1.5-6.el8.noarch 2021-05-25T11:58:36.244 INFO:teuthology.orchestra.run.smithi181.stdout: python3-babel-2.5.1-5.el8.noarch 2021-05-25T11:58:36.245 INFO:teuthology.orchestra.run.smithi181.stdout: python3-execnet-1.7.1-1.el8.noarch 2021-05-25T11:58:36.245 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2021-05-25T11:58:36.245 INFO:teuthology.orchestra.run.smithi181.stdout: python3-remoto-1.1.4-4.el8.noarch 2021-05-25T11:58:36.245 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:58:36.245 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2021-05-25T11:58:36.267 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-mgr-cephadm 2021-05-25T11:58:36.621 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-fuse 2021-05-25T11:58:36.805 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:46 ago on Tue 25 May 2021 11:56:50 AM UTC. 2021-05-25T11:58:37.165 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:48 ago on Tue 25 May 2021 11:56:49 AM UTC. 2021-05-25T11:58:37.507 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2021-05-25T11:58:37.507 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-05-25T11:58:37.508 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2021-05-25T11:58:37.508 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-05-25T11:58:37.508 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2021-05-25T11:58:37.508 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-cephadm noarch 2:17.0.0-4502.gece5ed1a.el8 ceph-noarch 85 k 2021-05-25T11:58:37.509 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2021-05-25T11:58:37.509 INFO:teuthology.orchestra.run.smithi102.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2021-05-25T11:58:37.509 INFO:teuthology.orchestra.run.smithi102.stdout: python3-babel noarch 2.5.1-5.el8 CentOS-AppStream 4.8 M 2021-05-25T11:58:37.509 INFO:teuthology.orchestra.run.smithi102.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2021-05-25T11:58:37.510 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 CentOS-AppStream 538 k 2021-05-25T11:58:37.510 INFO:teuthology.orchestra.run.smithi102.stdout: python3-remoto noarch 1.1.4-4.el8 epel 48 k 2021-05-25T11:58:37.510 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:58:37.510 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2021-05-25T11:58:37.511 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-05-25T11:58:37.511 INFO:teuthology.orchestra.run.smithi102.stdout:Install 6 Packages 2021-05-25T11:58:37.511 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:58:37.512 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 5.8 M 2021-05-25T11:58:37.512 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 24 M 2021-05-25T11:58:37.512 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2021-05-25T11:58:37.561 INFO:teuthology.orchestra.run.smithi102.stdout:(1/6): python3-jinja2-2.10.1-2.el8_0.noarch.rpm 10 MB/s | 538 kB 00:00 2021-05-25T11:58:37.603 INFO:teuthology.orchestra.run.smithi102.stdout:(2/6): python3-babel-2.5.1-5.el8.noarch.rpm 51 MB/s | 4.8 MB 00:00 2021-05-25T11:58:37.642 INFO:teuthology.orchestra.run.smithi102.stdout:(3/6): python3-apipkg-1.5-6.el8.noarch.rpm 262 kB/s | 21 kB 00:00 2021-05-25T11:58:37.653 INFO:teuthology.orchestra.run.smithi102.stdout:(4/6): ceph-mgr-cephadm-17.0.0-4502.gece5ed1a.e 591 kB/s | 85 kB 00:00 2021-05-25T11:58:37.696 INFO:teuthology.orchestra.run.smithi102.stdout:(5/6): python3-remoto-1.1.4-4.el8.noarch.rpm 898 kB/s | 48 kB 00:00 2021-05-25T11:58:37.795 INFO:teuthology.orchestra.run.smithi102.stdout:(6/6): python3-execnet-1.7.1-1.el8.noarch.rpm 1.8 MB/s | 355 kB 00:00 2021-05-25T11:58:37.798 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2021-05-25T11:58:37.798 INFO:teuthology.orchestra.run.smithi102.stdout:Total 20 MB/s | 5.8 MB 00:00 2021-05-25T11:58:37.800 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2021-05-25T11:58:37.810 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2021-05-25T11:58:37.810 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2021-05-25T11:58:37.825 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2021-05-25T11:58:37.825 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2021-05-25T11:58:37.826 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2021-05-25T11:58:37.826 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2021-05-25T11:58:37.826 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2021-05-25T11:58:37.826 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-fuse x86_64 2:17.0.0-4502.gece5ed1a.el8 ceph 838 k 2021-05-25T11:58:37.827 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2021-05-25T11:58:37.827 INFO:teuthology.orchestra.run.smithi181.stdout: fuse x86_64 2.9.7-12.el8 CentOS-Base 82 k 2021-05-25T11:58:37.827 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:58:37.827 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2021-05-25T11:58:37.828 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2021-05-25T11:58:37.828 INFO:teuthology.orchestra.run.smithi181.stdout:Install 2 Packages 2021-05-25T11:58:37.828 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:58:37.829 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 920 k 2021-05-25T11:58:37.829 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 3.3 M 2021-05-25T11:58:37.829 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2021-05-25T11:58:37.860 INFO:teuthology.orchestra.run.smithi181.stdout:(1/2): fuse-2.9.7-12.el8.x86_64.rpm 2.5 MB/s | 82 kB 00:00 2021-05-25T11:58:37.889 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2021-05-25T11:58:37.892 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2021-05-25T11:58:38.069 INFO:teuthology.orchestra.run.smithi181.stdout:(2/2): ceph-fuse-17.0.0-4502.gece5ed1a.el8.x86_ 3.4 MB/s | 838 kB 00:00 2021-05-25T11:58:38.071 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2021-05-25T11:58:38.071 INFO:teuthology.orchestra.run.smithi181.stdout:Total 3.7 MB/s | 920 kB 00:00 2021-05-25T11:58:38.072 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2021-05-25T11:58:38.093 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2021-05-25T11:58:38.094 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2021-05-25T11:58:38.170 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2021-05-25T11:58:38.174 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2021-05-25T11:58:38.271 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2021-05-25T11:58:38.333 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2021-05-25T11:58:38.574 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2021-05-25T11:58:38.657 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : fuse-2.9.7-12.el8.x86_64 1/2 2021-05-25T11:58:38.831 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2021-05-25T11:58:38.887 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-fuse-2:17.0.0-4502.gece5ed1a.el8.x86_64 2/2 2021-05-25T11:58:39.269 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-fuse-2:17.0.0-4502.gece5ed1a.el8.x86_64 2/2 2021-05-25T11:58:39.270 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-fuse-2:17.0.0-4502.gece5ed1a.el8.x86_64 1/2 2021-05-25T11:58:39.519 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/2 2021-05-25T11:58:39.519 INFO:teuthology.orchestra.run.smithi181.stdout:Installed products updated. 2021-05-25T11:58:39.753 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:58:39.753 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2021-05-25T11:58:39.754 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-fuse-2:17.0.0-4502.gece5ed1a.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2021-05-25T11:58:39.754 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:58:39.754 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2021-05-25T11:58:39.780 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-remoto-1.1.4-4.el8.noarch 3/6 2021-05-25T11:58:39.955 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install librados-devel 2021-05-25T11:58:40.120 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 4/6 2021-05-25T11:58:40.359 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 5/6 2021-05-25T11:58:40.506 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:51 ago on Tue 25 May 2021 11:56:49 AM UTC. 2021-05-25T11:58:40.610 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-cephadm-2:17.0.0-4502.gece5ed1a.el8.noarch 6/6 2021-05-25T11:58:40.847 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.0.0-4502.gece5ed1a.el8.noarch 6/6 2021-05-25T11:58:40.847 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-cephadm-2:17.0.0-4502.gece5ed1a.el8.noarch 1/6 2021-05-25T11:58:40.848 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 2/6 2021-05-25T11:58:40.848 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 3/6 2021-05-25T11:58:40.848 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2021-05-25T11:58:40.848 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2021-05-25T11:58:41.135 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-remoto-1.1.4-4.el8.noarch 6/6 2021-05-25T11:58:41.135 INFO:teuthology.orchestra.run.smithi102.stdout:Installed products updated. 2021-05-25T11:58:41.176 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2021-05-25T11:58:41.177 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2021-05-25T11:58:41.177 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repo Size 2021-05-25T11:58:41.178 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2021-05-25T11:58:41.178 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2021-05-25T11:58:41.178 INFO:teuthology.orchestra.run.smithi181.stdout: librados-devel x86_64 2:17.0.0-4502.gece5ed1a.el8 ceph 125 k 2021-05-25T11:58:41.178 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:58:41.179 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2021-05-25T11:58:41.179 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2021-05-25T11:58:41.179 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2021-05-25T11:58:41.179 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:58:41.180 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 125 k 2021-05-25T11:58:41.180 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 439 k 2021-05-25T11:58:41.180 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2021-05-25T11:58:41.346 INFO:teuthology.orchestra.run.smithi181.stdout:librados-devel-17.0.0-4502.gece5ed1a.el8.x86_64 744 kB/s | 125 kB 00:00 2021-05-25T11:58:41.347 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2021-05-25T11:58:41.347 INFO:teuthology.orchestra.run.smithi181.stdout:Total 738 kB/s | 125 kB 00:00 2021-05-25T11:58:41.347 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2021-05-25T11:58:41.354 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2021-05-25T11:58:41.354 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2021-05-25T11:58:41.375 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:58:41.375 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2021-05-25T11:58:41.375 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-cephadm-2:17.0.0-4502.gece5ed1a.el8.noarch 2021-05-25T11:58:41.376 INFO:teuthology.orchestra.run.smithi102.stdout: python3-apipkg-1.5-6.el8.noarch 2021-05-25T11:58:41.376 INFO:teuthology.orchestra.run.smithi102.stdout: python3-babel-2.5.1-5.el8.noarch 2021-05-25T11:58:41.376 INFO:teuthology.orchestra.run.smithi102.stdout: python3-execnet-1.7.1-1.el8.noarch 2021-05-25T11:58:41.376 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2021-05-25T11:58:41.377 INFO:teuthology.orchestra.run.smithi102.stdout: python3-remoto-1.1.4-4.el8.noarch 2021-05-25T11:58:41.377 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:58:41.377 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2021-05-25T11:58:41.406 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2021-05-25T11:58:41.409 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2021-05-25T11:58:41.469 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-fuse 2021-05-25T11:58:41.520 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2021-05-25T11:58:41.760 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : librados-devel-2:17.0.0-4502.gece5ed1a.el8.x86_64 1/1 2021-05-25T11:58:41.977 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librados-devel-2:17.0.0-4502.gece5ed1a.el8.x86_64 1/1 2021-05-25T11:58:42.007 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:52 ago on Tue 25 May 2021 11:56:50 AM UTC. 2021-05-25T11:58:42.267 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librados-devel-2:17.0.0-4502.gece5ed1a.el8.x86_64 1/1 2021-05-25T11:58:42.267 INFO:teuthology.orchestra.run.smithi181.stdout:Installed products updated. 2021-05-25T11:58:42.484 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:58:42.484 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2021-05-25T11:58:42.484 INFO:teuthology.orchestra.run.smithi181.stdout: librados-devel-2:17.0.0-4502.gece5ed1a.el8.x86_64 2021-05-25T11:58:42.484 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:58:42.485 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2021-05-25T11:58:42.657 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2021-05-25T11:58:42.658 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-05-25T11:58:42.659 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2021-05-25T11:58:42.659 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-05-25T11:58:42.659 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2021-05-25T11:58:42.659 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-fuse x86_64 2:17.0.0-4502.gece5ed1a.el8 ceph 838 k 2021-05-25T11:58:42.660 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2021-05-25T11:58:42.660 INFO:teuthology.orchestra.run.smithi102.stdout: fuse x86_64 2.9.7-12.el8 CentOS-Base 82 k 2021-05-25T11:58:42.660 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:58:42.661 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2021-05-25T11:58:42.661 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-05-25T11:58:42.661 INFO:teuthology.orchestra.run.smithi102.stdout:Install 2 Packages 2021-05-25T11:58:42.662 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:58:42.662 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 920 k 2021-05-25T11:58:42.662 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 3.3 M 2021-05-25T11:58:42.663 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2021-05-25T11:58:42.677 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install libcephfs2 2021-05-25T11:58:42.692 INFO:teuthology.orchestra.run.smithi102.stdout:(1/2): fuse-2.9.7-12.el8.x86_64.rpm 2.5 MB/s | 82 kB 00:00 2021-05-25T11:58:42.897 INFO:teuthology.orchestra.run.smithi102.stdout:(2/2): ceph-fuse-17.0.0-4502.gece5ed1a.el8.x86_ 3.4 MB/s | 838 kB 00:00 2021-05-25T11:58:42.898 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2021-05-25T11:58:42.899 INFO:teuthology.orchestra.run.smithi102.stdout:Total 3.7 MB/s | 920 kB 00:00 2021-05-25T11:58:42.900 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2021-05-25T11:58:42.921 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2021-05-25T11:58:42.922 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2021-05-25T11:58:42.997 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2021-05-25T11:58:43.000 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2021-05-25T11:58:43.162 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2021-05-25T11:58:43.224 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:54 ago on Tue 25 May 2021 11:56:49 AM UTC. 2021-05-25T11:58:43.517 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : fuse-2.9.7-12.el8.x86_64 1/2 2021-05-25T11:58:43.709 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-fuse-2:17.0.0-4502.gece5ed1a.el8.x86_64 2/2 2021-05-25T11:58:43.757 INFO:teuthology.orchestra.run.smithi181.stdout:Package libcephfs2-2:17.0.0-4502.gece5ed1a.el8.x86_64 is already installed. 2021-05-25T11:58:43.926 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2021-05-25T11:58:43.927 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2021-05-25T11:58:43.927 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2021-05-25T11:58:43.988 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install libcephfs-devel 2021-05-25T11:58:44.063 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-fuse-2:17.0.0-4502.gece5ed1a.el8.x86_64 2/2 2021-05-25T11:58:44.063 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-fuse-2:17.0.0-4502.gece5ed1a.el8.x86_64 1/2 2021-05-25T11:58:44.326 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/2 2021-05-25T11:58:44.326 INFO:teuthology.orchestra.run.smithi102.stdout:Installed products updated. 2021-05-25T11:58:44.532 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:55 ago on Tue 25 May 2021 11:56:49 AM UTC. 2021-05-25T11:58:44.568 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:58:44.568 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2021-05-25T11:58:44.569 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-fuse-2:17.0.0-4502.gece5ed1a.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2021-05-25T11:58:44.569 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:58:44.569 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2021-05-25T11:58:44.744 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install librados-devel 2021-05-25T11:58:45.240 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2021-05-25T11:58:45.241 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2021-05-25T11:58:45.242 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repo Size 2021-05-25T11:58:45.242 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2021-05-25T11:58:45.242 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2021-05-25T11:58:45.242 INFO:teuthology.orchestra.run.smithi181.stdout: libcephfs-devel x86_64 2:17.0.0-4502.gece5ed1a.el8 ceph 24 k 2021-05-25T11:58:45.242 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:58:45.243 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2021-05-25T11:58:45.243 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2021-05-25T11:58:45.243 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2021-05-25T11:58:45.243 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:58:45.244 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 24 k 2021-05-25T11:58:45.244 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 101 k 2021-05-25T11:58:45.244 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2021-05-25T11:58:45.295 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:55 ago on Tue 25 May 2021 11:56:50 AM UTC. 2021-05-25T11:58:45.364 INFO:teuthology.orchestra.run.smithi181.stdout:libcephfs-devel-17.0.0-4502.gece5ed1a.el8.x86_6 195 kB/s | 24 kB 00:00 2021-05-25T11:58:45.365 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2021-05-25T11:58:45.365 INFO:teuthology.orchestra.run.smithi181.stdout:Total 193 kB/s | 24 kB 00:00 2021-05-25T11:58:45.365 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2021-05-25T11:58:45.369 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2021-05-25T11:58:45.369 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2021-05-25T11:58:45.371 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2021-05-25T11:58:45.374 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2021-05-25T11:58:45.452 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2021-05-25T11:58:45.668 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libcephfs-devel-2:17.0.0-4502.gece5ed1a.el8.x86_64 1/1 2021-05-25T11:58:45.868 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libcephfs-devel-2:17.0.0-4502.gece5ed1a.el8.x86_64 1/1 2021-05-25T11:58:45.950 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2021-05-25T11:58:45.951 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-05-25T11:58:45.951 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repo Size 2021-05-25T11:58:45.952 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-05-25T11:58:45.952 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2021-05-25T11:58:45.952 INFO:teuthology.orchestra.run.smithi102.stdout: librados-devel x86_64 2:17.0.0-4502.gece5ed1a.el8 ceph 125 k 2021-05-25T11:58:45.952 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:58:45.953 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2021-05-25T11:58:45.953 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-05-25T11:58:45.953 INFO:teuthology.orchestra.run.smithi102.stdout:Install 1 Package 2021-05-25T11:58:45.953 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:58:45.953 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 125 k 2021-05-25T11:58:45.954 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 439 k 2021-05-25T11:58:45.954 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2021-05-25T11:58:46.091 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libcephfs-devel-2:17.0.0-4502.gece5ed1a.el8.x86_64 1/1 2021-05-25T11:58:46.092 INFO:teuthology.orchestra.run.smithi181.stdout:Installed products updated. 2021-05-25T11:58:46.119 INFO:teuthology.orchestra.run.smithi102.stdout:librados-devel-17.0.0-4502.gece5ed1a.el8.x86_64 746 kB/s | 125 kB 00:00 2021-05-25T11:58:46.120 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2021-05-25T11:58:46.120 INFO:teuthology.orchestra.run.smithi102.stdout:Total 740 kB/s | 125 kB 00:00 2021-05-25T11:58:46.121 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2021-05-25T11:58:46.127 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2021-05-25T11:58:46.128 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2021-05-25T11:58:46.181 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2021-05-25T11:58:46.183 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2021-05-25T11:58:46.318 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2021-05-25T11:58:46.319 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:58:46.319 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2021-05-25T11:58:46.319 INFO:teuthology.orchestra.run.smithi181.stdout: libcephfs-devel-2:17.0.0-4502.gece5ed1a.el8.x86_64 2021-05-25T11:58:46.320 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:58:46.320 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2021-05-25T11:58:46.384 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install librados2 2021-05-25T11:58:46.576 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : librados-devel-2:17.0.0-4502.gece5ed1a.el8.x86_64 1/1 2021-05-25T11:58:46.819 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librados-devel-2:17.0.0-4502.gece5ed1a.el8.x86_64 1/1 2021-05-25T11:58:46.928 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:57 ago on Tue 25 May 2021 11:56:49 AM UTC. 2021-05-25T11:58:47.053 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librados-devel-2:17.0.0-4502.gece5ed1a.el8.x86_64 1/1 2021-05-25T11:58:47.053 INFO:teuthology.orchestra.run.smithi102.stdout:Installed products updated. 2021-05-25T11:58:47.285 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:58:47.285 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2021-05-25T11:58:47.285 INFO:teuthology.orchestra.run.smithi102.stdout: librados-devel-2:17.0.0-4502.gece5ed1a.el8.x86_64 2021-05-25T11:58:47.285 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:58:47.286 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2021-05-25T11:58:47.423 INFO:teuthology.orchestra.run.smithi181.stdout:Package librados2-2:17.0.0-4502.gece5ed1a.el8.x86_64 is already installed. 2021-05-25T11:58:47.485 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install libcephfs2 2021-05-25T11:58:47.592 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2021-05-25T11:58:47.593 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2021-05-25T11:58:47.593 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2021-05-25T11:58:47.905 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install librbd1 2021-05-25T11:58:48.036 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:58 ago on Tue 25 May 2021 11:56:50 AM UTC. 2021-05-25T11:58:48.455 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:59 ago on Tue 25 May 2021 11:56:49 AM UTC. 2021-05-25T11:58:48.751 INFO:teuthology.orchestra.run.smithi102.stdout:Package libcephfs2-2:17.0.0-4502.gece5ed1a.el8.x86_64 is already installed. 2021-05-25T11:58:48.915 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2021-05-25T11:58:48.916 INFO:teuthology.orchestra.run.smithi102.stdout:Nothing to do. 2021-05-25T11:58:48.916 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2021-05-25T11:58:48.939 INFO:teuthology.orchestra.run.smithi181.stdout:Package librbd1-2:17.0.0-4502.gece5ed1a.el8.x86_64 is already installed. 2021-05-25T11:58:48.986 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install libcephfs-devel 2021-05-25T11:58:49.108 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2021-05-25T11:58:49.109 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2021-05-25T11:58:49.109 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2021-05-25T11:58:49.171 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install python3-rados 2021-05-25T11:58:49.527 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:59 ago on Tue 25 May 2021 11:56:50 AM UTC. 2021-05-25T11:58:49.715 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:02:00 ago on Tue 25 May 2021 11:56:49 AM UTC. 2021-05-25T11:58:50.200 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2021-05-25T11:58:50.201 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-05-25T11:58:50.202 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repo Size 2021-05-25T11:58:50.202 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-05-25T11:58:50.202 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2021-05-25T11:58:50.202 INFO:teuthology.orchestra.run.smithi102.stdout: libcephfs-devel x86_64 2:17.0.0-4502.gece5ed1a.el8 ceph 24 k 2021-05-25T11:58:50.202 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:58:50.203 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2021-05-25T11:58:50.203 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-05-25T11:58:50.203 INFO:teuthology.orchestra.run.smithi102.stdout:Install 1 Package 2021-05-25T11:58:50.203 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:58:50.204 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 24 k 2021-05-25T11:58:50.204 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 101 k 2021-05-25T11:58:50.204 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2021-05-25T11:58:50.222 INFO:teuthology.orchestra.run.smithi181.stdout:Package python3-rados-2:17.0.0-4502.gece5ed1a.el8.x86_64 is already installed. 2021-05-25T11:58:50.323 INFO:teuthology.orchestra.run.smithi102.stdout:libcephfs-devel-17.0.0-4502.gece5ed1a.el8.x86_6 196 kB/s | 24 kB 00:00 2021-05-25T11:58:50.324 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2021-05-25T11:58:50.324 INFO:teuthology.orchestra.run.smithi102.stdout:Total 194 kB/s | 24 kB 00:00 2021-05-25T11:58:50.324 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2021-05-25T11:58:50.328 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2021-05-25T11:58:50.329 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2021-05-25T11:58:50.330 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2021-05-25T11:58:50.334 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2021-05-25T11:58:50.383 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2021-05-25T11:58:50.391 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2021-05-25T11:58:50.392 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2021-05-25T11:58:50.392 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2021-05-25T11:58:50.454 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install python3-rgw 2021-05-25T11:58:50.574 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libcephfs-devel-2:17.0.0-4502.gece5ed1a.el8.x86_64 1/1 2021-05-25T11:58:50.785 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: libcephfs-devel-2:17.0.0-4502.gece5ed1a.el8.x86_64 1/1 2021-05-25T11:58:50.998 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:02:01 ago on Tue 25 May 2021 11:56:49 AM UTC. 2021-05-25T11:58:51.004 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libcephfs-devel-2:17.0.0-4502.gece5ed1a.el8.x86_64 1/1 2021-05-25T11:58:51.004 INFO:teuthology.orchestra.run.smithi102.stdout:Installed products updated. 2021-05-25T11:58:51.243 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:58:51.243 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2021-05-25T11:58:51.244 INFO:teuthology.orchestra.run.smithi102.stdout: libcephfs-devel-2:17.0.0-4502.gece5ed1a.el8.x86_64 2021-05-25T11:58:51.244 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:58:51.244 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2021-05-25T11:58:51.328 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install librados2 2021-05-25T11:58:51.489 INFO:teuthology.orchestra.run.smithi181.stdout:Package python3-rgw-2:17.0.0-4502.gece5ed1a.el8.x86_64 is already installed. 2021-05-25T11:58:51.657 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2021-05-25T11:58:51.658 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2021-05-25T11:58:51.658 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2021-05-25T11:58:51.722 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install python3-cephfs 2021-05-25T11:58:51.872 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:02:01 ago on Tue 25 May 2021 11:56:50 AM UTC. 2021-05-25T11:58:52.263 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:02:03 ago on Tue 25 May 2021 11:56:49 AM UTC. 2021-05-25T11:58:52.372 INFO:teuthology.orchestra.run.smithi102.stdout:Package librados2-2:17.0.0-4502.gece5ed1a.el8.x86_64 is already installed. 2021-05-25T11:58:52.543 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2021-05-25T11:58:52.544 INFO:teuthology.orchestra.run.smithi102.stdout:Nothing to do. 2021-05-25T11:58:52.544 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2021-05-25T11:58:52.621 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install librbd1 2021-05-25T11:58:52.764 INFO:teuthology.orchestra.run.smithi181.stdout:Package python3-cephfs-2:17.0.0-4502.gece5ed1a.el8.x86_64 is already installed. 2021-05-25T11:58:52.938 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2021-05-25T11:58:52.939 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2021-05-25T11:58:52.939 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2021-05-25T11:58:53.007 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install python3-rbd 2021-05-25T11:58:53.162 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:02:03 ago on Tue 25 May 2021 11:56:50 AM UTC. 2021-05-25T11:58:53.556 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:02:04 ago on Tue 25 May 2021 11:56:49 AM UTC. 2021-05-25T11:58:53.672 INFO:teuthology.orchestra.run.smithi102.stdout:Package librbd1-2:17.0.0-4502.gece5ed1a.el8.x86_64 is already installed. 2021-05-25T11:58:53.846 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2021-05-25T11:58:53.847 INFO:teuthology.orchestra.run.smithi102.stdout:Nothing to do. 2021-05-25T11:58:53.847 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2021-05-25T11:58:53.930 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install python3-rados 2021-05-25T11:58:54.397 INFO:teuthology.orchestra.run.smithi181.stdout:Package python3-rbd-2:17.0.0-4502.gece5ed1a.el8.x86_64 is already installed. 2021-05-25T11:58:54.475 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:02:04 ago on Tue 25 May 2021 11:56:50 AM UTC. 2021-05-25T11:58:54.566 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2021-05-25T11:58:54.567 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2021-05-25T11:58:54.567 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2021-05-25T11:58:54.628 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install rbd-fuse 2021-05-25T11:58:55.171 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:02:06 ago on Tue 25 May 2021 11:56:49 AM UTC. 2021-05-25T11:58:55.197 INFO:teuthology.orchestra.run.smithi102.stdout:Package python3-rados-2:17.0.0-4502.gece5ed1a.el8.x86_64 is already installed. 2021-05-25T11:58:55.371 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2021-05-25T11:58:55.372 INFO:teuthology.orchestra.run.smithi102.stdout:Nothing to do. 2021-05-25T11:58:55.372 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2021-05-25T11:58:55.454 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install python3-rgw 2021-05-25T11:58:55.857 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2021-05-25T11:58:55.858 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2021-05-25T11:58:55.858 INFO:teuthology.orchestra.run.smithi181.stdout: Package Architecture Version Repository Size 2021-05-25T11:58:55.858 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2021-05-25T11:58:55.859 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2021-05-25T11:58:55.859 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-fuse x86_64 2:17.0.0-4502.gece5ed1a.el8 ceph 90 k 2021-05-25T11:58:55.859 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:58:55.860 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2021-05-25T11:58:55.860 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2021-05-25T11:58:55.860 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2021-05-25T11:58:55.861 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:58:55.861 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 90 k 2021-05-25T11:58:55.861 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 320 k 2021-05-25T11:58:55.861 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2021-05-25T11:58:55.996 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:02:05 ago on Tue 25 May 2021 11:56:50 AM UTC. 2021-05-25T11:58:56.026 INFO:teuthology.orchestra.run.smithi181.stdout:rbd-fuse-17.0.0-4502.gece5ed1a.el8.x86_64.rpm 538 kB/s | 90 kB 00:00 2021-05-25T11:58:56.027 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2021-05-25T11:58:56.028 INFO:teuthology.orchestra.run.smithi181.stdout:Total 534 kB/s | 90 kB 00:00 2021-05-25T11:58:56.028 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2021-05-25T11:58:56.039 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2021-05-25T11:58:56.039 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2021-05-25T11:58:56.090 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2021-05-25T11:58:56.093 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2021-05-25T11:58:56.212 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2021-05-25T11:58:56.416 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : rbd-fuse-2:17.0.0-4502.gece5ed1a.el8.x86_64 1/1 2021-05-25T11:58:56.497 INFO:teuthology.orchestra.run.smithi102.stdout:Package python3-rgw-2:17.0.0-4502.gece5ed1a.el8.x86_64 is already installed. 2021-05-25T11:58:56.661 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: rbd-fuse-2:17.0.0-4502.gece5ed1a.el8.x86_64 1/1 2021-05-25T11:58:56.666 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2021-05-25T11:58:56.667 INFO:teuthology.orchestra.run.smithi102.stdout:Nothing to do. 2021-05-25T11:58:56.668 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2021-05-25T11:58:56.738 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install python3-cephfs 2021-05-25T11:58:56.914 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : rbd-fuse-2:17.0.0-4502.gece5ed1a.el8.x86_64 1/1 2021-05-25T11:58:56.915 INFO:teuthology.orchestra.run.smithi181.stdout:Installed products updated. 2021-05-25T11:58:57.187 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:58:57.187 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2021-05-25T11:58:57.187 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-fuse-2:17.0.0-4502.gece5ed1a.el8.x86_64 2021-05-25T11:58:57.188 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:58:57.188 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2021-05-25T11:58:57.278 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:02:07 ago on Tue 25 May 2021 11:56:50 AM UTC. 2021-05-25T11:58:57.404 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install rbd-mirror 2021-05-25T11:58:57.780 INFO:teuthology.orchestra.run.smithi102.stdout:Package python3-cephfs-2:17.0.0-4502.gece5ed1a.el8.x86_64 is already installed. 2021-05-25T11:58:57.953 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2021-05-25T11:58:57.954 INFO:teuthology.orchestra.run.smithi102.stdout:Nothing to do. 2021-05-25T11:58:57.954 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2021-05-25T11:58:57.955 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:02:08 ago on Tue 25 May 2021 11:56:49 AM UTC. 2021-05-25T11:58:58.038 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install python3-rbd 2021-05-25T11:58:58.578 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:02:08 ago on Tue 25 May 2021 11:56:50 AM UTC. 2021-05-25T11:58:58.630 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2021-05-25T11:58:58.631 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2021-05-25T11:58:58.631 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repo Size 2021-05-25T11:58:58.632 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2021-05-25T11:58:58.632 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2021-05-25T11:58:58.632 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-mirror x86_64 2:17.0.0-4502.gece5ed1a.el8 ceph 3.7 M 2021-05-25T11:58:58.632 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:58:58.633 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2021-05-25T11:58:58.633 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2021-05-25T11:58:58.633 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2021-05-25T11:58:58.633 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:58:58.633 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 3.7 M 2021-05-25T11:58:58.634 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 21 M 2021-05-25T11:58:58.634 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2021-05-25T11:58:58.979 INFO:teuthology.orchestra.run.smithi181.stdout:rbd-mirror-17.0.0-4502.gece5ed1a.el8.x86_64.rpm 11 MB/s | 3.7 MB 00:00 2021-05-25T11:58:58.979 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2021-05-25T11:58:58.980 INFO:teuthology.orchestra.run.smithi181.stdout:Total 11 MB/s | 3.7 MB 00:00 2021-05-25T11:58:58.980 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2021-05-25T11:58:58.994 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2021-05-25T11:58:58.994 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2021-05-25T11:58:59.058 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2021-05-25T11:58:59.061 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2021-05-25T11:58:59.096 INFO:teuthology.orchestra.run.smithi102.stdout:Package python3-rbd-2:17.0.0-4502.gece5ed1a.el8.x86_64 is already installed. 2021-05-25T11:58:59.271 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2021-05-25T11:58:59.273 INFO:teuthology.orchestra.run.smithi102.stdout:Nothing to do. 2021-05-25T11:58:59.273 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2021-05-25T11:58:59.346 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install rbd-fuse 2021-05-25T11:58:59.555 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2021-05-25T11:58:59.809 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : rbd-mirror-2:17.0.0-4502.gece5ed1a.el8.x86_64 1/1 2021-05-25T11:58:59.887 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:02:09 ago on Tue 25 May 2021 11:56:50 AM UTC. 2021-05-25T11:59:00.206 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: rbd-mirror-2:17.0.0-4502.gece5ed1a.el8.x86_64 1/1 2021-05-25T11:59:00.461 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : rbd-mirror-2:17.0.0-4502.gece5ed1a.el8.x86_64 1/1 2021-05-25T11:59:00.461 INFO:teuthology.orchestra.run.smithi181.stdout:Installed products updated. 2021-05-25T11:59:00.648 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2021-05-25T11:59:00.649 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-05-25T11:59:00.649 INFO:teuthology.orchestra.run.smithi102.stdout: Package Architecture Version Repository Size 2021-05-25T11:59:00.650 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-05-25T11:59:00.650 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2021-05-25T11:59:00.650 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-fuse x86_64 2:17.0.0-4502.gece5ed1a.el8 ceph 90 k 2021-05-25T11:59:00.650 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:59:00.651 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2021-05-25T11:59:00.651 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-05-25T11:59:00.651 INFO:teuthology.orchestra.run.smithi102.stdout:Install 1 Package 2021-05-25T11:59:00.651 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:59:00.651 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 90 k 2021-05-25T11:59:00.652 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 320 k 2021-05-25T11:59:00.652 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2021-05-25T11:59:00.709 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:59:00.709 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2021-05-25T11:59:00.710 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-mirror-2:17.0.0-4502.gece5ed1a.el8.x86_64 2021-05-25T11:59:00.710 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:59:00.710 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2021-05-25T11:59:00.795 INFO:teuthology.orchestra.run.smithi102.stdout:rbd-fuse-17.0.0-4502.gece5ed1a.el8.x86_64.rpm 621 kB/s | 90 kB 00:00 2021-05-25T11:59:00.796 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2021-05-25T11:59:00.796 INFO:teuthology.orchestra.run.smithi102.stdout:Total 615 kB/s | 90 kB 00:00 2021-05-25T11:59:00.797 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2021-05-25T11:59:00.808 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2021-05-25T11:59:00.808 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2021-05-25T11:59:00.858 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2021-05-25T11:59:00.861 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2021-05-25T11:59:00.922 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install rbd-nbd 2021-05-25T11:59:00.968 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2021-05-25T11:59:01.190 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : rbd-fuse-2:17.0.0-4502.gece5ed1a.el8.x86_64 1/1 2021-05-25T11:59:01.435 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: rbd-fuse-2:17.0.0-4502.gece5ed1a.el8.x86_64 1/1 2021-05-25T11:59:01.472 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:02:12 ago on Tue 25 May 2021 11:56:49 AM UTC. 2021-05-25T11:59:01.678 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : rbd-fuse-2:17.0.0-4502.gece5ed1a.el8.x86_64 1/1 2021-05-25T11:59:01.679 INFO:teuthology.orchestra.run.smithi102.stdout:Installed products updated. 2021-05-25T11:59:01.946 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:59:01.946 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2021-05-25T11:59:01.946 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-fuse-2:17.0.0-4502.gece5ed1a.el8.x86_64 2021-05-25T11:59:01.946 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:59:01.947 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2021-05-25T11:59:02.121 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2021-05-25T11:59:02.122 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2021-05-25T11:59:02.122 INFO:teuthology.orchestra.run.smithi181.stdout: Package Architecture Version Repository Size 2021-05-25T11:59:02.122 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2021-05-25T11:59:02.123 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2021-05-25T11:59:02.123 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-nbd x86_64 2:17.0.0-4502.gece5ed1a.el8 ceph 172 k 2021-05-25T11:59:02.123 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:59:02.123 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2021-05-25T11:59:02.124 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2021-05-25T11:59:02.124 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2021-05-25T11:59:02.124 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:59:02.124 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 172 k 2021-05-25T11:59:02.125 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 671 k 2021-05-25T11:59:02.125 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2021-05-25T11:59:02.132 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install rbd-mirror 2021-05-25T11:59:02.294 INFO:teuthology.orchestra.run.smithi181.stdout:rbd-nbd-17.0.0-4502.gece5ed1a.el8.x86_64.rpm 1.0 MB/s | 172 kB 00:00 2021-05-25T11:59:02.295 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2021-05-25T11:59:02.295 INFO:teuthology.orchestra.run.smithi181.stdout:Total 996 kB/s | 172 kB 00:00 2021-05-25T11:59:02.295 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2021-05-25T11:59:02.308 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2021-05-25T11:59:02.308 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2021-05-25T11:59:02.359 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2021-05-25T11:59:02.362 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2021-05-25T11:59:02.519 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2021-05-25T11:59:02.686 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:02:12 ago on Tue 25 May 2021 11:56:50 AM UTC. 2021-05-25T11:59:02.732 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : rbd-nbd-2:17.0.0-4502.gece5ed1a.el8.x86_64 1/1 2021-05-25T11:59:02.971 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: rbd-nbd-2:17.0.0-4502.gece5ed1a.el8.x86_64 1/1 2021-05-25T11:59:03.187 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : rbd-nbd-2:17.0.0-4502.gece5ed1a.el8.x86_64 1/1 2021-05-25T11:59:03.188 INFO:teuthology.orchestra.run.smithi181.stdout:Installed products updated. 2021-05-25T11:59:03.357 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2021-05-25T11:59:03.358 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-05-25T11:59:03.359 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repo Size 2021-05-25T11:59:03.359 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-05-25T11:59:03.359 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2021-05-25T11:59:03.359 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-mirror x86_64 2:17.0.0-4502.gece5ed1a.el8 ceph 3.7 M 2021-05-25T11:59:03.359 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:59:03.360 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2021-05-25T11:59:03.360 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-05-25T11:59:03.360 INFO:teuthology.orchestra.run.smithi102.stdout:Install 1 Package 2021-05-25T11:59:03.360 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:59:03.361 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 3.7 M 2021-05-25T11:59:03.361 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 21 M 2021-05-25T11:59:03.361 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2021-05-25T11:59:03.426 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:59:03.426 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2021-05-25T11:59:03.426 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-nbd-2:17.0.0-4502.gece5ed1a.el8.x86_64 2021-05-25T11:59:03.426 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:59:03.427 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2021-05-25T11:59:03.634 DEBUG:teuthology.parallel:result is None 2021-05-25T11:59:03.702 INFO:teuthology.orchestra.run.smithi102.stdout:rbd-mirror-17.0.0-4502.gece5ed1a.el8.x86_64.rpm 11 MB/s | 3.7 MB 00:00 2021-05-25T11:59:03.703 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2021-05-25T11:59:03.703 INFO:teuthology.orchestra.run.smithi102.stdout:Total 11 MB/s | 3.7 MB 00:00 2021-05-25T11:59:03.703 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2021-05-25T11:59:03.717 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2021-05-25T11:59:03.717 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2021-05-25T11:59:03.781 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2021-05-25T11:59:03.784 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2021-05-25T11:59:04.266 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2021-05-25T11:59:04.499 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : rbd-mirror-2:17.0.0-4502.gece5ed1a.el8.x86_64 1/1 2021-05-25T11:59:04.849 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: rbd-mirror-2:17.0.0-4502.gece5ed1a.el8.x86_64 1/1 2021-05-25T11:59:05.090 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : rbd-mirror-2:17.0.0-4502.gece5ed1a.el8.x86_64 1/1 2021-05-25T11:59:05.091 INFO:teuthology.orchestra.run.smithi102.stdout:Installed products updated. 2021-05-25T11:59:05.315 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:59:05.316 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2021-05-25T11:59:05.316 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-mirror-2:17.0.0-4502.gece5ed1a.el8.x86_64 2021-05-25T11:59:05.316 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:59:05.317 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2021-05-25T11:59:06.158 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install rbd-nbd 2021-05-25T11:59:06.710 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:02:16 ago on Tue 25 May 2021 11:56:50 AM UTC. 2021-05-25T11:59:07.388 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2021-05-25T11:59:07.389 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-05-25T11:59:07.389 INFO:teuthology.orchestra.run.smithi102.stdout: Package Architecture Version Repository Size 2021-05-25T11:59:07.389 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-05-25T11:59:07.390 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2021-05-25T11:59:07.390 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-nbd x86_64 2:17.0.0-4502.gece5ed1a.el8 ceph 172 k 2021-05-25T11:59:07.390 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:59:07.390 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2021-05-25T11:59:07.390 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-05-25T11:59:07.391 INFO:teuthology.orchestra.run.smithi102.stdout:Install 1 Package 2021-05-25T11:59:07.391 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:59:07.391 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 172 k 2021-05-25T11:59:07.391 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 671 k 2021-05-25T11:59:07.392 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2021-05-25T11:59:07.557 INFO:teuthology.orchestra.run.smithi102.stdout:rbd-nbd-17.0.0-4502.gece5ed1a.el8.x86_64.rpm 1.0 MB/s | 172 kB 00:00 2021-05-25T11:59:07.558 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2021-05-25T11:59:07.559 INFO:teuthology.orchestra.run.smithi102.stdout:Total 1.0 MB/s | 172 kB 00:00 2021-05-25T11:59:07.559 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2021-05-25T11:59:07.571 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2021-05-25T11:59:07.571 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2021-05-25T11:59:07.622 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2021-05-25T11:59:07.625 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2021-05-25T11:59:07.771 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2021-05-25T11:59:07.964 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : rbd-nbd-2:17.0.0-4502.gece5ed1a.el8.x86_64 1/1 2021-05-25T11:59:08.211 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: rbd-nbd-2:17.0.0-4502.gece5ed1a.el8.x86_64 1/1 2021-05-25T11:59:08.447 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : rbd-nbd-2:17.0.0-4502.gece5ed1a.el8.x86_64 1/1 2021-05-25T11:59:08.447 INFO:teuthology.orchestra.run.smithi102.stdout:Installed products updated. 2021-05-25T11:59:08.678 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:59:08.679 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2021-05-25T11:59:08.679 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-nbd-2:17.0.0-4502.gece5ed1a.el8.x86_64 2021-05-25T11:59:08.679 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:59:08.679 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2021-05-25T11:59:08.895 DEBUG:teuthology.parallel:result is None 2021-05-25T11:59:08.896 DEBUG:teuthology.orchestra.run.smithi102:> sudo lsb_release -is 2021-05-25T11:59:08.929 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS 2021-05-25T11:59:08.930 DEBUG:teuthology.misc:System to be installed: CentOS 2021-05-25T11:59:08.931 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=ece5ed1ac99f43a50411c04b4ca92c148baf0cd1 2021-05-25T11:59:09.080 DEBUG:teuthology.orchestra.run.smithi102:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2021-05-25T11:59:09.099 INFO:teuthology.orchestra.run.smithi102.stdout:17.0.0-4502.gece5ed1a.el8 2021-05-25T11:59:09.100 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-4502.gece5ed1a.el8 2021-05-25T11:59:09.100 INFO:teuthology.task.install:The correct ceph version 17.0.0-4502.gece5ed1a is installed. 2021-05-25T11:59:09.101 DEBUG:teuthology.orchestra.run.smithi181:> sudo lsb_release -is 2021-05-25T11:59:09.141 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS 2021-05-25T11:59:09.143 DEBUG:teuthology.misc:System to be installed: CentOS 2021-05-25T11:59:09.143 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=ece5ed1ac99f43a50411c04b4ca92c148baf0cd1 2021-05-25T11:59:09.287 DEBUG:teuthology.orchestra.run.smithi181:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2021-05-25T11:59:09.310 INFO:teuthology.orchestra.run.smithi181.stdout:17.0.0-4502.gece5ed1a.el8 2021-05-25T11:59:09.310 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-4502.gece5ed1a.el8 2021-05-25T11:59:09.310 INFO:teuthology.task.install:The correct ceph version 17.0.0-4502.gece5ed1a is installed. 2021-05-25T11:59:09.311 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2021-05-25T11:59:09.311 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2021-05-25T11:59:09.312 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2021-05-25T11:59:09.337 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2021-05-25T11:59:09.337 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2021-05-25T11:59:09.376 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2021-05-25T11:59:09.377 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2021-05-25T11:59:09.377 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/usr/bin/daemon-helper 2021-05-25T11:59:09.401 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod a=rx -- /usr/bin/daemon-helper 2021-05-25T11:59:09.465 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2021-05-25T11:59:09.465 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/usr/bin/daemon-helper 2021-05-25T11:59:09.493 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod a=rx -- /usr/bin/daemon-helper 2021-05-25T11:59:09.560 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2021-05-25T11:59:09.560 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2021-05-25T11:59:09.561 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/usr/bin/adjust-ulimits 2021-05-25T11:59:09.584 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2021-05-25T11:59:09.648 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2021-05-25T11:59:09.648 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/usr/bin/adjust-ulimits 2021-05-25T11:59:09.675 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2021-05-25T11:59:09.742 INFO:teuthology.run_tasks:Running task ceph... 2021-05-25T11:59:09.806 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2021-05-25T11:59:09.807 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod 777 /var/log/ceph 2021-05-25T11:59:09.809 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod 777 /var/log/ceph 2021-05-25T11:59:09.837 INFO:tasks.ceph:Disabling ceph logrotate... 2021-05-25T11:59:09.837 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -f -- /etc/logrotate.d/ceph 2021-05-25T11:59:09.873 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -f -- /etc/logrotate.d/ceph 2021-05-25T11:59:09.902 INFO:tasks.ceph:Creating extra log directories... 2021-05-25T11:59:09.903 DEBUG:teuthology.orchestra.run.smithi102:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2021-05-25T11:59:09.937 DEBUG:teuthology.orchestra.run.smithi181:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2021-05-25T11:59:09.978 INFO:tasks.ceph:Creating ceph cluster ceph... 2021-05-25T11:59:09.978 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20, 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 5}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon client ping interval': 4, 'mon client ping timeout': 12, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_FULL\\)', '\\(SLOW_OPS\\)', '\\(MON_DOWN\\)', '\\(PG_', '\\(POOL_APP_NOT_ENABLED\\)', '\\(SMALLER_PGP_NUM\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2021-05-25T11:59:09.979 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/kchai-2021-05-25_11:32:10-rados-master-distro-basic-smithi/6134024', 'branch': 'master', 'description': 'rados/monthrash/{ceph clusters/3-mons mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8} thrashers/many workloads/rados_api_tests}', 'email': None, 'first_in_suite': False, 'job_id': '6134024', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'kchai-2021-05-25_11:32:10-rados-master-distro-basic-smithi', 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 10}}], 'os_type': 'centos', 'os_version': '8.3', 'overrides': {'admin_socket': {'branch': 'master'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20, 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 5}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon client ping interval': 4, 'mon client ping timeout': 12, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_FULL\\)', '\\(SLOW_OPS\\)', '\\(MON_DOWN\\)', '\\(PG_', '\\(POOL_APP_NOT_ENABLED\\)', '\\(SMALLER_PGP_NUM\\)', 'slow request'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'mon_bind_msgr2': False, 'sha1': 'ece5ed1ac99f43a50411c04b4ca92c148baf0cd1'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'sha1': 'ece5ed1ac99f43a50411c04b4ca92c148baf0cd1'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-kefu-testing-2021-05-25-1039', 'sha1': 'a16ab1caacd8b41a7fe74bc40a10ec74ee8f5802'}}, 'owner': 'scheduled_kchai@teuthology', 'priority': 60, 'repo': 'git://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'], ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0']], 'sha1': 'ece5ed1ac99f43a50411c04b4ca92c148baf0cd1', 'sleep_before_teardown': 0, 'suite': 'rados', 'suite_branch': 'wip-kefu-testing-2021-05-25-1039', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_a16ab1caacd8b41a7fe74bc40a10ec74ee8f5802/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'a16ab1caacd8b41a7fe74bc40a10ec74ee8f5802', 'targets': {'smithi102.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDPYTl+tVRlf0cNuMvygA/Yga0FJP6usxKwc8vN1QFMnOQPq3Hsoo1dLLjJSi/E932f0QySRVdI3MFop8if8qvKTFPgBkabdxHhSuca6vtL5KQM3riUwm2KX88rvHIkTWaeJURSRDCWN9tZI9frx9NJmi685km7RCn2ZfaNzDXXyV32ZrHp2b24yFJr9Pt4kTWUDZY0CdahLVlg88ITssRlbybE/Ms/kcus2JobIHApn7ZXKdoF2snUpocKm3yjYozagGLiNoNzjLOhpEPd2q+5iJ+YznN5BLxBPK093EtDmOUvlF0Mhv/Py9rSsmEKPcSMNu8Z+FVnMRO5st6KU3Y2IzJmahfB2nodNlfC+AnOcbpMl92cL02WVLAaIsVF2tg4xieaUGJrn3kuNCTq78rLjwfqO+Du3ZHbgbLelV3xHzaWbBJUxGddoS/jJiqU9j49pbofTkantWkmh99dAkJ0YWL/8VCVkHXtfxvLa2cWHNrvc290A4BpuC2RPIMHQHU=', 'smithi181.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDb98fBVldQtZu7X/AXNeYvvo5iK2gGpwuPGtcT6Zal8pQTgaF0XFDDfAbxA2tY42m3wJA56o2D5PAcNvLpLaAjUPH+uSsClydVoLbbYRJM2syXkop0jH28lKW7uCDf3cv25VLGodjKS8la5KHw4xRRoqeBfKy4rbUx0AsEi5AHv3Xut/akBhVyX7UiE3z2FuRcIyM9e1/BOH9m+3Oumu6vZ/1xU/MHdPzpdpJHovFMEr8tYMEc/H8eMsZE8wzJDMZREt401+N3T8c9k94Vdan7IqqWzWxahq/JiJO34LoIpUWvc1fkReFuG657d95rqfpLVzd+acyOCcY/QOX4uGO7bnxqG/M+wmn/iwN8Uat9cn00jj9sMutvsqQDp9bgs7yCqGFpEAGPKzblljp8IoAVZB9ixJumFiAFGlOQc2iFQgnW6peJZsDNbNEvnln9FmaBykybWqOtIm3zXp/sWGC8JZO1Jrq97V2kH8JHmYYoVzaMA1Zxqfucbs0Ye7N2Mu8='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'mon_thrash': {'freeze_mon_duration': 20, 'freeze_mon_probability': 10, 'revive_delay': 20, 'thrash_delay': 1, 'thrash_many': True}}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}}}], 'teuthology_branch': 'master', 'teuthology_sha1': 'd92ec1407c30d60262a45a2844110a5c94d567cc', 'timestamp': '2021-05-25_11:32:10', 'tube': 'smithi', 'user': 'kchai', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.8274'} 2021-05-25T11:59:09.980 DEBUG:teuthology.orchestra.run.smithi102:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2021-05-25T11:59:10.001 DEBUG:teuthology.orchestra.run.smithi181:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2021-05-25T11:59:10.024 DEBUG:teuthology.orchestra.run.smithi102:> sudo install -d -m0777 -- /var/run/ceph 2021-05-25T11:59:10.056 DEBUG:teuthology.orchestra.run.smithi181:> sudo install -d -m0777 -- /var/run/ceph 2021-05-25T11:59:10.089 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2021-05-25T11:59:10.089 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/scratch_devs of=/dev/stdout 2021-05-25T11:59:10.131 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2021-05-25T11:59:10.131 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2021-05-25T11:59:10.184 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2021-05-25T11:59:10.185 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-05-25T11:59:10.185 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 6h/6d Inode: 103011 Links: 1 2021-05-25T11:59:10.185 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-05-25T11:59:10.186 INFO:teuthology.orchestra.run.smithi102.stdout:Context: system_u:object_r:device_t:s0 2021-05-25T11:59:10.186 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2021-05-25 11:59:04.657076144 +0000 2021-05-25T11:59:10.186 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2021-05-25 11:56:14.732912821 +0000 2021-05-25T11:59:10.186 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2021-05-25 11:56:14.732912821 +0000 2021-05-25T11:59:10.187 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: - 2021-05-25T11:59:10.194 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2021-05-25T11:59:10.194 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2021-05-25T11:59:10.195 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.000236482 s, 2.2 MB/s 2021-05-25T11:59:10.197 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2021-05-25T11:59:10.251 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2021-05-25T11:59:10.251 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-05-25T11:59:10.252 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 6h/6d Inode: 103067 Links: 1 2021-05-25T11:59:10.252 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-05-25T11:59:10.252 INFO:teuthology.orchestra.run.smithi102.stdout:Context: system_u:object_r:device_t:s0 2021-05-25T11:59:10.252 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2021-05-25 11:59:04.657076144 +0000 2021-05-25T11:59:10.253 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2021-05-25 11:56:15.049906604 +0000 2021-05-25T11:59:10.253 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2021-05-25 11:56:15.049906604 +0000 2021-05-25T11:59:10.253 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: - 2021-05-25T11:59:10.261 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2021-05-25T11:59:10.261 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2021-05-25T11:59:10.261 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.000212642 s, 2.4 MB/s 2021-05-25T11:59:10.263 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2021-05-25T11:59:10.317 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2021-05-25T11:59:10.318 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-05-25T11:59:10.318 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 6h/6d Inode: 106715 Links: 1 2021-05-25T11:59:10.318 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-05-25T11:59:10.318 INFO:teuthology.orchestra.run.smithi102.stdout:Context: system_u:object_r:device_t:s0 2021-05-25T11:59:10.319 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2021-05-25 11:59:04.657076144 +0000 2021-05-25T11:59:10.319 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2021-05-25 11:56:15.375900211 +0000 2021-05-25T11:59:10.319 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2021-05-25 11:56:15.375900211 +0000 2021-05-25T11:59:10.319 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: - 2021-05-25T11:59:10.327 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2021-05-25T11:59:10.327 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2021-05-25T11:59:10.327 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.000212643 s, 2.4 MB/s 2021-05-25T11:59:10.329 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2021-05-25T11:59:10.383 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2021-05-25T11:59:10.383 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-05-25T11:59:10.383 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 6h/6d Inode: 108644 Links: 1 2021-05-25T11:59:10.384 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-05-25T11:59:10.384 INFO:teuthology.orchestra.run.smithi102.stdout:Context: system_u:object_r:device_t:s0 2021-05-25T11:59:10.384 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2021-05-25 11:59:04.657076144 +0000 2021-05-25T11:59:10.384 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2021-05-25 11:56:15.717893504 +0000 2021-05-25T11:59:10.385 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2021-05-25 11:56:15.717893504 +0000 2021-05-25T11:59:10.385 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: - 2021-05-25T11:59:10.392 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2021-05-25T11:59:10.393 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2021-05-25T11:59:10.393 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.000198258 s, 2.6 MB/s 2021-05-25T11:59:10.395 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2021-05-25T11:59:10.396 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2021-05-25T11:59:10.396 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/scratch_devs of=/dev/stdout 2021-05-25T11:59:10.411 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2021-05-25T11:59:10.412 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2021-05-25T11:59:10.467 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2021-05-25T11:59:10.467 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-05-25T11:59:10.468 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 6h/6d Inode: 108565 Links: 1 2021-05-25T11:59:10.468 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-05-25T11:59:10.468 INFO:teuthology.orchestra.run.smithi181.stdout:Context: system_u:object_r:device_t:s0 2021-05-25T11:59:10.468 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2021-05-25 11:59:00.014709134 +0000 2021-05-25T11:59:10.469 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2021-05-25 11:56:14.705763590 +0000 2021-05-25T11:59:10.469 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2021-05-25 11:56:14.705763590 +0000 2021-05-25T11:59:10.469 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: - 2021-05-25T11:59:10.477 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2021-05-25T11:59:10.478 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2021-05-25T11:59:10.478 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.000253041 s, 2.0 MB/s 2021-05-25T11:59:10.481 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2021-05-25T11:59:10.536 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2021-05-25T11:59:10.536 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-05-25T11:59:10.536 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 6h/6d Inode: 102337 Links: 1 2021-05-25T11:59:10.536 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-05-25T11:59:10.537 INFO:teuthology.orchestra.run.smithi181.stdout:Context: system_u:object_r:device_t:s0 2021-05-25T11:59:10.537 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2021-05-25 11:59:00.014709134 +0000 2021-05-25T11:59:10.537 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2021-05-25 11:56:15.029757522 +0000 2021-05-25T11:59:10.537 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2021-05-25 11:56:15.029757522 +0000 2021-05-25T11:59:10.538 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: - 2021-05-25T11:59:10.546 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2021-05-25T11:59:10.546 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2021-05-25T11:59:10.546 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.00021954 s, 2.3 MB/s 2021-05-25T11:59:10.548 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2021-05-25T11:59:10.603 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2021-05-25T11:59:10.604 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-05-25T11:59:10.604 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 6h/6d Inode: 104957 Links: 1 2021-05-25T11:59:10.604 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-05-25T11:59:10.604 INFO:teuthology.orchestra.run.smithi181.stdout:Context: system_u:object_r:device_t:s0 2021-05-25T11:59:10.604 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2021-05-25 11:59:00.015709115 +0000 2021-05-25T11:59:10.605 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2021-05-25 11:56:15.365751230 +0000 2021-05-25T11:59:10.605 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2021-05-25 11:56:15.365751230 +0000 2021-05-25T11:59:10.605 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: - 2021-05-25T11:59:10.613 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2021-05-25T11:59:10.613 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2021-05-25T11:59:10.614 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.000220518 s, 2.3 MB/s 2021-05-25T11:59:10.616 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2021-05-25T11:59:10.670 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2021-05-25T11:59:10.671 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-05-25T11:59:10.671 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 6h/6d Inode: 108641 Links: 1 2021-05-25T11:59:10.671 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-05-25T11:59:10.671 INFO:teuthology.orchestra.run.smithi181.stdout:Context: system_u:object_r:device_t:s0 2021-05-25T11:59:10.672 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2021-05-25 11:59:00.015709115 +0000 2021-05-25T11:59:10.672 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2021-05-25 11:56:15.674745444 +0000 2021-05-25T11:59:10.672 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2021-05-25 11:56:15.674745444 +0000 2021-05-25T11:59:10.672 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: - 2021-05-25T11:59:10.680 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2021-05-25T11:59:10.680 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2021-05-25T11:59:10.681 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.000229335 s, 2.2 MB/s 2021-05-25T11:59:10.683 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2021-05-25T11:59:10.683 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi102.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi181.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2021-05-25T11:59:10.684 INFO:tasks.ceph:Generating config... 2021-05-25T11:59:10.685 INFO:tasks.ceph:[client] debug monc = 20 2021-05-25T11:59:10.685 INFO:tasks.ceph:[client] debug ms = 1 2021-05-25T11:59:10.686 INFO:tasks.ceph:[global] debug ms = 1 2021-05-25T11:59:10.686 INFO:tasks.ceph:[global] debug objecter = 20 2021-05-25T11:59:10.686 INFO:tasks.ceph:[global] debug rados = 20 2021-05-25T11:59:10.687 INFO:tasks.ceph:[global] mon client directed command retry = 5 2021-05-25T11:59:10.687 INFO:tasks.ceph:[global] mon election default strategy = 3 2021-05-25T11:59:10.687 INFO:tasks.ceph:[global] ms bind msgr2 = False 2021-05-25T11:59:10.688 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2021-05-25T11:59:10.688 INFO:tasks.ceph:[global] ms type = async 2021-05-25T11:59:10.688 INFO:tasks.ceph:[mgr] debug mgr = 20 2021-05-25T11:59:10.688 INFO:tasks.ceph:[mgr] debug ms = 1 2021-05-25T11:59:10.689 INFO:tasks.ceph:[mon] debug mon = 20 2021-05-25T11:59:10.689 INFO:tasks.ceph:[mon] debug ms = 1 2021-05-25T11:59:10.689 INFO:tasks.ceph:[mon] debug paxos = 20 2021-05-25T11:59:10.690 INFO:tasks.ceph:[mon] mon min osdmap epochs = 25 2021-05-25T11:59:10.690 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2021-05-25T11:59:10.690 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2021-05-25T11:59:10.691 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2021-05-25T11:59:10.691 INFO:tasks.ceph:[mon] mon scrub inject crc mismatch = 0.01 2021-05-25T11:59:10.691 INFO:tasks.ceph:[mon] mon scrub inject missing keys = 0.05 2021-05-25T11:59:10.691 INFO:tasks.ceph:[mon] mon scrub interval = 300 2021-05-25T11:59:10.692 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2021-05-25T11:59:10.692 INFO:tasks.ceph:[mon] paxos service trim min = 5 2021-05-25T11:59:10.692 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2021-05-25T11:59:10.693 INFO:tasks.ceph:[osd] bluestore compression algorithm = zlib 2021-05-25T11:59:10.693 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2021-05-25T11:59:10.693 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2021-05-25T11:59:10.694 INFO:tasks.ceph:[osd] debug bluefs = 20 2021-05-25T11:59:10.694 INFO:tasks.ceph:[osd] debug bluestore = 20 2021-05-25T11:59:10.694 INFO:tasks.ceph:[osd] debug ms = 1 2021-05-25T11:59:10.694 INFO:tasks.ceph:[osd] debug osd = 20 2021-05-25T11:59:10.695 INFO:tasks.ceph:[osd] debug rocksdb = 10 2021-05-25T11:59:10.695 INFO:tasks.ceph:[osd] mon client ping interval = 4 2021-05-25T11:59:10.695 INFO:tasks.ceph:[osd] mon client ping timeout = 12 2021-05-25T11:59:10.696 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2021-05-25T11:59:10.696 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2021-05-25T11:59:10.696 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2021-05-25T11:59:10.697 INFO:tasks.ceph:[osd] osd class default list = * 2021-05-25T11:59:10.697 INFO:tasks.ceph:[osd] osd class load list = * 2021-05-25T11:59:10.697 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2021-05-25T11:59:10.697 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2021-05-25T11:59:10.698 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2021-05-25T11:59:10.698 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2021-05-25T11:59:10.698 INFO:tasks.ceph:[osd] osd op queue = debug_random 2021-05-25T11:59:10.699 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2021-05-25T11:59:10.699 INFO:tasks.ceph:Setting up mon.a... 2021-05-25T11:59:10.699 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2021-05-25T11:59:10.772 INFO:teuthology.orchestra.run.smithi102.stdout:creating /etc/ceph/ceph.keyring 2021-05-25T11:59:10.774 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2021-05-25T11:59:10.812 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod 0644 /etc/ceph/ceph.keyring 2021-05-25T11:59:10.876 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.102:6789]'), ('mon.c', '[v1:172.21.15.102:6790]'), ('mon.b', '[v1:172.21.15.181:6789]')] 2021-05-25T11:59:10.876 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.102:6789],[v1:172.21.15.102:6790],[v1:172.21.15.181:6789]', 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20, 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon client ping interval': 4, 'mon client ping timeout': 12, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 5}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2021-05-25T11:59:10.878 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2021-05-25T11:59:10.878 DEBUG:teuthology.orchestra.run.smithi102:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2021-05-25T11:59:10.931 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.102:6789]' --addv c '[v1:172.21.15.102:6790]' --addv b '[v1:172.21.15.181:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2021-05-25T11:59:11.012 INFO:teuthology.orchestra.run.smithi102.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2021-05-25T11:59:11.012 INFO:teuthology.orchestra.run.smithi102.stdout:monmaptool: generated fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:11.012 INFO:teuthology.orchestra.run.smithi102.stdout:setting min_mon_release = octopus 2021-05-25T11:59:11.012 INFO:teuthology.orchestra.run.smithi102.stdout:epoch 0 2021-05-25T11:59:11.013 INFO:teuthology.orchestra.run.smithi102.stdout:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:11.013 INFO:teuthology.orchestra.run.smithi102.stdout:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:11.013 INFO:teuthology.orchestra.run.smithi102.stdout:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:11.013 INFO:teuthology.orchestra.run.smithi102.stdout:min_mon_release 15 (octopus) 2021-05-25T11:59:11.013 INFO:teuthology.orchestra.run.smithi102.stdout:election_strategy: 3 2021-05-25T11:59:11.014 INFO:teuthology.orchestra.run.smithi102.stdout:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:11.014 INFO:teuthology.orchestra.run.smithi102.stdout:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:11.014 INFO:teuthology.orchestra.run.smithi102.stdout:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:11.015 INFO:teuthology.orchestra.run.smithi102.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2021-05-25T11:59:11.017 DEBUG:teuthology.orchestra.run.smithi102:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2021-05-25T11:59:11.070 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 20ee91b6-8935-4b98-b868-85c03789ccc4... 2021-05-25T11:59:11.072 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2021-05-25T11:59:11.113 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2021-05-25T11:59:11.152 INFO:teuthology.orchestra.run.smithi102.stdout:[global] 2021-05-25T11:59:11.153 INFO:teuthology.orchestra.run.smithi102.stdout: chdir = "" 2021-05-25T11:59:11.153 INFO:teuthology.orchestra.run.smithi102.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-05-25T11:59:11.153 INFO:teuthology.orchestra.run.smithi102.stdout: auth supported = cephx 2021-05-25T11:59:11.153 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:59:11.154 INFO:teuthology.orchestra.run.smithi102.stdout: filestore xattr use omap = true 2021-05-25T11:59:11.154 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:59:11.154 INFO:teuthology.orchestra.run.smithi102.stdout: mon clock drift allowed = 1.000 2021-05-25T11:59:11.154 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:59:11.155 INFO:teuthology.orchestra.run.smithi102.stdout: osd crush chooseleaf type = 0 2021-05-25T11:59:11.155 INFO:teuthology.orchestra.run.smithi102.stdout: auth debug = true 2021-05-25T11:59:11.155 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:59:11.155 INFO:teuthology.orchestra.run.smithi102.stdout: ms die on old message = true 2021-05-25T11:59:11.155 INFO:teuthology.orchestra.run.smithi102.stdout: ms die on bug = true 2021-05-25T11:59:11.156 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:59:11.156 INFO:teuthology.orchestra.run.smithi102.stdout: mon max pg per osd = 10000 # >= luminous 2021-05-25T11:59:11.156 INFO:teuthology.orchestra.run.smithi102.stdout: mon pg warn max object skew = 0 2021-05-25T11:59:11.156 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:59:11.156 INFO:teuthology.orchestra.run.smithi102.stdout: # disable pg_autoscaler by default for new pools 2021-05-25T11:59:11.157 INFO:teuthology.orchestra.run.smithi102.stdout: osd_pool_default_pg_autoscale_mode = off 2021-05-25T11:59:11.157 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:59:11.157 INFO:teuthology.orchestra.run.smithi102.stdout: osd pool default size = 2 2021-05-25T11:59:11.157 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:59:11.158 INFO:teuthology.orchestra.run.smithi102.stdout: mon osd allow primary affinity = true 2021-05-25T11:59:11.158 INFO:teuthology.orchestra.run.smithi102.stdout: mon osd allow pg remap = true 2021-05-25T11:59:11.158 INFO:teuthology.orchestra.run.smithi102.stdout: mon warn on legacy crush tunables = false 2021-05-25T11:59:11.158 INFO:teuthology.orchestra.run.smithi102.stdout: mon warn on crush straw calc version zero = false 2021-05-25T11:59:11.158 INFO:teuthology.orchestra.run.smithi102.stdout: mon warn on no sortbitwise = false 2021-05-25T11:59:11.159 INFO:teuthology.orchestra.run.smithi102.stdout: mon warn on osd down out interval zero = false 2021-05-25T11:59:11.159 INFO:teuthology.orchestra.run.smithi102.stdout: mon warn on too few osds = false 2021-05-25T11:59:11.159 INFO:teuthology.orchestra.run.smithi102.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-05-25T11:59:11.159 INFO:teuthology.orchestra.run.smithi102.stdout: mon_warn_on_pool_no_redundancy = false 2021-05-25T11:59:11.159 INFO:teuthology.orchestra.run.smithi102.stdout: mon_allow_pool_size_one = true 2021-05-25T11:59:11.160 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:59:11.160 INFO:teuthology.orchestra.run.smithi102.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-05-25T11:59:11.160 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:59:11.160 INFO:teuthology.orchestra.run.smithi102.stdout: osd default data pool replay window = 5 2021-05-25T11:59:11.160 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:59:11.161 INFO:teuthology.orchestra.run.smithi102.stdout: mon allow pool delete = true 2021-05-25T11:59:11.161 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:59:11.161 INFO:teuthology.orchestra.run.smithi102.stdout: mon cluster log file level = debug 2021-05-25T11:59:11.161 INFO:teuthology.orchestra.run.smithi102.stdout: debug asserts on shutdown = true 2021-05-25T11:59:11.162 INFO:teuthology.orchestra.run.smithi102.stdout: mon health detail to clog = false 2021-05-25T11:59:11.162 INFO:teuthology.orchestra.run.smithi102.stdout: mon host = "[v1:172.21.15.102:6789],[v1:172.21.15.102:6790],[v1:172.21.15.181:6789]" 2021-05-25T11:59:11.162 INFO:teuthology.orchestra.run.smithi102.stdout: debug ms = 1 2021-05-25T11:59:11.162 INFO:teuthology.orchestra.run.smithi102.stdout: debug objecter = 20 2021-05-25T11:59:11.162 INFO:teuthology.orchestra.run.smithi102.stdout: debug rados = 20 2021-05-25T11:59:11.163 INFO:teuthology.orchestra.run.smithi102.stdout: mon client directed command retry = 5 2021-05-25T11:59:11.163 INFO:teuthology.orchestra.run.smithi102.stdout: mon election default strategy = 3 2021-05-25T11:59:11.163 INFO:teuthology.orchestra.run.smithi102.stdout: ms bind msgr2 = False 2021-05-25T11:59:11.163 INFO:teuthology.orchestra.run.smithi102.stdout: ms inject socket failures = 5000 2021-05-25T11:59:11.163 INFO:teuthology.orchestra.run.smithi102.stdout: ms type = async 2021-05-25T11:59:11.164 INFO:teuthology.orchestra.run.smithi102.stdout: fsid = 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:11.164 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:59:11.164 INFO:teuthology.orchestra.run.smithi102.stdout:[osd] 2021-05-25T11:59:11.164 INFO:teuthology.orchestra.run.smithi102.stdout: osd journal size = 100 2021-05-25T11:59:11.164 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:59:11.165 INFO:teuthology.orchestra.run.smithi102.stdout: osd scrub load threshold = 5.0 2021-05-25T11:59:11.165 INFO:teuthology.orchestra.run.smithi102.stdout: osd scrub max interval = 600 2021-05-25T11:59:11.165 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:59:11.165 INFO:teuthology.orchestra.run.smithi102.stdout: osd recover clone overlap = true 2021-05-25T11:59:11.166 INFO:teuthology.orchestra.run.smithi102.stdout: osd recovery max chunk = 1048576 2021-05-25T11:59:11.166 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:59:11.166 INFO:teuthology.orchestra.run.smithi102.stdout: osd debug shutdown = true 2021-05-25T11:59:11.166 INFO:teuthology.orchestra.run.smithi102.stdout: osd debug op order = true 2021-05-25T11:59:11.166 INFO:teuthology.orchestra.run.smithi102.stdout: osd debug verify stray on activate = true 2021-05-25T11:59:11.167 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:59:11.167 INFO:teuthology.orchestra.run.smithi102.stdout: osd open classes on start = true 2021-05-25T11:59:11.167 INFO:teuthology.orchestra.run.smithi102.stdout: osd debug pg log writeout = true 2021-05-25T11:59:11.167 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:59:11.168 INFO:teuthology.orchestra.run.smithi102.stdout: osd deep scrub update digest min age = 30 2021-05-25T11:59:11.168 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:59:11.168 INFO:teuthology.orchestra.run.smithi102.stdout: osd map max advance = 10 2021-05-25T11:59:11.168 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:59:11.168 INFO:teuthology.orchestra.run.smithi102.stdout: journal zero on create = true 2021-05-25T11:59:11.169 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:59:11.169 INFO:teuthology.orchestra.run.smithi102.stdout: filestore ondisk finisher threads = 3 2021-05-25T11:59:11.169 INFO:teuthology.orchestra.run.smithi102.stdout: filestore apply finisher threads = 3 2021-05-25T11:59:11.169 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:59:11.169 INFO:teuthology.orchestra.run.smithi102.stdout: bdev debug aio = true 2021-05-25T11:59:11.170 INFO:teuthology.orchestra.run.smithi102.stdout: osd debug misdirected ops = true 2021-05-25T11:59:11.170 INFO:teuthology.orchestra.run.smithi102.stdout: bluestore block size = 96636764160 2021-05-25T11:59:11.170 INFO:teuthology.orchestra.run.smithi102.stdout: bluestore compression algorithm = zlib 2021-05-25T11:59:11.170 INFO:teuthology.orchestra.run.smithi102.stdout: bluestore compression mode = aggressive 2021-05-25T11:59:11.171 INFO:teuthology.orchestra.run.smithi102.stdout: bluestore fsck on mount = True 2021-05-25T11:59:11.171 INFO:teuthology.orchestra.run.smithi102.stdout: debug bluefs = 20 2021-05-25T11:59:11.171 INFO:teuthology.orchestra.run.smithi102.stdout: debug bluestore = 20 2021-05-25T11:59:11.171 INFO:teuthology.orchestra.run.smithi102.stdout: debug ms = 1 2021-05-25T11:59:11.171 INFO:teuthology.orchestra.run.smithi102.stdout: debug osd = 20 2021-05-25T11:59:11.172 INFO:teuthology.orchestra.run.smithi102.stdout: debug rocksdb = 10 2021-05-25T11:59:11.172 INFO:teuthology.orchestra.run.smithi102.stdout: mon client ping interval = 4 2021-05-25T11:59:11.172 INFO:teuthology.orchestra.run.smithi102.stdout: mon client ping timeout = 12 2021-05-25T11:59:11.172 INFO:teuthology.orchestra.run.smithi102.stdout: mon osd backfillfull_ratio = 0.85 2021-05-25T11:59:11.173 INFO:teuthology.orchestra.run.smithi102.stdout: mon osd full ratio = 0.9 2021-05-25T11:59:11.173 INFO:teuthology.orchestra.run.smithi102.stdout: mon osd nearfull ratio = 0.8 2021-05-25T11:59:11.173 INFO:teuthology.orchestra.run.smithi102.stdout: osd class default list = * 2021-05-25T11:59:11.173 INFO:teuthology.orchestra.run.smithi102.stdout: osd class load list = * 2021-05-25T11:59:11.173 INFO:teuthology.orchestra.run.smithi102.stdout: osd debug verify cached snaps = True 2021-05-25T11:59:11.174 INFO:teuthology.orchestra.run.smithi102.stdout: osd debug verify missing on start = True 2021-05-25T11:59:11.174 INFO:teuthology.orchestra.run.smithi102.stdout: osd failsafe full ratio = 0.95 2021-05-25T11:59:11.174 INFO:teuthology.orchestra.run.smithi102.stdout: osd objectstore = bluestore 2021-05-25T11:59:11.174 INFO:teuthology.orchestra.run.smithi102.stdout: osd op queue = debug_random 2021-05-25T11:59:11.175 INFO:teuthology.orchestra.run.smithi102.stdout: osd op queue cut off = debug_random 2021-05-25T11:59:11.175 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:59:11.175 INFO:teuthology.orchestra.run.smithi102.stdout:[mgr] 2021-05-25T11:59:11.175 INFO:teuthology.orchestra.run.smithi102.stdout: debug ms = 1 2021-05-25T11:59:11.175 INFO:teuthology.orchestra.run.smithi102.stdout: debug mgr = 20 2021-05-25T11:59:11.176 INFO:teuthology.orchestra.run.smithi102.stdout: debug mon = 20 2021-05-25T11:59:11.176 INFO:teuthology.orchestra.run.smithi102.stdout: debug auth = 20 2021-05-25T11:59:11.176 INFO:teuthology.orchestra.run.smithi102.stdout: mon reweight min pgs per osd = 4 2021-05-25T11:59:11.176 INFO:teuthology.orchestra.run.smithi102.stdout: mon reweight min bytes per osd = 10 2021-05-25T11:59:11.176 INFO:teuthology.orchestra.run.smithi102.stdout: mgr/telemetry/nag = false 2021-05-25T11:59:11.177 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:59:11.177 INFO:teuthology.orchestra.run.smithi102.stdout:[mon] 2021-05-25T11:59:11.177 INFO:teuthology.orchestra.run.smithi102.stdout: debug ms = 1 2021-05-25T11:59:11.177 INFO:teuthology.orchestra.run.smithi102.stdout: debug mon = 20 2021-05-25T11:59:11.178 INFO:teuthology.orchestra.run.smithi102.stdout: debug paxos = 20 2021-05-25T11:59:11.178 INFO:teuthology.orchestra.run.smithi102.stdout: debug auth = 20 2021-05-25T11:59:11.178 INFO:teuthology.orchestra.run.smithi102.stdout: mon data avail warn = 5 2021-05-25T11:59:11.178 INFO:teuthology.orchestra.run.smithi102.stdout: mon mgr mkfs grace = 240 2021-05-25T11:59:11.178 INFO:teuthology.orchestra.run.smithi102.stdout: mon reweight min pgs per osd = 4 2021-05-25T11:59:11.179 INFO:teuthology.orchestra.run.smithi102.stdout: mon osd reporter subtree level = osd 2021-05-25T11:59:11.179 INFO:teuthology.orchestra.run.smithi102.stdout: mon osd prime pg temp = true 2021-05-25T11:59:11.179 INFO:teuthology.orchestra.run.smithi102.stdout: mon reweight min bytes per osd = 10 2021-05-25T11:59:11.179 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:59:11.179 INFO:teuthology.orchestra.run.smithi102.stdout: # rotate auth tickets quickly to exercise renewal paths 2021-05-25T11:59:11.180 INFO:teuthology.orchestra.run.smithi102.stdout: auth mon ticket ttl = 660 # 11m 2021-05-25T11:59:11.180 INFO:teuthology.orchestra.run.smithi102.stdout: auth service ticket ttl = 240 # 4m 2021-05-25T11:59:11.180 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:59:11.180 INFO:teuthology.orchestra.run.smithi102.stdout: # don't complain about insecure global_id in the test suite 2021-05-25T11:59:11.181 INFO:teuthology.orchestra.run.smithi102.stdout: mon_warn_on_insecure_global_id_reclaim = false 2021-05-25T11:59:11.181 INFO:teuthology.orchestra.run.smithi102.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2021-05-25T11:59:11.181 INFO:teuthology.orchestra.run.smithi102.stdout: mon min osdmap epochs = 25 2021-05-25T11:59:11.181 INFO:teuthology.orchestra.run.smithi102.stdout: mon osdmap full prune interval = 2 2021-05-25T11:59:11.181 INFO:teuthology.orchestra.run.smithi102.stdout: mon osdmap full prune min = 15 2021-05-25T11:59:11.182 INFO:teuthology.orchestra.run.smithi102.stdout: mon osdmap full prune txsize = 2 2021-05-25T11:59:11.182 INFO:teuthology.orchestra.run.smithi102.stdout: mon scrub inject crc mismatch = 0.01 2021-05-25T11:59:11.182 INFO:teuthology.orchestra.run.smithi102.stdout: mon scrub inject missing keys = 0.05 2021-05-25T11:59:11.182 INFO:teuthology.orchestra.run.smithi102.stdout: mon scrub interval = 300 2021-05-25T11:59:11.182 INFO:teuthology.orchestra.run.smithi102.stdout: mon warn on pool no app = False 2021-05-25T11:59:11.183 INFO:teuthology.orchestra.run.smithi102.stdout: paxos service trim min = 5 2021-05-25T11:59:11.183 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:59:11.183 INFO:teuthology.orchestra.run.smithi102.stdout:[client] 2021-05-25T11:59:11.183 INFO:teuthology.orchestra.run.smithi102.stdout: rgw cache enabled = true 2021-05-25T11:59:11.183 INFO:teuthology.orchestra.run.smithi102.stdout: rgw enable ops log = true 2021-05-25T11:59:11.184 INFO:teuthology.orchestra.run.smithi102.stdout: rgw enable usage log = true 2021-05-25T11:59:11.184 INFO:teuthology.orchestra.run.smithi102.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-05-25T11:59:11.184 INFO:teuthology.orchestra.run.smithi102.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-05-25T11:59:11.184 INFO:teuthology.orchestra.run.smithi102.stdout: debug monc = 20 2021-05-25T11:59:11.185 INFO:teuthology.orchestra.run.smithi102.stdout: debug ms = 1 2021-05-25T11:59:11.185 INFO:teuthology.orchestra.run.smithi102.stdout:[mon.a] 2021-05-25T11:59:11.185 INFO:teuthology.orchestra.run.smithi102.stdout:[mon.c] 2021-05-25T11:59:11.185 INFO:teuthology.orchestra.run.smithi102.stdout:[mon.b] 2021-05-25T11:59:11.187 INFO:teuthology.orchestra.run.smithi181.stdout:[global] 2021-05-25T11:59:11.187 INFO:teuthology.orchestra.run.smithi181.stdout: chdir = "" 2021-05-25T11:59:11.188 INFO:teuthology.orchestra.run.smithi181.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-05-25T11:59:11.188 INFO:teuthology.orchestra.run.smithi181.stdout: auth supported = cephx 2021-05-25T11:59:11.188 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:59:11.188 INFO:teuthology.orchestra.run.smithi181.stdout: filestore xattr use omap = true 2021-05-25T11:59:11.188 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:59:11.189 INFO:teuthology.orchestra.run.smithi181.stdout: mon clock drift allowed = 1.000 2021-05-25T11:59:11.189 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:59:11.189 INFO:teuthology.orchestra.run.smithi181.stdout: osd crush chooseleaf type = 0 2021-05-25T11:59:11.189 INFO:teuthology.orchestra.run.smithi181.stdout: auth debug = true 2021-05-25T11:59:11.190 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:59:11.190 INFO:teuthology.orchestra.run.smithi181.stdout: ms die on old message = true 2021-05-25T11:59:11.190 INFO:teuthology.orchestra.run.smithi181.stdout: ms die on bug = true 2021-05-25T11:59:11.190 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:59:11.190 INFO:teuthology.orchestra.run.smithi181.stdout: mon max pg per osd = 10000 # >= luminous 2021-05-25T11:59:11.191 INFO:teuthology.orchestra.run.smithi181.stdout: mon pg warn max object skew = 0 2021-05-25T11:59:11.191 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:59:11.191 INFO:teuthology.orchestra.run.smithi181.stdout: # disable pg_autoscaler by default for new pools 2021-05-25T11:59:11.191 INFO:teuthology.orchestra.run.smithi181.stdout: osd_pool_default_pg_autoscale_mode = off 2021-05-25T11:59:11.192 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:59:11.192 INFO:teuthology.orchestra.run.smithi181.stdout: osd pool default size = 2 2021-05-25T11:59:11.192 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:59:11.192 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd allow primary affinity = true 2021-05-25T11:59:11.193 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd allow pg remap = true 2021-05-25T11:59:11.193 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on legacy crush tunables = false 2021-05-25T11:59:11.193 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on crush straw calc version zero = false 2021-05-25T11:59:11.193 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on no sortbitwise = false 2021-05-25T11:59:11.193 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on osd down out interval zero = false 2021-05-25T11:59:11.194 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on too few osds = false 2021-05-25T11:59:11.194 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-05-25T11:59:11.194 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_pool_no_redundancy = false 2021-05-25T11:59:11.194 INFO:teuthology.orchestra.run.smithi181.stdout: mon_allow_pool_size_one = true 2021-05-25T11:59:11.194 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:59:11.195 INFO:teuthology.orchestra.run.smithi181.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-05-25T11:59:11.195 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:59:11.195 INFO:teuthology.orchestra.run.smithi181.stdout: osd default data pool replay window = 5 2021-05-25T11:59:11.195 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:59:11.196 INFO:teuthology.orchestra.run.smithi181.stdout: mon allow pool delete = true 2021-05-25T11:59:11.196 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:59:11.196 INFO:teuthology.orchestra.run.smithi181.stdout: mon cluster log file level = debug 2021-05-25T11:59:11.196 INFO:teuthology.orchestra.run.smithi181.stdout: debug asserts on shutdown = true 2021-05-25T11:59:11.196 INFO:teuthology.orchestra.run.smithi181.stdout: mon health detail to clog = false 2021-05-25T11:59:11.197 INFO:teuthology.orchestra.run.smithi181.stdout: mon host = "[v1:172.21.15.102:6789],[v1:172.21.15.102:6790],[v1:172.21.15.181:6789]" 2021-05-25T11:59:11.197 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2021-05-25T11:59:11.197 INFO:teuthology.orchestra.run.smithi181.stdout: debug objecter = 20 2021-05-25T11:59:11.197 INFO:teuthology.orchestra.run.smithi181.stdout: debug rados = 20 2021-05-25T11:59:11.198 INFO:teuthology.orchestra.run.smithi181.stdout: mon client directed command retry = 5 2021-05-25T11:59:11.198 INFO:teuthology.orchestra.run.smithi181.stdout: mon election default strategy = 3 2021-05-25T11:59:11.198 INFO:teuthology.orchestra.run.smithi181.stdout: ms bind msgr2 = False 2021-05-25T11:59:11.198 INFO:teuthology.orchestra.run.smithi181.stdout: ms inject socket failures = 5000 2021-05-25T11:59:11.198 INFO:teuthology.orchestra.run.smithi181.stdout: ms type = async 2021-05-25T11:59:11.199 INFO:teuthology.orchestra.run.smithi181.stdout: fsid = 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:11.199 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:59:11.199 INFO:teuthology.orchestra.run.smithi181.stdout:[osd] 2021-05-25T11:59:11.199 INFO:teuthology.orchestra.run.smithi181.stdout: osd journal size = 100 2021-05-25T11:59:11.199 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:59:11.200 INFO:teuthology.orchestra.run.smithi181.stdout: osd scrub load threshold = 5.0 2021-05-25T11:59:11.200 INFO:teuthology.orchestra.run.smithi181.stdout: osd scrub max interval = 600 2021-05-25T11:59:11.200 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:59:11.200 INFO:teuthology.orchestra.run.smithi181.stdout: osd recover clone overlap = true 2021-05-25T11:59:11.201 INFO:teuthology.orchestra.run.smithi181.stdout: osd recovery max chunk = 1048576 2021-05-25T11:59:11.201 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:59:11.201 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug shutdown = true 2021-05-25T11:59:11.201 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug op order = true 2021-05-25T11:59:11.201 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug verify stray on activate = true 2021-05-25T11:59:11.202 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:59:11.202 INFO:teuthology.orchestra.run.smithi181.stdout: osd open classes on start = true 2021-05-25T11:59:11.202 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug pg log writeout = true 2021-05-25T11:59:11.202 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:59:11.202 INFO:teuthology.orchestra.run.smithi181.stdout: osd deep scrub update digest min age = 30 2021-05-25T11:59:11.203 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:59:11.203 INFO:teuthology.orchestra.run.smithi181.stdout: osd map max advance = 10 2021-05-25T11:59:11.203 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:59:11.203 INFO:teuthology.orchestra.run.smithi181.stdout: journal zero on create = true 2021-05-25T11:59:11.204 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:59:11.204 INFO:teuthology.orchestra.run.smithi181.stdout: filestore ondisk finisher threads = 3 2021-05-25T11:59:11.204 INFO:teuthology.orchestra.run.smithi181.stdout: filestore apply finisher threads = 3 2021-05-25T11:59:11.204 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:59:11.204 INFO:teuthology.orchestra.run.smithi181.stdout: bdev debug aio = true 2021-05-25T11:59:11.205 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug misdirected ops = true 2021-05-25T11:59:11.205 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore block size = 96636764160 2021-05-25T11:59:11.205 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore compression algorithm = zlib 2021-05-25T11:59:11.205 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore compression mode = aggressive 2021-05-25T11:59:11.205 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore fsck on mount = True 2021-05-25T11:59:11.206 INFO:teuthology.orchestra.run.smithi181.stdout: debug bluefs = 20 2021-05-25T11:59:11.206 INFO:teuthology.orchestra.run.smithi181.stdout: debug bluestore = 20 2021-05-25T11:59:11.206 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2021-05-25T11:59:11.206 INFO:teuthology.orchestra.run.smithi181.stdout: debug osd = 20 2021-05-25T11:59:11.207 INFO:teuthology.orchestra.run.smithi181.stdout: debug rocksdb = 10 2021-05-25T11:59:11.207 INFO:teuthology.orchestra.run.smithi181.stdout: mon client ping interval = 4 2021-05-25T11:59:11.207 INFO:teuthology.orchestra.run.smithi181.stdout: mon client ping timeout = 12 2021-05-25T11:59:11.207 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd backfillfull_ratio = 0.85 2021-05-25T11:59:11.207 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd full ratio = 0.9 2021-05-25T11:59:11.208 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd nearfull ratio = 0.8 2021-05-25T11:59:11.208 INFO:teuthology.orchestra.run.smithi181.stdout: osd class default list = * 2021-05-25T11:59:11.208 INFO:teuthology.orchestra.run.smithi181.stdout: osd class load list = * 2021-05-25T11:59:11.208 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug verify cached snaps = True 2021-05-25T11:59:11.209 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug verify missing on start = True 2021-05-25T11:59:11.209 INFO:teuthology.orchestra.run.smithi181.stdout: osd failsafe full ratio = 0.95 2021-05-25T11:59:11.209 INFO:teuthology.orchestra.run.smithi181.stdout: osd objectstore = bluestore 2021-05-25T11:59:11.209 INFO:teuthology.orchestra.run.smithi181.stdout: osd op queue = debug_random 2021-05-25T11:59:11.209 INFO:teuthology.orchestra.run.smithi181.stdout: osd op queue cut off = debug_random 2021-05-25T11:59:11.210 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:59:11.210 INFO:teuthology.orchestra.run.smithi181.stdout:[mgr] 2021-05-25T11:59:11.210 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2021-05-25T11:59:11.210 INFO:teuthology.orchestra.run.smithi181.stdout: debug mgr = 20 2021-05-25T11:59:11.211 INFO:teuthology.orchestra.run.smithi181.stdout: debug mon = 20 2021-05-25T11:59:11.211 INFO:teuthology.orchestra.run.smithi181.stdout: debug auth = 20 2021-05-25T11:59:11.211 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min pgs per osd = 4 2021-05-25T11:59:11.211 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min bytes per osd = 10 2021-05-25T11:59:11.211 INFO:teuthology.orchestra.run.smithi181.stdout: mgr/telemetry/nag = false 2021-05-25T11:59:11.212 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:59:11.212 INFO:teuthology.orchestra.run.smithi181.stdout:[mon] 2021-05-25T11:59:11.212 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2021-05-25T11:59:11.212 INFO:teuthology.orchestra.run.smithi181.stdout: debug mon = 20 2021-05-25T11:59:11.213 INFO:teuthology.orchestra.run.smithi181.stdout: debug paxos = 20 2021-05-25T11:59:11.213 INFO:teuthology.orchestra.run.smithi181.stdout: debug auth = 20 2021-05-25T11:59:11.213 INFO:teuthology.orchestra.run.smithi181.stdout: mon data avail warn = 5 2021-05-25T11:59:11.213 INFO:teuthology.orchestra.run.smithi181.stdout: mon mgr mkfs grace = 240 2021-05-25T11:59:11.213 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min pgs per osd = 4 2021-05-25T11:59:11.214 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd reporter subtree level = osd 2021-05-25T11:59:11.214 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd prime pg temp = true 2021-05-25T11:59:11.214 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min bytes per osd = 10 2021-05-25T11:59:11.214 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:59:11.215 INFO:teuthology.orchestra.run.smithi181.stdout: # rotate auth tickets quickly to exercise renewal paths 2021-05-25T11:59:11.215 INFO:teuthology.orchestra.run.smithi181.stdout: auth mon ticket ttl = 660 # 11m 2021-05-25T11:59:11.215 INFO:teuthology.orchestra.run.smithi181.stdout: auth service ticket ttl = 240 # 4m 2021-05-25T11:59:11.215 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:59:11.215 INFO:teuthology.orchestra.run.smithi181.stdout: # don't complain about insecure global_id in the test suite 2021-05-25T11:59:11.216 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_insecure_global_id_reclaim = false 2021-05-25T11:59:11.216 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2021-05-25T11:59:11.216 INFO:teuthology.orchestra.run.smithi181.stdout: mon min osdmap epochs = 25 2021-05-25T11:59:11.216 INFO:teuthology.orchestra.run.smithi181.stdout: mon osdmap full prune interval = 2 2021-05-25T11:59:11.216 INFO:teuthology.orchestra.run.smithi181.stdout: mon osdmap full prune min = 15 2021-05-25T11:59:11.217 INFO:teuthology.orchestra.run.smithi181.stdout: mon osdmap full prune txsize = 2 2021-05-25T11:59:11.217 INFO:teuthology.orchestra.run.smithi181.stdout: mon scrub inject crc mismatch = 0.01 2021-05-25T11:59:11.217 INFO:teuthology.orchestra.run.smithi181.stdout: mon scrub inject missing keys = 0.05 2021-05-25T11:59:11.217 INFO:teuthology.orchestra.run.smithi181.stdout: mon scrub interval = 300 2021-05-25T11:59:11.218 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on pool no app = False 2021-05-25T11:59:11.218 INFO:teuthology.orchestra.run.smithi181.stdout: paxos service trim min = 5 2021-05-25T11:59:11.218 INFO:teuthology.orchestra.run.smithi181.stdout: 2021-05-25T11:59:11.218 INFO:teuthology.orchestra.run.smithi181.stdout:[client] 2021-05-25T11:59:11.218 INFO:teuthology.orchestra.run.smithi181.stdout: rgw cache enabled = true 2021-05-25T11:59:11.219 INFO:teuthology.orchestra.run.smithi181.stdout: rgw enable ops log = true 2021-05-25T11:59:11.219 INFO:teuthology.orchestra.run.smithi181.stdout: rgw enable usage log = true 2021-05-25T11:59:11.219 INFO:teuthology.orchestra.run.smithi181.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-05-25T11:59:11.219 INFO:teuthology.orchestra.run.smithi181.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-05-25T11:59:11.219 INFO:teuthology.orchestra.run.smithi181.stdout: debug monc = 20 2021-05-25T11:59:11.220 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2021-05-25T11:59:11.220 INFO:teuthology.orchestra.run.smithi181.stdout:[mon.a] 2021-05-25T11:59:11.220 INFO:teuthology.orchestra.run.smithi181.stdout:[mon.c] 2021-05-25T11:59:11.220 INFO:teuthology.orchestra.run.smithi181.stdout:[mon.b] 2021-05-25T11:59:11.221 INFO:tasks.ceph:Creating admin key on mon.a... 2021-05-25T11:59:11.221 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2021-05-25T11:59:11.275 INFO:tasks.ceph:Copying monmap to all nodes... 2021-05-25T11:59:11.275 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2021-05-25T11:59:11.276 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2021-05-25T11:59:11.290 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2021-05-25T11:59:11.291 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2021-05-25T11:59:11.344 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi102.front.sepia.ceph.com 2021-05-25T11:59:11.345 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2021-05-25T11:59:11.345 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/etc/ceph/ceph.keyring 2021-05-25T11:59:11.345 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod 0644 /etc/ceph/ceph.keyring 2021-05-25T11:59:11.417 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2021-05-25T11:59:11.418 DEBUG:teuthology.orchestra.run.smithi102:> dd of=/home/ubuntu/cephtest/ceph.monmap 2021-05-25T11:59:11.482 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi181.front.sepia.ceph.com 2021-05-25T11:59:11.482 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2021-05-25T11:59:11.483 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/ceph/ceph.keyring 2021-05-25T11:59:11.483 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod 0644 /etc/ceph/ceph.keyring 2021-05-25T11:59:11.519 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2021-05-25T11:59:11.519 DEBUG:teuthology.orchestra.run.smithi181:> dd of=/home/ubuntu/cephtest/ceph.monmap 2021-05-25T11:59:11.574 INFO:tasks.ceph:Setting up mon nodes... 2021-05-25T11:59:11.574 INFO:tasks.ceph:Setting up mgr nodes... 2021-05-25T11:59:11.574 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2021-05-25T11:59:11.752 INFO:teuthology.orchestra.run.smithi181.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2021-05-25T11:59:11.754 INFO:tasks.ceph:Setting up mds nodes... 2021-05-25T11:59:11.755 INFO:tasks.ceph_client:Setting up client nodes... 2021-05-25T11:59:11.755 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2021-05-25T11:59:11.792 INFO:teuthology.orchestra.run.smithi181.stdout:creating /etc/ceph/ceph.client.0.keyring 2021-05-25T11:59:11.805 INFO:tasks.ceph:Running mkfs on osd nodes... 2021-05-25T11:59:11.805 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi102.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi181.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2021-05-25T11:59:11.806 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2021-05-25T11:59:11.832 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2021-05-25T11:59:11.832 INFO:tasks.ceph:role: osd.0 2021-05-25T11:59:11.832 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi102.front.sepia.ceph.com 2021-05-25T11:59:11.833 DEBUG:teuthology.orchestra.run.smithi102:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2021-05-25T11:59:11.904 INFO:teuthology.orchestra.run.smithi102.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2021-05-25T11:59:11.905 INFO:teuthology.orchestra.run.smithi102.stdout: = sectsz=512 attr=2, projid32bit=1 2021-05-25T11:59:11.905 INFO:teuthology.orchestra.run.smithi102.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2021-05-25T11:59:11.905 INFO:teuthology.orchestra.run.smithi102.stdout: = reflink=1 2021-05-25T11:59:11.905 INFO:teuthology.orchestra.run.smithi102.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-05-25T11:59:11.906 INFO:teuthology.orchestra.run.smithi102.stdout: = sunit=0 swidth=0 blks 2021-05-25T11:59:11.906 INFO:teuthology.orchestra.run.smithi102.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2021-05-25T11:59:11.907 INFO:teuthology.orchestra.run.smithi102.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-05-25T11:59:11.907 INFO:teuthology.orchestra.run.smithi102.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-05-25T11:59:11.907 INFO:teuthology.orchestra.run.smithi102.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-05-25T11:59:12.264 INFO:teuthology.orchestra.run.smithi102.stdout:Discarding blocks...Done. 2021-05-25T11:59:12.266 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi102.front.sepia.ceph.com -o noatime 2021-05-25T11:59:12.266 DEBUG:teuthology.orchestra.run.smithi102:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2021-05-25T11:59:12.399 DEBUG:teuthology.orchestra.run.smithi102:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2021-05-25T11:59:12.429 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2021-05-25T11:59:12.495 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2021-05-25T11:59:12.495 INFO:tasks.ceph:role: osd.1 2021-05-25T11:59:12.495 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi102.front.sepia.ceph.com 2021-05-25T11:59:12.496 DEBUG:teuthology.orchestra.run.smithi102:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2021-05-25T11:59:12.567 INFO:teuthology.orchestra.run.smithi102.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2021-05-25T11:59:12.567 INFO:teuthology.orchestra.run.smithi102.stdout: = sectsz=512 attr=2, projid32bit=1 2021-05-25T11:59:12.567 INFO:teuthology.orchestra.run.smithi102.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2021-05-25T11:59:12.568 INFO:teuthology.orchestra.run.smithi102.stdout: = reflink=1 2021-05-25T11:59:12.568 INFO:teuthology.orchestra.run.smithi102.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-05-25T11:59:12.568 INFO:teuthology.orchestra.run.smithi102.stdout: = sunit=0 swidth=0 blks 2021-05-25T11:59:12.568 INFO:teuthology.orchestra.run.smithi102.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2021-05-25T11:59:12.568 INFO:teuthology.orchestra.run.smithi102.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-05-25T11:59:12.569 INFO:teuthology.orchestra.run.smithi102.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-05-25T11:59:12.569 INFO:teuthology.orchestra.run.smithi102.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-05-25T11:59:12.929 INFO:teuthology.orchestra.run.smithi102.stdout:Discarding blocks...Done. 2021-05-25T11:59:12.931 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi102.front.sepia.ceph.com -o noatime 2021-05-25T11:59:12.931 DEBUG:teuthology.orchestra.run.smithi102:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2021-05-25T11:59:13.022 DEBUG:teuthology.orchestra.run.smithi102:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2021-05-25T11:59:13.049 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2021-05-25T11:59:13.113 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2021-05-25T11:59:13.114 INFO:tasks.ceph:role: osd.2 2021-05-25T11:59:13.114 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi102.front.sepia.ceph.com 2021-05-25T11:59:13.114 DEBUG:teuthology.orchestra.run.smithi102:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2021-05-25T11:59:13.184 INFO:teuthology.orchestra.run.smithi102.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2021-05-25T11:59:13.184 INFO:teuthology.orchestra.run.smithi102.stdout: = sectsz=512 attr=2, projid32bit=1 2021-05-25T11:59:13.184 INFO:teuthology.orchestra.run.smithi102.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2021-05-25T11:59:13.185 INFO:teuthology.orchestra.run.smithi102.stdout: = reflink=1 2021-05-25T11:59:13.185 INFO:teuthology.orchestra.run.smithi102.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-05-25T11:59:13.185 INFO:teuthology.orchestra.run.smithi102.stdout: = sunit=0 swidth=0 blks 2021-05-25T11:59:13.185 INFO:teuthology.orchestra.run.smithi102.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2021-05-25T11:59:13.186 INFO:teuthology.orchestra.run.smithi102.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-05-25T11:59:13.186 INFO:teuthology.orchestra.run.smithi102.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-05-25T11:59:13.186 INFO:teuthology.orchestra.run.smithi102.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-05-25T11:59:13.547 INFO:teuthology.orchestra.run.smithi102.stdout:Discarding blocks...Done. 2021-05-25T11:59:13.549 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi102.front.sepia.ceph.com -o noatime 2021-05-25T11:59:13.549 DEBUG:teuthology.orchestra.run.smithi102:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2021-05-25T11:59:13.662 DEBUG:teuthology.orchestra.run.smithi102:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2021-05-25T11:59:13.691 DEBUG:teuthology.orchestra.run.smithi102:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2021-05-25T11:59:13.794 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:13.792+0000 7fa26d81b0c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2021-05-25T11:59:13.795 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:13.792+0000 7fa26d81b0c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2021-05-25T11:59:13.795 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:13.792+0000 7fa26d81b0c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2021-05-25T11:59:13.796 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:13.792+0000 7fa26d81b0c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2021-05-25T11:59:13.796 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:13.792+0000 7fa26d81b0c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2021-05-25T11:59:13.797 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:13.796+0000 7fa26d81b0c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2021-05-25T11:59:16.643 DEBUG:teuthology.orchestra.run.smithi102:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2021-05-25T11:59:16.672 DEBUG:teuthology.orchestra.run.smithi102:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2021-05-25T11:59:16.757 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:16.755+0000 7fc3ad7650c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2021-05-25T11:59:16.758 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:16.755+0000 7fc3ad7650c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2021-05-25T11:59:16.759 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:16.755+0000 7fc3ad7650c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2021-05-25T11:59:16.759 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:16.755+0000 7fc3ad7650c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2021-05-25T11:59:16.759 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:16.755+0000 7fc3ad7650c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2021-05-25T11:59:16.760 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:16.756+0000 7fc3ad7650c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2021-05-25T11:59:19.598 DEBUG:teuthology.orchestra.run.smithi102:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2021-05-25T11:59:19.666 DEBUG:teuthology.orchestra.run.smithi102:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2021-05-25T11:59:19.755 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:19.752+0000 7f384dac00c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2021-05-25T11:59:19.755 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:19.753+0000 7f384dac00c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2021-05-25T11:59:19.755 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:19.753+0000 7f384dac00c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2021-05-25T11:59:19.755 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:19.753+0000 7f384dac00c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2021-05-25T11:59:19.756 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:19.753+0000 7f384dac00c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2021-05-25T11:59:19.756 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:19.753+0000 7f384dac00c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2021-05-25T11:59:22.594 DEBUG:teuthology.orchestra.run.smithi102:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2021-05-25T11:59:22.623 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2021-05-25T11:59:22.652 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2021-05-25T11:59:22.652 INFO:tasks.ceph:role: osd.3 2021-05-25T11:59:22.652 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi181.front.sepia.ceph.com 2021-05-25T11:59:22.653 DEBUG:teuthology.orchestra.run.smithi181:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2021-05-25T11:59:22.727 INFO:teuthology.orchestra.run.smithi181.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2021-05-25T11:59:22.727 INFO:teuthology.orchestra.run.smithi181.stdout: = sectsz=512 attr=2, projid32bit=1 2021-05-25T11:59:22.727 INFO:teuthology.orchestra.run.smithi181.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2021-05-25T11:59:22.727 INFO:teuthology.orchestra.run.smithi181.stdout: = reflink=1 2021-05-25T11:59:22.728 INFO:teuthology.orchestra.run.smithi181.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-05-25T11:59:22.728 INFO:teuthology.orchestra.run.smithi181.stdout: = sunit=0 swidth=0 blks 2021-05-25T11:59:22.728 INFO:teuthology.orchestra.run.smithi181.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2021-05-25T11:59:22.728 INFO:teuthology.orchestra.run.smithi181.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-05-25T11:59:22.729 INFO:teuthology.orchestra.run.smithi181.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-05-25T11:59:22.729 INFO:teuthology.orchestra.run.smithi181.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-05-25T11:59:23.089 INFO:teuthology.orchestra.run.smithi181.stdout:Discarding blocks...Done. 2021-05-25T11:59:23.091 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi181.front.sepia.ceph.com -o noatime 2021-05-25T11:59:23.091 DEBUG:teuthology.orchestra.run.smithi181:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2021-05-25T11:59:23.219 DEBUG:teuthology.orchestra.run.smithi181:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2021-05-25T11:59:23.249 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2021-05-25T11:59:23.316 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2021-05-25T11:59:23.316 INFO:tasks.ceph:role: osd.4 2021-05-25T11:59:23.317 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi181.front.sepia.ceph.com 2021-05-25T11:59:23.317 DEBUG:teuthology.orchestra.run.smithi181:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2021-05-25T11:59:23.388 INFO:teuthology.orchestra.run.smithi181.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2021-05-25T11:59:23.389 INFO:teuthology.orchestra.run.smithi181.stdout: = sectsz=512 attr=2, projid32bit=1 2021-05-25T11:59:23.389 INFO:teuthology.orchestra.run.smithi181.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2021-05-25T11:59:23.389 INFO:teuthology.orchestra.run.smithi181.stdout: = reflink=1 2021-05-25T11:59:23.390 INFO:teuthology.orchestra.run.smithi181.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-05-25T11:59:23.390 INFO:teuthology.orchestra.run.smithi181.stdout: = sunit=0 swidth=0 blks 2021-05-25T11:59:23.390 INFO:teuthology.orchestra.run.smithi181.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2021-05-25T11:59:23.390 INFO:teuthology.orchestra.run.smithi181.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-05-25T11:59:23.391 INFO:teuthology.orchestra.run.smithi181.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-05-25T11:59:23.391 INFO:teuthology.orchestra.run.smithi181.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-05-25T11:59:23.753 INFO:teuthology.orchestra.run.smithi181.stdout:Discarding blocks...Done. 2021-05-25T11:59:23.755 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi181.front.sepia.ceph.com -o noatime 2021-05-25T11:59:23.756 DEBUG:teuthology.orchestra.run.smithi181:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2021-05-25T11:59:23.845 DEBUG:teuthology.orchestra.run.smithi181:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2021-05-25T11:59:23.874 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2021-05-25T11:59:23.939 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2021-05-25T11:59:23.939 INFO:tasks.ceph:role: osd.5 2021-05-25T11:59:23.940 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi181.front.sepia.ceph.com 2021-05-25T11:59:23.940 DEBUG:teuthology.orchestra.run.smithi181:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2021-05-25T11:59:24.010 INFO:teuthology.orchestra.run.smithi181.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2021-05-25T11:59:24.011 INFO:teuthology.orchestra.run.smithi181.stdout: = sectsz=512 attr=2, projid32bit=1 2021-05-25T11:59:24.011 INFO:teuthology.orchestra.run.smithi181.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2021-05-25T11:59:24.011 INFO:teuthology.orchestra.run.smithi181.stdout: = reflink=1 2021-05-25T11:59:24.012 INFO:teuthology.orchestra.run.smithi181.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-05-25T11:59:24.012 INFO:teuthology.orchestra.run.smithi181.stdout: = sunit=0 swidth=0 blks 2021-05-25T11:59:24.013 INFO:teuthology.orchestra.run.smithi181.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2021-05-25T11:59:24.013 INFO:teuthology.orchestra.run.smithi181.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-05-25T11:59:24.013 INFO:teuthology.orchestra.run.smithi181.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-05-25T11:59:24.014 INFO:teuthology.orchestra.run.smithi181.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-05-25T11:59:24.376 INFO:teuthology.orchestra.run.smithi181.stdout:Discarding blocks...Done. 2021-05-25T11:59:24.378 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi181.front.sepia.ceph.com -o noatime 2021-05-25T11:59:24.378 DEBUG:teuthology.orchestra.run.smithi181:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-5 2021-05-25T11:59:24.502 DEBUG:teuthology.orchestra.run.smithi181:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2021-05-25T11:59:24.529 DEBUG:teuthology.orchestra.run.smithi181:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2021-05-25T11:59:24.623 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:24.621+0000 7f6b94e240c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2021-05-25T11:59:24.624 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:24.623+0000 7f6b94e240c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2021-05-25T11:59:24.624 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:24.623+0000 7f6b94e240c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2021-05-25T11:59:24.625 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:24.623+0000 7f6b94e240c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2021-05-25T11:59:24.625 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:24.623+0000 7f6b94e240c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2021-05-25T11:59:24.625 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:24.624+0000 7f6b94e240c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2021-05-25T11:59:27.480 DEBUG:teuthology.orchestra.run.smithi181:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2021-05-25T11:59:27.509 DEBUG:teuthology.orchestra.run.smithi181:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2021-05-25T11:59:27.594 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:27.593+0000 7f0ea64620c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2021-05-25T11:59:27.595 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:27.593+0000 7f0ea64620c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2021-05-25T11:59:27.595 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:27.593+0000 7f0ea64620c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2021-05-25T11:59:27.595 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:27.593+0000 7f0ea64620c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2021-05-25T11:59:27.595 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:27.593+0000 7f0ea64620c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2021-05-25T11:59:27.596 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:27.593+0000 7f0ea64620c0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2021-05-25T11:59:30.440 DEBUG:teuthology.orchestra.run.smithi181:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2021-05-25T11:59:30.469 DEBUG:teuthology.orchestra.run.smithi181:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2021-05-25T11:59:30.555 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:30.553+0000 7f2f3431a0c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2021-05-25T11:59:30.555 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:30.553+0000 7f2f3431a0c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2021-05-25T11:59:30.556 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:30.554+0000 7f2f3431a0c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2021-05-25T11:59:30.556 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:30.554+0000 7f2f3431a0c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2021-05-25T11:59:30.556 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:30.554+0000 7f2f3431a0c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2021-05-25T11:59:30.556 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:30.554+0000 7f2f3431a0c0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2021-05-25T11:59:33.404 DEBUG:teuthology.orchestra.run.smithi181:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2021-05-25T11:59:33.433 INFO:tasks.ceph:Reading keys from all nodes... 2021-05-25T11:59:33.434 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2021-05-25T11:59:33.434 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2021-05-25T11:59:33.462 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2021-05-25T11:59:33.462 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2021-05-25T11:59:33.529 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2021-05-25T11:59:33.529 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2021-05-25T11:59:33.594 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2021-05-25T11:59:33.595 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2021-05-25T11:59:33.624 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2021-05-25T11:59:33.624 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2021-05-25T11:59:33.693 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2021-05-25T11:59:33.693 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2021-05-25T11:59:33.760 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2021-05-25T11:59:33.760 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2021-05-25T11:59:33.826 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2021-05-25T11:59:33.826 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2021-05-25T11:59:33.881 INFO:tasks.ceph:Adding keys to all mons... 2021-05-25T11:59:33.882 DEBUG:teuthology.orchestra.run.smithi102:> sudo tee -a /etc/ceph/ceph.keyring 2021-05-25T11:59:33.884 DEBUG:teuthology.orchestra.run.smithi181:> sudo tee -a /etc/ceph/ceph.keyring 2021-05-25T11:59:33.924 INFO:teuthology.orchestra.run.smithi102.stdout:[osd.0] 2021-05-25T11:59:33.924 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQCR5qxgUp5WLxAAxILgEmzGuoxwitZYU04u2Q== 2021-05-25T11:59:33.924 INFO:teuthology.orchestra.run.smithi102.stdout:[osd.1] 2021-05-25T11:59:33.925 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQCU5qxgrNASLRAABicK2C2TZ157syrdDyeoqg== 2021-05-25T11:59:33.925 INFO:teuthology.orchestra.run.smithi102.stdout:[osd.2] 2021-05-25T11:59:33.925 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQCX5qxgu1jvLBAAPNtkWsz4OPjPxnUNcN2yRQ== 2021-05-25T11:59:33.925 INFO:teuthology.orchestra.run.smithi102.stdout:[mgr.x] 2021-05-25T11:59:33.925 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQCP5qxg40/TLBAAnZ9rRtojIQ8Ba3+etIMU1g== 2021-05-25T11:59:33.926 INFO:teuthology.orchestra.run.smithi102.stdout:[osd.3] 2021-05-25T11:59:33.926 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQCc5qxgB10aJRAAB7c821FXx1f8xVlw9O1BXw== 2021-05-25T11:59:33.926 INFO:teuthology.orchestra.run.smithi102.stdout:[osd.4] 2021-05-25T11:59:33.926 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQCf5qxgZsRlIxAA/jBMflLASozcFY1YK06k6g== 2021-05-25T11:59:33.926 INFO:teuthology.orchestra.run.smithi102.stdout:[osd.5] 2021-05-25T11:59:33.927 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQCi5qxggDYMIRAAk6SBx0lM7vmue95dG1SuJA== 2021-05-25T11:59:33.927 INFO:teuthology.orchestra.run.smithi102.stdout:[client.0] 2021-05-25T11:59:33.927 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQCP5qxg0OI+LxAApTRSUyVMRvpgF3/RY3e3RA== 2021-05-25T11:59:33.945 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.0] 2021-05-25T11:59:33.945 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQCR5qxgUp5WLxAAxILgEmzGuoxwitZYU04u2Q== 2021-05-25T11:59:33.945 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.1] 2021-05-25T11:59:33.945 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQCU5qxgrNASLRAABicK2C2TZ157syrdDyeoqg== 2021-05-25T11:59:33.946 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.2] 2021-05-25T11:59:33.946 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQCX5qxgu1jvLBAAPNtkWsz4OPjPxnUNcN2yRQ== 2021-05-25T11:59:33.946 INFO:teuthology.orchestra.run.smithi181.stdout:[mgr.x] 2021-05-25T11:59:33.946 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQCP5qxg40/TLBAAnZ9rRtojIQ8Ba3+etIMU1g== 2021-05-25T11:59:33.946 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.3] 2021-05-25T11:59:33.947 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQCc5qxgB10aJRAAB7c821FXx1f8xVlw9O1BXw== 2021-05-25T11:59:33.947 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.4] 2021-05-25T11:59:33.947 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQCf5qxgZsRlIxAA/jBMflLASozcFY1YK06k6g== 2021-05-25T11:59:33.947 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.5] 2021-05-25T11:59:33.948 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQCi5qxggDYMIRAAk6SBx0lM7vmue95dG1SuJA== 2021-05-25T11:59:33.948 INFO:teuthology.orchestra.run.smithi181.stdout:[client.0] 2021-05-25T11:59:33.948 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQCP5qxg0OI+LxAApTRSUyVMRvpgF3/RY3e3RA== 2021-05-25T11:59:33.949 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-05-25T11:59:33.970 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-05-25T11:59:34.036 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-05-25T11:59:34.039 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-05-25T11:59:34.085 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-05-25T11:59:34.087 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-05-25T11:59:34.133 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2021-05-25T11:59:34.135 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2021-05-25T11:59:34.180 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-05-25T11:59:34.183 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-05-25T11:59:34.229 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-05-25T11:59:34.269 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-05-25T11:59:34.315 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-05-25T11:59:34.318 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-05-25T11:59:34.363 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-05-25T11:59:34.366 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-05-25T11:59:34.414 INFO:tasks.ceph:Running mkfs on mon nodes... 2021-05-25T11:59:34.414 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2021-05-25T11:59:34.439 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2021-05-25T11:59:34.532 DEBUG:teuthology.orchestra.run.smithi102:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2021-05-25T11:59:34.556 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2021-05-25T11:59:34.620 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2021-05-25T11:59:34.711 DEBUG:teuthology.orchestra.run.smithi102:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2021-05-25T11:59:34.736 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2021-05-25T11:59:34.761 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2021-05-25T11:59:34.855 DEBUG:teuthology.orchestra.run.smithi181:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2021-05-25T11:59:34.879 DEBUG:teuthology.orchestra.run.smithi102:> rm -- /home/ubuntu/cephtest/ceph.monmap 2021-05-25T11:59:34.881 DEBUG:teuthology.orchestra.run.smithi181:> rm -- /home/ubuntu/cephtest/ceph.monmap 2021-05-25T11:59:34.933 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2021-05-25T11:59:34.933 INFO:tasks.ceph.mon.a:Restarting daemon 2021-05-25T11:59:34.934 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2021-05-25T11:59:34.936 INFO:tasks.ceph.mon.a:Started 2021-05-25T11:59:34.937 INFO:tasks.ceph.mon.c:Restarting daemon 2021-05-25T11:59:34.937 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2021-05-25T11:59:34.939 INFO:tasks.ceph.mon.c:Started 2021-05-25T11:59:34.939 INFO:tasks.ceph.mon.b:Restarting daemon 2021-05-25T11:59:34.940 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2021-05-25T11:59:34.975 INFO:tasks.ceph.mon.b:Started 2021-05-25T11:59:34.976 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2021-05-25T11:59:34.976 INFO:tasks.ceph.mgr.x:Restarting daemon 2021-05-25T11:59:34.976 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2021-05-25T11:59:34.979 INFO:tasks.ceph.mgr.x:Started 2021-05-25T11:59:34.979 INFO:tasks.ceph:Setting crush tunables to default 2021-05-25T11:59:34.979 DEBUG:teuthology.orchestra.run.smithi102:> sudo ceph --cluster ceph osd crush tunables default 2021-05-25T11:59:35.090 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:35.089+0000 7f747cb1f700 10 monclient: get_monmap_and_config 2021-05-25T11:59:35.091 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:35.089+0000 7f747cb1f700 10 monclient: build_initial_monmap 2021-05-25T11:59:35.091 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:35.089+0000 7f747cb1f700 10 monclient: monmap: 2021-05-25T11:59:35.091 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T11:59:35.091 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:35.092 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:35.090141+0000 2021-05-25T11:59:35.092 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:35.090141+0000 2021-05-25T11:59:35.092 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T11:59:35.092 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:35.092 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T11:59:35.093 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T11:59:35.093 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T11:59:35.093 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:35.096 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:35.089+0000 7f747cb1f700 1 Processor -- start 2021-05-25T11:59:35.096 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:35.089+0000 7f747cb1f700 1 -- start start 2021-05-25T11:59:35.096 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:35.089+0000 7f747cb1f700 10 monclient: init 2021-05-25T11:59:35.096 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:35.090+0000 7f747cb1f700 10 monclient: _reopen_session rank -1 2021-05-25T11:59:35.097 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:35.090+0000 7f747cb1f700 10 monclient: _add_conns ranks=[0,1,2] 2021-05-25T11:59:35.097 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:35.090+0000 7f747cb1f700 10 monclient(hunting): picked mon.noname-a con 0x7f7478096130 addr v1:172.21.15.102:6789/0 2021-05-25T11:59:35.097 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:35.090+0000 7f747cb1f700 10 monclient(hunting): picked mon.noname-c con 0x7f7478096510 addr v1:172.21.15.181:6789/0 2021-05-25T11:59:35.097 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:35.090+0000 7f747cb1f700 10 monclient(hunting): picked mon.noname-b con 0x7f747812e3b0 addr v1:172.21.15.102:6790/0 2021-05-25T11:59:35.097 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:35.090+0000 7f747cb1f700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74780926f0 con 0x7f7478096130 2021-05-25T11:59:35.098 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:35.090+0000 7f747cb1f700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7478092990 con 0x7f7478096510 2021-05-25T11:59:35.098 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:35.090+0000 7f747cb1f700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7478092c30 con 0x7f747812e3b0 2021-05-25T11:59:35.098 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:35.090+0000 7f747cb1f700 10 monclient(hunting): _renew_subs 2021-05-25T11:59:35.098 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:35.090+0000 7f747cb1f700 10 monclient(hunting): authenticate will time out at 775.687521s 2021-05-25T11:59:35.099 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:35.090+0000 7f74777fe700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f7478096130 0x7f7478093290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:58460/0 (socket says 172.21.15.102:58460) 2021-05-25T11:59:35.099 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:35.090+0000 7f7477fff700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f747812e3b0 0x7f74781307a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:53340/0 (socket says 172.21.15.102:53340) 2021-05-25T11:59:35.099 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:35.090+0000 7f74777fe700 1 -- 172.21.15.102:0/2327458690 learned_addr learned my addr 172.21.15.102:0/2327458690 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T11:59:38.092 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:38.090+0000 7f74757fa700 10 monclient(hunting): tick 2021-05-25T11:59:38.093 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:38.090+0000 7f74757fa700 1 monclient(hunting): continuing hunt 2021-05-25T11:59:38.093 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:38.090+0000 7f74757fa700 10 monclient(hunting): _reopen_session rank -1 2021-05-25T11:59:38.093 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:38.090+0000 7f74757fa700 1 -- 172.21.15.102:0/2327458690 >> v1:172.21.15.102:6790/0 conn(0x7f747812e3b0 legacy=0x7f74781307a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:38.093 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:38.090+0000 7f74757fa700 1 -- 172.21.15.102:0/2327458690 >> v1:172.21.15.181:6789/0 conn(0x7f7478096510 legacy=0x7f74780969c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:38.094 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:38.090+0000 7f74757fa700 1 -- 172.21.15.102:0/2327458690 >> v1:172.21.15.102:6789/0 conn(0x7f7478096130 legacy=0x7f7478093290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:38.094 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:38.090+0000 7f74757fa700 10 monclient: _add_conns ranks=[1,2,0] 2021-05-25T11:59:38.094 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:38.090+0000 7f74757fa700 10 monclient(hunting): picked mon.noname-c con 0x7f7460003330 addr v1:172.21.15.181:6789/0 2021-05-25T11:59:38.094 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:38.090+0000 7f74757fa700 10 monclient(hunting): picked mon.noname-b con 0x7f747812e3b0 addr v1:172.21.15.102:6790/0 2021-05-25T11:59:38.095 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:38.090+0000 7f74757fa700 10 monclient(hunting): picked mon.noname-a con 0x7f746000ccd0 addr v1:172.21.15.102:6789/0 2021-05-25T11:59:38.095 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:38.090+0000 7f74757fa700 1 -- 172.21.15.102:0/2327458690 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7478092ed0 con 0x7f746000ccd0 2021-05-25T11:59:38.095 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:38.090+0000 7f74757fa700 1 -- 172.21.15.102:0/2327458690 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74600117f0 con 0x7f7460003330 2021-05-25T11:59:38.095 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:38.090+0000 7f74757fa700 1 -- 172.21.15.102:0/2327458690 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7460012a00 con 0x7f747812e3b0 2021-05-25T11:59:38.095 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:38.090+0000 7f74757fa700 10 monclient(hunting): _renew_subs 2021-05-25T11:59:41.092 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.091+0000 7f74757fa700 10 monclient(hunting): tick 2021-05-25T11:59:41.093 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.091+0000 7f74757fa700 1 monclient(hunting): continuing hunt 2021-05-25T11:59:41.093 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.091+0000 7f74757fa700 10 monclient(hunting): _reopen_session rank -1 2021-05-25T11:59:41.093 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.091+0000 7f74757fa700 1 -- 172.21.15.102:0/2327458690 >> v1:172.21.15.102:6790/0 conn(0x7f747812e3b0 legacy=0x7f746000b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:41.093 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.091+0000 7f74757fa700 1 -- 172.21.15.102:0/2327458690 >> v1:172.21.15.181:6789/0 conn(0x7f7460003330 legacy=0x7f7460003750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:41.094 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.091+0000 7f74757fa700 1 -- 172.21.15.102:0/2327458690 >> v1:172.21.15.102:6789/0 conn(0x7f746000ccd0 legacy=0x7f746000f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:41.094 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.091+0000 7f74757fa700 10 monclient: _add_conns ranks=[0,2,1] 2021-05-25T11:59:41.094 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.091+0000 7f74757fa700 10 monclient(hunting): picked mon.noname-a con 0x7f7460012d90 addr v1:172.21.15.102:6789/0 2021-05-25T11:59:41.094 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.091+0000 7f74757fa700 10 monclient(hunting): picked mon.noname-b con 0x7f74600107e0 addr v1:172.21.15.102:6790/0 2021-05-25T11:59:41.095 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.091+0000 7f74757fa700 10 monclient(hunting): picked mon.noname-c con 0x7f7460003330 addr v1:172.21.15.181:6789/0 2021-05-25T11:59:41.095 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.091+0000 7f74757fa700 1 -- 172.21.15.102:0/2327458690 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7460013cd0 con 0x7f7460012d90 2021-05-25T11:59:41.095 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.091+0000 7f74757fa700 1 -- 172.21.15.102:0/2327458690 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7460011370 con 0x7f7460003330 2021-05-25T11:59:41.095 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.091+0000 7f74757fa700 1 -- 172.21.15.102:0/2327458690 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7460011550 con 0x7f74600107e0 2021-05-25T11:59:41.096 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.091+0000 7f74757fa700 10 monclient(hunting): _renew_subs 2021-05-25T11:59:41.254 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.249+0000 7f74767fc700 1 -- 172.21.15.102:0/2327458690 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 103764805 0 0) 0x7f7460013cd0 con 0x7f7460012d90 2021-05-25T11:59:41.254 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.249+0000 7f74767fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:41.254 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.249+0000 7f74767fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:41.255 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.249+0000 7f74767fc700 10 monclient(hunting): my global_id is 4098 2021-05-25T11:59:41.255 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.249+0000 7f74767fc700 1 -- 172.21.15.102:0/2327458690 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f745c003270 con 0x7f7460012d90 2021-05-25T11:59:41.255 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.249+0000 7f74767fc700 1 -- 172.21.15.102:0/2327458690 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1422492182 0 0) 0x7f7460011370 con 0x7f7460003330 2021-05-25T11:59:41.255 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.249+0000 7f74767fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:41.255 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.249+0000 7f74767fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:41.256 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.249+0000 7f74767fc700 10 monclient(hunting): my global_id is 4102 2021-05-25T11:59:41.256 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.249+0000 7f74767fc700 1 -- 172.21.15.102:0/2327458690 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7460013cd0 con 0x7f7460003330 2021-05-25T11:59:41.256 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.250+0000 7f74767fc700 1 -- 172.21.15.102:0/2327458690 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1448375990 0 0) 0x7f7460011550 con 0x7f74600107e0 2021-05-25T11:59:41.256 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.250+0000 7f74767fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:41.257 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.250+0000 7f74767fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:41.257 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.250+0000 7f74767fc700 10 monclient(hunting): my global_id is 4103 2021-05-25T11:59:41.257 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.250+0000 7f74767fc700 1 -- 172.21.15.102:0/2327458690 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7460011370 con 0x7f74600107e0 2021-05-25T11:59:41.257 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.250+0000 7f74767fc700 1 -- 172.21.15.102:0/2327458690 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1763153868 0 0) 0x7f745c003270 con 0x7f7460012d90 2021-05-25T11:59:41.257 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.250+0000 7f74767fc700 1 -- 172.21.15.102:0/2327458690 >> v1:172.21.15.102:6790/0 conn(0x7f74600107e0 legacy=0x7f7460010c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:41.258 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.250+0000 7f74767fc700 1 -- 172.21.15.102:0/2327458690 >> v1:172.21.15.181:6789/0 conn(0x7f7460003330 legacy=0x7f7460003750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:41.258 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.250+0000 7f74767fc700 10 monclient: _finish_hunting 0 2021-05-25T11:59:41.258 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.250+0000 7f74767fc700 1 monclient: found mon.noname-a 2021-05-25T11:59:41.258 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.250+0000 7f74767fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T11:59:41.259 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.250+0000 7f74767fc700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T11:59:41.259 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.250+0000 7f74767fc700 1 -- 172.21.15.102:0/2327458690 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f746000c5d0 con 0x7f7460012d90 2021-05-25T11:59:41.259 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.250+0000 7f74767fc700 10 monclient: _finish_auth 0 2021-05-25T11:59:41.259 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.250+0000 7f74767fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T11:59:41.259 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.250+0000 7f74767fc700 1 -- 172.21.15.102:0/2327458690 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f7464001590 con 0x7f7460012d90 2021-05-25T11:59:41.260 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.250+0000 7f74767fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:41.260 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.250+0000 7f74767fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T11:59:41.260 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.250+0000 7f74767fc700 10 monclient: dump: 2021-05-25T11:59:41.260 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:41.260 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:41.261 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:41.261 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:41.261 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:41.261 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:41.261 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:41.262 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:41.262 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:41.262 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:41.262 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.250+0000 7f747cb1f700 5 monclient: authenticate success, global_id 4098 2021-05-25T11:59:41.262 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.250+0000 7f747cb1f700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T11:59:41.263 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.250+0000 7f74767fc700 1 -- 172.21.15.102:0/2327458690 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7464001830 con 0x7f7460012d90 2021-05-25T11:59:41.263 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.250+0000 7f74767fc700 10 monclient: handle_config config(0 keys) v1 2021-05-25T11:59:41.263 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.250+0000 7f74767fc700 1 -- 172.21.15.102:0/2327458690 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f7464001c00 con 0x7f7460012d90 2021-05-25T11:59:41.263 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.250+0000 7f747cb1f700 10 monclient: get_monmap_and_config success 2021-05-25T11:59:41.264 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.250+0000 7f747cb1f700 4 set_mon_vals no callback set 2021-05-25T11:59:41.264 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.250+0000 7f74767fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:41.264 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.250+0000 7f74767fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T11:59:41.264 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.250+0000 7f74767fc700 10 monclient: dump: 2021-05-25T11:59:41.264 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:41.265 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:41.265 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:41.265 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:41.265 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:41.265 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:41.265 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:41.266 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:41.266 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:41.266 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:41.266 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.251+0000 7f747cb1f700 10 monclient: shutdown 2021-05-25T11:59:41.267 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.251+0000 7f747cb1f700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T11:59:41.267 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.251+0000 7f747cb1f700 1 -- 172.21.15.102:0/2327458690 >> v1:172.21.15.102:6789/0 conn(0x7f7460012d90 legacy=0x7f74600131d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:41.267 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.251+0000 7f747cb1f700 1 -- 172.21.15.102:0/2327458690 shutdown_connections 2021-05-25T11:59:41.267 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.251+0000 7f747cb1f700 1 -- 172.21.15.102:0/2327458690 >> 172.21.15.102:0/2327458690 conn(0x7f7478097cc0 msgr2=0x7f7478061b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T11:59:41.267 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.251+0000 7f747cb1f700 1 -- 172.21.15.102:0/2327458690 shutdown_connections 2021-05-25T11:59:41.268 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.251+0000 7f747cb1f700 1 -- 172.21.15.102:0/2327458690 wait complete. 2021-05-25T11:59:41.268 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.252+0000 7f747cb1f700 10 monclient: build_initial_monmap 2021-05-25T11:59:41.268 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.252+0000 7f747cb1f700 10 monclient: monmap: 2021-05-25T11:59:41.268 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T11:59:41.268 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T11:59:41.269 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T11:59:41.269 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T11:59:41.269 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T11:59:41.269 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T11:59:41.269 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T11:59:41.270 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T11:59:41.270 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T11:59:41.270 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:41.270 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.252+0000 7f747cb1f700 1 librados: starting msgr at 2021-05-25T11:59:41.270 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.252+0000 7f747cb1f700 1 librados: starting objecter 2021-05-25T11:59:41.271 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.252+0000 7f747cb1f700 1 Processor -- start 2021-05-25T11:59:41.271 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.252+0000 7f747cb1f700 1 -- start start 2021-05-25T11:59:41.271 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.252+0000 7f747cb1f700 1 librados: setting wanted keys 2021-05-25T11:59:41.271 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.252+0000 7f747cb1f700 1 librados: calling monclient init 2021-05-25T11:59:41.271 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.252+0000 7f747cb1f700 10 monclient: init 2021-05-25T11:59:41.276 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.253+0000 7f747cb1f700 10 monclient: _reopen_session rank -1 2021-05-25T11:59:41.276 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.253+0000 7f747cb1f700 10 monclient: _add_conns ranks=[2,1,0] 2021-05-25T11:59:41.277 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.253+0000 7f747cb1f700 10 monclient(hunting): picked mon.noname-c con 0x7f7460003330 addr v1:172.21.15.102:6790/0 2021-05-25T11:59:41.277 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.253+0000 7f747cb1f700 10 monclient(hunting): picked mon.noname-b con 0x7f74600107e0 addr v1:172.21.15.181:6789/0 2021-05-25T11:59:41.277 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.253+0000 7f747cb1f700 10 monclient(hunting): picked mon.noname-a con 0x7f7460012d90 addr v1:172.21.15.102:6789/0 2021-05-25T11:59:41.277 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.253+0000 7f747cb1f700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f747812d990 con 0x7f7460012d90 2021-05-25T11:59:41.277 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.253+0000 7f747cb1f700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7478112580 con 0x7f74600107e0 2021-05-25T11:59:41.278 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.253+0000 7f747cb1f700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7478128d50 con 0x7f7460003330 2021-05-25T11:59:41.278 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.253+0000 7f747cb1f700 10 monclient(hunting): _renew_subs 2021-05-25T11:59:41.278 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.253+0000 7f747cb1f700 10 monclient(hunting): authenticate will time out at 781.850518s 2021-05-25T11:59:41.278 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.253+0000 7f74777fe700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f7460003330 0x7f7478067ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:53364/0 (socket says 172.21.15.102:53364) 2021-05-25T11:59:41.278 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.253+0000 7f74777fe700 1 -- 172.21.15.102:0/2236435465 learned_addr learned my addr 172.21.15.102:0/2236435465 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T11:59:41.279 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.253+0000 7f745b7fe700 10 client.?.objecter ms_handle_connect 0x7f7460003330 2021-05-25T11:59:41.279 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.253+0000 7f745b7fe700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:41.279 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.253+0000 7f745b7fe700 10 client.?.objecter ms_handle_connect 0x7f74600107e0 2021-05-25T11:59:41.279 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.253+0000 7f745b7fe700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:41.279 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.253+0000 7f745b7fe700 10 client.?.objecter ms_handle_connect 0x7f7460012d90 2021-05-25T11:59:41.280 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.253+0000 7f745b7fe700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:41.280 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.254+0000 7f745b7fe700 1 -- 172.21.15.102:0/2236435465 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 478457746 0 0) 0x7f7478112580 con 0x7f74600107e0 2021-05-25T11:59:41.280 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.254+0000 7f745b7fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:41.280 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.254+0000 7f745b7fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:41.281 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.254+0000 7f745b7fe700 10 monclient(hunting): my global_id is 4108 2021-05-25T11:59:41.281 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.254+0000 7f745b7fe700 1 -- 172.21.15.102:0/2236435465 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f746000ca70 con 0x7f74600107e0 2021-05-25T11:59:41.281 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.254+0000 7f745b7fe700 1 -- 172.21.15.102:0/2236435465 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3556552318 0 0) 0x7f746000ca70 con 0x7f74600107e0 2021-05-25T11:59:41.281 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.254+0000 7f745b7fe700 1 -- 172.21.15.102:0/2236435465 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7478112580 con 0x7f74600107e0 2021-05-25T11:59:41.281 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.254+0000 7f745b7fe700 1 -- 172.21.15.102:0/2236435465 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f7468001880 con 0x7f74600107e0 2021-05-25T11:59:41.282 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.254+0000 7f745b7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:41.282 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.254+0000 7f745b7fe700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T11:59:41.282 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.254+0000 7f745b7fe700 10 monclient(hunting): dump: 2021-05-25T11:59:41.282 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:41.282 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:41.283 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:41.283 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:41.283 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:41.283 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:41.283 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:41.284 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:41.284 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:41.284 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:41.284 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.254+0000 7f745b7fe700 10 monclient(hunting): _finish_auth 0 2021-05-25T11:59:41.284 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.254+0000 7f745b7fe700 1 -- 172.21.15.102:0/2236435465 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 809228603 0 0) 0x7f747812d990 con 0x7f7460012d90 2021-05-25T11:59:41.285 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.254+0000 7f745b7fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:41.285 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.254+0000 7f745b7fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:41.285 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.254+0000 7f745b7fe700 10 monclient(hunting): my global_id is 4107 2021-05-25T11:59:41.285 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.254+0000 7f745b7fe700 1 -- 172.21.15.102:0/2236435465 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f746000ca70 con 0x7f7460012d90 2021-05-25T11:59:41.285 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.254+0000 7f745b7fe700 1 -- 172.21.15.102:0/2236435465 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2119185284 0 0) 0x7f7478112580 con 0x7f74600107e0 2021-05-25T11:59:41.286 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.254+0000 7f745b7fe700 1 -- 172.21.15.102:0/2236435465 >> v1:172.21.15.102:6790/0 conn(0x7f7460003330 legacy=0x7f7478067ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:41.286 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.254+0000 7f745b7fe700 1 -- 172.21.15.102:0/2236435465 >> v1:172.21.15.102:6789/0 conn(0x7f7460012d90 legacy=0x7f747813f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:41.286 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.254+0000 7f745b7fe700 10 monclient: _finish_hunting 0 2021-05-25T11:59:41.286 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.254+0000 7f745b7fe700 1 monclient: found mon.b 2021-05-25T11:59:41.286 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.254+0000 7f745b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T11:59:41.287 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.254+0000 7f745b7fe700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T11:59:41.287 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.254+0000 7f745b7fe700 1 -- 172.21.15.102:0/2236435465 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7478129170 con 0x7f74600107e0 2021-05-25T11:59:41.287 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.254+0000 7f745b7fe700 10 monclient: _finish_auth 0 2021-05-25T11:59:41.287 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.254+0000 7f745b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T11:59:41.287 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.254+0000 7f747cb1f700 5 monclient: authenticate success, global_id 4108 2021-05-25T11:59:41.288 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.254+0000 7f747cb1f700 10 monclient: _renew_subs 2021-05-25T11:59:41.288 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.254+0000 7f747cb1f700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T11:59:41.288 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.254+0000 7f747cb1f700 1 -- 172.21.15.102:0/2236435465 --> v1:172.21.15.181:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74781130b0 con 0x7f74600107e0 2021-05-25T11:59:41.288 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.254+0000 7f747cb1f700 10 client.4108.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T11:59:41.288 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.254+0000 7f747cb1f700 10 monclient: _renew_subs 2021-05-25T11:59:41.289 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.254+0000 7f747cb1f700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T11:59:41.289 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.254+0000 7f747cb1f700 1 -- 172.21.15.102:0/2236435465 --> v1:172.21.15.181:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7478068630 con 0x7f74600107e0 2021-05-25T11:59:41.289 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.254+0000 7f747cb1f700 1 librados: init done 2021-05-25T11:59:41.289 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.255+0000 7f745b7fe700 1 -- 172.21.15.102:0/2236435465 <== mon.1 v1:172.21.15.181:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7468001ec0 con 0x7f74600107e0 2021-05-25T11:59:41.289 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.255+0000 7f745b7fe700 10 monclient: handle_config config(0 keys) v1 2021-05-25T11:59:41.290 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.255+0000 7f745b7fe700 1 -- 172.21.15.102:0/2236435465 <== mon.1 v1:172.21.15.181:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f7468002030 con 0x7f74600107e0 2021-05-25T11:59:41.290 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.255+0000 7f745b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:41.290 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.255+0000 7f745b7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T11:59:41.290 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.255+0000 7f7475ffb700 4 set_mon_vals no callback set 2021-05-25T11:59:41.290 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.255+0000 7f745b7fe700 10 monclient: dump: 2021-05-25T11:59:41.291 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:41.291 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:41.291 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:41.291 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:41.292 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:41.292 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:41.292 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:41.292 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:41.292 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:41.292 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:41.293 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.255+0000 7f745b7fe700 1 -- 172.21.15.102:0/2236435465 <== mon.1 v1:172.21.15.181:6789/0 7 ==== mgrmap(e 1) v1 ==== 623+0+0 (unknown 1065898531 0 0) 0x7f74680024b0 con 0x7f74600107e0 2021-05-25T11:59:41.293 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.255+0000 7f745b7fe700 1 -- 172.21.15.102:0/2236435465 <== mon.1 v1:172.21.15.181:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (unknown 1288797299 0 0) 0x7f7468003630 con 0x7f74600107e0 2021-05-25T11:59:41.293 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.255+0000 7f745b7fe700 10 client.4108.objecter ms_dispatch 0x7f7478001550 osd_map(1..1 src has 1..1) v4 2021-05-25T11:59:41.293 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.255+0000 7f745b7fe700 3 client.4108.objecter handle_osd_map got epochs [1,1] > 0 2021-05-25T11:59:41.293 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.255+0000 7f745b7fe700 3 client.4108.objecter handle_osd_map decoding full epoch 1 2021-05-25T11:59:41.294 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.255+0000 7f745b7fe700 20 client.4108.objecter dump_active .. 0 homeless 2021-05-25T11:59:41.294 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.255+0000 7f747cb1f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:41.294 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.255+0000 7f747cb1f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:41.294 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.255+0000 7f747cb1f700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T11:59:41.294 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.255+0000 7f747cb1f700 1 -- 172.21.15.102:0/2236435465 --> v1:172.21.15.181:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7454004db0 con 0x7f74600107e0 2021-05-25T11:59:41.295 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.259+0000 7f745b7fe700 1 -- 172.21.15.102:0/2236435465 <== mon.1 v1:172.21.15.181:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72237 (unknown 1092875540 0 1415944542) 0x7f7468002a30 con 0x7f74600107e0 2021-05-25T11:59:41.295 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.259+0000 7f745b7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:41.295 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.259+0000 7f745b7fe700 10 monclient: _finish_command 1 = system:0 2021-05-25T11:59:41.332 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.330+0000 7f747cb1f700 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2021-05-25T11:59:41.332 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.330+0000 7f747cb1f700 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2021-05-25T11:59:41.332 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.330+0000 7f747cb1f700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T11:59:41.332 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.330+0000 7f747cb1f700 1 -- 172.21.15.102:0/2236435465 --> v1:172.21.15.181:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f7454005060 con 0x7f74600107e0 2021-05-25T11:59:41.339 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.338+0000 7f745b7fe700 1 -- 172.21.15.102:0/2236435465 <== mon.1 v1:172.21.15.181:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7f7468003930 con 0x7f74600107e0 2021-05-25T11:59:41.340 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.338+0000 7f745b7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2021-05-25T11:59:41.340 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.338+0000 7f745b7fe700 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2021-05-25T11:59:41.341 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.339+0000 7f747cb1f700 10 librados: watch_flush enter 2021-05-25T11:59:41.342 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.339+0000 7f747cb1f700 10 librados: watch_flush exit 2021-05-25T11:59:41.342 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.339+0000 7f747cb1f700 20 client.4108.objecter shutdown clearing up homeless session... 2021-05-25T11:59:41.342 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.339+0000 7f747cb1f700 10 client.4108.objecter successfully canceled tick 2021-05-25T11:59:41.342 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.339+0000 7f747cb1f700 10 monclient: shutdown 2021-05-25T11:59:41.342 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.339+0000 7f747cb1f700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T11:59:41.343 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.339+0000 7f747cb1f700 1 -- 172.21.15.102:0/2236435465 >> v1:172.21.15.181:6789/0 conn(0x7f74600107e0 legacy=0x7f7478128640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:41.343 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.339+0000 7f747cb1f700 1 -- 172.21.15.102:0/2236435465 shutdown_connections 2021-05-25T11:59:41.343 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.339+0000 7f747cb1f700 1 -- 172.21.15.102:0/2236435465 >> 172.21.15.102:0/2236435465 conn(0x7f7478097cc0 msgr2=0x7f7478093ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T11:59:41.343 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.339+0000 7f747cb1f700 1 -- 172.21.15.102:0/2236435465 shutdown_connections 2021-05-25T11:59:41.344 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.339+0000 7f747cb1f700 1 -- 172.21.15.102:0/2236435465 wait complete. 2021-05-25T11:59:41.344 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:41.339+0000 7f747cb1f700 1 librados: shutdown 2021-05-25T11:59:41.344 INFO:teuthology.orchestra.run.smithi102.stderr:adjusted tunables profile to default 2021-05-25T11:59:41.352 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2021-05-25T11:59:41.353 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2021-05-25T11:59:41.353 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2021-05-25T11:59:41.378 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2021-05-25T11:59:41.378 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2021-05-25T11:59:41.442 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2021-05-25T11:59:41.442 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2021-05-25T11:59:41.505 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2021-05-25T11:59:41.506 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2021-05-25T11:59:41.531 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2021-05-25T11:59:41.531 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2021-05-25T11:59:41.595 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2021-05-25T11:59:41.595 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2021-05-25T11:59:41.659 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster ceph osd new 8f1f285f-7041-4c53-ab09-f638ab670036 0 2021-05-25T11:59:41.809 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.807+0000 7f041f2ff700 10 monclient: get_monmap_and_config 2021-05-25T11:59:41.810 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.807+0000 7f041f2ff700 10 monclient: build_initial_monmap 2021-05-25T11:59:41.810 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.807+0000 7f041f2ff700 10 monclient: monmap: 2021-05-25T11:59:41.810 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 0 2021-05-25T11:59:41.811 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:41.811 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 2021-05-25T11:59:41.808949+0000 2021-05-25T11:59:41.811 INFO:teuthology.orchestra.run.smithi181.stderr:created 2021-05-25T11:59:41.808949+0000 2021-05-25T11:59:41.811 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 0 (unknown) 2021-05-25T11:59:41.811 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 3 2021-05-25T11:59:41.812 INFO:teuthology.orchestra.run.smithi181.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T11:59:41.812 INFO:teuthology.orchestra.run.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T11:59:41.812 INFO:teuthology.orchestra.run.smithi181.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T11:59:41.812 INFO:teuthology.orchestra.run.smithi181.stderr: 2021-05-25T11:59:41.812 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.807+0000 7f041f2ff700 1 Processor -- start 2021-05-25T11:59:41.813 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.807+0000 7f041f2ff700 1 -- start start 2021-05-25T11:59:41.813 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.807+0000 7f041f2ff700 10 monclient: init 2021-05-25T11:59:41.823 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.808+0000 7f041f2ff700 10 monclient: _reopen_session rank -1 2021-05-25T11:59:41.824 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.808+0000 7f041f2ff700 10 monclient: _add_conns ranks=[2,0,1] 2021-05-25T11:59:41.824 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.808+0000 7f041f2ff700 10 monclient(hunting): picked mon.noname-b con 0x7f0418096b00 addr v1:172.21.15.102:6790/0 2021-05-25T11:59:41.824 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.808+0000 7f041f2ff700 10 monclient(hunting): picked mon.noname-a con 0x7f0418096ee0 addr v1:172.21.15.102:6789/0 2021-05-25T11:59:41.824 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.808+0000 7f041f2ff700 10 monclient(hunting): picked mon.noname-c con 0x7f041812e390 addr v1:172.21.15.181:6789/0 2021-05-25T11:59:41.824 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.808+0000 7f041f2ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0418092800 con 0x7f0418096ee0 2021-05-25T11:59:41.825 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.808+0000 7f041f2ff700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0418092aa0 con 0x7f041812e390 2021-05-25T11:59:41.825 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.808+0000 7f041f2ff700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0418092d40 con 0x7f0418096b00 2021-05-25T11:59:41.825 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.808+0000 7f041f2ff700 10 monclient(hunting): _renew_subs 2021-05-25T11:59:41.825 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.808+0000 7f041f2ff700 10 monclient(hunting): authenticate will time out at 781.292464s 2021-05-25T11:59:41.825 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.808+0000 7f041d89c700 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f041812e390 0x7f0418130780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:34546/0 (socket says 172.21.15.181:34546) 2021-05-25T11:59:41.826 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.808+0000 7f041d89c700 1 -- 172.21.15.181:0/657832412 learned_addr learned my addr 172.21.15.181:0/657832412 (peer_addr_for_me v1:172.21.15.181:0/0) 2021-05-25T11:59:41.826 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.808+0000 7f0407fff700 1 -- 172.21.15.181:0/657832412 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4083477126 0 0) 0x7f0418092aa0 con 0x7f041812e390 2021-05-25T11:59:41.826 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.808+0000 7f0407fff700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:41.826 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.808+0000 7f0407fff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:41.826 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.808+0000 7f0407fff700 10 monclient(hunting): my global_id is 4111 2021-05-25T11:59:41.827 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.808+0000 7f0407fff700 1 -- 172.21.15.181:0/657832412 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0400003270 con 0x7f041812e390 2021-05-25T11:59:41.827 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.808+0000 7f0407fff700 1 -- 172.21.15.181:0/657832412 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2126934292 0 0) 0x7f0418092d40 con 0x7f0418096b00 2021-05-25T11:59:41.827 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.808+0000 7f0407fff700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:41.827 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.808+0000 7f0407fff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:41.827 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.808+0000 7f0407fff700 10 monclient(hunting): my global_id is 4109 2021-05-25T11:59:41.828 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.808+0000 7f0407fff700 1 -- 172.21.15.181:0/657832412 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0418092aa0 con 0x7f0418096b00 2021-05-25T11:59:41.828 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.808+0000 7f0407fff700 1 -- 172.21.15.181:0/657832412 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2949775327 0 0) 0x7f0418092800 con 0x7f0418096ee0 2021-05-25T11:59:41.828 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.808+0000 7f0407fff700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:41.828 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.808+0000 7f0407fff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:41.828 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.808+0000 7f0407fff700 10 monclient(hunting): my global_id is 4110 2021-05-25T11:59:41.829 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.808+0000 7f0407fff700 1 -- 172.21.15.181:0/657832412 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0418092d40 con 0x7f0418096ee0 2021-05-25T11:59:41.829 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.809+0000 7f0407fff700 1 -- 172.21.15.181:0/657832412 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2278202486 0 0) 0x7f0400003270 con 0x7f041812e390 2021-05-25T11:59:41.829 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.809+0000 7f0407fff700 1 -- 172.21.15.181:0/657832412 >> v1:172.21.15.102:6790/0 conn(0x7f0418096b00 legacy=0x7f04180933a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:41.829 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.809+0000 7f0407fff700 1 -- 172.21.15.181:0/657832412 >> v1:172.21.15.102:6789/0 conn(0x7f0418096ee0 legacy=0x7f0418097360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:41.829 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.809+0000 7f0407fff700 10 monclient: _finish_hunting 0 2021-05-25T11:59:41.830 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.809+0000 7f0407fff700 1 monclient: found mon.noname-c 2021-05-25T11:59:41.830 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.809+0000 7f0407fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T11:59:41.830 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.809+0000 7f0407fff700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.181:6789/0 2021-05-25T11:59:41.830 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.809+0000 7f0407fff700 1 -- 172.21.15.181:0/657832412 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0418092fe0 con 0x7f041812e390 2021-05-25T11:59:41.831 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.809+0000 7f0407fff700 10 monclient: _finish_auth 0 2021-05-25T11:59:41.831 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.809+0000 7f0407fff700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T11:59:41.831 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.809+0000 7f041f2ff700 5 monclient: authenticate success, global_id 4111 2021-05-25T11:59:41.831 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.809+0000 7f0407fff700 1 -- 172.21.15.181:0/657832412 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f0414001990 con 0x7f041812e390 2021-05-25T11:59:41.831 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.809+0000 7f0407fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:41.832 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.809+0000 7f0407fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T11:59:41.832 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.809+0000 7f0407fff700 10 monclient: dump: 2021-05-25T11:59:41.832 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 1 2021-05-25T11:59:41.832 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:41.832 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:41.833 INFO:teuthology.orchestra.run.smithi181.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:41.833 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:41.833 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 3 2021-05-25T11:59:41.833 INFO:teuthology.orchestra.run.smithi181.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:41.834 INFO:teuthology.orchestra.run.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:41.834 INFO:teuthology.orchestra.run.smithi181.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:41.834 INFO:teuthology.orchestra.run.smithi181.stderr: 2021-05-25T11:59:41.834 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.809+0000 7f041f2ff700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T11:59:41.834 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.809+0000 7f0407fff700 1 -- 172.21.15.181:0/657832412 <== mon.1 v1:172.21.15.181:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0414001c00 con 0x7f041812e390 2021-05-25T11:59:41.835 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.809+0000 7f0407fff700 10 monclient: handle_config config(0 keys) v1 2021-05-25T11:59:41.835 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.809+0000 7f041f2ff700 10 monclient: get_monmap_and_config success 2021-05-25T11:59:41.835 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.809+0000 7f041f2ff700 4 set_mon_vals no callback set 2021-05-25T11:59:41.835 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.809+0000 7f0407fff700 1 -- 172.21.15.181:0/657832412 <== mon.1 v1:172.21.15.181:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f0414001f90 con 0x7f041812e390 2021-05-25T11:59:41.836 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.809+0000 7f0407fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:41.836 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.809+0000 7f0407fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T11:59:41.836 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.809+0000 7f0407fff700 10 monclient: dump: 2021-05-25T11:59:41.836 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 1 2021-05-25T11:59:41.836 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:41.836 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:41.837 INFO:teuthology.orchestra.run.smithi181.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:41.837 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:41.837 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 3 2021-05-25T11:59:41.837 INFO:teuthology.orchestra.run.smithi181.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:41.837 INFO:teuthology.orchestra.run.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:41.838 INFO:teuthology.orchestra.run.smithi181.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:41.838 INFO:teuthology.orchestra.run.smithi181.stderr: 2021-05-25T11:59:41.838 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.809+0000 7f041f2ff700 10 monclient: shutdown 2021-05-25T11:59:41.838 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.809+0000 7f041f2ff700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T11:59:41.839 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.809+0000 7f041f2ff700 1 -- 172.21.15.181:0/657832412 >> v1:172.21.15.181:6789/0 conn(0x7f041812e390 legacy=0x7f0418130780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:41.839 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.809+0000 7f041f2ff700 1 -- 172.21.15.181:0/657832412 shutdown_connections 2021-05-25T11:59:41.839 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.809+0000 7f041f2ff700 1 -- 172.21.15.181:0/657832412 >> 172.21.15.181:0/657832412 conn(0x7f0418061f00 msgr2=0x7f0418062310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T11:59:41.839 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.809+0000 7f041f2ff700 1 -- 172.21.15.181:0/657832412 shutdown_connections 2021-05-25T11:59:41.839 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.809+0000 7f041f2ff700 1 -- 172.21.15.181:0/657832412 wait complete. 2021-05-25T11:59:41.840 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.810+0000 7f041f2ff700 10 monclient: build_initial_monmap 2021-05-25T11:59:41.840 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.810+0000 7f041f2ff700 10 monclient: monmap: 2021-05-25T11:59:41.840 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 0 2021-05-25T11:59:41.840 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T11:59:41.840 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 0.000000 2021-05-25T11:59:41.841 INFO:teuthology.orchestra.run.smithi181.stderr:created 0.000000 2021-05-25T11:59:41.841 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 0 (unknown) 2021-05-25T11:59:41.841 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 1 2021-05-25T11:59:41.841 INFO:teuthology.orchestra.run.smithi181.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T11:59:41.842 INFO:teuthology.orchestra.run.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T11:59:41.842 INFO:teuthology.orchestra.run.smithi181.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T11:59:41.842 INFO:teuthology.orchestra.run.smithi181.stderr: 2021-05-25T11:59:41.842 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.810+0000 7f041f2ff700 1 librados: starting msgr at 2021-05-25T11:59:41.842 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.810+0000 7f041f2ff700 1 librados: starting objecter 2021-05-25T11:59:41.843 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.810+0000 7f041f2ff700 1 Processor -- start 2021-05-25T11:59:41.843 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.810+0000 7f041f2ff700 1 -- start start 2021-05-25T11:59:41.843 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.810+0000 7f041f2ff700 1 librados: setting wanted keys 2021-05-25T11:59:41.843 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.810+0000 7f041f2ff700 1 librados: calling monclient init 2021-05-25T11:59:41.843 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.810+0000 7f041f2ff700 10 monclient: init 2021-05-25T11:59:41.844 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.811+0000 7f041f2ff700 10 monclient: _reopen_session rank -1 2021-05-25T11:59:41.844 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.811+0000 7f041f2ff700 10 monclient: _add_conns ranks=[0,2,1] 2021-05-25T11:59:41.844 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.811+0000 7f041f2ff700 10 monclient(hunting): picked mon.noname-a con 0x7f0418096b00 addr v1:172.21.15.102:6789/0 2021-05-25T11:59:41.844 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.811+0000 7f041f2ff700 10 monclient(hunting): picked mon.noname-c con 0x7f0418096ee0 addr v1:172.21.15.102:6790/0 2021-05-25T11:59:41.844 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.811+0000 7f041f2ff700 10 monclient(hunting): picked mon.noname-b con 0x7f041812e390 addr v1:172.21.15.181:6789/0 2021-05-25T11:59:41.845 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.811+0000 7f041f2ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04180922e0 con 0x7f0418096b00 2021-05-25T11:59:41.845 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.811+0000 7f041f2ff700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0418119a10 con 0x7f041812e390 2021-05-25T11:59:41.845 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.811+0000 7f041f2ff700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0418119cb0 con 0x7f0418096ee0 2021-05-25T11:59:41.845 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.811+0000 7f041f2ff700 10 monclient(hunting): _renew_subs 2021-05-25T11:59:41.845 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.811+0000 7f041f2ff700 10 monclient(hunting): authenticate will time out at 781.295357s 2021-05-25T11:59:41.846 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.811+0000 7f041d89c700 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f041812e390 0x7f0418140f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:34552/0 (socket says 172.21.15.181:34552) 2021-05-25T11:59:41.846 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.811+0000 7f041d89c700 1 -- 172.21.15.181:0/136456739 learned_addr learned my addr 172.21.15.181:0/136456739 (peer_addr_for_me v1:172.21.15.181:0/0) 2021-05-25T11:59:41.846 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.811+0000 7f04057fa700 10 client.?.objecter ms_handle_connect 0x7f041812e390 2021-05-25T11:59:41.846 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.811+0000 7f04057fa700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:41.847 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.811+0000 7f04057fa700 10 client.?.objecter ms_handle_connect 0x7f0418096ee0 2021-05-25T11:59:41.847 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.811+0000 7f04057fa700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:41.847 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.811+0000 7f04057fa700 1 -- 172.21.15.181:0/136456739 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2974366763 0 0) 0x7f0418119a10 con 0x7f041812e390 2021-05-25T11:59:41.847 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.811+0000 7f04057fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:41.847 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.811+0000 7f04057fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:41.848 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.811+0000 7f04057fa700 10 monclient(hunting): my global_id is 4114 2021-05-25T11:59:41.848 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.811+0000 7f04057fa700 1 -- 172.21.15.181:0/136456739 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03f8004c10 con 0x7f041812e390 2021-05-25T11:59:41.848 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.811+0000 7f04057fa700 10 client.?.objecter ms_handle_connect 0x7f0418096b00 2021-05-25T11:59:41.848 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.811+0000 7f04057fa700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:41.849 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.811+0000 7f04057fa700 1 -- 172.21.15.181:0/136456739 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3966925136 0 0) 0x7f0418119cb0 con 0x7f0418096ee0 2021-05-25T11:59:41.849 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.811+0000 7f04057fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:41.849 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.811+0000 7f04057fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:41.849 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.811+0000 7f04057fa700 10 monclient(hunting): my global_id is 4112 2021-05-25T11:59:41.849 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.811+0000 7f04057fa700 1 -- 172.21.15.181:0/136456739 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0418119a10 con 0x7f0418096ee0 2021-05-25T11:59:41.850 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.811+0000 7f04057fa700 1 -- 172.21.15.181:0/136456739 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1417995120 0 0) 0x7f04180922e0 con 0x7f0418096b00 2021-05-25T11:59:41.850 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.811+0000 7f04057fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:41.850 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.811+0000 7f04057fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:41.850 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.811+0000 7f04057fa700 10 monclient(hunting): my global_id is 4113 2021-05-25T11:59:41.851 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.811+0000 7f04057fa700 1 -- 172.21.15.181:0/136456739 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0418119cb0 con 0x7f0418096b00 2021-05-25T11:59:41.851 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.811+0000 7f04057fa700 1 -- 172.21.15.181:0/136456739 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2610005659 0 0) 0x7f03f8004c10 con 0x7f041812e390 2021-05-25T11:59:41.851 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.811+0000 7f04057fa700 1 -- 172.21.15.181:0/136456739 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f04180922e0 con 0x7f041812e390 2021-05-25T11:59:41.851 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.811+0000 7f04057fa700 1 -- 172.21.15.181:0/136456739 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f0414001950 con 0x7f041812e390 2021-05-25T11:59:41.851 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.811+0000 7f04057fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:41.852 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.811+0000 7f04057fa700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T11:59:41.852 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.811+0000 7f04057fa700 10 monclient(hunting): dump: 2021-05-25T11:59:41.852 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 1 2021-05-25T11:59:41.852 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:41.852 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:41.853 INFO:teuthology.orchestra.run.smithi181.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:41.853 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:41.853 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 3 2021-05-25T11:59:41.853 INFO:teuthology.orchestra.run.smithi181.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:41.853 INFO:teuthology.orchestra.run.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:41.853 INFO:teuthology.orchestra.run.smithi181.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:41.854 INFO:teuthology.orchestra.run.smithi181.stderr: 2021-05-25T11:59:41.854 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.811+0000 7f04057fa700 10 monclient(hunting): _finish_auth 0 2021-05-25T11:59:41.854 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.812+0000 7f04057fa700 1 -- 172.21.15.181:0/136456739 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1518795293 0 0) 0x7f0418119a10 con 0x7f0418096ee0 2021-05-25T11:59:41.854 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.812+0000 7f04057fa700 1 -- 172.21.15.181:0/136456739 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f03f8004c10 con 0x7f0418096ee0 2021-05-25T11:59:41.855 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.812+0000 7f04057fa700 1 -- 172.21.15.181:0/136456739 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f040c001870 con 0x7f0418096ee0 2021-05-25T11:59:41.855 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.812+0000 7f04057fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:41.855 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.812+0000 7f04057fa700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2021-05-25T11:59:41.855 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.812+0000 7f04057fa700 10 monclient(hunting): dump: 2021-05-25T11:59:41.855 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 1 2021-05-25T11:59:41.856 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:41.856 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:41.856 INFO:teuthology.orchestra.run.smithi181.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:41.856 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:41.856 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 3 2021-05-25T11:59:41.857 INFO:teuthology.orchestra.run.smithi181.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:41.857 INFO:teuthology.orchestra.run.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:41.857 INFO:teuthology.orchestra.run.smithi181.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:41.857 INFO:teuthology.orchestra.run.smithi181.stderr: 2021-05-25T11:59:41.857 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.812+0000 7f04057fa700 1 -- 172.21.15.181:0/136456739 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3146758911 0 0) 0x7f04180922e0 con 0x7f041812e390 2021-05-25T11:59:41.857 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.812+0000 7f04057fa700 1 -- 172.21.15.181:0/136456739 >> v1:172.21.15.102:6790/0 conn(0x7f0418096ee0 legacy=0x7f041812d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:41.858 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.812+0000 7f04057fa700 1 -- 172.21.15.181:0/136456739 >> v1:172.21.15.102:6789/0 conn(0x7f0418096b00 legacy=0x7f04181164e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:41.858 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.812+0000 7f04057fa700 10 monclient: _finish_hunting 0 2021-05-25T11:59:41.858 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.812+0000 7f04057fa700 1 monclient: found mon.b 2021-05-25T11:59:41.858 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.812+0000 7f04057fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T11:59:41.859 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.812+0000 7f04057fa700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T11:59:41.859 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.812+0000 7f04057fa700 1 -- 172.21.15.181:0/136456739 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f041812d990 con 0x7f041812e390 2021-05-25T11:59:41.859 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.812+0000 7f04057fa700 10 monclient: _finish_auth 0 2021-05-25T11:59:41.859 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.812+0000 7f04057fa700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T11:59:41.859 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.812+0000 7f041f2ff700 5 monclient: authenticate success, global_id 4114 2021-05-25T11:59:41.860 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.812+0000 7f041f2ff700 10 monclient: _renew_subs 2021-05-25T11:59:41.860 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.812+0000 7f041f2ff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T11:59:41.860 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.812+0000 7f041f2ff700 1 -- 172.21.15.181:0/136456739 --> v1:172.21.15.181:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04181187e0 con 0x7f041812e390 2021-05-25T11:59:41.860 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.812+0000 7f041f2ff700 10 client.4114.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T11:59:41.860 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.812+0000 7f041f2ff700 10 monclient: _renew_subs 2021-05-25T11:59:41.861 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.812+0000 7f041f2ff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T11:59:41.861 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.812+0000 7f041f2ff700 1 -- 172.21.15.181:0/136456739 --> v1:172.21.15.181:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f04181171b0 con 0x7f041812e390 2021-05-25T11:59:41.861 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.812+0000 7f041f2ff700 1 librados: init done 2021-05-25T11:59:41.861 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.812+0000 7f04057fa700 1 -- 172.21.15.181:0/136456739 <== mon.1 v1:172.21.15.181:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0414004b60 con 0x7f041812e390 2021-05-25T11:59:41.861 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.812+0000 7f04057fa700 10 monclient: handle_config config(0 keys) v1 2021-05-25T11:59:41.862 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.812+0000 7f04057fa700 1 -- 172.21.15.181:0/136456739 <== mon.1 v1:172.21.15.181:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f0414004e70 con 0x7f041812e390 2021-05-25T11:59:41.862 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.812+0000 7f04057fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:41.862 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.812+0000 7f04057fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T11:59:41.862 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.812+0000 7f04077fe700 4 set_mon_vals no callback set 2021-05-25T11:59:41.863 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.812+0000 7f04057fa700 10 monclient: dump: 2021-05-25T11:59:41.863 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 1 2021-05-25T11:59:41.863 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:41.863 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:41.863 INFO:teuthology.orchestra.run.smithi181.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:41.864 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:41.864 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 3 2021-05-25T11:59:41.864 INFO:teuthology.orchestra.run.smithi181.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:41.864 INFO:teuthology.orchestra.run.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:41.865 INFO:teuthology.orchestra.run.smithi181.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:41.865 INFO:teuthology.orchestra.run.smithi181.stderr: 2021-05-25T11:59:41.865 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.812+0000 7f04057fa700 1 -- 172.21.15.181:0/136456739 <== mon.1 v1:172.21.15.181:6789/0 7 ==== mgrmap(e 1) v1 ==== 623+0+0 (unknown 1065898531 0 0) 0x7f04140052b0 con 0x7f041812e390 2021-05-25T11:59:41.865 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.812+0000 7f04057fa700 1 -- 172.21.15.181:0/136456739 <== mon.1 v1:172.21.15.181:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 754+0+0 (unknown 171113607 0 0) 0x7f0414006460 con 0x7f041812e390 2021-05-25T11:59:41.865 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.812+0000 7f04057fa700 10 client.4114.objecter ms_dispatch 0x7f0418001550 osd_map(2..2 src has 1..2) v4 2021-05-25T11:59:41.866 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.812+0000 7f04057fa700 3 client.4114.objecter handle_osd_map got epochs [2,2] > 0 2021-05-25T11:59:41.866 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.812+0000 7f04057fa700 3 client.4114.objecter handle_osd_map decoding full epoch 2 2021-05-25T11:59:41.866 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.812+0000 7f04057fa700 20 client.4114.objecter dump_active .. 0 homeless 2021-05-25T11:59:41.866 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.813+0000 7f041f2ff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:41.867 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.813+0000 7f041f2ff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:41.867 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.813+0000 7f041f2ff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T11:59:41.867 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.813+0000 7f041f2ff700 1 -- 172.21.15.181:0/136456739 --> v1:172.21.15.181:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03f0004db0 con 0x7f041812e390 2021-05-25T11:59:41.867 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.815+0000 7f04057fa700 1 -- 172.21.15.181:0/136456739 <== mon.1 v1:172.21.15.181:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72237 (unknown 1092875540 0 1415944542) 0x7f0414005720 con 0x7f041812e390 2021-05-25T11:59:41.867 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.815+0000 7f04057fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:41.868 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.815+0000 7f04057fa700 10 monclient: _finish_command 1 = system:0 2021-05-25T11:59:41.896 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.894+0000 7f041f2ff700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "8f1f285f-7041-4c53-ab09-f638ab670036", "id": 0}] 2021-05-25T11:59:41.897 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.894+0000 7f041f2ff700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "8f1f285f-7041-4c53-ab09-f638ab670036", "id": 0}] 2021-05-25T11:59:41.897 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.894+0000 7f041f2ff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T11:59:41.897 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.894+0000 7f041f2ff700 1 -- 172.21.15.181:0/136456739 --> v1:172.21.15.181:6789/0 -- mon_command({"prefix": "osd new", "uuid": "8f1f285f-7041-4c53-ab09-f638ab670036", "id": 0} v 0) v1 -- 0x7f03f00055e0 con 0x7f041812e390 2021-05-25T11:59:41.902 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.900+0000 7f04057fa700 1 -- 172.21.15.181:0/136456739 <== mon.1 v1:172.21.15.181:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8f1f285f-7041-4c53-ab09-f638ab670036", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 2365491003 0 732707515) 0x7f0414005a20 con 0x7f041812e390 2021-05-25T11:59:41.902 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.900+0000 7f04057fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "8f1f285f-7041-4c53-ab09-f638ab670036", "id": 0}] 2021-05-25T11:59:41.902 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.900+0000 7f04057fa700 10 monclient: _finish_command 2 = system:0 2021-05-25T11:59:41.903 INFO:teuthology.orchestra.run.smithi181.stdout:0 2021-05-25T11:59:41.904 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.901+0000 7f041f2ff700 10 librados: watch_flush enter 2021-05-25T11:59:41.904 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.901+0000 7f041f2ff700 10 librados: watch_flush exit 2021-05-25T11:59:41.905 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.901+0000 7f041f2ff700 20 client.4114.objecter shutdown clearing up homeless session... 2021-05-25T11:59:41.905 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.901+0000 7f041f2ff700 10 client.4114.objecter successfully canceled tick 2021-05-25T11:59:41.905 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.901+0000 7f041f2ff700 10 monclient: shutdown 2021-05-25T11:59:41.905 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.901+0000 7f041f2ff700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T11:59:41.906 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.901+0000 7f041f2ff700 1 -- 172.21.15.181:0/136456739 >> v1:172.21.15.181:6789/0 conn(0x7f041812e390 legacy=0x7f0418140f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:41.906 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.901+0000 7f041f2ff700 1 -- 172.21.15.181:0/136456739 shutdown_connections 2021-05-25T11:59:41.906 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.901+0000 7f041f2ff700 1 -- 172.21.15.181:0/136456739 >> 172.21.15.181:0/136456739 conn(0x7f0418061f00 msgr2=0x7f0418067710 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T11:59:41.906 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.901+0000 7f041f2ff700 1 -- 172.21.15.181:0/136456739 shutdown_connections 2021-05-25T11:59:41.906 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.901+0000 7f041f2ff700 1 -- 172.21.15.181:0/136456739 wait complete. 2021-05-25T11:59:41.907 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:41.901+0000 7f041f2ff700 1 librados: shutdown 2021-05-25T11:59:41.915 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster ceph osd new d0d92491-3d97-4e4e-acf4-9dc7c0e4701f 1 2021-05-25T11:59:42.026 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.023+0000 7f65993de700 10 monclient: get_monmap_and_config 2021-05-25T11:59:42.026 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.023+0000 7f65993de700 10 monclient: build_initial_monmap 2021-05-25T11:59:42.026 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.023+0000 7f65993de700 10 monclient: monmap: 2021-05-25T11:59:42.026 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 0 2021-05-25T11:59:42.027 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:42.027 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 2021-05-25T11:59:42.025455+0000 2021-05-25T11:59:42.027 INFO:teuthology.orchestra.run.smithi181.stderr:created 2021-05-25T11:59:42.025455+0000 2021-05-25T11:59:42.027 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 0 (unknown) 2021-05-25T11:59:42.027 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 3 2021-05-25T11:59:42.028 INFO:teuthology.orchestra.run.smithi181.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T11:59:42.028 INFO:teuthology.orchestra.run.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T11:59:42.028 INFO:teuthology.orchestra.run.smithi181.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T11:59:42.028 INFO:teuthology.orchestra.run.smithi181.stderr: 2021-05-25T11:59:42.039 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.024+0000 7f65993de700 1 Processor -- start 2021-05-25T11:59:42.039 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.024+0000 7f65993de700 1 -- start start 2021-05-25T11:59:42.039 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.024+0000 7f65993de700 10 monclient: init 2021-05-25T11:59:42.040 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.024+0000 7f65993de700 10 monclient: _reopen_session rank -1 2021-05-25T11:59:42.040 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.024+0000 7f65993de700 10 monclient: _add_conns ranks=[1,2,0] 2021-05-25T11:59:42.040 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.024+0000 7f65993de700 10 monclient(hunting): picked mon.noname-c con 0x7f6594063ad0 addr v1:172.21.15.181:6789/0 2021-05-25T11:59:42.040 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.024+0000 7f65993de700 10 monclient(hunting): picked mon.noname-b con 0x7f6594128660 addr v1:172.21.15.102:6790/0 2021-05-25T11:59:42.041 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.024+0000 7f65993de700 10 monclient(hunting): picked mon.noname-a con 0x7f6594128b10 addr v1:172.21.15.102:6789/0 2021-05-25T11:59:42.041 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.024+0000 7f65993de700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f659409ae40 con 0x7f6594128b10 2021-05-25T11:59:42.041 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.024+0000 7f65993de700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f659409b0e0 con 0x7f6594063ad0 2021-05-25T11:59:42.041 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.024+0000 7f65993de700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6594090aa0 con 0x7f6594128660 2021-05-25T11:59:42.041 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.024+0000 7f65993de700 10 monclient(hunting): _renew_subs 2021-05-25T11:59:42.041 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.024+0000 7f65993de700 10 monclient(hunting): authenticate will time out at 781.508964s 2021-05-25T11:59:42.042 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.025+0000 7f6593fff700 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f6594063ad0 0x7f6594097fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:34554/0 (socket says 172.21.15.181:34554) 2021-05-25T11:59:42.042 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.025+0000 7f6593fff700 1 -- 172.21.15.181:0/404321641 learned_addr learned my addr 172.21.15.181:0/404321641 (peer_addr_for_me v1:172.21.15.181:0/0) 2021-05-25T11:59:42.042 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.025+0000 7f6592ffd700 1 -- 172.21.15.181:0/404321641 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2154253403 0 0) 0x7f659409b0e0 con 0x7f6594063ad0 2021-05-25T11:59:42.042 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.025+0000 7f6592ffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:42.043 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.025+0000 7f6592ffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:42.043 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.025+0000 7f6592ffd700 10 monclient(hunting): my global_id is 4117 2021-05-25T11:59:42.043 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.025+0000 7f6592ffd700 1 -- 172.21.15.181:0/404321641 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6578003270 con 0x7f6594063ad0 2021-05-25T11:59:42.043 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.025+0000 7f6592ffd700 1 -- 172.21.15.181:0/404321641 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2582729506 0 0) 0x7f6578003270 con 0x7f6594063ad0 2021-05-25T11:59:42.043 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.025+0000 7f6592ffd700 1 -- 172.21.15.181:0/404321641 >> v1:172.21.15.102:6790/0 conn(0x7f6594128660 legacy=0x7f6594130ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:42.044 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.025+0000 7f6592ffd700 1 -- 172.21.15.181:0/404321641 >> v1:172.21.15.102:6789/0 conn(0x7f6594128b10 legacy=0x7f65941341f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:42.044 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.025+0000 7f6592ffd700 10 monclient: _finish_hunting 0 2021-05-25T11:59:42.044 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.025+0000 7f6592ffd700 1 monclient: found mon.noname-c 2021-05-25T11:59:42.044 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.025+0000 7f6592ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T11:59:42.044 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.025+0000 7f6592ffd700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.181:6789/0 2021-05-25T11:59:42.045 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.025+0000 7f6592ffd700 1 -- 172.21.15.181:0/404321641 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6594063eb0 con 0x7f6594063ad0 2021-05-25T11:59:42.045 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.025+0000 7f6592ffd700 10 monclient: _finish_auth 0 2021-05-25T11:59:42.045 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.025+0000 7f6592ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T11:59:42.045 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.025+0000 7f6592ffd700 1 -- 172.21.15.181:0/404321641 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f6584001610 con 0x7f6594063ad0 2021-05-25T11:59:42.046 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.025+0000 7f65993de700 5 monclient: authenticate success, global_id 4117 2021-05-25T11:59:42.046 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.025+0000 7f65993de700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T11:59:42.046 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.025+0000 7f6592ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:42.046 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.025+0000 7f6592ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T11:59:42.046 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.025+0000 7f6592ffd700 10 monclient: dump: 2021-05-25T11:59:42.047 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 1 2021-05-25T11:59:42.047 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:42.047 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:42.047 INFO:teuthology.orchestra.run.smithi181.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:42.047 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:42.048 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 3 2021-05-25T11:59:42.048 INFO:teuthology.orchestra.run.smithi181.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:42.048 INFO:teuthology.orchestra.run.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:42.048 INFO:teuthology.orchestra.run.smithi181.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:42.048 INFO:teuthology.orchestra.run.smithi181.stderr: 2021-05-25T11:59:42.048 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.025+0000 7f65993de700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T11:59:42.049 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.025+0000 7f6592ffd700 1 -- 172.21.15.181:0/404321641 <== mon.1 v1:172.21.15.181:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f65840018c0 con 0x7f6594063ad0 2021-05-25T11:59:42.049 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.025+0000 7f6592ffd700 10 monclient: handle_config config(0 keys) v1 2021-05-25T11:59:42.049 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.025+0000 7f65993de700 10 monclient: get_monmap_and_config success 2021-05-25T11:59:42.049 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.025+0000 7f65993de700 4 set_mon_vals no callback set 2021-05-25T11:59:42.049 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.025+0000 7f6592ffd700 1 -- 172.21.15.181:0/404321641 <== mon.1 v1:172.21.15.181:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f6584001c10 con 0x7f6594063ad0 2021-05-25T11:59:42.050 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.025+0000 7f6592ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:42.050 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.025+0000 7f6592ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T11:59:42.050 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.025+0000 7f6592ffd700 10 monclient: dump: 2021-05-25T11:59:42.050 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 1 2021-05-25T11:59:42.050 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:42.051 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:42.051 INFO:teuthology.orchestra.run.smithi181.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:42.051 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:42.051 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 3 2021-05-25T11:59:42.051 INFO:teuthology.orchestra.run.smithi181.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:42.052 INFO:teuthology.orchestra.run.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:42.052 INFO:teuthology.orchestra.run.smithi181.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:42.052 INFO:teuthology.orchestra.run.smithi181.stderr: 2021-05-25T11:59:42.052 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.026+0000 7f65993de700 10 monclient: shutdown 2021-05-25T11:59:42.052 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.026+0000 7f65993de700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T11:59:42.053 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.026+0000 7f65993de700 1 -- 172.21.15.181:0/404321641 >> v1:172.21.15.181:6789/0 conn(0x7f6594063ad0 legacy=0x7f6594097fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:42.053 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.026+0000 7f65993de700 1 -- 172.21.15.181:0/404321641 shutdown_connections 2021-05-25T11:59:42.053 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.026+0000 7f65993de700 1 -- 172.21.15.181:0/404321641 >> 172.21.15.181:0/404321641 conn(0x7f659409eab0 msgr2=0x7f659409eec0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T11:59:42.053 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.026+0000 7f65993de700 1 -- 172.21.15.181:0/404321641 shutdown_connections 2021-05-25T11:59:42.053 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.026+0000 7f65993de700 1 -- 172.21.15.181:0/404321641 wait complete. 2021-05-25T11:59:42.053 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.026+0000 7f65993de700 10 monclient: build_initial_monmap 2021-05-25T11:59:42.054 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.026+0000 7f65993de700 10 monclient: monmap: 2021-05-25T11:59:42.054 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 0 2021-05-25T11:59:42.054 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T11:59:42.054 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 0.000000 2021-05-25T11:59:42.054 INFO:teuthology.orchestra.run.smithi181.stderr:created 0.000000 2021-05-25T11:59:42.055 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 0 (unknown) 2021-05-25T11:59:42.055 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 1 2021-05-25T11:59:42.055 INFO:teuthology.orchestra.run.smithi181.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T11:59:42.055 INFO:teuthology.orchestra.run.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T11:59:42.056 INFO:teuthology.orchestra.run.smithi181.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T11:59:42.056 INFO:teuthology.orchestra.run.smithi181.stderr: 2021-05-25T11:59:42.057 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.026+0000 7f65993de700 1 librados: starting msgr at 2021-05-25T11:59:42.057 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.026+0000 7f65993de700 1 librados: starting objecter 2021-05-25T11:59:42.057 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.027+0000 7f65993de700 1 Processor -- start 2021-05-25T11:59:42.058 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.027+0000 7f65993de700 1 -- start start 2021-05-25T11:59:42.058 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.027+0000 7f65993de700 1 librados: setting wanted keys 2021-05-25T11:59:42.058 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.027+0000 7f65993de700 1 librados: calling monclient init 2021-05-25T11:59:42.059 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.027+0000 7f65993de700 10 monclient: init 2021-05-25T11:59:42.059 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.027+0000 7f65993de700 10 monclient: _reopen_session rank -1 2021-05-25T11:59:42.060 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.027+0000 7f65993de700 10 monclient: _add_conns ranks=[2,0,1] 2021-05-25T11:59:42.060 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.027+0000 7f65993de700 10 monclient(hunting): picked mon.noname-c con 0x7f6594063ad0 addr v1:172.21.15.102:6790/0 2021-05-25T11:59:42.060 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.027+0000 7f65993de700 10 monclient(hunting): picked mon.noname-a con 0x7f6594128660 addr v1:172.21.15.102:6789/0 2021-05-25T11:59:42.061 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.027+0000 7f65993de700 10 monclient(hunting): picked mon.noname-b con 0x7f6594128b10 addr v1:172.21.15.181:6789/0 2021-05-25T11:59:42.061 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.027+0000 7f65993de700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6594116700 con 0x7f6594128660 2021-05-25T11:59:42.062 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.027+0000 7f65993de700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f659413c250 con 0x7f6594128b10 2021-05-25T11:59:42.062 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.027+0000 7f65993de700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65941184f0 con 0x7f6594063ad0 2021-05-25T11:59:42.062 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.027+0000 7f65993de700 10 monclient(hunting): _renew_subs 2021-05-25T11:59:42.063 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.027+0000 7f65993de700 10 monclient(hunting): authenticate will time out at 781.511763s 2021-05-25T11:59:42.063 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.027+0000 7f6598bdd700 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f6594128b10 0x7f659413bb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:34564/0 (socket says 172.21.15.181:34564) 2021-05-25T11:59:42.064 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.027+0000 7f6593fff700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f6594063ad0 0x7f6594118b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.181:53584/0 (socket says 172.21.15.181:53584) 2021-05-25T11:59:42.064 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.027+0000 7f6598bdd700 1 -- 172.21.15.181:0/2137520127 learned_addr learned my addr 172.21.15.181:0/2137520127 (peer_addr_for_me v1:172.21.15.181:0/0) 2021-05-25T11:59:42.064 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.027+0000 7f65937fe700 1 --1- 172.21.15.181:0/2137520127 >> v1:172.21.15.102:6789/0 conn(0x7f6594128660 0x7f659413b430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.181:34752/0 (socket says 172.21.15.181:34752) 2021-05-25T11:59:42.065 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.027+0000 7f657ffff700 10 client.?.objecter ms_handle_connect 0x7f6594128b10 2021-05-25T11:59:42.065 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.027+0000 7f657ffff700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:42.065 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.027+0000 7f657ffff700 10 client.?.objecter ms_handle_connect 0x7f6594128660 2021-05-25T11:59:42.066 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.027+0000 7f657ffff700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:42.066 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.027+0000 7f657ffff700 10 client.?.objecter ms_handle_connect 0x7f6594063ad0 2021-05-25T11:59:42.067 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.027+0000 7f657ffff700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:42.067 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.028+0000 7f657ffff700 1 -- 172.21.15.181:0/2137520127 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1512119828 0 0) 0x7f659413c250 con 0x7f6594128b10 2021-05-25T11:59:42.067 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.028+0000 7f657ffff700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:42.068 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.028+0000 7f657ffff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:42.068 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.028+0000 7f657ffff700 10 monclient(hunting): my global_id is 4120 2021-05-25T11:59:42.068 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.028+0000 7f657ffff700 1 -- 172.21.15.181:0/2137520127 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6574004c10 con 0x7f6594128b10 2021-05-25T11:59:42.069 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.028+0000 7f657ffff700 1 -- 172.21.15.181:0/2137520127 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1920321989 0 0) 0x7f6594116700 con 0x7f6594128660 2021-05-25T11:59:42.069 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.028+0000 7f657ffff700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:42.070 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.028+0000 7f657ffff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:42.070 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.028+0000 7f657ffff700 10 monclient(hunting): my global_id is 4119 2021-05-25T11:59:42.070 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.028+0000 7f657ffff700 1 -- 172.21.15.181:0/2137520127 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f659413c250 con 0x7f6594128660 2021-05-25T11:59:42.071 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.028+0000 7f657ffff700 1 -- 172.21.15.181:0/2137520127 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1974633097 0 0) 0x7f65941184f0 con 0x7f6594063ad0 2021-05-25T11:59:42.071 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.028+0000 7f657ffff700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:42.072 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.028+0000 7f657ffff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:42.072 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.028+0000 7f657ffff700 10 monclient(hunting): my global_id is 4118 2021-05-25T11:59:42.072 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.028+0000 7f657ffff700 1 -- 172.21.15.181:0/2137520127 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6594116700 con 0x7f6594063ad0 2021-05-25T11:59:42.073 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.028+0000 7f657ffff700 1 -- 172.21.15.181:0/2137520127 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1638600747 0 0) 0x7f6574004c10 con 0x7f6594128b10 2021-05-25T11:59:42.073 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.028+0000 7f657ffff700 1 -- 172.21.15.181:0/2137520127 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f65941184f0 con 0x7f6594128b10 2021-05-25T11:59:42.074 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.028+0000 7f657ffff700 1 -- 172.21.15.181:0/2137520127 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2909415149 0 0) 0x7f659413c250 con 0x7f6594128660 2021-05-25T11:59:42.074 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.028+0000 7f657ffff700 1 -- 172.21.15.181:0/2137520127 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6574004c10 con 0x7f6594128660 2021-05-25T11:59:42.074 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.028+0000 7f657ffff700 1 -- 172.21.15.181:0/2137520127 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f6588001c80 con 0x7f6594128b10 2021-05-25T11:59:42.075 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.028+0000 7f657ffff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:42.075 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.028+0000 7f657ffff700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T11:59:42.076 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.028+0000 7f657ffff700 10 monclient(hunting): dump: 2021-05-25T11:59:42.076 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 1 2021-05-25T11:59:42.076 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:42.077 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:42.077 INFO:teuthology.orchestra.run.smithi181.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:42.077 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:42.078 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 3 2021-05-25T11:59:42.078 INFO:teuthology.orchestra.run.smithi181.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:42.079 INFO:teuthology.orchestra.run.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:42.079 INFO:teuthology.orchestra.run.smithi181.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:42.079 INFO:teuthology.orchestra.run.smithi181.stderr: 2021-05-25T11:59:42.080 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.028+0000 7f657ffff700 10 monclient(hunting): _finish_auth 0 2021-05-25T11:59:42.080 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.028+0000 7f657ffff700 1 -- 172.21.15.181:0/2137520127 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 243266200 0 0) 0x7f6594116700 con 0x7f6594063ad0 2021-05-25T11:59:42.081 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.028+0000 7f657ffff700 1 -- 172.21.15.181:0/2137520127 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f659413c250 con 0x7f6594063ad0 2021-05-25T11:59:42.081 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.028+0000 7f657ffff700 1 -- 172.21.15.181:0/2137520127 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f6580001840 con 0x7f6594128660 2021-05-25T11:59:42.081 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.028+0000 7f657ffff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:42.082 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.028+0000 7f657ffff700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2021-05-25T11:59:42.082 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.028+0000 7f657ffff700 10 monclient(hunting): dump: 2021-05-25T11:59:42.082 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 1 2021-05-25T11:59:42.083 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:42.083 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:42.084 INFO:teuthology.orchestra.run.smithi181.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:42.084 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:42.084 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 3 2021-05-25T11:59:42.084 INFO:teuthology.orchestra.run.smithi181.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:42.085 INFO:teuthology.orchestra.run.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:42.085 INFO:teuthology.orchestra.run.smithi181.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:42.085 INFO:teuthology.orchestra.run.smithi181.stderr: 2021-05-25T11:59:42.086 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.028+0000 7f657ffff700 1 -- 172.21.15.181:0/2137520127 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f6584001430 con 0x7f6594063ad0 2021-05-25T11:59:42.086 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.028+0000 7f657ffff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:42.087 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.028+0000 7f657ffff700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2021-05-25T11:59:42.087 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.028+0000 7f657ffff700 10 monclient(hunting): dump: 2021-05-25T11:59:42.087 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 1 2021-05-25T11:59:42.087 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:42.087 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:42.088 INFO:teuthology.orchestra.run.smithi181.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:42.088 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:42.088 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 3 2021-05-25T11:59:42.088 INFO:teuthology.orchestra.run.smithi181.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:42.088 INFO:teuthology.orchestra.run.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:42.089 INFO:teuthology.orchestra.run.smithi181.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:42.089 INFO:teuthology.orchestra.run.smithi181.stderr: 2021-05-25T11:59:42.089 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.028+0000 7f657ffff700 1 -- 172.21.15.181:0/2137520127 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2294662153 0 0) 0x7f65941184f0 con 0x7f6594128b10 2021-05-25T11:59:42.089 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.028+0000 7f657ffff700 1 -- 172.21.15.181:0/2137520127 >> v1:172.21.15.102:6790/0 conn(0x7f6594063ad0 legacy=0x7f6594118b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:42.090 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.028+0000 7f657ffff700 1 -- 172.21.15.181:0/2137520127 >> v1:172.21.15.102:6789/0 conn(0x7f6594128660 legacy=0x7f659413b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:42.090 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.028+0000 7f657ffff700 10 monclient: _finish_hunting 0 2021-05-25T11:59:42.090 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.028+0000 7f657ffff700 1 monclient: found mon.b 2021-05-25T11:59:42.090 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.028+0000 7f657ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T11:59:42.090 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.028+0000 7f657ffff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T11:59:42.091 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.028+0000 7f657ffff700 1 -- 172.21.15.181:0/2137520127 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6594118790 con 0x7f6594128b10 2021-05-25T11:59:42.091 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.028+0000 7f657ffff700 10 monclient: _finish_auth 0 2021-05-25T11:59:42.091 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.028+0000 7f657ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T11:59:42.091 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.028+0000 7f65993de700 5 monclient: authenticate success, global_id 4120 2021-05-25T11:59:42.091 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.028+0000 7f65993de700 10 monclient: _renew_subs 2021-05-25T11:59:42.092 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.028+0000 7f65993de700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T11:59:42.092 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.028+0000 7f65993de700 1 -- 172.21.15.181:0/2137520127 --> v1:172.21.15.181:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6594117c90 con 0x7f6594128b10 2021-05-25T11:59:42.092 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.028+0000 7f65993de700 10 client.4120.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T11:59:42.092 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.028+0000 7f65993de700 10 monclient: _renew_subs 2021-05-25T11:59:42.093 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.028+0000 7f65993de700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T11:59:42.093 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.028+0000 7f65993de700 1 -- 172.21.15.181:0/2137520127 --> v1:172.21.15.181:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f65941446a0 con 0x7f6594128b10 2021-05-25T11:59:42.093 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.028+0000 7f65993de700 1 librados: init done 2021-05-25T11:59:42.093 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.028+0000 7f657ffff700 1 -- 172.21.15.181:0/2137520127 <== mon.1 v1:172.21.15.181:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6588001fc0 con 0x7f6594128b10 2021-05-25T11:59:42.093 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.028+0000 7f657ffff700 10 monclient: handle_config config(0 keys) v1 2021-05-25T11:59:42.093 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.028+0000 7f657ffff700 1 -- 172.21.15.181:0/2137520127 <== mon.1 v1:172.21.15.181:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f65880025b0 con 0x7f6594128b10 2021-05-25T11:59:42.094 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.028+0000 7f657ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:42.094 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.028+0000 7f65927fc700 4 set_mon_vals no callback set 2021-05-25T11:59:42.094 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.028+0000 7f657ffff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T11:59:42.094 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.028+0000 7f657ffff700 10 monclient: dump: 2021-05-25T11:59:42.095 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 1 2021-05-25T11:59:42.095 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:42.095 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:42.095 INFO:teuthology.orchestra.run.smithi181.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:42.095 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:42.096 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 3 2021-05-25T11:59:42.096 INFO:teuthology.orchestra.run.smithi181.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:42.096 INFO:teuthology.orchestra.run.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:42.096 INFO:teuthology.orchestra.run.smithi181.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:42.096 INFO:teuthology.orchestra.run.smithi181.stderr: 2021-05-25T11:59:42.096 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.028+0000 7f657ffff700 1 -- 172.21.15.181:0/2137520127 <== mon.1 v1:172.21.15.181:6789/0 7 ==== mgrmap(e 1) v1 ==== 623+0+0 (unknown 1065898531 0 0) 0x7f6588002a70 con 0x7f6594128b10 2021-05-25T11:59:42.097 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.028+0000 7f657ffff700 1 -- 172.21.15.181:0/2137520127 <== mon.1 v1:172.21.15.181:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 869+0+0 (unknown 419303084 0 0) 0x7f6588006250 con 0x7f6594128b10 2021-05-25T11:59:42.097 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.028+0000 7f657ffff700 10 client.4120.objecter ms_dispatch 0x7f6594001550 osd_map(3..3 src has 1..3) v4 2021-05-25T11:59:42.097 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.028+0000 7f657ffff700 3 client.4120.objecter handle_osd_map got epochs [3,3] > 0 2021-05-25T11:59:42.097 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.028+0000 7f657ffff700 3 client.4120.objecter handle_osd_map decoding full epoch 3 2021-05-25T11:59:42.097 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.028+0000 7f657ffff700 20 client.4120.objecter dump_active .. 0 homeless 2021-05-25T11:59:42.098 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.029+0000 7f65993de700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:42.098 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.029+0000 7f65993de700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:42.098 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.029+0000 7f65993de700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T11:59:42.098 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.029+0000 7f65993de700 1 -- 172.21.15.181:0/2137520127 --> v1:172.21.15.181:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f656c004db0 con 0x7f6594128b10 2021-05-25T11:59:42.098 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.031+0000 7f657ffff700 1 -- 172.21.15.181:0/2137520127 <== mon.1 v1:172.21.15.181:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72237 (unknown 1092875540 0 1415944542) 0x7f6588002f60 con 0x7f6594128b10 2021-05-25T11:59:42.099 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.031+0000 7f657ffff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:42.099 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.031+0000 7f657ffff700 10 monclient: _finish_command 1 = system:0 2021-05-25T11:59:42.111 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.108+0000 7f65993de700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "d0d92491-3d97-4e4e-acf4-9dc7c0e4701f", "id": 1}] 2021-05-25T11:59:42.111 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.108+0000 7f65993de700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "d0d92491-3d97-4e4e-acf4-9dc7c0e4701f", "id": 1}] 2021-05-25T11:59:42.111 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.108+0000 7f65993de700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T11:59:42.111 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.108+0000 7f65993de700 1 -- 172.21.15.181:0/2137520127 --> v1:172.21.15.181:6789/0 -- mon_command({"prefix": "osd new", "uuid": "d0d92491-3d97-4e4e-acf4-9dc7c0e4701f", "id": 1} v 0) v1 -- 0x7f656c0055e0 con 0x7f6594128b10 2021-05-25T11:59:42.118 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.116+0000 7f657ffff700 1 -- 172.21.15.181:0/2137520127 <== mon.1 v1:172.21.15.181:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d0d92491-3d97-4e4e-acf4-9dc7c0e4701f", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 319252186 0 940483276) 0x7f6588006530 con 0x7f6594128b10 2021-05-25T11:59:42.119 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.116+0000 7f657ffff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "d0d92491-3d97-4e4e-acf4-9dc7c0e4701f", "id": 1}] 2021-05-25T11:59:42.119 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.116+0000 7f657ffff700 10 monclient: _finish_command 2 = system:0 2021-05-25T11:59:42.119 INFO:teuthology.orchestra.run.smithi181.stdout:1 2021-05-25T11:59:42.121 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.117+0000 7f65993de700 10 librados: watch_flush enter 2021-05-25T11:59:42.121 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.117+0000 7f65993de700 10 librados: watch_flush exit 2021-05-25T11:59:42.121 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.117+0000 7f65993de700 20 client.4120.objecter shutdown clearing up homeless session... 2021-05-25T11:59:42.121 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.117+0000 7f65993de700 10 client.4120.objecter successfully canceled tick 2021-05-25T11:59:42.122 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.117+0000 7f65993de700 10 monclient: shutdown 2021-05-25T11:59:42.122 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.117+0000 7f65993de700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T11:59:42.122 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.117+0000 7f65993de700 1 -- 172.21.15.181:0/2137520127 >> v1:172.21.15.181:6789/0 conn(0x7f6594128b10 legacy=0x7f659413bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:42.122 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.117+0000 7f65993de700 1 -- 172.21.15.181:0/2137520127 shutdown_connections 2021-05-25T11:59:42.122 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.117+0000 7f65993de700 1 -- 172.21.15.181:0/2137520127 >> 172.21.15.181:0/2137520127 conn(0x7f659409eab0 msgr2=0x7f659412a870 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T11:59:42.123 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.117+0000 7f65993de700 1 -- 172.21.15.181:0/2137520127 shutdown_connections 2021-05-25T11:59:42.123 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.117+0000 7f65993de700 1 -- 172.21.15.181:0/2137520127 wait complete. 2021-05-25T11:59:42.123 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.117+0000 7f65993de700 1 librados: shutdown 2021-05-25T11:59:42.130 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster ceph osd new 80a3eb90-b36e-4035-af91-3917fbf8dc2a 2 2021-05-25T11:59:42.241 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.239+0000 7fa564b5a700 10 monclient: get_monmap_and_config 2021-05-25T11:59:42.241 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.239+0000 7fa564b5a700 10 monclient: build_initial_monmap 2021-05-25T11:59:42.241 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.239+0000 7fa564b5a700 10 monclient: monmap: 2021-05-25T11:59:42.242 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 0 2021-05-25T11:59:42.242 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:42.242 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 2021-05-25T11:59:42.240901+0000 2021-05-25T11:59:42.242 INFO:teuthology.orchestra.run.smithi181.stderr:created 2021-05-25T11:59:42.240901+0000 2021-05-25T11:59:42.243 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 0 (unknown) 2021-05-25T11:59:42.243 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 3 2021-05-25T11:59:42.243 INFO:teuthology.orchestra.run.smithi181.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T11:59:42.243 INFO:teuthology.orchestra.run.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T11:59:42.243 INFO:teuthology.orchestra.run.smithi181.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T11:59:42.244 INFO:teuthology.orchestra.run.smithi181.stderr: 2021-05-25T11:59:42.255 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.239+0000 7fa564b5a700 1 Processor -- start 2021-05-25T11:59:42.255 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.239+0000 7fa564b5a700 1 -- start start 2021-05-25T11:59:42.255 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.239+0000 7fa564b5a700 10 monclient: init 2021-05-25T11:59:42.255 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.240+0000 7fa564b5a700 10 monclient: _reopen_session rank -1 2021-05-25T11:59:42.256 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.240+0000 7fa564b5a700 10 monclient: _add_conns ranks=[1,0,2] 2021-05-25T11:59:42.256 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.240+0000 7fa564b5a700 10 monclient(hunting): picked mon.noname-c con 0x7fa560064180 addr v1:172.21.15.181:6789/0 2021-05-25T11:59:42.256 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.240+0000 7fa564b5a700 10 monclient(hunting): picked mon.noname-a con 0x7fa560128980 addr v1:172.21.15.102:6789/0 2021-05-25T11:59:42.256 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.240+0000 7fa564b5a700 10 monclient(hunting): picked mon.noname-b con 0x7fa5601323e0 addr v1:172.21.15.102:6790/0 2021-05-25T11:59:42.257 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.240+0000 7fa564b5a700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5600988f0 con 0x7fa560128980 2021-05-25T11:59:42.257 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.240+0000 7fa564b5a700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa560064560 con 0x7fa560064180 2021-05-25T11:59:42.257 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.240+0000 7fa564b5a700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa560128e30 con 0x7fa5601323e0 2021-05-25T11:59:42.257 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.240+0000 7fa564b5a700 10 monclient(hunting): _renew_subs 2021-05-25T11:59:42.257 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.240+0000 7fa564b5a700 10 monclient(hunting): authenticate will time out at 781.724390s 2021-05-25T11:59:42.258 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.240+0000 7fa55e59c700 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7fa560064180 0x7fa5600981e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:34566/0 (socket says 172.21.15.181:34566) 2021-05-25T11:59:42.258 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.240+0000 7fa55e59c700 1 -- 172.21.15.181:0/3927838112 learned_addr learned my addr 172.21.15.181:0/3927838112 (peer_addr_for_me v1:172.21.15.181:0/0) 2021-05-25T11:59:42.258 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.240+0000 7fa55d59a700 1 -- 172.21.15.181:0/3927838112 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3125583143 0 0) 0x7fa560064560 con 0x7fa560064180 2021-05-25T11:59:42.258 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.240+0000 7fa55d59a700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:42.258 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.240+0000 7fa55d59a700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:42.259 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.240+0000 7fa55d59a700 10 monclient(hunting): my global_id is 4123 2021-05-25T11:59:42.259 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.240+0000 7fa55d59a700 1 -- 172.21.15.181:0/3927838112 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa544003270 con 0x7fa560064180 2021-05-25T11:59:42.259 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.240+0000 7fa55d59a700 1 -- 172.21.15.181:0/3927838112 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1891389911 0 0) 0x7fa560128e30 con 0x7fa5601323e0 2021-05-25T11:59:42.259 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.240+0000 7fa55d59a700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:42.260 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.240+0000 7fa55d59a700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:42.260 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.240+0000 7fa55d59a700 10 monclient(hunting): my global_id is 4121 2021-05-25T11:59:42.260 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.240+0000 7fa55d59a700 1 -- 172.21.15.181:0/3927838112 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa560064560 con 0x7fa5601323e0 2021-05-25T11:59:42.260 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.240+0000 7fa55d59a700 1 -- 172.21.15.181:0/3927838112 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 679935405 0 0) 0x7fa5600988f0 con 0x7fa560128980 2021-05-25T11:59:42.260 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.240+0000 7fa55d59a700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:42.261 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.240+0000 7fa55d59a700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:42.261 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.240+0000 7fa55d59a700 10 monclient(hunting): my global_id is 4122 2021-05-25T11:59:42.261 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.240+0000 7fa55d59a700 1 -- 172.21.15.181:0/3927838112 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa560128e30 con 0x7fa560128980 2021-05-25T11:59:42.261 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.241+0000 7fa55d59a700 1 -- 172.21.15.181:0/3927838112 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4213138921 0 0) 0x7fa544003270 con 0x7fa560064180 2021-05-25T11:59:42.261 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.241+0000 7fa55d59a700 1 -- 172.21.15.181:0/3927838112 >> v1:172.21.15.102:6790/0 conn(0x7fa5601323e0 legacy=0x7fa5601347d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:42.262 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.241+0000 7fa55d59a700 1 -- 172.21.15.181:0/3927838112 >> v1:172.21.15.102:6789/0 conn(0x7fa560128980 legacy=0x7fa560130cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:42.262 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.241+0000 7fa55d59a700 10 monclient: _finish_hunting 0 2021-05-25T11:59:42.262 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.241+0000 7fa55d59a700 1 monclient: found mon.noname-c 2021-05-25T11:59:42.262 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.241+0000 7fa55d59a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T11:59:42.263 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.241+0000 7fa55d59a700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.181:6789/0 2021-05-25T11:59:42.263 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.241+0000 7fa55d59a700 1 -- 172.21.15.181:0/3927838112 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa560098f90 con 0x7fa560064180 2021-05-25T11:59:42.263 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.241+0000 7fa55d59a700 10 monclient: _finish_auth 0 2021-05-25T11:59:42.263 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.241+0000 7fa55d59a700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T11:59:42.263 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.241+0000 7fa55d59a700 1 -- 172.21.15.181:0/3927838112 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fa548001610 con 0x7fa560064180 2021-05-25T11:59:42.264 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.241+0000 7fa564b5a700 5 monclient: authenticate success, global_id 4123 2021-05-25T11:59:42.264 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.241+0000 7fa564b5a700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T11:59:42.264 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.241+0000 7fa55d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:42.264 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.241+0000 7fa55d59a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T11:59:42.264 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.241+0000 7fa55d59a700 10 monclient: dump: 2021-05-25T11:59:42.265 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 1 2021-05-25T11:59:42.265 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:42.265 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:42.265 INFO:teuthology.orchestra.run.smithi181.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:42.266 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:42.266 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 3 2021-05-25T11:59:42.266 INFO:teuthology.orchestra.run.smithi181.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:42.266 INFO:teuthology.orchestra.run.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:42.266 INFO:teuthology.orchestra.run.smithi181.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:42.267 INFO:teuthology.orchestra.run.smithi181.stderr: 2021-05-25T11:59:42.267 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.241+0000 7fa564b5a700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T11:59:42.267 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.241+0000 7fa55d59a700 1 -- 172.21.15.181:0/3927838112 <== mon.1 v1:172.21.15.181:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa5480018f0 con 0x7fa560064180 2021-05-25T11:59:42.268 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.241+0000 7fa55d59a700 10 monclient: handle_config config(0 keys) v1 2021-05-25T11:59:42.268 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.241+0000 7fa564b5a700 10 monclient: get_monmap_and_config success 2021-05-25T11:59:42.268 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.241+0000 7fa55d59a700 1 -- 172.21.15.181:0/3927838112 <== mon.1 v1:172.21.15.181:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fa548001c80 con 0x7fa560064180 2021-05-25T11:59:42.269 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.241+0000 7fa55d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:42.269 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.241+0000 7fa564b5a700 4 set_mon_vals no callback set 2021-05-25T11:59:42.269 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.241+0000 7fa55d59a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T11:59:42.270 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.241+0000 7fa55d59a700 10 monclient: dump: 2021-05-25T11:59:42.270 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 1 2021-05-25T11:59:42.270 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:42.271 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:42.271 INFO:teuthology.orchestra.run.smithi181.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:42.271 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:42.272 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 3 2021-05-25T11:59:42.272 INFO:teuthology.orchestra.run.smithi181.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:42.272 INFO:teuthology.orchestra.run.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:42.273 INFO:teuthology.orchestra.run.smithi181.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:42.273 INFO:teuthology.orchestra.run.smithi181.stderr: 2021-05-25T11:59:42.273 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.241+0000 7fa564b5a700 10 monclient: shutdown 2021-05-25T11:59:42.274 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.241+0000 7fa564b5a700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T11:59:42.274 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.241+0000 7fa564b5a700 1 -- 172.21.15.181:0/3927838112 >> v1:172.21.15.181:6789/0 conn(0x7fa560064180 legacy=0x7fa5600981e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:42.274 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.241+0000 7fa564b5a700 1 -- 172.21.15.181:0/3927838112 shutdown_connections 2021-05-25T11:59:42.275 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.241+0000 7fa564b5a700 1 -- 172.21.15.181:0/3927838112 >> 172.21.15.181:0/3927838112 conn(0x7fa56009b800 msgr2=0x7fa56009bc10 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T11:59:42.275 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.241+0000 7fa564b5a700 1 -- 172.21.15.181:0/3927838112 shutdown_connections 2021-05-25T11:59:42.276 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.241+0000 7fa564b5a700 1 -- 172.21.15.181:0/3927838112 wait complete. 2021-05-25T11:59:42.276 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.242+0000 7fa564b5a700 10 monclient: build_initial_monmap 2021-05-25T11:59:42.276 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.242+0000 7fa564b5a700 10 monclient: monmap: 2021-05-25T11:59:42.276 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 0 2021-05-25T11:59:42.277 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T11:59:42.277 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 0.000000 2021-05-25T11:59:42.278 INFO:teuthology.orchestra.run.smithi181.stderr:created 0.000000 2021-05-25T11:59:42.278 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 0 (unknown) 2021-05-25T11:59:42.278 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 1 2021-05-25T11:59:42.279 INFO:teuthology.orchestra.run.smithi181.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T11:59:42.279 INFO:teuthology.orchestra.run.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T11:59:42.279 INFO:teuthology.orchestra.run.smithi181.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T11:59:42.279 INFO:teuthology.orchestra.run.smithi181.stderr: 2021-05-25T11:59:42.280 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.242+0000 7fa564b5a700 1 librados: starting msgr at 2021-05-25T11:59:42.280 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.242+0000 7fa564b5a700 1 librados: starting objecter 2021-05-25T11:59:42.280 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.242+0000 7fa564b5a700 1 Processor -- start 2021-05-25T11:59:42.280 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.242+0000 7fa564b5a700 1 -- start start 2021-05-25T11:59:42.280 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.242+0000 7fa564b5a700 1 librados: setting wanted keys 2021-05-25T11:59:42.281 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.242+0000 7fa564b5a700 1 librados: calling monclient init 2021-05-25T11:59:42.281 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.242+0000 7fa564b5a700 10 monclient: init 2021-05-25T11:59:42.281 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.243+0000 7fa564b5a700 10 monclient: _reopen_session rank -1 2021-05-25T11:59:42.281 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.243+0000 7fa564b5a700 10 monclient: _add_conns ranks=[0,1,2] 2021-05-25T11:59:42.282 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.243+0000 7fa564b5a700 10 monclient(hunting): picked mon.noname-a con 0x7fa560064180 addr v1:172.21.15.102:6789/0 2021-05-25T11:59:42.282 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.243+0000 7fa564b5a700 10 monclient(hunting): picked mon.noname-b con 0x7fa560128980 addr v1:172.21.15.181:6789/0 2021-05-25T11:59:42.282 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.243+0000 7fa564b5a700 10 monclient(hunting): picked mon.noname-c con 0x7fa5601323e0 addr v1:172.21.15.102:6790/0 2021-05-25T11:59:42.282 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.243+0000 7fa564b5a700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa56009a7b0 con 0x7fa560064180 2021-05-25T11:59:42.282 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.243+0000 7fa564b5a700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa560098710 con 0x7fa560128980 2021-05-25T11:59:42.283 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.243+0000 7fa564b5a700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa56012c9f0 con 0x7fa5601323e0 2021-05-25T11:59:42.283 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.243+0000 7fa564b5a700 10 monclient(hunting): _renew_subs 2021-05-25T11:59:42.283 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.243+0000 7fa564b5a700 10 monclient(hunting): authenticate will time out at 781.727206s 2021-05-25T11:59:42.283 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.243+0000 7fa55dd9b700 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7fa560128980 0x7fa560130ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:34576/0 (socket says 172.21.15.181:34576) 2021-05-25T11:59:42.283 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.243+0000 7fa55dd9b700 1 -- 172.21.15.181:0/1747216706 learned_addr learned my addr 172.21.15.181:0/1747216706 (peer_addr_for_me v1:172.21.15.181:0/0) 2021-05-25T11:59:42.284 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.243+0000 7fa54e7fc700 10 client.?.objecter ms_handle_connect 0x7fa560128980 2021-05-25T11:59:42.284 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.243+0000 7fa54e7fc700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:42.284 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.243+0000 7fa54e7fc700 10 client.?.objecter ms_handle_connect 0x7fa560064180 2021-05-25T11:59:42.284 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.243+0000 7fa54e7fc700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:42.284 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.243+0000 7fa54e7fc700 10 client.?.objecter ms_handle_connect 0x7fa5601323e0 2021-05-25T11:59:42.285 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.243+0000 7fa54e7fc700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:42.285 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.243+0000 7fa54e7fc700 1 -- 172.21.15.181:0/1747216706 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3324296961 0 0) 0x7fa560098710 con 0x7fa560128980 2021-05-25T11:59:42.285 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.243+0000 7fa54e7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:42.285 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.243+0000 7fa54e7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:42.286 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.243+0000 7fa54e7fc700 10 monclient(hunting): my global_id is 4126 2021-05-25T11:59:42.286 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.243+0000 7fa54e7fc700 1 -- 172.21.15.181:0/1747216706 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa53c004c10 con 0x7fa560128980 2021-05-25T11:59:42.286 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.243+0000 7fa54e7fc700 1 -- 172.21.15.181:0/1747216706 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1285914852 0 0) 0x7fa56009a7b0 con 0x7fa560064180 2021-05-25T11:59:42.286 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.243+0000 7fa54e7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:42.286 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.243+0000 7fa54e7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:42.287 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.243+0000 7fa54e7fc700 10 monclient(hunting): my global_id is 4125 2021-05-25T11:59:42.287 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.243+0000 7fa54e7fc700 1 -- 172.21.15.181:0/1747216706 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa560098710 con 0x7fa560064180 2021-05-25T11:59:42.287 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.243+0000 7fa54e7fc700 1 -- 172.21.15.181:0/1747216706 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 482239490 0 0) 0x7fa56012c9f0 con 0x7fa5601323e0 2021-05-25T11:59:42.287 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.243+0000 7fa54e7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:42.287 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.243+0000 7fa54e7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:42.288 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.243+0000 7fa54e7fc700 10 monclient(hunting): my global_id is 4124 2021-05-25T11:59:42.288 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.243+0000 7fa54e7fc700 1 -- 172.21.15.181:0/1747216706 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa56009a7b0 con 0x7fa5601323e0 2021-05-25T11:59:42.288 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.243+0000 7fa54e7fc700 1 -- 172.21.15.181:0/1747216706 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3904096553 0 0) 0x7fa56009a7b0 con 0x7fa5601323e0 2021-05-25T11:59:42.288 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.243+0000 7fa54e7fc700 1 -- 172.21.15.181:0/1747216706 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa56012c9f0 con 0x7fa5601323e0 2021-05-25T11:59:42.289 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.243+0000 7fa54e7fc700 1 -- 172.21.15.181:0/1747216706 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3785301276 0 0) 0x7fa560098710 con 0x7fa560064180 2021-05-25T11:59:42.289 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.243+0000 7fa54e7fc700 1 -- 172.21.15.181:0/1747216706 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa56009a7b0 con 0x7fa560064180 2021-05-25T11:59:42.289 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.243+0000 7fa54e7fc700 1 -- 172.21.15.181:0/1747216706 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3216734528 0 0) 0x7fa53c004c10 con 0x7fa560128980 2021-05-25T11:59:42.289 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.243+0000 7fa54e7fc700 1 -- 172.21.15.181:0/1747216706 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa560098710 con 0x7fa560128980 2021-05-25T11:59:42.290 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.243+0000 7fa54e7fc700 1 -- 172.21.15.181:0/1747216706 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fa550001b80 con 0x7fa5601323e0 2021-05-25T11:59:42.290 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.243+0000 7fa54e7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:42.290 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.243+0000 7fa54e7fc700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T11:59:42.290 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.243+0000 7fa54e7fc700 10 monclient(hunting): dump: 2021-05-25T11:59:42.291 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 1 2021-05-25T11:59:42.291 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:42.291 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:42.291 INFO:teuthology.orchestra.run.smithi181.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:42.292 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:42.292 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 3 2021-05-25T11:59:42.292 INFO:teuthology.orchestra.run.smithi181.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:42.292 INFO:teuthology.orchestra.run.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:42.293 INFO:teuthology.orchestra.run.smithi181.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:42.293 INFO:teuthology.orchestra.run.smithi181.stderr: 2021-05-25T11:59:42.293 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.243+0000 7fa54e7fc700 10 monclient(hunting): _finish_auth 0 2021-05-25T11:59:42.293 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.243+0000 7fa54e7fc700 1 -- 172.21.15.181:0/1747216706 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fa5480015d0 con 0x7fa560064180 2021-05-25T11:59:42.293 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.243+0000 7fa54e7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:42.294 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.243+0000 7fa54e7fc700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2021-05-25T11:59:42.294 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.243+0000 7fa54e7fc700 10 monclient(hunting): dump: 2021-05-25T11:59:42.294 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 1 2021-05-25T11:59:42.294 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:42.295 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:42.295 INFO:teuthology.orchestra.run.smithi181.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:42.295 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:42.295 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 3 2021-05-25T11:59:42.295 INFO:teuthology.orchestra.run.smithi181.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:42.296 INFO:teuthology.orchestra.run.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:42.296 INFO:teuthology.orchestra.run.smithi181.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:42.296 INFO:teuthology.orchestra.run.smithi181.stderr: 2021-05-25T11:59:42.296 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.243+0000 7fa54e7fc700 1 -- 172.21.15.181:0/1747216706 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fa554002950 con 0x7fa560128980 2021-05-25T11:59:42.297 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.243+0000 7fa54e7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:42.297 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.243+0000 7fa54e7fc700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2021-05-25T11:59:42.297 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.243+0000 7fa54e7fc700 10 monclient(hunting): dump: 2021-05-25T11:59:42.297 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 1 2021-05-25T11:59:42.297 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:42.298 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:42.298 INFO:teuthology.orchestra.run.smithi181.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:42.298 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:42.298 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 3 2021-05-25T11:59:42.299 INFO:teuthology.orchestra.run.smithi181.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:42.299 INFO:teuthology.orchestra.run.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:42.299 INFO:teuthology.orchestra.run.smithi181.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:42.299 INFO:teuthology.orchestra.run.smithi181.stderr: 2021-05-25T11:59:42.299 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.243+0000 7fa54e7fc700 1 -- 172.21.15.181:0/1747216706 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 198318796 0 0) 0x7fa56012c9f0 con 0x7fa5601323e0 2021-05-25T11:59:42.300 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.243+0000 7fa54e7fc700 1 -- 172.21.15.181:0/1747216706 >> v1:172.21.15.181:6789/0 conn(0x7fa560128980 legacy=0x7fa560130ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:42.300 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.243+0000 7fa54e7fc700 1 -- 172.21.15.181:0/1747216706 >> v1:172.21.15.102:6789/0 conn(0x7fa560064180 legacy=0x7fa560137950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:42.300 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.244+0000 7fa54e7fc700 10 monclient: _finish_hunting 0 2021-05-25T11:59:42.300 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.244+0000 7fa54e7fc700 1 monclient: found mon.c 2021-05-25T11:59:42.300 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.244+0000 7fa54e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T11:59:42.301 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.244+0000 7fa54e7fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T11:59:42.301 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.244+0000 7fa54e7fc700 1 -- 172.21.15.181:0/1747216706 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa56012bf50 con 0x7fa5601323e0 2021-05-25T11:59:42.301 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.244+0000 7fa54e7fc700 10 monclient: _finish_auth 0 2021-05-25T11:59:42.301 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.244+0000 7fa54e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T11:59:42.301 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.244+0000 7fa564b5a700 5 monclient: authenticate success, global_id 4124 2021-05-25T11:59:42.302 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.244+0000 7fa564b5a700 10 monclient: _renew_subs 2021-05-25T11:59:42.302 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.244+0000 7fa564b5a700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T11:59:42.302 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.244+0000 7fa564b5a700 1 -- 172.21.15.181:0/1747216706 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa56012b4b0 con 0x7fa5601323e0 2021-05-25T11:59:42.302 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.244+0000 7fa564b5a700 10 client.4124.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T11:59:42.303 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.244+0000 7fa564b5a700 10 monclient: _renew_subs 2021-05-25T11:59:42.303 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.244+0000 7fa564b5a700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T11:59:42.303 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.244+0000 7fa564b5a700 1 -- 172.21.15.181:0/1747216706 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa560138600 con 0x7fa5601323e0 2021-05-25T11:59:42.303 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.244+0000 7fa564b5a700 1 librados: init done 2021-05-25T11:59:42.304 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.244+0000 7fa54e7fc700 1 -- 172.21.15.181:0/1747216706 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa550001f90 con 0x7fa5601323e0 2021-05-25T11:59:42.304 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.244+0000 7fa54e7fc700 10 monclient: handle_config config(0 keys) v1 2021-05-25T11:59:42.304 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.244+0000 7fa54e7fc700 1 -- 172.21.15.181:0/1747216706 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fa5500022a0 con 0x7fa5601323e0 2021-05-25T11:59:42.304 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.244+0000 7fa54e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:42.305 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.244+0000 7fa54e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T11:59:42.305 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.244+0000 7fa55cd99700 4 set_mon_vals no callback set 2021-05-25T11:59:42.305 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.244+0000 7fa54e7fc700 10 monclient: dump: 2021-05-25T11:59:42.306 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 1 2021-05-25T11:59:42.306 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:42.306 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:42.306 INFO:teuthology.orchestra.run.smithi181.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:42.307 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:42.307 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 3 2021-05-25T11:59:42.307 INFO:teuthology.orchestra.run.smithi181.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:42.307 INFO:teuthology.orchestra.run.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:42.307 INFO:teuthology.orchestra.run.smithi181.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:42.308 INFO:teuthology.orchestra.run.smithi181.stderr: 2021-05-25T11:59:42.308 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.244+0000 7fa54e7fc700 1 -- 172.21.15.181:0/1747216706 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 1) v1 ==== 623+0+0 (unknown 1065898531 0 0) 0x7fa5500027e0 con 0x7fa5601323e0 2021-05-25T11:59:42.308 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.244+0000 7fa54e7fc700 1 -- 172.21.15.181:0/1747216706 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 984+0+0 (unknown 327445535 0 0) 0x7fa550003b20 con 0x7fa5601323e0 2021-05-25T11:59:42.308 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.244+0000 7fa54e7fc700 10 client.4124.objecter ms_dispatch 0x7fa560001550 osd_map(4..4 src has 1..4) v4 2021-05-25T11:59:42.308 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.244+0000 7fa54e7fc700 3 client.4124.objecter handle_osd_map got epochs [4,4] > 0 2021-05-25T11:59:42.309 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.244+0000 7fa54e7fc700 3 client.4124.objecter handle_osd_map decoding full epoch 4 2021-05-25T11:59:42.309 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.244+0000 7fa54e7fc700 20 client.4124.objecter dump_active .. 0 homeless 2021-05-25T11:59:42.309 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.244+0000 7fa564b5a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:42.309 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.244+0000 7fa564b5a700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:42.310 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.244+0000 7fa564b5a700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T11:59:42.310 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.244+0000 7fa564b5a700 1 -- 172.21.15.181:0/1747216706 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa540004db0 con 0x7fa5601323e0 2021-05-25T11:59:42.310 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.247+0000 7fa54e7fc700 1 -- 172.21.15.181:0/1747216706 <== mon.2 v1:172.21.15.102:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72237 (unknown 1092875540 0 1415944542) 0x7fa5500037f0 con 0x7fa5601323e0 2021-05-25T11:59:42.310 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.247+0000 7fa54e7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:42.310 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.247+0000 7fa54e7fc700 10 monclient: _finish_command 1 = system:0 2021-05-25T11:59:42.328 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.326+0000 7fa564b5a700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "80a3eb90-b36e-4035-af91-3917fbf8dc2a", "id": 2}] 2021-05-25T11:59:42.329 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.326+0000 7fa564b5a700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "80a3eb90-b36e-4035-af91-3917fbf8dc2a", "id": 2}] 2021-05-25T11:59:42.329 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.326+0000 7fa564b5a700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T11:59:42.329 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.326+0000 7fa564b5a700 1 -- 172.21.15.181:0/1747216706 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "osd new", "uuid": "80a3eb90-b36e-4035-af91-3917fbf8dc2a", "id": 2} v 0) v1 -- 0x7fa5400055e0 con 0x7fa5601323e0 2021-05-25T11:59:42.331 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.329+0000 7fa54e7fc700 1 -- 172.21.15.181:0/1747216706 <== mon.2 v1:172.21.15.102:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "80a3eb90-b36e-4035-af91-3917fbf8dc2a", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 2024643355 0 216599125) 0x7fa550002ca0 con 0x7fa5601323e0 2021-05-25T11:59:42.331 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.329+0000 7fa54e7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "80a3eb90-b36e-4035-af91-3917fbf8dc2a", "id": 2}] 2021-05-25T11:59:42.332 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.329+0000 7fa54e7fc700 10 monclient: _finish_command 2 = system:0 2021-05-25T11:59:42.332 INFO:teuthology.orchestra.run.smithi181.stdout:2 2021-05-25T11:59:42.334 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.330+0000 7fa564b5a700 10 librados: watch_flush enter 2021-05-25T11:59:42.334 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.330+0000 7fa564b5a700 10 librados: watch_flush exit 2021-05-25T11:59:42.334 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.330+0000 7fa564b5a700 20 client.4124.objecter shutdown clearing up homeless session... 2021-05-25T11:59:42.334 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.330+0000 7fa564b5a700 10 client.4124.objecter successfully canceled tick 2021-05-25T11:59:42.334 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.330+0000 7fa564b5a700 10 monclient: shutdown 2021-05-25T11:59:42.335 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.330+0000 7fa564b5a700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T11:59:42.335 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.330+0000 7fa564b5a700 1 -- 172.21.15.181:0/1747216706 >> v1:172.21.15.102:6790/0 conn(0x7fa5601323e0 legacy=0x7fa5601431e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:42.335 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.330+0000 7fa564b5a700 1 -- 172.21.15.181:0/1747216706 shutdown_connections 2021-05-25T11:59:42.335 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.330+0000 7fa564b5a700 1 -- 172.21.15.181:0/1747216706 >> 172.21.15.181:0/1747216706 conn(0x7fa56009b800 msgr2=0x7fa560131ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T11:59:42.336 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.330+0000 7fa564b5a700 1 -- 172.21.15.181:0/1747216706 shutdown_connections 2021-05-25T11:59:42.336 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.330+0000 7fa564b5a700 1 -- 172.21.15.181:0/1747216706 wait complete. 2021-05-25T11:59:42.336 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.330+0000 7fa564b5a700 1 librados: shutdown 2021-05-25T11:59:42.344 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster ceph osd new 45889d6a-dcfb-4670-afef-cee9bbb0476a 3 2021-05-25T11:59:42.454 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.451+0000 7fac430cc700 10 monclient: get_monmap_and_config 2021-05-25T11:59:42.454 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.451+0000 7fac430cc700 10 monclient: build_initial_monmap 2021-05-25T11:59:42.454 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.451+0000 7fac430cc700 10 monclient: monmap: 2021-05-25T11:59:42.455 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 0 2021-05-25T11:59:42.455 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:42.455 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 2021-05-25T11:59:42.453556+0000 2021-05-25T11:59:42.455 INFO:teuthology.orchestra.run.smithi181.stderr:created 2021-05-25T11:59:42.453556+0000 2021-05-25T11:59:42.455 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 0 (unknown) 2021-05-25T11:59:42.456 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 3 2021-05-25T11:59:42.456 INFO:teuthology.orchestra.run.smithi181.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T11:59:42.456 INFO:teuthology.orchestra.run.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T11:59:42.457 INFO:teuthology.orchestra.run.smithi181.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T11:59:42.457 INFO:teuthology.orchestra.run.smithi181.stderr: 2021-05-25T11:59:42.468 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.452+0000 7fac430cc700 1 Processor -- start 2021-05-25T11:59:42.468 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.452+0000 7fac430cc700 1 -- start start 2021-05-25T11:59:42.469 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.452+0000 7fac430cc700 10 monclient: init 2021-05-25T11:59:42.469 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.452+0000 7fac430cc700 10 monclient: _reopen_session rank -1 2021-05-25T11:59:42.469 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.452+0000 7fac430cc700 10 monclient: _add_conns ranks=[1,2,0] 2021-05-25T11:59:42.469 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.452+0000 7fac430cc700 10 monclient(hunting): picked mon.noname-c con 0x7fac3c098c10 addr v1:172.21.15.181:6789/0 2021-05-25T11:59:42.470 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.452+0000 7fac430cc700 10 monclient(hunting): picked mon.noname-b con 0x7fac3c098140 addr v1:172.21.15.102:6790/0 2021-05-25T11:59:42.470 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.452+0000 7fac430cc700 10 monclient(hunting): picked mon.noname-a con 0x7fac3c0985d0 addr v1:172.21.15.102:6789/0 2021-05-25T11:59:42.470 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.452+0000 7fac430cc700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac3c1291a0 con 0x7fac3c0985d0 2021-05-25T11:59:42.470 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.453+0000 7fac430cc700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac3c09b2d0 con 0x7fac3c098c10 2021-05-25T11:59:42.471 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.453+0000 7fac430cc700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac3c090990 con 0x7fac3c098140 2021-05-25T11:59:42.471 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.453+0000 7fac430cc700 10 monclient(hunting): _renew_subs 2021-05-25T11:59:42.471 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.453+0000 7fac430cc700 10 monclient(hunting): authenticate will time out at 781.937037s 2021-05-25T11:59:42.471 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.453+0000 7fac40e68700 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7fac3c098c10 0x7fac3c128a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:34578/0 (socket says 172.21.15.181:34578) 2021-05-25T11:59:42.471 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.453+0000 7fac40e68700 1 -- 172.21.15.181:0/3682540091 learned_addr learned my addr 172.21.15.181:0/3682540091 (peer_addr_for_me v1:172.21.15.181:0/0) 2021-05-25T11:59:42.472 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.453+0000 7fac337fe700 1 -- 172.21.15.181:0/3682540091 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 34230156 0 0) 0x7fac3c09b2d0 con 0x7fac3c098c10 2021-05-25T11:59:42.472 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.453+0000 7fac337fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:42.472 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.453+0000 7fac337fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:42.472 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.453+0000 7fac337fe700 10 monclient(hunting): my global_id is 4129 2021-05-25T11:59:42.473 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.453+0000 7fac337fe700 1 -- 172.21.15.181:0/3682540091 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac28003270 con 0x7fac3c098c10 2021-05-25T11:59:42.473 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.453+0000 7fac337fe700 1 -- 172.21.15.181:0/3682540091 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4286315559 0 0) 0x7fac3c090990 con 0x7fac3c098140 2021-05-25T11:59:42.473 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.453+0000 7fac337fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:42.473 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.453+0000 7fac337fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:42.473 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.453+0000 7fac337fe700 10 monclient(hunting): my global_id is 4127 2021-05-25T11:59:42.474 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.453+0000 7fac337fe700 1 -- 172.21.15.181:0/3682540091 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac3c09b2d0 con 0x7fac3c098140 2021-05-25T11:59:42.474 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.453+0000 7fac337fe700 1 -- 172.21.15.181:0/3682540091 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3202655005 0 0) 0x7fac3c1291a0 con 0x7fac3c0985d0 2021-05-25T11:59:42.474 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.453+0000 7fac337fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:42.474 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.453+0000 7fac337fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:42.475 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.453+0000 7fac337fe700 10 monclient(hunting): my global_id is 4128 2021-05-25T11:59:42.475 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.453+0000 7fac337fe700 1 -- 172.21.15.181:0/3682540091 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac3c090990 con 0x7fac3c0985d0 2021-05-25T11:59:42.475 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.453+0000 7fac337fe700 1 -- 172.21.15.181:0/3682540091 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 297220043 0 0) 0x7fac28003270 con 0x7fac3c098c10 2021-05-25T11:59:42.475 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.453+0000 7fac337fe700 1 -- 172.21.15.181:0/3682540091 >> v1:172.21.15.102:6790/0 conn(0x7fac3c098140 legacy=0x7fac3c130c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:42.475 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.453+0000 7fac337fe700 1 -- 172.21.15.181:0/3682540091 >> v1:172.21.15.102:6789/0 conn(0x7fac3c0985d0 legacy=0x7fac3c134330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:42.476 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.453+0000 7fac337fe700 10 monclient: _finish_hunting 0 2021-05-25T11:59:42.476 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.453+0000 7fac337fe700 1 monclient: found mon.noname-c 2021-05-25T11:59:42.476 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.453+0000 7fac337fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T11:59:42.476 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.453+0000 7fac337fe700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.181:6789/0 2021-05-25T11:59:42.476 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.453+0000 7fac337fe700 1 -- 172.21.15.181:0/3682540091 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac3c090c30 con 0x7fac3c098c10 2021-05-25T11:59:42.477 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.453+0000 7fac337fe700 10 monclient: _finish_auth 0 2021-05-25T11:59:42.477 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.453+0000 7fac337fe700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T11:59:42.477 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.453+0000 7fac337fe700 1 -- 172.21.15.181:0/3682540091 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fac2c001590 con 0x7fac3c098c10 2021-05-25T11:59:42.477 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.453+0000 7fac430cc700 5 monclient: authenticate success, global_id 4129 2021-05-25T11:59:42.478 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.453+0000 7fac337fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:42.478 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.453+0000 7fac337fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T11:59:42.478 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.453+0000 7fac337fe700 10 monclient: dump: 2021-05-25T11:59:42.478 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 1 2021-05-25T11:59:42.479 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:42.479 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:42.479 INFO:teuthology.orchestra.run.smithi181.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:42.479 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:42.480 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 3 2021-05-25T11:59:42.480 INFO:teuthology.orchestra.run.smithi181.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:42.480 INFO:teuthology.orchestra.run.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:42.480 INFO:teuthology.orchestra.run.smithi181.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:42.480 INFO:teuthology.orchestra.run.smithi181.stderr: 2021-05-25T11:59:42.481 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.453+0000 7fac430cc700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T11:59:42.481 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.453+0000 7fac337fe700 1 -- 172.21.15.181:0/3682540091 <== mon.1 v1:172.21.15.181:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fac2c0017b0 con 0x7fac3c098c10 2021-05-25T11:59:42.481 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.453+0000 7fac337fe700 10 monclient: handle_config config(0 keys) v1 2021-05-25T11:59:42.481 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.453+0000 7fac337fe700 1 -- 172.21.15.181:0/3682540091 <== mon.1 v1:172.21.15.181:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fac2c001ac0 con 0x7fac3c098c10 2021-05-25T11:59:42.481 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.453+0000 7fac430cc700 10 monclient: get_monmap_and_config success 2021-05-25T11:59:42.482 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.453+0000 7fac430cc700 4 set_mon_vals no callback set 2021-05-25T11:59:42.482 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.453+0000 7fac337fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:42.482 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.453+0000 7fac337fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T11:59:42.482 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.453+0000 7fac337fe700 10 monclient: dump: 2021-05-25T11:59:42.483 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 1 2021-05-25T11:59:42.483 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:42.483 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:42.483 INFO:teuthology.orchestra.run.smithi181.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:42.483 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:42.484 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 3 2021-05-25T11:59:42.484 INFO:teuthology.orchestra.run.smithi181.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:42.484 INFO:teuthology.orchestra.run.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:42.484 INFO:teuthology.orchestra.run.smithi181.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:42.484 INFO:teuthology.orchestra.run.smithi181.stderr: 2021-05-25T11:59:42.485 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.454+0000 7fac430cc700 10 monclient: shutdown 2021-05-25T11:59:42.485 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.454+0000 7fac430cc700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T11:59:42.485 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.454+0000 7fac430cc700 1 -- 172.21.15.181:0/3682540091 >> v1:172.21.15.181:6789/0 conn(0x7fac3c098c10 legacy=0x7fac3c128a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:42.485 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.454+0000 7fac430cc700 1 -- 172.21.15.181:0/3682540091 shutdown_connections 2021-05-25T11:59:42.486 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.454+0000 7fac430cc700 1 -- 172.21.15.181:0/3682540091 >> 172.21.15.181:0/3682540091 conn(0x7fac3c09b640 msgr2=0x7fac3c09ba50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T11:59:42.486 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.454+0000 7fac430cc700 1 -- 172.21.15.181:0/3682540091 shutdown_connections 2021-05-25T11:59:42.486 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.454+0000 7fac430cc700 1 -- 172.21.15.181:0/3682540091 wait complete. 2021-05-25T11:59:42.486 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.454+0000 7fac430cc700 10 monclient: build_initial_monmap 2021-05-25T11:59:42.486 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.454+0000 7fac430cc700 10 monclient: monmap: 2021-05-25T11:59:42.487 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 0 2021-05-25T11:59:42.487 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T11:59:42.487 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 0.000000 2021-05-25T11:59:42.487 INFO:teuthology.orchestra.run.smithi181.stderr:created 0.000000 2021-05-25T11:59:42.488 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 0 (unknown) 2021-05-25T11:59:42.488 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 1 2021-05-25T11:59:42.488 INFO:teuthology.orchestra.run.smithi181.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T11:59:42.488 INFO:teuthology.orchestra.run.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T11:59:42.488 INFO:teuthology.orchestra.run.smithi181.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T11:59:42.489 INFO:teuthology.orchestra.run.smithi181.stderr: 2021-05-25T11:59:42.489 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.455+0000 7fac430cc700 1 librados: starting msgr at 2021-05-25T11:59:42.489 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.455+0000 7fac430cc700 1 librados: starting objecter 2021-05-25T11:59:42.489 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.455+0000 7fac430cc700 1 Processor -- start 2021-05-25T11:59:42.489 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.455+0000 7fac430cc700 1 -- start start 2021-05-25T11:59:42.490 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.455+0000 7fac430cc700 1 librados: setting wanted keys 2021-05-25T11:59:42.490 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.455+0000 7fac430cc700 1 librados: calling monclient init 2021-05-25T11:59:42.490 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.455+0000 7fac430cc700 10 monclient: init 2021-05-25T11:59:42.490 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.456+0000 7fac430cc700 10 monclient: _reopen_session rank -1 2021-05-25T11:59:42.491 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.456+0000 7fac430cc700 10 monclient: _add_conns ranks=[0,1,2] 2021-05-25T11:59:42.491 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.456+0000 7fac430cc700 10 monclient(hunting): picked mon.noname-a con 0x7fac3c098140 addr v1:172.21.15.102:6789/0 2021-05-25T11:59:42.491 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.456+0000 7fac430cc700 10 monclient(hunting): picked mon.noname-b con 0x7fac3c0985d0 addr v1:172.21.15.181:6789/0 2021-05-25T11:59:42.491 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.456+0000 7fac430cc700 10 monclient(hunting): picked mon.noname-c con 0x7fac3c098c10 addr v1:172.21.15.102:6790/0 2021-05-25T11:59:42.491 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.456+0000 7fac430cc700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac3c126840 con 0x7fac3c098140 2021-05-25T11:59:42.492 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.456+0000 7fac430cc700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac3c1213d0 con 0x7fac3c0985d0 2021-05-25T11:59:42.492 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.456+0000 7fac430cc700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac3c09a720 con 0x7fac3c098c10 2021-05-25T11:59:42.492 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.456+0000 7fac430cc700 10 monclient(hunting): _renew_subs 2021-05-25T11:59:42.492 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.456+0000 7fac430cc700 10 monclient(hunting): authenticate will time out at 781.940330s 2021-05-25T11:59:42.492 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.456+0000 7fac33fff700 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7fac3c0985d0 0x7fac3c120320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:34584/0 (socket says 172.21.15.181:34584) 2021-05-25T11:59:42.493 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.456+0000 7fac33fff700 1 -- 172.21.15.181:0/955568681 learned_addr learned my addr 172.21.15.181:0/955568681 (peer_addr_for_me v1:172.21.15.181:0/0) 2021-05-25T11:59:42.493 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.456+0000 7fac30ff9700 10 client.?.objecter ms_handle_connect 0x7fac3c0985d0 2021-05-25T11:59:42.493 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.456+0000 7fac30ff9700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:42.493 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.456+0000 7fac30ff9700 10 client.?.objecter ms_handle_connect 0x7fac3c098c10 2021-05-25T11:59:42.493 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.456+0000 7fac30ff9700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:42.494 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.456+0000 7fac30ff9700 10 client.?.objecter ms_handle_connect 0x7fac3c098140 2021-05-25T11:59:42.494 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.456+0000 7fac30ff9700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:42.494 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.456+0000 7fac30ff9700 1 -- 172.21.15.181:0/955568681 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4231314855 0 0) 0x7fac3c1213d0 con 0x7fac3c0985d0 2021-05-25T11:59:42.494 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.456+0000 7fac30ff9700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:42.495 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.456+0000 7fac30ff9700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:42.495 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.456+0000 7fac30ff9700 10 monclient(hunting): my global_id is 4132 2021-05-25T11:59:42.495 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.456+0000 7fac30ff9700 1 -- 172.21.15.181:0/955568681 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac1c004c10 con 0x7fac3c0985d0 2021-05-25T11:59:42.495 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.456+0000 7fac30ff9700 1 -- 172.21.15.181:0/955568681 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2690376642 0 0) 0x7fac3c09a720 con 0x7fac3c098c10 2021-05-25T11:59:42.496 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.456+0000 7fac30ff9700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:42.496 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.456+0000 7fac30ff9700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:42.496 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.456+0000 7fac30ff9700 10 monclient(hunting): my global_id is 4130 2021-05-25T11:59:42.496 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.456+0000 7fac30ff9700 1 -- 172.21.15.181:0/955568681 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac3c1213d0 con 0x7fac3c098c10 2021-05-25T11:59:42.496 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.456+0000 7fac30ff9700 1 -- 172.21.15.181:0/955568681 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4158057604 0 0) 0x7fac3c126840 con 0x7fac3c098140 2021-05-25T11:59:42.497 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.456+0000 7fac30ff9700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:42.497 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.456+0000 7fac30ff9700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:42.497 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.456+0000 7fac30ff9700 10 monclient(hunting): my global_id is 4131 2021-05-25T11:59:42.497 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.456+0000 7fac30ff9700 1 -- 172.21.15.181:0/955568681 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac3c09a720 con 0x7fac3c098140 2021-05-25T11:59:42.497 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.456+0000 7fac30ff9700 1 -- 172.21.15.181:0/955568681 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1698964500 0 0) 0x7fac1c004c10 con 0x7fac3c0985d0 2021-05-25T11:59:42.498 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.456+0000 7fac30ff9700 1 -- 172.21.15.181:0/955568681 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fac3c126840 con 0x7fac3c0985d0 2021-05-25T11:59:42.498 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.456+0000 7fac30ff9700 1 -- 172.21.15.181:0/955568681 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fac240029d0 con 0x7fac3c0985d0 2021-05-25T11:59:42.498 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.456+0000 7fac30ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:42.498 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.456+0000 7fac30ff9700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T11:59:42.499 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.456+0000 7fac30ff9700 10 monclient(hunting): dump: 2021-05-25T11:59:42.499 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 1 2021-05-25T11:59:42.499 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:42.499 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:42.499 INFO:teuthology.orchestra.run.smithi181.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:42.500 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:42.500 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 3 2021-05-25T11:59:42.500 INFO:teuthology.orchestra.run.smithi181.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:42.500 INFO:teuthology.orchestra.run.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:42.500 INFO:teuthology.orchestra.run.smithi181.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:42.501 INFO:teuthology.orchestra.run.smithi181.stderr: 2021-05-25T11:59:42.501 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.456+0000 7fac30ff9700 10 monclient(hunting): _finish_auth 0 2021-05-25T11:59:42.501 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.456+0000 7fac30ff9700 1 -- 172.21.15.181:0/955568681 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1278729717 0 0) 0x7fac3c1213d0 con 0x7fac3c098c10 2021-05-25T11:59:42.501 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.457+0000 7fac30ff9700 1 -- 172.21.15.181:0/955568681 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fac1c004c10 con 0x7fac3c098c10 2021-05-25T11:59:42.501 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.457+0000 7fac30ff9700 1 -- 172.21.15.181:0/955568681 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fac38001af0 con 0x7fac3c098c10 2021-05-25T11:59:42.502 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.457+0000 7fac30ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:42.502 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.457+0000 7fac30ff9700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2021-05-25T11:59:42.502 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.457+0000 7fac30ff9700 10 monclient(hunting): dump: 2021-05-25T11:59:42.502 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 1 2021-05-25T11:59:42.503 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:42.503 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:42.503 INFO:teuthology.orchestra.run.smithi181.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:42.503 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:42.503 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 3 2021-05-25T11:59:42.504 INFO:teuthology.orchestra.run.smithi181.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:42.504 INFO:teuthology.orchestra.run.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:42.504 INFO:teuthology.orchestra.run.smithi181.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:42.504 INFO:teuthology.orchestra.run.smithi181.stderr: 2021-05-25T11:59:42.504 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.457+0000 7fac30ff9700 1 -- 172.21.15.181:0/955568681 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 191967922 0 0) 0x7fac3c09a720 con 0x7fac3c098140 2021-05-25T11:59:42.505 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.457+0000 7fac30ff9700 1 -- 172.21.15.181:0/955568681 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fac3c1213d0 con 0x7fac3c098140 2021-05-25T11:59:42.505 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.457+0000 7fac30ff9700 1 -- 172.21.15.181:0/955568681 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fac2c001510 con 0x7fac3c098140 2021-05-25T11:59:42.505 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.457+0000 7fac30ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:42.505 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.457+0000 7fac30ff9700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2021-05-25T11:59:42.506 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.457+0000 7fac30ff9700 10 monclient(hunting): dump: 2021-05-25T11:59:42.506 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 1 2021-05-25T11:59:42.506 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:42.506 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:42.506 INFO:teuthology.orchestra.run.smithi181.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:42.507 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:42.507 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 3 2021-05-25T11:59:42.507 INFO:teuthology.orchestra.run.smithi181.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:42.507 INFO:teuthology.orchestra.run.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:42.508 INFO:teuthology.orchestra.run.smithi181.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:42.508 INFO:teuthology.orchestra.run.smithi181.stderr: 2021-05-25T11:59:42.508 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.457+0000 7fac30ff9700 1 -- 172.21.15.181:0/955568681 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 306722057 0 0) 0x7fac3c126840 con 0x7fac3c0985d0 2021-05-25T11:59:42.508 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.457+0000 7fac30ff9700 1 -- 172.21.15.181:0/955568681 >> v1:172.21.15.102:6790/0 conn(0x7fac3c098c10 legacy=0x7fac3c120a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:42.508 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.457+0000 7fac30ff9700 1 -- 172.21.15.181:0/955568681 >> v1:172.21.15.102:6789/0 conn(0x7fac3c098140 legacy=0x7fac3c126e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:42.509 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.457+0000 7fac30ff9700 10 monclient: _finish_hunting 0 2021-05-25T11:59:42.509 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.457+0000 7fac30ff9700 1 monclient: found mon.b 2021-05-25T11:59:42.509 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.457+0000 7fac30ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T11:59:42.509 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.457+0000 7fac30ff9700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T11:59:42.509 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.457+0000 7fac30ff9700 1 -- 172.21.15.181:0/955568681 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac3c125c90 con 0x7fac3c0985d0 2021-05-25T11:59:42.510 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.457+0000 7fac30ff9700 10 monclient: _finish_auth 0 2021-05-25T11:59:42.510 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.457+0000 7fac30ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T11:59:42.510 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.457+0000 7fac430cc700 5 monclient: authenticate success, global_id 4132 2021-05-25T11:59:42.510 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.457+0000 7fac430cc700 10 monclient: _renew_subs 2021-05-25T11:59:42.511 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.457+0000 7fac430cc700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T11:59:42.511 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.457+0000 7fac430cc700 1 -- 172.21.15.181:0/955568681 --> v1:172.21.15.181:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac3c1211a0 con 0x7fac3c0985d0 2021-05-25T11:59:42.511 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.457+0000 7fac430cc700 10 client.4132.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T11:59:42.511 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.457+0000 7fac430cc700 10 monclient: _renew_subs 2021-05-25T11:59:42.511 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.457+0000 7fac430cc700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T11:59:42.512 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.457+0000 7fac430cc700 1 -- 172.21.15.181:0/955568681 --> v1:172.21.15.181:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fac3c144310 con 0x7fac3c0985d0 2021-05-25T11:59:42.512 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.457+0000 7fac430cc700 1 librados: init done 2021-05-25T11:59:42.512 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.457+0000 7fac30ff9700 1 -- 172.21.15.181:0/955568681 <== mon.1 v1:172.21.15.181:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fac24002da0 con 0x7fac3c0985d0 2021-05-25T11:59:42.512 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.457+0000 7fac30ff9700 10 monclient: handle_config config(0 keys) v1 2021-05-25T11:59:42.512 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.457+0000 7fac30ff9700 1 -- 172.21.15.181:0/955568681 <== mon.1 v1:172.21.15.181:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fac240017c0 con 0x7fac3c0985d0 2021-05-25T11:59:42.513 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.457+0000 7fac30ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:42.513 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.457+0000 7fac30ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T11:59:42.513 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.457+0000 7fac30ff9700 10 monclient: dump: 2021-05-25T11:59:42.513 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 1 2021-05-25T11:59:42.513 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:42.514 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:42.514 INFO:teuthology.orchestra.run.smithi181.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:42.514 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:42.514 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 3 2021-05-25T11:59:42.515 INFO:teuthology.orchestra.run.smithi181.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:42.515 INFO:teuthology.orchestra.run.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:42.515 INFO:teuthology.orchestra.run.smithi181.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:42.515 INFO:teuthology.orchestra.run.smithi181.stderr: 2021-05-25T11:59:42.515 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.457+0000 7fac32ffd700 4 set_mon_vals no callback set 2021-05-25T11:59:42.516 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.457+0000 7fac30ff9700 1 -- 172.21.15.181:0/955568681 <== mon.1 v1:172.21.15.181:6789/0 7 ==== mgrmap(e 1) v1 ==== 623+0+0 (unknown 1065898531 0 0) 0x7fac24001c80 con 0x7fac3c0985d0 2021-05-25T11:59:42.516 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.457+0000 7fac30ff9700 1 -- 172.21.15.181:0/955568681 <== mon.1 v1:172.21.15.181:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1099+0+0 (unknown 2004052759 0 0) 0x7fac24006220 con 0x7fac3c0985d0 2021-05-25T11:59:42.516 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.457+0000 7fac30ff9700 10 client.4132.objecter ms_dispatch 0x7fac3c001550 osd_map(5..5 src has 1..5) v4 2021-05-25T11:59:42.516 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.457+0000 7fac30ff9700 3 client.4132.objecter handle_osd_map got epochs [5,5] > 0 2021-05-25T11:59:42.516 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.457+0000 7fac430cc700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:42.517 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.457+0000 7fac30ff9700 3 client.4132.objecter handle_osd_map decoding full epoch 5 2021-05-25T11:59:42.517 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.458+0000 7fac430cc700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:42.517 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.458+0000 7fac430cc700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T11:59:42.517 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.458+0000 7fac30ff9700 20 client.4132.objecter dump_active .. 0 homeless 2021-05-25T11:59:42.518 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.458+0000 7fac430cc700 1 -- 172.21.15.181:0/955568681 --> v1:172.21.15.181:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac14004db0 con 0x7fac3c0985d0 2021-05-25T11:59:42.518 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.460+0000 7fac30ff9700 1 -- 172.21.15.181:0/955568681 <== mon.1 v1:172.21.15.181:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72237 (unknown 1092875540 0 1415944542) 0x7fac24005d60 con 0x7fac3c0985d0 2021-05-25T11:59:42.518 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.460+0000 7fac30ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:42.518 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.460+0000 7fac30ff9700 10 monclient: _finish_command 1 = system:0 2021-05-25T11:59:42.539 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.536+0000 7fac430cc700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "45889d6a-dcfb-4670-afef-cee9bbb0476a", "id": 3}] 2021-05-25T11:59:42.539 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.536+0000 7fac430cc700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "45889d6a-dcfb-4670-afef-cee9bbb0476a", "id": 3}] 2021-05-25T11:59:42.539 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.536+0000 7fac430cc700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T11:59:42.539 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.536+0000 7fac430cc700 1 -- 172.21.15.181:0/955568681 --> v1:172.21.15.181:6789/0 -- mon_command({"prefix": "osd new", "uuid": "45889d6a-dcfb-4670-afef-cee9bbb0476a", "id": 3} v 0) v1 -- 0x7fac14005060 con 0x7fac3c0985d0 2021-05-25T11:59:42.546 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.544+0000 7fac30ff9700 1 -- 172.21.15.181:0/955568681 <== mon.1 v1:172.21.15.181:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "45889d6a-dcfb-4670-afef-cee9bbb0476a", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 839137081 0 525046306) 0x7fac24005f80 con 0x7fac3c0985d0 2021-05-25T11:59:42.547 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.544+0000 7fac30ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "45889d6a-dcfb-4670-afef-cee9bbb0476a", "id": 3}] 2021-05-25T11:59:42.547 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.544+0000 7fac30ff9700 10 monclient: _finish_command 2 = system:0 2021-05-25T11:59:42.547 INFO:teuthology.orchestra.run.smithi181.stdout:3 2021-05-25T11:59:42.549 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.545+0000 7fac430cc700 10 librados: watch_flush enter 2021-05-25T11:59:42.549 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.545+0000 7fac430cc700 10 librados: watch_flush exit 2021-05-25T11:59:42.549 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.545+0000 7fac430cc700 20 client.4132.objecter shutdown clearing up homeless session... 2021-05-25T11:59:42.549 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.545+0000 7fac430cc700 10 client.4132.objecter successfully canceled tick 2021-05-25T11:59:42.550 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.545+0000 7fac430cc700 10 monclient: shutdown 2021-05-25T11:59:42.550 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.545+0000 7fac430cc700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T11:59:42.550 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.545+0000 7fac430cc700 1 -- 172.21.15.181:0/955568681 >> v1:172.21.15.181:6789/0 conn(0x7fac3c0985d0 legacy=0x7fac3c120320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:42.550 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.545+0000 7fac430cc700 1 -- 172.21.15.181:0/955568681 shutdown_connections 2021-05-25T11:59:42.551 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.545+0000 7fac430cc700 1 -- 172.21.15.181:0/955568681 >> 172.21.15.181:0/955568681 conn(0x7fac3c09b640 msgr2=0x7fac3c127a80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T11:59:42.551 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.545+0000 7fac430cc700 1 -- 172.21.15.181:0/955568681 shutdown_connections 2021-05-25T11:59:42.551 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.545+0000 7fac430cc700 1 -- 172.21.15.181:0/955568681 wait complete. 2021-05-25T11:59:42.551 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.545+0000 7fac430cc700 1 librados: shutdown 2021-05-25T11:59:42.559 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster ceph osd new 69e097b7-1472-442f-af42-225c0e40ad5c 4 2021-05-25T11:59:42.671 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.669+0000 7fe41daaa700 10 monclient: get_monmap_and_config 2021-05-25T11:59:42.672 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.669+0000 7fe41daaa700 10 monclient: build_initial_monmap 2021-05-25T11:59:42.672 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.669+0000 7fe41daaa700 10 monclient: monmap: 2021-05-25T11:59:42.672 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 0 2021-05-25T11:59:42.672 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:42.673 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 2021-05-25T11:59:42.671301+0000 2021-05-25T11:59:42.673 INFO:teuthology.orchestra.run.smithi181.stderr:created 2021-05-25T11:59:42.671301+0000 2021-05-25T11:59:42.673 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 0 (unknown) 2021-05-25T11:59:42.673 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 3 2021-05-25T11:59:42.673 INFO:teuthology.orchestra.run.smithi181.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T11:59:42.674 INFO:teuthology.orchestra.run.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T11:59:42.674 INFO:teuthology.orchestra.run.smithi181.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T11:59:42.674 INFO:teuthology.orchestra.run.smithi181.stderr: 2021-05-25T11:59:42.685 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.670+0000 7fe41daaa700 1 Processor -- start 2021-05-25T11:59:42.685 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.670+0000 7fe41daaa700 1 -- start start 2021-05-25T11:59:42.685 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.670+0000 7fe41daaa700 10 monclient: init 2021-05-25T11:59:42.685 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.670+0000 7fe41daaa700 10 monclient: _reopen_session rank -1 2021-05-25T11:59:42.686 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.670+0000 7fe41daaa700 10 monclient: _add_conns ranks=[2,0,1] 2021-05-25T11:59:42.686 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.670+0000 7fe41daaa700 10 monclient(hunting): picked mon.noname-b con 0x7fe418098280 addr v1:172.21.15.102:6790/0 2021-05-25T11:59:42.686 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.670+0000 7fe41daaa700 10 monclient(hunting): picked mon.noname-a con 0x7fe41812bde0 addr v1:172.21.15.102:6789/0 2021-05-25T11:59:42.686 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.670+0000 7fe41daaa700 10 monclient(hunting): picked mon.noname-c con 0x7fe41812c260 addr v1:172.21.15.181:6789/0 2021-05-25T11:59:42.687 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.670+0000 7fe41daaa700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe41809add0 con 0x7fe41812bde0 2021-05-25T11:59:42.687 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.670+0000 7fe41daaa700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe41809b070 con 0x7fe41812c260 2021-05-25T11:59:42.687 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.670+0000 7fe41daaa700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe418090aa0 con 0x7fe418098280 2021-05-25T11:59:42.687 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.670+0000 7fe41daaa700 10 monclient(hunting): _renew_subs 2021-05-25T11:59:42.687 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.670+0000 7fe41daaa700 10 monclient(hunting): authenticate will time out at 782.154745s 2021-05-25T11:59:42.688 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.670+0000 7fe41d2a9700 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7fe41812c260 0x7fe418134190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:34594/0 (socket says 172.21.15.181:34594) 2021-05-25T11:59:42.688 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.670+0000 7fe41d2a9700 1 -- 172.21.15.181:0/2845721795 learned_addr learned my addr 172.21.15.181:0/2845721795 (peer_addr_for_me v1:172.21.15.181:0/0) 2021-05-25T11:59:42.688 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.671+0000 7fe40f7fe700 1 -- 172.21.15.181:0/2845721795 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 139112832 0 0) 0x7fe41809b070 con 0x7fe41812c260 2021-05-25T11:59:42.688 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.671+0000 7fe40f7fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:42.688 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.671+0000 7fe40f7fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:42.689 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.671+0000 7fe40f7fe700 10 monclient(hunting): my global_id is 4135 2021-05-25T11:59:42.689 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.671+0000 7fe40f7fe700 1 -- 172.21.15.181:0/2845721795 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe404003270 con 0x7fe41812c260 2021-05-25T11:59:42.689 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.671+0000 7fe40f7fe700 1 -- 172.21.15.181:0/2845721795 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2395217706 0 0) 0x7fe404003270 con 0x7fe41812c260 2021-05-25T11:59:42.689 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.671+0000 7fe40f7fe700 1 -- 172.21.15.181:0/2845721795 >> v1:172.21.15.102:6790/0 conn(0x7fe418098280 legacy=0x7fe41812b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:42.690 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.671+0000 7fe40f7fe700 1 -- 172.21.15.181:0/2845721795 >> v1:172.21.15.102:6789/0 conn(0x7fe41812bde0 legacy=0x7fe418130a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:42.690 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.671+0000 7fe40f7fe700 10 monclient: _finish_hunting 0 2021-05-25T11:59:42.690 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.671+0000 7fe40f7fe700 1 monclient: found mon.noname-c 2021-05-25T11:59:42.690 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.671+0000 7fe40f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T11:59:42.690 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.671+0000 7fe40f7fe700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.181:6789/0 2021-05-25T11:59:42.691 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.671+0000 7fe40f7fe700 1 -- 172.21.15.181:0/2845721795 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe418098660 con 0x7fe41812c260 2021-05-25T11:59:42.691 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.671+0000 7fe40f7fe700 10 monclient: _finish_auth 0 2021-05-25T11:59:42.691 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.671+0000 7fe40f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T11:59:42.691 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.671+0000 7fe40f7fe700 1 -- 172.21.15.181:0/2845721795 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fe414001990 con 0x7fe41812c260 2021-05-25T11:59:42.692 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.671+0000 7fe40f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:42.692 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.671+0000 7fe40f7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T11:59:42.692 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.671+0000 7fe40f7fe700 10 monclient: dump: 2021-05-25T11:59:42.692 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 1 2021-05-25T11:59:42.692 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:42.693 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:42.693 INFO:teuthology.orchestra.run.smithi181.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:42.693 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:42.693 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 3 2021-05-25T11:59:42.693 INFO:teuthology.orchestra.run.smithi181.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:42.694 INFO:teuthology.orchestra.run.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:42.694 INFO:teuthology.orchestra.run.smithi181.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:42.694 INFO:teuthology.orchestra.run.smithi181.stderr: 2021-05-25T11:59:42.694 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.671+0000 7fe41daaa700 5 monclient: authenticate success, global_id 4135 2021-05-25T11:59:42.695 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.671+0000 7fe41daaa700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T11:59:42.695 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.671+0000 7fe40f7fe700 1 -- 172.21.15.181:0/2845721795 <== mon.1 v1:172.21.15.181:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe414001c40 con 0x7fe41812c260 2021-05-25T11:59:42.695 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.671+0000 7fe40f7fe700 10 monclient: handle_config config(0 keys) v1 2021-05-25T11:59:42.695 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.671+0000 7fe40f7fe700 1 -- 172.21.15.181:0/2845721795 <== mon.1 v1:172.21.15.181:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fe414001fd0 con 0x7fe41812c260 2021-05-25T11:59:42.695 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.671+0000 7fe41daaa700 10 monclient: get_monmap_and_config success 2021-05-25T11:59:42.696 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.671+0000 7fe41daaa700 4 set_mon_vals no callback set 2021-05-25T11:59:42.696 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.671+0000 7fe40f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:42.696 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.671+0000 7fe40f7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T11:59:42.696 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.671+0000 7fe40f7fe700 10 monclient: dump: 2021-05-25T11:59:42.697 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 1 2021-05-25T11:59:42.697 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:42.697 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:42.697 INFO:teuthology.orchestra.run.smithi181.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:42.697 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:42.698 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 3 2021-05-25T11:59:42.698 INFO:teuthology.orchestra.run.smithi181.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:42.698 INFO:teuthology.orchestra.run.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:42.698 INFO:teuthology.orchestra.run.smithi181.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:42.699 INFO:teuthology.orchestra.run.smithi181.stderr: 2021-05-25T11:59:42.699 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.672+0000 7fe41daaa700 10 monclient: shutdown 2021-05-25T11:59:42.699 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.672+0000 7fe41daaa700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T11:59:42.699 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.672+0000 7fe41daaa700 1 -- 172.21.15.181:0/2845721795 >> v1:172.21.15.181:6789/0 conn(0x7fe41812c260 legacy=0x7fe418134190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:42.699 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.672+0000 7fe41daaa700 1 -- 172.21.15.181:0/2845721795 shutdown_connections 2021-05-25T11:59:42.700 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.672+0000 7fe41daaa700 1 -- 172.21.15.181:0/2845721795 >> 172.21.15.181:0/2845721795 conn(0x7fe41809ea40 msgr2=0x7fe41809ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T11:59:42.700 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.672+0000 7fe41daaa700 1 -- 172.21.15.181:0/2845721795 shutdown_connections 2021-05-25T11:59:42.700 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.672+0000 7fe41daaa700 1 -- 172.21.15.181:0/2845721795 wait complete. 2021-05-25T11:59:42.700 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.672+0000 7fe41daaa700 10 monclient: build_initial_monmap 2021-05-25T11:59:42.700 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.672+0000 7fe41daaa700 10 monclient: monmap: 2021-05-25T11:59:42.701 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 0 2021-05-25T11:59:42.701 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T11:59:42.701 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 0.000000 2021-05-25T11:59:42.701 INFO:teuthology.orchestra.run.smithi181.stderr:created 0.000000 2021-05-25T11:59:42.701 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 0 (unknown) 2021-05-25T11:59:42.702 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 1 2021-05-25T11:59:42.702 INFO:teuthology.orchestra.run.smithi181.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T11:59:42.702 INFO:teuthology.orchestra.run.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T11:59:42.702 INFO:teuthology.orchestra.run.smithi181.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T11:59:42.703 INFO:teuthology.orchestra.run.smithi181.stderr: 2021-05-25T11:59:42.703 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.672+0000 7fe41daaa700 1 librados: starting msgr at 2021-05-25T11:59:42.703 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.672+0000 7fe41daaa700 1 librados: starting objecter 2021-05-25T11:59:42.703 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.672+0000 7fe41daaa700 1 Processor -- start 2021-05-25T11:59:42.703 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.673+0000 7fe41daaa700 1 -- start start 2021-05-25T11:59:42.704 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.673+0000 7fe41daaa700 1 librados: setting wanted keys 2021-05-25T11:59:42.704 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.673+0000 7fe41daaa700 1 librados: calling monclient init 2021-05-25T11:59:42.704 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.673+0000 7fe41daaa700 10 monclient: init 2021-05-25T11:59:42.704 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.673+0000 7fe41daaa700 10 monclient: _reopen_session rank -1 2021-05-25T11:59:42.704 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.673+0000 7fe41daaa700 10 monclient: _add_conns ranks=[0,2,1] 2021-05-25T11:59:42.705 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.673+0000 7fe41daaa700 10 monclient(hunting): picked mon.noname-a con 0x7fe418098280 addr v1:172.21.15.102:6789/0 2021-05-25T11:59:42.705 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.673+0000 7fe41daaa700 10 monclient(hunting): picked mon.noname-c con 0x7fe41812bde0 addr v1:172.21.15.102:6790/0 2021-05-25T11:59:42.705 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.673+0000 7fe41daaa700 10 monclient(hunting): picked mon.noname-b con 0x7fe41812c260 addr v1:172.21.15.181:6789/0 2021-05-25T11:59:42.705 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.673+0000 7fe41daaa700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe418115220 con 0x7fe418098280 2021-05-25T11:59:42.705 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.673+0000 7fe41daaa700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe418115450 con 0x7fe41812c260 2021-05-25T11:59:42.706 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.673+0000 7fe41daaa700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4181156f0 con 0x7fe41812bde0 2021-05-25T11:59:42.706 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.673+0000 7fe41daaa700 10 monclient(hunting): _renew_subs 2021-05-25T11:59:42.706 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.673+0000 7fe41daaa700 10 monclient(hunting): authenticate will time out at 782.157681s 2021-05-25T11:59:42.706 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.673+0000 7fe41d2a9700 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7fe41812c260 0x7fe418143720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:34600/0 (socket says 172.21.15.181:34600) 2021-05-25T11:59:42.707 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.673+0000 7fe41d2a9700 1 -- 172.21.15.181:0/3171285207 learned_addr learned my addr 172.21.15.181:0/3171285207 (peer_addr_for_me v1:172.21.15.181:0/0) 2021-05-25T11:59:42.707 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.673+0000 7fe40cff9700 10 client.?.objecter ms_handle_connect 0x7fe41812c260 2021-05-25T11:59:42.707 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.673+0000 7fe40cff9700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:42.707 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.673+0000 7fe40cff9700 10 client.?.objecter ms_handle_connect 0x7fe41812bde0 2021-05-25T11:59:42.707 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.673+0000 7fe40cff9700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:42.708 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.673+0000 7fe40cff9700 10 client.?.objecter ms_handle_connect 0x7fe418098280 2021-05-25T11:59:42.708 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.673+0000 7fe40cff9700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:42.708 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.673+0000 7fe40cff9700 1 -- 172.21.15.181:0/3171285207 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1014538807 0 0) 0x7fe418115450 con 0x7fe41812c260 2021-05-25T11:59:42.708 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.673+0000 7fe40cff9700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:42.709 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.673+0000 7fe40cff9700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:42.709 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.673+0000 7fe40cff9700 10 monclient(hunting): my global_id is 4138 2021-05-25T11:59:42.709 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.673+0000 7fe40cff9700 1 -- 172.21.15.181:0/3171285207 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3f8004c10 con 0x7fe41812c260 2021-05-25T11:59:42.709 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.674+0000 7fe40cff9700 1 -- 172.21.15.181:0/3171285207 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2410526689 0 0) 0x7fe4181156f0 con 0x7fe41812bde0 2021-05-25T11:59:42.709 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.674+0000 7fe40cff9700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:42.710 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.674+0000 7fe40cff9700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:42.710 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.674+0000 7fe40cff9700 10 monclient(hunting): my global_id is 4136 2021-05-25T11:59:42.710 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.674+0000 7fe40cff9700 1 -- 172.21.15.181:0/3171285207 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe418115450 con 0x7fe41812bde0 2021-05-25T11:59:42.710 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.674+0000 7fe40cff9700 1 -- 172.21.15.181:0/3171285207 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1553058046 0 0) 0x7fe418115220 con 0x7fe418098280 2021-05-25T11:59:42.710 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.674+0000 7fe40cff9700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:42.711 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.674+0000 7fe40cff9700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:42.711 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.674+0000 7fe40cff9700 10 monclient(hunting): my global_id is 4137 2021-05-25T11:59:42.711 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.674+0000 7fe40cff9700 1 -- 172.21.15.181:0/3171285207 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4181156f0 con 0x7fe418098280 2021-05-25T11:59:42.711 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.674+0000 7fe40cff9700 1 -- 172.21.15.181:0/3171285207 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2680121530 0 0) 0x7fe3f8004c10 con 0x7fe41812c260 2021-05-25T11:59:42.711 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.674+0000 7fe40cff9700 1 -- 172.21.15.181:0/3171285207 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe418115220 con 0x7fe41812c260 2021-05-25T11:59:42.712 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.674+0000 7fe40cff9700 1 -- 172.21.15.181:0/3171285207 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fe414001950 con 0x7fe41812c260 2021-05-25T11:59:42.712 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.674+0000 7fe40cff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:42.712 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.674+0000 7fe40cff9700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T11:59:42.712 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.674+0000 7fe40cff9700 10 monclient(hunting): dump: 2021-05-25T11:59:42.713 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 1 2021-05-25T11:59:42.713 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:42.713 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:42.713 INFO:teuthology.orchestra.run.smithi181.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:42.713 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:42.714 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 3 2021-05-25T11:59:42.714 INFO:teuthology.orchestra.run.smithi181.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:42.714 INFO:teuthology.orchestra.run.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:42.714 INFO:teuthology.orchestra.run.smithi181.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:42.715 INFO:teuthology.orchestra.run.smithi181.stderr: 2021-05-25T11:59:42.715 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.674+0000 7fe40cff9700 10 monclient(hunting): _finish_auth 0 2021-05-25T11:59:42.715 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.674+0000 7fe40cff9700 1 -- 172.21.15.181:0/3171285207 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2834166850 0 0) 0x7fe418115450 con 0x7fe41812bde0 2021-05-25T11:59:42.715 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.674+0000 7fe40cff9700 1 -- 172.21.15.181:0/3171285207 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe3f8004c10 con 0x7fe41812bde0 2021-05-25T11:59:42.715 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.674+0000 7fe40cff9700 1 -- 172.21.15.181:0/3171285207 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fe4000018c0 con 0x7fe41812bde0 2021-05-25T11:59:42.716 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.674+0000 7fe40cff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:42.716 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.674+0000 7fe40cff9700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2021-05-25T11:59:42.716 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.674+0000 7fe40cff9700 10 monclient(hunting): dump: 2021-05-25T11:59:42.716 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 1 2021-05-25T11:59:42.716 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:42.717 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:42.717 INFO:teuthology.orchestra.run.smithi181.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:42.717 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:42.717 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 3 2021-05-25T11:59:42.718 INFO:teuthology.orchestra.run.smithi181.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:42.718 INFO:teuthology.orchestra.run.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:42.718 INFO:teuthology.orchestra.run.smithi181.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:42.718 INFO:teuthology.orchestra.run.smithi181.stderr: 2021-05-25T11:59:42.718 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.674+0000 7fe40cff9700 1 -- 172.21.15.181:0/3171285207 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2307262296 0 0) 0x7fe4181156f0 con 0x7fe418098280 2021-05-25T11:59:42.719 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.674+0000 7fe40cff9700 1 -- 172.21.15.181:0/3171285207 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe418115450 con 0x7fe418098280 2021-05-25T11:59:42.719 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.674+0000 7fe40cff9700 1 -- 172.21.15.181:0/3171285207 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fe4080018c0 con 0x7fe418098280 2021-05-25T11:59:42.719 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.674+0000 7fe40cff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:42.719 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.674+0000 7fe40cff9700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2021-05-25T11:59:42.719 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.674+0000 7fe40cff9700 10 monclient(hunting): dump: 2021-05-25T11:59:42.720 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 1 2021-05-25T11:59:42.720 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:42.720 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:42.720 INFO:teuthology.orchestra.run.smithi181.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:42.721 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:42.721 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 3 2021-05-25T11:59:42.721 INFO:teuthology.orchestra.run.smithi181.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:42.721 INFO:teuthology.orchestra.run.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:42.721 INFO:teuthology.orchestra.run.smithi181.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:42.722 INFO:teuthology.orchestra.run.smithi181.stderr: 2021-05-25T11:59:42.722 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.674+0000 7fe40cff9700 1 -- 172.21.15.181:0/3171285207 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2039915222 0 0) 0x7fe418115220 con 0x7fe41812c260 2021-05-25T11:59:42.722 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.674+0000 7fe40cff9700 1 -- 172.21.15.181:0/3171285207 >> v1:172.21.15.102:6790/0 conn(0x7fe41812bde0 legacy=0x7fe41813fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:42.722 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.674+0000 7fe40cff9700 1 -- 172.21.15.181:0/3171285207 >> v1:172.21.15.102:6789/0 conn(0x7fe418098280 legacy=0x7fe41813b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:42.723 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.674+0000 7fe40cff9700 10 monclient: _finish_hunting 0 2021-05-25T11:59:42.723 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.674+0000 7fe40cff9700 1 monclient: found mon.b 2021-05-25T11:59:42.723 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.674+0000 7fe40cff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T11:59:42.723 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.674+0000 7fe40cff9700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T11:59:42.723 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.674+0000 7fe40cff9700 1 -- 172.21.15.181:0/3171285207 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe418115990 con 0x7fe41812c260 2021-05-25T11:59:42.724 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.674+0000 7fe40cff9700 10 monclient: _finish_auth 0 2021-05-25T11:59:42.724 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.674+0000 7fe40cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T11:59:42.724 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.674+0000 7fe41daaa700 5 monclient: authenticate success, global_id 4138 2021-05-25T11:59:42.724 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.674+0000 7fe41daaa700 10 monclient: _renew_subs 2021-05-25T11:59:42.725 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.674+0000 7fe41daaa700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T11:59:42.725 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.674+0000 7fe41daaa700 1 -- 172.21.15.181:0/3171285207 --> v1:172.21.15.181:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe418096800 con 0x7fe41812c260 2021-05-25T11:59:42.725 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.674+0000 7fe41daaa700 10 client.4138.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T11:59:42.725 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.674+0000 7fe41daaa700 10 monclient: _renew_subs 2021-05-25T11:59:42.725 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.674+0000 7fe41daaa700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T11:59:42.726 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.674+0000 7fe41daaa700 1 -- 172.21.15.181:0/3171285207 --> v1:172.21.15.181:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe41813ba50 con 0x7fe41812c260 2021-05-25T11:59:42.726 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.674+0000 7fe41daaa700 1 librados: init done 2021-05-25T11:59:42.726 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.674+0000 7fe40cff9700 1 -- 172.21.15.181:0/3171285207 <== mon.1 v1:172.21.15.181:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe414004cc0 con 0x7fe41812c260 2021-05-25T11:59:42.726 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.674+0000 7fe40cff9700 10 monclient: handle_config config(0 keys) v1 2021-05-25T11:59:42.727 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.674+0000 7fe40cff9700 1 -- 172.21.15.181:0/3171285207 <== mon.1 v1:172.21.15.181:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fe414004fd0 con 0x7fe41812c260 2021-05-25T11:59:42.727 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.674+0000 7fe40cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:42.727 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.674+0000 7fe40cff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T11:59:42.727 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.674+0000 7fe40cff9700 10 monclient: dump: 2021-05-25T11:59:42.727 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 1 2021-05-25T11:59:42.728 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:42.728 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:42.728 INFO:teuthology.orchestra.run.smithi181.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:42.728 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:42.729 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 3 2021-05-25T11:59:42.729 INFO:teuthology.orchestra.run.smithi181.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:42.729 INFO:teuthology.orchestra.run.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:42.729 INFO:teuthology.orchestra.run.smithi181.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:42.729 INFO:teuthology.orchestra.run.smithi181.stderr: 2021-05-25T11:59:42.730 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.674+0000 7fe40cff9700 1 -- 172.21.15.181:0/3171285207 <== mon.1 v1:172.21.15.181:6789/0 7 ==== mgrmap(e 1) v1 ==== 623+0+0 (unknown 1065898531 0 0) 0x7fe414005410 con 0x7fe41812c260 2021-05-25T11:59:42.730 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.674+0000 7fe40effd700 4 set_mon_vals no callback set 2021-05-25T11:59:42.730 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.674+0000 7fe40cff9700 1 -- 172.21.15.181:0/3171285207 <== mon.1 v1:172.21.15.181:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1214+0+0 (unknown 4174556018 0 0) 0x7fe414006600 con 0x7fe41812c260 2021-05-25T11:59:42.730 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.674+0000 7fe40cff9700 10 client.4138.objecter ms_dispatch 0x7fe418001550 osd_map(6..6 src has 1..6) v4 2021-05-25T11:59:42.731 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.674+0000 7fe40cff9700 3 client.4138.objecter handle_osd_map got epochs [6,6] > 0 2021-05-25T11:59:42.731 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.674+0000 7fe40cff9700 3 client.4138.objecter handle_osd_map decoding full epoch 6 2021-05-25T11:59:42.731 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.674+0000 7fe40cff9700 20 client.4138.objecter dump_active .. 0 homeless 2021-05-25T11:59:42.731 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.675+0000 7fe41daaa700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:42.731 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.675+0000 7fe41daaa700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:42.732 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.675+0000 7fe41daaa700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T11:59:42.732 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.675+0000 7fe41daaa700 1 -- 172.21.15.181:0/3171285207 --> v1:172.21.15.181:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3f0004db0 con 0x7fe41812c260 2021-05-25T11:59:42.732 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.677+0000 7fe40cff9700 1 -- 172.21.15.181:0/3171285207 <== mon.1 v1:172.21.15.181:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72237 (unknown 1092875540 0 1415944542) 0x7fe4140061e0 con 0x7fe41812c260 2021-05-25T11:59:42.732 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.677+0000 7fe40cff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:42.732 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.677+0000 7fe40cff9700 10 monclient: _finish_command 1 = system:0 2021-05-25T11:59:42.757 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.755+0000 7fe41daaa700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "69e097b7-1472-442f-af42-225c0e40ad5c", "id": 4}] 2021-05-25T11:59:42.758 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.755+0000 7fe41daaa700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "69e097b7-1472-442f-af42-225c0e40ad5c", "id": 4}] 2021-05-25T11:59:42.758 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.755+0000 7fe41daaa700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T11:59:42.758 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.755+0000 7fe41daaa700 1 -- 172.21.15.181:0/3171285207 --> v1:172.21.15.181:6789/0 -- mon_command({"prefix": "osd new", "uuid": "69e097b7-1472-442f-af42-225c0e40ad5c", "id": 4} v 0) v1 -- 0x7fe3f00055e0 con 0x7fe41812c260 2021-05-25T11:59:42.765 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.763+0000 7fe40cff9700 1 -- 172.21.15.181:0/3171285207 <== mon.1 v1:172.21.15.181:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "69e097b7-1472-442f-af42-225c0e40ad5c", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 1256831665 0 1697012583) 0x7fe4140058d0 con 0x7fe41812c260 2021-05-25T11:59:42.765 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.763+0000 7fe40cff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "69e097b7-1472-442f-af42-225c0e40ad5c", "id": 4}] 2021-05-25T11:59:42.766 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.763+0000 7fe40cff9700 10 monclient: _finish_command 2 = system:0 2021-05-25T11:59:42.766 INFO:teuthology.orchestra.run.smithi181.stdout:4 2021-05-25T11:59:42.767 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.764+0000 7fe41daaa700 10 librados: watch_flush enter 2021-05-25T11:59:42.768 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.764+0000 7fe41daaa700 10 librados: watch_flush exit 2021-05-25T11:59:42.768 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.764+0000 7fe41daaa700 20 client.4138.objecter shutdown clearing up homeless session... 2021-05-25T11:59:42.768 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.764+0000 7fe41daaa700 10 client.4138.objecter successfully canceled tick 2021-05-25T11:59:42.768 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.764+0000 7fe41daaa700 10 monclient: shutdown 2021-05-25T11:59:42.769 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.764+0000 7fe41daaa700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T11:59:42.769 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.764+0000 7fe41daaa700 1 -- 172.21.15.181:0/3171285207 >> v1:172.21.15.181:6789/0 conn(0x7fe41812c260 legacy=0x7fe418143720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:42.769 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.764+0000 7fe41daaa700 1 -- 172.21.15.181:0/3171285207 shutdown_connections 2021-05-25T11:59:42.769 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.764+0000 7fe41daaa700 1 -- 172.21.15.181:0/3171285207 >> 172.21.15.181:0/3171285207 conn(0x7fe41809ea40 msgr2=0x7fe41812b4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T11:59:42.769 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.764+0000 7fe41daaa700 1 -- 172.21.15.181:0/3171285207 shutdown_connections 2021-05-25T11:59:42.770 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.764+0000 7fe41daaa700 1 -- 172.21.15.181:0/3171285207 wait complete. 2021-05-25T11:59:42.770 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.764+0000 7fe41daaa700 1 librados: shutdown 2021-05-25T11:59:42.777 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster ceph osd new f1045691-db58-471e-943a-e4d99129a917 5 2021-05-25T11:59:42.887 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.885+0000 7f34cc0fa700 10 monclient: get_monmap_and_config 2021-05-25T11:59:42.887 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.885+0000 7f34cc0fa700 10 monclient: build_initial_monmap 2021-05-25T11:59:42.888 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.885+0000 7f34cc0fa700 10 monclient: monmap: 2021-05-25T11:59:42.888 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 0 2021-05-25T11:59:42.888 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:42.888 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 2021-05-25T11:59:42.886774+0000 2021-05-25T11:59:42.889 INFO:teuthology.orchestra.run.smithi181.stderr:created 2021-05-25T11:59:42.886774+0000 2021-05-25T11:59:42.889 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 0 (unknown) 2021-05-25T11:59:42.889 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 3 2021-05-25T11:59:42.889 INFO:teuthology.orchestra.run.smithi181.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T11:59:42.889 INFO:teuthology.orchestra.run.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T11:59:42.890 INFO:teuthology.orchestra.run.smithi181.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T11:59:42.890 INFO:teuthology.orchestra.run.smithi181.stderr: 2021-05-25T11:59:42.890 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.885+0000 7f34cc0fa700 1 Processor -- start 2021-05-25T11:59:42.901 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.885+0000 7f34cc0fa700 1 -- start start 2021-05-25T11:59:42.901 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.885+0000 7f34cc0fa700 10 monclient: init 2021-05-25T11:59:42.901 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.886+0000 7f34cc0fa700 10 monclient: _reopen_session rank -1 2021-05-25T11:59:42.902 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.886+0000 7f34cc0fa700 10 monclient: _add_conns ranks=[0,1,2] 2021-05-25T11:59:42.902 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.886+0000 7f34cc0fa700 10 monclient(hunting): picked mon.noname-a con 0x7f34c412ba40 addr v1:172.21.15.102:6789/0 2021-05-25T11:59:42.902 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.886+0000 7f34cc0fa700 10 monclient(hunting): picked mon.noname-c con 0x7f34c409a500 addr v1:172.21.15.181:6789/0 2021-05-25T11:59:42.902 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.886+0000 7f34cc0fa700 10 monclient(hunting): picked mon.noname-b con 0x7f34c412c330 addr v1:172.21.15.102:6790/0 2021-05-25T11:59:42.903 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.886+0000 7f34cc0fa700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34c40989d0 con 0x7f34c412ba40 2021-05-25T11:59:42.903 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.886+0000 7f34cc0fa700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34c4064380 con 0x7f34c409a500 2021-05-25T11:59:42.903 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.886+0000 7f34cc0fa700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34c409b200 con 0x7f34c412c330 2021-05-25T11:59:42.903 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.886+0000 7f34cc0fa700 10 monclient(hunting): _renew_subs 2021-05-25T11:59:42.903 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.886+0000 7f34cc0fa700 10 monclient(hunting): authenticate will time out at 782.370317s 2021-05-25T11:59:42.904 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.886+0000 7f34c9695700 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f34c409a500 0x7f34c4130dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:34604/0 (socket says 172.21.15.181:34604) 2021-05-25T11:59:42.904 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.886+0000 7f34c9695700 1 -- 172.21.15.181:0/3238879306 learned_addr learned my addr 172.21.15.181:0/3238879306 (peer_addr_for_me v1:172.21.15.181:0/0) 2021-05-25T11:59:42.904 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.886+0000 7f34c8e94700 1 -- 172.21.15.181:0/3238879306 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4203455199 0 0) 0x7f34c4064380 con 0x7f34c409a500 2021-05-25T11:59:42.904 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.886+0000 7f34c8e94700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:42.905 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.886+0000 7f34c8e94700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:42.905 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.886+0000 7f34c8e94700 10 monclient(hunting): my global_id is 4141 2021-05-25T11:59:42.905 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.886+0000 7f34c8e94700 1 -- 172.21.15.181:0/3238879306 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34ac003270 con 0x7f34c409a500 2021-05-25T11:59:42.905 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.886+0000 7f34c8e94700 1 -- 172.21.15.181:0/3238879306 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3357001037 0 0) 0x7f34c409b200 con 0x7f34c412c330 2021-05-25T11:59:42.905 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.886+0000 7f34c8e94700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:42.906 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.886+0000 7f34c8e94700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:42.906 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.886+0000 7f34c8e94700 10 monclient(hunting): my global_id is 4139 2021-05-25T11:59:42.906 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.886+0000 7f34c8e94700 1 -- 172.21.15.181:0/3238879306 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34c4064380 con 0x7f34c412c330 2021-05-25T11:59:42.906 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.886+0000 7f34c8e94700 1 -- 172.21.15.181:0/3238879306 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2982066686 0 0) 0x7f34ac003270 con 0x7f34c409a500 2021-05-25T11:59:42.907 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.886+0000 7f34c8e94700 1 -- 172.21.15.181:0/3238879306 >> v1:172.21.15.102:6790/0 conn(0x7f34c412c330 legacy=0x7f34c4134500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:42.907 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.886+0000 7f34c8e94700 1 -- 172.21.15.181:0/3238879306 >> v1:172.21.15.102:6789/0 conn(0x7f34c412ba40 legacy=0x7f34c40982c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:42.907 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.886+0000 7f34c8e94700 10 monclient: _finish_hunting 0 2021-05-25T11:59:42.907 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.886+0000 7f34c8e94700 1 monclient: found mon.noname-c 2021-05-25T11:59:42.908 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.886+0000 7f34c8e94700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T11:59:42.908 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.886+0000 7f34c8e94700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.181:6789/0 2021-05-25T11:59:42.908 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.886+0000 7f34c8e94700 1 -- 172.21.15.181:0/3238879306 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34c4090b10 con 0x7f34c409a500 2021-05-25T11:59:42.908 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.886+0000 7f34c8e94700 10 monclient: _finish_auth 0 2021-05-25T11:59:42.908 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.886+0000 7f34c8e94700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T11:59:42.909 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.886+0000 7f34cc0fa700 5 monclient: authenticate success, global_id 4141 2021-05-25T11:59:42.909 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.886+0000 7f34c8e94700 1 -- 172.21.15.181:0/3238879306 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f34b4001610 con 0x7f34c409a500 2021-05-25T11:59:42.909 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.886+0000 7f34cc0fa700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T11:59:42.909 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.886+0000 7f34c8e94700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:42.910 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.886+0000 7f34c8e94700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T11:59:42.910 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.886+0000 7f34c8e94700 10 monclient: dump: 2021-05-25T11:59:42.910 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 1 2021-05-25T11:59:42.910 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:42.910 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:42.911 INFO:teuthology.orchestra.run.smithi181.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:42.911 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:42.911 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 3 2021-05-25T11:59:42.911 INFO:teuthology.orchestra.run.smithi181.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:42.912 INFO:teuthology.orchestra.run.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:42.912 INFO:teuthology.orchestra.run.smithi181.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:42.912 INFO:teuthology.orchestra.run.smithi181.stderr: 2021-05-25T11:59:42.912 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.887+0000 7f34cc0fa700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T11:59:42.912 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.887+0000 7f34c8e94700 1 -- 172.21.15.181:0/3238879306 <== mon.1 v1:172.21.15.181:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f34b4001830 con 0x7f34c409a500 2021-05-25T11:59:42.913 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.887+0000 7f34c8e94700 10 monclient: handle_config config(0 keys) v1 2021-05-25T11:59:42.913 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.887+0000 7f34cc0fa700 10 monclient: get_monmap_and_config success 2021-05-25T11:59:42.913 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.887+0000 7f34c8e94700 1 -- 172.21.15.181:0/3238879306 <== mon.1 v1:172.21.15.181:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f34b4001b40 con 0x7f34c409a500 2021-05-25T11:59:42.913 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.887+0000 7f34cc0fa700 4 set_mon_vals no callback set 2021-05-25T11:59:42.914 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.887+0000 7f34c8e94700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:42.914 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.887+0000 7f34c8e94700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T11:59:42.914 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.887+0000 7f34c8e94700 10 monclient: dump: 2021-05-25T11:59:42.914 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 1 2021-05-25T11:59:42.914 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:42.915 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:42.915 INFO:teuthology.orchestra.run.smithi181.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:42.915 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:42.915 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 3 2021-05-25T11:59:42.916 INFO:teuthology.orchestra.run.smithi181.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:42.916 INFO:teuthology.orchestra.run.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:42.916 INFO:teuthology.orchestra.run.smithi181.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:42.916 INFO:teuthology.orchestra.run.smithi181.stderr: 2021-05-25T11:59:42.916 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.887+0000 7f34cc0fa700 10 monclient: shutdown 2021-05-25T11:59:42.917 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.887+0000 7f34cc0fa700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T11:59:42.917 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.887+0000 7f34cc0fa700 1 -- 172.21.15.181:0/3238879306 >> v1:172.21.15.181:6789/0 conn(0x7f34c409a500 legacy=0x7f34c4130dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:42.917 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.887+0000 7f34cc0fa700 1 -- 172.21.15.181:0/3238879306 shutdown_connections 2021-05-25T11:59:42.917 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.887+0000 7f34cc0fa700 1 -- 172.21.15.181:0/3238879306 >> 172.21.15.181:0/3238879306 conn(0x7f34c409b640 msgr2=0x7f34c409ba50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T11:59:42.917 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.887+0000 7f34cc0fa700 1 -- 172.21.15.181:0/3238879306 shutdown_connections 2021-05-25T11:59:42.918 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.887+0000 7f34cc0fa700 1 -- 172.21.15.181:0/3238879306 wait complete. 2021-05-25T11:59:42.918 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.888+0000 7f34cc0fa700 10 monclient: build_initial_monmap 2021-05-25T11:59:42.918 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.888+0000 7f34cc0fa700 10 monclient: monmap: 2021-05-25T11:59:42.918 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 0 2021-05-25T11:59:42.918 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T11:59:42.919 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 0.000000 2021-05-25T11:59:42.919 INFO:teuthology.orchestra.run.smithi181.stderr:created 0.000000 2021-05-25T11:59:42.919 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 0 (unknown) 2021-05-25T11:59:42.919 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 1 2021-05-25T11:59:42.919 INFO:teuthology.orchestra.run.smithi181.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T11:59:42.920 INFO:teuthology.orchestra.run.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T11:59:42.920 INFO:teuthology.orchestra.run.smithi181.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T11:59:42.920 INFO:teuthology.orchestra.run.smithi181.stderr: 2021-05-25T11:59:42.920 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.888+0000 7f34cc0fa700 1 librados: starting msgr at 2021-05-25T11:59:42.920 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.888+0000 7f34cc0fa700 1 librados: starting objecter 2021-05-25T11:59:42.921 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.888+0000 7f34cc0fa700 1 Processor -- start 2021-05-25T11:59:42.921 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.888+0000 7f34cc0fa700 1 -- start start 2021-05-25T11:59:42.921 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.888+0000 7f34cc0fa700 1 librados: setting wanted keys 2021-05-25T11:59:42.921 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.888+0000 7f34cc0fa700 1 librados: calling monclient init 2021-05-25T11:59:42.921 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.888+0000 7f34cc0fa700 10 monclient: init 2021-05-25T11:59:42.922 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.888+0000 7f34cc0fa700 10 monclient: _reopen_session rank -1 2021-05-25T11:59:42.922 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.888+0000 7f34cc0fa700 10 monclient: _add_conns ranks=[2,0,1] 2021-05-25T11:59:42.922 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.888+0000 7f34cc0fa700 10 monclient(hunting): picked mon.noname-c con 0x7f34c409a500 addr v1:172.21.15.102:6790/0 2021-05-25T11:59:42.922 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.888+0000 7f34cc0fa700 10 monclient(hunting): picked mon.noname-a con 0x7f34c412ba40 addr v1:172.21.15.102:6789/0 2021-05-25T11:59:42.923 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.888+0000 7f34cc0fa700 10 monclient(hunting): picked mon.noname-b con 0x7f34c412c330 addr v1:172.21.15.181:6789/0 2021-05-25T11:59:42.923 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.888+0000 7f34cc0fa700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34c40984f0 con 0x7f34c412ba40 2021-05-25T11:59:42.923 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.888+0000 7f34cc0fa700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34c4138b00 con 0x7f34c412c330 2021-05-25T11:59:42.923 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.888+0000 7f34cc0fa700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34c4124020 con 0x7f34c409a500 2021-05-25T11:59:42.923 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.888+0000 7f34cc0fa700 10 monclient(hunting): _renew_subs 2021-05-25T11:59:42.924 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.888+0000 7f34cc0fa700 10 monclient(hunting): authenticate will time out at 782.372915s 2021-05-25T11:59:42.924 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.889+0000 7f34c9e96700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f34c409a500 0x7f34c41287a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.181:53632/0 (socket says 172.21.15.181:53632) 2021-05-25T11:59:42.924 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.889+0000 7f34c9e96700 1 -- 172.21.15.181:0/3303977313 learned_addr learned my addr 172.21.15.181:0/3303977313 (peer_addr_for_me v1:172.21.15.181:0/0) 2021-05-25T11:59:42.924 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.889+0000 7f34ca697700 1 --1- 172.21.15.181:0/3303977313 >> v1:172.21.15.181:6789/0 conn(0x7f34c412c330 0x7f34c41439a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:34612/0 (socket says 172.21.15.181:34612) 2021-05-25T11:59:42.924 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.889+0000 7f34b1ffb700 10 client.?.objecter ms_handle_connect 0x7f34c412c330 2021-05-25T11:59:42.925 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.889+0000 7f34b1ffb700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:42.925 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.889+0000 7f34b1ffb700 10 client.?.objecter ms_handle_connect 0x7f34c409a500 2021-05-25T11:59:42.925 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.889+0000 7f34b1ffb700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:42.925 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.889+0000 7f34b1ffb700 10 client.?.objecter ms_handle_connect 0x7f34c412ba40 2021-05-25T11:59:42.925 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.889+0000 7f34b1ffb700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:42.926 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.889+0000 7f34b1ffb700 1 -- 172.21.15.181:0/3303977313 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3642624101 0 0) 0x7f34c4138b00 con 0x7f34c412c330 2021-05-25T11:59:42.926 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.889+0000 7f34b1ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:42.926 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.889+0000 7f34b1ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:42.926 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.889+0000 7f34b1ffb700 10 monclient(hunting): my global_id is 4144 2021-05-25T11:59:42.926 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.889+0000 7f34b1ffb700 1 -- 172.21.15.181:0/3303977313 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34a4004c10 con 0x7f34c412c330 2021-05-25T11:59:42.927 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.889+0000 7f34b1ffb700 1 -- 172.21.15.181:0/3303977313 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3856541071 0 0) 0x7f34c40984f0 con 0x7f34c412ba40 2021-05-25T11:59:42.927 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.889+0000 7f34b1ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:42.927 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.889+0000 7f34b1ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:42.927 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.889+0000 7f34b1ffb700 10 monclient(hunting): my global_id is 4143 2021-05-25T11:59:42.928 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.889+0000 7f34b1ffb700 1 -- 172.21.15.181:0/3303977313 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34c4138b00 con 0x7f34c412ba40 2021-05-25T11:59:42.928 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.889+0000 7f34b1ffb700 1 -- 172.21.15.181:0/3303977313 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1886427756 0 0) 0x7f34c4124020 con 0x7f34c409a500 2021-05-25T11:59:42.928 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.889+0000 7f34b1ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:42.928 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.889+0000 7f34b1ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:42.928 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.889+0000 7f34b1ffb700 10 monclient(hunting): my global_id is 4142 2021-05-25T11:59:42.929 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.889+0000 7f34b1ffb700 1 -- 172.21.15.181:0/3303977313 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34c40984f0 con 0x7f34c409a500 2021-05-25T11:59:42.929 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.889+0000 7f34b1ffb700 1 -- 172.21.15.181:0/3303977313 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2520773016 0 0) 0x7f34a4004c10 con 0x7f34c412c330 2021-05-25T11:59:42.929 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.889+0000 7f34b1ffb700 1 -- 172.21.15.181:0/3303977313 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f34c4124020 con 0x7f34c412c330 2021-05-25T11:59:42.929 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.889+0000 7f34b1ffb700 1 -- 172.21.15.181:0/3303977313 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f34c0001d10 con 0x7f34c412c330 2021-05-25T11:59:42.930 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.889+0000 7f34b1ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:42.930 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.889+0000 7f34b1ffb700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T11:59:42.930 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.889+0000 7f34b1ffb700 10 monclient(hunting): dump: 2021-05-25T11:59:42.930 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 1 2021-05-25T11:59:42.930 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:42.931 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:42.931 INFO:teuthology.orchestra.run.smithi181.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:42.931 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:42.931 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 3 2021-05-25T11:59:42.932 INFO:teuthology.orchestra.run.smithi181.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:42.932 INFO:teuthology.orchestra.run.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:42.932 INFO:teuthology.orchestra.run.smithi181.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:42.932 INFO:teuthology.orchestra.run.smithi181.stderr: 2021-05-25T11:59:42.932 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.889+0000 7f34b1ffb700 10 monclient(hunting): _finish_auth 0 2021-05-25T11:59:42.933 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.889+0000 7f34b1ffb700 1 -- 172.21.15.181:0/3303977313 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2552840889 0 0) 0x7f34c4138b00 con 0x7f34c412ba40 2021-05-25T11:59:42.933 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.889+0000 7f34b1ffb700 1 -- 172.21.15.181:0/3303977313 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f34a4004c10 con 0x7f34c412ba40 2021-05-25T11:59:42.933 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.889+0000 7f34b1ffb700 1 -- 172.21.15.181:0/3303977313 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f34b4001510 con 0x7f34c412ba40 2021-05-25T11:59:42.933 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.889+0000 7f34b1ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:42.933 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.889+0000 7f34b1ffb700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2021-05-25T11:59:42.934 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.889+0000 7f34b1ffb700 10 monclient(hunting): dump: 2021-05-25T11:59:42.934 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 1 2021-05-25T11:59:42.934 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:42.934 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:42.935 INFO:teuthology.orchestra.run.smithi181.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:42.935 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:42.935 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 3 2021-05-25T11:59:42.936 INFO:teuthology.orchestra.run.smithi181.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:42.936 INFO:teuthology.orchestra.run.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:42.936 INFO:teuthology.orchestra.run.smithi181.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:42.936 INFO:teuthology.orchestra.run.smithi181.stderr: 2021-05-25T11:59:42.936 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.889+0000 7f34b1ffb700 1 -- 172.21.15.181:0/3303977313 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4280007698 0 0) 0x7f34c40984f0 con 0x7f34c409a500 2021-05-25T11:59:42.937 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.889+0000 7f34b1ffb700 1 -- 172.21.15.181:0/3303977313 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f34c4138b00 con 0x7f34c409a500 2021-05-25T11:59:42.937 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.889+0000 7f34b1ffb700 1 -- 172.21.15.181:0/3303977313 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f34b8001990 con 0x7f34c409a500 2021-05-25T11:59:42.937 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.889+0000 7f34b1ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:42.937 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.889+0000 7f34b1ffb700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2021-05-25T11:59:42.937 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.889+0000 7f34b1ffb700 10 monclient(hunting): dump: 2021-05-25T11:59:42.938 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 1 2021-05-25T11:59:42.938 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:42.938 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:42.938 INFO:teuthology.orchestra.run.smithi181.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:42.939 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:42.939 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 3 2021-05-25T11:59:42.939 INFO:teuthology.orchestra.run.smithi181.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:42.939 INFO:teuthology.orchestra.run.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:42.939 INFO:teuthology.orchestra.run.smithi181.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:42.940 INFO:teuthology.orchestra.run.smithi181.stderr: 2021-05-25T11:59:42.940 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.889+0000 7f34b1ffb700 1 -- 172.21.15.181:0/3303977313 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 92775684 0 0) 0x7f34c4124020 con 0x7f34c412c330 2021-05-25T11:59:42.940 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.889+0000 7f34b1ffb700 1 -- 172.21.15.181:0/3303977313 >> v1:172.21.15.102:6790/0 conn(0x7f34c409a500 legacy=0x7f34c41287a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:42.940 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.889+0000 7f34b1ffb700 1 -- 172.21.15.181:0/3303977313 >> v1:172.21.15.102:6789/0 conn(0x7f34c412ba40 legacy=0x7f34c4140270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:42.941 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.889+0000 7f34b1ffb700 10 monclient: _finish_hunting 0 2021-05-25T11:59:42.941 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.889+0000 7f34b1ffb700 1 monclient: found mon.b 2021-05-25T11:59:42.941 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.889+0000 7f34b1ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T11:59:42.941 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.889+0000 7f34b1ffb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T11:59:42.941 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.889+0000 7f34b1ffb700 1 -- 172.21.15.181:0/3303977313 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34c41242c0 con 0x7f34c412c330 2021-05-25T11:59:42.942 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.889+0000 7f34b1ffb700 10 monclient: _finish_auth 0 2021-05-25T11:59:42.942 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.889+0000 7f34b1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T11:59:42.942 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.889+0000 7f34cc0fa700 5 monclient: authenticate success, global_id 4144 2021-05-25T11:59:42.942 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.889+0000 7f34cc0fa700 10 monclient: _renew_subs 2021-05-25T11:59:42.942 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.889+0000 7f34cc0fa700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T11:59:42.943 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.889+0000 7f34cc0fa700 1 -- 172.21.15.181:0/3303977313 --> v1:172.21.15.181:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34c4122250 con 0x7f34c412c330 2021-05-25T11:59:42.943 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.889+0000 7f34b1ffb700 1 -- 172.21.15.181:0/3303977313 <== mon.1 v1:172.21.15.181:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f34c0002ff0 con 0x7f34c412c330 2021-05-25T11:59:42.943 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.889+0000 7f34b1ffb700 10 monclient: handle_config config(0 keys) v1 2021-05-25T11:59:42.943 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.889+0000 7f34b1ffb700 1 -- 172.21.15.181:0/3303977313 <== mon.1 v1:172.21.15.181:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f34c00023d0 con 0x7f34c412c330 2021-05-25T11:59:42.943 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.889+0000 7f34b1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:42.944 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.889+0000 7f34b1ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T11:59:42.944 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.889+0000 7f34b1ffb700 10 monclient: dump: 2021-05-25T11:59:42.944 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 1 2021-05-25T11:59:42.944 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:42.945 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:42.945 INFO:teuthology.orchestra.run.smithi181.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:42.945 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:42.945 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 3 2021-05-25T11:59:42.945 INFO:teuthology.orchestra.run.smithi181.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:42.946 INFO:teuthology.orchestra.run.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:42.946 INFO:teuthology.orchestra.run.smithi181.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:42.946 INFO:teuthology.orchestra.run.smithi181.stderr: 2021-05-25T11:59:42.946 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.889+0000 7f34b1ffb700 1 -- 172.21.15.181:0/3303977313 <== mon.1 v1:172.21.15.181:6789/0 7 ==== mgrmap(e 1) v1 ==== 623+0+0 (unknown 1065898531 0 0) 0x7f34c0002890 con 0x7f34c412c330 2021-05-25T11:59:42.946 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.889+0000 7f34cc0fa700 10 client.4144.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T11:59:42.947 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.889+0000 7f34b3fff700 4 set_mon_vals no callback set 2021-05-25T11:59:42.947 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.889+0000 7f34cc0fa700 10 monclient: _renew_subs 2021-05-25T11:59:42.947 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.889+0000 7f34cc0fa700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T11:59:42.947 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.889+0000 7f34cc0fa700 1 -- 172.21.15.181:0/3303977313 --> v1:172.21.15.181:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f34c4129070 con 0x7f34c412c330 2021-05-25T11:59:42.948 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.889+0000 7f34cc0fa700 1 librados: init done 2021-05-25T11:59:42.948 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.890+0000 7f34b1ffb700 1 -- 172.21.15.181:0/3303977313 <== mon.1 v1:172.21.15.181:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1329+0+0 (unknown 1205360234 0 0) 0x7f34c0002da0 con 0x7f34c412c330 2021-05-25T11:59:42.948 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.890+0000 7f34b1ffb700 10 client.4144.objecter ms_dispatch 0x7f34c4001550 osd_map(7..7 src has 1..7) v4 2021-05-25T11:59:42.948 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.890+0000 7f34b1ffb700 3 client.4144.objecter handle_osd_map got epochs [7,7] > 0 2021-05-25T11:59:42.948 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.890+0000 7f34b1ffb700 3 client.4144.objecter handle_osd_map decoding full epoch 7 2021-05-25T11:59:42.949 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.890+0000 7f34b1ffb700 20 client.4144.objecter dump_active .. 0 homeless 2021-05-25T11:59:42.949 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.890+0000 7f34cc0fa700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:42.949 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.890+0000 7f34cc0fa700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:42.949 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.890+0000 7f34cc0fa700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T11:59:42.949 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.890+0000 7f34cc0fa700 1 -- 172.21.15.181:0/3303977313 --> v1:172.21.15.181:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34c40997d0 con 0x7f34c412c330 2021-05-25T11:59:42.950 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.892+0000 7f34b1ffb700 1 -- 172.21.15.181:0/3303977313 <== mon.1 v1:172.21.15.181:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72237 (unknown 1092875540 0 1415944542) 0x7f34c0005890 con 0x7f34c412c330 2021-05-25T11:59:42.950 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.892+0000 7f34b1ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:42.950 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.892+0000 7f34b1ffb700 10 monclient: _finish_command 1 = system:0 2021-05-25T11:59:42.972 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.969+0000 7f34cc0fa700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "f1045691-db58-471e-943a-e4d99129a917", "id": 5}] 2021-05-25T11:59:42.972 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.970+0000 7f34cc0fa700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "f1045691-db58-471e-943a-e4d99129a917", "id": 5}] 2021-05-25T11:59:42.972 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.970+0000 7f34cc0fa700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T11:59:42.972 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.970+0000 7f34cc0fa700 1 -- 172.21.15.181:0/3303977313 --> v1:172.21.15.181:6789/0 -- mon_command({"prefix": "osd new", "uuid": "f1045691-db58-471e-943a-e4d99129a917", "id": 5} v 0) v1 -- 0x7f34c4122da0 con 0x7f34c412c330 2021-05-25T11:59:42.979 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.977+0000 7f34b1ffb700 1 -- 172.21.15.181:0/3303977313 <== mon.1 v1:172.21.15.181:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f1045691-db58-471e-943a-e4d99129a917", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 3657875854 0 1988412176) 0x7f34c0006010 con 0x7f34c412c330 2021-05-25T11:59:42.979 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.977+0000 7f34b1ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "f1045691-db58-471e-943a-e4d99129a917", "id": 5}] 2021-05-25T11:59:42.980 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.977+0000 7f34b1ffb700 10 monclient: _finish_command 2 = system:0 2021-05-25T11:59:42.980 INFO:teuthology.orchestra.run.smithi181.stdout:5 2021-05-25T11:59:42.982 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.978+0000 7f34cc0fa700 10 librados: watch_flush enter 2021-05-25T11:59:42.982 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.978+0000 7f34cc0fa700 10 librados: watch_flush exit 2021-05-25T11:59:42.982 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.978+0000 7f34cc0fa700 20 client.4144.objecter shutdown clearing up homeless session... 2021-05-25T11:59:42.982 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.978+0000 7f34cc0fa700 10 client.4144.objecter successfully canceled tick 2021-05-25T11:59:42.983 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.978+0000 7f34cc0fa700 10 monclient: shutdown 2021-05-25T11:59:42.983 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.978+0000 7f34cc0fa700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T11:59:42.983 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.978+0000 7f34cc0fa700 1 -- 172.21.15.181:0/3303977313 >> v1:172.21.15.181:6789/0 conn(0x7f34c412c330 legacy=0x7f34c41439a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:42.983 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.978+0000 7f34cc0fa700 1 -- 172.21.15.181:0/3303977313 shutdown_connections 2021-05-25T11:59:42.983 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.978+0000 7f34cc0fa700 1 -- 172.21.15.181:0/3303977313 >> 172.21.15.181:0/3303977313 conn(0x7f34c409b640 msgr2=0x7f34c409c950 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T11:59:42.984 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.978+0000 7f34cc0fa700 1 -- 172.21.15.181:0/3303977313 shutdown_connections 2021-05-25T11:59:42.984 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.978+0000 7f34cc0fa700 1 -- 172.21.15.181:0/3303977313 wait complete. 2021-05-25T11:59:42.984 INFO:teuthology.orchestra.run.smithi181.stderr:2021-05-25T11:59:42.978+0000 7f34cc0fa700 1 librados: shutdown 2021-05-25T11:59:42.992 INFO:tasks.ceph.osd.0:Restarting daemon 2021-05-25T11:59:42.992 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2021-05-25T11:59:42.995 INFO:tasks.ceph.osd.0:Started 2021-05-25T11:59:42.995 INFO:tasks.ceph.osd.1:Restarting daemon 2021-05-25T11:59:42.996 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2021-05-25T11:59:42.998 INFO:tasks.ceph.osd.1:Started 2021-05-25T11:59:42.998 INFO:tasks.ceph.osd.2:Restarting daemon 2021-05-25T11:59:42.999 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2021-05-25T11:59:43.001 INFO:tasks.ceph.osd.2:Started 2021-05-25T11:59:43.001 INFO:tasks.ceph.osd.3:Restarting daemon 2021-05-25T11:59:43.002 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2021-05-25T11:59:43.004 INFO:tasks.ceph.osd.3:Started 2021-05-25T11:59:43.005 INFO:tasks.ceph.osd.4:Restarting daemon 2021-05-25T11:59:43.005 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2021-05-25T11:59:43.007 INFO:tasks.ceph.osd.4:Started 2021-05-25T11:59:43.008 INFO:tasks.ceph.osd.5:Restarting daemon 2021-05-25T11:59:43.008 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2021-05-25T11:59:43.010 INFO:tasks.ceph.osd.5:Started 2021-05-25T11:59:43.011 INFO:tasks.ceph:Waiting for OSDs to come up 2021-05-25T11:59:43.592 INFO:tasks.ceph.osd.4.smithi181.stderr:2021-05-25T11:59:43.589+0000 7f003195d0c0 -1 Falling back to public interface 2021-05-25T11:59:43.592 INFO:tasks.ceph.osd.1.smithi102.stderr:2021-05-25T11:59:43.591+0000 7f2deb2ad0c0 -1 Falling back to public interface 2021-05-25T11:59:43.593 INFO:tasks.ceph.osd.2.smithi102.stderr:2021-05-25T11:59:43.591+0000 7f9d6b84f0c0 -1 Falling back to public interface 2021-05-25T11:59:43.593 INFO:tasks.ceph.osd.1.smithi102.stderr:2021-05-25T11:59:43.591+0000 7f2de523b700 -1 PosixStack listen unable to listen on v1:0.0.0.0:6800/0: (98) Address already in use 2021-05-25T11:59:43.594 INFO:tasks.ceph.osd.0.smithi102.stderr:2021-05-25T11:59:43.592+0000 7f39076370c0 -1 Falling back to public interface 2021-05-25T11:59:43.594 INFO:tasks.ceph.osd.3.smithi181.stderr:2021-05-25T11:59:43.590+0000 7f150db600c0 -1 Falling back to public interface 2021-05-25T11:59:43.604 INFO:tasks.ceph.osd.5.smithi181.stderr:2021-05-25T11:59:43.602+0000 7f049afb30c0 -1 Falling back to public interface 2021-05-25T11:59:43.616 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2021-05-25T11:59:43.727 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.726+0000 7f1d487e0700 10 monclient: get_monmap_and_config 2021-05-25T11:59:43.728 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.726+0000 7f1d487e0700 10 monclient: build_initial_monmap 2021-05-25T11:59:43.728 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.726+0000 7f1d487e0700 10 monclient: monmap: 2021-05-25T11:59:43.729 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2021-05-25T11:59:43.729 INFO:teuthology.misc.health.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:43.729 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2021-05-25T11:59:43.727126+0000 2021-05-25T11:59:43.729 INFO:teuthology.misc.health.smithi102.stderr:created 2021-05-25T11:59:43.727126+0000 2021-05-25T11:59:43.729 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T11:59:43.730 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:43.730 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T11:59:43.730 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T11:59:43.730 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T11:59:43.731 INFO:teuthology.misc.health.smithi102.stderr: 2021-05-25T11:59:43.731 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.726+0000 7f1d487e0700 1 Processor -- start 2021-05-25T11:59:43.731 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.726+0000 7f1d487e0700 1 -- start start 2021-05-25T11:59:43.731 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.726+0000 7f1d487e0700 10 monclient: init 2021-05-25T11:59:43.742 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.727+0000 7f1d487e0700 10 monclient: _reopen_session rank -1 2021-05-25T11:59:43.742 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.727+0000 7f1d487e0700 10 monclient: _add_conns ranks=[2,1,0] 2021-05-25T11:59:43.742 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.727+0000 7f1d487e0700 10 monclient(hunting): picked mon.noname-b con 0x7f1d40098330 addr v1:172.21.15.102:6790/0 2021-05-25T11:59:43.742 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.727+0000 7f1d487e0700 10 monclient(hunting): picked mon.noname-c con 0x7f1d4012be70 addr v1:172.21.15.181:6789/0 2021-05-25T11:59:43.743 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.727+0000 7f1d487e0700 10 monclient(hunting): picked mon.noname-a con 0x7f1d4012c320 addr v1:172.21.15.102:6789/0 2021-05-25T11:59:43.743 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.727+0000 7f1d487e0700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d40064b90 con 0x7f1d4012c320 2021-05-25T11:59:43.743 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.727+0000 7f1d487e0700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d40090b50 con 0x7f1d4012be70 2021-05-25T11:59:43.743 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.727+0000 7f1d487e0700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d4009af80 con 0x7f1d40098330 2021-05-25T11:59:43.744 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.727+0000 7f1d487e0700 10 monclient(hunting): _renew_subs 2021-05-25T11:59:43.744 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.727+0000 7f1d487e0700 10 monclient(hunting): authenticate will time out at 784.324468s 2021-05-25T11:59:43.744 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.727+0000 7f1d46d7d700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f1d4012c320 0x7f1d40134220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:58516/0 (socket says 172.21.15.102:58516) 2021-05-25T11:59:43.744 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.727+0000 7f1d46d7d700 1 -- 172.21.15.102:0/1738651076 learned_addr learned my addr 172.21.15.102:0/1738651076 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T11:59:43.744 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.727+0000 7f1d4657c700 1 --1- 172.21.15.102:0/1738651076 >> v1:172.21.15.102:6790/0 conn(0x7f1d40098330 0x7f1d4012b760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:53394/0 (socket says 172.21.15.102:53394) 2021-05-25T11:59:43.745 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.727+0000 7f1d4557a700 1 -- 172.21.15.102:0/1738651076 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2012508456 0 0) 0x7f1d40064b90 con 0x7f1d4012c320 2021-05-25T11:59:43.745 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.727+0000 7f1d4557a700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:43.745 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.727+0000 7f1d4557a700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:43.745 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.727+0000 7f1d4557a700 10 monclient(hunting): my global_id is 4164 2021-05-25T11:59:43.745 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.727+0000 7f1d4557a700 1 -- 172.21.15.102:0/1738651076 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d2c003270 con 0x7f1d4012c320 2021-05-25T11:59:43.746 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.727+0000 7f1d4557a700 1 -- 172.21.15.102:0/1738651076 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4004631247 0 0) 0x7f1d4009af80 con 0x7f1d40098330 2021-05-25T11:59:43.746 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.727+0000 7f1d4557a700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:43.746 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.727+0000 7f1d4557a700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:43.747 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.727+0000 7f1d4557a700 10 monclient(hunting): my global_id is 4163 2021-05-25T11:59:43.747 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.727+0000 7f1d4557a700 1 -- 172.21.15.102:0/1738651076 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d40064b90 con 0x7f1d40098330 2021-05-25T11:59:43.747 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.727+0000 7f1d4557a700 1 -- 172.21.15.102:0/1738651076 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2156385170 0 0) 0x7f1d40090b50 con 0x7f1d4012be70 2021-05-25T11:59:43.748 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.727+0000 7f1d4557a700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:43.748 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.727+0000 7f1d4557a700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:43.749 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.727+0000 7f1d4557a700 10 monclient(hunting): my global_id is 4165 2021-05-25T11:59:43.749 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.728+0000 7f1d4557a700 1 -- 172.21.15.102:0/1738651076 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d4009af80 con 0x7f1d4012be70 2021-05-25T11:59:43.749 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.728+0000 7f1d4557a700 1 -- 172.21.15.102:0/1738651076 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 616217004 0 0) 0x7f1d2c003270 con 0x7f1d4012c320 2021-05-25T11:59:43.750 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.728+0000 7f1d4557a700 1 -- 172.21.15.102:0/1738651076 >> v1:172.21.15.102:6790/0 conn(0x7f1d40098330 legacy=0x7f1d4012b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:43.750 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.728+0000 7f1d4557a700 1 -- 172.21.15.102:0/1738651076 >> v1:172.21.15.181:6789/0 conn(0x7f1d4012be70 legacy=0x7f1d40130af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:43.751 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.728+0000 7f1d4557a700 10 monclient: _finish_hunting 0 2021-05-25T11:59:43.751 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.728+0000 7f1d4557a700 1 monclient: found mon.noname-a 2021-05-25T11:59:43.752 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.728+0000 7f1d4557a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T11:59:43.752 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.728+0000 7f1d4557a700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T11:59:43.752 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.728+0000 7f1d4557a700 1 -- 172.21.15.102:0/1738651076 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d4009b220 con 0x7f1d4012c320 2021-05-25T11:59:43.753 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.728+0000 7f1d4557a700 10 monclient: _finish_auth 0 2021-05-25T11:59:43.753 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.728+0000 7f1d4557a700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T11:59:43.754 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.728+0000 7f1d487e0700 5 monclient: authenticate success, global_id 4164 2021-05-25T11:59:43.754 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.728+0000 7f1d487e0700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T11:59:43.754 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.728+0000 7f1d4557a700 1 -- 172.21.15.102:0/1738651076 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f1d3c001990 con 0x7f1d4012c320 2021-05-25T11:59:43.755 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.728+0000 7f1d4557a700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:43.755 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.728+0000 7f1d4557a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T11:59:43.756 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.728+0000 7f1d4557a700 10 monclient: dump: 2021-05-25T11:59:43.756 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2021-05-25T11:59:43.756 INFO:teuthology.misc.health.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:43.757 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:43.757 INFO:teuthology.misc.health.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:43.758 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:43.758 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:43.758 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:43.759 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:43.759 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:43.760 INFO:teuthology.misc.health.smithi102.stderr: 2021-05-25T11:59:43.760 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.728+0000 7f1d487e0700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T11:59:43.761 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.728+0000 7f1d4557a700 1 -- 172.21.15.102:0/1738651076 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1d3c001b80 con 0x7f1d4012c320 2021-05-25T11:59:43.761 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.728+0000 7f1d4557a700 10 monclient: handle_config config(0 keys) v1 2021-05-25T11:59:43.761 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.728+0000 7f1d4557a700 1 -- 172.21.15.102:0/1738651076 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f1d3c001f10 con 0x7f1d4012c320 2021-05-25T11:59:43.762 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.728+0000 7f1d487e0700 10 monclient: get_monmap_and_config success 2021-05-25T11:59:43.762 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.728+0000 7f1d487e0700 4 set_mon_vals no callback set 2021-05-25T11:59:43.762 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.728+0000 7f1d4557a700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:43.763 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.728+0000 7f1d4557a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T11:59:43.763 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.728+0000 7f1d4557a700 10 monclient: dump: 2021-05-25T11:59:43.763 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2021-05-25T11:59:43.764 INFO:teuthology.misc.health.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:43.764 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:43.764 INFO:teuthology.misc.health.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:43.765 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:43.765 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:43.765 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:43.766 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:43.766 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:43.766 INFO:teuthology.misc.health.smithi102.stderr: 2021-05-25T11:59:43.766 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.728+0000 7f1d487e0700 10 monclient: shutdown 2021-05-25T11:59:43.767 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.728+0000 7f1d487e0700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T11:59:43.767 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.728+0000 7f1d487e0700 1 -- 172.21.15.102:0/1738651076 >> v1:172.21.15.102:6789/0 conn(0x7f1d4012c320 legacy=0x7f1d40134220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:43.767 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.728+0000 7f1d487e0700 1 -- 172.21.15.102:0/1738651076 shutdown_connections 2021-05-25T11:59:43.767 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.728+0000 7f1d487e0700 1 -- 172.21.15.102:0/1738651076 >> 172.21.15.102:0/1738651076 conn(0x7f1d4009b800 msgr2=0x7f1d4009bc10 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T11:59:43.767 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.728+0000 7f1d487e0700 1 -- 172.21.15.102:0/1738651076 shutdown_connections 2021-05-25T11:59:43.768 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.728+0000 7f1d487e0700 1 -- 172.21.15.102:0/1738651076 wait complete. 2021-05-25T11:59:43.768 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.729+0000 7f1d487e0700 10 monclient: build_initial_monmap 2021-05-25T11:59:43.768 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.729+0000 7f1d487e0700 10 monclient: monmap: 2021-05-25T11:59:43.768 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2021-05-25T11:59:43.768 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T11:59:43.769 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2021-05-25T11:59:43.769 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2021-05-25T11:59:43.769 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T11:59:43.769 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2021-05-25T11:59:43.769 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T11:59:43.770 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T11:59:43.770 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T11:59:43.770 INFO:teuthology.misc.health.smithi102.stderr: 2021-05-25T11:59:43.771 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.729+0000 7f1d487e0700 1 librados: starting msgr at 2021-05-25T11:59:43.771 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.729+0000 7f1d487e0700 1 librados: starting objecter 2021-05-25T11:59:43.771 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.729+0000 7f1d487e0700 1 Processor -- start 2021-05-25T11:59:43.772 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.729+0000 7f1d487e0700 1 -- start start 2021-05-25T11:59:43.772 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.729+0000 7f1d487e0700 1 librados: setting wanted keys 2021-05-25T11:59:43.772 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.729+0000 7f1d487e0700 1 librados: calling monclient init 2021-05-25T11:59:43.773 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.729+0000 7f1d487e0700 10 monclient: init 2021-05-25T11:59:43.773 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.730+0000 7f1d487e0700 10 monclient: _reopen_session rank -1 2021-05-25T11:59:43.773 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.730+0000 7f1d487e0700 10 monclient: _add_conns ranks=[1,0,2] 2021-05-25T11:59:43.774 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.730+0000 7f1d487e0700 10 monclient(hunting): picked mon.noname-b con 0x7f1d40098330 addr v1:172.21.15.181:6789/0 2021-05-25T11:59:43.774 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.730+0000 7f1d487e0700 10 monclient(hunting): picked mon.noname-a con 0x7f1d4012be70 addr v1:172.21.15.102:6789/0 2021-05-25T11:59:43.774 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.730+0000 7f1d487e0700 10 monclient(hunting): picked mon.noname-c con 0x7f1d4012c320 addr v1:172.21.15.102:6790/0 2021-05-25T11:59:43.775 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.730+0000 7f1d487e0700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d40115980 con 0x7f1d4012be70 2021-05-25T11:59:43.775 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.730+0000 7f1d487e0700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d40115c20 con 0x7f1d40098330 2021-05-25T11:59:43.775 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.730+0000 7f1d487e0700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d401151a0 con 0x7f1d4012c320 2021-05-25T11:59:43.776 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.730+0000 7f1d487e0700 10 monclient(hunting): _renew_subs 2021-05-25T11:59:43.776 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.730+0000 7f1d487e0700 10 monclient(hunting): authenticate will time out at 784.327286s 2021-05-25T11:59:43.777 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.730+0000 7f1d45d7b700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f1d4012be70 0x7f1d40140020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:58522/0 (socket says 172.21.15.102:58522) 2021-05-25T11:59:43.777 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.730+0000 7f1d45d7b700 1 -- 172.21.15.102:0/1953497115 learned_addr learned my addr 172.21.15.102:0/1953497115 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T11:59:43.777 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.730+0000 7f1d367fc700 10 client.?.objecter ms_handle_connect 0x7f1d4012be70 2021-05-25T11:59:43.778 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.730+0000 7f1d367fc700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:43.778 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.730+0000 7f1d367fc700 10 client.?.objecter ms_handle_connect 0x7f1d4012c320 2021-05-25T11:59:43.778 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.730+0000 7f1d367fc700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:43.779 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.730+0000 7f1d367fc700 1 -- 172.21.15.102:0/1953497115 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1095742529 0 0) 0x7f1d40115980 con 0x7f1d4012be70 2021-05-25T11:59:43.779 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.730+0000 7f1d367fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:43.779 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.730+0000 7f1d367fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:43.780 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.730+0000 7f1d367fc700 10 monclient(hunting): my global_id is 4167 2021-05-25T11:59:43.780 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.730+0000 7f1d367fc700 1 -- 172.21.15.102:0/1953497115 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d20004c10 con 0x7f1d4012be70 2021-05-25T11:59:43.780 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.730+0000 7f1d367fc700 10 client.?.objecter ms_handle_connect 0x7f1d40098330 2021-05-25T11:59:43.781 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.730+0000 7f1d367fc700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:43.781 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.730+0000 7f1d367fc700 1 -- 172.21.15.102:0/1953497115 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2497679710 0 0) 0x7f1d401151a0 con 0x7f1d4012c320 2021-05-25T11:59:43.781 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.730+0000 7f1d367fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:43.782 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.730+0000 7f1d367fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:43.782 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.730+0000 7f1d367fc700 10 monclient(hunting): my global_id is 4166 2021-05-25T11:59:43.783 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.730+0000 7f1d367fc700 1 -- 172.21.15.102:0/1953497115 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d40115980 con 0x7f1d4012c320 2021-05-25T11:59:43.783 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.730+0000 7f1d367fc700 1 -- 172.21.15.102:0/1953497115 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4177462568 0 0) 0x7f1d40115c20 con 0x7f1d40098330 2021-05-25T11:59:43.783 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.730+0000 7f1d367fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:43.783 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.730+0000 7f1d367fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:43.784 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.730+0000 7f1d367fc700 10 monclient(hunting): my global_id is 4168 2021-05-25T11:59:43.784 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.730+0000 7f1d367fc700 1 -- 172.21.15.102:0/1953497115 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d401151a0 con 0x7f1d40098330 2021-05-25T11:59:43.784 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.730+0000 7f1d367fc700 1 -- 172.21.15.102:0/1953497115 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3805941364 0 0) 0x7f1d20004c10 con 0x7f1d4012be70 2021-05-25T11:59:43.784 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.730+0000 7f1d367fc700 1 -- 172.21.15.102:0/1953497115 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1d40115c20 con 0x7f1d4012be70 2021-05-25T11:59:43.785 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.730+0000 7f1d367fc700 1 -- 172.21.15.102:0/1953497115 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f1d28001980 con 0x7f1d4012be70 2021-05-25T11:59:43.785 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.730+0000 7f1d367fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:43.785 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.730+0000 7f1d367fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T11:59:43.785 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.730+0000 7f1d367fc700 10 monclient(hunting): dump: 2021-05-25T11:59:43.785 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2021-05-25T11:59:43.786 INFO:teuthology.misc.health.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:43.786 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:43.786 INFO:teuthology.misc.health.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:43.786 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:43.786 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:43.787 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:43.787 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:43.787 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:43.788 INFO:teuthology.misc.health.smithi102.stderr: 2021-05-25T11:59:43.788 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.730+0000 7f1d367fc700 10 monclient(hunting): _finish_auth 0 2021-05-25T11:59:43.788 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.730+0000 7f1d367fc700 1 -- 172.21.15.102:0/1953497115 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2502315006 0 0) 0x7f1d40115980 con 0x7f1d4012c320 2021-05-25T11:59:43.788 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.730+0000 7f1d367fc700 1 -- 172.21.15.102:0/1953497115 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1d20004c10 con 0x7f1d4012c320 2021-05-25T11:59:43.788 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.730+0000 7f1d367fc700 1 -- 172.21.15.102:0/1953497115 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f1d3c001890 con 0x7f1d4012c320 2021-05-25T11:59:43.789 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.730+0000 7f1d367fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:43.789 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.730+0000 7f1d367fc700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2021-05-25T11:59:43.789 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.730+0000 7f1d367fc700 10 monclient(hunting): dump: 2021-05-25T11:59:43.789 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2021-05-25T11:59:43.789 INFO:teuthology.misc.health.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:43.790 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:43.790 INFO:teuthology.misc.health.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:43.790 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:43.790 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:43.791 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:43.791 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:43.791 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:43.791 INFO:teuthology.misc.health.smithi102.stderr: 2021-05-25T11:59:43.791 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.730+0000 7f1d367fc700 1 -- 172.21.15.102:0/1953497115 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1827419141 0 0) 0x7f1d401151a0 con 0x7f1d40098330 2021-05-25T11:59:43.792 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.731+0000 7f1d367fc700 1 -- 172.21.15.102:0/1953497115 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1d40115980 con 0x7f1d40098330 2021-05-25T11:59:43.792 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.731+0000 7f1d367fc700 1 -- 172.21.15.102:0/1953497115 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f1d300019d0 con 0x7f1d40098330 2021-05-25T11:59:43.792 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.731+0000 7f1d367fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:43.792 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.731+0000 7f1d367fc700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2021-05-25T11:59:43.792 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.731+0000 7f1d367fc700 10 monclient(hunting): dump: 2021-05-25T11:59:43.793 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2021-05-25T11:59:43.793 INFO:teuthology.misc.health.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:43.793 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:43.793 INFO:teuthology.misc.health.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:43.794 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:43.794 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:43.794 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:43.794 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:43.794 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:43.795 INFO:teuthology.misc.health.smithi102.stderr: 2021-05-25T11:59:43.795 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.731+0000 7f1d367fc700 1 -- 172.21.15.102:0/1953497115 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2479348361 0 0) 0x7f1d40115c20 con 0x7f1d4012be70 2021-05-25T11:59:43.795 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.731+0000 7f1d367fc700 1 -- 172.21.15.102:0/1953497115 >> v1:172.21.15.102:6790/0 conn(0x7f1d4012c320 legacy=0x7f1d40143750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:43.795 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.731+0000 7f1d367fc700 1 -- 172.21.15.102:0/1953497115 >> v1:172.21.15.181:6789/0 conn(0x7f1d40098330 legacy=0x7f1d401281e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:43.796 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.731+0000 7f1d367fc700 10 monclient: _finish_hunting 0 2021-05-25T11:59:43.796 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.731+0000 7f1d367fc700 1 monclient: found mon.a 2021-05-25T11:59:43.796 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.731+0000 7f1d367fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T11:59:43.796 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.731+0000 7f1d367fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:43.796 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.731+0000 7f1d367fc700 1 -- 172.21.15.102:0/1953497115 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d40097470 con 0x7f1d4012be70 2021-05-25T11:59:43.797 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.731+0000 7f1d367fc700 10 monclient: _finish_auth 0 2021-05-25T11:59:43.797 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.731+0000 7f1d367fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T11:59:43.797 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.731+0000 7f1d487e0700 5 monclient: authenticate success, global_id 4167 2021-05-25T11:59:43.797 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.731+0000 7f1d487e0700 10 monclient: _renew_subs 2021-05-25T11:59:43.797 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.731+0000 7f1d487e0700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:43.798 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.731+0000 7f1d487e0700 1 -- 172.21.15.102:0/1953497115 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d40096990 con 0x7f1d4012be70 2021-05-25T11:59:43.798 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.731+0000 7f1d487e0700 10 client.4167.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T11:59:43.798 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.731+0000 7f1d487e0700 10 monclient: _renew_subs 2021-05-25T11:59:43.798 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.731+0000 7f1d487e0700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:43.798 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.731+0000 7f1d487e0700 1 -- 172.21.15.102:0/1953497115 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d40130350 con 0x7f1d4012be70 2021-05-25T11:59:43.799 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.731+0000 7f1d487e0700 1 librados: init done 2021-05-25T11:59:43.799 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.731+0000 7f1d367fc700 1 -- 172.21.15.102:0/1953497115 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1d28001c60 con 0x7f1d4012be70 2021-05-25T11:59:43.799 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.731+0000 7f1d367fc700 10 monclient: handle_config config(0 keys) v1 2021-05-25T11:59:43.799 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.731+0000 7f1d367fc700 1 -- 172.21.15.102:0/1953497115 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f1d28001f70 con 0x7f1d4012be70 2021-05-25T11:59:43.800 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.731+0000 7f1d367fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:43.800 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.731+0000 7f1d367fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T11:59:43.800 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.731+0000 7f1d44d79700 4 set_mon_vals no callback set 2021-05-25T11:59:43.800 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.731+0000 7f1d367fc700 10 monclient: dump: 2021-05-25T11:59:43.800 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2021-05-25T11:59:43.801 INFO:teuthology.misc.health.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:43.801 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:43.801 INFO:teuthology.misc.health.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:43.801 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:43.801 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:43.802 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:43.802 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:43.802 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:43.802 INFO:teuthology.misc.health.smithi102.stderr: 2021-05-25T11:59:43.802 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.731+0000 7f1d367fc700 1 -- 172.21.15.102:0/1953497115 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 1) v1 ==== 623+0+0 (unknown 1065898531 0 0) 0x7f1d280023b0 con 0x7f1d4012be70 2021-05-25T11:59:43.803 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.731+0000 7f1d367fc700 1 -- 172.21.15.102:0/1953497115 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1444+0+0 (unknown 2753780754 0 0) 0x7f1d28006190 con 0x7f1d4012be70 2021-05-25T11:59:43.803 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.731+0000 7f1d367fc700 10 client.4167.objecter ms_dispatch 0x7f1d40001550 osd_map(8..8 src has 1..8) v4 2021-05-25T11:59:43.803 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.731+0000 7f1d367fc700 3 client.4167.objecter handle_osd_map got epochs [8,8] > 0 2021-05-25T11:59:43.803 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.731+0000 7f1d367fc700 3 client.4167.objecter handle_osd_map decoding full epoch 8 2021-05-25T11:59:43.803 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.731+0000 7f1d367fc700 20 client.4167.objecter dump_active .. 0 homeless 2021-05-25T11:59:43.804 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.731+0000 7f1d487e0700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:43.804 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.731+0000 7f1d487e0700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:43.804 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.731+0000 7f1d487e0700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:43.804 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.731+0000 7f1d487e0700 1 -- 172.21.15.102:0/1953497115 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d24004db0 con 0x7f1d4012be70 2021-05-25T11:59:43.804 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.734+0000 7f1d367fc700 1 -- 172.21.15.102:0/1953497115 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72237 (unknown 1092875540 0 1415944542) 0x7f1d28005620 con 0x7f1d4012be70 2021-05-25T11:59:43.805 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.734+0000 7f1d367fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:43.805 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.734+0000 7f1d367fc700 10 monclient: _finish_command 1 = system:0 2021-05-25T11:59:43.806 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.796+0000 7f1d487e0700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2021-05-25T11:59:43.806 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.796+0000 7f1d487e0700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2021-05-25T11:59:43.806 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.796+0000 7f1d487e0700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:43.807 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.796+0000 7f1d487e0700 1 -- 172.21.15.102:0/1953497115 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1d24005060 con 0x7f1d4012be70 2021-05-25T11:59:43.807 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.797+0000 7f1d367fc700 1 -- 172.21.15.102:0/1953497115 <== mon.0 v1:172.21.15.102:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) v1 ==== 74+0+5570 (unknown 150273500 0 3572596026) 0x7f1d28002980 con 0x7f1d4012be70 2021-05-25T11:59:43.807 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.797+0000 7f1d367fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2021-05-25T11:59:43.807 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.797+0000 7f1d367fc700 10 monclient: _finish_command 2 = system:0 2021-05-25T11:59:43.808 INFO:teuthology.misc.health.smithi102.stdout: 2021-05-25T11:59:43.808 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":8,"fsid":"20ee91b6-8935-4b98-b868-85c03789ccc4","created":"2021-05-25T11:59:41.238908+0000","modified":"2021-05-25T11:59:42.972556+0000","last_up_change":"0.000000","last_in_change":"2021-05-25T11:59:42.972556+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"8f1f285f-7041-4c53-ab09-f638ab670036","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d0d92491-3d97-4e4e-acf4-9dc7c0e4701f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"80a3eb90-b36e-4035-af91-3917fbf8dc2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45889d6a-dcfb-4670-afef-cee9bbb0476a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"69e097b7-1472-442f-af42-225c0e40ad5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f1045691-db58-471e-943a-e4d99129a917","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-05-25T11:59:43.810 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.797+0000 7f1d487e0700 10 librados: watch_flush enter 2021-05-25T11:59:43.810 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.797+0000 7f1d487e0700 10 librados: watch_flush exit 2021-05-25T11:59:43.810 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.797+0000 7f1d487e0700 20 client.4167.objecter shutdown clearing up homeless session... 2021-05-25T11:59:43.811 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.797+0000 7f1d487e0700 10 client.4167.objecter successfully canceled tick 2021-05-25T11:59:43.811 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.797+0000 7f1d487e0700 10 monclient: shutdown 2021-05-25T11:59:43.811 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.797+0000 7f1d487e0700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T11:59:43.811 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.797+0000 7f1d487e0700 1 -- 172.21.15.102:0/1953497115 >> v1:172.21.15.102:6789/0 conn(0x7f1d4012be70 legacy=0x7f1d40140020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:43.812 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.797+0000 7f1d487e0700 1 -- 172.21.15.102:0/1953497115 shutdown_connections 2021-05-25T11:59:43.812 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.797+0000 7f1d487e0700 1 -- 172.21.15.102:0/1953497115 >> 172.21.15.102:0/1953497115 conn(0x7f1d4009b800 msgr2=0x7f1d4012ea90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T11:59:43.812 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.797+0000 7f1d487e0700 1 -- 172.21.15.102:0/1953497115 shutdown_connections 2021-05-25T11:59:43.812 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.797+0000 7f1d487e0700 1 -- 172.21.15.102:0/1953497115 wait complete. 2021-05-25T11:59:43.812 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:43.797+0000 7f1d487e0700 1 librados: shutdown 2021-05-25T11:59:43.813 DEBUG:teuthology.misc:0 of 6 OSDs are up 2021-05-25T11:59:47.029 INFO:tasks.ceph.osd.2.smithi102.stderr:2021-05-25T11:59:47.027+0000 7f9d6b84f0c0 -1 osd.2 0 log_to_monitors {default=true} 2021-05-25T11:59:47.042 INFO:tasks.ceph.osd.4.smithi181.stderr:2021-05-25T11:59:47.040+0000 7f003195d0c0 -1 osd.4 0 log_to_monitors {default=true} 2021-05-25T11:59:47.049 INFO:tasks.ceph.osd.3.smithi181.stderr:2021-05-25T11:59:47.047+0000 7f150db600c0 -1 osd.3 0 log_to_monitors {default=true} 2021-05-25T11:59:47.056 INFO:tasks.ceph.osd.1.smithi102.stderr:2021-05-25T11:59:47.055+0000 7f2deb2ad0c0 -1 osd.1 0 log_to_monitors {default=true} 2021-05-25T11:59:47.075 INFO:tasks.ceph.osd.5.smithi181.stderr:2021-05-25T11:59:47.072+0000 7f049afb30c0 -1 osd.5 0 log_to_monitors {default=true} 2021-05-25T11:59:47.083 INFO:tasks.ceph.osd.0.smithi102.stderr:2021-05-25T11:59:47.082+0000 7f39076370c0 -1 osd.0 0 log_to_monitors {default=true} 2021-05-25T11:59:48.570 INFO:tasks.ceph.osd.4.smithi181.stderr:2021-05-25T11:59:48.567+0000 7f00280e4700 -1 osd.4 0 waiting for initial osdmap 2021-05-25T11:59:48.571 INFO:tasks.ceph.osd.3.smithi181.stderr:2021-05-25T11:59:48.567+0000 7f15042e7700 -1 osd.3 0 waiting for initial osdmap 2021-05-25T11:59:48.571 INFO:tasks.ceph.osd.5.smithi181.stderr:2021-05-25T11:59:48.568+0000 7f049173a700 -1 osd.5 0 waiting for initial osdmap 2021-05-25T11:59:48.572 INFO:tasks.ceph.osd.2.smithi102.stderr:2021-05-25T11:59:48.570+0000 7f9d61fd6700 -1 osd.2 0 waiting for initial osdmap 2021-05-25T11:59:48.573 INFO:tasks.ceph.osd.1.smithi102.stderr:2021-05-25T11:59:48.570+0000 7f2de1a34700 -1 osd.1 0 waiting for initial osdmap 2021-05-25T11:59:48.573 INFO:tasks.ceph.osd.0.smithi102.stderr:2021-05-25T11:59:48.571+0000 7f38fddbe700 -1 osd.0 0 waiting for initial osdmap 2021-05-25T11:59:48.575 INFO:tasks.ceph.osd.3.smithi181.stderr:2021-05-25T11:59:48.572+0000 7f14fec8d700 -1 osd.3 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2021-05-25T11:59:48.576 INFO:tasks.ceph.osd.4.smithi181.stderr:2021-05-25T11:59:48.573+0000 7f0022a8a700 -1 osd.4 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2021-05-25T11:59:48.577 INFO:tasks.ceph.osd.0.smithi102.stderr:2021-05-25T11:59:48.575+0000 7f38f7f63700 -1 osd.0 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2021-05-25T11:59:48.577 INFO:tasks.ceph.osd.1.smithi102.stderr:2021-05-25T11:59:48.576+0000 7f2ddc3da700 -1 osd.1 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2021-05-25T11:59:48.578 INFO:tasks.ceph.osd.5.smithi181.stderr:2021-05-25T11:59:48.576+0000 7f048b8df700 -1 osd.5 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2021-05-25T11:59:48.579 INFO:tasks.ceph.osd.2.smithi102.stderr:2021-05-25T11:59:48.575+0000 7f9d5c97c700 -1 osd.2 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2021-05-25T11:59:50.417 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2021-05-25T11:59:50.526 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.525+0000 7fb8daa3d700 10 monclient: get_monmap_and_config 2021-05-25T11:59:50.527 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.525+0000 7fb8daa3d700 10 monclient: build_initial_monmap 2021-05-25T11:59:50.527 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.525+0000 7fb8daa3d700 10 monclient: monmap: 2021-05-25T11:59:50.528 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2021-05-25T11:59:50.528 INFO:teuthology.misc.health.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:50.528 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2021-05-25T11:59:50.526260+0000 2021-05-25T11:59:50.528 INFO:teuthology.misc.health.smithi102.stderr:created 2021-05-25T11:59:50.526260+0000 2021-05-25T11:59:50.529 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T11:59:50.529 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:50.529 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T11:59:50.529 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T11:59:50.529 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T11:59:50.530 INFO:teuthology.misc.health.smithi102.stderr: 2021-05-25T11:59:50.540 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.526+0000 7fb8daa3d700 1 Processor -- start 2021-05-25T11:59:50.540 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.526+0000 7fb8daa3d700 1 -- start start 2021-05-25T11:59:50.541 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.526+0000 7fb8daa3d700 10 monclient: init 2021-05-25T11:59:50.541 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.526+0000 7fb8daa3d700 10 monclient: _reopen_session rank -1 2021-05-25T11:59:50.541 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.526+0000 7fb8daa3d700 10 monclient: _add_conns ranks=[1,0,2] 2021-05-25T11:59:50.541 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.526+0000 7fb8daa3d700 10 monclient(hunting): picked mon.noname-c con 0x7fb8d4094820 addr v1:172.21.15.181:6789/0 2021-05-25T11:59:50.541 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.526+0000 7fb8daa3d700 10 monclient(hunting): picked mon.noname-a con 0x7fb8d4091bd0 addr v1:172.21.15.102:6789/0 2021-05-25T11:59:50.542 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.526+0000 7fb8daa3d700 10 monclient(hunting): picked mon.noname-b con 0x7fb8d412e290 addr v1:172.21.15.102:6790/0 2021-05-25T11:59:50.542 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.526+0000 7fb8daa3d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8d4128d70 con 0x7fb8d4091bd0 2021-05-25T11:59:50.542 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.526+0000 7fb8daa3d700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8d40928c0 con 0x7fb8d4094820 2021-05-25T11:59:50.542 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.526+0000 7fb8daa3d700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8d4092b60 con 0x7fb8d412e290 2021-05-25T11:59:50.542 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.526+0000 7fb8daa3d700 10 monclient(hunting): _renew_subs 2021-05-25T11:59:50.543 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.526+0000 7fb8daa3d700 10 monclient(hunting): authenticate will time out at 791.123602s 2021-05-25T11:59:50.543 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.526+0000 7fb8d8fda700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fb8d412e290 0x7fb8d4130680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:53496/0 (socket says 172.21.15.102:53496) 2021-05-25T11:59:50.543 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.526+0000 7fb8d8fda700 1 -- 172.21.15.102:0/39973982 learned_addr learned my addr 172.21.15.102:0/39973982 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T11:59:50.543 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.526+0000 7fb8d37fe700 1 --1- 172.21.15.102:0/39973982 >> v1:172.21.15.102:6789/0 conn(0x7fb8d4091bd0 0x7fb8d4128660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:58618/0 (socket says 172.21.15.102:58618) 2021-05-25T11:59:50.543 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.526+0000 7fb8d2ffd700 1 -- 172.21.15.102:0/39973982 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 699178808 0 0) 0x7fb8d4128d70 con 0x7fb8d4091bd0 2021-05-25T11:59:50.544 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.526+0000 7fb8d2ffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:50.544 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.526+0000 7fb8d2ffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:50.544 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.526+0000 7fb8d2ffd700 10 monclient(hunting): my global_id is 4206 2021-05-25T11:59:50.544 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.527+0000 7fb8d2ffd700 1 -- 172.21.15.102:0/39973982 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8bc003270 con 0x7fb8d4091bd0 2021-05-25T11:59:50.545 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.527+0000 7fb8d2ffd700 1 -- 172.21.15.102:0/39973982 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2132147030 0 0) 0x7fb8d4092b60 con 0x7fb8d412e290 2021-05-25T11:59:50.545 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.527+0000 7fb8d2ffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:50.545 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.527+0000 7fb8d2ffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:50.545 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.527+0000 7fb8d2ffd700 10 monclient(hunting): my global_id is 4199 2021-05-25T11:59:50.545 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.527+0000 7fb8d2ffd700 1 -- 172.21.15.102:0/39973982 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8d4128d70 con 0x7fb8d412e290 2021-05-25T11:59:50.545 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.527+0000 7fb8d2ffd700 1 -- 172.21.15.102:0/39973982 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1291540170 0 0) 0x7fb8bc003270 con 0x7fb8d4091bd0 2021-05-25T11:59:50.546 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.527+0000 7fb8d2ffd700 1 -- 172.21.15.102:0/39973982 >> v1:172.21.15.102:6790/0 conn(0x7fb8d412e290 legacy=0x7fb8d4130680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:50.546 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.527+0000 7fb8d2ffd700 1 -- 172.21.15.102:0/39973982 >> v1:172.21.15.181:6789/0 conn(0x7fb8d4094820 legacy=0x7fb8d4127f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:50.546 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.527+0000 7fb8d2ffd700 10 monclient: _finish_hunting 0 2021-05-25T11:59:50.546 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.527+0000 7fb8d2ffd700 1 monclient: found mon.noname-a 2021-05-25T11:59:50.547 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.527+0000 7fb8d2ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T11:59:50.547 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.527+0000 7fb8d2ffd700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T11:59:50.547 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.527+0000 7fb8d2ffd700 1 -- 172.21.15.102:0/39973982 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8d4092e00 con 0x7fb8d4091bd0 2021-05-25T11:59:50.547 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.527+0000 7fb8d2ffd700 10 monclient: _finish_auth 0 2021-05-25T11:59:50.548 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.527+0000 7fb8d2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T11:59:50.548 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.527+0000 7fb8daa3d700 5 monclient: authenticate success, global_id 4206 2021-05-25T11:59:50.549 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.527+0000 7fb8d2ffd700 1 -- 172.21.15.102:0/39973982 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fb8c0001610 con 0x7fb8d4091bd0 2021-05-25T11:59:50.549 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.527+0000 7fb8d2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:50.549 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.527+0000 7fb8d2ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T11:59:50.549 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.527+0000 7fb8d2ffd700 10 monclient: dump: 2021-05-25T11:59:50.549 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2021-05-25T11:59:50.550 INFO:teuthology.misc.health.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:50.550 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:50.550 INFO:teuthology.misc.health.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:50.550 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:50.551 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:50.551 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:50.551 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:50.551 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:50.552 INFO:teuthology.misc.health.smithi102.stderr: 2021-05-25T11:59:50.552 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.527+0000 7fb8daa3d700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T11:59:50.552 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.527+0000 7fb8d2ffd700 1 -- 172.21.15.102:0/39973982 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb8c0001880 con 0x7fb8d4091bd0 2021-05-25T11:59:50.552 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.527+0000 7fb8d2ffd700 10 monclient: handle_config config(0 keys) v1 2021-05-25T11:59:50.552 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.527+0000 7fb8d2ffd700 1 -- 172.21.15.102:0/39973982 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fb8c0001c10 con 0x7fb8d4091bd0 2021-05-25T11:59:50.553 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.527+0000 7fb8daa3d700 10 monclient: get_monmap_and_config success 2021-05-25T11:59:50.553 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.527+0000 7fb8daa3d700 4 set_mon_vals no callback set 2021-05-25T11:59:50.553 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.527+0000 7fb8d2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:50.553 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.527+0000 7fb8d2ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T11:59:50.553 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.527+0000 7fb8d2ffd700 10 monclient: dump: 2021-05-25T11:59:50.554 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2021-05-25T11:59:50.554 INFO:teuthology.misc.health.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:50.554 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:50.554 INFO:teuthology.misc.health.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:50.555 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:50.555 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:50.555 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:50.555 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:50.555 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:50.556 INFO:teuthology.misc.health.smithi102.stderr: 2021-05-25T11:59:50.556 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.527+0000 7fb8daa3d700 10 monclient: shutdown 2021-05-25T11:59:50.556 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.527+0000 7fb8daa3d700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T11:59:50.556 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.527+0000 7fb8daa3d700 1 -- 172.21.15.102:0/39973982 >> v1:172.21.15.102:6789/0 conn(0x7fb8d4091bd0 legacy=0x7fb8d4128660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:50.557 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.527+0000 7fb8daa3d700 1 -- 172.21.15.102:0/39973982 shutdown_connections 2021-05-25T11:59:50.557 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.527+0000 7fb8daa3d700 1 -- 172.21.15.102:0/39973982 >> 172.21.15.102:0/39973982 conn(0x7fb8d40971b0 msgr2=0x7fb8d40975c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T11:59:50.557 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.527+0000 7fb8daa3d700 1 -- 172.21.15.102:0/39973982 shutdown_connections 2021-05-25T11:59:50.557 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.527+0000 7fb8daa3d700 1 -- 172.21.15.102:0/39973982 wait complete. 2021-05-25T11:59:50.557 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.528+0000 7fb8daa3d700 10 monclient: build_initial_monmap 2021-05-25T11:59:50.558 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.528+0000 7fb8daa3d700 10 monclient: monmap: 2021-05-25T11:59:50.558 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2021-05-25T11:59:50.558 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T11:59:50.558 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2021-05-25T11:59:50.558 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2021-05-25T11:59:50.559 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T11:59:50.559 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2021-05-25T11:59:50.559 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T11:59:50.559 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T11:59:50.559 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T11:59:50.560 INFO:teuthology.misc.health.smithi102.stderr: 2021-05-25T11:59:50.560 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.528+0000 7fb8daa3d700 1 librados: starting msgr at 2021-05-25T11:59:50.560 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.528+0000 7fb8daa3d700 1 librados: starting objecter 2021-05-25T11:59:50.560 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.528+0000 7fb8daa3d700 1 Processor -- start 2021-05-25T11:59:50.560 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.529+0000 7fb8daa3d700 1 -- start start 2021-05-25T11:59:50.561 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.529+0000 7fb8daa3d700 1 librados: setting wanted keys 2021-05-25T11:59:50.561 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.529+0000 7fb8daa3d700 1 librados: calling monclient init 2021-05-25T11:59:50.561 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.529+0000 7fb8daa3d700 10 monclient: init 2021-05-25T11:59:50.561 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.529+0000 7fb8daa3d700 10 monclient: _reopen_session rank -1 2021-05-25T11:59:50.562 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.529+0000 7fb8daa3d700 10 monclient: _add_conns ranks=[0,2,1] 2021-05-25T11:59:50.562 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.529+0000 7fb8daa3d700 10 monclient(hunting): picked mon.noname-a con 0x7fb8d4091bd0 addr v1:172.21.15.102:6789/0 2021-05-25T11:59:50.562 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.529+0000 7fb8daa3d700 10 monclient(hunting): picked mon.noname-c con 0x7fb8d4094820 addr v1:172.21.15.102:6790/0 2021-05-25T11:59:50.562 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.529+0000 7fb8daa3d700 10 monclient(hunting): picked mon.noname-b con 0x7fb8d412e290 addr v1:172.21.15.181:6789/0 2021-05-25T11:59:50.562 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.529+0000 7fb8daa3d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8d4117220 con 0x7fb8d4091bd0 2021-05-25T11:59:50.563 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.529+0000 7fb8daa3d700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8d4119be0 con 0x7fb8d412e290 2021-05-25T11:59:50.563 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.529+0000 7fb8daa3d700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8d4119e80 con 0x7fb8d4094820 2021-05-25T11:59:50.563 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.529+0000 7fb8daa3d700 10 monclient(hunting): _renew_subs 2021-05-25T11:59:50.563 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.529+0000 7fb8daa3d700 10 monclient(hunting): authenticate will time out at 791.126482s 2021-05-25T11:59:50.563 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.529+0000 7fb8d3fff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fb8d4091bd0 0x7fb8d4116400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:58622/0 (socket says 172.21.15.102:58622) 2021-05-25T11:59:50.564 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.529+0000 7fb8d3fff700 1 -- 172.21.15.102:0/4230658653 learned_addr learned my addr 172.21.15.102:0/4230658653 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T11:59:50.564 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.529+0000 7fb8d37fe700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fb8d4094820 0x7fb8d4116b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:53502/0 (socket says 172.21.15.102:53502) 2021-05-25T11:59:50.564 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.529+0000 7fb8c7fff700 10 client.?.objecter ms_handle_connect 0x7fb8d4094820 2021-05-25T11:59:50.564 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.529+0000 7fb8c7fff700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:50.565 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.529+0000 7fb8c7fff700 10 client.?.objecter ms_handle_connect 0x7fb8d4091bd0 2021-05-25T11:59:50.565 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.529+0000 7fb8c7fff700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:50.565 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.529+0000 7fb8c7fff700 1 -- 172.21.15.102:0/4230658653 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3042497068 0 0) 0x7fb8d4117220 con 0x7fb8d4091bd0 2021-05-25T11:59:50.565 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.529+0000 7fb8c7fff700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:50.565 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.529+0000 7fb8c7fff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:50.566 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.529+0000 7fb8c7fff700 10 monclient(hunting): my global_id is 4209 2021-05-25T11:59:50.566 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.529+0000 7fb8c7fff700 1 -- 172.21.15.102:0/4230658653 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8b0004c10 con 0x7fb8d4091bd0 2021-05-25T11:59:50.566 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.529+0000 7fb8c7fff700 10 client.?.objecter ms_handle_connect 0x7fb8d412e290 2021-05-25T11:59:50.566 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.529+0000 7fb8c7fff700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:50.566 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.529+0000 7fb8c7fff700 1 -- 172.21.15.102:0/4230658653 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4161475853 0 0) 0x7fb8d4119e80 con 0x7fb8d4094820 2021-05-25T11:59:50.567 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.529+0000 7fb8c7fff700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:50.567 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.529+0000 7fb8c7fff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:50.567 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.529+0000 7fb8c7fff700 10 monclient(hunting): my global_id is 4202 2021-05-25T11:59:50.567 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.529+0000 7fb8c7fff700 1 -- 172.21.15.102:0/4230658653 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8d4117220 con 0x7fb8d4094820 2021-05-25T11:59:50.568 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.529+0000 7fb8c7fff700 1 -- 172.21.15.102:0/4230658653 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1559261287 0 0) 0x7fb8d4119be0 con 0x7fb8d412e290 2021-05-25T11:59:50.568 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.529+0000 7fb8c7fff700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:50.568 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.529+0000 7fb8c7fff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:50.568 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.529+0000 7fb8c7fff700 10 monclient(hunting): my global_id is 4204 2021-05-25T11:59:50.568 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.529+0000 7fb8c7fff700 1 -- 172.21.15.102:0/4230658653 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8d4119e80 con 0x7fb8d412e290 2021-05-25T11:59:50.569 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.530+0000 7fb8c7fff700 1 -- 172.21.15.102:0/4230658653 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3163042581 0 0) 0x7fb8b0004c10 con 0x7fb8d4091bd0 2021-05-25T11:59:50.569 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.530+0000 7fb8c7fff700 1 -- 172.21.15.102:0/4230658653 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb8d4119be0 con 0x7fb8d4091bd0 2021-05-25T11:59:50.569 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.530+0000 7fb8c7fff700 1 -- 172.21.15.102:0/4230658653 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fb8b8001990 con 0x7fb8d4091bd0 2021-05-25T11:59:50.569 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.530+0000 7fb8c7fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:50.569 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.530+0000 7fb8c7fff700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T11:59:50.570 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.530+0000 7fb8c7fff700 10 monclient(hunting): dump: 2021-05-25T11:59:50.570 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2021-05-25T11:59:50.570 INFO:teuthology.misc.health.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:50.570 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:50.571 INFO:teuthology.misc.health.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:50.571 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:50.571 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:50.571 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:50.571 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:50.572 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:50.572 INFO:teuthology.misc.health.smithi102.stderr: 2021-05-25T11:59:50.572 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.530+0000 7fb8c7fff700 10 monclient(hunting): _finish_auth 0 2021-05-25T11:59:50.572 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.530+0000 7fb8c7fff700 1 -- 172.21.15.102:0/4230658653 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 740974469 0 0) 0x7fb8d4117220 con 0x7fb8d4094820 2021-05-25T11:59:50.572 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.530+0000 7fb8c7fff700 1 -- 172.21.15.102:0/4230658653 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb8b0004c10 con 0x7fb8d4094820 2021-05-25T11:59:50.573 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.530+0000 7fb8c7fff700 1 -- 172.21.15.102:0/4230658653 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fb8c00015d0 con 0x7fb8d4094820 2021-05-25T11:59:50.573 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.530+0000 7fb8c7fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:50.573 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.530+0000 7fb8c7fff700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2021-05-25T11:59:50.573 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.530+0000 7fb8c7fff700 10 monclient(hunting): dump: 2021-05-25T11:59:50.573 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2021-05-25T11:59:50.574 INFO:teuthology.misc.health.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:50.574 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:50.574 INFO:teuthology.misc.health.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:50.574 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:50.575 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:50.575 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:50.575 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:50.575 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:50.576 INFO:teuthology.misc.health.smithi102.stderr: 2021-05-25T11:59:50.576 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.530+0000 7fb8c7fff700 1 -- 172.21.15.102:0/4230658653 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 952853823 0 0) 0x7fb8d4119be0 con 0x7fb8d4091bd0 2021-05-25T11:59:50.576 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.530+0000 7fb8c7fff700 1 -- 172.21.15.102:0/4230658653 >> v1:172.21.15.102:6790/0 conn(0x7fb8d4094820 legacy=0x7fb8d4116b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:50.576 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.530+0000 7fb8c7fff700 1 -- 172.21.15.102:0/4230658653 >> v1:172.21.15.181:6789/0 conn(0x7fb8d412e290 legacy=0x7fb8d4143790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:50.576 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.530+0000 7fb8c7fff700 10 monclient: _finish_hunting 0 2021-05-25T11:59:50.577 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.530+0000 7fb8c7fff700 1 monclient: found mon.a 2021-05-25T11:59:50.577 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.530+0000 7fb8c7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T11:59:50.577 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.530+0000 7fb8c7fff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:50.577 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.530+0000 7fb8c7fff700 1 -- 172.21.15.102:0/4230658653 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8d411a120 con 0x7fb8d4091bd0 2021-05-25T11:59:50.578 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.530+0000 7fb8c7fff700 10 monclient: _finish_auth 0 2021-05-25T11:59:50.578 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.530+0000 7fb8c7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T11:59:50.578 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.530+0000 7fb8daa3d700 5 monclient: authenticate success, global_id 4209 2021-05-25T11:59:50.578 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.530+0000 7fb8daa3d700 10 monclient: _renew_subs 2021-05-25T11:59:50.579 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.530+0000 7fb8daa3d700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:50.579 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.530+0000 7fb8daa3d700 1 -- 172.21.15.102:0/4230658653 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8d4117fb0 con 0x7fb8d4091bd0 2021-05-25T11:59:50.579 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.530+0000 7fb8daa3d700 10 client.4209.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T11:59:50.579 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.530+0000 7fb8daa3d700 10 monclient: _renew_subs 2021-05-25T11:59:50.579 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.530+0000 7fb8daa3d700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:50.580 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.530+0000 7fb8daa3d700 1 -- 172.21.15.102:0/4230658653 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8d412bab0 con 0x7fb8d4091bd0 2021-05-25T11:59:50.580 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.530+0000 7fb8daa3d700 1 librados: init done 2021-05-25T11:59:50.580 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.530+0000 7fb8c7fff700 1 -- 172.21.15.102:0/4230658653 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb8b8001d20 con 0x7fb8d4091bd0 2021-05-25T11:59:50.580 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.530+0000 7fb8c7fff700 10 monclient: handle_config config(0 keys) v1 2021-05-25T11:59:50.581 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.530+0000 7fb8c7fff700 1 -- 172.21.15.102:0/4230658653 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fb8b8002350 con 0x7fb8d4091bd0 2021-05-25T11:59:50.581 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.530+0000 7fb8c7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:50.581 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.530+0000 7fb8d27fc700 4 set_mon_vals no callback set 2021-05-25T11:59:50.581 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.530+0000 7fb8c7fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T11:59:50.581 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.530+0000 7fb8c7fff700 10 monclient: dump: 2021-05-25T11:59:50.582 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2021-05-25T11:59:50.582 INFO:teuthology.misc.health.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:50.582 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:50.582 INFO:teuthology.misc.health.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:50.583 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:50.583 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:50.583 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:50.583 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:50.584 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:50.584 INFO:teuthology.misc.health.smithi102.stderr: 2021-05-25T11:59:50.584 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.531+0000 7fb8c7fff700 1 -- 172.21.15.102:0/4230658653 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7fb8b80024b0 con 0x7fb8d4091bd0 2021-05-25T11:59:50.584 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.531+0000 7fb8daa3d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:50.584 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.531+0000 7fb8daa3d700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:50.585 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.531+0000 7fb8daa3d700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:50.585 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.531+0000 7fb8daa3d700 1 -- 172.21.15.102:0/4230658653 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8b4004db0 con 0x7fb8d4091bd0 2021-05-25T11:59:50.585 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.531+0000 7fb8c7fff700 1 -- 172.21.15.102:0/4230658653 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 2796+0+0 (unknown 1222120465 0 0) 0x7fb8b803cbb0 con 0x7fb8d4091bd0 2021-05-25T11:59:50.585 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.531+0000 7fb8c7fff700 10 client.4209.objecter ms_dispatch 0x7fb8d4001550 osd_map(11..11 src has 1..11) v4 2021-05-25T11:59:50.585 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.531+0000 7fb8c7fff700 3 client.4209.objecter handle_osd_map got epochs [11,11] > 0 2021-05-25T11:59:50.586 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.531+0000 7fb8c7fff700 3 client.4209.objecter handle_osd_map decoding full epoch 11 2021-05-25T11:59:50.586 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.531+0000 7fb8c7fff700 20 client.4209.objecter dump_active .. 0 homeless 2021-05-25T11:59:50.586 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.531+0000 7fb8d37fe700 10 monclient: get_auth_request con 0x7fb8b0033240 auth_method 0 2021-05-25T11:59:50.586 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.531+0000 7fb8c7fff700 10 client.4209.objecter ms_handle_connect 0x7fb8b0033240 2021-05-25T11:59:50.587 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.535+0000 7fb8c7fff700 1 -- 172.21.15.102:0/4230658653 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7fb8b8012da0 con 0x7fb8d4091bd0 2021-05-25T11:59:50.587 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.535+0000 7fb8c7fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:50.587 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.535+0000 7fb8c7fff700 10 monclient: _finish_command 1 = system:0 2021-05-25T11:59:50.691 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.690+0000 7fb8daa3d700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2021-05-25T11:59:50.691 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.690+0000 7fb8daa3d700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2021-05-25T11:59:50.692 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.690+0000 7fb8daa3d700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:50.692 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.690+0000 7fb8daa3d700 1 -- 172.21.15.102:0/4230658653 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb8b4005060 con 0x7fb8d4091bd0 2021-05-25T11:59:50.693 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.690+0000 7fb8c7fff700 1 -- 172.21.15.102:0/4230658653 <== mon.0 v1:172.21.15.102:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+8496 (unknown 4250787923 0 3840188472) 0x7fb8b8012990 con 0x7fb8d4091bd0 2021-05-25T11:59:50.693 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.690+0000 7fb8c7fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2021-05-25T11:59:50.693 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.690+0000 7fb8c7fff700 10 monclient: _finish_command 2 = system:0 2021-05-25T11:59:50.694 INFO:teuthology.misc.health.smithi102.stdout: 2021-05-25T11:59:50.694 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":12,"fsid":"20ee91b6-8935-4b98-b868-85c03789ccc4","created":"2021-05-25T11:59:41.238908+0000","modified":"2021-05-25T11:59:50.578675+0000","last_up_change":"2021-05-25T11:59:49.572449+0000","last_in_change":"2021-05-25T11:59:42.972556+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-05-25T11:59:49.609415+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"8f1f285f-7041-4c53-ab09-f638ab670036","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6807","nonce":29973}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6809","nonce":29973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6811","nonce":29973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6810","nonce":29973}]},"public_addr":"172.21.15.102:6807/29973","cluster_addr":"172.21.15.102:6809/29973","heartbeat_back_addr":"172.21.15.102:6811/29973","heartbeat_front_addr":"172.21.15.102:6810/29973","state":["exists","up"]},{"osd":1,"uuid":"d0d92491-3d97-4e4e-acf4-9dc7c0e4701f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6802","nonce":29974}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6804","nonce":29974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6808","nonce":29974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6806","nonce":29974}]},"public_addr":"172.21.15.102:6802/29974","cluster_addr":"172.21.15.102:6804/29974","heartbeat_back_addr":"172.21.15.102:6808/29974","heartbeat_front_addr":"172.21.15.102:6806/29974","state":["exists","up"]},{"osd":2,"uuid":"80a3eb90-b36e-4035-af91-3917fbf8dc2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6800","nonce":29975}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6801","nonce":29975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6805","nonce":29975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6803","nonce":29975}]},"public_addr":"172.21.15.102:6800/29975","cluster_addr":"172.21.15.102:6801/29975","heartbeat_back_addr":"172.21.15.102:6805/29975","heartbeat_front_addr":"172.21.15.102:6803/29975","state":["exists","up"]},{"osd":3,"uuid":"45889d6a-dcfb-4670-afef-cee9bbb0476a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6804","nonce":30005}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6805","nonce":30005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6807","nonce":30005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6806","nonce":30005}]},"public_addr":"172.21.15.181:6804/30005","cluster_addr":"172.21.15.181:6805/30005","heartbeat_back_addr":"172.21.15.181:6807/30005","heartbeat_front_addr":"172.21.15.181:6806/30005","state":["exists","up"]},{"osd":4,"uuid":"69e097b7-1472-442f-af42-225c0e40ad5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6800","nonce":30006}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6801","nonce":30006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6803","nonce":30006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6802","nonce":30006}]},"public_addr":"172.21.15.181:6800/30006","cluster_addr":"172.21.15.181:6801/30006","heartbeat_back_addr":"172.21.15.181:6803/30006","heartbeat_front_addr":"172.21.15.181:6802/30006","state":["exists","up"]},{"osd":5,"uuid":"f1045691-db58-471e-943a-e4d99129a917","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6808","nonce":30008}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6809","nonce":30008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6811","nonce":30008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6810","nonce":30008}]},"public_addr":"172.21.15.181:6808/30008","cluster_addr":"172.21.15.181:6809/30008","heartbeat_back_addr":"172.21.15.181:6811/30008","heartbeat_front_addr":"172.21.15.181:6810/30008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-05-25T11:59:48.129055+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-05-25T11:59:48.090298+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-05-25T11:59:48.049016+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-05-25T11:59:48.069527+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-05-25T11:59:48.002930+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-05-25T11:59:48.052367+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-05-25T11:59:50.695 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.692+0000 7fb8daa3d700 10 librados: watch_flush enter 2021-05-25T11:59:50.696 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.692+0000 7fb8daa3d700 10 librados: watch_flush exit 2021-05-25T11:59:50.696 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.692+0000 7fb8daa3d700 20 client.4209.objecter shutdown clearing up homeless session... 2021-05-25T11:59:50.696 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.692+0000 7fb8daa3d700 10 client.4209.objecter successfully canceled tick 2021-05-25T11:59:50.696 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.692+0000 7fb8daa3d700 1 -- 172.21.15.102:0/4230658653 >> v1:172.21.15.181:6812/29582 conn(0x7fb8b0033240 legacy=0x7fb8b0035700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:50.697 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.692+0000 7fb8daa3d700 10 monclient: shutdown 2021-05-25T11:59:50.697 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.692+0000 7fb8daa3d700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T11:59:50.697 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.692+0000 7fb8daa3d700 1 -- 172.21.15.102:0/4230658653 >> v1:172.21.15.102:6789/0 conn(0x7fb8d4091bd0 legacy=0x7fb8d4116400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:50.697 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.692+0000 7fb8daa3d700 1 -- 172.21.15.102:0/4230658653 shutdown_connections 2021-05-25T11:59:50.698 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.692+0000 7fb8daa3d700 1 -- 172.21.15.102:0/4230658653 >> 172.21.15.102:0/4230658653 conn(0x7fb8d40971b0 msgr2=0x7fb8d4096700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T11:59:50.698 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.692+0000 7fb8daa3d700 1 -- 172.21.15.102:0/4230658653 shutdown_connections 2021-05-25T11:59:50.698 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.692+0000 7fb8daa3d700 1 -- 172.21.15.102:0/4230658653 wait complete. 2021-05-25T11:59:50.698 INFO:teuthology.misc.health.smithi102.stderr:2021-05-25T11:59:50.692+0000 7fb8daa3d700 1 librados: shutdown 2021-05-25T11:59:50.703 DEBUG:teuthology.misc:6 of 6 OSDs are up 2021-05-25T11:59:50.704 INFO:tasks.ceph:Creating RBD pool 2021-05-25T11:59:50.704 DEBUG:teuthology.orchestra.run.smithi102:> sudo ceph --cluster ceph osd pool create rbd 8 2021-05-25T11:59:50.813 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.812+0000 7fd76759e700 10 monclient: get_monmap_and_config 2021-05-25T11:59:50.813 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.812+0000 7fd76759e700 10 monclient: build_initial_monmap 2021-05-25T11:59:50.813 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.812+0000 7fd76759e700 10 monclient: monmap: 2021-05-25T11:59:50.814 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T11:59:50.814 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:50.814 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:50.813135+0000 2021-05-25T11:59:50.814 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:50.813135+0000 2021-05-25T11:59:50.815 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T11:59:50.815 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:50.815 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T11:59:50.815 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T11:59:50.816 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T11:59:50.816 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:50.825 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.812+0000 7fd76759e700 1 Processor -- start 2021-05-25T11:59:50.826 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.812+0000 7fd76759e700 1 -- start start 2021-05-25T11:59:50.826 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.812+0000 7fd76759e700 10 monclient: init 2021-05-25T11:59:50.826 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.813+0000 7fd76759e700 10 monclient: _reopen_session rank -1 2021-05-25T11:59:50.826 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.813+0000 7fd76759e700 10 monclient: _add_conns ranks=[1,0,2] 2021-05-25T11:59:50.827 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.813+0000 7fd76759e700 10 monclient(hunting): picked mon.noname-c con 0x7fd768098280 addr v1:172.21.15.181:6789/0 2021-05-25T11:59:50.827 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.813+0000 7fd76759e700 10 monclient(hunting): picked mon.noname-a con 0x7fd76812bde0 addr v1:172.21.15.102:6789/0 2021-05-25T11:59:50.827 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.813+0000 7fd76759e700 10 monclient(hunting): picked mon.noname-b con 0x7fd76812c260 addr v1:172.21.15.102:6790/0 2021-05-25T11:59:50.827 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.813+0000 7fd76759e700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd76809add0 con 0x7fd76812bde0 2021-05-25T11:59:50.827 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.813+0000 7fd76759e700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd76809b070 con 0x7fd768098280 2021-05-25T11:59:50.828 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.813+0000 7fd76759e700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd768090aa0 con 0x7fd76812c260 2021-05-25T11:59:50.828 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.813+0000 7fd76759e700 10 monclient(hunting): _renew_subs 2021-05-25T11:59:50.828 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.813+0000 7fd76759e700 10 monclient(hunting): authenticate will time out at 791.410412s 2021-05-25T11:59:50.828 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.813+0000 7fd765d9b700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fd76812bde0 0x7fd768130a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:58658/0 (socket says 172.21.15.102:58658) 2021-05-25T11:59:50.829 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.813+0000 7fd766d9d700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fd76812c260 0x7fd768134190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:53534/0 (socket says 172.21.15.102:53534) 2021-05-25T11:59:50.829 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.813+0000 7fd765d9b700 1 -- 172.21.15.102:0/3631572193 learned_addr learned my addr 172.21.15.102:0/3631572193 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T11:59:50.829 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.813+0000 7fd76559a700 1 -- 172.21.15.102:0/3631572193 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2477115994 0 0) 0x7fd76809add0 con 0x7fd76812bde0 2021-05-25T11:59:50.829 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.813+0000 7fd76559a700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:50.829 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.813+0000 7fd76559a700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:50.830 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.813+0000 7fd76559a700 10 monclient(hunting): my global_id is 4212 2021-05-25T11:59:50.830 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.813+0000 7fd76559a700 1 -- 172.21.15.102:0/3631572193 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd74c003270 con 0x7fd76812bde0 2021-05-25T11:59:50.830 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.813+0000 7fd76559a700 1 -- 172.21.15.102:0/3631572193 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1992374371 0 0) 0x7fd768090aa0 con 0x7fd76812c260 2021-05-25T11:59:50.830 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.813+0000 7fd76559a700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:50.830 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.813+0000 7fd76559a700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:50.831 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.813+0000 7fd76559a700 10 monclient(hunting): my global_id is 4205 2021-05-25T11:59:50.831 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.813+0000 7fd76559a700 1 -- 172.21.15.102:0/3631572193 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd76809add0 con 0x7fd76812c260 2021-05-25T11:59:50.831 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.814+0000 7fd76559a700 1 -- 172.21.15.102:0/3631572193 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3980501178 0 0) 0x7fd74c003270 con 0x7fd76812bde0 2021-05-25T11:59:50.831 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.814+0000 7fd76559a700 1 -- 172.21.15.102:0/3631572193 >> v1:172.21.15.102:6790/0 conn(0x7fd76812c260 legacy=0x7fd768134190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:50.832 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.814+0000 7fd76559a700 1 -- 172.21.15.102:0/3631572193 >> v1:172.21.15.181:6789/0 conn(0x7fd768098280 legacy=0x7fd76812b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:50.832 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.814+0000 7fd76559a700 10 monclient: _finish_hunting 0 2021-05-25T11:59:50.832 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.814+0000 7fd76559a700 1 monclient: found mon.noname-a 2021-05-25T11:59:50.832 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.814+0000 7fd76559a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T11:59:50.832 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.814+0000 7fd76559a700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T11:59:50.833 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.814+0000 7fd76559a700 1 -- 172.21.15.102:0/3631572193 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd768098660 con 0x7fd76812bde0 2021-05-25T11:59:50.833 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.814+0000 7fd76559a700 10 monclient: _finish_auth 0 2021-05-25T11:59:50.833 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.814+0000 7fd76559a700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T11:59:50.833 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.814+0000 7fd76759e700 5 monclient: authenticate success, global_id 4212 2021-05-25T11:59:50.833 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.814+0000 7fd76759e700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T11:59:50.834 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.814+0000 7fd76559a700 1 -- 172.21.15.102:0/3631572193 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fd754001610 con 0x7fd76812bde0 2021-05-25T11:59:50.834 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.814+0000 7fd76559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:50.834 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.814+0000 7fd76559a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T11:59:50.834 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.814+0000 7fd76559a700 10 monclient: dump: 2021-05-25T11:59:50.835 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:50.835 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:50.835 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:50.835 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:50.835 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:50.836 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:50.836 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:50.836 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:50.836 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:50.836 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:50.836 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.814+0000 7fd76759e700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T11:59:50.837 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.814+0000 7fd76559a700 1 -- 172.21.15.102:0/3631572193 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd754001830 con 0x7fd76812bde0 2021-05-25T11:59:50.837 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.814+0000 7fd76559a700 10 monclient: handle_config config(0 keys) v1 2021-05-25T11:59:50.837 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.814+0000 7fd76759e700 10 monclient: get_monmap_and_config success 2021-05-25T11:59:50.837 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.814+0000 7fd76559a700 1 -- 172.21.15.102:0/3631572193 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fd754001b40 con 0x7fd76812bde0 2021-05-25T11:59:50.838 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.814+0000 7fd76559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:50.838 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.814+0000 7fd76759e700 4 set_mon_vals no callback set 2021-05-25T11:59:50.838 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.814+0000 7fd76559a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T11:59:50.838 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.814+0000 7fd76559a700 10 monclient: dump: 2021-05-25T11:59:50.838 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:50.839 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:50.839 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:50.839 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:50.839 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:50.839 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:50.840 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:50.840 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:50.840 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:50.840 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:50.840 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.814+0000 7fd76759e700 10 monclient: shutdown 2021-05-25T11:59:50.841 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.814+0000 7fd76759e700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T11:59:50.841 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.814+0000 7fd76759e700 1 -- 172.21.15.102:0/3631572193 >> v1:172.21.15.102:6789/0 conn(0x7fd76812bde0 legacy=0x7fd768130a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:50.841 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.814+0000 7fd76759e700 1 -- 172.21.15.102:0/3631572193 shutdown_connections 2021-05-25T11:59:50.841 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.814+0000 7fd76759e700 1 -- 172.21.15.102:0/3631572193 >> 172.21.15.102:0/3631572193 conn(0x7fd76809ea40 msgr2=0x7fd76809ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T11:59:50.841 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.814+0000 7fd76759e700 1 -- 172.21.15.102:0/3631572193 shutdown_connections 2021-05-25T11:59:50.842 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.814+0000 7fd76759e700 1 -- 172.21.15.102:0/3631572193 wait complete. 2021-05-25T11:59:50.842 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.815+0000 7fd76759e700 10 monclient: build_initial_monmap 2021-05-25T11:59:50.842 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.815+0000 7fd76759e700 10 monclient: monmap: 2021-05-25T11:59:50.842 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T11:59:50.842 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T11:59:50.843 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T11:59:50.843 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T11:59:50.843 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T11:59:50.843 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T11:59:50.843 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T11:59:50.844 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T11:59:50.844 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T11:59:50.844 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:50.844 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.815+0000 7fd76759e700 1 librados: starting msgr at 2021-05-25T11:59:50.844 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.815+0000 7fd76759e700 1 librados: starting objecter 2021-05-25T11:59:50.845 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.815+0000 7fd76759e700 1 Processor -- start 2021-05-25T11:59:50.845 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.815+0000 7fd76759e700 1 -- start start 2021-05-25T11:59:50.845 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.815+0000 7fd76759e700 1 librados: setting wanted keys 2021-05-25T11:59:50.845 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.815+0000 7fd76759e700 1 librados: calling monclient init 2021-05-25T11:59:50.846 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.815+0000 7fd76759e700 10 monclient: init 2021-05-25T11:59:50.846 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.815+0000 7fd76759e700 10 monclient: _reopen_session rank -1 2021-05-25T11:59:50.846 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.815+0000 7fd76759e700 10 monclient: _add_conns ranks=[2,1,0] 2021-05-25T11:59:50.846 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.815+0000 7fd76759e700 10 monclient(hunting): picked mon.noname-c con 0x7fd768098280 addr v1:172.21.15.102:6790/0 2021-05-25T11:59:50.846 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.815+0000 7fd76759e700 10 monclient(hunting): picked mon.noname-b con 0x7fd76812bde0 addr v1:172.21.15.181:6789/0 2021-05-25T11:59:50.847 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.815+0000 7fd76759e700 10 monclient(hunting): picked mon.noname-a con 0x7fd76812c260 addr v1:172.21.15.102:6789/0 2021-05-25T11:59:50.847 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.815+0000 7fd76759e700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd76809e520 con 0x7fd76812c260 2021-05-25T11:59:50.847 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.815+0000 7fd76759e700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd76813b920 con 0x7fd76812bde0 2021-05-25T11:59:50.847 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.815+0000 7fd76759e700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd768118aa0 con 0x7fd768098280 2021-05-25T11:59:50.848 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.815+0000 7fd76759e700 10 monclient(hunting): _renew_subs 2021-05-25T11:59:50.848 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.815+0000 7fd76759e700 10 monclient(hunting): authenticate will time out at 791.412892s 2021-05-25T11:59:50.848 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.815+0000 7fd76659c700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fd768098280 0x7fd76813aa50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:53538/0 (socket says 172.21.15.102:53538) 2021-05-25T11:59:50.848 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.815+0000 7fd76659c700 1 -- 172.21.15.102:0/3146761752 learned_addr learned my addr 172.21.15.102:0/3146761752 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T11:59:50.849 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.816+0000 7fd7527fc700 10 client.?.objecter ms_handle_connect 0x7fd76812c260 2021-05-25T11:59:50.849 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.816+0000 7fd7527fc700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:50.849 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.816+0000 7fd7527fc700 10 client.?.objecter ms_handle_connect 0x7fd768098280 2021-05-25T11:59:50.849 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.816+0000 7fd7527fc700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:50.849 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.816+0000 7fd7527fc700 10 client.?.objecter ms_handle_connect 0x7fd76812bde0 2021-05-25T11:59:50.850 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.816+0000 7fd7527fc700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:50.850 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.816+0000 7fd7527fc700 1 -- 172.21.15.102:0/3146761752 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4213822056 0 0) 0x7fd76809e520 con 0x7fd76812c260 2021-05-25T11:59:50.850 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.816+0000 7fd7527fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:50.850 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.816+0000 7fd7527fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:50.851 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.816+0000 7fd7527fc700 10 monclient(hunting): my global_id is 4215 2021-05-25T11:59:50.851 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.816+0000 7fd7527fc700 1 -- 172.21.15.102:0/3146761752 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd744004c10 con 0x7fd76812c260 2021-05-25T11:59:50.851 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.816+0000 7fd7527fc700 1 -- 172.21.15.102:0/3146761752 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2783356670 0 0) 0x7fd768118aa0 con 0x7fd768098280 2021-05-25T11:59:50.851 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.816+0000 7fd7527fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:50.852 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.816+0000 7fd7527fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:50.852 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.816+0000 7fd7527fc700 10 monclient(hunting): my global_id is 4208 2021-05-25T11:59:50.852 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.816+0000 7fd7527fc700 1 -- 172.21.15.102:0/3146761752 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd76809e520 con 0x7fd768098280 2021-05-25T11:59:50.852 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.816+0000 7fd7527fc700 1 -- 172.21.15.102:0/3146761752 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1527673221 0 0) 0x7fd76813b920 con 0x7fd76812bde0 2021-05-25T11:59:50.853 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.816+0000 7fd7527fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:50.853 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.816+0000 7fd7527fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:50.853 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.816+0000 7fd7527fc700 10 monclient(hunting): my global_id is 4210 2021-05-25T11:59:50.853 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.816+0000 7fd7527fc700 1 -- 172.21.15.102:0/3146761752 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd768118aa0 con 0x7fd76812bde0 2021-05-25T11:59:50.853 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.816+0000 7fd7527fc700 1 -- 172.21.15.102:0/3146761752 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 610475096 0 0) 0x7fd744004c10 con 0x7fd76812c260 2021-05-25T11:59:50.854 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.816+0000 7fd7527fc700 1 -- 172.21.15.102:0/3146761752 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd76813b920 con 0x7fd76812c260 2021-05-25T11:59:50.854 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.816+0000 7fd7527fc700 1 -- 172.21.15.102:0/3146761752 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fd758001d70 con 0x7fd76812c260 2021-05-25T11:59:50.854 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.816+0000 7fd7527fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:50.854 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.816+0000 7fd7527fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T11:59:50.854 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.816+0000 7fd7527fc700 10 monclient(hunting): dump: 2021-05-25T11:59:50.855 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:50.855 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:50.855 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:50.855 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:50.855 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:50.856 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:50.856 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:50.856 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:50.856 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:50.857 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:50.857 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.816+0000 7fd7527fc700 10 monclient(hunting): _finish_auth 0 2021-05-25T11:59:50.857 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.816+0000 7fd7527fc700 1 -- 172.21.15.102:0/3146761752 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1044608762 0 0) 0x7fd76809e520 con 0x7fd768098280 2021-05-25T11:59:50.857 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.816+0000 7fd7527fc700 1 -- 172.21.15.102:0/3146761752 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd744004c10 con 0x7fd768098280 2021-05-25T11:59:50.857 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.816+0000 7fd7527fc700 1 -- 172.21.15.102:0/3146761752 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fd75c001910 con 0x7fd768098280 2021-05-25T11:59:50.858 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.816+0000 7fd7527fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:50.858 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.816+0000 7fd7527fc700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2021-05-25T11:59:50.858 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.816+0000 7fd7527fc700 10 monclient(hunting): dump: 2021-05-25T11:59:50.858 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:50.858 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:50.859 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:50.859 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:50.859 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:50.859 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:50.859 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:50.860 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:50.860 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:50.860 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:50.860 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.816+0000 7fd7527fc700 1 -- 172.21.15.102:0/3146761752 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3929091511 0 0) 0x7fd768118aa0 con 0x7fd76812bde0 2021-05-25T11:59:50.860 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.816+0000 7fd7527fc700 1 -- 172.21.15.102:0/3146761752 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd76809e520 con 0x7fd76812bde0 2021-05-25T11:59:50.861 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.816+0000 7fd7527fc700 1 -- 172.21.15.102:0/3146761752 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fd754001510 con 0x7fd76812bde0 2021-05-25T11:59:50.861 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.816+0000 7fd7527fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:50.861 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.816+0000 7fd7527fc700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2021-05-25T11:59:50.861 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.816+0000 7fd7527fc700 10 monclient(hunting): dump: 2021-05-25T11:59:50.861 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:50.862 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:50.862 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:50.862 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:50.862 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:50.863 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:50.863 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:50.863 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:50.863 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:50.863 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:50.864 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.816+0000 7fd7527fc700 1 -- 172.21.15.102:0/3146761752 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1059907862 0 0) 0x7fd76813b920 con 0x7fd76812c260 2021-05-25T11:59:50.864 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.816+0000 7fd7527fc700 1 -- 172.21.15.102:0/3146761752 >> v1:172.21.15.102:6790/0 conn(0x7fd768098280 legacy=0x7fd76813aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:50.864 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.816+0000 7fd7527fc700 1 -- 172.21.15.102:0/3146761752 >> v1:172.21.15.181:6789/0 conn(0x7fd76812bde0 legacy=0x7fd7681282d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:50.864 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.816+0000 7fd7527fc700 10 monclient: _finish_hunting 0 2021-05-25T11:59:50.865 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.816+0000 7fd7527fc700 1 monclient: found mon.a 2021-05-25T11:59:50.865 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.816+0000 7fd7527fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T11:59:50.865 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.816+0000 7fd7527fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:50.865 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.816+0000 7fd7527fc700 1 -- 172.21.15.102:0/3146761752 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd768118d40 con 0x7fd76812c260 2021-05-25T11:59:50.865 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.816+0000 7fd7527fc700 10 monclient: _finish_auth 0 2021-05-25T11:59:50.866 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.816+0000 7fd7527fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T11:59:50.866 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.816+0000 7fd76759e700 5 monclient: authenticate success, global_id 4215 2021-05-25T11:59:50.866 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.816+0000 7fd76759e700 10 monclient: _renew_subs 2021-05-25T11:59:50.866 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.816+0000 7fd76759e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:50.867 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.816+0000 7fd76759e700 1 -- 172.21.15.102:0/3146761752 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd768117750 con 0x7fd76812c260 2021-05-25T11:59:50.867 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.816+0000 7fd76759e700 10 client.4215.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T11:59:50.867 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.816+0000 7fd76759e700 10 monclient: _renew_subs 2021-05-25T11:59:50.867 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.816+0000 7fd76759e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:50.867 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.816+0000 7fd76759e700 1 -- 172.21.15.102:0/3146761752 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd76813b380 con 0x7fd76812c260 2021-05-25T11:59:50.868 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.816+0000 7fd76759e700 1 librados: init done 2021-05-25T11:59:50.868 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.816+0000 7fd7527fc700 1 -- 172.21.15.102:0/3146761752 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd758001fe0 con 0x7fd76812c260 2021-05-25T11:59:50.868 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.816+0000 7fd7527fc700 10 monclient: handle_config config(0 keys) v1 2021-05-25T11:59:50.868 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.816+0000 7fd7527fc700 1 -- 172.21.15.102:0/3146761752 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fd758004d00 con 0x7fd76812c260 2021-05-25T11:59:50.869 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.816+0000 7fd7527fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:50.869 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.816+0000 7fd764d99700 4 set_mon_vals no callback set 2021-05-25T11:59:50.869 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.816+0000 7fd7527fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T11:59:50.869 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.816+0000 7fd7527fc700 10 monclient: dump: 2021-05-25T11:59:50.869 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:50.870 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:50.870 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:50.870 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:50.870 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:50.871 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:50.871 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:50.871 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:50.871 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:50.871 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:50.872 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.817+0000 7fd7527fc700 1 -- 172.21.15.102:0/3146761752 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7fd75800e020 con 0x7fd76812c260 2021-05-25T11:59:50.872 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.817+0000 7fd76759e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:50.872 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.817+0000 7fd76759e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:50.872 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.817+0000 7fd76759e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:50.872 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.817+0000 7fd76759e700 1 -- 172.21.15.102:0/3146761752 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd748004db0 con 0x7fd76812c260 2021-05-25T11:59:50.872 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.817+0000 7fd7527fc700 1 -- 172.21.15.102:0/3146761752 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(12..12 src has 1..12) v4 ==== 3172+0+0 (unknown 2473022558 0 0) 0x7fd75803cf20 con 0x7fd76812c260 2021-05-25T11:59:50.873 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.817+0000 7fd7527fc700 10 client.4215.objecter ms_dispatch 0x7fd768001550 osd_map(12..12 src has 1..12) v4 2021-05-25T11:59:50.873 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.817+0000 7fd7527fc700 3 client.4215.objecter handle_osd_map got epochs [12,12] > 0 2021-05-25T11:59:50.873 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.817+0000 7fd7527fc700 3 client.4215.objecter handle_osd_map decoding full epoch 12 2021-05-25T11:59:50.873 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.817+0000 7fd7527fc700 20 client.4215.objecter dump_active .. 0 homeless 2021-05-25T11:59:50.873 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.818+0000 7fd76659c700 10 monclient: get_auth_request con 0x7fd744033520 auth_method 0 2021-05-25T11:59:50.874 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.818+0000 7fd7527fc700 10 client.4215.objecter ms_handle_connect 0x7fd744033520 2021-05-25T11:59:50.874 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.821+0000 7fd7527fc700 1 -- 172.21.15.102:0/3146761752 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7fd758012f30 con 0x7fd76812c260 2021-05-25T11:59:50.874 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.821+0000 7fd7527fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:50.874 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.821+0000 7fd7527fc700 10 monclient: _finish_command 1 = system:0 2021-05-25T11:59:50.972 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.970+0000 7fd76759e700 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-05-25T11:59:50.972 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.970+0000 7fd76759e700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-05-25T11:59:50.973 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.970+0000 7fd76759e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:50.973 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:50.970+0000 7fd76759e700 1 -- 172.21.15.102:0/3146761752 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7fd7480055e0 con 0x7fd76812c260 2021-05-25T11:59:51.589 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.587+0000 7fd7527fc700 1 -- 172.21.15.102:0/3146761752 <== mon.0 v1:172.21.15.102:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v13) v1 ==== 109+0+0 (unknown 2984082091 0 0) 0x7fd758012b20 con 0x7fd76812c260 2021-05-25T11:59:51.589 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.587+0000 7fd7527fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-05-25T11:59:51.590 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.587+0000 7fd7527fc700 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2021-05-25T11:59:51.593 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.591+0000 7fd76759e700 10 librados: watch_flush enter 2021-05-25T11:59:51.593 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.591+0000 7fd76759e700 10 librados: watch_flush exit 2021-05-25T11:59:51.593 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.591+0000 7fd76759e700 20 client.4215.objecter shutdown clearing up homeless session... 2021-05-25T11:59:51.594 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.591+0000 7fd76759e700 10 client.4215.objecter successfully canceled tick 2021-05-25T11:59:51.594 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.591+0000 7fd76759e700 1 -- 172.21.15.102:0/3146761752 >> v1:172.21.15.181:6812/29582 conn(0x7fd744033520 legacy=0x7fd7440359e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:51.594 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.591+0000 7fd76759e700 10 monclient: shutdown 2021-05-25T11:59:51.594 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.591+0000 7fd76759e700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T11:59:51.594 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.591+0000 7fd76759e700 1 -- 172.21.15.102:0/3146761752 >> v1:172.21.15.102:6789/0 conn(0x7fd76812c260 legacy=0x7fd768142dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:51.595 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.591+0000 7fd76759e700 1 -- 172.21.15.102:0/3146761752 shutdown_connections 2021-05-25T11:59:51.595 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.591+0000 7fd76759e700 1 -- 172.21.15.102:0/3146761752 >> 172.21.15.102:0/3146761752 conn(0x7fd76809ea40 msgr2=0x7fd76812a0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T11:59:51.595 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.591+0000 7fd76759e700 1 -- 172.21.15.102:0/3146761752 shutdown_connections 2021-05-25T11:59:51.595 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.591+0000 7fd76759e700 1 -- 172.21.15.102:0/3146761752 wait complete. 2021-05-25T11:59:51.595 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.591+0000 7fd76759e700 1 librados: shutdown 2021-05-25T11:59:51.596 INFO:teuthology.orchestra.run.smithi102.stderr:pool 'rbd' created 2021-05-25T11:59:51.605 DEBUG:teuthology.orchestra.run.smithi102:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2021-05-25T11:59:51.719 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.717+0000 7f694a5a1700 10 monclient: get_monmap_and_config 2021-05-25T11:59:51.719 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.717+0000 7f694a5a1700 10 monclient: build_initial_monmap 2021-05-25T11:59:51.719 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.717+0000 7f694a5a1700 10 monclient: monmap: 2021-05-25T11:59:51.720 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T11:59:51.720 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:51.720 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:51.718651+0000 2021-05-25T11:59:51.720 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:51.718651+0000 2021-05-25T11:59:51.721 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T11:59:51.721 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:51.721 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T11:59:51.721 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T11:59:51.722 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T11:59:51.722 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:51.741 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.718+0000 7f694a5a1700 1 Processor -- start 2021-05-25T11:59:51.742 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.718+0000 7f694a5a1700 1 -- start start 2021-05-25T11:59:51.742 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.718+0000 7f694a5a1700 10 monclient: init 2021-05-25T11:59:51.743 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.718+0000 7f694a5a1700 10 monclient: _reopen_session rank -1 2021-05-25T11:59:51.743 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.718+0000 7f694a5a1700 10 monclient: _add_conns ranks=[1,0,2] 2021-05-25T11:59:51.743 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.718+0000 7f694a5a1700 10 monclient(hunting): picked mon.noname-c con 0x7f6944093620 addr v1:172.21.15.181:6789/0 2021-05-25T11:59:51.744 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.718+0000 7f694a5a1700 10 monclient(hunting): picked mon.noname-a con 0x7f6944093a00 addr v1:172.21.15.102:6789/0 2021-05-25T11:59:51.744 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.718+0000 7f694a5a1700 10 monclient(hunting): picked mon.noname-b con 0x7f694412e300 addr v1:172.21.15.102:6790/0 2021-05-25T11:59:51.745 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.718+0000 7f694a5a1700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6944092f50 con 0x7f6944093a00 2021-05-25T11:59:51.745 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.718+0000 7f694a5a1700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6944093180 con 0x7f6944093620 2021-05-25T11:59:51.745 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.718+0000 7f694a5a1700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69440925c0 con 0x7f694412e300 2021-05-25T11:59:51.746 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.718+0000 7f694a5a1700 10 monclient(hunting): _renew_subs 2021-05-25T11:59:51.746 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.718+0000 7f694a5a1700 10 monclient(hunting): authenticate will time out at 792.315960s 2021-05-25T11:59:51.747 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.719+0000 7f6948b3e700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f694412e300 0x7f69441306f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:53558/0 (socket says 172.21.15.102:53558) 2021-05-25T11:59:51.747 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.719+0000 7f6948b3e700 1 -- 172.21.15.102:0/1152192225 learned_addr learned my addr 172.21.15.102:0/1152192225 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T11:59:51.747 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.719+0000 7f69437fe700 1 --1- 172.21.15.102:0/1152192225 >> v1:172.21.15.102:6789/0 conn(0x7f6944093a00 0x7f6944093eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:58678/0 (socket says 172.21.15.102:58678) 2021-05-25T11:59:51.748 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.719+0000 7f6942ffd700 1 -- 172.21.15.102:0/1152192225 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3782691218 0 0) 0x7f6944092f50 con 0x7f6944093a00 2021-05-25T11:59:51.748 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.719+0000 7f6942ffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:51.749 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.719+0000 7f6942ffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:51.749 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.719+0000 7f6942ffd700 10 monclient(hunting): my global_id is 4218 2021-05-25T11:59:51.749 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.719+0000 7f6942ffd700 1 -- 172.21.15.102:0/1152192225 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f692c003270 con 0x7f6944093a00 2021-05-25T11:59:51.750 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.719+0000 7f6942ffd700 1 -- 172.21.15.102:0/1152192225 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1550732019 0 0) 0x7f69440925c0 con 0x7f694412e300 2021-05-25T11:59:51.750 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.719+0000 7f6942ffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:51.751 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.719+0000 7f6942ffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:51.751 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.719+0000 7f6942ffd700 10 monclient(hunting): my global_id is 4211 2021-05-25T11:59:51.751 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.719+0000 7f6942ffd700 1 -- 172.21.15.102:0/1152192225 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6944092f50 con 0x7f694412e300 2021-05-25T11:59:51.752 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.719+0000 7f6942ffd700 1 -- 172.21.15.102:0/1152192225 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3166741165 0 0) 0x7f6944092f50 con 0x7f694412e300 2021-05-25T11:59:51.752 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.719+0000 7f6942ffd700 1 -- 172.21.15.102:0/1152192225 >> v1:172.21.15.181:6789/0 conn(0x7f6944093620 legacy=0x7f6944090990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:51.753 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.719+0000 7f6942ffd700 1 -- 172.21.15.102:0/1152192225 >> v1:172.21.15.102:6789/0 conn(0x7f6944093a00 legacy=0x7f6944093eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:51.753 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.719+0000 7f6942ffd700 10 monclient: _finish_hunting 0 2021-05-25T11:59:51.754 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.719+0000 7f6942ffd700 1 monclient: found mon.noname-b 2021-05-25T11:59:51.754 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.719+0000 7f6942ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T11:59:51.754 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.719+0000 7f6942ffd700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T11:59:51.755 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.719+0000 7f6942ffd700 1 -- 172.21.15.102:0/1152192225 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6944099760 con 0x7f694412e300 2021-05-25T11:59:51.755 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.719+0000 7f6942ffd700 10 monclient: _finish_auth 0 2021-05-25T11:59:51.756 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.719+0000 7f6942ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T11:59:51.756 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.719+0000 7f6942ffd700 1 -- 172.21.15.102:0/1152192225 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f6938001990 con 0x7f694412e300 2021-05-25T11:59:51.756 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.719+0000 7f694a5a1700 5 monclient: authenticate success, global_id 4211 2021-05-25T11:59:51.757 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.719+0000 7f694a5a1700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T11:59:51.757 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.719+0000 7f6942ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:51.758 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.719+0000 7f6942ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T11:59:51.758 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.719+0000 7f6942ffd700 10 monclient: dump: 2021-05-25T11:59:51.759 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:51.759 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:51.759 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:51.760 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:51.760 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:51.761 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:51.761 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:51.762 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:51.762 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:51.762 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:51.763 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.719+0000 7f694a5a1700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T11:59:51.763 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.719+0000 7f6942ffd700 1 -- 172.21.15.102:0/1152192225 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6938001bb0 con 0x7f694412e300 2021-05-25T11:59:51.764 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.719+0000 7f6942ffd700 10 monclient: handle_config config(0 keys) v1 2021-05-25T11:59:51.764 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.719+0000 7f694a5a1700 10 monclient: get_monmap_and_config success 2021-05-25T11:59:51.765 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.719+0000 7f6942ffd700 1 -- 172.21.15.102:0/1152192225 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f6938001f40 con 0x7f694412e300 2021-05-25T11:59:51.765 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.719+0000 7f6942ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:51.765 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.719+0000 7f694a5a1700 4 set_mon_vals no callback set 2021-05-25T11:59:51.766 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.719+0000 7f6942ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T11:59:51.766 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.719+0000 7f6942ffd700 10 monclient: dump: 2021-05-25T11:59:51.767 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:51.767 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:51.767 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:51.768 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:51.768 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:51.768 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:51.769 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:51.769 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:51.769 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:51.769 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:51.769 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.720+0000 7f694a5a1700 10 monclient: shutdown 2021-05-25T11:59:51.770 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.720+0000 7f694a5a1700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T11:59:51.770 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.720+0000 7f694a5a1700 1 -- 172.21.15.102:0/1152192225 >> v1:172.21.15.102:6790/0 conn(0x7f694412e300 legacy=0x7f69441306f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:51.770 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.720+0000 7f694a5a1700 1 -- 172.21.15.102:0/1152192225 shutdown_connections 2021-05-25T11:59:51.770 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.720+0000 7f694a5a1700 1 -- 172.21.15.102:0/1152192225 >> 172.21.15.102:0/1152192225 conn(0x7f6944097c50 msgr2=0x7f6944061b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T11:59:51.771 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.720+0000 7f694a5a1700 1 -- 172.21.15.102:0/1152192225 shutdown_connections 2021-05-25T11:59:51.771 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.720+0000 7f694a5a1700 1 -- 172.21.15.102:0/1152192225 wait complete. 2021-05-25T11:59:51.771 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.720+0000 7f694a5a1700 10 monclient: build_initial_monmap 2021-05-25T11:59:51.771 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.720+0000 7f694a5a1700 10 monclient: monmap: 2021-05-25T11:59:51.771 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T11:59:51.772 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T11:59:51.772 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T11:59:51.772 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T11:59:51.772 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T11:59:51.772 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T11:59:51.773 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T11:59:51.773 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T11:59:51.773 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T11:59:51.773 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:51.773 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.720+0000 7f694a5a1700 1 librados: starting msgr at 2021-05-25T11:59:51.774 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.720+0000 7f694a5a1700 1 librados: starting objecter 2021-05-25T11:59:51.774 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.720+0000 7f694a5a1700 1 Processor -- start 2021-05-25T11:59:51.774 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.721+0000 7f694a5a1700 1 -- start start 2021-05-25T11:59:51.774 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.721+0000 7f694a5a1700 1 librados: setting wanted keys 2021-05-25T11:59:51.775 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.721+0000 7f694a5a1700 1 librados: calling monclient init 2021-05-25T11:59:51.775 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.721+0000 7f694a5a1700 10 monclient: init 2021-05-25T11:59:51.775 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.721+0000 7f694a5a1700 10 monclient: _reopen_session rank -1 2021-05-25T11:59:51.775 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.721+0000 7f694a5a1700 10 monclient: _add_conns ranks=[0,1,2] 2021-05-25T11:59:51.775 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.721+0000 7f694a5a1700 10 monclient(hunting): picked mon.noname-a con 0x7f6944093620 addr v1:172.21.15.102:6789/0 2021-05-25T11:59:51.776 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.721+0000 7f694a5a1700 10 monclient(hunting): picked mon.noname-b con 0x7f6944093a00 addr v1:172.21.15.181:6789/0 2021-05-25T11:59:51.776 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.721+0000 7f694a5a1700 10 monclient(hunting): picked mon.noname-c con 0x7f694412e300 addr v1:172.21.15.102:6790/0 2021-05-25T11:59:51.776 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.721+0000 7f694a5a1700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69440680a0 con 0x7f6944093620 2021-05-25T11:59:51.776 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.721+0000 7f694a5a1700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69441378d0 con 0x7f6944093a00 2021-05-25T11:59:51.776 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.721+0000 7f694a5a1700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6944111820 con 0x7f694412e300 2021-05-25T11:59:51.777 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.721+0000 7f694a5a1700 10 monclient(hunting): _renew_subs 2021-05-25T11:59:51.777 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.721+0000 7f694a5a1700 10 monclient(hunting): authenticate will time out at 792.318629s 2021-05-25T11:59:51.777 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.721+0000 7f6943fff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f6944093620 0x7f6944136ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:58682/0 (socket says 172.21.15.102:58682) 2021-05-25T11:59:51.777 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.721+0000 7f6943fff700 1 -- 172.21.15.102:0/193442493 learned_addr learned my addr 172.21.15.102:0/193442493 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T11:59:51.778 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.721+0000 7f6937fff700 10 client.?.objecter ms_handle_connect 0x7f6944093620 2021-05-25T11:59:51.778 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.721+0000 7f6937fff700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:51.778 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.721+0000 7f6937fff700 10 client.?.objecter ms_handle_connect 0x7f694412e300 2021-05-25T11:59:51.778 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.721+0000 7f6937fff700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:51.778 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.721+0000 7f6937fff700 10 client.?.objecter ms_handle_connect 0x7f6944093a00 2021-05-25T11:59:51.779 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.721+0000 7f6937fff700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:51.779 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.721+0000 7f6937fff700 1 -- 172.21.15.102:0/193442493 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3761231432 0 0) 0x7f69440680a0 con 0x7f6944093620 2021-05-25T11:59:51.779 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.721+0000 7f6937fff700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:51.779 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.721+0000 7f6937fff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:51.779 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.721+0000 7f6937fff700 10 monclient(hunting): my global_id is 4221 2021-05-25T11:59:51.780 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.721+0000 7f6937fff700 1 -- 172.21.15.102:0/193442493 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6920004c10 con 0x7f6944093620 2021-05-25T11:59:51.780 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.721+0000 7f6937fff700 1 -- 172.21.15.102:0/193442493 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2201632813 0 0) 0x7f6944111820 con 0x7f694412e300 2021-05-25T11:59:51.780 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.721+0000 7f6937fff700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:51.780 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.721+0000 7f6937fff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:51.781 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.721+0000 7f6937fff700 10 monclient(hunting): my global_id is 4214 2021-05-25T11:59:51.781 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.721+0000 7f6937fff700 1 -- 172.21.15.102:0/193442493 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69440680a0 con 0x7f694412e300 2021-05-25T11:59:51.781 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.721+0000 7f6937fff700 1 -- 172.21.15.102:0/193442493 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3135556526 0 0) 0x7f69441378d0 con 0x7f6944093a00 2021-05-25T11:59:51.781 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.722+0000 7f6937fff700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:51.781 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.722+0000 7f6937fff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:51.782 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.722+0000 7f6937fff700 10 monclient(hunting): my global_id is 4216 2021-05-25T11:59:51.782 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.722+0000 7f6937fff700 1 -- 172.21.15.102:0/193442493 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6944111820 con 0x7f6944093a00 2021-05-25T11:59:51.782 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.722+0000 7f6937fff700 1 -- 172.21.15.102:0/193442493 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 162078818 0 0) 0x7f6920004c10 con 0x7f6944093620 2021-05-25T11:59:51.782 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.722+0000 7f6937fff700 1 -- 172.21.15.102:0/193442493 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f69441378d0 con 0x7f6944093620 2021-05-25T11:59:51.782 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.722+0000 7f6937fff700 1 -- 172.21.15.102:0/193442493 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f6928001900 con 0x7f6944093620 2021-05-25T11:59:51.783 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.722+0000 7f6937fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:51.783 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.722+0000 7f6937fff700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T11:59:51.783 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.722+0000 7f6937fff700 10 monclient(hunting): dump: 2021-05-25T11:59:51.783 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:51.783 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:51.784 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:51.784 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:51.784 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:51.784 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:51.784 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:51.785 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:51.785 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:51.785 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:51.785 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.722+0000 7f6937fff700 10 monclient(hunting): _finish_auth 0 2021-05-25T11:59:51.786 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.722+0000 7f6937fff700 1 -- 172.21.15.102:0/193442493 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1637091627 0 0) 0x7f69440680a0 con 0x7f694412e300 2021-05-25T11:59:51.786 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.722+0000 7f6937fff700 1 -- 172.21.15.102:0/193442493 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6920004c10 con 0x7f694412e300 2021-05-25T11:59:51.786 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.722+0000 7f6937fff700 1 -- 172.21.15.102:0/193442493 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f6938001950 con 0x7f694412e300 2021-05-25T11:59:51.786 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.722+0000 7f6937fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:51.786 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.722+0000 7f6937fff700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2021-05-25T11:59:51.787 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.722+0000 7f6937fff700 10 monclient(hunting): dump: 2021-05-25T11:59:51.787 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:51.787 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:51.787 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:51.787 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:51.788 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:51.788 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:51.788 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:51.788 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:51.788 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:51.789 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:51.789 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.722+0000 7f6937fff700 1 -- 172.21.15.102:0/193442493 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2356600697 0 0) 0x7f6944111820 con 0x7f6944093a00 2021-05-25T11:59:51.789 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.722+0000 7f6937fff700 1 -- 172.21.15.102:0/193442493 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f69440680a0 con 0x7f6944093a00 2021-05-25T11:59:51.789 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.722+0000 7f6937fff700 1 -- 172.21.15.102:0/193442493 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f6930001a70 con 0x7f6944093a00 2021-05-25T11:59:51.790 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.722+0000 7f6937fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:51.790 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.722+0000 7f6937fff700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2021-05-25T11:59:51.790 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.722+0000 7f6937fff700 10 monclient(hunting): dump: 2021-05-25T11:59:51.790 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:51.790 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:51.791 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:51.791 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:51.791 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:51.791 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:51.791 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:51.792 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:51.792 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:51.792 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:51.792 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.722+0000 7f6937fff700 1 -- 172.21.15.102:0/193442493 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1661201946 0 0) 0x7f69441378d0 con 0x7f6944093620 2021-05-25T11:59:51.793 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.722+0000 7f6937fff700 1 -- 172.21.15.102:0/193442493 >> v1:172.21.15.102:6790/0 conn(0x7f694412e300 legacy=0x7f694413ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:51.793 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.722+0000 7f6937fff700 1 -- 172.21.15.102:0/193442493 >> v1:172.21.15.181:6789/0 conn(0x7f6944093a00 legacy=0x7f69441371c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:51.793 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.722+0000 7f6937fff700 10 monclient: _finish_hunting 0 2021-05-25T11:59:51.793 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.722+0000 7f6937fff700 1 monclient: found mon.a 2021-05-25T11:59:51.793 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.722+0000 7f6937fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T11:59:51.794 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.722+0000 7f6937fff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:51.794 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.722+0000 7f6937fff700 1 -- 172.21.15.102:0/193442493 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6944111ac0 con 0x7f6944093620 2021-05-25T11:59:51.794 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.722+0000 7f6937fff700 10 monclient: _finish_auth 0 2021-05-25T11:59:51.794 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.722+0000 7f6937fff700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T11:59:51.794 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.722+0000 7f694a5a1700 5 monclient: authenticate success, global_id 4221 2021-05-25T11:59:51.795 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.722+0000 7f694a5a1700 10 monclient: _renew_subs 2021-05-25T11:59:51.795 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.722+0000 7f694a5a1700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:51.795 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.722+0000 7f694a5a1700 1 -- 172.21.15.102:0/193442493 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69440679d0 con 0x7f6944093620 2021-05-25T11:59:51.795 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.722+0000 7f694a5a1700 10 client.4221.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T11:59:51.795 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.722+0000 7f694a5a1700 10 monclient: _renew_subs 2021-05-25T11:59:51.796 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.722+0000 7f694a5a1700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:51.796 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.722+0000 7f694a5a1700 1 -- 172.21.15.102:0/193442493 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f69440673d0 con 0x7f6944093620 2021-05-25T11:59:51.796 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.722+0000 7f694a5a1700 1 librados: init done 2021-05-25T11:59:51.796 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.722+0000 7f6937fff700 1 -- 172.21.15.102:0/193442493 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6928001c90 con 0x7f6944093620 2021-05-25T11:59:51.797 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.722+0000 7f6937fff700 10 monclient: handle_config config(0 keys) v1 2021-05-25T11:59:51.797 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.722+0000 7f6937fff700 1 -- 172.21.15.102:0/193442493 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f6928002410 con 0x7f6944093620 2021-05-25T11:59:51.797 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.722+0000 7f6937fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:51.797 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.722+0000 7f6937fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T11:59:51.797 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.722+0000 7f69427fc700 4 set_mon_vals no callback set 2021-05-25T11:59:51.798 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.722+0000 7f6937fff700 10 monclient: dump: 2021-05-25T11:59:51.798 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:51.798 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:51.798 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:51.798 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:51.799 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:51.799 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:51.799 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:51.799 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:51.800 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:51.800 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:51.800 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.723+0000 7f6937fff700 1 -- 172.21.15.102:0/193442493 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f6928002570 con 0x7f6944093620 2021-05-25T11:59:51.800 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.723+0000 7f694a5a1700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:51.800 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.723+0000 7f694a5a1700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:51.801 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.723+0000 7f694a5a1700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:51.801 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.723+0000 7f694a5a1700 1 -- 172.21.15.102:0/193442493 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6924004db0 con 0x7f6944093620 2021-05-25T11:59:51.801 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.723+0000 7f6937fff700 1 -- 172.21.15.102:0/193442493 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(13..13 src has 1..13) v4 ==== 3538+0+0 (unknown 110960687 0 0) 0x7f692803cec0 con 0x7f6944093620 2021-05-25T11:59:51.801 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.723+0000 7f6937fff700 10 client.4221.objecter ms_dispatch 0x7f6944001550 osd_map(13..13 src has 1..13) v4 2021-05-25T11:59:51.802 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.723+0000 7f6937fff700 3 client.4221.objecter handle_osd_map got epochs [13,13] > 0 2021-05-25T11:59:51.802 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.723+0000 7f6937fff700 3 client.4221.objecter handle_osd_map decoding full epoch 13 2021-05-25T11:59:51.802 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.723+0000 7f6937fff700 20 client.4221.objecter dump_active .. 0 homeless 2021-05-25T11:59:51.802 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.723+0000 7f69437fe700 10 monclient: get_auth_request con 0x7f6920033550 auth_method 0 2021-05-25T11:59:51.802 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.723+0000 7f6937fff700 10 client.4221.objecter ms_handle_connect 0x7f6920033550 2021-05-25T11:59:51.803 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.727+0000 7f6937fff700 1 -- 172.21.15.102:0/193442493 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7f6928012ba0 con 0x7f6944093620 2021-05-25T11:59:51.803 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.727+0000 7f6937fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:51.803 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.727+0000 7f6937fff700 10 monclient: _finish_command 1 = system:0 2021-05-25T11:59:51.879 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.878+0000 7f694a5a1700 10 start_mon_command cmd=[{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2021-05-25T11:59:51.880 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.878+0000 7f694a5a1700 10 monclient: _send_command 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2021-05-25T11:59:51.880 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.878+0000 7f694a5a1700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:51.880 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:51.878+0000 7f694a5a1700 1 -- 172.21.15.102:0/193442493 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f69240055e0 con 0x7f6944093620 2021-05-25T11:59:52.593 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.591+0000 7f6937fff700 1 -- 172.21.15.102:0/193442493 <== mon.0 v1:172.21.15.102:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v14) v1 ==== 173+0+0 (unknown 3289125425 0 0) 0x7f69280129c0 con 0x7f6944093620 2021-05-25T11:59:52.593 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.591+0000 7f6937fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2021-05-25T11:59:52.593 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.591+0000 7f6937fff700 10 monclient: _finish_command 2 = system:0 enabled application 'rbd' on pool 'rbd' 2021-05-25T11:59:52.596 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.594+0000 7f694a5a1700 10 librados: watch_flush enter 2021-05-25T11:59:52.597 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.595+0000 7f694a5a1700 10 librados: watch_flush exit 2021-05-25T11:59:52.597 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.595+0000 7f694a5a1700 20 client.4221.objecter shutdown clearing up homeless session... 2021-05-25T11:59:52.597 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.595+0000 7f694a5a1700 10 client.4221.objecter successfully canceled tick 2021-05-25T11:59:52.597 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.595+0000 7f694a5a1700 1 -- 172.21.15.102:0/193442493 >> v1:172.21.15.181:6812/29582 conn(0x7f6920033550 legacy=0x7f6920035a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:52.598 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.595+0000 7f694a5a1700 10 monclient: shutdown 2021-05-25T11:59:52.598 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.595+0000 7f694a5a1700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T11:59:52.598 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.595+0000 7f694a5a1700 1 -- 172.21.15.102:0/193442493 >> v1:172.21.15.102:6789/0 conn(0x7f6944093620 legacy=0x7f6944136ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:52.598 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.595+0000 7f694a5a1700 1 -- 172.21.15.102:0/193442493 shutdown_connections 2021-05-25T11:59:52.599 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.595+0000 7f694a5a1700 1 -- 172.21.15.102:0/193442493 >> 172.21.15.102:0/193442493 conn(0x7f6944097c50 msgr2=0x7f69440969e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T11:59:52.599 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.595+0000 7f694a5a1700 1 -- 172.21.15.102:0/193442493 shutdown_connections 2021-05-25T11:59:52.599 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.595+0000 7f694a5a1700 1 -- 172.21.15.102:0/193442493 wait complete. 2021-05-25T11:59:52.599 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.595+0000 7f694a5a1700 1 librados: shutdown 2021-05-25T11:59:52.600 INFO:teuthology.orchestra.run.smithi102.stderr:enabled application 'rbd' on pool 'rbd' 2021-05-25T11:59:52.612 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2021-05-25T11:59:52.613 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-05-25T11:59:52.614 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2021-05-25T11:59:52.727 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.726+0000 7f933fc9d700 10 monclient: get_monmap_and_config 2021-05-25T11:59:52.727 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.726+0000 7f933fc9d700 10 monclient: build_initial_monmap 2021-05-25T11:59:52.728 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.726+0000 7f933fc9d700 10 monclient: monmap: 2021-05-25T11:59:52.728 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T11:59:52.728 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:52.728 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:52.726980+0000 2021-05-25T11:59:52.728 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:52.726980+0000 2021-05-25T11:59:52.729 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T11:59:52.729 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:52.729 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T11:59:52.729 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T11:59:52.729 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T11:59:52.730 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:52.740 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.726+0000 7f933fc9d700 1 Processor -- start 2021-05-25T11:59:52.741 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.726+0000 7f933fc9d700 1 -- start start 2021-05-25T11:59:52.741 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.726+0000 7f933fc9d700 10 monclient: init 2021-05-25T11:59:52.741 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.727+0000 7f933fc9d700 10 monclient: _reopen_session rank -1 2021-05-25T11:59:52.741 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.727+0000 7f933fc9d700 10 monclient: _add_conns ranks=[1,2,0] 2021-05-25T11:59:52.742 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.727+0000 7f933fc9d700 10 monclient(hunting): picked mon.noname-c con 0x7f933809a640 addr v1:172.21.15.181:6789/0 2021-05-25T11:59:52.742 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.727+0000 7f933fc9d700 10 monclient(hunting): picked mon.noname-b con 0x7f9338063bb0 addr v1:172.21.15.102:6790/0 2021-05-25T11:59:52.742 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.727+0000 7f933fc9d700 10 monclient(hunting): picked mon.noname-a con 0x7f93381325c0 addr v1:172.21.15.102:6789/0 2021-05-25T11:59:52.742 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.727+0000 7f933fc9d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9338096b20 con 0x7f93381325c0 2021-05-25T11:59:52.743 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.727+0000 7f933fc9d700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f933809a210 con 0x7f933809a640 2021-05-25T11:59:52.743 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.727+0000 7f933fc9d700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f933809b460 con 0x7f9338063bb0 2021-05-25T11:59:52.743 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.727+0000 7f933fc9d700 10 monclient(hunting): _renew_subs 2021-05-25T11:59:52.743 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.727+0000 7f933fc9d700 10 monclient(hunting): authenticate will time out at 793.324290s 2021-05-25T11:59:52.743 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.727+0000 7f933d238700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f9338063bb0 0x7f9338064030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:53572/0 (socket says 172.21.15.102:53572) 2021-05-25T11:59:52.744 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.727+0000 7f933e23a700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f93381325c0 0x7f93381349b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:58696/0 (socket says 172.21.15.102:58696) 2021-05-25T11:59:52.744 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.727+0000 7f933d238700 1 -- 172.21.15.102:0/1985240008 learned_addr learned my addr 172.21.15.102:0/1985240008 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T11:59:52.744 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.727+0000 7f933ca37700 1 -- 172.21.15.102:0/1985240008 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3321622283 0 0) 0x7f933809b460 con 0x7f9338063bb0 2021-05-25T11:59:52.744 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.727+0000 7f933ca37700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:52.745 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.727+0000 7f933ca37700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:52.745 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.727+0000 7f933ca37700 10 monclient(hunting): my global_id is 4217 2021-05-25T11:59:52.745 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.727+0000 7f933ca37700 1 -- 172.21.15.102:0/1985240008 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9320003270 con 0x7f9338063bb0 2021-05-25T11:59:52.745 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.727+0000 7f933ca37700 1 -- 172.21.15.102:0/1985240008 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 435818066 0 0) 0x7f9338096b20 con 0x7f93381325c0 2021-05-25T11:59:52.745 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.727+0000 7f933ca37700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:52.746 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.727+0000 7f933ca37700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:52.746 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.727+0000 7f933ca37700 10 monclient(hunting): my global_id is 4224 2021-05-25T11:59:52.746 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.727+0000 7f933ca37700 1 -- 172.21.15.102:0/1985240008 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f933809b460 con 0x7f93381325c0 2021-05-25T11:59:52.746 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.727+0000 7f933ca37700 1 -- 172.21.15.102:0/1985240008 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 280425699 0 0) 0x7f933809a210 con 0x7f933809a640 2021-05-25T11:59:52.747 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.727+0000 7f933ca37700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:52.747 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.727+0000 7f933ca37700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:52.747 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.727+0000 7f933ca37700 10 monclient(hunting): my global_id is 4219 2021-05-25T11:59:52.747 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.727+0000 7f933ca37700 1 -- 172.21.15.102:0/1985240008 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9338096b20 con 0x7f933809a640 2021-05-25T11:59:52.747 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.727+0000 7f933ca37700 1 -- 172.21.15.102:0/1985240008 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1640365924 0 0) 0x7f9320003270 con 0x7f9338063bb0 2021-05-25T11:59:52.748 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.727+0000 7f933ca37700 1 -- 172.21.15.102:0/1985240008 >> v1:172.21.15.181:6789/0 conn(0x7f933809a640 legacy=0x7f93380634a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:52.748 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.727+0000 7f933ca37700 1 -- 172.21.15.102:0/1985240008 >> v1:172.21.15.102:6789/0 conn(0x7f93381325c0 legacy=0x7f93381349b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:52.748 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.727+0000 7f933ca37700 10 monclient: _finish_hunting 0 2021-05-25T11:59:52.748 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.727+0000 7f933ca37700 1 monclient: found mon.noname-b 2021-05-25T11:59:52.748 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.727+0000 7f933ca37700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T11:59:52.749 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.727+0000 7f933ca37700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T11:59:52.749 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.727+0000 7f933ca37700 1 -- 172.21.15.102:0/1985240008 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93380964c0 con 0x7f9338063bb0 2021-05-25T11:59:52.749 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.727+0000 7f933ca37700 10 monclient: _finish_auth 0 2021-05-25T11:59:52.749 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.727+0000 7f933ca37700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T11:59:52.750 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.727+0000 7f933ca37700 1 -- 172.21.15.102:0/1985240008 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f9328001610 con 0x7f9338063bb0 2021-05-25T11:59:52.750 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.727+0000 7f933fc9d700 5 monclient: authenticate success, global_id 4217 2021-05-25T11:59:52.750 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.727+0000 7f933fc9d700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T11:59:52.750 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.727+0000 7f933ca37700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:52.750 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.727+0000 7f933ca37700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T11:59:52.751 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.727+0000 7f933ca37700 10 monclient: dump: 2021-05-25T11:59:52.751 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:52.751 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:52.751 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:52.752 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:52.752 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:52.752 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:52.752 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:52.752 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:52.753 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:52.753 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:52.753 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.728+0000 7f933fc9d700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T11:59:52.754 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.728+0000 7f933ca37700 1 -- 172.21.15.102:0/1985240008 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f93280018f0 con 0x7f9338063bb0 2021-05-25T11:59:52.754 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.728+0000 7f933ca37700 10 monclient: handle_config config(0 keys) v1 2021-05-25T11:59:52.754 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.728+0000 7f933ca37700 1 -- 172.21.15.102:0/1985240008 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f9328001c80 con 0x7f9338063bb0 2021-05-25T11:59:52.754 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.728+0000 7f933fc9d700 10 monclient: get_monmap_and_config success 2021-05-25T11:59:52.755 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.728+0000 7f933fc9d700 4 set_mon_vals no callback set 2021-05-25T11:59:52.755 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.728+0000 7f933ca37700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:52.755 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.728+0000 7f933ca37700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T11:59:52.755 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.728+0000 7f933ca37700 10 monclient: dump: 2021-05-25T11:59:52.755 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:52.756 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:52.756 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:52.756 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:52.756 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:52.756 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:52.757 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:52.757 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:52.757 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:52.757 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:52.757 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.728+0000 7f933fc9d700 10 monclient: shutdown 2021-05-25T11:59:52.758 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.728+0000 7f933fc9d700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T11:59:52.758 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.728+0000 7f933fc9d700 1 -- 172.21.15.102:0/1985240008 >> v1:172.21.15.102:6790/0 conn(0x7f9338063bb0 legacy=0x7f9338064030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:52.758 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.728+0000 7f933fc9d700 1 -- 172.21.15.102:0/1985240008 shutdown_connections 2021-05-25T11:59:52.758 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.728+0000 7f933fc9d700 1 -- 172.21.15.102:0/1985240008 >> 172.21.15.102:0/1985240008 conn(0x7f933809ef50 msgr2=0x7f933809f360 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T11:59:52.759 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.728+0000 7f933fc9d700 1 -- 172.21.15.102:0/1985240008 shutdown_connections 2021-05-25T11:59:52.759 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.728+0000 7f933fc9d700 1 -- 172.21.15.102:0/1985240008 wait complete. 2021-05-25T11:59:52.759 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.728+0000 7f933fc9d700 10 monclient: build_initial_monmap 2021-05-25T11:59:52.759 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.729+0000 7f933fc9d700 10 monclient: monmap: 2021-05-25T11:59:52.759 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T11:59:52.760 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T11:59:52.760 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T11:59:52.760 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T11:59:52.760 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T11:59:52.760 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T11:59:52.761 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T11:59:52.761 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T11:59:52.761 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T11:59:52.761 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:52.762 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.729+0000 7f933fc9d700 1 librados: starting msgr at 2021-05-25T11:59:52.762 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.729+0000 7f933fc9d700 1 librados: starting objecter 2021-05-25T11:59:52.762 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.729+0000 7f933fc9d700 1 Processor -- start 2021-05-25T11:59:52.762 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.729+0000 7f933fc9d700 1 -- start start 2021-05-25T11:59:52.762 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.729+0000 7f933fc9d700 1 librados: setting wanted keys 2021-05-25T11:59:52.763 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.729+0000 7f933fc9d700 1 librados: calling monclient init 2021-05-25T11:59:52.763 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.729+0000 7f933fc9d700 10 monclient: init 2021-05-25T11:59:52.763 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.729+0000 7f933fc9d700 10 monclient: _reopen_session rank -1 2021-05-25T11:59:52.763 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.729+0000 7f933fc9d700 10 monclient: _add_conns ranks=[2,0,1] 2021-05-25T11:59:52.763 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.729+0000 7f933fc9d700 10 monclient(hunting): picked mon.noname-c con 0x7f9338063bb0 addr v1:172.21.15.102:6790/0 2021-05-25T11:59:52.764 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.729+0000 7f933fc9d700 10 monclient(hunting): picked mon.noname-a con 0x7f933809a640 addr v1:172.21.15.102:6789/0 2021-05-25T11:59:52.764 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.729+0000 7f933fc9d700 10 monclient(hunting): picked mon.noname-b con 0x7f93381325c0 addr v1:172.21.15.181:6789/0 2021-05-25T11:59:52.764 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.729+0000 7f933fc9d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9338095030 con 0x7f933809a640 2021-05-25T11:59:52.764 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.729+0000 7f933fc9d700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9338115f00 con 0x7f93381325c0 2021-05-25T11:59:52.764 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.729+0000 7f933fc9d700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93381161a0 con 0x7f9338063bb0 2021-05-25T11:59:52.765 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.729+0000 7f933fc9d700 10 monclient(hunting): _renew_subs 2021-05-25T11:59:52.765 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.729+0000 7f933fc9d700 10 monclient(hunting): authenticate will time out at 793.326827s 2021-05-25T11:59:52.765 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.729+0000 7f933da39700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f9338063bb0 0x7f93380922a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:53576/0 (socket says 172.21.15.102:53576) 2021-05-25T11:59:52.765 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.729+0000 7f933da39700 1 -- 172.21.15.102:0/3462805309 learned_addr learned my addr 172.21.15.102:0/3462805309 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T11:59:52.766 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.729+0000 7f9325ffb700 10 client.?.objecter ms_handle_connect 0x7f933809a640 2021-05-25T11:59:52.766 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.729+0000 7f9325ffb700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:52.766 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.729+0000 7f9325ffb700 10 client.?.objecter ms_handle_connect 0x7f9338063bb0 2021-05-25T11:59:52.766 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.729+0000 7f9325ffb700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:52.766 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.730+0000 7f9325ffb700 1 -- 172.21.15.102:0/3462805309 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2613205868 0 0) 0x7f9338095030 con 0x7f933809a640 2021-05-25T11:59:52.767 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.730+0000 7f9325ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:52.767 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.730+0000 7f9325ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:52.767 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.730+0000 7f9325ffb700 10 monclient(hunting): my global_id is 4227 2021-05-25T11:59:52.767 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.730+0000 7f9325ffb700 1 -- 172.21.15.102:0/3462805309 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9318004c10 con 0x7f933809a640 2021-05-25T11:59:52.768 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.730+0000 7f9325ffb700 10 client.?.objecter ms_handle_connect 0x7f93381325c0 2021-05-25T11:59:52.768 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.730+0000 7f9325ffb700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:52.768 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.730+0000 7f9325ffb700 1 -- 172.21.15.102:0/3462805309 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3863222736 0 0) 0x7f93381161a0 con 0x7f9338063bb0 2021-05-25T11:59:52.768 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.730+0000 7f9325ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:52.768 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.730+0000 7f9325ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:52.769 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.730+0000 7f9325ffb700 10 monclient(hunting): my global_id is 4220 2021-05-25T11:59:52.769 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.730+0000 7f9325ffb700 1 -- 172.21.15.102:0/3462805309 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9338095030 con 0x7f9338063bb0 2021-05-25T11:59:52.769 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.730+0000 7f9325ffb700 1 -- 172.21.15.102:0/3462805309 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1675758283 0 0) 0x7f9338115f00 con 0x7f93381325c0 2021-05-25T11:59:52.769 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.730+0000 7f9325ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:52.770 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.730+0000 7f9325ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:52.770 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.730+0000 7f9325ffb700 10 monclient(hunting): my global_id is 4222 2021-05-25T11:59:52.770 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.730+0000 7f9325ffb700 1 -- 172.21.15.102:0/3462805309 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93381161a0 con 0x7f93381325c0 2021-05-25T11:59:52.770 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.730+0000 7f9325ffb700 1 -- 172.21.15.102:0/3462805309 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1860443533 0 0) 0x7f9318004c10 con 0x7f933809a640 2021-05-25T11:59:52.770 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.730+0000 7f9325ffb700 1 -- 172.21.15.102:0/3462805309 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9338115f00 con 0x7f933809a640 2021-05-25T11:59:52.771 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.730+0000 7f9325ffb700 1 -- 172.21.15.102:0/3462805309 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f93280015d0 con 0x7f933809a640 2021-05-25T11:59:52.771 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.730+0000 7f9325ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:52.771 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.730+0000 7f9325ffb700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T11:59:52.771 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.730+0000 7f9325ffb700 10 monclient(hunting): dump: 2021-05-25T11:59:52.771 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:52.772 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:52.772 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:52.772 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:52.772 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:52.773 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:52.773 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:52.773 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:52.773 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:52.773 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:52.774 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.730+0000 7f9325ffb700 10 monclient(hunting): _finish_auth 0 2021-05-25T11:59:52.774 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.730+0000 7f9325ffb700 1 -- 172.21.15.102:0/3462805309 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 945917945 0 0) 0x7f9338095030 con 0x7f9338063bb0 2021-05-25T11:59:52.774 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.730+0000 7f9325ffb700 1 -- 172.21.15.102:0/3462805309 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9318004c10 con 0x7f9338063bb0 2021-05-25T11:59:52.774 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.730+0000 7f9325ffb700 1 -- 172.21.15.102:0/3462805309 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f932c001980 con 0x7f9338063bb0 2021-05-25T11:59:52.775 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.730+0000 7f9325ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:52.775 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.730+0000 7f9325ffb700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2021-05-25T11:59:52.775 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.730+0000 7f9325ffb700 10 monclient(hunting): dump: 2021-05-25T11:59:52.775 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:52.775 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:52.776 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:52.776 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:52.776 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:52.776 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:52.777 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:52.777 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:52.777 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:52.777 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:52.777 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.730+0000 7f9325ffb700 1 -- 172.21.15.102:0/3462805309 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 545587540 0 0) 0x7f93381161a0 con 0x7f93381325c0 2021-05-25T11:59:52.778 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.730+0000 7f9325ffb700 1 -- 172.21.15.102:0/3462805309 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9338095030 con 0x7f93381325c0 2021-05-25T11:59:52.778 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.730+0000 7f9325ffb700 1 -- 172.21.15.102:0/3462805309 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f9334001d50 con 0x7f93381325c0 2021-05-25T11:59:52.778 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.730+0000 7f9325ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:52.778 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.730+0000 7f9325ffb700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2021-05-25T11:59:52.779 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.730+0000 7f9325ffb700 10 monclient(hunting): dump: 2021-05-25T11:59:52.779 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:52.779 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:52.779 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:52.779 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:52.780 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:52.780 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:52.780 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:52.780 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:52.780 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:52.781 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:52.781 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.730+0000 7f9325ffb700 1 -- 172.21.15.102:0/3462805309 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1147102252 0 0) 0x7f9338115f00 con 0x7f933809a640 2021-05-25T11:59:52.781 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.730+0000 7f9325ffb700 1 -- 172.21.15.102:0/3462805309 >> v1:172.21.15.102:6790/0 conn(0x7f9338063bb0 legacy=0x7f93380922a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:52.781 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.730+0000 7f9325ffb700 1 -- 172.21.15.102:0/3462805309 >> v1:172.21.15.181:6789/0 conn(0x7f93381325c0 legacy=0x7f93380930c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:52.782 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.730+0000 7f9325ffb700 10 monclient: _finish_hunting 0 2021-05-25T11:59:52.782 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.730+0000 7f9325ffb700 1 monclient: found mon.a 2021-05-25T11:59:52.782 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.730+0000 7f9325ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T11:59:52.782 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.730+0000 7f9325ffb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:52.782 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.730+0000 7f9325ffb700 1 -- 172.21.15.102:0/3462805309 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9338116440 con 0x7f933809a640 2021-05-25T11:59:52.783 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.730+0000 7f9325ffb700 10 monclient: _finish_auth 0 2021-05-25T11:59:52.783 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.730+0000 7f9325ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T11:59:52.783 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.730+0000 7f933fc9d700 5 monclient: authenticate success, global_id 4227 2021-05-25T11:59:52.783 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.730+0000 7f933fc9d700 10 monclient: _renew_subs 2021-05-25T11:59:52.783 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.730+0000 7f933fc9d700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:52.784 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.730+0000 7f933fc9d700 1 -- 172.21.15.102:0/3462805309 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f933812b350 con 0x7f933809a640 2021-05-25T11:59:52.784 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.730+0000 7f9325ffb700 1 -- 172.21.15.102:0/3462805309 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9328004940 con 0x7f933809a640 2021-05-25T11:59:52.784 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.730+0000 7f9325ffb700 10 monclient: handle_config config(0 keys) v1 2021-05-25T11:59:52.784 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.730+0000 7f9325ffb700 1 -- 172.21.15.102:0/3462805309 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f9328004c50 con 0x7f933809a640 2021-05-25T11:59:52.784 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.730+0000 7f9325ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:52.785 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.730+0000 7f9325ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T11:59:52.785 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.730+0000 7f9325ffb700 10 monclient: dump: 2021-05-25T11:59:52.785 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:52.785 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:52.785 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:52.786 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:52.786 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:52.786 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:52.786 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:52.787 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:52.787 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:52.787 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:52.787 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.730+0000 7f933fc9d700 10 client.4227.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T11:59:52.787 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.730+0000 7f9327fff700 4 set_mon_vals no callback set 2021-05-25T11:59:52.788 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.730+0000 7f933fc9d700 10 monclient: _renew_subs 2021-05-25T11:59:52.788 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.730+0000 7f933fc9d700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:52.788 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.730+0000 7f933fc9d700 1 -- 172.21.15.102:0/3462805309 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9338144a90 con 0x7f933809a640 2021-05-25T11:59:52.788 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.730+0000 7f933fc9d700 1 librados: init done 2021-05-25T11:59:52.789 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.731+0000 7f9325ffb700 1 -- 172.21.15.102:0/3462805309 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f932800df70 con 0x7f933809a640 2021-05-25T11:59:52.789 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.731+0000 7f9325ffb700 1 -- 172.21.15.102:0/3462805309 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(14..14 src has 1..14) v4 ==== 3549+0+0 (unknown 1641242595 0 0) 0x7f932803cee0 con 0x7f933809a640 2021-05-25T11:59:52.789 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.731+0000 7f9325ffb700 10 client.4227.objecter ms_dispatch 0x7f9338001550 osd_map(14..14 src has 1..14) v4 2021-05-25T11:59:52.789 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.731+0000 7f9325ffb700 3 client.4227.objecter handle_osd_map got epochs [14,14] > 0 2021-05-25T11:59:52.789 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.731+0000 7f9325ffb700 3 client.4227.objecter handle_osd_map decoding full epoch 14 2021-05-25T11:59:52.790 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.731+0000 7f9325ffb700 20 client.4227.objecter dump_active .. 0 homeless 2021-05-25T11:59:52.790 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.731+0000 7f933fc9d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:52.790 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.731+0000 7f933fc9d700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:52.790 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.731+0000 7f933fc9d700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:52.791 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.731+0000 7f933fc9d700 1 -- 172.21.15.102:0/3462805309 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9338099cb0 con 0x7f933809a640 2021-05-25T11:59:52.791 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.731+0000 7f933da39700 10 monclient: get_auth_request con 0x7f9318033540 auth_method 0 2021-05-25T11:59:52.791 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.732+0000 7f9325ffb700 10 client.4227.objecter ms_handle_connect 0x7f9318033540 2021-05-25T11:59:52.791 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.735+0000 7f9325ffb700 1 -- 172.21.15.102:0/3462805309 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7f9328012c70 con 0x7f933809a640 2021-05-25T11:59:52.792 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.735+0000 7f9325ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:52.792 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.735+0000 7f9325ffb700 10 monclient: _finish_command 1 = system:0 2021-05-25T11:59:52.887 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.885+0000 7f933fc9d700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2021-05-25T11:59:52.887 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.886+0000 7f933fc9d700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2021-05-25T11:59:52.888 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.886+0000 7f933fc9d700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:52.888 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.886+0000 7f933fc9d700 1 -- 172.21.15.102:0/3462805309 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9338059730 con 0x7f933809a640 2021-05-25T11:59:52.889 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.886+0000 7f9325ffb700 1 -- 172.21.15.102:0/3462805309 <== mon.0 v1:172.21.15.102:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) v1 ==== 74+0+10037 (unknown 92231148 0 3395287620) 0x7f9328012a90 con 0x7f933809a640 2021-05-25T11:59:52.889 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.886+0000 7f9325ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2021-05-25T11:59:52.889 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.886+0000 7f9325ffb700 10 monclient: _finish_command 2 = system:0 2021-05-25T11:59:52.890 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:59:52.890 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":14,"fsid":"20ee91b6-8935-4b98-b868-85c03789ccc4","created":"2021-05-25T11:59:41.238908+0000","modified":"2021-05-25T11:59:52.587219+0000","last_up_change":"2021-05-25T11:59:49.572449+0000","last_in_change":"2021-05-25T11:59:42.972556+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-05-25T11:59:49.609415+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-05-25T11:59:50.997242+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"8f1f285f-7041-4c53-ab09-f638ab670036","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6807","nonce":29973}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6809","nonce":29973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6811","nonce":29973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6810","nonce":29973}]},"public_addr":"172.21.15.102:6807/29973","cluster_addr":"172.21.15.102:6809/29973","heartbeat_back_addr":"172.21.15.102:6811/29973","heartbeat_front_addr":"172.21.15.102:6810/29973","state":["exists","up"]},{"osd":1,"uuid":"d0d92491-3d97-4e4e-acf4-9dc7c0e4701f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6802","nonce":29974}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6804","nonce":29974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6808","nonce":29974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6806","nonce":29974}]},"public_addr":"172.21.15.102:6802/29974","cluster_addr":"172.21.15.102:6804/29974","heartbeat_back_addr":"172.21.15.102:6808/29974","heartbeat_front_addr":"172.21.15.102:6806/29974","state":["exists","up"]},{"osd":2,"uuid":"80a3eb90-b36e-4035-af91-3917fbf8dc2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6800","nonce":29975}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6801","nonce":29975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6805","nonce":29975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6803","nonce":29975}]},"public_addr":"172.21.15.102:6800/29975","cluster_addr":"172.21.15.102:6801/29975","heartbeat_back_addr":"172.21.15.102:6805/29975","heartbeat_front_addr":"172.21.15.102:6803/29975","state":["exists","up"]},{"osd":3,"uuid":"45889d6a-dcfb-4670-afef-cee9bbb0476a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6804","nonce":30005}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6805","nonce":30005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6807","nonce":30005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6806","nonce":30005}]},"public_addr":"172.21.15.181:6804/30005","cluster_addr":"172.21.15.181:6805/30005","heartbeat_back_addr":"172.21.15.181:6807/30005","heartbeat_front_addr":"172.21.15.181:6806/30005","state":["exists","up"]},{"osd":4,"uuid":"69e097b7-1472-442f-af42-225c0e40ad5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6800","nonce":30006}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6801","nonce":30006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6803","nonce":30006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6802","nonce":30006}]},"public_addr":"172.21.15.181:6800/30006","cluster_addr":"172.21.15.181:6801/30006","heartbeat_back_addr":"172.21.15.181:6803/30006","heartbeat_front_addr":"172.21.15.181:6802/30006","state":["exists","up"]},{"osd":5,"uuid":"f1045691-db58-471e-943a-e4d99129a917","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6808","nonce":30008}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6809","nonce":30008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6811","nonce":30008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6810","nonce":30008}]},"public_addr":"172.21.15.181:6808/30008","cluster_addr":"172.21.15.181:6809/30008","heartbeat_back_addr":"172.21.15.181:6811/30008","heartbeat_front_addr":"172.21.15.181:6810/30008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-05-25T11:59:48.129055+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-05-25T11:59:48.090298+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-05-25T11:59:48.049016+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-05-25T11:59:48.069527+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-05-25T11:59:48.002930+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-05-25T11:59:48.052367+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-05-25T11:59:52.891 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.888+0000 7f933fc9d700 10 librados: watch_flush enter 2021-05-25T11:59:52.892 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.888+0000 7f933fc9d700 10 librados: watch_flush exit 2021-05-25T11:59:52.892 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.888+0000 7f933fc9d700 20 client.4227.objecter shutdown clearing up homeless session... 2021-05-25T11:59:52.892 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.888+0000 7f933fc9d700 10 client.4227.objecter successfully canceled tick 2021-05-25T11:59:52.892 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.888+0000 7f933fc9d700 1 -- 172.21.15.102:0/3462805309 >> v1:172.21.15.181:6812/29582 conn(0x7f9318033540 legacy=0x7f9318035a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:52.892 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.888+0000 7f933fc9d700 10 monclient: shutdown 2021-05-25T11:59:52.893 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.888+0000 7f933fc9d700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T11:59:52.893 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.888+0000 7f933fc9d700 1 -- 172.21.15.102:0/3462805309 >> v1:172.21.15.102:6789/0 conn(0x7f933809a640 legacy=0x7f93380929b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:52.893 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.888+0000 7f933fc9d700 1 -- 172.21.15.102:0/3462805309 shutdown_connections 2021-05-25T11:59:52.893 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.888+0000 7f933fc9d700 1 -- 172.21.15.102:0/3462805309 >> 172.21.15.102:0/3462805309 conn(0x7f933809ef50 msgr2=0x7f933809b700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T11:59:52.894 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.888+0000 7f933fc9d700 1 -- 172.21.15.102:0/3462805309 shutdown_connections 2021-05-25T11:59:52.894 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.888+0000 7f933fc9d700 1 -- 172.21.15.102:0/3462805309 wait complete. 2021-05-25T11:59:52.894 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:52.888+0000 7f933fc9d700 1 librados: shutdown 2021-05-25T11:59:52.901 INFO:tasks.ceph.ceph_manager.ceph:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2021-05-25T11:59:49.609415+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '14', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2021-05-25T11:59:50.997242+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '14', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-05-25T11:59:52.901 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2021-05-25T11:59:53.012 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.010+0000 7ff6ef1ac700 10 monclient: get_monmap_and_config 2021-05-25T11:59:53.012 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.011+0000 7ff6ef1ac700 10 monclient: build_initial_monmap 2021-05-25T11:59:53.013 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.011+0000 7ff6ef1ac700 10 monclient: monmap: 2021-05-25T11:59:53.013 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T11:59:53.013 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:53.013 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:53.011904+0000 2021-05-25T11:59:53.014 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:53.011904+0000 2021-05-25T11:59:53.014 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T11:59:53.014 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:53.015 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T11:59:53.015 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T11:59:53.015 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T11:59:53.015 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:53.026 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.011+0000 7ff6ef1ac700 1 Processor -- start 2021-05-25T11:59:53.026 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.011+0000 7ff6ef1ac700 1 -- start start 2021-05-25T11:59:53.027 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.011+0000 7ff6ef1ac700 10 monclient: init 2021-05-25T11:59:53.027 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.012+0000 7ff6ef1ac700 10 monclient: _reopen_session rank -1 2021-05-25T11:59:53.027 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.012+0000 7ff6ef1ac700 10 monclient: _add_conns ranks=[1,2,0] 2021-05-25T11:59:53.027 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.012+0000 7ff6ef1ac700 10 monclient(hunting): picked mon.noname-c con 0x7ff6e812c1d0 addr v1:172.21.15.181:6789/0 2021-05-25T11:59:53.028 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.012+0000 7ff6ef1ac700 10 monclient(hunting): picked mon.noname-b con 0x7ff6e8068da0 addr v1:172.21.15.102:6790/0 2021-05-25T11:59:53.028 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.012+0000 7ff6ef1ac700 10 monclient(hunting): picked mon.noname-a con 0x7ff6e8069220 addr v1:172.21.15.102:6789/0 2021-05-25T11:59:53.028 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.012+0000 7ff6ef1ac700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6e8097930 con 0x7ff6e8069220 2021-05-25T11:59:53.028 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.012+0000 7ff6ef1ac700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6e8097bd0 con 0x7ff6e812c1d0 2021-05-25T11:59:53.028 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.012+0000 7ff6ef1ac700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6e8097e70 con 0x7ff6e8068da0 2021-05-25T11:59:53.029 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.012+0000 7ff6ef1ac700 10 monclient(hunting): _renew_subs 2021-05-25T11:59:53.029 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.012+0000 7ff6ef1ac700 10 monclient(hunting): authenticate will time out at 793.609247s 2021-05-25T11:59:53.029 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.012+0000 7ff6dffff700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7ff6e8068da0 0x7ff6e8130af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:53588/0 (socket says 172.21.15.102:53588) 2021-05-25T11:59:53.029 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.012+0000 7ff6ed749700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7ff6e8069220 0x7ff6e8134220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:58712/0 (socket says 172.21.15.102:58712) 2021-05-25T11:59:53.029 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.012+0000 7ff6dffff700 1 -- 172.21.15.102:0/2340997794 learned_addr learned my addr 172.21.15.102:0/2340997794 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T11:59:53.030 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.012+0000 7ff6df7fe700 1 -- 172.21.15.102:0/2340997794 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 114753144 0 0) 0x7ff6e8097930 con 0x7ff6e8069220 2021-05-25T11:59:53.030 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.012+0000 7ff6df7fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:53.030 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.012+0000 7ff6df7fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:53.030 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.012+0000 7ff6df7fe700 10 monclient(hunting): my global_id is 4230 2021-05-25T11:59:53.031 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.012+0000 7ff6df7fe700 1 -- 172.21.15.102:0/2340997794 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6d4003270 con 0x7ff6e8069220 2021-05-25T11:59:53.031 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.012+0000 7ff6df7fe700 1 -- 172.21.15.102:0/2340997794 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1422119929 0 0) 0x7ff6e8097e70 con 0x7ff6e8068da0 2021-05-25T11:59:53.031 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.012+0000 7ff6df7fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:53.031 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.012+0000 7ff6df7fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:53.031 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.012+0000 7ff6df7fe700 10 monclient(hunting): my global_id is 4223 2021-05-25T11:59:53.032 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.012+0000 7ff6df7fe700 1 -- 172.21.15.102:0/2340997794 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6e8097930 con 0x7ff6e8068da0 2021-05-25T11:59:53.032 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.012+0000 7ff6df7fe700 1 -- 172.21.15.102:0/2340997794 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 843712920 0 0) 0x7ff6e8097bd0 con 0x7ff6e812c1d0 2021-05-25T11:59:53.032 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.012+0000 7ff6df7fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:53.032 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.012+0000 7ff6df7fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:53.032 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.012+0000 7ff6df7fe700 10 monclient(hunting): my global_id is 4225 2021-05-25T11:59:53.033 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.012+0000 7ff6df7fe700 1 -- 172.21.15.102:0/2340997794 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6e8097e70 con 0x7ff6e812c1d0 2021-05-25T11:59:53.033 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.012+0000 7ff6df7fe700 1 -- 172.21.15.102:0/2340997794 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1674808668 0 0) 0x7ff6e8097930 con 0x7ff6e8068da0 2021-05-25T11:59:53.033 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.012+0000 7ff6df7fe700 1 -- 172.21.15.102:0/2340997794 >> v1:172.21.15.181:6789/0 conn(0x7ff6e812c1d0 legacy=0x7ff6e812b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:53.033 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.012+0000 7ff6df7fe700 1 -- 172.21.15.102:0/2340997794 >> v1:172.21.15.102:6789/0 conn(0x7ff6e8069220 legacy=0x7ff6e8134220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:53.034 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.012+0000 7ff6df7fe700 10 monclient: _finish_hunting 0 2021-05-25T11:59:53.034 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.012+0000 7ff6df7fe700 1 monclient: found mon.noname-b 2021-05-25T11:59:53.034 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.012+0000 7ff6df7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T11:59:53.034 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.012+0000 7ff6df7fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T11:59:53.034 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.012+0000 7ff6df7fe700 1 -- 172.21.15.102:0/2340997794 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6e8069840 con 0x7ff6e8068da0 2021-05-25T11:59:53.035 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.012+0000 7ff6df7fe700 10 monclient: _finish_auth 0 2021-05-25T11:59:53.035 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.012+0000 7ff6df7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T11:59:53.035 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.012+0000 7ff6ef1ac700 5 monclient: authenticate success, global_id 4223 2021-05-25T11:59:53.035 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.012+0000 7ff6df7fe700 1 -- 172.21.15.102:0/2340997794 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7ff6d00015f0 con 0x7ff6e8068da0 2021-05-25T11:59:53.036 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.012+0000 7ff6df7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:53.036 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.012+0000 7ff6df7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T11:59:53.036 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.012+0000 7ff6df7fe700 10 monclient: dump: 2021-05-25T11:59:53.036 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:53.036 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:53.037 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:53.037 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:53.037 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:53.037 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:53.037 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:53.038 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:53.038 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:53.038 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:53.038 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.012+0000 7ff6ef1ac700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T11:59:53.039 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.012+0000 7ff6df7fe700 1 -- 172.21.15.102:0/2340997794 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff6d0001820 con 0x7ff6e8068da0 2021-05-25T11:59:53.039 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.012+0000 7ff6df7fe700 10 monclient: handle_config config(0 keys) v1 2021-05-25T11:59:53.039 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.012+0000 7ff6ef1ac700 10 monclient: get_monmap_and_config success 2021-05-25T11:59:53.039 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.012+0000 7ff6ef1ac700 4 set_mon_vals no callback set 2021-05-25T11:59:53.039 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.012+0000 7ff6df7fe700 1 -- 172.21.15.102:0/2340997794 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7ff6d0001bb0 con 0x7ff6e8068da0 2021-05-25T11:59:53.040 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.012+0000 7ff6df7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:53.040 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.012+0000 7ff6df7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T11:59:53.040 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.012+0000 7ff6df7fe700 10 monclient: dump: 2021-05-25T11:59:53.040 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:53.041 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:53.041 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:53.041 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:53.041 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:53.041 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:53.042 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:53.042 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:53.042 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:53.042 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:53.043 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.013+0000 7ff6ef1ac700 10 monclient: shutdown 2021-05-25T11:59:53.043 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.013+0000 7ff6ef1ac700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T11:59:53.043 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.013+0000 7ff6ef1ac700 1 -- 172.21.15.102:0/2340997794 >> v1:172.21.15.102:6790/0 conn(0x7ff6e8068da0 legacy=0x7ff6e8130af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:53.043 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.013+0000 7ff6ef1ac700 1 -- 172.21.15.102:0/2340997794 shutdown_connections 2021-05-25T11:59:53.043 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.013+0000 7ff6ef1ac700 1 -- 172.21.15.102:0/2340997794 >> 172.21.15.102:0/2340997794 conn(0x7ff6e80984c0 msgr2=0x7ff6e80988d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T11:59:53.044 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.013+0000 7ff6ef1ac700 1 -- 172.21.15.102:0/2340997794 shutdown_connections 2021-05-25T11:59:53.044 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.013+0000 7ff6ef1ac700 1 -- 172.21.15.102:0/2340997794 wait complete. 2021-05-25T11:59:53.044 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.013+0000 7ff6ef1ac700 10 monclient: build_initial_monmap 2021-05-25T11:59:53.044 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.013+0000 7ff6ef1ac700 10 monclient: monmap: 2021-05-25T11:59:53.045 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T11:59:53.045 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T11:59:53.045 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T11:59:53.045 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T11:59:53.045 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T11:59:53.046 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T11:59:53.046 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T11:59:53.046 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T11:59:53.046 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T11:59:53.046 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:53.047 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.014+0000 7ff6ef1ac700 1 librados: starting msgr at 2021-05-25T11:59:53.047 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.014+0000 7ff6ef1ac700 1 librados: starting objecter 2021-05-25T11:59:53.047 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.014+0000 7ff6ef1ac700 1 Processor -- start 2021-05-25T11:59:53.047 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.014+0000 7ff6ef1ac700 1 -- start start 2021-05-25T11:59:53.048 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.014+0000 7ff6ef1ac700 1 librados: setting wanted keys 2021-05-25T11:59:53.048 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.014+0000 7ff6ef1ac700 1 librados: calling monclient init 2021-05-25T11:59:53.048 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.014+0000 7ff6ef1ac700 10 monclient: init 2021-05-25T11:59:53.048 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.014+0000 7ff6ef1ac700 10 monclient: _reopen_session rank -1 2021-05-25T11:59:53.048 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.014+0000 7ff6ef1ac700 10 monclient: _add_conns ranks=[2,0,1] 2021-05-25T11:59:53.048 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.014+0000 7ff6ef1ac700 10 monclient(hunting): picked mon.noname-c con 0x7ff6e8068da0 addr v1:172.21.15.102:6790/0 2021-05-25T11:59:53.049 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.014+0000 7ff6ef1ac700 10 monclient(hunting): picked mon.noname-a con 0x7ff6e8069220 addr v1:172.21.15.102:6789/0 2021-05-25T11:59:53.049 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.014+0000 7ff6ef1ac700 10 monclient(hunting): picked mon.noname-b con 0x7ff6e812c1d0 addr v1:172.21.15.181:6789/0 2021-05-25T11:59:53.049 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.014+0000 7ff6ef1ac700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6e812e800 con 0x7ff6e8069220 2021-05-25T11:59:53.049 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.014+0000 7ff6ef1ac700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6e8115770 con 0x7ff6e812c1d0 2021-05-25T11:59:53.049 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.014+0000 7ff6ef1ac700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6e8115a10 con 0x7ff6e8068da0 2021-05-25T11:59:53.050 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.014+0000 7ff6ef1ac700 10 monclient(hunting): _renew_subs 2021-05-25T11:59:53.050 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.014+0000 7ff6ef1ac700 10 monclient(hunting): authenticate will time out at 793.611885s 2021-05-25T11:59:53.050 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.014+0000 7ff6ecf48700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7ff6e8068da0 0x7ff6e813aea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:53592/0 (socket says 172.21.15.102:53592) 2021-05-25T11:59:53.050 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.014+0000 7ff6ecf48700 1 -- 172.21.15.102:0/3993460785 learned_addr learned my addr 172.21.15.102:0/3993460785 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T11:59:53.051 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.015+0000 7ff6dcff9700 10 client.?.objecter ms_handle_connect 0x7ff6e8068da0 2021-05-25T11:59:53.051 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.015+0000 7ff6dcff9700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:53.051 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.015+0000 7ff6dcff9700 10 client.?.objecter ms_handle_connect 0x7ff6e8069220 2021-05-25T11:59:53.051 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.015+0000 7ff6dcff9700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:53.051 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.015+0000 7ff6dcff9700 1 -- 172.21.15.102:0/3993460785 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4254188785 0 0) 0x7ff6e8115a10 con 0x7ff6e8068da0 2021-05-25T11:59:53.052 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.015+0000 7ff6dcff9700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:53.052 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.015+0000 7ff6dcff9700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:53.052 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.015+0000 7ff6dcff9700 10 monclient(hunting): my global_id is 4226 2021-05-25T11:59:53.052 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.015+0000 7ff6dcff9700 1 -- 172.21.15.102:0/3993460785 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6c8004c10 con 0x7ff6e8068da0 2021-05-25T11:59:53.052 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.015+0000 7ff6dcff9700 10 client.?.objecter ms_handle_connect 0x7ff6e812c1d0 2021-05-25T11:59:53.053 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.015+0000 7ff6dcff9700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:53.053 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.015+0000 7ff6dcff9700 1 -- 172.21.15.102:0/3993460785 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3795957525 0 0) 0x7ff6e812e800 con 0x7ff6e8069220 2021-05-25T11:59:53.053 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.015+0000 7ff6dcff9700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:53.053 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.015+0000 7ff6dcff9700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:53.053 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.015+0000 7ff6dcff9700 10 monclient(hunting): my global_id is 4233 2021-05-25T11:59:53.054 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.015+0000 7ff6dcff9700 1 -- 172.21.15.102:0/3993460785 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6e8115a10 con 0x7ff6e8069220 2021-05-25T11:59:53.054 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.015+0000 7ff6dcff9700 1 -- 172.21.15.102:0/3993460785 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 996855501 0 0) 0x7ff6e8115770 con 0x7ff6e812c1d0 2021-05-25T11:59:53.054 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.015+0000 7ff6dcff9700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:53.054 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.015+0000 7ff6dcff9700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:53.054 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.015+0000 7ff6dcff9700 10 monclient(hunting): my global_id is 4228 2021-05-25T11:59:53.055 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.015+0000 7ff6dcff9700 1 -- 172.21.15.102:0/3993460785 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6e812e800 con 0x7ff6e812c1d0 2021-05-25T11:59:53.055 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.015+0000 7ff6dcff9700 1 -- 172.21.15.102:0/3993460785 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2428928474 0 0) 0x7ff6c8004c10 con 0x7ff6e8068da0 2021-05-25T11:59:53.055 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.015+0000 7ff6dcff9700 1 -- 172.21.15.102:0/3993460785 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff6e8115770 con 0x7ff6e8068da0 2021-05-25T11:59:53.055 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.015+0000 7ff6dcff9700 1 -- 172.21.15.102:0/3993460785 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7ff6d8002a00 con 0x7ff6e8068da0 2021-05-25T11:59:53.055 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.015+0000 7ff6dcff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:53.056 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.015+0000 7ff6dcff9700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T11:59:53.056 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.015+0000 7ff6dcff9700 10 monclient(hunting): dump: 2021-05-25T11:59:53.056 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:53.056 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:53.056 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:53.057 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:53.057 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:53.057 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:53.057 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:53.057 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:53.057 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:53.058 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:53.058 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.015+0000 7ff6dcff9700 10 monclient(hunting): _finish_auth 0 2021-05-25T11:59:53.058 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.015+0000 7ff6dcff9700 1 -- 172.21.15.102:0/3993460785 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 958800101 0 0) 0x7ff6e8115a10 con 0x7ff6e8069220 2021-05-25T11:59:53.058 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.015+0000 7ff6dcff9700 1 -- 172.21.15.102:0/3993460785 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff6c8004c10 con 0x7ff6e8069220 2021-05-25T11:59:53.059 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.015+0000 7ff6dcff9700 1 -- 172.21.15.102:0/3993460785 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7ff6d0001b30 con 0x7ff6e8069220 2021-05-25T11:59:53.059 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.015+0000 7ff6dcff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:53.059 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.015+0000 7ff6dcff9700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2021-05-25T11:59:53.059 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.015+0000 7ff6dcff9700 10 monclient(hunting): dump: 2021-05-25T11:59:53.059 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:53.060 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:53.060 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:53.060 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:53.060 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:53.060 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:53.061 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:53.061 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:53.061 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:53.061 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:53.061 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.015+0000 7ff6dcff9700 1 -- 172.21.15.102:0/3993460785 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2846280761 0 0) 0x7ff6e812e800 con 0x7ff6e812c1d0 2021-05-25T11:59:53.062 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.015+0000 7ff6dcff9700 1 -- 172.21.15.102:0/3993460785 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff6e8115a10 con 0x7ff6e812c1d0 2021-05-25T11:59:53.062 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.015+0000 7ff6dcff9700 1 -- 172.21.15.102:0/3993460785 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1359990058 0 0) 0x7ff6e8115770 con 0x7ff6e8068da0 2021-05-25T11:59:53.062 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.015+0000 7ff6dcff9700 1 -- 172.21.15.102:0/3993460785 >> v1:172.21.15.181:6789/0 conn(0x7ff6e812c1d0 legacy=0x7ff6e8142fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:53.062 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.015+0000 7ff6dcff9700 1 -- 172.21.15.102:0/3993460785 >> v1:172.21.15.102:6789/0 conn(0x7ff6e8069220 legacy=0x7ff6e813b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:53.063 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.015+0000 7ff6dcff9700 10 monclient: _finish_hunting 0 2021-05-25T11:59:53.063 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.015+0000 7ff6dcff9700 1 monclient: found mon.c 2021-05-25T11:59:53.063 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.015+0000 7ff6dcff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T11:59:53.063 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.015+0000 7ff6dcff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T11:59:53.063 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.015+0000 7ff6dcff9700 1 -- 172.21.15.102:0/3993460785 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6e81138c0 con 0x7ff6e8068da0 2021-05-25T11:59:53.064 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.015+0000 7ff6dcff9700 10 monclient: _finish_auth 0 2021-05-25T11:59:53.064 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.015+0000 7ff6dcff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T11:59:53.064 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.015+0000 7ff6ef1ac700 5 monclient: authenticate success, global_id 4226 2021-05-25T11:59:53.064 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.015+0000 7ff6ef1ac700 10 monclient: _renew_subs 2021-05-25T11:59:53.064 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.015+0000 7ff6ef1ac700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T11:59:53.065 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.015+0000 7ff6ef1ac700 1 -- 172.21.15.102:0/3993460785 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6e8113010 con 0x7ff6e8068da0 2021-05-25T11:59:53.065 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.015+0000 7ff6ef1ac700 10 client.4226.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T11:59:53.065 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.015+0000 7ff6ef1ac700 10 monclient: _renew_subs 2021-05-25T11:59:53.065 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.015+0000 7ff6ef1ac700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T11:59:53.065 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.015+0000 7ff6ef1ac700 1 -- 172.21.15.102:0/3993460785 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6e8116370 con 0x7ff6e8068da0 2021-05-25T11:59:53.066 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.015+0000 7ff6ef1ac700 1 librados: init done 2021-05-25T11:59:53.066 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.015+0000 7ff6dcff9700 1 -- 172.21.15.102:0/3993460785 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff6d8002d40 con 0x7ff6e8068da0 2021-05-25T11:59:53.066 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.015+0000 7ff6dcff9700 10 monclient: handle_config config(0 keys) v1 2021-05-25T11:59:53.066 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.015+0000 7ff6dcff9700 1 -- 172.21.15.102:0/3993460785 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7ff6d8001910 con 0x7ff6e8068da0 2021-05-25T11:59:53.067 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.015+0000 7ff6dcff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:53.067 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.015+0000 7ff6deffd700 4 set_mon_vals no callback set 2021-05-25T11:59:53.067 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.015+0000 7ff6dcff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T11:59:53.067 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.015+0000 7ff6dcff9700 10 monclient: dump: 2021-05-25T11:59:53.067 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:53.068 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:53.068 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:53.068 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:53.068 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:53.069 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:53.069 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:53.069 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:53.069 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:53.069 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:53.070 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.016+0000 7ff6dcff9700 1 -- 172.21.15.102:0/3993460785 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7ff6d8001a70 con 0x7ff6e8068da0 2021-05-25T11:59:53.070 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.016+0000 7ff6dcff9700 1 -- 172.21.15.102:0/3993460785 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(14..14 src has 1..14) v4 ==== 3549+0+0 (unknown 1641242595 0 0) 0x7ff6d803cde0 con 0x7ff6e8068da0 2021-05-25T11:59:53.070 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.016+0000 7ff6dcff9700 10 client.4226.objecter ms_dispatch 0x7ff6e8001550 osd_map(14..14 src has 1..14) v4 2021-05-25T11:59:53.070 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.016+0000 7ff6dcff9700 3 client.4226.objecter handle_osd_map got epochs [14,14] > 0 2021-05-25T11:59:53.071 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.016+0000 7ff6dcff9700 3 client.4226.objecter handle_osd_map decoding full epoch 14 2021-05-25T11:59:53.071 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.016+0000 7ff6ef1ac700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:53.071 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.016+0000 7ff6ef1ac700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:53.071 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.016+0000 7ff6ef1ac700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T11:59:53.071 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.016+0000 7ff6ef1ac700 1 -- 172.21.15.102:0/3993460785 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6c0004db0 con 0x7ff6e8068da0 2021-05-25T11:59:53.072 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.016+0000 7ff6dcff9700 20 client.4226.objecter dump_active .. 0 homeless 2021-05-25T11:59:53.072 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.016+0000 7ff6dffff700 10 monclient: get_auth_request con 0x7ff6c80334c0 auth_method 0 2021-05-25T11:59:53.072 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.016+0000 7ff6dcff9700 10 client.4226.objecter ms_handle_connect 0x7ff6c80334c0 2021-05-25T11:59:53.072 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.020+0000 7ff6dcff9700 1 -- 172.21.15.102:0/3993460785 <== mon.2 v1:172.21.15.102:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7ff6d8012de0 con 0x7ff6e8068da0 2021-05-25T11:59:53.073 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.020+0000 7ff6dcff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:53.073 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.020+0000 7ff6dcff9700 10 monclient: _finish_command 1 = system:0 2021-05-25T11:59:53.171 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.169+0000 7ff6ef1ac700 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}] 2021-05-25T11:59:53.172 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.169+0000 7ff6ef1ac700 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}] 2021-05-25T11:59:53.172 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.169+0000 7ff6ef1ac700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T11:59:53.172 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.169+0000 7ff6ef1ac700 1 -- 172.21.15.102:0/3993460785 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7ff6c00055e0 con 0x7ff6e8068da0 2021-05-25T11:59:53.172 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.170+0000 7ff6dcff9700 1 -- 172.21.15.102:0/3993460785 <== mon.2 v1:172.21.15.102:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v14) v1 ==== 110+0+10 (unknown 349952732 0 2170607528) 0x7ff6d80129d0 con 0x7ff6e8068da0 2021-05-25T11:59:53.173 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.170+0000 7ff6dcff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}] 2021-05-25T11:59:53.173 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.170+0000 7ff6dcff9700 10 monclient: _finish_command 2 = system:0 2021-05-25T11:59:53.173 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2021-05-25T11:59:53.175 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.171+0000 7ff6ef1ac700 10 librados: watch_flush enter 2021-05-25T11:59:53.175 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.171+0000 7ff6ef1ac700 10 librados: watch_flush exit 2021-05-25T11:59:53.175 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.171+0000 7ff6ef1ac700 20 client.4226.objecter shutdown clearing up homeless session... 2021-05-25T11:59:53.176 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.171+0000 7ff6ef1ac700 10 client.4226.objecter successfully canceled tick 2021-05-25T11:59:53.176 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.171+0000 7ff6ef1ac700 1 -- 172.21.15.102:0/3993460785 >> v1:172.21.15.181:6812/29582 conn(0x7ff6c80334c0 legacy=0x7ff6c8035980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:53.176 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.171+0000 7ff6ef1ac700 10 monclient: shutdown 2021-05-25T11:59:53.176 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.171+0000 7ff6ef1ac700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T11:59:53.176 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.171+0000 7ff6ef1ac700 1 -- 172.21.15.102:0/3993460785 >> v1:172.21.15.102:6790/0 conn(0x7ff6e8068da0 legacy=0x7ff6e813aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:53.177 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.171+0000 7ff6ef1ac700 1 -- 172.21.15.102:0/3993460785 shutdown_connections 2021-05-25T11:59:53.177 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.171+0000 7ff6ef1ac700 1 -- 172.21.15.102:0/3993460785 >> 172.21.15.102:0/3993460785 conn(0x7ff6e80984c0 msgr2=0x7ff6e812ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T11:59:53.177 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.171+0000 7ff6ef1ac700 1 -- 172.21.15.102:0/3993460785 shutdown_connections 2021-05-25T11:59:53.177 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.171+0000 7ff6ef1ac700 1 -- 172.21.15.102:0/3993460785 wait complete. 2021-05-25T11:59:53.177 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.171+0000 7ff6ef1ac700 1 librados: shutdown 2021-05-25T11:59:53.184 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2021-05-25T11:59:53.296 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.294+0000 7f8b184be700 10 monclient: get_monmap_and_config 2021-05-25T11:59:53.296 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.294+0000 7f8b184be700 10 monclient: build_initial_monmap 2021-05-25T11:59:53.297 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.294+0000 7f8b184be700 10 monclient: monmap: 2021-05-25T11:59:53.297 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T11:59:53.297 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:53.297 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:53.295615+0000 2021-05-25T11:59:53.297 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:53.295615+0000 2021-05-25T11:59:53.298 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T11:59:53.298 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:53.298 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T11:59:53.298 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T11:59:53.299 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T11:59:53.299 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:53.299 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.295+0000 7f8b184be700 1 Processor -- start 2021-05-25T11:59:53.299 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.295+0000 7f8b184be700 1 -- start start 2021-05-25T11:59:53.299 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.295+0000 7f8b184be700 10 monclient: init 2021-05-25T11:59:53.311 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.295+0000 7f8b184be700 10 monclient: _reopen_session rank -1 2021-05-25T11:59:53.311 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.295+0000 7f8b184be700 10 monclient: _add_conns ranks=[2,1,0] 2021-05-25T11:59:53.311 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.295+0000 7f8b184be700 10 monclient(hunting): picked mon.noname-b con 0x7f8b10097c10 addr v1:172.21.15.102:6790/0 2021-05-25T11:59:53.312 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.295+0000 7f8b184be700 10 monclient(hunting): picked mon.noname-c con 0x7f8b1012af50 addr v1:172.21.15.181:6789/0 2021-05-25T11:59:53.312 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.295+0000 7f8b184be700 10 monclient(hunting): picked mon.noname-a con 0x7f8b1012ea60 addr v1:172.21.15.102:6789/0 2021-05-25T11:59:53.312 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.295+0000 7f8b184be700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b10092d70 con 0x7f8b1012ea60 2021-05-25T11:59:53.312 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.295+0000 7f8b184be700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b10092fa0 con 0x7f8b1012af50 2021-05-25T11:59:53.312 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.295+0000 7f8b184be700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b10093240 con 0x7f8b10097c10 2021-05-25T11:59:53.313 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.295+0000 7f8b184be700 10 monclient(hunting): _renew_subs 2021-05-25T11:59:53.313 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.295+0000 7f8b184be700 10 monclient(hunting): authenticate will time out at 793.892825s 2021-05-25T11:59:53.313 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.295+0000 7f8b16a5b700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f8b1012ea60 0x7f8b10130e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:58722/0 (socket says 172.21.15.102:58722) 2021-05-25T11:59:53.313 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.295+0000 7f8b16a5b700 1 -- 172.21.15.102:0/1985019669 learned_addr learned my addr 172.21.15.102:0/1985019669 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T11:59:53.314 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.295+0000 7f8b1625a700 1 --1- 172.21.15.102:0/1985019669 >> v1:172.21.15.102:6790/0 conn(0x7f8b10097c10 0x7f8b10068eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:53600/0 (socket says 172.21.15.102:53600) 2021-05-25T11:59:53.314 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.296+0000 7f8b15258700 1 -- 172.21.15.102:0/1985019669 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 492743241 0 0) 0x7f8b10092d70 con 0x7f8b1012ea60 2021-05-25T11:59:53.314 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.296+0000 7f8b15258700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:53.314 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.296+0000 7f8b15258700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:53.315 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.296+0000 7f8b15258700 10 monclient(hunting): my global_id is 4236 2021-05-25T11:59:53.315 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.296+0000 7f8b15258700 1 -- 172.21.15.102:0/1985019669 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8af8003270 con 0x7f8b1012ea60 2021-05-25T11:59:53.315 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.296+0000 7f8b15258700 1 -- 172.21.15.102:0/1985019669 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2412990015 0 0) 0x7f8b10093240 con 0x7f8b10097c10 2021-05-25T11:59:53.315 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.296+0000 7f8b15258700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:53.315 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.296+0000 7f8b15258700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:53.316 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.296+0000 7f8b15258700 10 monclient(hunting): my global_id is 4229 2021-05-25T11:59:53.316 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.296+0000 7f8b15258700 1 -- 172.21.15.102:0/1985019669 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b10092d70 con 0x7f8b10097c10 2021-05-25T11:59:53.316 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.296+0000 7f8b15258700 1 -- 172.21.15.102:0/1985019669 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 955475436 0 0) 0x7f8b10092fa0 con 0x7f8b1012af50 2021-05-25T11:59:53.316 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.296+0000 7f8b15258700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:53.316 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.296+0000 7f8b15258700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:53.317 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.296+0000 7f8b15258700 10 monclient(hunting): my global_id is 4231 2021-05-25T11:59:53.317 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.296+0000 7f8b15258700 1 -- 172.21.15.102:0/1985019669 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b10093240 con 0x7f8b1012af50 2021-05-25T11:59:53.317 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.296+0000 7f8b15258700 1 -- 172.21.15.102:0/1985019669 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 525252044 0 0) 0x7f8af8003270 con 0x7f8b1012ea60 2021-05-25T11:59:53.317 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.296+0000 7f8b15258700 1 -- 172.21.15.102:0/1985019669 >> v1:172.21.15.102:6790/0 conn(0x7f8b10097c10 legacy=0x7f8b10068eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:53.318 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.296+0000 7f8b15258700 1 -- 172.21.15.102:0/1985019669 >> v1:172.21.15.181:6789/0 conn(0x7f8b1012af50 legacy=0x7f8b1012d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:53.318 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.296+0000 7f8b15258700 10 monclient: _finish_hunting 0 2021-05-25T11:59:53.318 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.296+0000 7f8b15258700 1 monclient: found mon.noname-a 2021-05-25T11:59:53.318 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.296+0000 7f8b15258700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T11:59:53.319 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.296+0000 7f8b15258700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T11:59:53.319 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.296+0000 7f8b15258700 1 -- 172.21.15.102:0/1985019669 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b100934e0 con 0x7f8b1012ea60 2021-05-25T11:59:53.319 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.296+0000 7f8b15258700 10 monclient: _finish_auth 0 2021-05-25T11:59:53.319 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.296+0000 7f8b15258700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T11:59:53.320 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.296+0000 7f8b15258700 1 -- 172.21.15.102:0/1985019669 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f8b0c0018c0 con 0x7f8b1012ea60 2021-05-25T11:59:53.320 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.296+0000 7f8b15258700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:53.320 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.296+0000 7f8b15258700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T11:59:53.320 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.296+0000 7f8b15258700 10 monclient: dump: 2021-05-25T11:59:53.320 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:53.321 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:53.321 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:53.321 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:53.321 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:53.322 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:53.322 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:53.322 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:53.322 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:53.322 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:53.323 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.296+0000 7f8b184be700 5 monclient: authenticate success, global_id 4236 2021-05-25T11:59:53.323 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.296+0000 7f8b184be700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T11:59:53.323 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.296+0000 7f8b15258700 1 -- 172.21.15.102:0/1985019669 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8b0c001ab0 con 0x7f8b1012ea60 2021-05-25T11:59:53.323 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.296+0000 7f8b15258700 10 monclient: handle_config config(0 keys) v1 2021-05-25T11:59:53.323 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.296+0000 7f8b184be700 10 monclient: get_monmap_and_config success 2021-05-25T11:59:53.324 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.296+0000 7f8b15258700 1 -- 172.21.15.102:0/1985019669 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f8b0c001e40 con 0x7f8b1012ea60 2021-05-25T11:59:53.324 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.296+0000 7f8b184be700 4 set_mon_vals no callback set 2021-05-25T11:59:53.324 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.296+0000 7f8b15258700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:53.324 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.296+0000 7f8b15258700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T11:59:53.324 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.296+0000 7f8b15258700 10 monclient: dump: 2021-05-25T11:59:53.325 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:53.325 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:53.325 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:53.325 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:53.326 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:53.326 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:53.326 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:53.326 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:53.326 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:53.327 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:53.327 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.297+0000 7f8b184be700 10 monclient: shutdown 2021-05-25T11:59:53.327 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.297+0000 7f8b184be700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T11:59:53.327 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.297+0000 7f8b184be700 1 -- 172.21.15.102:0/1985019669 >> v1:172.21.15.102:6789/0 conn(0x7f8b1012ea60 legacy=0x7f8b10130e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:53.327 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.297+0000 7f8b184be700 1 -- 172.21.15.102:0/1985019669 shutdown_connections 2021-05-25T11:59:53.328 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.297+0000 7f8b184be700 1 -- 172.21.15.102:0/1985019669 >> 172.21.15.102:0/1985019669 conn(0x7f8b1009b3c0 msgr2=0x7f8b10061b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T11:59:53.328 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.297+0000 7f8b184be700 1 -- 172.21.15.102:0/1985019669 shutdown_connections 2021-05-25T11:59:53.328 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.297+0000 7f8b184be700 1 -- 172.21.15.102:0/1985019669 wait complete. 2021-05-25T11:59:53.328 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.297+0000 7f8b184be700 10 monclient: build_initial_monmap 2021-05-25T11:59:53.329 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.297+0000 7f8b184be700 10 monclient: monmap: 2021-05-25T11:59:53.329 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T11:59:53.329 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T11:59:53.329 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T11:59:53.330 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T11:59:53.330 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T11:59:53.330 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T11:59:53.330 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T11:59:53.330 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T11:59:53.331 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T11:59:53.331 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:53.331 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.297+0000 7f8b184be700 1 librados: starting msgr at 2021-05-25T11:59:53.331 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.297+0000 7f8b184be700 1 librados: starting objecter 2021-05-25T11:59:53.332 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.297+0000 7f8b184be700 1 Processor -- start 2021-05-25T11:59:53.332 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.297+0000 7f8b184be700 1 -- start start 2021-05-25T11:59:53.332 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.297+0000 7f8b184be700 1 librados: setting wanted keys 2021-05-25T11:59:53.332 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.297+0000 7f8b184be700 1 librados: calling monclient init 2021-05-25T11:59:53.333 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.297+0000 7f8b184be700 10 monclient: init 2021-05-25T11:59:53.333 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.298+0000 7f8b184be700 10 monclient: _reopen_session rank -1 2021-05-25T11:59:53.333 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.298+0000 7f8b184be700 10 monclient: _add_conns ranks=[0,2,1] 2021-05-25T11:59:53.333 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.298+0000 7f8b184be700 10 monclient(hunting): picked mon.noname-a con 0x7f8b10097c10 addr v1:172.21.15.102:6789/0 2021-05-25T11:59:53.334 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.298+0000 7f8b184be700 10 monclient(hunting): picked mon.noname-c con 0x7f8b1012af50 addr v1:172.21.15.102:6790/0 2021-05-25T11:59:53.334 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.298+0000 7f8b184be700 10 monclient(hunting): picked mon.noname-b con 0x7f8b1012ea60 addr v1:172.21.15.181:6789/0 2021-05-25T11:59:53.334 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.298+0000 7f8b184be700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b10111be0 con 0x7f8b10097c10 2021-05-25T11:59:53.334 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.298+0000 7f8b184be700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b10113ec0 con 0x7f8b1012ea60 2021-05-25T11:59:53.335 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.298+0000 7f8b184be700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b10114160 con 0x7f8b1012af50 2021-05-25T11:59:53.335 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.298+0000 7f8b184be700 10 monclient(hunting): _renew_subs 2021-05-25T11:59:53.335 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.298+0000 7f8b184be700 10 monclient(hunting): authenticate will time out at 793.895365s 2021-05-25T11:59:53.335 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.298+0000 7f8b1625a700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f8b10097c10 0x7f8b10110dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:58728/0 (socket says 172.21.15.102:58728) 2021-05-25T11:59:53.336 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.298+0000 7f8b1625a700 1 -- 172.21.15.102:0/421796044 learned_addr learned my addr 172.21.15.102:0/421796044 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T11:59:53.336 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.298+0000 7f8afe7fc700 10 client.?.objecter ms_handle_connect 0x7f8b10097c10 2021-05-25T11:59:53.336 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.298+0000 7f8afe7fc700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:53.336 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.298+0000 7f8afe7fc700 10 client.?.objecter ms_handle_connect 0x7f8b1012af50 2021-05-25T11:59:53.336 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.298+0000 7f8afe7fc700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:53.337 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.298+0000 7f8afe7fc700 1 -- 172.21.15.102:0/421796044 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1972293857 0 0) 0x7f8b10111be0 con 0x7f8b10097c10 2021-05-25T11:59:53.337 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.298+0000 7f8afe7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:53.337 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.298+0000 7f8afe7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:53.337 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.298+0000 7f8afe7fc700 10 monclient(hunting): my global_id is 4239 2021-05-25T11:59:53.338 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.298+0000 7f8afe7fc700 1 -- 172.21.15.102:0/421796044 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8af0004c10 con 0x7f8b10097c10 2021-05-25T11:59:53.338 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.298+0000 7f8afe7fc700 1 -- 172.21.15.102:0/421796044 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3766364626 0 0) 0x7f8b10114160 con 0x7f8b1012af50 2021-05-25T11:59:53.338 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.298+0000 7f8afe7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:53.338 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.298+0000 7f8afe7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:53.339 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.298+0000 7f8afe7fc700 10 monclient(hunting): my global_id is 4232 2021-05-25T11:59:53.339 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.298+0000 7f8afe7fc700 1 -- 172.21.15.102:0/421796044 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b10111be0 con 0x7f8b1012af50 2021-05-25T11:59:53.339 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.298+0000 7f8afe7fc700 10 client.?.objecter ms_handle_connect 0x7f8b1012ea60 2021-05-25T11:59:53.339 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.298+0000 7f8afe7fc700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:53.339 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.298+0000 7f8afe7fc700 1 -- 172.21.15.102:0/421796044 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2340133412 0 0) 0x7f8af0004c10 con 0x7f8b10097c10 2021-05-25T11:59:53.340 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.298+0000 7f8afe7fc700 1 -- 172.21.15.102:0/421796044 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8b10114160 con 0x7f8b10097c10 2021-05-25T11:59:53.340 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.298+0000 7f8afe7fc700 1 -- 172.21.15.102:0/421796044 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f8b000011e0 con 0x7f8b10097c10 2021-05-25T11:59:53.340 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.298+0000 7f8afe7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:53.340 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.298+0000 7f8afe7fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T11:59:53.340 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.298+0000 7f8afe7fc700 10 monclient(hunting): dump: 2021-05-25T11:59:53.341 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:53.341 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:53.341 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:53.341 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:53.341 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:53.342 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:53.342 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:53.342 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:53.342 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:53.342 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:53.343 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.298+0000 7f8afe7fc700 10 monclient(hunting): _finish_auth 0 2021-05-25T11:59:53.343 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.298+0000 7f8afe7fc700 1 -- 172.21.15.102:0/421796044 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1764056886 0 0) 0x7f8b10111be0 con 0x7f8b1012af50 2021-05-25T11:59:53.343 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.298+0000 7f8afe7fc700 1 -- 172.21.15.102:0/421796044 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8af0004c10 con 0x7f8b1012af50 2021-05-25T11:59:53.343 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.298+0000 7f8afe7fc700 1 -- 172.21.15.102:0/421796044 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1554297951 0 0) 0x7f8b10113ec0 con 0x7f8b1012ea60 2021-05-25T11:59:53.343 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.298+0000 7f8afe7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:53.344 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.298+0000 7f8afe7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:53.344 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.298+0000 7f8afe7fc700 10 monclient(hunting): my global_id is 4234 2021-05-25T11:59:53.344 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.298+0000 7f8afe7fc700 1 -- 172.21.15.102:0/421796044 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b10111be0 con 0x7f8b1012ea60 2021-05-25T11:59:53.344 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.298+0000 7f8afe7fc700 1 -- 172.21.15.102:0/421796044 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f8b040018c0 con 0x7f8b1012af50 2021-05-25T11:59:53.345 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.298+0000 7f8afe7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:53.345 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.298+0000 7f8afe7fc700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2021-05-25T11:59:53.345 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.298+0000 7f8afe7fc700 10 monclient(hunting): dump: 2021-05-25T11:59:53.345 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:53.345 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:53.346 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:53.346 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:53.346 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:53.346 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:53.346 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:53.347 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:53.347 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:53.347 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:53.347 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.299+0000 7f8afe7fc700 1 -- 172.21.15.102:0/421796044 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2520434957 0 0) 0x7f8b10114160 con 0x7f8b10097c10 2021-05-25T11:59:53.347 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.299+0000 7f8afe7fc700 1 -- 172.21.15.102:0/421796044 >> v1:172.21.15.102:6790/0 conn(0x7f8b1012af50 legacy=0x7f8b101114d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:53.348 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.299+0000 7f8afe7fc700 1 -- 172.21.15.102:0/421796044 >> v1:172.21.15.181:6789/0 conn(0x7f8b1012ea60 legacy=0x7f8b1013ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:53.348 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.299+0000 7f8afe7fc700 10 monclient: _finish_hunting 0 2021-05-25T11:59:53.348 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.299+0000 7f8afe7fc700 1 monclient: found mon.a 2021-05-25T11:59:53.348 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.299+0000 7f8afe7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T11:59:53.348 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.299+0000 7f8afe7fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:53.349 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.299+0000 7f8afe7fc700 1 -- 172.21.15.102:0/421796044 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b10114400 con 0x7f8b10097c10 2021-05-25T11:59:53.349 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.299+0000 7f8afe7fc700 10 monclient: _finish_auth 0 2021-05-25T11:59:53.349 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.299+0000 7f8afe7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T11:59:53.349 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.299+0000 7f8b184be700 5 monclient: authenticate success, global_id 4239 2021-05-25T11:59:53.349 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.299+0000 7f8b184be700 10 monclient: _renew_subs 2021-05-25T11:59:53.350 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.299+0000 7f8b184be700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:53.350 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.299+0000 7f8b184be700 1 -- 172.21.15.102:0/421796044 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b10112290 con 0x7f8b10097c10 2021-05-25T11:59:53.350 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.299+0000 7f8b184be700 10 client.4239.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T11:59:53.350 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.299+0000 7f8b184be700 10 monclient: _renew_subs 2021-05-25T11:59:53.351 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.299+0000 7f8b184be700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:53.351 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.299+0000 7f8b184be700 1 -- 172.21.15.102:0/421796044 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b101147c0 con 0x7f8b10097c10 2021-05-25T11:59:53.351 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.299+0000 7f8b184be700 1 librados: init done 2021-05-25T11:59:53.351 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.299+0000 7f8afe7fc700 1 -- 172.21.15.102:0/421796044 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8b00004530 con 0x7f8b10097c10 2021-05-25T11:59:53.351 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.299+0000 7f8afe7fc700 10 monclient: handle_config config(0 keys) v1 2021-05-25T11:59:53.352 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.299+0000 7f8afe7fc700 1 -- 172.21.15.102:0/421796044 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f8b00004840 con 0x7f8b10097c10 2021-05-25T11:59:53.352 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.299+0000 7f8afe7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:53.352 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.299+0000 7f8afe7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T11:59:53.352 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.299+0000 7f8b14a57700 4 set_mon_vals no callback set 2021-05-25T11:59:53.353 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.299+0000 7f8afe7fc700 10 monclient: dump: 2021-05-25T11:59:53.353 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:53.353 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:53.353 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:53.353 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:53.354 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:53.354 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:53.354 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:53.354 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:53.355 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:53.355 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:53.355 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.299+0000 7f8afe7fc700 1 -- 172.21.15.102:0/421796044 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f8b0000dbe0 con 0x7f8b10097c10 2021-05-25T11:59:53.356 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.300+0000 7f8b184be700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:53.356 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.300+0000 7f8b184be700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:53.356 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.300+0000 7f8b184be700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:53.357 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.300+0000 7f8afe7fc700 1 -- 172.21.15.102:0/421796044 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(14..14 src has 1..14) v4 ==== 3549+0+0 (unknown 1641242595 0 0) 0x7f8b0003ce80 con 0x7f8b10097c10 2021-05-25T11:59:53.357 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.300+0000 7f8afe7fc700 10 client.4239.objecter ms_dispatch 0x7f8b10001550 osd_map(14..14 src has 1..14) v4 2021-05-25T11:59:53.358 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.300+0000 7f8afe7fc700 3 client.4239.objecter handle_osd_map got epochs [14,14] > 0 2021-05-25T11:59:53.358 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.300+0000 7f8afe7fc700 3 client.4239.objecter handle_osd_map decoding full epoch 14 2021-05-25T11:59:53.359 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.300+0000 7f8b184be700 1 -- 172.21.15.102:0/421796044 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8af4004db0 con 0x7f8b10097c10 2021-05-25T11:59:53.359 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.300+0000 7f8afe7fc700 20 client.4239.objecter dump_active .. 0 homeless 2021-05-25T11:59:53.359 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.300+0000 7f8b15a59700 10 monclient: get_auth_request con 0x7f8af0033450 auth_method 0 2021-05-25T11:59:53.360 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.300+0000 7f8afe7fc700 10 client.4239.objecter ms_handle_connect 0x7f8af0033450 2021-05-25T11:59:53.360 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.304+0000 7f8afe7fc700 1 -- 172.21.15.102:0/421796044 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7f8b00012d80 con 0x7f8b10097c10 2021-05-25T11:59:53.361 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.304+0000 7f8afe7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:53.361 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.304+0000 7f8afe7fc700 10 monclient: _finish_command 1 = system:0 2021-05-25T11:59:53.454 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.452+0000 7f8b184be700 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2021-05-25T11:59:53.454 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.452+0000 7f8b184be700 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2021-05-25T11:59:53.455 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.452+0000 7f8b184be700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:53.455 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.452+0000 7f8b184be700 1 -- 172.21.15.102:0/421796044 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f8af40055e0 con 0x7f8b10097c10 2021-05-25T11:59:53.455 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.453+0000 7f8afe7fc700 1 -- 172.21.15.102:0/421796044 <== mon.0 v1:172.21.15.102:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v14) v1 ==== 92+0+10 (unknown 185327234 0 265718887) 0x7f8b00012970 con 0x7f8b10097c10 2021-05-25T11:59:53.456 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.453+0000 7f8afe7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2021-05-25T11:59:53.456 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.453+0000 7f8afe7fc700 10 monclient: _finish_command 2 = system:0 2021-05-25T11:59:53.456 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 8 2021-05-25T11:59:53.458 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.454+0000 7f8b184be700 10 librados: watch_flush enter 2021-05-25T11:59:53.458 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.454+0000 7f8b184be700 10 librados: watch_flush exit 2021-05-25T11:59:53.458 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.454+0000 7f8b184be700 20 client.4239.objecter shutdown clearing up homeless session... 2021-05-25T11:59:53.459 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.454+0000 7f8b184be700 10 client.4239.objecter successfully canceled tick 2021-05-25T11:59:53.459 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.454+0000 7f8b184be700 1 -- 172.21.15.102:0/421796044 >> v1:172.21.15.181:6812/29582 conn(0x7f8af0033450 legacy=0x7f8af0035910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:53.459 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.454+0000 7f8b184be700 10 monclient: shutdown 2021-05-25T11:59:53.459 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.454+0000 7f8b184be700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T11:59:53.459 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.454+0000 7f8b184be700 1 -- 172.21.15.102:0/421796044 >> v1:172.21.15.102:6789/0 conn(0x7f8b10097c10 legacy=0x7f8b10110dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:53.460 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.454+0000 7f8b184be700 1 -- 172.21.15.102:0/421796044 shutdown_connections 2021-05-25T11:59:53.460 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.454+0000 7f8b184be700 1 -- 172.21.15.102:0/421796044 >> 172.21.15.102:0/421796044 conn(0x7f8b1009b3c0 msgr2=0x7f8b1012e2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T11:59:53.460 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.454+0000 7f8b184be700 1 -- 172.21.15.102:0/421796044 shutdown_connections 2021-05-25T11:59:53.460 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.454+0000 7f8b184be700 1 -- 172.21.15.102:0/421796044 wait complete. 2021-05-25T11:59:53.461 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.454+0000 7f8b184be700 1 librados: shutdown 2021-05-25T11:59:53.467 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2021-05-25T11:59:53.467 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2021-05-25T11:59:53.468 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2021-05-25T11:59:53.577 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.576+0000 7f5d945f5700 10 monclient: get_monmap_and_config 2021-05-25T11:59:53.578 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.576+0000 7f5d945f5700 10 monclient: build_initial_monmap 2021-05-25T11:59:53.578 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.576+0000 7f5d945f5700 10 monclient: monmap: 2021-05-25T11:59:53.578 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T11:59:53.579 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:53.579 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:53.577047+0000 2021-05-25T11:59:53.579 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:53.577047+0000 2021-05-25T11:59:53.579 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T11:59:53.579 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:53.580 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T11:59:53.580 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T11:59:53.580 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T11:59:53.580 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:53.580 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.576+0000 7f5d945f5700 1 Processor -- start 2021-05-25T11:59:53.581 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.576+0000 7f5d945f5700 1 -- start start 2021-05-25T11:59:53.581 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.576+0000 7f5d945f5700 10 monclient: init 2021-05-25T11:59:53.590 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.577+0000 7f5d945f5700 10 monclient: _reopen_session rank -1 2021-05-25T11:59:53.591 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.577+0000 7f5d945f5700 10 monclient: _add_conns ranks=[1,2,0] 2021-05-25T11:59:53.591 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.577+0000 7f5d945f5700 10 monclient(hunting): picked mon.noname-c con 0x7f5d8c097fa0 addr v1:172.21.15.181:6789/0 2021-05-25T11:59:53.591 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.577+0000 7f5d945f5700 10 monclient(hunting): picked mon.noname-b con 0x7f5d8c063b70 addr v1:172.21.15.102:6790/0 2021-05-25T11:59:53.591 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.577+0000 7f5d945f5700 10 monclient(hunting): picked mon.noname-a con 0x7f5d8c1325c0 addr v1:172.21.15.102:6789/0 2021-05-25T11:59:53.592 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.577+0000 7f5d945f5700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d8c096b20 con 0x7f5d8c1325c0 2021-05-25T11:59:53.592 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.577+0000 7f5d945f5700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d8c09b460 con 0x7f5d8c097fa0 2021-05-25T11:59:53.592 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.577+0000 7f5d945f5700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d8c0964c0 con 0x7f5d8c063b70 2021-05-25T11:59:53.592 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.577+0000 7f5d945f5700 10 monclient(hunting): _renew_subs 2021-05-25T11:59:53.593 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.577+0000 7f5d945f5700 10 monclient(hunting): authenticate will time out at 794.174374s 2021-05-25T11:59:53.593 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.577+0000 7f5d91b90700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f5d8c063b70 0x7f5d8c064020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:53616/0 (socket says 172.21.15.102:53616) 2021-05-25T11:59:53.593 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.577+0000 7f5d92b92700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f5d8c1325c0 0x7f5d8c1349b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:58740/0 (socket says 172.21.15.102:58740) 2021-05-25T11:59:53.593 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.577+0000 7f5d91b90700 1 -- 172.21.15.102:0/3890579563 learned_addr learned my addr 172.21.15.102:0/3890579563 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T11:59:53.593 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.577+0000 7f5d9138f700 1 -- 172.21.15.102:0/3890579563 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3223622862 0 0) 0x7f5d8c0964c0 con 0x7f5d8c063b70 2021-05-25T11:59:53.594 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.577+0000 7f5d9138f700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:53.594 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.577+0000 7f5d9138f700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:53.594 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.577+0000 7f5d9138f700 10 monclient(hunting): my global_id is 4235 2021-05-25T11:59:53.594 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.577+0000 7f5d9138f700 1 -- 172.21.15.102:0/3890579563 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d74003270 con 0x7f5d8c063b70 2021-05-25T11:59:53.595 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.577+0000 7f5d9138f700 1 -- 172.21.15.102:0/3890579563 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1992556623 0 0) 0x7f5d8c096b20 con 0x7f5d8c1325c0 2021-05-25T11:59:53.595 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.577+0000 7f5d9138f700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:53.595 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.577+0000 7f5d9138f700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:53.595 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.577+0000 7f5d9138f700 10 monclient(hunting): my global_id is 4242 2021-05-25T11:59:53.595 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.577+0000 7f5d9138f700 1 -- 172.21.15.102:0/3890579563 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d8c0964c0 con 0x7f5d8c1325c0 2021-05-25T11:59:53.596 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.577+0000 7f5d9138f700 1 -- 172.21.15.102:0/3890579563 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2416644339 0 0) 0x7f5d8c0964c0 con 0x7f5d8c1325c0 2021-05-25T11:59:53.596 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.577+0000 7f5d9138f700 1 -- 172.21.15.102:0/3890579563 >> v1:172.21.15.102:6790/0 conn(0x7f5d8c063b70 legacy=0x7f5d8c064020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:53.596 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.577+0000 7f5d9138f700 1 -- 172.21.15.102:0/3890579563 >> v1:172.21.15.181:6789/0 conn(0x7f5d8c097fa0 legacy=0x7f5d8c063460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:53.596 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.577+0000 7f5d9138f700 10 monclient: _finish_hunting 0 2021-05-25T11:59:53.596 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.577+0000 7f5d9138f700 1 monclient: found mon.noname-a 2021-05-25T11:59:53.597 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.578+0000 7f5d9138f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T11:59:53.597 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.578+0000 7f5d9138f700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T11:59:53.597 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.578+0000 7f5d9138f700 1 -- 172.21.15.102:0/3890579563 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d8c096760 con 0x7f5d8c1325c0 2021-05-25T11:59:53.597 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.578+0000 7f5d9138f700 10 monclient: _finish_auth 0 2021-05-25T11:59:53.597 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.578+0000 7f5d9138f700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T11:59:53.598 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.578+0000 7f5d945f5700 5 monclient: authenticate success, global_id 4242 2021-05-25T11:59:53.598 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.578+0000 7f5d9138f700 1 -- 172.21.15.102:0/3890579563 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f5d88001910 con 0x7f5d8c1325c0 2021-05-25T11:59:53.598 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.578+0000 7f5d945f5700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T11:59:53.598 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.578+0000 7f5d9138f700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:53.599 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.578+0000 7f5d9138f700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T11:59:53.599 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.578+0000 7f5d9138f700 10 monclient: dump: 2021-05-25T11:59:53.599 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:53.599 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:53.599 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:53.600 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:53.600 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:53.600 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:53.600 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:53.600 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:53.601 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:53.601 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:53.601 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.578+0000 7f5d945f5700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T11:59:53.601 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.578+0000 7f5d9138f700 1 -- 172.21.15.102:0/3890579563 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5d88001b40 con 0x7f5d8c1325c0 2021-05-25T11:59:53.601 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.578+0000 7f5d9138f700 10 monclient: handle_config config(0 keys) v1 2021-05-25T11:59:53.602 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.578+0000 7f5d945f5700 10 monclient: get_monmap_and_config success 2021-05-25T11:59:53.602 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.578+0000 7f5d945f5700 4 set_mon_vals no callback set 2021-05-25T11:59:53.602 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.578+0000 7f5d9138f700 1 -- 172.21.15.102:0/3890579563 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f5d88001ed0 con 0x7f5d8c1325c0 2021-05-25T11:59:53.602 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.578+0000 7f5d9138f700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:53.603 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.578+0000 7f5d9138f700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T11:59:53.603 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.578+0000 7f5d9138f700 10 monclient: dump: 2021-05-25T11:59:53.603 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:53.603 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:53.603 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:53.604 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:53.604 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:53.604 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:53.604 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:53.604 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:53.605 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:53.605 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:53.605 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.578+0000 7f5d945f5700 10 monclient: shutdown 2021-05-25T11:59:53.605 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.578+0000 7f5d945f5700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T11:59:53.605 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.578+0000 7f5d945f5700 1 -- 172.21.15.102:0/3890579563 >> v1:172.21.15.102:6789/0 conn(0x7f5d8c1325c0 legacy=0x7f5d8c1349b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:53.606 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.578+0000 7f5d945f5700 1 -- 172.21.15.102:0/3890579563 shutdown_connections 2021-05-25T11:59:53.606 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.578+0000 7f5d945f5700 1 -- 172.21.15.102:0/3890579563 >> 172.21.15.102:0/3890579563 conn(0x7f5d8c09ef50 msgr2=0x7f5d8c09f360 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T11:59:53.606 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.578+0000 7f5d945f5700 1 -- 172.21.15.102:0/3890579563 shutdown_connections 2021-05-25T11:59:53.606 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.578+0000 7f5d945f5700 1 -- 172.21.15.102:0/3890579563 wait complete. 2021-05-25T11:59:53.607 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.579+0000 7f5d945f5700 10 monclient: build_initial_monmap 2021-05-25T11:59:53.607 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.579+0000 7f5d945f5700 10 monclient: monmap: 2021-05-25T11:59:53.607 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T11:59:53.607 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T11:59:53.607 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T11:59:53.608 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T11:59:53.608 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T11:59:53.608 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T11:59:53.608 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T11:59:53.608 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T11:59:53.609 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T11:59:53.609 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:53.609 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.579+0000 7f5d945f5700 1 librados: starting msgr at 2021-05-25T11:59:53.609 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.579+0000 7f5d945f5700 1 librados: starting objecter 2021-05-25T11:59:53.609 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.579+0000 7f5d945f5700 1 Processor -- start 2021-05-25T11:59:53.610 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.579+0000 7f5d945f5700 1 -- start start 2021-05-25T11:59:53.610 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.579+0000 7f5d945f5700 1 librados: setting wanted keys 2021-05-25T11:59:53.610 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.579+0000 7f5d945f5700 1 librados: calling monclient init 2021-05-25T11:59:53.610 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.579+0000 7f5d945f5700 10 monclient: init 2021-05-25T11:59:53.611 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.579+0000 7f5d945f5700 10 monclient: _reopen_session rank -1 2021-05-25T11:59:53.611 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.579+0000 7f5d945f5700 10 monclient: _add_conns ranks=[2,0,1] 2021-05-25T11:59:53.611 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.579+0000 7f5d945f5700 10 monclient(hunting): picked mon.noname-c con 0x7f5d8c063b70 addr v1:172.21.15.102:6790/0 2021-05-25T11:59:53.611 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.579+0000 7f5d945f5700 10 monclient(hunting): picked mon.noname-a con 0x7f5d8c097fa0 addr v1:172.21.15.102:6789/0 2021-05-25T11:59:53.611 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.579+0000 7f5d945f5700 10 monclient(hunting): picked mon.noname-b con 0x7f5d8c1325c0 addr v1:172.21.15.181:6789/0 2021-05-25T11:59:53.612 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.579+0000 7f5d945f5700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d8c12c0a0 con 0x7f5d8c097fa0 2021-05-25T11:59:53.612 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.579+0000 7f5d945f5700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d8c09ab70 con 0x7f5d8c1325c0 2021-05-25T11:59:53.612 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.579+0000 7f5d945f5700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d8c116100 con 0x7f5d8c063b70 2021-05-25T11:59:53.613 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.579+0000 7f5d945f5700 10 monclient(hunting): _renew_subs 2021-05-25T11:59:53.613 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.579+0000 7f5d945f5700 10 monclient(hunting): authenticate will time out at 794.176922s 2021-05-25T11:59:53.613 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.580+0000 7f5d91b90700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f5d8c097fa0 0x7f5d8c12b990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:58744/0 (socket says 172.21.15.102:58744) 2021-05-25T11:59:53.613 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.580+0000 7f5d91b90700 1 -- 172.21.15.102:0/389089671 learned_addr learned my addr 172.21.15.102:0/389089671 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T11:59:53.613 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.580+0000 7f5d7a7fc700 10 client.?.objecter ms_handle_connect 0x7f5d8c063b70 2021-05-25T11:59:53.614 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.580+0000 7f5d7a7fc700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:53.614 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.580+0000 7f5d7a7fc700 10 client.?.objecter ms_handle_connect 0x7f5d8c097fa0 2021-05-25T11:59:53.614 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.580+0000 7f5d7a7fc700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:53.614 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.580+0000 7f5d7a7fc700 1 -- 172.21.15.102:0/389089671 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 770942409 0 0) 0x7f5d8c116100 con 0x7f5d8c063b70 2021-05-25T11:59:53.615 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.580+0000 7f5d7a7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:53.615 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.580+0000 7f5d7a7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:53.615 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.580+0000 7f5d7a7fc700 10 monclient(hunting): my global_id is 4238 2021-05-25T11:59:53.615 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.580+0000 7f5d7a7fc700 1 -- 172.21.15.102:0/389089671 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d6c004c10 con 0x7f5d8c063b70 2021-05-25T11:59:53.615 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.580+0000 7f5d7a7fc700 1 -- 172.21.15.102:0/389089671 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3343690585 0 0) 0x7f5d8c12c0a0 con 0x7f5d8c097fa0 2021-05-25T11:59:53.616 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.580+0000 7f5d7a7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:53.616 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.580+0000 7f5d7a7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:53.616 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.580+0000 7f5d7a7fc700 10 monclient(hunting): my global_id is 4245 2021-05-25T11:59:53.616 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.580+0000 7f5d7a7fc700 1 -- 172.21.15.102:0/389089671 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d8c116100 con 0x7f5d8c097fa0 2021-05-25T11:59:53.617 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.580+0000 7f5d7a7fc700 10 client.?.objecter ms_handle_connect 0x7f5d8c1325c0 2021-05-25T11:59:53.617 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.580+0000 7f5d7a7fc700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:53.617 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.580+0000 7f5d7a7fc700 1 -- 172.21.15.102:0/389089671 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1629148504 0 0) 0x7f5d8c09ab70 con 0x7f5d8c1325c0 2021-05-25T11:59:53.617 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.580+0000 7f5d7a7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:53.617 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.580+0000 7f5d7a7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:53.618 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.580+0000 7f5d7a7fc700 10 monclient(hunting): my global_id is 4240 2021-05-25T11:59:53.618 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.580+0000 7f5d7a7fc700 1 -- 172.21.15.102:0/389089671 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d8c12c0a0 con 0x7f5d8c1325c0 2021-05-25T11:59:53.618 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.580+0000 7f5d7a7fc700 1 -- 172.21.15.102:0/389089671 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2655796138 0 0) 0x7f5d6c004c10 con 0x7f5d8c063b70 2021-05-25T11:59:53.618 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.580+0000 7f5d7a7fc700 1 -- 172.21.15.102:0/389089671 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5d8c09ab70 con 0x7f5d8c063b70 2021-05-25T11:59:53.619 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.580+0000 7f5d7a7fc700 1 -- 172.21.15.102:0/389089671 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f5d80001860 con 0x7f5d8c063b70 2021-05-25T11:59:53.619 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.580+0000 7f5d7a7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:53.619 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.580+0000 7f5d7a7fc700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T11:59:53.619 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.580+0000 7f5d7a7fc700 10 monclient(hunting): dump: 2021-05-25T11:59:53.619 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:53.620 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:53.620 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:53.620 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:53.620 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:53.621 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:53.621 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:53.621 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:53.621 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:53.621 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:53.622 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.580+0000 7f5d7a7fc700 10 monclient(hunting): _finish_auth 0 2021-05-25T11:59:53.622 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.580+0000 7f5d7a7fc700 1 -- 172.21.15.102:0/389089671 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4155964460 0 0) 0x7f5d8c116100 con 0x7f5d8c097fa0 2021-05-25T11:59:53.622 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.580+0000 7f5d7a7fc700 1 -- 172.21.15.102:0/389089671 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5d6c004c10 con 0x7f5d8c097fa0 2021-05-25T11:59:53.622 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.580+0000 7f5d7a7fc700 1 -- 172.21.15.102:0/389089671 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f5d7c001960 con 0x7f5d8c097fa0 2021-05-25T11:59:53.622 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.580+0000 7f5d7a7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:53.623 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.580+0000 7f5d7a7fc700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2021-05-25T11:59:53.623 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.580+0000 7f5d7a7fc700 10 monclient(hunting): dump: 2021-05-25T11:59:53.623 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:53.623 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:53.623 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:53.624 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:53.624 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:53.624 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:53.624 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:53.625 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:53.625 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:53.625 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:53.625 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.580+0000 7f5d7a7fc700 1 -- 172.21.15.102:0/389089671 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 920032101 0 0) 0x7f5d8c09ab70 con 0x7f5d8c063b70 2021-05-25T11:59:53.625 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.580+0000 7f5d7a7fc700 1 -- 172.21.15.102:0/389089671 >> v1:172.21.15.181:6789/0 conn(0x7f5d8c1325c0 legacy=0x7f5d8c143190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:53.626 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.580+0000 7f5d7a7fc700 1 -- 172.21.15.102:0/389089671 >> v1:172.21.15.102:6789/0 conn(0x7f5d8c097fa0 legacy=0x7f5d8c12b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:53.626 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.580+0000 7f5d7a7fc700 10 monclient: _finish_hunting 0 2021-05-25T11:59:53.626 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.580+0000 7f5d7a7fc700 1 monclient: found mon.c 2021-05-25T11:59:53.626 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.580+0000 7f5d7a7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T11:59:53.626 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.580+0000 7f5d7a7fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T11:59:53.627 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.580+0000 7f5d7a7fc700 1 -- 172.21.15.102:0/389089671 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d8c1163a0 con 0x7f5d8c063b70 2021-05-25T11:59:53.627 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.580+0000 7f5d7a7fc700 10 monclient: _finish_auth 0 2021-05-25T11:59:53.627 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.580+0000 7f5d7a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T11:59:53.627 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.580+0000 7f5d945f5700 5 monclient: authenticate success, global_id 4238 2021-05-25T11:59:53.628 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.580+0000 7f5d945f5700 10 monclient: _renew_subs 2021-05-25T11:59:53.628 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.580+0000 7f5d945f5700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T11:59:53.628 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.580+0000 7f5d945f5700 1 -- 172.21.15.102:0/389089671 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d8c094510 con 0x7f5d8c063b70 2021-05-25T11:59:53.628 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.580+0000 7f5d945f5700 10 client.4238.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T11:59:53.628 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.580+0000 7f5d945f5700 10 monclient: _renew_subs 2021-05-25T11:59:53.629 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.580+0000 7f5d945f5700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T11:59:53.629 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.580+0000 7f5d945f5700 1 -- 172.21.15.102:0/389089671 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d8c092f00 con 0x7f5d8c063b70 2021-05-25T11:59:53.629 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.580+0000 7f5d945f5700 1 librados: init done 2021-05-25T11:59:53.629 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.580+0000 7f5d7a7fc700 1 -- 172.21.15.102:0/389089671 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5d80001ad0 con 0x7f5d8c063b70 2021-05-25T11:59:53.629 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.580+0000 7f5d7a7fc700 10 monclient: handle_config config(0 keys) v1 2021-05-25T11:59:53.630 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.580+0000 7f5d7a7fc700 1 -- 172.21.15.102:0/389089671 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f5d80001e60 con 0x7f5d8c063b70 2021-05-25T11:59:53.630 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.580+0000 7f5d7a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:53.630 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.580+0000 7f5d7a7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T11:59:53.630 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.580+0000 7f5d90b8e700 4 set_mon_vals no callback set 2021-05-25T11:59:53.630 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.580+0000 7f5d7a7fc700 10 monclient: dump: 2021-05-25T11:59:53.631 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:53.631 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:53.631 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:53.631 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:53.632 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:53.632 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:53.632 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:53.632 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:53.632 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:53.633 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:53.633 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.581+0000 7f5d7a7fc700 1 -- 172.21.15.102:0/389089671 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f5d80002040 con 0x7f5d8c063b70 2021-05-25T11:59:53.633 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.581+0000 7f5d945f5700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:53.633 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.581+0000 7f5d945f5700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:53.633 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.581+0000 7f5d945f5700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T11:59:53.634 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.581+0000 7f5d945f5700 1 -- 172.21.15.102:0/389089671 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d70004db0 con 0x7f5d8c063b70 2021-05-25T11:59:53.634 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.581+0000 7f5d7a7fc700 1 -- 172.21.15.102:0/389089671 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(14..14 src has 1..14) v4 ==== 3549+0+0 (unknown 1641242595 0 0) 0x7f5d8003a800 con 0x7f5d8c063b70 2021-05-25T11:59:53.634 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.581+0000 7f5d7a7fc700 10 client.4238.objecter ms_dispatch 0x7f5d8c001550 osd_map(14..14 src has 1..14) v4 2021-05-25T11:59:53.634 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.581+0000 7f5d7a7fc700 3 client.4238.objecter handle_osd_map got epochs [14,14] > 0 2021-05-25T11:59:53.635 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.581+0000 7f5d7a7fc700 3 client.4238.objecter handle_osd_map decoding full epoch 14 2021-05-25T11:59:53.635 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.581+0000 7f5d7a7fc700 20 client.4238.objecter dump_active .. 0 homeless 2021-05-25T11:59:53.635 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.581+0000 7f5d91b90700 10 monclient: get_auth_request con 0x7f5d6c0333c0 auth_method 0 2021-05-25T11:59:53.635 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.582+0000 7f5d7a7fc700 10 client.4238.objecter ms_handle_connect 0x7f5d6c0333c0 2021-05-25T11:59:53.636 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.585+0000 7f5d7a7fc700 1 -- 172.21.15.102:0/389089671 <== mon.2 v1:172.21.15.102:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7f5d80002870 con 0x7f5d8c063b70 2021-05-25T11:59:53.636 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.585+0000 7f5d7a7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:53.636 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.585+0000 7f5d7a7fc700 10 monclient: _finish_command 1 = system:0 2021-05-25T11:59:53.778 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.776+0000 7f5d945f5700 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2021-05-25T11:59:53.778 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.776+0000 7f5d945f5700 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2021-05-25T11:59:53.778 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.776+0000 7f5d945f5700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T11:59:53.779 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.776+0000 7f5d945f5700 1 -- 172.21.15.102:0/389089671 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f5d70005060 con 0x7f5d8c063b70 2021-05-25T11:59:53.780 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.778+0000 7f5d7a7fc700 1 -- 172.21.15.102:0/389089671 <== mon.2 v1:172.21.15.102:6790/0 10 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v4) v1 ==== 74+0+73761 (unknown 3797009233 0 2193875798) 0x7f5d800104d0 con 0x7f5d8c063b70 2021-05-25T11:59:53.780 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.778+0000 7f5d7a7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2021-05-25T11:59:53.780 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.778+0000 7f5d7a7fc700 10 monclient: _finish_command 2 = system:0 2021-05-25T11:59:53.781 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:59:53.783 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.779+0000 7f5d945f5700 10 librados: watch_flush enter 2021-05-25T11:59:53.783 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.779+0000 7f5d945f5700 10 librados: watch_flush exit 2021-05-25T11:59:53.783 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.779+0000 7f5d945f5700 20 client.4238.objecter shutdown clearing up homeless session... 2021-05-25T11:59:53.784 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.779+0000 7f5d945f5700 10 client.4238.objecter successfully canceled tick 2021-05-25T11:59:53.784 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.779+0000 7f5d945f5700 1 -- 172.21.15.102:0/389089671 >> v1:172.21.15.181:6812/29582 conn(0x7f5d6c0333c0 legacy=0x7f5d6c035880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:53.784 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.779+0000 7f5d945f5700 10 monclient: shutdown 2021-05-25T11:59:53.784 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.779+0000 7f5d945f5700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T11:59:53.785 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.779+0000 7f5d945f5700 1 -- 172.21.15.102:0/389089671 >> v1:172.21.15.102:6790/0 conn(0x7f5d8c063b70 legacy=0x7f5d8c0921d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:53.785 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.779+0000 7f5d945f5700 1 -- 172.21.15.102:0/389089671 shutdown_connections 2021-05-25T11:59:53.785 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.780+0000 7f5d945f5700 1 -- 172.21.15.102:0/389089671 >> 172.21.15.102:0/389089671 conn(0x7f5d8c09ef50 msgr2=0x7f5d8c131a00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T11:59:53.785 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.780+0000 7f5d945f5700 1 -- 172.21.15.102:0/389089671 shutdown_connections 2021-05-25T11:59:53.785 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.780+0000 7f5d945f5700 1 -- 172.21.15.102:0/389089671 wait complete. 2021-05-25T11:59:53.786 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.780+0000 7f5d945f5700 1 librados: shutdown 2021-05-25T11:59:53.792 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":4,"active_gid":4105,"active_name":"x","active_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6812","nonce":29582}]},"active_addr":"172.21.15.181:6812/29582","active_change":"2021-05-25T11:59:45.549365+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[],"modules":["iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Registry to which we should normalize unqualified image names","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"str","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"autoscale_profile":{"name":"autoscale_profile","type":"str","level":"advanced","flags":1,"default_value":"scale-up","min":"","max":"","enum_allowed":[],"desc":"pg_autoscale profiler","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.181:0","nonce":2642639533}]},{"addrvec":[{"type":"v2","addr":"172.21.15.181:0","nonce":1064236234}]},{"addrvec":[{"type":"v2","addr":"172.21.15.181:0","nonce":2578341040}]}]}} 2021-05-25T11:59:53.794 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2021-05-25T11:59:53.795 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2021-05-25T11:59:53.795 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-05-25T11:59:53.903 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.902+0000 7fbe04393700 10 monclient: get_monmap_and_config 2021-05-25T11:59:53.904 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.902+0000 7fbe04393700 10 monclient: build_initial_monmap 2021-05-25T11:59:53.904 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.902+0000 7fbe04393700 10 monclient: monmap: 2021-05-25T11:59:53.904 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T11:59:53.905 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:53.905 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:53.903101+0000 2021-05-25T11:59:53.905 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:53.903101+0000 2021-05-25T11:59:53.905 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T11:59:53.905 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:53.906 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T11:59:53.906 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T11:59:53.906 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T11:59:53.907 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:53.907 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.902+0000 7fbe04393700 1 Processor -- start 2021-05-25T11:59:53.907 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.902+0000 7fbe04393700 1 -- start start 2021-05-25T11:59:53.907 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.902+0000 7fbe04393700 10 monclient: init 2021-05-25T11:59:53.917 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.903+0000 7fbe04393700 10 monclient: _reopen_session rank -1 2021-05-25T11:59:53.917 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.903+0000 7fbe04393700 10 monclient: _add_conns ranks=[1,0,2] 2021-05-25T11:59:53.918 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.903+0000 7fbe04393700 10 monclient(hunting): picked mon.noname-c con 0x7fbdfc068d50 addr v1:172.21.15.181:6789/0 2021-05-25T11:59:53.918 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.903+0000 7fbe04393700 10 monclient(hunting): picked mon.noname-a con 0x7fbdfc069130 addr v1:172.21.15.102:6789/0 2021-05-25T11:59:53.918 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.903+0000 7fbe04393700 10 monclient(hunting): picked mon.noname-b con 0x7fbdfc0695e0 addr v1:172.21.15.102:6790/0 2021-05-25T11:59:53.918 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.903+0000 7fbe04393700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdfc094380 con 0x7fbdfc069130 2021-05-25T11:59:53.919 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.903+0000 7fbe04393700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdfc094620 con 0x7fbdfc068d50 2021-05-25T11:59:53.919 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.903+0000 7fbe04393700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdfc0948c0 con 0x7fbdfc0695e0 2021-05-25T11:59:53.919 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.903+0000 7fbe04393700 10 monclient(hunting): _renew_subs 2021-05-25T11:59:53.919 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.903+0000 7fbe04393700 10 monclient(hunting): authenticate will time out at 794.500401s 2021-05-25T11:59:53.919 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.903+0000 7fbe02930700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fbdfc0695e0 0x7fbdfc132160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:53632/0 (socket says 172.21.15.102:53632) 2021-05-25T11:59:53.920 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.903+0000 7fbe02930700 1 -- 172.21.15.102:0/4187406066 learned_addr learned my addr 172.21.15.102:0/4187406066 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T11:59:53.920 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.903+0000 7fbe0112d700 1 -- 172.21.15.102:0/4187406066 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1036000026 0 0) 0x7fbdfc0948c0 con 0x7fbdfc0695e0 2021-05-25T11:59:53.920 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.903+0000 7fbe0112d700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:53.920 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.903+0000 7fbe0112d700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:53.920 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.903+0000 7fbe0112d700 10 monclient(hunting): my global_id is 4241 2021-05-25T11:59:53.921 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.903+0000 7fbe0112d700 1 -- 172.21.15.102:0/4187406066 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbde4003270 con 0x7fbdfc0695e0 2021-05-25T11:59:53.921 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.903+0000 7fbe0112d700 1 -- 172.21.15.102:0/4187406066 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 405406520 0 0) 0x7fbdfc094380 con 0x7fbdfc069130 2021-05-25T11:59:53.921 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.903+0000 7fbe0112d700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:53.921 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.903+0000 7fbe0112d700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:53.922 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.903+0000 7fbe0112d700 10 monclient(hunting): my global_id is 4248 2021-05-25T11:59:53.922 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.903+0000 7fbe0112d700 1 -- 172.21.15.102:0/4187406066 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdfc0948c0 con 0x7fbdfc069130 2021-05-25T11:59:53.922 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.903+0000 7fbe0112d700 1 -- 172.21.15.102:0/4187406066 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4253691198 0 0) 0x7fbde4003270 con 0x7fbdfc0695e0 2021-05-25T11:59:53.922 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.903+0000 7fbe0112d700 1 -- 172.21.15.102:0/4187406066 >> v1:172.21.15.181:6789/0 conn(0x7fbdfc068d50 legacy=0x7fbdfc129c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:53.922 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.903+0000 7fbe0112d700 1 -- 172.21.15.102:0/4187406066 >> v1:172.21.15.102:6789/0 conn(0x7fbdfc069130 legacy=0x7fbdfc12ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:53.923 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.903+0000 7fbe0112d700 10 monclient: _finish_hunting 0 2021-05-25T11:59:53.923 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.903+0000 7fbe0112d700 1 monclient: found mon.noname-b 2021-05-25T11:59:53.923 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.903+0000 7fbe0112d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T11:59:53.924 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.904+0000 7fbe0112d700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T11:59:53.924 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.904+0000 7fbe0112d700 1 -- 172.21.15.102:0/4187406066 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdfc094c80 con 0x7fbdfc0695e0 2021-05-25T11:59:53.924 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.904+0000 7fbe0112d700 10 monclient: _finish_auth 0 2021-05-25T11:59:53.924 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.904+0000 7fbe0112d700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T11:59:53.924 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.904+0000 7fbe04393700 5 monclient: authenticate success, global_id 4241 2021-05-25T11:59:53.925 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.904+0000 7fbe0112d700 1 -- 172.21.15.102:0/4187406066 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fbdf8001990 con 0x7fbdfc0695e0 2021-05-25T11:59:53.925 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.904+0000 7fbe0112d700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:53.925 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.904+0000 7fbe0112d700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T11:59:53.925 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.904+0000 7fbe0112d700 10 monclient: dump: 2021-05-25T11:59:53.925 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:53.926 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:53.926 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:53.926 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:53.926 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:53.926 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:53.927 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:53.927 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:53.927 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:53.928 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:53.928 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.904+0000 7fbe04393700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T11:59:53.928 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.904+0000 7fbe0112d700 1 -- 172.21.15.102:0/4187406066 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbdf8001c30 con 0x7fbdfc0695e0 2021-05-25T11:59:53.928 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.904+0000 7fbe0112d700 10 monclient: handle_config config(0 keys) v1 2021-05-25T11:59:53.929 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.904+0000 7fbe04393700 10 monclient: get_monmap_and_config success 2021-05-25T11:59:53.929 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.904+0000 7fbe0112d700 1 -- 172.21.15.102:0/4187406066 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fbdf8001fc0 con 0x7fbdfc0695e0 2021-05-25T11:59:53.929 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.904+0000 7fbe04393700 4 set_mon_vals no callback set 2021-05-25T11:59:53.929 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.904+0000 7fbe0112d700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:53.929 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.904+0000 7fbe0112d700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T11:59:53.930 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.904+0000 7fbe0112d700 10 monclient: dump: 2021-05-25T11:59:53.930 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:53.930 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:53.930 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:53.931 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:53.931 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:53.931 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:53.931 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:53.932 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:53.932 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:53.932 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:53.932 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.904+0000 7fbe04393700 10 monclient: shutdown 2021-05-25T11:59:53.932 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.904+0000 7fbe04393700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T11:59:53.933 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.904+0000 7fbe04393700 1 -- 172.21.15.102:0/4187406066 >> v1:172.21.15.102:6790/0 conn(0x7fbdfc0695e0 legacy=0x7fbdfc132160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:53.933 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.904+0000 7fbe04393700 1 -- 172.21.15.102:0/4187406066 shutdown_connections 2021-05-25T11:59:53.933 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.904+0000 7fbe04393700 1 -- 172.21.15.102:0/4187406066 >> 172.21.15.102:0/4187406066 conn(0x7fbdfc09b370 msgr2=0x7fbdfc097430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T11:59:53.933 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.904+0000 7fbe04393700 1 -- 172.21.15.102:0/4187406066 shutdown_connections 2021-05-25T11:59:53.934 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.904+0000 7fbe04393700 1 -- 172.21.15.102:0/4187406066 wait complete. 2021-05-25T11:59:53.934 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.905+0000 7fbe04393700 10 monclient: build_initial_monmap 2021-05-25T11:59:53.934 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.905+0000 7fbe04393700 10 monclient: monmap: 2021-05-25T11:59:53.934 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T11:59:53.934 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T11:59:53.935 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T11:59:53.935 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T11:59:53.935 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T11:59:53.935 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T11:59:53.935 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T11:59:53.936 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T11:59:53.936 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T11:59:53.936 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:53.936 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.905+0000 7fbe04393700 1 librados: starting msgr at 2021-05-25T11:59:53.936 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.905+0000 7fbe04393700 1 librados: starting objecter 2021-05-25T11:59:53.937 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.905+0000 7fbe04393700 1 Processor -- start 2021-05-25T11:59:53.937 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.905+0000 7fbe04393700 1 -- start start 2021-05-25T11:59:53.937 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.905+0000 7fbe04393700 1 librados: setting wanted keys 2021-05-25T11:59:53.938 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.905+0000 7fbe04393700 1 librados: calling monclient init 2021-05-25T11:59:53.938 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.905+0000 7fbe04393700 10 monclient: init 2021-05-25T11:59:53.938 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.905+0000 7fbe04393700 10 monclient: _reopen_session rank -1 2021-05-25T11:59:53.938 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.905+0000 7fbe04393700 10 monclient: _add_conns ranks=[1,0,2] 2021-05-25T11:59:53.939 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.905+0000 7fbe04393700 10 monclient(hunting): picked mon.noname-b con 0x7fbdfc068d50 addr v1:172.21.15.181:6789/0 2021-05-25T11:59:53.939 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.905+0000 7fbe04393700 10 monclient(hunting): picked mon.noname-a con 0x7fbdfc069130 addr v1:172.21.15.102:6789/0 2021-05-25T11:59:53.939 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.905+0000 7fbe04393700 10 monclient(hunting): picked mon.noname-c con 0x7fbdfc0695e0 addr v1:172.21.15.102:6790/0 2021-05-25T11:59:53.940 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.905+0000 7fbe04393700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdfc1220b0 con 0x7fbdfc069130 2021-05-25T11:59:53.940 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.905+0000 7fbe04393700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdfc122350 con 0x7fbdfc068d50 2021-05-25T11:59:53.940 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.905+0000 7fbe04393700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdfc125f30 con 0x7fbdfc0695e0 2021-05-25T11:59:53.941 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.905+0000 7fbe04393700 10 monclient(hunting): _renew_subs 2021-05-25T11:59:53.941 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.905+0000 7fbe04393700 10 monclient(hunting): authenticate will time out at 794.502971s 2021-05-25T11:59:53.942 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.906+0000 7fbe0192e700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fbdfc069130 0x7fbdfc139100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:58758/0 (socket says 172.21.15.102:58758) 2021-05-25T11:59:53.942 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.906+0000 7fbe0192e700 1 -- 172.21.15.102:0/952713175 learned_addr learned my addr 172.21.15.102:0/952713175 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T11:59:53.943 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.906+0000 7fbdea7fc700 10 client.?.objecter ms_handle_connect 0x7fbdfc0695e0 2021-05-25T11:59:53.943 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.906+0000 7fbdea7fc700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:53.943 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.906+0000 7fbdea7fc700 10 client.?.objecter ms_handle_connect 0x7fbdfc069130 2021-05-25T11:59:53.944 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.906+0000 7fbdea7fc700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:53.944 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.906+0000 7fbdea7fc700 10 client.?.objecter ms_handle_connect 0x7fbdfc068d50 2021-05-25T11:59:53.945 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.906+0000 7fbdea7fc700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:53.945 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.906+0000 7fbdea7fc700 1 -- 172.21.15.102:0/952713175 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2527678720 0 0) 0x7fbdfc125f30 con 0x7fbdfc0695e0 2021-05-25T11:59:53.945 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.906+0000 7fbdea7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:53.946 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.906+0000 7fbdea7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:53.946 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.906+0000 7fbdea7fc700 10 monclient(hunting): my global_id is 4244 2021-05-25T11:59:53.947 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.906+0000 7fbdea7fc700 1 -- 172.21.15.102:0/952713175 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbddc004c10 con 0x7fbdfc0695e0 2021-05-25T11:59:53.947 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.906+0000 7fbdea7fc700 1 -- 172.21.15.102:0/952713175 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3208121755 0 0) 0x7fbdfc1220b0 con 0x7fbdfc069130 2021-05-25T11:59:53.948 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.906+0000 7fbdea7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:53.948 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.906+0000 7fbdea7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:53.948 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.906+0000 7fbdea7fc700 10 monclient(hunting): my global_id is 4251 2021-05-25T11:59:53.949 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.906+0000 7fbdea7fc700 1 -- 172.21.15.102:0/952713175 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdfc125f30 con 0x7fbdfc069130 2021-05-25T11:59:53.949 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.906+0000 7fbdea7fc700 1 -- 172.21.15.102:0/952713175 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3344776651 0 0) 0x7fbdfc122350 con 0x7fbdfc068d50 2021-05-25T11:59:53.949 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.906+0000 7fbdea7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:53.949 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.906+0000 7fbdea7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:53.950 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.906+0000 7fbdea7fc700 10 monclient(hunting): my global_id is 4246 2021-05-25T11:59:53.950 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.906+0000 7fbdea7fc700 1 -- 172.21.15.102:0/952713175 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdfc1220b0 con 0x7fbdfc068d50 2021-05-25T11:59:53.950 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.906+0000 7fbdea7fc700 1 -- 172.21.15.102:0/952713175 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 179651595 0 0) 0x7fbdfc125f30 con 0x7fbdfc069130 2021-05-25T11:59:53.950 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.906+0000 7fbdea7fc700 1 -- 172.21.15.102:0/952713175 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbdfc122350 con 0x7fbdfc069130 2021-05-25T11:59:53.951 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.906+0000 7fbdea7fc700 1 -- 172.21.15.102:0/952713175 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1734650011 0 0) 0x7fbddc004c10 con 0x7fbdfc0695e0 2021-05-25T11:59:53.951 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.906+0000 7fbdea7fc700 1 -- 172.21.15.102:0/952713175 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbdfc125f30 con 0x7fbdfc0695e0 2021-05-25T11:59:53.951 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.906+0000 7fbdea7fc700 1 -- 172.21.15.102:0/952713175 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fbdf0002990 con 0x7fbdfc069130 2021-05-25T11:59:53.951 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.906+0000 7fbdea7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:53.951 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.906+0000 7fbdea7fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T11:59:53.952 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.906+0000 7fbdea7fc700 10 monclient(hunting): dump: 2021-05-25T11:59:53.952 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:53.952 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:53.952 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:53.952 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:53.953 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:53.953 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:53.953 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:53.953 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:53.953 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:53.954 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:53.954 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.906+0000 7fbdea7fc700 10 monclient(hunting): _finish_auth 0 2021-05-25T11:59:53.954 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.906+0000 7fbdea7fc700 1 -- 172.21.15.102:0/952713175 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fbdf8001da0 con 0x7fbdfc0695e0 2021-05-25T11:59:53.954 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.906+0000 7fbdea7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:53.955 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.906+0000 7fbdea7fc700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2021-05-25T11:59:53.955 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.906+0000 7fbdea7fc700 10 monclient(hunting): dump: 2021-05-25T11:59:53.955 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:53.955 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:53.955 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:53.956 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:53.956 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:53.956 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:53.957 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:53.957 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:53.957 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:53.957 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:53.957 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.906+0000 7fbdea7fc700 1 -- 172.21.15.102:0/952713175 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2894361976 0 0) 0x7fbdfc1220b0 con 0x7fbdfc068d50 2021-05-25T11:59:53.958 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.906+0000 7fbdea7fc700 1 -- 172.21.15.102:0/952713175 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbddc004c10 con 0x7fbdfc068d50 2021-05-25T11:59:53.958 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.906+0000 7fbdea7fc700 1 -- 172.21.15.102:0/952713175 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fbdec001800 con 0x7fbdfc068d50 2021-05-25T11:59:53.958 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.906+0000 7fbdea7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:53.958 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.906+0000 7fbdea7fc700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2021-05-25T11:59:53.959 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.906+0000 7fbdea7fc700 10 monclient(hunting): dump: 2021-05-25T11:59:53.959 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:53.960 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:53.960 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:53.960 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:53.961 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:53.961 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:53.961 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:53.961 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:53.961 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:53.962 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:53.962 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.906+0000 7fbdea7fc700 1 -- 172.21.15.102:0/952713175 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1533651509 0 0) 0x7fbdfc122350 con 0x7fbdfc069130 2021-05-25T11:59:53.962 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.906+0000 7fbdea7fc700 1 -- 172.21.15.102:0/952713175 >> v1:172.21.15.102:6790/0 conn(0x7fbdfc0695e0 legacy=0x7fbdfc141290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:53.962 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.906+0000 7fbdea7fc700 1 -- 172.21.15.102:0/952713175 >> v1:172.21.15.181:6789/0 conn(0x7fbdfc068d50 legacy=0x7fbdfc125820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:53.963 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.906+0000 7fbdea7fc700 10 monclient: _finish_hunting 0 2021-05-25T11:59:53.963 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.906+0000 7fbdea7fc700 1 monclient: found mon.a 2021-05-25T11:59:53.963 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.906+0000 7fbdea7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T11:59:53.963 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.906+0000 7fbdea7fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:53.963 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.906+0000 7fbdea7fc700 1 -- 172.21.15.102:0/952713175 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdfc139810 con 0x7fbdfc069130 2021-05-25T11:59:53.964 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.906+0000 7fbdea7fc700 10 monclient: _finish_auth 0 2021-05-25T11:59:53.964 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.906+0000 7fbdea7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T11:59:53.964 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.906+0000 7fbe04393700 5 monclient: authenticate success, global_id 4251 2021-05-25T11:59:53.964 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.906+0000 7fbe04393700 10 monclient: _renew_subs 2021-05-25T11:59:53.964 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.906+0000 7fbe04393700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:53.965 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.906+0000 7fbe04393700 1 -- 172.21.15.102:0/952713175 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdfc068a20 con 0x7fbdfc069130 2021-05-25T11:59:53.965 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.906+0000 7fbe04393700 10 client.4251.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T11:59:53.965 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.906+0000 7fbe04393700 10 monclient: _renew_subs 2021-05-25T11:59:53.965 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.906+0000 7fbe04393700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:53.966 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.906+0000 7fbe04393700 1 -- 172.21.15.102:0/952713175 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdfc068160 con 0x7fbdfc069130 2021-05-25T11:59:53.966 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.906+0000 7fbe04393700 1 librados: init done 2021-05-25T11:59:53.966 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.906+0000 7fbdea7fc700 1 -- 172.21.15.102:0/952713175 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbdf0002d20 con 0x7fbdfc069130 2021-05-25T11:59:53.966 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.906+0000 7fbdea7fc700 10 monclient: handle_config config(0 keys) v1 2021-05-25T11:59:53.967 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.906+0000 7fbdea7fc700 1 -- 172.21.15.102:0/952713175 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fbdf0001820 con 0x7fbdfc069130 2021-05-25T11:59:53.967 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.906+0000 7fbdea7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:53.967 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.906+0000 7fbdea7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T11:59:53.967 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.906+0000 7fbe0092c700 4 set_mon_vals no callback set 2021-05-25T11:59:53.968 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.906+0000 7fbdea7fc700 10 monclient: dump: 2021-05-25T11:59:53.968 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:53.968 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:53.968 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:53.968 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:53.969 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:53.969 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:53.969 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:53.969 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:53.969 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:53.970 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:53.970 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.907+0000 7fbdea7fc700 1 -- 172.21.15.102:0/952713175 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7fbdf0001a00 con 0x7fbdfc069130 2021-05-25T11:59:53.970 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.907+0000 7fbdea7fc700 1 -- 172.21.15.102:0/952713175 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(15..15 src has 1..15) v4 ==== 3549+0+0 (unknown 3948819498 0 0) 0x7fbdf003ce10 con 0x7fbdfc069130 2021-05-25T11:59:53.971 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.907+0000 7fbdea7fc700 10 client.4251.objecter ms_dispatch 0x7fbdfc001550 osd_map(15..15 src has 1..15) v4 2021-05-25T11:59:53.971 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.907+0000 7fbdea7fc700 3 client.4251.objecter handle_osd_map got epochs [15,15] > 0 2021-05-25T11:59:53.971 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.907+0000 7fbdea7fc700 3 client.4251.objecter handle_osd_map decoding full epoch 15 2021-05-25T11:59:53.971 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.907+0000 7fbdea7fc700 20 client.4251.objecter dump_active .. 0 homeless 2021-05-25T11:59:53.971 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.907+0000 7fbe04393700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:53.972 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.907+0000 7fbe04393700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:53.972 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.907+0000 7fbe04393700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:53.972 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.907+0000 7fbe04393700 1 -- 172.21.15.102:0/952713175 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbde0004db0 con 0x7fbdfc069130 2021-05-25T11:59:53.972 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.907+0000 7fbe0212f700 10 monclient: get_auth_request con 0x7fbddc033510 auth_method 0 2021-05-25T11:59:53.973 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.908+0000 7fbdea7fc700 10 client.4251.objecter ms_handle_connect 0x7fbddc033510 2021-05-25T11:59:53.973 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.911+0000 7fbdea7fc700 1 -- 172.21.15.102:0/952713175 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7fbdf0012db0 con 0x7fbdfc069130 2021-05-25T11:59:53.973 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.911+0000 7fbdea7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:53.973 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:53.911+0000 7fbdea7fc700 10 monclient: _finish_command 1 = system:0 2021-05-25T11:59:54.064 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.062+0000 7fbe04393700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2021-05-25T11:59:54.064 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.062+0000 7fbe04393700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2021-05-25T11:59:54.065 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.062+0000 7fbe04393700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:54.065 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.062+0000 7fbe04393700 1 -- 172.21.15.102:0/952713175 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbde0005060 con 0x7fbdfc069130 2021-05-25T11:59:54.065 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.063+0000 7fbdea7fc700 1 -- 172.21.15.102:0/952713175 <== mon.0 v1:172.21.15.102:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) v1 ==== 74+0+10024 (unknown 4221056075 0 1374345216) 0x7fbdf00129a0 con 0x7fbdfc069130 2021-05-25T11:59:54.066 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.063+0000 7fbdea7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2021-05-25T11:59:54.066 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.063+0000 7fbdea7fc700 10 monclient: _finish_command 2 = system:0 2021-05-25T11:59:54.066 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:59:54.067 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":15,"fsid":"20ee91b6-8935-4b98-b868-85c03789ccc4","created":"2021-05-25T11:59:41.238908+0000","modified":"2021-05-25T11:59:53.591171+0000","last_up_change":"2021-05-25T11:59:49.572449+0000","last_in_change":"2021-05-25T11:59:42.972556+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-05-25T11:59:49.609415+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-05-25T11:59:50.997242+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"8f1f285f-7041-4c53-ab09-f638ab670036","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6807","nonce":29973}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6809","nonce":29973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6811","nonce":29973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6810","nonce":29973}]},"public_addr":"172.21.15.102:6807/29973","cluster_addr":"172.21.15.102:6809/29973","heartbeat_back_addr":"172.21.15.102:6811/29973","heartbeat_front_addr":"172.21.15.102:6810/29973","state":["exists","up"]},{"osd":1,"uuid":"d0d92491-3d97-4e4e-acf4-9dc7c0e4701f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6802","nonce":29974}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6804","nonce":29974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6808","nonce":29974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6806","nonce":29974}]},"public_addr":"172.21.15.102:6802/29974","cluster_addr":"172.21.15.102:6804/29974","heartbeat_back_addr":"172.21.15.102:6808/29974","heartbeat_front_addr":"172.21.15.102:6806/29974","state":["exists","up"]},{"osd":2,"uuid":"80a3eb90-b36e-4035-af91-3917fbf8dc2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6800","nonce":29975}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6801","nonce":29975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6805","nonce":29975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6803","nonce":29975}]},"public_addr":"172.21.15.102:6800/29975","cluster_addr":"172.21.15.102:6801/29975","heartbeat_back_addr":"172.21.15.102:6805/29975","heartbeat_front_addr":"172.21.15.102:6803/29975","state":["exists","up"]},{"osd":3,"uuid":"45889d6a-dcfb-4670-afef-cee9bbb0476a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6804","nonce":30005}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6805","nonce":30005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6807","nonce":30005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6806","nonce":30005}]},"public_addr":"172.21.15.181:6804/30005","cluster_addr":"172.21.15.181:6805/30005","heartbeat_back_addr":"172.21.15.181:6807/30005","heartbeat_front_addr":"172.21.15.181:6806/30005","state":["exists","up"]},{"osd":4,"uuid":"69e097b7-1472-442f-af42-225c0e40ad5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6800","nonce":30006}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6801","nonce":30006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6803","nonce":30006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6802","nonce":30006}]},"public_addr":"172.21.15.181:6800/30006","cluster_addr":"172.21.15.181:6801/30006","heartbeat_back_addr":"172.21.15.181:6803/30006","heartbeat_front_addr":"172.21.15.181:6802/30006","state":["exists","up"]},{"osd":5,"uuid":"f1045691-db58-471e-943a-e4d99129a917","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6808","nonce":30008}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6809","nonce":30008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6811","nonce":30008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6810","nonce":30008}]},"public_addr":"172.21.15.181:6808/30008","cluster_addr":"172.21.15.181:6809/30008","heartbeat_back_addr":"172.21.15.181:6811/30008","heartbeat_front_addr":"172.21.15.181:6810/30008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-05-25T11:59:48.129055+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-05-25T11:59:48.090298+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-05-25T11:59:48.049016+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-05-25T11:59:48.069527+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-05-25T11:59:48.002930+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-05-25T11:59:48.052367+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-05-25T11:59:54.068 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.065+0000 7fbe04393700 10 librados: watch_flush enter 2021-05-25T11:59:54.069 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.065+0000 7fbe04393700 10 librados: watch_flush exit 2021-05-25T11:59:54.069 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.065+0000 7fbe04393700 20 client.4251.objecter shutdown clearing up homeless session... 2021-05-25T11:59:54.069 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.065+0000 7fbe04393700 10 client.4251.objecter successfully canceled tick 2021-05-25T11:59:54.069 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.065+0000 7fbe04393700 1 -- 172.21.15.102:0/952713175 >> v1:172.21.15.181:6812/29582 conn(0x7fbddc033510 legacy=0x7fbddc0359d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:54.070 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.065+0000 7fbe04393700 10 monclient: shutdown 2021-05-25T11:59:54.070 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.065+0000 7fbe04393700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T11:59:54.070 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.065+0000 7fbe04393700 1 -- 172.21.15.102:0/952713175 >> v1:172.21.15.102:6789/0 conn(0x7fbdfc069130 legacy=0x7fbdfc139100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:54.070 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.065+0000 7fbe04393700 1 -- 172.21.15.102:0/952713175 shutdown_connections 2021-05-25T11:59:54.070 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.065+0000 7fbe04393700 1 -- 172.21.15.102:0/952713175 >> 172.21.15.102:0/952713175 conn(0x7fbdfc09b370 msgr2=0x7fbdfc0989c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T11:59:54.071 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.065+0000 7fbe04393700 1 -- 172.21.15.102:0/952713175 shutdown_connections 2021-05-25T11:59:54.071 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.065+0000 7fbe04393700 1 -- 172.21.15.102:0/952713175 wait complete. 2021-05-25T11:59:54.071 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.065+0000 7fbe04393700 1 librados: shutdown 2021-05-25T11:59:54.077 INFO:tasks.ceph.ceph_manager.ceph:all up! 2021-05-25T11:59:54.078 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-05-25T11:59:54.186 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.185+0000 7fe826514700 10 monclient: get_monmap_and_config 2021-05-25T11:59:54.187 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.185+0000 7fe826514700 10 monclient: build_initial_monmap 2021-05-25T11:59:54.187 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.185+0000 7fe826514700 10 monclient: monmap: 2021-05-25T11:59:54.187 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T11:59:54.187 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:54.187 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:54.186040+0000 2021-05-25T11:59:54.188 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:54.186040+0000 2021-05-25T11:59:54.188 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T11:59:54.188 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:54.188 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T11:59:54.188 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T11:59:54.189 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T11:59:54.189 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:54.199 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.185+0000 7fe826514700 1 Processor -- start 2021-05-25T11:59:54.199 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.185+0000 7fe826514700 1 -- start start 2021-05-25T11:59:54.199 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.185+0000 7fe826514700 10 monclient: init 2021-05-25T11:59:54.200 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.186+0000 7fe826514700 10 monclient: _reopen_session rank -1 2021-05-25T11:59:54.200 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.186+0000 7fe826514700 10 monclient: _add_conns ranks=[0,2,1] 2021-05-25T11:59:54.200 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.186+0000 7fe826514700 10 monclient(hunting): picked mon.noname-a con 0x7fe8201283d0 addr v1:172.21.15.102:6789/0 2021-05-25T11:59:54.200 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.186+0000 7fe826514700 10 monclient(hunting): picked mon.noname-b con 0x7fe820069100 addr v1:172.21.15.102:6790/0 2021-05-25T11:59:54.200 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.186+0000 7fe826514700 10 monclient(hunting): picked mon.noname-c con 0x7fe8200695b0 addr v1:172.21.15.181:6789/0 2021-05-25T11:59:54.201 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.186+0000 7fe826514700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8200664d0 con 0x7fe8201283d0 2021-05-25T11:59:54.201 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.186+0000 7fe826514700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe820097b50 con 0x7fe8200695b0 2021-05-25T11:59:54.201 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.186+0000 7fe826514700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe820129080 con 0x7fe820069100 2021-05-25T11:59:54.201 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.186+0000 7fe826514700 10 monclient(hunting): _renew_subs 2021-05-25T11:59:54.202 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.186+0000 7fe826514700 10 monclient(hunting): authenticate will time out at 794.783332s 2021-05-25T11:59:54.202 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.186+0000 7fe81ffff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fe8201283d0 0x7fe820128970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:58764/0 (socket says 172.21.15.102:58764) 2021-05-25T11:59:54.202 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.186+0000 7fe81f7fe700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fe820069100 0x7fe82012d740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:53644/0 (socket says 172.21.15.102:53644) 2021-05-25T11:59:54.202 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.186+0000 7fe81ffff700 1 -- 172.21.15.102:0/2337581786 learned_addr learned my addr 172.21.15.102:0/2337581786 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T11:59:54.203 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.186+0000 7fe81effd700 1 -- 172.21.15.102:0/2337581786 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2170765546 0 0) 0x7fe820129080 con 0x7fe820069100 2021-05-25T11:59:54.203 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.186+0000 7fe81effd700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:54.203 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.186+0000 7fe81effd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:54.203 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.186+0000 7fe81effd700 10 monclient(hunting): my global_id is 4247 2021-05-25T11:59:54.203 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.186+0000 7fe81effd700 1 -- 172.21.15.102:0/2337581786 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe808003270 con 0x7fe820069100 2021-05-25T11:59:54.204 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.186+0000 7fe81effd700 1 -- 172.21.15.102:0/2337581786 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 613379737 0 0) 0x7fe8200664d0 con 0x7fe8201283d0 2021-05-25T11:59:54.204 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.186+0000 7fe81effd700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:54.204 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.186+0000 7fe81effd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:54.204 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.186+0000 7fe81effd700 10 monclient(hunting): my global_id is 4254 2021-05-25T11:59:54.204 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.186+0000 7fe81effd700 1 -- 172.21.15.102:0/2337581786 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe820129080 con 0x7fe8201283d0 2021-05-25T11:59:54.205 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.186+0000 7fe81effd700 1 -- 172.21.15.102:0/2337581786 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2549223536 0 0) 0x7fe808003270 con 0x7fe820069100 2021-05-25T11:59:54.205 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.186+0000 7fe81effd700 1 -- 172.21.15.102:0/2337581786 >> v1:172.21.15.181:6789/0 conn(0x7fe8200695b0 legacy=0x7fe820130e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:54.205 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.186+0000 7fe81effd700 1 -- 172.21.15.102:0/2337581786 >> v1:172.21.15.102:6789/0 conn(0x7fe8201283d0 legacy=0x7fe820128970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:54.205 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.186+0000 7fe81effd700 10 monclient: _finish_hunting 0 2021-05-25T11:59:54.206 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.186+0000 7fe81effd700 1 monclient: found mon.noname-b 2021-05-25T11:59:54.206 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.186+0000 7fe81effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T11:59:54.206 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.186+0000 7fe81effd700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T11:59:54.206 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.186+0000 7fe81effd700 1 -- 172.21.15.102:0/2337581786 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe820066700 con 0x7fe820069100 2021-05-25T11:59:54.206 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.186+0000 7fe81effd700 10 monclient: _finish_auth 0 2021-05-25T11:59:54.207 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.186+0000 7fe81effd700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T11:59:54.207 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.186+0000 7fe81effd700 1 -- 172.21.15.102:0/2337581786 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fe80c001610 con 0x7fe820069100 2021-05-25T11:59:54.207 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.186+0000 7fe826514700 5 monclient: authenticate success, global_id 4247 2021-05-25T11:59:54.207 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.186+0000 7fe826514700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T11:59:54.208 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.186+0000 7fe81effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:54.208 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.186+0000 7fe81effd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T11:59:54.208 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.187+0000 7fe81effd700 10 monclient: dump: 2021-05-25T11:59:54.208 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:54.208 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:54.209 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:54.209 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:54.209 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:54.209 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:54.209 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:54.210 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:54.210 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:54.210 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:54.210 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.187+0000 7fe826514700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T11:59:54.211 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.187+0000 7fe81effd700 1 -- 172.21.15.102:0/2337581786 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe80c001840 con 0x7fe820069100 2021-05-25T11:59:54.211 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.187+0000 7fe81effd700 10 monclient: handle_config config(0 keys) v1 2021-05-25T11:59:54.211 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.187+0000 7fe826514700 10 monclient: get_monmap_and_config success 2021-05-25T11:59:54.211 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.187+0000 7fe81effd700 1 -- 172.21.15.102:0/2337581786 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fe80c001bd0 con 0x7fe820069100 2021-05-25T11:59:54.211 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.187+0000 7fe826514700 4 set_mon_vals no callback set 2021-05-25T11:59:54.212 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.187+0000 7fe81effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:54.212 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.187+0000 7fe81effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T11:59:54.212 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.187+0000 7fe81effd700 10 monclient: dump: 2021-05-25T11:59:54.212 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:54.212 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:54.213 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:54.213 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:54.213 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:54.213 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:54.213 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:54.214 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:54.214 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:54.214 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:54.214 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.187+0000 7fe826514700 10 monclient: shutdown 2021-05-25T11:59:54.215 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.187+0000 7fe826514700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T11:59:54.215 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.187+0000 7fe826514700 1 -- 172.21.15.102:0/2337581786 >> v1:172.21.15.102:6790/0 conn(0x7fe820069100 legacy=0x7fe82012d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:54.215 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.187+0000 7fe826514700 1 -- 172.21.15.102:0/2337581786 shutdown_connections 2021-05-25T11:59:54.215 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.187+0000 7fe826514700 1 -- 172.21.15.102:0/2337581786 >> 172.21.15.102:0/2337581786 conn(0x7fe82009b920 msgr2=0x7fe82009bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T11:59:54.215 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.187+0000 7fe826514700 1 -- 172.21.15.102:0/2337581786 shutdown_connections 2021-05-25T11:59:54.216 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.187+0000 7fe826514700 1 -- 172.21.15.102:0/2337581786 wait complete. 2021-05-25T11:59:54.216 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.188+0000 7fe826514700 10 monclient: build_initial_monmap 2021-05-25T11:59:54.216 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.188+0000 7fe826514700 10 monclient: monmap: 2021-05-25T11:59:54.216 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T11:59:54.216 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T11:59:54.217 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T11:59:54.217 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T11:59:54.217 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T11:59:54.217 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T11:59:54.218 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T11:59:54.218 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T11:59:54.218 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T11:59:54.218 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:54.218 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.188+0000 7fe826514700 1 librados: starting msgr at 2021-05-25T11:59:54.219 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.188+0000 7fe826514700 1 librados: starting objecter 2021-05-25T11:59:54.219 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.188+0000 7fe826514700 1 Processor -- start 2021-05-25T11:59:54.219 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.188+0000 7fe826514700 1 -- start start 2021-05-25T11:59:54.219 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.188+0000 7fe826514700 1 librados: setting wanted keys 2021-05-25T11:59:54.220 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.188+0000 7fe826514700 1 librados: calling monclient init 2021-05-25T11:59:54.220 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.188+0000 7fe826514700 10 monclient: init 2021-05-25T11:59:54.220 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.188+0000 7fe826514700 10 monclient: _reopen_session rank -1 2021-05-25T11:59:54.220 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.188+0000 7fe826514700 10 monclient: _add_conns ranks=[0,2,1] 2021-05-25T11:59:54.220 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.188+0000 7fe826514700 10 monclient(hunting): picked mon.noname-a con 0x7fe820069100 addr v1:172.21.15.102:6789/0 2021-05-25T11:59:54.221 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.188+0000 7fe826514700 10 monclient(hunting): picked mon.noname-c con 0x7fe8200695b0 addr v1:172.21.15.102:6790/0 2021-05-25T11:59:54.221 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.189+0000 7fe826514700 10 monclient(hunting): picked mon.noname-b con 0x7fe8201283d0 addr v1:172.21.15.181:6789/0 2021-05-25T11:59:54.221 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.189+0000 7fe826514700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe820065840 con 0x7fe820069100 2021-05-25T11:59:54.221 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.189+0000 7fe826514700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe82009abb0 con 0x7fe8201283d0 2021-05-25T11:59:54.222 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.189+0000 7fe826514700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe820062980 con 0x7fe8200695b0 2021-05-25T11:59:54.222 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.189+0000 7fe826514700 10 monclient(hunting): _renew_subs 2021-05-25T11:59:54.222 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.189+0000 7fe826514700 10 monclient(hunting): authenticate will time out at 794.786041s 2021-05-25T11:59:54.222 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.189+0000 7fe81ffff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fe820069100 0x7fe820061b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:58770/0 (socket says 172.21.15.102:58770) 2021-05-25T11:59:54.222 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.189+0000 7fe81f7fe700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fe8200695b0 0x7fe820062230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:53650/0 (socket says 172.21.15.102:53650) 2021-05-25T11:59:54.223 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.189+0000 7fe81ffff700 1 -- 172.21.15.102:0/4271982146 learned_addr learned my addr 172.21.15.102:0/4271982146 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T11:59:54.223 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.189+0000 7fe813fff700 10 client.?.objecter ms_handle_connect 0x7fe820069100 2021-05-25T11:59:54.223 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.189+0000 7fe813fff700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:54.223 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.189+0000 7fe813fff700 10 client.?.objecter ms_handle_connect 0x7fe8200695b0 2021-05-25T11:59:54.224 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.189+0000 7fe813fff700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:54.224 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.189+0000 7fe813fff700 1 -- 172.21.15.102:0/4271982146 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2799530865 0 0) 0x7fe820065840 con 0x7fe820069100 2021-05-25T11:59:54.224 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.189+0000 7fe813fff700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:54.224 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.189+0000 7fe813fff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:54.224 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.189+0000 7fe813fff700 10 monclient(hunting): my global_id is 4257 2021-05-25T11:59:54.225 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.189+0000 7fe813fff700 1 -- 172.21.15.102:0/4271982146 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7fc004c10 con 0x7fe820069100 2021-05-25T11:59:54.225 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.189+0000 7fe813fff700 1 -- 172.21.15.102:0/4271982146 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2506942460 0 0) 0x7fe820062980 con 0x7fe8200695b0 2021-05-25T11:59:54.225 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.189+0000 7fe813fff700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:54.225 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.189+0000 7fe813fff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:54.225 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.189+0000 7fe813fff700 10 monclient(hunting): my global_id is 4250 2021-05-25T11:59:54.226 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.189+0000 7fe813fff700 1 -- 172.21.15.102:0/4271982146 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe820065840 con 0x7fe8200695b0 2021-05-25T11:59:54.226 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.189+0000 7fe813fff700 10 client.?.objecter ms_handle_connect 0x7fe8201283d0 2021-05-25T11:59:54.226 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.189+0000 7fe813fff700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:54.226 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.189+0000 7fe813fff700 1 -- 172.21.15.102:0/4271982146 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3984552197 0 0) 0x7fe82009abb0 con 0x7fe8201283d0 2021-05-25T11:59:54.227 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.189+0000 7fe813fff700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:54.227 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.189+0000 7fe813fff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:54.227 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.189+0000 7fe813fff700 10 monclient(hunting): my global_id is 4252 2021-05-25T11:59:54.227 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.189+0000 7fe813fff700 1 -- 172.21.15.102:0/4271982146 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe820062980 con 0x7fe8201283d0 2021-05-25T11:59:54.227 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.189+0000 7fe813fff700 1 -- 172.21.15.102:0/4271982146 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4151581434 0 0) 0x7fe7fc004c10 con 0x7fe820069100 2021-05-25T11:59:54.228 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.189+0000 7fe813fff700 1 -- 172.21.15.102:0/4271982146 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe82009abb0 con 0x7fe820069100 2021-05-25T11:59:54.228 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.189+0000 7fe813fff700 1 -- 172.21.15.102:0/4271982146 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fe8040019f0 con 0x7fe820069100 2021-05-25T11:59:54.228 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.189+0000 7fe813fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:54.228 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.189+0000 7fe813fff700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T11:59:54.229 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.189+0000 7fe813fff700 10 monclient(hunting): dump: 2021-05-25T11:59:54.229 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:54.229 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:54.229 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:54.229 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:54.230 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:54.230 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:54.230 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:54.230 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:54.231 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:54.231 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:54.231 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.189+0000 7fe813fff700 10 monclient(hunting): _finish_auth 0 2021-05-25T11:59:54.231 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.189+0000 7fe813fff700 1 -- 172.21.15.102:0/4271982146 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3768459707 0 0) 0x7fe820065840 con 0x7fe8200695b0 2021-05-25T11:59:54.231 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.189+0000 7fe813fff700 1 -- 172.21.15.102:0/4271982146 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe7fc004c10 con 0x7fe8200695b0 2021-05-25T11:59:54.232 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.189+0000 7fe813fff700 1 -- 172.21.15.102:0/4271982146 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fe80c001b50 con 0x7fe8200695b0 2021-05-25T11:59:54.232 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.189+0000 7fe813fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:54.232 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.189+0000 7fe813fff700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2021-05-25T11:59:54.232 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.189+0000 7fe813fff700 10 monclient(hunting): dump: 2021-05-25T11:59:54.232 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:54.233 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:54.233 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:54.233 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:54.233 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:54.233 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:54.234 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:54.234 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:54.234 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:54.234 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:54.235 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.189+0000 7fe813fff700 1 -- 172.21.15.102:0/4271982146 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1403518484 0 0) 0x7fe82009abb0 con 0x7fe820069100 2021-05-25T11:59:54.235 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.189+0000 7fe813fff700 1 -- 172.21.15.102:0/4271982146 >> v1:172.21.15.102:6790/0 conn(0x7fe8200695b0 legacy=0x7fe820062230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:54.235 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.189+0000 7fe813fff700 1 -- 172.21.15.102:0/4271982146 >> v1:172.21.15.181:6789/0 conn(0x7fe8201283d0 legacy=0x7fe8201416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:54.235 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.189+0000 7fe813fff700 10 monclient: _finish_hunting 0 2021-05-25T11:59:54.235 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.189+0000 7fe813fff700 1 monclient: found mon.a 2021-05-25T11:59:54.236 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.189+0000 7fe813fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T11:59:54.236 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.189+0000 7fe813fff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:54.236 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.189+0000 7fe813fff700 1 -- 172.21.15.102:0/4271982146 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe820124a80 con 0x7fe820069100 2021-05-25T11:59:54.236 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.189+0000 7fe813fff700 10 monclient: _finish_auth 0 2021-05-25T11:59:54.237 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.189+0000 7fe813fff700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T11:59:54.237 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.189+0000 7fe826514700 5 monclient: authenticate success, global_id 4257 2021-05-25T11:59:54.237 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.189+0000 7fe826514700 10 monclient: _renew_subs 2021-05-25T11:59:54.237 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.189+0000 7fe826514700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:54.237 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.189+0000 7fe826514700 1 -- 172.21.15.102:0/4271982146 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe820128190 con 0x7fe820069100 2021-05-25T11:59:54.238 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.189+0000 7fe826514700 10 client.4257.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T11:59:54.238 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.189+0000 7fe826514700 10 monclient: _renew_subs 2021-05-25T11:59:54.238 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.189+0000 7fe826514700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:54.238 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.189+0000 7fe826514700 1 -- 172.21.15.102:0/4271982146 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8200952d0 con 0x7fe820069100 2021-05-25T11:59:54.238 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.189+0000 7fe826514700 1 librados: init done 2021-05-25T11:59:54.239 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.189+0000 7fe813fff700 1 -- 172.21.15.102:0/4271982146 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe804001c80 con 0x7fe820069100 2021-05-25T11:59:54.239 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.189+0000 7fe813fff700 10 monclient: handle_config config(0 keys) v1 2021-05-25T11:59:54.239 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.189+0000 7fe813fff700 1 -- 172.21.15.102:0/4271982146 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fe804004980 con 0x7fe820069100 2021-05-25T11:59:54.239 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.189+0000 7fe813fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:54.240 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.189+0000 7fe81e7fc700 4 set_mon_vals no callback set 2021-05-25T11:59:54.240 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.189+0000 7fe813fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T11:59:54.240 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.189+0000 7fe813fff700 10 monclient: dump: 2021-05-25T11:59:54.240 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:54.240 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:54.241 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:54.241 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:54.241 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:54.241 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:54.242 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:54.242 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:54.242 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:54.242 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:54.242 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.190+0000 7fe813fff700 1 -- 172.21.15.102:0/4271982146 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7fe80400dca0 con 0x7fe820069100 2021-05-25T11:59:54.243 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.190+0000 7fe826514700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:54.243 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.190+0000 7fe826514700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:54.243 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.190+0000 7fe826514700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:54.243 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.190+0000 7fe826514700 1 -- 172.21.15.102:0/4271982146 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe800004db0 con 0x7fe820069100 2021-05-25T11:59:54.244 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.190+0000 7fe813fff700 1 -- 172.21.15.102:0/4271982146 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(15..15 src has 1..15) v4 ==== 3549+0+0 (unknown 3948819498 0 0) 0x7fe80403cde0 con 0x7fe820069100 2021-05-25T11:59:54.244 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.190+0000 7fe813fff700 10 client.4257.objecter ms_dispatch 0x7fe820001550 osd_map(15..15 src has 1..15) v4 2021-05-25T11:59:54.244 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.190+0000 7fe813fff700 3 client.4257.objecter handle_osd_map got epochs [15,15] > 0 2021-05-25T11:59:54.244 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.190+0000 7fe813fff700 3 client.4257.objecter handle_osd_map decoding full epoch 15 2021-05-25T11:59:54.244 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.190+0000 7fe813fff700 20 client.4257.objecter dump_active .. 0 homeless 2021-05-25T11:59:54.245 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.190+0000 7fe81f7fe700 10 monclient: get_auth_request con 0x7fe7fc033200 auth_method 0 2021-05-25T11:59:54.245 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.191+0000 7fe813fff700 10 client.4257.objecter ms_handle_connect 0x7fe7fc033200 2021-05-25T11:59:54.245 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.194+0000 7fe813fff700 1 -- 172.21.15.102:0/4271982146 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7fe804012bb0 con 0x7fe820069100 2021-05-25T11:59:54.245 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.194+0000 7fe813fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:54.245 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.194+0000 7fe813fff700 10 monclient: _finish_command 1 = system:0 2021-05-25T11:59:54.344 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.343+0000 7fe826514700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2021-05-25T11:59:54.345 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.343+0000 7fe826514700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2021-05-25T11:59:54.345 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.343+0000 7fe826514700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:54.345 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.343+0000 7fe826514700 1 -- 172.21.15.102:0/4271982146 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe800005060 con 0x7fe820069100 2021-05-25T11:59:54.346 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.344+0000 7fe813fff700 1 -- 172.21.15.102:0/4271982146 <== mon.0 v1:172.21.15.102:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) v1 ==== 74+0+10024 (unknown 4221056075 0 1374345216) 0x7fe804012bb0 con 0x7fe820069100 2021-05-25T11:59:54.346 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.344+0000 7fe813fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2021-05-25T11:59:54.346 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.344+0000 7fe813fff700 10 monclient: _finish_command 2 = system:0 2021-05-25T11:59:54.347 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:59:54.347 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":15,"fsid":"20ee91b6-8935-4b98-b868-85c03789ccc4","created":"2021-05-25T11:59:41.238908+0000","modified":"2021-05-25T11:59:53.591171+0000","last_up_change":"2021-05-25T11:59:49.572449+0000","last_in_change":"2021-05-25T11:59:42.972556+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-05-25T11:59:49.609415+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-05-25T11:59:50.997242+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"8f1f285f-7041-4c53-ab09-f638ab670036","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6807","nonce":29973}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6809","nonce":29973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6811","nonce":29973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6810","nonce":29973}]},"public_addr":"172.21.15.102:6807/29973","cluster_addr":"172.21.15.102:6809/29973","heartbeat_back_addr":"172.21.15.102:6811/29973","heartbeat_front_addr":"172.21.15.102:6810/29973","state":["exists","up"]},{"osd":1,"uuid":"d0d92491-3d97-4e4e-acf4-9dc7c0e4701f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6802","nonce":29974}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6804","nonce":29974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6808","nonce":29974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6806","nonce":29974}]},"public_addr":"172.21.15.102:6802/29974","cluster_addr":"172.21.15.102:6804/29974","heartbeat_back_addr":"172.21.15.102:6808/29974","heartbeat_front_addr":"172.21.15.102:6806/29974","state":["exists","up"]},{"osd":2,"uuid":"80a3eb90-b36e-4035-af91-3917fbf8dc2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6800","nonce":29975}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6801","nonce":29975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6805","nonce":29975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6803","nonce":29975}]},"public_addr":"172.21.15.102:6800/29975","cluster_addr":"172.21.15.102:6801/29975","heartbeat_back_addr":"172.21.15.102:6805/29975","heartbeat_front_addr":"172.21.15.102:6803/29975","state":["exists","up"]},{"osd":3,"uuid":"45889d6a-dcfb-4670-afef-cee9bbb0476a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6804","nonce":30005}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6805","nonce":30005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6807","nonce":30005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6806","nonce":30005}]},"public_addr":"172.21.15.181:6804/30005","cluster_addr":"172.21.15.181:6805/30005","heartbeat_back_addr":"172.21.15.181:6807/30005","heartbeat_front_addr":"172.21.15.181:6806/30005","state":["exists","up"]},{"osd":4,"uuid":"69e097b7-1472-442f-af42-225c0e40ad5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6800","nonce":30006}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6801","nonce":30006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6803","nonce":30006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6802","nonce":30006}]},"public_addr":"172.21.15.181:6800/30006","cluster_addr":"172.21.15.181:6801/30006","heartbeat_back_addr":"172.21.15.181:6803/30006","heartbeat_front_addr":"172.21.15.181:6802/30006","state":["exists","up"]},{"osd":5,"uuid":"f1045691-db58-471e-943a-e4d99129a917","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6808","nonce":30008}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6809","nonce":30008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6811","nonce":30008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6810","nonce":30008}]},"public_addr":"172.21.15.181:6808/30008","cluster_addr":"172.21.15.181:6809/30008","heartbeat_back_addr":"172.21.15.181:6811/30008","heartbeat_front_addr":"172.21.15.181:6810/30008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-05-25T11:59:48.129055+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-05-25T11:59:48.090298+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-05-25T11:59:48.049016+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-05-25T11:59:48.069527+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-05-25T11:59:48.002930+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-05-25T11:59:48.052367+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-05-25T11:59:54.348 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.345+0000 7fe826514700 10 librados: watch_flush enter 2021-05-25T11:59:54.349 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.345+0000 7fe826514700 10 librados: watch_flush exit 2021-05-25T11:59:54.349 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.345+0000 7fe826514700 20 client.4257.objecter shutdown clearing up homeless session... 2021-05-25T11:59:54.349 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.345+0000 7fe826514700 10 client.4257.objecter successfully canceled tick 2021-05-25T11:59:54.349 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.345+0000 7fe826514700 1 -- 172.21.15.102:0/4271982146 >> v1:172.21.15.181:6812/29582 conn(0x7fe7fc033200 legacy=0x7fe7fc0356c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:54.350 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.345+0000 7fe826514700 10 monclient: shutdown 2021-05-25T11:59:54.350 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.345+0000 7fe826514700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T11:59:54.350 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.345+0000 7fe826514700 1 -- 172.21.15.102:0/4271982146 >> v1:172.21.15.102:6789/0 conn(0x7fe820069100 legacy=0x7fe820061b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:54.350 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.345+0000 7fe826514700 1 -- 172.21.15.102:0/4271982146 shutdown_connections 2021-05-25T11:59:54.351 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.345+0000 7fe826514700 1 -- 172.21.15.102:0/4271982146 >> 172.21.15.102:0/4271982146 conn(0x7fe82009b920 msgr2=0x7fe82011f380 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T11:59:54.351 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.345+0000 7fe826514700 1 -- 172.21.15.102:0/4271982146 shutdown_connections 2021-05-25T11:59:54.351 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.345+0000 7fe826514700 1 -- 172.21.15.102:0/4271982146 wait complete. 2021-05-25T11:59:54.351 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.345+0000 7fe826514700 1 librados: shutdown 2021-05-25T11:59:54.358 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2021-05-25T11:59:54.467 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.466+0000 7ff2e91da700 10 monclient: get_monmap_and_config 2021-05-25T11:59:54.468 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.466+0000 7ff2e91da700 10 monclient: build_initial_monmap 2021-05-25T11:59:54.468 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.466+0000 7ff2e91da700 10 monclient: monmap: 2021-05-25T11:59:54.468 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T11:59:54.469 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:54.469 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:54.467469+0000 2021-05-25T11:59:54.469 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:54.467469+0000 2021-05-25T11:59:54.469 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T11:59:54.469 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:54.470 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T11:59:54.470 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T11:59:54.470 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T11:59:54.470 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:54.481 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.467+0000 7ff2e91da700 1 Processor -- start 2021-05-25T11:59:54.482 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.467+0000 7ff2e91da700 1 -- start start 2021-05-25T11:59:54.482 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.467+0000 7ff2e91da700 10 monclient: init 2021-05-25T11:59:54.482 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.467+0000 7ff2e91da700 10 monclient: _reopen_session rank -1 2021-05-25T11:59:54.482 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.467+0000 7ff2e91da700 10 monclient: _add_conns ranks=[0,1,2] 2021-05-25T11:59:54.482 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.467+0000 7ff2e91da700 10 monclient(hunting): picked mon.noname-a con 0x7ff2e4098930 addr v1:172.21.15.102:6789/0 2021-05-25T11:59:54.483 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.467+0000 7ff2e91da700 10 monclient(hunting): picked mon.noname-c con 0x7ff2e412bca0 addr v1:172.21.15.181:6789/0 2021-05-25T11:59:54.483 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.467+0000 7ff2e91da700 10 monclient(hunting): picked mon.noname-b con 0x7ff2e412c150 addr v1:172.21.15.102:6790/0 2021-05-25T11:59:54.483 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.467+0000 7ff2e91da700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2e4099b20 con 0x7ff2e4098930 2021-05-25T11:59:54.483 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.467+0000 7ff2e91da700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2e409a3a0 con 0x7ff2e412bca0 2021-05-25T11:59:54.484 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.467+0000 7ff2e91da700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2e409ac60 con 0x7ff2e412c150 2021-05-25T11:59:54.484 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.467+0000 7ff2e91da700 10 monclient(hunting): _renew_subs 2021-05-25T11:59:54.484 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.467+0000 7ff2e91da700 10 monclient(hunting): authenticate will time out at 795.064791s 2021-05-25T11:59:54.484 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.467+0000 7ff2e3fff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7ff2e4098930 0x7ff2e412b590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:58780/0 (socket says 172.21.15.102:58780) 2021-05-25T11:59:54.484 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.467+0000 7ff2e89d9700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7ff2e412c150 0x7ff2e4133ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:53658/0 (socket says 172.21.15.102:53658) 2021-05-25T11:59:54.485 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.467+0000 7ff2e3fff700 1 -- 172.21.15.102:0/3586146233 learned_addr learned my addr 172.21.15.102:0/3586146233 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T11:59:54.485 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.468+0000 7ff2e2ffd700 1 -- 172.21.15.102:0/3586146233 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 734677656 0 0) 0x7ff2e4099b20 con 0x7ff2e4098930 2021-05-25T11:59:54.485 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.468+0000 7ff2e2ffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:54.485 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.468+0000 7ff2e2ffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:54.485 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.468+0000 7ff2e2ffd700 10 monclient(hunting): my global_id is 4260 2021-05-25T11:59:54.486 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.468+0000 7ff2e2ffd700 1 -- 172.21.15.102:0/3586146233 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2c8003270 con 0x7ff2e4098930 2021-05-25T11:59:54.486 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.468+0000 7ff2e2ffd700 1 -- 172.21.15.102:0/3586146233 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1238869874 0 0) 0x7ff2e409ac60 con 0x7ff2e412c150 2021-05-25T11:59:54.486 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.468+0000 7ff2e2ffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:54.486 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.468+0000 7ff2e2ffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:54.487 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.468+0000 7ff2e2ffd700 10 monclient(hunting): my global_id is 4253 2021-05-25T11:59:54.490 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.468+0000 7ff2e2ffd700 1 -- 172.21.15.102:0/3586146233 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2e4099b20 con 0x7ff2e412c150 2021-05-25T11:59:54.491 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.468+0000 7ff2e2ffd700 1 -- 172.21.15.102:0/3586146233 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2214628196 0 0) 0x7ff2e409a3a0 con 0x7ff2e412bca0 2021-05-25T11:59:54.491 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.468+0000 7ff2e2ffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:54.491 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.468+0000 7ff2e2ffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:54.491 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.468+0000 7ff2e2ffd700 10 monclient(hunting): my global_id is 4255 2021-05-25T11:59:54.491 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.468+0000 7ff2e2ffd700 1 -- 172.21.15.102:0/3586146233 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2e409ac60 con 0x7ff2e412bca0 2021-05-25T11:59:54.492 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.468+0000 7ff2e2ffd700 1 -- 172.21.15.102:0/3586146233 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1289990930 0 0) 0x7ff2c8003270 con 0x7ff2e4098930 2021-05-25T11:59:54.492 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.468+0000 7ff2e2ffd700 1 -- 172.21.15.102:0/3586146233 >> v1:172.21.15.102:6790/0 conn(0x7ff2e412c150 legacy=0x7ff2e4133ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:54.492 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.468+0000 7ff2e2ffd700 1 -- 172.21.15.102:0/3586146233 >> v1:172.21.15.181:6789/0 conn(0x7ff2e412bca0 legacy=0x7ff2e41308c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:54.492 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.468+0000 7ff2e2ffd700 10 monclient: _finish_hunting 0 2021-05-25T11:59:54.493 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.468+0000 7ff2e2ffd700 1 monclient: found mon.noname-a 2021-05-25T11:59:54.493 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.468+0000 7ff2e2ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T11:59:54.493 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.468+0000 7ff2e2ffd700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T11:59:54.493 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.468+0000 7ff2e2ffd700 1 -- 172.21.15.102:0/3586146233 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2e409af00 con 0x7ff2e4098930 2021-05-25T11:59:54.493 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.468+0000 7ff2e2ffd700 10 monclient: _finish_auth 0 2021-05-25T11:59:54.494 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.468+0000 7ff2e2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T11:59:54.499 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.468+0000 7ff2e91da700 5 monclient: authenticate success, global_id 4260 2021-05-25T11:59:54.499 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.468+0000 7ff2e2ffd700 1 -- 172.21.15.102:0/3586146233 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7ff2d0001610 con 0x7ff2e4098930 2021-05-25T11:59:54.500 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.468+0000 7ff2e91da700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T11:59:54.500 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.468+0000 7ff2e2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:54.500 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.468+0000 7ff2e2ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T11:59:54.500 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.468+0000 7ff2e2ffd700 10 monclient: dump: 2021-05-25T11:59:54.500 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:54.501 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:54.501 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:54.501 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:54.501 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:54.501 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:54.502 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:54.502 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:54.502 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:54.502 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:54.503 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.468+0000 7ff2e91da700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T11:59:54.503 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.468+0000 7ff2e2ffd700 1 -- 172.21.15.102:0/3586146233 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff2d0001800 con 0x7ff2e4098930 2021-05-25T11:59:54.503 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.468+0000 7ff2e2ffd700 10 monclient: handle_config config(0 keys) v1 2021-05-25T11:59:54.503 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.468+0000 7ff2e91da700 10 monclient: get_monmap_and_config success 2021-05-25T11:59:54.503 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.468+0000 7ff2e2ffd700 1 -- 172.21.15.102:0/3586146233 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7ff2d0001b90 con 0x7ff2e4098930 2021-05-25T11:59:54.504 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.468+0000 7ff2e91da700 4 set_mon_vals no callback set 2021-05-25T11:59:54.504 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.468+0000 7ff2e2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:54.504 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.468+0000 7ff2e2ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T11:59:54.504 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.468+0000 7ff2e2ffd700 10 monclient: dump: 2021-05-25T11:59:54.504 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:54.505 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:54.506 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:54.506 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:54.506 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:54.506 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:54.506 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:54.507 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:54.507 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:54.507 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:54.507 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.468+0000 7ff2e91da700 10 monclient: shutdown 2021-05-25T11:59:54.507 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.468+0000 7ff2e91da700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T11:59:54.508 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.468+0000 7ff2e91da700 1 -- 172.21.15.102:0/3586146233 >> v1:172.21.15.102:6789/0 conn(0x7ff2e4098930 legacy=0x7ff2e412b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:54.508 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.468+0000 7ff2e91da700 1 -- 172.21.15.102:0/3586146233 shutdown_connections 2021-05-25T11:59:54.508 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.468+0000 7ff2e91da700 1 -- 172.21.15.102:0/3586146233 >> 172.21.15.102:0/3586146233 conn(0x7ff2e409e9d0 msgr2=0x7ff2e409ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T11:59:54.508 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.469+0000 7ff2e91da700 1 -- 172.21.15.102:0/3586146233 shutdown_connections 2021-05-25T11:59:54.509 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.469+0000 7ff2e91da700 1 -- 172.21.15.102:0/3586146233 wait complete. 2021-05-25T11:59:54.509 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.469+0000 7ff2e91da700 10 monclient: build_initial_monmap 2021-05-25T11:59:54.509 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.469+0000 7ff2e91da700 10 monclient: monmap: 2021-05-25T11:59:54.509 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T11:59:54.509 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T11:59:54.510 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T11:59:54.510 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T11:59:54.510 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T11:59:54.510 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T11:59:54.511 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T11:59:54.511 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T11:59:54.511 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T11:59:54.511 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:54.511 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.469+0000 7ff2e91da700 1 librados: starting msgr at 2021-05-25T11:59:54.512 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.469+0000 7ff2e91da700 1 librados: starting objecter 2021-05-25T11:59:54.512 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.469+0000 7ff2e91da700 1 Processor -- start 2021-05-25T11:59:54.512 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.469+0000 7ff2e91da700 1 -- start start 2021-05-25T11:59:54.512 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.469+0000 7ff2e91da700 1 librados: setting wanted keys 2021-05-25T11:59:54.513 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.469+0000 7ff2e91da700 1 librados: calling monclient init 2021-05-25T11:59:54.513 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.469+0000 7ff2e91da700 10 monclient: init 2021-05-25T11:59:54.513 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.470+0000 7ff2e91da700 10 monclient: _reopen_session rank -1 2021-05-25T11:59:54.513 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.470+0000 7ff2e91da700 10 monclient: _add_conns ranks=[0,2,1] 2021-05-25T11:59:54.513 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.470+0000 7ff2e91da700 10 monclient(hunting): picked mon.noname-a con 0x7ff2e4098930 addr v1:172.21.15.102:6789/0 2021-05-25T11:59:54.514 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.470+0000 7ff2e91da700 10 monclient(hunting): picked mon.noname-c con 0x7ff2e412bca0 addr v1:172.21.15.102:6790/0 2021-05-25T11:59:54.514 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.470+0000 7ff2e91da700 10 monclient(hunting): picked mon.noname-b con 0x7ff2e412c150 addr v1:172.21.15.181:6789/0 2021-05-25T11:59:54.514 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.470+0000 7ff2e91da700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2e4094710 con 0x7ff2e4098930 2021-05-25T11:59:54.514 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.470+0000 7ff2e91da700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2e40949b0 con 0x7ff2e412c150 2021-05-25T11:59:54.514 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.470+0000 7ff2e91da700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2e4094c50 con 0x7ff2e412bca0 2021-05-25T11:59:54.515 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.470+0000 7ff2e91da700 10 monclient(hunting): _renew_subs 2021-05-25T11:59:54.515 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.470+0000 7ff2e91da700 10 monclient(hunting): authenticate will time out at 795.067376s 2021-05-25T11:59:54.515 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.470+0000 7ff2e3fff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7ff2e4098930 0x7ff2e4097110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:58784/0 (socket says 172.21.15.102:58784) 2021-05-25T11:59:54.515 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.470+0000 7ff2e3fff700 1 -- 172.21.15.102:0/3869869503 learned_addr learned my addr 172.21.15.102:0/3869869503 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T11:59:54.516 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.470+0000 7ff2e37fe700 1 --1- 172.21.15.102:0/3869869503 >> v1:172.21.15.102:6790/0 conn(0x7ff2e412bca0 0x7ff2e413a9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:53664/0 (socket says 172.21.15.102:53664) 2021-05-25T11:59:54.516 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.470+0000 7ff2cffff700 10 client.?.objecter ms_handle_connect 0x7ff2e4098930 2021-05-25T11:59:54.516 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.470+0000 7ff2cffff700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:54.516 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.470+0000 7ff2cffff700 10 client.?.objecter ms_handle_connect 0x7ff2e412bca0 2021-05-25T11:59:54.516 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.470+0000 7ff2cffff700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:54.517 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.470+0000 7ff2cffff700 1 -- 172.21.15.102:0/3869869503 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2781720564 0 0) 0x7ff2e4094710 con 0x7ff2e4098930 2021-05-25T11:59:54.517 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.470+0000 7ff2cffff700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:54.517 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.470+0000 7ff2cffff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:54.517 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.470+0000 7ff2cffff700 10 monclient(hunting): my global_id is 4263 2021-05-25T11:59:54.518 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.470+0000 7ff2cffff700 1 -- 172.21.15.102:0/3869869503 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2c4004c10 con 0x7ff2e4098930 2021-05-25T11:59:54.518 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.470+0000 7ff2cffff700 10 client.?.objecter ms_handle_connect 0x7ff2e412c150 2021-05-25T11:59:54.518 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.470+0000 7ff2cffff700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:54.518 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.470+0000 7ff2cffff700 1 -- 172.21.15.102:0/3869869503 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2253345627 0 0) 0x7ff2e4094c50 con 0x7ff2e412bca0 2021-05-25T11:59:54.518 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.470+0000 7ff2cffff700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:54.519 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.470+0000 7ff2cffff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:54.519 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.470+0000 7ff2cffff700 10 monclient(hunting): my global_id is 4256 2021-05-25T11:59:54.519 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.470+0000 7ff2cffff700 1 -- 172.21.15.102:0/3869869503 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2e4094710 con 0x7ff2e412bca0 2021-05-25T11:59:54.519 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.470+0000 7ff2cffff700 1 -- 172.21.15.102:0/3869869503 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2312148271 0 0) 0x7ff2e40949b0 con 0x7ff2e412c150 2021-05-25T11:59:54.519 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.470+0000 7ff2cffff700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:54.520 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.470+0000 7ff2cffff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:54.520 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.470+0000 7ff2cffff700 10 monclient(hunting): my global_id is 4258 2021-05-25T11:59:54.520 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.470+0000 7ff2cffff700 1 -- 172.21.15.102:0/3869869503 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2e4094c50 con 0x7ff2e412c150 2021-05-25T11:59:54.520 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.470+0000 7ff2cffff700 1 -- 172.21.15.102:0/3869869503 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2233897429 0 0) 0x7ff2c4004c10 con 0x7ff2e4098930 2021-05-25T11:59:54.520 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.470+0000 7ff2cffff700 1 -- 172.21.15.102:0/3869869503 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff2e40949b0 con 0x7ff2e4098930 2021-05-25T11:59:54.521 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.470+0000 7ff2cffff700 1 -- 172.21.15.102:0/3869869503 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7ff2d0001510 con 0x7ff2e4098930 2021-05-25T11:59:54.521 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.470+0000 7ff2cffff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:54.521 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.470+0000 7ff2cffff700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T11:59:54.521 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.470+0000 7ff2cffff700 10 monclient(hunting): dump: 2021-05-25T11:59:54.522 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:54.522 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:54.522 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:54.522 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:54.522 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:54.523 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:54.523 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:54.523 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:54.523 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:54.523 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:54.524 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.470+0000 7ff2cffff700 10 monclient(hunting): _finish_auth 0 2021-05-25T11:59:54.524 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.470+0000 7ff2cffff700 1 -- 172.21.15.102:0/3869869503 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2554488436 0 0) 0x7ff2e4094710 con 0x7ff2e412bca0 2021-05-25T11:59:54.524 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.470+0000 7ff2cffff700 1 -- 172.21.15.102:0/3869869503 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff2c4004c10 con 0x7ff2e412bca0 2021-05-25T11:59:54.525 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.470+0000 7ff2cffff700 1 -- 172.21.15.102:0/3869869503 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7ff2d4001880 con 0x7ff2e412bca0 2021-05-25T11:59:54.525 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.470+0000 7ff2cffff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:54.525 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.470+0000 7ff2cffff700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2021-05-25T11:59:54.525 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.470+0000 7ff2cffff700 10 monclient(hunting): dump: 2021-05-25T11:59:54.525 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:54.526 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:54.526 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:54.526 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:54.526 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:54.527 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:54.527 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:54.527 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:54.527 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:54.527 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:54.528 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.471+0000 7ff2cffff700 1 -- 172.21.15.102:0/3869869503 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2826355852 0 0) 0x7ff2e4094c50 con 0x7ff2e412c150 2021-05-25T11:59:54.528 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.471+0000 7ff2cffff700 1 -- 172.21.15.102:0/3869869503 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff2e4094710 con 0x7ff2e412c150 2021-05-25T11:59:54.528 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.471+0000 7ff2cffff700 1 -- 172.21.15.102:0/3869869503 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7ff2d8001c80 con 0x7ff2e412c150 2021-05-25T11:59:54.528 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.471+0000 7ff2cffff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:54.528 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.471+0000 7ff2cffff700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2021-05-25T11:59:54.529 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.471+0000 7ff2cffff700 10 monclient(hunting): dump: 2021-05-25T11:59:54.529 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:54.529 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:54.529 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:54.529 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:54.530 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:54.530 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:54.530 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:54.530 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:54.531 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:54.531 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:54.531 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.471+0000 7ff2cffff700 1 -- 172.21.15.102:0/3869869503 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1888882972 0 0) 0x7ff2e40949b0 con 0x7ff2e4098930 2021-05-25T11:59:54.531 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.471+0000 7ff2cffff700 1 -- 172.21.15.102:0/3869869503 >> v1:172.21.15.102:6790/0 conn(0x7ff2e412bca0 legacy=0x7ff2e413a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:54.531 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.471+0000 7ff2cffff700 1 -- 172.21.15.102:0/3869869503 >> v1:172.21.15.181:6789/0 conn(0x7ff2e412c150 legacy=0x7ff2e4142d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:54.532 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.471+0000 7ff2cffff700 10 monclient: _finish_hunting 0 2021-05-25T11:59:54.532 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.471+0000 7ff2cffff700 1 monclient: found mon.a 2021-05-25T11:59:54.532 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.471+0000 7ff2cffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T11:59:54.532 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.471+0000 7ff2cffff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:54.532 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.471+0000 7ff2cffff700 1 -- 172.21.15.102:0/3869869503 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2e4097820 con 0x7ff2e4098930 2021-05-25T11:59:54.533 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.471+0000 7ff2cffff700 10 monclient: _finish_auth 0 2021-05-25T11:59:54.533 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.471+0000 7ff2cffff700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T11:59:54.533 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.471+0000 7ff2e91da700 5 monclient: authenticate success, global_id 4263 2021-05-25T11:59:54.533 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.471+0000 7ff2e91da700 10 monclient: _renew_subs 2021-05-25T11:59:54.533 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.471+0000 7ff2e91da700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:54.534 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.471+0000 7ff2e91da700 1 -- 172.21.15.102:0/3869869503 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2e4097a70 con 0x7ff2e4098930 2021-05-25T11:59:54.534 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.471+0000 7ff2e91da700 10 client.4263.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T11:59:54.534 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.471+0000 7ff2e91da700 10 monclient: _renew_subs 2021-05-25T11:59:54.534 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.471+0000 7ff2e91da700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:54.535 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.471+0000 7ff2e91da700 1 -- 172.21.15.102:0/3869869503 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2e413b2b0 con 0x7ff2e4098930 2021-05-25T11:59:54.535 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.471+0000 7ff2e91da700 1 librados: init done 2021-05-25T11:59:54.535 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.471+0000 7ff2cffff700 1 -- 172.21.15.102:0/3869869503 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff2d00047e0 con 0x7ff2e4098930 2021-05-25T11:59:54.535 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.471+0000 7ff2cffff700 10 monclient: handle_config config(0 keys) v1 2021-05-25T11:59:54.535 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.471+0000 7ff2cffff700 1 -- 172.21.15.102:0/3869869503 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7ff2d0004af0 con 0x7ff2e4098930 2021-05-25T11:59:54.536 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.471+0000 7ff2cffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:54.536 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.471+0000 7ff2cffff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T11:59:54.536 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.471+0000 7ff2e27fc700 4 set_mon_vals no callback set 2021-05-25T11:59:54.536 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.471+0000 7ff2cffff700 10 monclient: dump: 2021-05-25T11:59:54.537 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:54.537 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:54.537 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:54.537 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:54.537 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:54.538 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:54.538 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:54.538 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:54.538 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:54.538 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:54.539 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.471+0000 7ff2cffff700 1 -- 172.21.15.102:0/3869869503 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7ff2d0004d10 con 0x7ff2e4098930 2021-05-25T11:59:54.539 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.471+0000 7ff2e91da700 10 client.4263.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:54.539 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.471+0000 7ff2e91da700 15 client.4263.objecter _session_command_op_assign -1 1 2021-05-25T11:59:54.539 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.471+0000 7ff2e91da700 20 client.4263.objecter _get_session osd=-1 returning homeless 2021-05-25T11:59:54.539 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.471+0000 7ff2e91da700 10 client.4263.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T11:59:54.540 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.471+0000 7ff2e91da700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:54.540 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.471+0000 7ff2e91da700 1 -- 172.21.15.102:0/3869869503 --> v1:172.21.15.102:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff2bc0028c0 con 0x7ff2e4098930 2021-05-25T11:59:54.540 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.472+0000 7ff2cffff700 1 -- 172.21.15.102:0/3869869503 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(15..15 src has 1..15) v4 ==== 3549+0+0 (unknown 3948819498 0 0) 0x7ff2d003cfb0 con 0x7ff2e4098930 2021-05-25T11:59:54.540 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.472+0000 7ff2cffff700 10 client.4263.objecter ms_dispatch 0x7ff2e4001550 osd_map(15..15 src has 1..15) v4 2021-05-25T11:59:54.541 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.472+0000 7ff2cffff700 3 client.4263.objecter handle_osd_map got epochs [15,15] > 0 2021-05-25T11:59:54.541 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.472+0000 7ff2cffff700 3 client.4263.objecter handle_osd_map decoding full epoch 15 2021-05-25T11:59:54.541 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.472+0000 7ff2cffff700 10 client.4263.objecter checking command 1 2021-05-25T11:59:54.541 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.472+0000 7ff2cffff700 20 client.4263.objecter _get_session s=0x7ff2c4039cc0 osd=0 3 2021-05-25T11:59:54.541 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.472+0000 7ff2cffff700 20 client.4263.objecter put_session s=0x7ff2c4039cc0 osd=0 3 2021-05-25T11:59:54.542 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.472+0000 7ff2cffff700 15 client.4263.objecter _session_command_op_remove -1 1 2021-05-25T11:59:54.542 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.472+0000 7ff2cffff700 20 client.4263.objecter _get_session s=0x7ff2c4039cc0 osd=0 3 2021-05-25T11:59:54.542 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.472+0000 7ff2cffff700 20 client.4263.objecter get_session s=0x7ff2c4039cc0 osd=0 3 2021-05-25T11:59:54.542 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.472+0000 7ff2cffff700 15 client.4263.objecter _session_command_op_assign 0 1 2021-05-25T11:59:54.542 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.472+0000 7ff2cffff700 20 client.4263.objecter put_session s=0x7ff2c4039cc0 osd=0 4 2021-05-25T11:59:54.543 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.472+0000 7ff2cffff700 10 client.4263.objecter _send_command 1 2021-05-25T11:59:54.543 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.472+0000 7ff2cffff700 1 -- 172.21.15.102:0/3869869503 --> v1:172.21.15.102:6807/29973 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff2c403e040 con 0x7ff2c403a370 2021-05-25T11:59:54.543 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.472+0000 7ff2cffff700 20 client.4263.objecter dump_active .. 0 homeless 2021-05-25T11:59:54.543 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.472+0000 7ff2cffff700 1 -- 172.21.15.102:0/3869869503 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_get_version_reply(handle=1 version=15) v2 ==== 24+0+0 (unknown 2942983318 0 0) 0x7ff2d003d3e0 con 0x7ff2e4098930 2021-05-25T11:59:54.544 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.472+0000 7ff2cffff700 10 monclient: handle_get_version_reply finishing 1 version 15 2021-05-25T11:59:54.544 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.472+0000 7ff2e37fe700 10 monclient: get_auth_request con 0x7ff2c4033550 auth_method 0 2021-05-25T11:59:54.544 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.472+0000 7ff2e89d9700 10 monclient: get_auth_request con 0x7ff2c403a370 auth_method 0 2021-05-25T11:59:54.544 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.472+0000 7ff2cffff700 10 client.4263.objecter ms_handle_connect 0x7ff2c403a370 2021-05-25T11:59:54.544 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.472+0000 7ff2cffff700 10 client.4263.objecter ms_handle_connect 0x7ff2c4033550 2021-05-25T11:59:54.545 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.473+0000 7ff2cffff700 1 -- 172.21.15.102:0/3869869503 <== osd.0 v1:172.21.15.102:6807/29973 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23336 (unknown 0 0 3933607479) 0x7ff2c403e040 con 0x7ff2c403a370 2021-05-25T11:59:54.545 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.473+0000 7ff2cffff700 10 client.4263.objecter ms_dispatch 0x7ff2e4001550 command_reply(tid 1: 0 ) v1 2021-05-25T11:59:54.545 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.473+0000 7ff2cffff700 10 client.4263.objecter _finish_command 1 = system:0 2021-05-25T11:59:54.545 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.473+0000 7ff2cffff700 20 client.4263.objecter put_session s=0x7ff2c4039cc0 osd=0 4 2021-05-25T11:59:54.545 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.473+0000 7ff2cffff700 15 client.4263.objecter _session_command_op_remove 0 1 2021-05-25T11:59:54.547 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.487+0000 7ff2e91da700 10 client.4263.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2021-05-25T11:59:54.547 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.487+0000 7ff2e91da700 15 client.4263.objecter _session_command_op_assign -1 2 2021-05-25T11:59:54.548 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.487+0000 7ff2e91da700 20 client.4263.objecter _get_session s=0x7ff2c4039cc0 osd=0 3 2021-05-25T11:59:54.548 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.487+0000 7ff2e91da700 20 client.4263.objecter put_session s=0x7ff2c4039cc0 osd=0 3 2021-05-25T11:59:54.548 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.487+0000 7ff2e91da700 20 client.4263.objecter _get_session s=0x7ff2c4039cc0 osd=0 3 2021-05-25T11:59:54.548 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.487+0000 7ff2e91da700 15 client.4263.objecter _session_command_op_remove -1 2 2021-05-25T11:59:54.549 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.487+0000 7ff2e91da700 20 client.4263.objecter get_session s=0x7ff2c4039cc0 osd=0 3 2021-05-25T11:59:54.549 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.487+0000 7ff2e91da700 15 client.4263.objecter _session_command_op_assign 0 2 2021-05-25T11:59:54.549 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.487+0000 7ff2e91da700 20 client.4263.objecter put_session s=0x7ff2c4039cc0 osd=0 4 2021-05-25T11:59:54.549 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.487+0000 7ff2e91da700 10 client.4263.objecter _send_command 2 2021-05-25T11:59:54.549 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.487+0000 7ff2e91da700 1 -- 172.21.15.102:0/3869869503 --> v1:172.21.15.102:6807/29973 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff2bc001150 con 0x7ff2c403a370 2021-05-25T11:59:54.550 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.487+0000 7ff2cffff700 1 -- 172.21.15.102:0/3869869503 <== osd.0 v1:172.21.15.102:6807/29973 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (unknown 0 0 1800444376) 0x7ff2bc001150 con 0x7ff2c403a370 2021-05-25T11:59:54.550 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.487+0000 7ff2cffff700 10 client.4263.objecter ms_dispatch 0x7ff2e4001550 command_reply(tid 2: 0 ) v1 2021-05-25T11:59:54.550 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.487+0000 7ff2cffff700 10 client.4263.objecter _finish_command 2 = system:0 2021-05-25T11:59:54.550 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.487+0000 7ff2cffff700 20 client.4263.objecter put_session s=0x7ff2c4039cc0 osd=0 4 2021-05-25T11:59:54.551 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.487+0000 7ff2cffff700 15 client.4263.objecter _session_command_op_remove 0 2 2021-05-25T11:59:54.553 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.487+0000 7ff2e91da700 10 librados: watch_flush enter 2021-05-25T11:59:54.554 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.487+0000 7ff2e91da700 10 librados: watch_flush exit 2021-05-25T11:59:54.554 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.487+0000 7ff2e91da700 10 client.4263.objecter close_session for osd.0 2021-05-25T11:59:54.554 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.487+0000 7ff2e91da700 1 -- 172.21.15.102:0/3869869503 >> v1:172.21.15.102:6807/29973 conn(0x7ff2c403a370 legacy=0x7ff2c403c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:54.554 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.487+0000 7ff2e91da700 20 client.4263.objecter put_session s=0x7ff2c4039cc0 osd=0 1 2021-05-25T11:59:54.554 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.487+0000 7ff2e91da700 20 client.4263.objecter shutdown clearing up homeless session... 2021-05-25T11:59:54.555 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.487+0000 7ff2e91da700 10 client.4263.objecter successfully canceled tick 2021-05-25T11:59:54.555 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.487+0000 7ff2e91da700 1 -- 172.21.15.102:0/3869869503 >> v1:172.21.15.181:6812/29582 conn(0x7ff2c4033550 legacy=0x7ff2c4035a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:54.555 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.487+0000 7ff2e91da700 10 monclient: shutdown 2021-05-25T11:59:54.555 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.487+0000 7ff2e91da700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T11:59:54.556 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.487+0000 7ff2e91da700 1 -- 172.21.15.102:0/3869869503 >> v1:172.21.15.102:6789/0 conn(0x7ff2e4098930 legacy=0x7ff2e4097110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:54.556 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.487+0000 7ff2e89d9700 1 -- 172.21.15.102:0/3869869503 reap_dead start 2021-05-25T11:59:54.556 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.487+0000 7ff2e91da700 1 -- 172.21.15.102:0/3869869503 shutdown_connections 2021-05-25T11:59:54.556 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.487+0000 7ff2e91da700 1 -- 172.21.15.102:0/3869869503 >> 172.21.15.102:0/3869869503 conn(0x7ff2e409e9d0 msgr2=0x7ff2e41285c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T11:59:54.556 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.487+0000 7ff2e91da700 1 -- 172.21.15.102:0/3869869503 shutdown_connections 2021-05-25T11:59:54.557 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.487+0000 7ff2e91da700 1 -- 172.21.15.102:0/3869869503 wait complete. 2021-05-25T11:59:54.557 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.487+0000 7ff2e91da700 1 librados: shutdown 2021-05-25T11:59:54.557 INFO:teuthology.orchestra.run.smithi102.stdout:47244640259 2021-05-25T11:59:54.557 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2021-05-25T11:59:54.668 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.666+0000 7f511f0c7700 10 monclient: get_monmap_and_config 2021-05-25T11:59:54.668 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.666+0000 7f511f0c7700 10 monclient: build_initial_monmap 2021-05-25T11:59:54.669 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.667+0000 7f511f0c7700 10 monclient: monmap: 2021-05-25T11:59:54.669 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T11:59:54.669 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:54.669 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:54.667876+0000 2021-05-25T11:59:54.669 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:54.667876+0000 2021-05-25T11:59:54.670 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T11:59:54.670 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:54.670 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T11:59:54.670 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T11:59:54.671 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T11:59:54.671 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:54.682 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.667+0000 7f511f0c7700 1 Processor -- start 2021-05-25T11:59:54.682 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.667+0000 7f511f0c7700 1 -- start start 2021-05-25T11:59:54.682 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.667+0000 7f511f0c7700 10 monclient: init 2021-05-25T11:59:54.682 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.668+0000 7f511f0c7700 10 monclient: _reopen_session rank -1 2021-05-25T11:59:54.683 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.668+0000 7f511f0c7700 10 monclient: _add_conns ranks=[1,2,0] 2021-05-25T11:59:54.683 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.668+0000 7f511f0c7700 10 monclient(hunting): picked mon.noname-c con 0x7f5118128d10 addr v1:172.21.15.181:6789/0 2021-05-25T11:59:54.683 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.668+0000 7f511f0c7700 10 monclient(hunting): picked mon.noname-b con 0x7f5118096810 addr v1:172.21.15.102:6790/0 2021-05-25T11:59:54.683 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.668+0000 7f511f0c7700 10 monclient(hunting): picked mon.noname-a con 0x7f51181321f0 addr v1:172.21.15.102:6789/0 2021-05-25T11:59:54.683 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.668+0000 7f511f0c7700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5118069b10 con 0x7f51181321f0 2021-05-25T11:59:54.684 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.668+0000 7f511f0c7700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51180976c0 con 0x7f5118128d10 2021-05-25T11:59:54.684 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.668+0000 7f511f0c7700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5118097960 con 0x7f5118096810 2021-05-25T11:59:54.684 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.668+0000 7f511f0c7700 10 monclient(hunting): _renew_subs 2021-05-25T11:59:54.684 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.668+0000 7f511f0c7700 10 monclient(hunting): authenticate will time out at 795.265236s 2021-05-25T11:59:54.685 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.668+0000 7f511d664700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f51181321f0 0x7f51181345e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:58798/0 (socket says 172.21.15.102:58798) 2021-05-25T11:59:54.685 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.668+0000 7f510ffff700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f5118096810 0x7f5118130ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:53674/0 (socket says 172.21.15.102:53674) 2021-05-25T11:59:54.685 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.668+0000 7f511d664700 1 -- 172.21.15.102:0/3659731609 learned_addr learned my addr 172.21.15.102:0/3659731609 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T11:59:54.685 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.668+0000 7f510f7fe700 1 -- 172.21.15.102:0/3659731609 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2130696918 0 0) 0x7f5118069b10 con 0x7f51181321f0 2021-05-25T11:59:54.685 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.668+0000 7f510f7fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:54.686 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.668+0000 7f510f7fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:54.686 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.668+0000 7f510f7fe700 10 monclient(hunting): my global_id is 4266 2021-05-25T11:59:54.686 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.668+0000 7f510f7fe700 1 -- 172.21.15.102:0/3659731609 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5104003270 con 0x7f51181321f0 2021-05-25T11:59:54.686 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.668+0000 7f510f7fe700 1 -- 172.21.15.102:0/3659731609 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1451074873 0 0) 0x7f5118097960 con 0x7f5118096810 2021-05-25T11:59:54.687 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.668+0000 7f510f7fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:54.687 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.668+0000 7f510f7fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:54.687 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.668+0000 7f510f7fe700 10 monclient(hunting): my global_id is 4259 2021-05-25T11:59:54.687 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.668+0000 7f510f7fe700 1 -- 172.21.15.102:0/3659731609 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5118069b10 con 0x7f5118096810 2021-05-25T11:59:54.687 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.668+0000 7f510f7fe700 1 -- 172.21.15.102:0/3659731609 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 10480129 0 0) 0x7f5104003270 con 0x7f51181321f0 2021-05-25T11:59:54.688 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.668+0000 7f510f7fe700 1 -- 172.21.15.102:0/3659731609 >> v1:172.21.15.102:6790/0 conn(0x7f5118096810 legacy=0x7f5118130ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:54.688 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.668+0000 7f510f7fe700 1 -- 172.21.15.102:0/3659731609 >> v1:172.21.15.181:6789/0 conn(0x7f5118128d10 legacy=0x7f5118068d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:54.688 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.668+0000 7f510f7fe700 10 monclient: _finish_hunting 0 2021-05-25T11:59:54.688 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.668+0000 7f510f7fe700 1 monclient: found mon.noname-a 2021-05-25T11:59:54.689 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.668+0000 7f510f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T11:59:54.689 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.668+0000 7f510f7fe700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T11:59:54.689 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.668+0000 7f510f7fe700 1 -- 172.21.15.102:0/3659731609 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5118097c00 con 0x7f51181321f0 2021-05-25T11:59:54.689 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.668+0000 7f510f7fe700 10 monclient: _finish_auth 0 2021-05-25T11:59:54.689 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.668+0000 7f510f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T11:59:54.690 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.668+0000 7f511f0c7700 5 monclient: authenticate success, global_id 4266 2021-05-25T11:59:54.690 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.668+0000 7f511f0c7700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T11:59:54.690 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.668+0000 7f510f7fe700 1 -- 172.21.15.102:0/3659731609 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f5114001990 con 0x7f51181321f0 2021-05-25T11:59:54.690 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.668+0000 7f510f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:54.691 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.668+0000 7f510f7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T11:59:54.691 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.668+0000 7f510f7fe700 10 monclient: dump: 2021-05-25T11:59:54.691 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:54.691 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:54.691 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:54.692 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:54.692 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:54.692 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:54.692 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:54.693 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:54.693 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:54.693 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:54.693 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.668+0000 7f511f0c7700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T11:59:54.693 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.669+0000 7f510f7fe700 1 -- 172.21.15.102:0/3659731609 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5114001b40 con 0x7f51181321f0 2021-05-25T11:59:54.694 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.669+0000 7f510f7fe700 10 monclient: handle_config config(0 keys) v1 2021-05-25T11:59:54.694 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.669+0000 7f511f0c7700 10 monclient: get_monmap_and_config success 2021-05-25T11:59:54.694 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.669+0000 7f511f0c7700 4 set_mon_vals no callback set 2021-05-25T11:59:54.694 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.669+0000 7f510f7fe700 1 -- 172.21.15.102:0/3659731609 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f5114001ed0 con 0x7f51181321f0 2021-05-25T11:59:54.695 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.669+0000 7f510f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:54.695 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.669+0000 7f510f7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T11:59:54.695 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.669+0000 7f510f7fe700 10 monclient: dump: 2021-05-25T11:59:54.695 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:54.695 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:54.696 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:54.696 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:54.696 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:54.696 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:54.696 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:54.697 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:54.697 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:54.697 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:54.697 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.669+0000 7f511f0c7700 10 monclient: shutdown 2021-05-25T11:59:54.698 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.669+0000 7f511f0c7700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T11:59:54.698 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.669+0000 7f511f0c7700 1 -- 172.21.15.102:0/3659731609 >> v1:172.21.15.102:6789/0 conn(0x7f51181321f0 legacy=0x7f51181345e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:54.698 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.669+0000 7f511f0c7700 1 -- 172.21.15.102:0/3659731609 shutdown_connections 2021-05-25T11:59:54.698 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.669+0000 7f511f0c7700 1 -- 172.21.15.102:0/3659731609 >> 172.21.15.102:0/3659731609 conn(0x7f511809b920 msgr2=0x7f511809bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T11:59:54.698 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.669+0000 7f511f0c7700 1 -- 172.21.15.102:0/3659731609 shutdown_connections 2021-05-25T11:59:54.699 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.669+0000 7f511f0c7700 1 -- 172.21.15.102:0/3659731609 wait complete. 2021-05-25T11:59:54.699 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.669+0000 7f511f0c7700 10 monclient: build_initial_monmap 2021-05-25T11:59:54.699 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.669+0000 7f511f0c7700 10 monclient: monmap: 2021-05-25T11:59:54.699 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T11:59:54.699 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T11:59:54.700 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T11:59:54.700 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T11:59:54.700 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T11:59:54.700 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T11:59:54.701 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T11:59:54.701 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T11:59:54.701 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T11:59:54.701 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:54.701 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.670+0000 7f511f0c7700 1 librados: starting msgr at 2021-05-25T11:59:54.702 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.670+0000 7f511f0c7700 1 librados: starting objecter 2021-05-25T11:59:54.702 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.670+0000 7f511f0c7700 1 Processor -- start 2021-05-25T11:59:54.702 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.670+0000 7f511f0c7700 1 -- start start 2021-05-25T11:59:54.702 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.670+0000 7f511f0c7700 1 librados: setting wanted keys 2021-05-25T11:59:54.703 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.670+0000 7f511f0c7700 1 librados: calling monclient init 2021-05-25T11:59:54.703 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.670+0000 7f511f0c7700 10 monclient: init 2021-05-25T11:59:54.703 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.670+0000 7f511f0c7700 10 monclient: _reopen_session rank -1 2021-05-25T11:59:54.703 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.670+0000 7f511f0c7700 10 monclient: _add_conns ranks=[2,1,0] 2021-05-25T11:59:54.703 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.670+0000 7f511f0c7700 10 monclient(hunting): picked mon.noname-c con 0x7f5118096810 addr v1:172.21.15.102:6790/0 2021-05-25T11:59:54.704 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.670+0000 7f511f0c7700 10 monclient(hunting): picked mon.noname-b con 0x7f5118128d10 addr v1:172.21.15.181:6789/0 2021-05-25T11:59:54.704 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.670+0000 7f511f0c7700 10 monclient(hunting): picked mon.noname-a con 0x7f51181321f0 addr v1:172.21.15.102:6789/0 2021-05-25T11:59:54.704 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.670+0000 7f511f0c7700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5118127b70 con 0x7f51181321f0 2021-05-25T11:59:54.704 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.670+0000 7f511f0c7700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f511813b6d0 con 0x7f5118128d10 2021-05-25T11:59:54.704 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.670+0000 7f511f0c7700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5118131af0 con 0x7f5118096810 2021-05-25T11:59:54.705 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.670+0000 7f511f0c7700 10 monclient(hunting): _renew_subs 2021-05-25T11:59:54.705 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.670+0000 7f511f0c7700 10 monclient(hunting): authenticate will time out at 795.267850s 2021-05-25T11:59:54.705 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.670+0000 7f511d664700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f51181321f0 0x7f5118142b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:58804/0 (socket says 172.21.15.102:58804) 2021-05-25T11:59:54.705 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.670+0000 7f511d664700 1 -- 172.21.15.102:0/11627548 learned_addr learned my addr 172.21.15.102:0/11627548 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T11:59:54.706 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.670+0000 7f511ce63700 1 --1- 172.21.15.102:0/11627548 >> v1:172.21.15.102:6790/0 conn(0x7f5118096810 0x7f51181313e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:53678/0 (socket says 172.21.15.102:53678) 2021-05-25T11:59:54.706 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.671+0000 7f510cff9700 10 client.?.objecter ms_handle_connect 0x7f51181321f0 2021-05-25T11:59:54.706 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.671+0000 7f510cff9700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:54.706 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.671+0000 7f510cff9700 10 client.?.objecter ms_handle_connect 0x7f5118096810 2021-05-25T11:59:54.706 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.671+0000 7f510cff9700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:54.707 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.671+0000 7f510cff9700 10 client.?.objecter ms_handle_connect 0x7f5118128d10 2021-05-25T11:59:54.707 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.671+0000 7f510cff9700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:54.707 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.671+0000 7f510cff9700 1 -- 172.21.15.102:0/11627548 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1859950675 0 0) 0x7f5118127b70 con 0x7f51181321f0 2021-05-25T11:59:54.707 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.671+0000 7f510cff9700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:54.708 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.671+0000 7f510cff9700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:54.708 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.671+0000 7f510cff9700 10 monclient(hunting): my global_id is 4269 2021-05-25T11:59:54.708 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.671+0000 7f510cff9700 1 -- 172.21.15.102:0/11627548 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50f8004c10 con 0x7f51181321f0 2021-05-25T11:59:54.708 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.671+0000 7f510cff9700 1 -- 172.21.15.102:0/11627548 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3592461663 0 0) 0x7f5118131af0 con 0x7f5118096810 2021-05-25T11:59:54.708 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.671+0000 7f510cff9700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:54.709 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.671+0000 7f510cff9700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:54.709 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.671+0000 7f510cff9700 10 monclient(hunting): my global_id is 4262 2021-05-25T11:59:54.709 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.671+0000 7f510cff9700 1 -- 172.21.15.102:0/11627548 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5118127b70 con 0x7f5118096810 2021-05-25T11:59:54.709 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.671+0000 7f510cff9700 1 -- 172.21.15.102:0/11627548 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3811460819 0 0) 0x7f511813b6d0 con 0x7f5118128d10 2021-05-25T11:59:54.709 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.671+0000 7f510cff9700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:54.710 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.671+0000 7f510cff9700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:54.710 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.671+0000 7f510cff9700 10 monclient(hunting): my global_id is 4264 2021-05-25T11:59:54.710 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.671+0000 7f510cff9700 1 -- 172.21.15.102:0/11627548 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5118131af0 con 0x7f5118128d10 2021-05-25T11:59:54.710 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.671+0000 7f510cff9700 1 -- 172.21.15.102:0/11627548 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3712845324 0 0) 0x7f50f8004c10 con 0x7f51181321f0 2021-05-25T11:59:54.711 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.671+0000 7f510cff9700 1 -- 172.21.15.102:0/11627548 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f511813b6d0 con 0x7f51181321f0 2021-05-25T11:59:54.711 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.671+0000 7f510cff9700 1 -- 172.21.15.102:0/11627548 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f5114001e50 con 0x7f51181321f0 2021-05-25T11:59:54.711 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.671+0000 7f510cff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:54.711 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.671+0000 7f510cff9700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T11:59:54.711 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.671+0000 7f510cff9700 10 monclient(hunting): dump: 2021-05-25T11:59:54.712 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:54.712 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:54.712 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:54.712 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:54.713 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:54.713 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:54.713 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:54.713 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:54.713 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:54.714 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:54.714 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.671+0000 7f510cff9700 10 monclient(hunting): _finish_auth 0 2021-05-25T11:59:54.714 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.671+0000 7f510cff9700 1 -- 172.21.15.102:0/11627548 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3394290729 0 0) 0x7f5118127b70 con 0x7f5118096810 2021-05-25T11:59:54.714 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.671+0000 7f510cff9700 1 -- 172.21.15.102:0/11627548 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f50f8004c10 con 0x7f5118096810 2021-05-25T11:59:54.714 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.671+0000 7f510cff9700 1 -- 172.21.15.102:0/11627548 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f5108001900 con 0x7f5118096810 2021-05-25T11:59:54.715 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.671+0000 7f510cff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:54.715 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.671+0000 7f510cff9700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2021-05-25T11:59:54.715 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.671+0000 7f510cff9700 10 monclient(hunting): dump: 2021-05-25T11:59:54.715 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:54.716 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:54.716 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:54.716 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:54.716 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:54.716 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:54.717 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:54.717 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:54.717 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:54.717 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:54.717 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.671+0000 7f510cff9700 1 -- 172.21.15.102:0/11627548 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3275520301 0 0) 0x7f5118131af0 con 0x7f5118128d10 2021-05-25T11:59:54.718 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.671+0000 7f510cff9700 1 -- 172.21.15.102:0/11627548 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5118127b70 con 0x7f5118128d10 2021-05-25T11:59:54.718 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.671+0000 7f510cff9700 1 -- 172.21.15.102:0/11627548 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f5100001920 con 0x7f5118128d10 2021-05-25T11:59:54.718 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.671+0000 7f510cff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:54.718 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.671+0000 7f510cff9700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2021-05-25T11:59:54.719 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.671+0000 7f510cff9700 10 monclient(hunting): dump: 2021-05-25T11:59:54.719 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:54.719 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:54.719 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:54.719 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:54.720 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:54.720 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:54.720 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:54.720 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:54.720 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:54.721 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:54.721 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.671+0000 7f510cff9700 1 -- 172.21.15.102:0/11627548 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1535957335 0 0) 0x7f50f8004c10 con 0x7f5118096810 2021-05-25T11:59:54.721 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.671+0000 7f510cff9700 1 -- 172.21.15.102:0/11627548 >> v1:172.21.15.181:6789/0 conn(0x7f5118128d10 legacy=0x7f511813a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:54.721 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.671+0000 7f510cff9700 1 -- 172.21.15.102:0/11627548 >> v1:172.21.15.102:6789/0 conn(0x7f51181321f0 legacy=0x7f5118142b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:54.721 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.671+0000 7f510cff9700 10 monclient: _finish_hunting 0 2021-05-25T11:59:54.722 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.671+0000 7f510cff9700 1 monclient: found mon.c 2021-05-25T11:59:54.722 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.671+0000 7f510cff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T11:59:54.722 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.671+0000 7f510cff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T11:59:54.722 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.671+0000 7f510cff9700 1 -- 172.21.15.102:0/11627548 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f511812fc90 con 0x7f5118096810 2021-05-25T11:59:54.723 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.671+0000 7f510cff9700 10 monclient: _finish_auth 0 2021-05-25T11:59:54.723 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.671+0000 7f510cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T11:59:54.723 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.671+0000 7f511f0c7700 5 monclient: authenticate success, global_id 4262 2021-05-25T11:59:54.723 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.671+0000 7f511f0c7700 10 monclient: _renew_subs 2021-05-25T11:59:54.723 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.671+0000 7f511f0c7700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T11:59:54.724 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.671+0000 7f511f0c7700 1 -- 172.21.15.102:0/11627548 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f511813af10 con 0x7f5118096810 2021-05-25T11:59:54.724 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.671+0000 7f511f0c7700 10 client.4262.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T11:59:54.724 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.671+0000 7f511f0c7700 10 monclient: _renew_subs 2021-05-25T11:59:54.724 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.671+0000 7f511f0c7700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T11:59:54.725 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.671+0000 7f511f0c7700 1 -- 172.21.15.102:0/11627548 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f511812ea60 con 0x7f5118096810 2021-05-25T11:59:54.725 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.671+0000 7f511f0c7700 1 librados: init done 2021-05-25T11:59:54.725 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.671+0000 7f510cff9700 1 -- 172.21.15.102:0/11627548 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5108001e50 con 0x7f5118096810 2021-05-25T11:59:54.725 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.671+0000 7f510cff9700 10 monclient: handle_config config(0 keys) v1 2021-05-25T11:59:54.725 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.671+0000 7f510cff9700 1 -- 172.21.15.102:0/11627548 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f51080021e0 con 0x7f5118096810 2021-05-25T11:59:54.726 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.671+0000 7f510cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:54.726 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.671+0000 7f510effd700 4 set_mon_vals no callback set 2021-05-25T11:59:54.726 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.671+0000 7f510cff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T11:59:54.726 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.671+0000 7f510cff9700 10 monclient: dump: 2021-05-25T11:59:54.727 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:54.727 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:54.727 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:54.727 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:54.727 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:54.728 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:54.728 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:54.728 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:54.728 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:54.728 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:54.729 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.672+0000 7f511f0c7700 10 client.4262.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:54.729 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.672+0000 7f511f0c7700 15 client.4262.objecter _session_command_op_assign -1 1 2021-05-25T11:59:54.729 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.672+0000 7f511f0c7700 20 client.4262.objecter _get_session osd=-1 returning homeless 2021-05-25T11:59:54.729 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.672+0000 7f511f0c7700 10 client.4262.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T11:59:54.730 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.672+0000 7f511f0c7700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T11:59:54.730 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.672+0000 7f511f0c7700 1 -- 172.21.15.102:0/11627548 --> v1:172.21.15.102:6790/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f50f00028c0 con 0x7f5118096810 2021-05-25T11:59:54.730 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.672+0000 7f510cff9700 1 -- 172.21.15.102:0/11627548 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f51080023c0 con 0x7f5118096810 2021-05-25T11:59:54.730 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.672+0000 7f510cff9700 1 -- 172.21.15.102:0/11627548 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(15..15 src has 1..15) v4 ==== 3549+0+0 (unknown 3948819498 0 0) 0x7f510803cc60 con 0x7f5118096810 2021-05-25T11:59:54.730 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.672+0000 7f510cff9700 10 client.4262.objecter ms_dispatch 0x7f5118001550 osd_map(15..15 src has 1..15) v4 2021-05-25T11:59:54.731 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.672+0000 7f510cff9700 3 client.4262.objecter handle_osd_map got epochs [15,15] > 0 2021-05-25T11:59:54.731 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.672+0000 7f510cff9700 3 client.4262.objecter handle_osd_map decoding full epoch 15 2021-05-25T11:59:54.731 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.672+0000 7f510cff9700 10 client.4262.objecter checking command 1 2021-05-25T11:59:54.731 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.672+0000 7f510cff9700 20 client.4262.objecter _get_session s=0x7f50f8039c90 osd=1 3 2021-05-25T11:59:54.731 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.672+0000 7f510cff9700 20 client.4262.objecter put_session s=0x7f50f8039c90 osd=1 3 2021-05-25T11:59:54.732 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.672+0000 7f510cff9700 15 client.4262.objecter _session_command_op_remove -1 1 2021-05-25T11:59:54.732 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.672+0000 7f510cff9700 20 client.4262.objecter _get_session s=0x7f50f8039c90 osd=1 3 2021-05-25T11:59:54.732 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.672+0000 7f510cff9700 20 client.4262.objecter get_session s=0x7f50f8039c90 osd=1 3 2021-05-25T11:59:54.732 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.672+0000 7f510cff9700 15 client.4262.objecter _session_command_op_assign 1 1 2021-05-25T11:59:54.733 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.672+0000 7f510cff9700 20 client.4262.objecter put_session s=0x7f50f8039c90 osd=1 4 2021-05-25T11:59:54.733 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.672+0000 7f510cff9700 10 client.4262.objecter _send_command 1 2021-05-25T11:59:54.733 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.672+0000 7f510cff9700 1 -- 172.21.15.102:0/11627548 --> v1:172.21.15.102:6802/29974 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f50f803e010 con 0x7f50f803a340 2021-05-25T11:59:54.733 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.672+0000 7f510cff9700 20 client.4262.objecter dump_active .. 0 homeless 2021-05-25T11:59:54.733 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.672+0000 7f510cff9700 1 -- 172.21.15.102:0/11627548 <== mon.2 v1:172.21.15.102:6790/0 9 ==== mon_get_version_reply(handle=1 version=15) v2 ==== 24+0+0 (unknown 2942983318 0 0) 0x7f510803d090 con 0x7f5118096810 2021-05-25T11:59:54.734 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.672+0000 7f510cff9700 10 monclient: handle_get_version_reply finishing 1 version 15 2021-05-25T11:59:54.734 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.672+0000 7f510ffff700 10 monclient: get_auth_request con 0x7f50f8033520 auth_method 0 2021-05-25T11:59:54.734 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.672+0000 7f511d664700 10 monclient: get_auth_request con 0x7f50f803a340 auth_method 0 2021-05-25T11:59:54.734 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.672+0000 7f510cff9700 10 client.4262.objecter ms_handle_connect 0x7f50f803a340 2021-05-25T11:59:54.734 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.672+0000 7f510cff9700 10 client.4262.objecter ms_handle_connect 0x7f50f8033520 2021-05-25T11:59:54.735 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.673+0000 7f510cff9700 1 -- 172.21.15.102:0/11627548 <== osd.1 v1:172.21.15.102:6802/29974 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23336 (unknown 0 0 3933607479) 0x7f50f803e010 con 0x7f50f803a340 2021-05-25T11:59:54.735 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.673+0000 7f510cff9700 10 client.4262.objecter ms_dispatch 0x7f5118001550 command_reply(tid 1: 0 ) v1 2021-05-25T11:59:54.735 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.673+0000 7f510cff9700 10 client.4262.objecter _finish_command 1 = system:0 2021-05-25T11:59:54.735 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.673+0000 7f510cff9700 20 client.4262.objecter put_session s=0x7f50f8039c90 osd=1 4 2021-05-25T11:59:54.736 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.673+0000 7f510cff9700 15 client.4262.objecter _session_command_op_remove 1 1 2021-05-25T11:59:54.737 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.687+0000 7f511f0c7700 10 client.4262.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2021-05-25T11:59:54.737 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.687+0000 7f511f0c7700 15 client.4262.objecter _session_command_op_assign -1 2 2021-05-25T11:59:54.737 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.687+0000 7f511f0c7700 20 client.4262.objecter _get_session s=0x7f50f8039c90 osd=1 3 2021-05-25T11:59:54.737 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.687+0000 7f511f0c7700 20 client.4262.objecter put_session s=0x7f50f8039c90 osd=1 3 2021-05-25T11:59:54.738 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.687+0000 7f511f0c7700 20 client.4262.objecter _get_session s=0x7f50f8039c90 osd=1 3 2021-05-25T11:59:54.738 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.687+0000 7f511f0c7700 15 client.4262.objecter _session_command_op_remove -1 2 2021-05-25T11:59:54.738 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.687+0000 7f511f0c7700 20 client.4262.objecter get_session s=0x7f50f8039c90 osd=1 3 2021-05-25T11:59:54.738 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.687+0000 7f511f0c7700 15 client.4262.objecter _session_command_op_assign 1 2 2021-05-25T11:59:54.739 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.687+0000 7f511f0c7700 20 client.4262.objecter put_session s=0x7f50f8039c90 osd=1 4 2021-05-25T11:59:54.739 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.687+0000 7f511f0c7700 10 client.4262.objecter _send_command 2 2021-05-25T11:59:54.739 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.687+0000 7f511f0c7700 1 -- 172.21.15.102:0/11627548 --> v1:172.21.15.102:6802/29974 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f50f0001150 con 0x7f50f803a340 2021-05-25T11:59:54.739 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.687+0000 7f510cff9700 1 -- 172.21.15.102:0/11627548 <== osd.1 v1:172.21.15.102:6802/29974 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (unknown 0 0 1800444376) 0x7f50f0001150 con 0x7f50f803a340 2021-05-25T11:59:54.739 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.687+0000 7f510cff9700 10 client.4262.objecter ms_dispatch 0x7f5118001550 command_reply(tid 2: 0 ) v1 2021-05-25T11:59:54.740 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.687+0000 7f510cff9700 10 client.4262.objecter _finish_command 2 = system:0 2021-05-25T11:59:54.740 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.687+0000 7f510cff9700 20 client.4262.objecter put_session s=0x7f50f8039c90 osd=1 4 2021-05-25T11:59:54.740 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.687+0000 7f510cff9700 15 client.4262.objecter _session_command_op_remove 1 2 2021-05-25T11:59:54.742 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.687+0000 7f511f0c7700 10 librados: watch_flush enter 2021-05-25T11:59:54.743 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.687+0000 7f511f0c7700 10 librados: watch_flush exit 2021-05-25T11:59:54.743 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.687+0000 7f511f0c7700 10 client.4262.objecter close_session for osd.1 2021-05-25T11:59:54.743 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.687+0000 7f511f0c7700 1 -- 172.21.15.102:0/11627548 >> v1:172.21.15.102:6802/29974 conn(0x7f50f803a340 legacy=0x7f50f803c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:54.743 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.687+0000 7f511f0c7700 20 client.4262.objecter put_session s=0x7f50f8039c90 osd=1 1 2021-05-25T11:59:54.744 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.687+0000 7f511f0c7700 20 client.4262.objecter shutdown clearing up homeless session... 2021-05-25T11:59:54.744 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.687+0000 7f511f0c7700 10 client.4262.objecter successfully canceled tick 2021-05-25T11:59:54.744 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.687+0000 7f511f0c7700 1 -- 172.21.15.102:0/11627548 >> v1:172.21.15.181:6812/29582 conn(0x7f50f8033520 legacy=0x7f50f80359e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:54.744 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.687+0000 7f511f0c7700 10 monclient: shutdown 2021-05-25T11:59:54.744 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.687+0000 7f511f0c7700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T11:59:54.745 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.687+0000 7f511f0c7700 1 -- 172.21.15.102:0/11627548 >> v1:172.21.15.102:6790/0 conn(0x7f5118096810 legacy=0x7f51181313e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:54.745 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.687+0000 7f511d664700 1 -- 172.21.15.102:0/11627548 reap_dead start 2021-05-25T11:59:54.745 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.687+0000 7f511f0c7700 1 -- 172.21.15.102:0/11627548 shutdown_connections 2021-05-25T11:59:54.745 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.687+0000 7f511f0c7700 1 -- 172.21.15.102:0/11627548 >> 172.21.15.102:0/11627548 conn(0x7f511809b920 msgr2=0x7f511809b1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T11:59:54.746 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.688+0000 7f511f0c7700 1 -- 172.21.15.102:0/11627548 shutdown_connections 2021-05-25T11:59:54.746 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.688+0000 7f511f0c7700 1 -- 172.21.15.102:0/11627548 wait complete. 2021-05-25T11:59:54.746 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.688+0000 7f511f0c7700 1 librados: shutdown 2021-05-25T11:59:54.746 INFO:teuthology.orchestra.run.smithi102.stdout:47244640259 2021-05-25T11:59:54.747 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2021-05-25T11:59:54.856 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.854+0000 7f94e5225700 10 monclient: get_monmap_and_config 2021-05-25T11:59:54.857 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.854+0000 7f94e5225700 10 monclient: build_initial_monmap 2021-05-25T11:59:54.857 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.854+0000 7f94e5225700 10 monclient: monmap: 2021-05-25T11:59:54.857 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T11:59:54.857 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:54.857 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:54.855638+0000 2021-05-25T11:59:54.858 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:54.855638+0000 2021-05-25T11:59:54.858 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T11:59:54.858 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:54.858 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T11:59:54.859 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T11:59:54.859 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T11:59:54.859 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:54.859 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.855+0000 7f94e5225700 1 Processor -- start 2021-05-25T11:59:54.860 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.855+0000 7f94e5225700 1 -- start start 2021-05-25T11:59:54.860 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.855+0000 7f94e5225700 10 monclient: init 2021-05-25T11:59:54.871 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.855+0000 7f94e5225700 10 monclient: _reopen_session rank -1 2021-05-25T11:59:54.871 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.855+0000 7f94e5225700 10 monclient: _add_conns ranks=[1,2,0] 2021-05-25T11:59:54.871 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.855+0000 7f94e5225700 10 monclient(hunting): picked mon.noname-c con 0x7f94e009a5e0 addr v1:172.21.15.181:6789/0 2021-05-25T11:59:54.872 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.855+0000 7f94e5225700 10 monclient(hunting): picked mon.noname-b con 0x7f94e0063de0 addr v1:172.21.15.102:6790/0 2021-05-25T11:59:54.872 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.855+0000 7f94e5225700 10 monclient(hunting): picked mon.noname-a con 0x7f94e01325c0 addr v1:172.21.15.102:6789/0 2021-05-25T11:59:54.872 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.855+0000 7f94e5225700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94e009a3a0 con 0x7f94e01325c0 2021-05-25T11:59:54.872 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.855+0000 7f94e5225700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94e009b400 con 0x7f94e009a5e0 2021-05-25T11:59:54.873 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.855+0000 7f94e5225700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94e00963f0 con 0x7f94e0063de0 2021-05-25T11:59:54.873 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.855+0000 7f94e5225700 10 monclient(hunting): _renew_subs 2021-05-25T11:59:54.873 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.855+0000 7f94e5225700 10 monclient(hunting): authenticate will time out at 795.452808s 2021-05-25T11:59:54.873 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.855+0000 7f94de59c700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f94e0063de0 0x7f94e0064260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:53688/0 (socket says 172.21.15.102:53688) 2021-05-25T11:59:54.874 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.855+0000 7f94df59e700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f94e01325c0 0x7f94e01349b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:58814/0 (socket says 172.21.15.102:58814) 2021-05-25T11:59:54.874 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.855+0000 7f94de59c700 1 -- 172.21.15.102:0/3539329019 learned_addr learned my addr 172.21.15.102:0/3539329019 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T11:59:54.874 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.856+0000 7f94ddd9b700 1 -- 172.21.15.102:0/3539329019 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1637318614 0 0) 0x7f94e00963f0 con 0x7f94e0063de0 2021-05-25T11:59:54.874 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.856+0000 7f94ddd9b700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:54.874 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.856+0000 7f94ddd9b700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:54.875 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.856+0000 7f94ddd9b700 10 monclient(hunting): my global_id is 4265 2021-05-25T11:59:54.875 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.856+0000 7f94ddd9b700 1 -- 172.21.15.102:0/3539329019 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94cc003270 con 0x7f94e0063de0 2021-05-25T11:59:54.875 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.856+0000 7f94ddd9b700 1 -- 172.21.15.102:0/3539329019 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3053042120 0 0) 0x7f94e009a3a0 con 0x7f94e01325c0 2021-05-25T11:59:54.875 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.856+0000 7f94ddd9b700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:54.876 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.856+0000 7f94ddd9b700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:54.876 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.856+0000 7f94ddd9b700 10 monclient(hunting): my global_id is 4272 2021-05-25T11:59:54.876 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.856+0000 7f94ddd9b700 1 -- 172.21.15.102:0/3539329019 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94e00963f0 con 0x7f94e01325c0 2021-05-25T11:59:54.876 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.856+0000 7f94ddd9b700 1 -- 172.21.15.102:0/3539329019 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 43021649 0 0) 0x7f94cc003270 con 0x7f94e0063de0 2021-05-25T11:59:54.876 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.856+0000 7f94ddd9b700 1 -- 172.21.15.102:0/3539329019 >> v1:172.21.15.181:6789/0 conn(0x7f94e009a5e0 legacy=0x7f94e00636d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:54.877 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.856+0000 7f94ddd9b700 1 -- 172.21.15.102:0/3539329019 >> v1:172.21.15.102:6789/0 conn(0x7f94e01325c0 legacy=0x7f94e01349b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:54.877 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.856+0000 7f94ddd9b700 10 monclient: _finish_hunting 0 2021-05-25T11:59:54.877 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.856+0000 7f94ddd9b700 1 monclient: found mon.noname-b 2021-05-25T11:59:54.877 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.856+0000 7f94ddd9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T11:59:54.877 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.856+0000 7f94ddd9b700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T11:59:54.878 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.856+0000 7f94ddd9b700 1 -- 172.21.15.102:0/3539329019 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94e0096690 con 0x7f94e0063de0 2021-05-25T11:59:54.878 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.856+0000 7f94ddd9b700 10 monclient: _finish_auth 0 2021-05-25T11:59:54.878 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.856+0000 7f94ddd9b700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T11:59:54.878 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.856+0000 7f94e5225700 5 monclient: authenticate success, global_id 4265 2021-05-25T11:59:54.879 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.856+0000 7f94ddd9b700 1 -- 172.21.15.102:0/3539329019 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f94c8001580 con 0x7f94e0063de0 2021-05-25T11:59:54.879 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.856+0000 7f94e5225700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T11:59:54.879 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.856+0000 7f94ddd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:54.879 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.856+0000 7f94ddd9b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T11:59:54.879 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.856+0000 7f94ddd9b700 10 monclient: dump: 2021-05-25T11:59:54.880 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:54.880 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:54.880 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:54.880 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:54.881 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:54.881 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:54.881 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:54.881 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:54.881 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:54.882 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:54.882 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.856+0000 7f94e5225700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T11:59:54.882 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.856+0000 7f94ddd9b700 1 -- 172.21.15.102:0/3539329019 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f94c80017a0 con 0x7f94e0063de0 2021-05-25T11:59:54.882 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.856+0000 7f94ddd9b700 10 monclient: handle_config config(0 keys) v1 2021-05-25T11:59:54.883 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.856+0000 7f94e5225700 10 monclient: get_monmap_and_config success 2021-05-25T11:59:54.883 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.856+0000 7f94ddd9b700 1 -- 172.21.15.102:0/3539329019 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f94c8001b30 con 0x7f94e0063de0 2021-05-25T11:59:54.883 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.856+0000 7f94e5225700 4 set_mon_vals no callback set 2021-05-25T11:59:54.883 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.856+0000 7f94ddd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:54.883 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.856+0000 7f94ddd9b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T11:59:54.884 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.856+0000 7f94ddd9b700 10 monclient: dump: 2021-05-25T11:59:54.884 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:54.884 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:54.884 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:54.885 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:54.885 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:54.885 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:54.885 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:54.885 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:54.886 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:54.886 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:54.886 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.856+0000 7f94e5225700 10 monclient: shutdown 2021-05-25T11:59:54.886 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.856+0000 7f94e5225700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T11:59:54.886 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.856+0000 7f94e5225700 1 -- 172.21.15.102:0/3539329019 >> v1:172.21.15.102:6790/0 conn(0x7f94e0063de0 legacy=0x7f94e0064260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:54.887 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.857+0000 7f94e5225700 1 -- 172.21.15.102:0/3539329019 shutdown_connections 2021-05-25T11:59:54.887 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.857+0000 7f94e5225700 1 -- 172.21.15.102:0/3539329019 >> 172.21.15.102:0/3539329019 conn(0x7f94e009bc30 msgr2=0x7f94e009c040 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T11:59:54.887 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.857+0000 7f94e5225700 1 -- 172.21.15.102:0/3539329019 shutdown_connections 2021-05-25T11:59:54.887 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.857+0000 7f94e5225700 1 -- 172.21.15.102:0/3539329019 wait complete. 2021-05-25T11:59:54.887 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.857+0000 7f94e5225700 10 monclient: build_initial_monmap 2021-05-25T11:59:54.888 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.857+0000 7f94e5225700 10 monclient: monmap: 2021-05-25T11:59:54.888 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T11:59:54.888 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T11:59:54.888 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T11:59:54.889 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T11:59:54.889 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T11:59:54.889 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T11:59:54.889 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T11:59:54.889 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T11:59:54.890 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T11:59:54.890 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:54.890 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.857+0000 7f94e5225700 1 librados: starting msgr at 2021-05-25T11:59:54.890 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.857+0000 7f94e5225700 1 librados: starting objecter 2021-05-25T11:59:54.890 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.858+0000 7f94e5225700 1 Processor -- start 2021-05-25T11:59:54.891 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.858+0000 7f94e5225700 1 -- start start 2021-05-25T11:59:54.891 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.858+0000 7f94e5225700 1 librados: setting wanted keys 2021-05-25T11:59:54.891 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.858+0000 7f94e5225700 1 librados: calling monclient init 2021-05-25T11:59:54.891 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.858+0000 7f94e5225700 10 monclient: init 2021-05-25T11:59:54.892 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.858+0000 7f94e5225700 10 monclient: _reopen_session rank -1 2021-05-25T11:59:54.892 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.858+0000 7f94e5225700 10 monclient: _add_conns ranks=[2,0,1] 2021-05-25T11:59:54.892 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.858+0000 7f94e5225700 10 monclient(hunting): picked mon.noname-c con 0x7f94e0063de0 addr v1:172.21.15.102:6790/0 2021-05-25T11:59:54.892 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.858+0000 7f94e5225700 10 monclient(hunting): picked mon.noname-a con 0x7f94e009a5e0 addr v1:172.21.15.102:6789/0 2021-05-25T11:59:54.892 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.858+0000 7f94e5225700 10 monclient(hunting): picked mon.noname-b con 0x7f94e01325c0 addr v1:172.21.15.181:6789/0 2021-05-25T11:59:54.893 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.858+0000 7f94e5225700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94e0116230 con 0x7f94e009a5e0 2021-05-25T11:59:54.893 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.858+0000 7f94e5225700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94e0118050 con 0x7f94e01325c0 2021-05-25T11:59:54.893 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.858+0000 7f94e5225700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94e01182f0 con 0x7f94e0063de0 2021-05-25T11:59:54.893 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.858+0000 7f94e5225700 10 monclient(hunting): _renew_subs 2021-05-25T11:59:54.893 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.858+0000 7f94e5225700 10 monclient(hunting): authenticate will time out at 795.455541s 2021-05-25T11:59:54.894 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.858+0000 7f94ded9d700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f94e0063de0 0x7f94e0115410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:53694/0 (socket says 172.21.15.102:53694) 2021-05-25T11:59:54.894 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.858+0000 7f94ded9d700 1 -- 172.21.15.102:0/3101110989 learned_addr learned my addr 172.21.15.102:0/3101110989 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T11:59:54.894 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.858+0000 7f94c6ffd700 10 client.?.objecter ms_handle_connect 0x7f94e0063de0 2021-05-25T11:59:54.894 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.858+0000 7f94c6ffd700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:54.895 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.858+0000 7f94c6ffd700 10 client.?.objecter ms_handle_connect 0x7f94e009a5e0 2021-05-25T11:59:54.895 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.858+0000 7f94c6ffd700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:54.895 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.858+0000 7f94c6ffd700 1 -- 172.21.15.102:0/3101110989 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3560131739 0 0) 0x7f94e01182f0 con 0x7f94e0063de0 2021-05-25T11:59:54.895 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.858+0000 7f94c6ffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:54.895 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.858+0000 7f94c6ffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:54.896 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.858+0000 7f94c6ffd700 10 monclient(hunting): my global_id is 4268 2021-05-25T11:59:54.896 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.858+0000 7f94c6ffd700 1 -- 172.21.15.102:0/3101110989 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94c0004c10 con 0x7f94e0063de0 2021-05-25T11:59:54.896 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.858+0000 7f94c6ffd700 10 client.?.objecter ms_handle_connect 0x7f94e01325c0 2021-05-25T11:59:54.896 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.858+0000 7f94c6ffd700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:54.896 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.858+0000 7f94c6ffd700 1 -- 172.21.15.102:0/3101110989 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1560189109 0 0) 0x7f94e0116230 con 0x7f94e009a5e0 2021-05-25T11:59:54.897 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.858+0000 7f94c6ffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:54.897 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.858+0000 7f94c6ffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:54.897 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.858+0000 7f94c6ffd700 10 monclient(hunting): my global_id is 4275 2021-05-25T11:59:54.897 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.858+0000 7f94c6ffd700 1 -- 172.21.15.102:0/3101110989 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94e01182f0 con 0x7f94e009a5e0 2021-05-25T11:59:54.898 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.858+0000 7f94c6ffd700 1 -- 172.21.15.102:0/3101110989 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3014020772 0 0) 0x7f94e0118050 con 0x7f94e01325c0 2021-05-25T11:59:54.898 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.858+0000 7f94c6ffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:54.898 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.858+0000 7f94c6ffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:54.898 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.858+0000 7f94c6ffd700 10 monclient(hunting): my global_id is 4270 2021-05-25T11:59:54.898 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.858+0000 7f94c6ffd700 1 -- 172.21.15.102:0/3101110989 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94e0116230 con 0x7f94e01325c0 2021-05-25T11:59:54.899 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.859+0000 7f94c6ffd700 1 -- 172.21.15.102:0/3101110989 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 189974374 0 0) 0x7f94c0004c10 con 0x7f94e0063de0 2021-05-25T11:59:54.899 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.859+0000 7f94c6ffd700 1 -- 172.21.15.102:0/3101110989 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f94e0118050 con 0x7f94e0063de0 2021-05-25T11:59:54.899 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.859+0000 7f94c6ffd700 1 -- 172.21.15.102:0/3101110989 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f94d4001990 con 0x7f94e0063de0 2021-05-25T11:59:54.899 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.859+0000 7f94c6ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:54.900 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.859+0000 7f94c6ffd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T11:59:54.900 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.859+0000 7f94c6ffd700 10 monclient(hunting): dump: 2021-05-25T11:59:54.900 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:54.900 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:54.900 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:54.901 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:54.901 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:54.901 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:54.901 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:54.901 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:54.902 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:54.902 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:54.902 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.859+0000 7f94c6ffd700 10 monclient(hunting): _finish_auth 0 2021-05-25T11:59:54.902 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.859+0000 7f94c6ffd700 1 -- 172.21.15.102:0/3101110989 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3099359515 0 0) 0x7f94e01182f0 con 0x7f94e009a5e0 2021-05-25T11:59:54.903 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.859+0000 7f94c6ffd700 1 -- 172.21.15.102:0/3101110989 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f94c0004c10 con 0x7f94e009a5e0 2021-05-25T11:59:54.903 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.859+0000 7f94c6ffd700 1 -- 172.21.15.102:0/3101110989 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f94c80013a0 con 0x7f94e009a5e0 2021-05-25T11:59:54.903 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.859+0000 7f94c6ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:54.903 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.859+0000 7f94c6ffd700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2021-05-25T11:59:54.903 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.859+0000 7f94c6ffd700 10 monclient(hunting): dump: 2021-05-25T11:59:54.904 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:54.904 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:54.904 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:54.904 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:54.904 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:54.905 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:54.905 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:54.905 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:54.905 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:54.905 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:54.906 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.859+0000 7f94c6ffd700 1 -- 172.21.15.102:0/3101110989 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1900771931 0 0) 0x7f94e0118050 con 0x7f94e0063de0 2021-05-25T11:59:54.906 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.859+0000 7f94c6ffd700 1 -- 172.21.15.102:0/3101110989 >> v1:172.21.15.181:6789/0 conn(0x7f94e01325c0 legacy=0x7f94e0143130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:54.906 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.859+0000 7f94c6ffd700 1 -- 172.21.15.102:0/3101110989 >> v1:172.21.15.102:6789/0 conn(0x7f94e009a5e0 legacy=0x7f94e0115b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:54.906 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.859+0000 7f94c6ffd700 10 monclient: _finish_hunting 0 2021-05-25T11:59:54.907 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.859+0000 7f94c6ffd700 1 monclient: found mon.c 2021-05-25T11:59:54.907 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.859+0000 7f94c6ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T11:59:54.907 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.859+0000 7f94c6ffd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T11:59:54.907 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.859+0000 7f94c6ffd700 1 -- 172.21.15.102:0/3101110989 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94e0095780 con 0x7f94e0063de0 2021-05-25T11:59:54.907 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.859+0000 7f94c6ffd700 10 monclient: _finish_auth 0 2021-05-25T11:59:54.908 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.859+0000 7f94c6ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T11:59:54.908 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.859+0000 7f94e5225700 5 monclient: authenticate success, global_id 4268 2021-05-25T11:59:54.908 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.859+0000 7f94e5225700 10 monclient: _renew_subs 2021-05-25T11:59:54.908 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.859+0000 7f94e5225700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T11:59:54.909 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.859+0000 7f94e5225700 1 -- 172.21.15.102:0/3101110989 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94e0094df0 con 0x7f94e0063de0 2021-05-25T11:59:54.909 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.859+0000 7f94e5225700 10 client.4268.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T11:59:54.909 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.859+0000 7f94e5225700 10 monclient: _renew_subs 2021-05-25T11:59:54.909 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.859+0000 7f94e5225700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T11:59:54.909 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.859+0000 7f94e5225700 1 -- 172.21.15.102:0/3101110989 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f94e012dd70 con 0x7f94e0063de0 2021-05-25T11:59:54.910 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.859+0000 7f94e5225700 1 librados: init done 2021-05-25T11:59:54.910 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.859+0000 7f94c6ffd700 1 -- 172.21.15.102:0/3101110989 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f94d4001d20 con 0x7f94e0063de0 2021-05-25T11:59:54.910 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.859+0000 7f94c6ffd700 10 monclient: handle_config config(0 keys) v1 2021-05-25T11:59:54.910 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.859+0000 7f94c6ffd700 1 -- 172.21.15.102:0/3101110989 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f94d4002410 con 0x7f94e0063de0 2021-05-25T11:59:54.911 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.859+0000 7f94c6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:54.911 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.859+0000 7f94c6ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T11:59:54.911 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.859+0000 7f94dd59a700 4 set_mon_vals no callback set 2021-05-25T11:59:54.911 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.859+0000 7f94c6ffd700 10 monclient: dump: 2021-05-25T11:59:54.911 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:54.912 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:54.912 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:54.912 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:54.912 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:54.912 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:54.913 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:54.913 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:54.913 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:54.913 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:54.914 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.859+0000 7f94e5225700 10 client.4268.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:54.914 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.859+0000 7f94e5225700 15 client.4268.objecter _session_command_op_assign -1 1 2021-05-25T11:59:54.914 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.859+0000 7f94e5225700 20 client.4268.objecter _get_session osd=-1 returning homeless 2021-05-25T11:59:54.914 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.859+0000 7f94e5225700 10 client.4268.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T11:59:54.914 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.859+0000 7f94e5225700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T11:59:54.915 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.859+0000 7f94e5225700 1 -- 172.21.15.102:0/3101110989 --> v1:172.21.15.102:6790/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f94b80028c0 con 0x7f94e0063de0 2021-05-25T11:59:54.915 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.859+0000 7f94c6ffd700 1 -- 172.21.15.102:0/3101110989 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f94d40025f0 con 0x7f94e0063de0 2021-05-25T11:59:54.915 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.860+0000 7f94c6ffd700 1 -- 172.21.15.102:0/3101110989 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(15..15 src has 1..15) v4 ==== 3549+0+0 (unknown 3948819498 0 0) 0x7f94d403cf80 con 0x7f94e0063de0 2021-05-25T11:59:54.915 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.860+0000 7f94c6ffd700 10 client.4268.objecter ms_dispatch 0x7f94e0001550 osd_map(15..15 src has 1..15) v4 2021-05-25T11:59:54.916 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.860+0000 7f94c6ffd700 3 client.4268.objecter handle_osd_map got epochs [15,15] > 0 2021-05-25T11:59:54.916 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.860+0000 7f94c6ffd700 3 client.4268.objecter handle_osd_map decoding full epoch 15 2021-05-25T11:59:54.916 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.860+0000 7f94c6ffd700 10 client.4268.objecter checking command 1 2021-05-25T11:59:54.916 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.860+0000 7f94c6ffd700 20 client.4268.objecter _get_session s=0x7f94c0039a30 osd=2 3 2021-05-25T11:59:54.916 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.860+0000 7f94c6ffd700 20 client.4268.objecter put_session s=0x7f94c0039a30 osd=2 3 2021-05-25T11:59:54.917 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.860+0000 7f94c6ffd700 15 client.4268.objecter _session_command_op_remove -1 1 2021-05-25T11:59:54.917 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.860+0000 7f94c6ffd700 20 client.4268.objecter _get_session s=0x7f94c0039a30 osd=2 3 2021-05-25T11:59:54.917 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.860+0000 7f94c6ffd700 20 client.4268.objecter get_session s=0x7f94c0039a30 osd=2 3 2021-05-25T11:59:54.917 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.860+0000 7f94c6ffd700 15 client.4268.objecter _session_command_op_assign 2 1 2021-05-25T11:59:54.917 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.860+0000 7f94c6ffd700 20 client.4268.objecter put_session s=0x7f94c0039a30 osd=2 4 2021-05-25T11:59:54.918 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.860+0000 7f94c6ffd700 10 client.4268.objecter _send_command 1 2021-05-25T11:59:54.918 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.860+0000 7f94c6ffd700 1 -- 172.21.15.102:0/3101110989 --> v1:172.21.15.102:6800/29975 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f94c003ddb0 con 0x7f94c003a0e0 2021-05-25T11:59:54.918 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.860+0000 7f94c6ffd700 20 client.4268.objecter dump_active .. 0 homeless 2021-05-25T11:59:54.918 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.860+0000 7f94c6ffd700 1 -- 172.21.15.102:0/3101110989 <== mon.2 v1:172.21.15.102:6790/0 9 ==== mon_get_version_reply(handle=1 version=15) v2 ==== 24+0+0 (unknown 2942983318 0 0) 0x7f94d403d3b0 con 0x7f94e0063de0 2021-05-25T11:59:54.919 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.860+0000 7f94c6ffd700 10 monclient: handle_get_version_reply finishing 1 version 15 2021-05-25T11:59:54.919 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.860+0000 7f94de59c700 10 monclient: get_auth_request con 0x7f94c0033220 auth_method 0 2021-05-25T11:59:54.919 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.860+0000 7f94df59e700 10 monclient: get_auth_request con 0x7f94c003a0e0 auth_method 0 2021-05-25T11:59:54.919 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.860+0000 7f94c6ffd700 10 client.4268.objecter ms_handle_connect 0x7f94c003a0e0 2021-05-25T11:59:54.919 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.860+0000 7f94c6ffd700 10 client.4268.objecter ms_handle_connect 0x7f94c0033220 2021-05-25T11:59:54.920 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.861+0000 7f94c6ffd700 1 -- 172.21.15.102:0/3101110989 <== osd.2 v1:172.21.15.102:6800/29975 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23336 (unknown 0 0 3933607479) 0x7f94c003ddb0 con 0x7f94c003a0e0 2021-05-25T11:59:54.920 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.861+0000 7f94c6ffd700 10 client.4268.objecter ms_dispatch 0x7f94e0001550 command_reply(tid 1: 0 ) v1 2021-05-25T11:59:54.920 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.861+0000 7f94c6ffd700 10 client.4268.objecter _finish_command 1 = system:0 2021-05-25T11:59:54.920 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.861+0000 7f94c6ffd700 20 client.4268.objecter put_session s=0x7f94c0039a30 osd=2 4 2021-05-25T11:59:54.920 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.861+0000 7f94c6ffd700 15 client.4268.objecter _session_command_op_remove 2 1 2021-05-25T11:59:54.922 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.874+0000 7f94e5225700 10 client.4268.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2021-05-25T11:59:54.922 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.874+0000 7f94e5225700 15 client.4268.objecter _session_command_op_assign -1 2 2021-05-25T11:59:54.922 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.874+0000 7f94e5225700 20 client.4268.objecter _get_session s=0x7f94c0039a30 osd=2 3 2021-05-25T11:59:54.922 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.874+0000 7f94e5225700 20 client.4268.objecter put_session s=0x7f94c0039a30 osd=2 3 2021-05-25T11:59:54.923 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.874+0000 7f94e5225700 20 client.4268.objecter _get_session s=0x7f94c0039a30 osd=2 3 2021-05-25T11:59:54.923 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.874+0000 7f94e5225700 15 client.4268.objecter _session_command_op_remove -1 2 2021-05-25T11:59:54.923 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.874+0000 7f94e5225700 20 client.4268.objecter get_session s=0x7f94c0039a30 osd=2 3 2021-05-25T11:59:54.923 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.874+0000 7f94e5225700 15 client.4268.objecter _session_command_op_assign 2 2 2021-05-25T11:59:54.924 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.874+0000 7f94e5225700 20 client.4268.objecter put_session s=0x7f94c0039a30 osd=2 4 2021-05-25T11:59:54.924 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.874+0000 7f94e5225700 10 client.4268.objecter _send_command 2 2021-05-25T11:59:54.924 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.874+0000 7f94e5225700 1 -- 172.21.15.102:0/3101110989 --> v1:172.21.15.102:6800/29975 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f94b8001150 con 0x7f94c003a0e0 2021-05-25T11:59:54.924 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.875+0000 7f94c6ffd700 1 -- 172.21.15.102:0/3101110989 <== osd.2 v1:172.21.15.102:6800/29975 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (unknown 0 0 1800444376) 0x7f94b8001150 con 0x7f94c003a0e0 2021-05-25T11:59:54.924 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.875+0000 7f94c6ffd700 10 client.4268.objecter ms_dispatch 0x7f94e0001550 command_reply(tid 2: 0 ) v1 2021-05-25T11:59:54.925 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.875+0000 7f94c6ffd700 10 client.4268.objecter _finish_command 2 = system:0 2021-05-25T11:59:54.925 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.875+0000 7f94c6ffd700 20 client.4268.objecter put_session s=0x7f94c0039a30 osd=2 4 2021-05-25T11:59:54.925 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.875+0000 7f94c6ffd700 15 client.4268.objecter _session_command_op_remove 2 2 2021-05-25T11:59:54.927 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.875+0000 7f94e5225700 10 librados: watch_flush enter 2021-05-25T11:59:54.927 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.875+0000 7f94e5225700 10 librados: watch_flush exit 2021-05-25T11:59:54.928 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.875+0000 7f94e5225700 10 client.4268.objecter close_session for osd.2 2021-05-25T11:59:54.928 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.875+0000 7f94e5225700 1 -- 172.21.15.102:0/3101110989 >> v1:172.21.15.102:6800/29975 conn(0x7f94c003a0e0 legacy=0x7f94c003c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:54.928 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.875+0000 7f94e5225700 20 client.4268.objecter put_session s=0x7f94c0039a30 osd=2 1 2021-05-25T11:59:54.928 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.875+0000 7f94e5225700 20 client.4268.objecter shutdown clearing up homeless session... 2021-05-25T11:59:54.928 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.875+0000 7f94e5225700 10 client.4268.objecter successfully canceled tick 2021-05-25T11:59:54.929 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.875+0000 7f94e5225700 1 -- 172.21.15.102:0/3101110989 >> v1:172.21.15.181:6812/29582 conn(0x7f94c0033220 legacy=0x7f94c00356e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:54.929 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.875+0000 7f94e5225700 10 monclient: shutdown 2021-05-25T11:59:54.929 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.875+0000 7f94e5225700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T11:59:54.929 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.875+0000 7f94e5225700 1 -- 172.21.15.102:0/3101110989 >> v1:172.21.15.102:6790/0 conn(0x7f94e0063de0 legacy=0x7f94e0115410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:54.930 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.875+0000 7f94df59e700 1 -- 172.21.15.102:0/3101110989 reap_dead start 2021-05-25T11:59:54.930 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.875+0000 7f94e5225700 1 -- 172.21.15.102:0/3101110989 shutdown_connections 2021-05-25T11:59:54.930 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.875+0000 7f94e5225700 1 -- 172.21.15.102:0/3101110989 >> 172.21.15.102:0/3101110989 conn(0x7f94e009bc30 msgr2=0x7f94e009e210 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T11:59:54.930 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.875+0000 7f94e5225700 1 -- 172.21.15.102:0/3101110989 shutdown_connections 2021-05-25T11:59:54.930 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.875+0000 7f94e5225700 1 -- 172.21.15.102:0/3101110989 wait complete. 2021-05-25T11:59:54.931 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:54.875+0000 7f94e5225700 1 librados: shutdown 2021-05-25T11:59:54.931 INFO:teuthology.orchestra.run.smithi102.stdout:47244640259 2021-05-25T11:59:54.931 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2021-05-25T11:59:55.043 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.041+0000 7f210bfff700 10 monclient: get_monmap_and_config 2021-05-25T11:59:55.043 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.041+0000 7f210bfff700 10 monclient: build_initial_monmap 2021-05-25T11:59:55.043 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.041+0000 7f210bfff700 10 monclient: monmap: 2021-05-25T11:59:55.044 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T11:59:55.044 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:55.044 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:55.042450+0000 2021-05-25T11:59:55.044 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:55.042450+0000 2021-05-25T11:59:55.045 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T11:59:55.045 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:55.045 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T11:59:55.045 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T11:59:55.045 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T11:59:55.046 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:55.046 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.042+0000 7f210bfff700 1 Processor -- start 2021-05-25T11:59:55.046 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.042+0000 7f210bfff700 1 -- start start 2021-05-25T11:59:55.046 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.042+0000 7f210bfff700 10 monclient: init 2021-05-25T11:59:55.058 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.042+0000 7f210bfff700 10 monclient: _reopen_session rank -1 2021-05-25T11:59:55.058 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.042+0000 7f210bfff700 10 monclient: _add_conns ranks=[2,0,1] 2021-05-25T11:59:55.059 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.042+0000 7f210bfff700 10 monclient(hunting): picked mon.noname-b con 0x7f210c09b4e0 addr v1:172.21.15.102:6790/0 2021-05-25T11:59:55.059 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.042+0000 7f210bfff700 10 monclient(hunting): picked mon.noname-a con 0x7f210c068b40 addr v1:172.21.15.102:6789/0 2021-05-25T11:59:55.059 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.042+0000 7f210bfff700 10 monclient(hunting): picked mon.noname-c con 0x7f210c068ff0 addr v1:172.21.15.181:6789/0 2021-05-25T11:59:55.059 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.042+0000 7f210bfff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f210c096d20 con 0x7f210c068b40 2021-05-25T11:59:55.059 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.042+0000 7f210bfff700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f210c096fc0 con 0x7f210c068ff0 2021-05-25T11:59:55.060 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.042+0000 7f210bfff700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f210c097260 con 0x7f210c09b4e0 2021-05-25T11:59:55.060 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.042+0000 7f210bfff700 10 monclient(hunting): _renew_subs 2021-05-25T11:59:55.060 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.042+0000 7f210bfff700 10 monclient(hunting): authenticate will time out at 795.639584s 2021-05-25T11:59:55.060 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.042+0000 7f210affd700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f210c09b4e0 0x7f210c068430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:53704/0 (socket says 172.21.15.102:53704) 2021-05-25T11:59:55.061 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.042+0000 7f210a7fc700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f210c068b40 0x7f210c1308d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:58826/0 (socket says 172.21.15.102:58826) 2021-05-25T11:59:55.061 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.042+0000 7f210affd700 1 -- 172.21.15.102:0/1923650733 learned_addr learned my addr 172.21.15.102:0/1923650733 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T11:59:55.061 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.042+0000 7f2109ffb700 1 -- 172.21.15.102:0/1923650733 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 614773470 0 0) 0x7f210c096d20 con 0x7f210c068b40 2021-05-25T11:59:55.061 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.042+0000 7f2109ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:55.062 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.042+0000 7f2109ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:55.062 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.042+0000 7f2109ffb700 10 monclient(hunting): my global_id is 4278 2021-05-25T11:59:55.062 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.042+0000 7f2109ffb700 1 -- 172.21.15.102:0/1923650733 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20f8003270 con 0x7f210c068b40 2021-05-25T11:59:55.062 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.043+0000 7f2109ffb700 1 -- 172.21.15.102:0/1923650733 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1508655830 0 0) 0x7f210c097260 con 0x7f210c09b4e0 2021-05-25T11:59:55.062 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.043+0000 7f2109ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:55.063 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.043+0000 7f2109ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:55.063 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.043+0000 7f2109ffb700 10 monclient(hunting): my global_id is 4271 2021-05-25T11:59:55.063 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.043+0000 7f2109ffb700 1 -- 172.21.15.102:0/1923650733 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f210c096d20 con 0x7f210c09b4e0 2021-05-25T11:59:55.063 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.043+0000 7f2109ffb700 1 -- 172.21.15.102:0/1923650733 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4124337620 0 0) 0x7f210c096fc0 con 0x7f210c068ff0 2021-05-25T11:59:55.064 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.043+0000 7f2109ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:55.064 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.043+0000 7f2109ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:55.064 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.043+0000 7f2109ffb700 10 monclient(hunting): my global_id is 4273 2021-05-25T11:59:55.064 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.043+0000 7f2109ffb700 1 -- 172.21.15.102:0/1923650733 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f210c097260 con 0x7f210c068ff0 2021-05-25T11:59:55.064 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.043+0000 7f2109ffb700 1 -- 172.21.15.102:0/1923650733 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 491302289 0 0) 0x7f20f8003270 con 0x7f210c068b40 2021-05-25T11:59:55.065 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.043+0000 7f2109ffb700 1 -- 172.21.15.102:0/1923650733 >> v1:172.21.15.102:6790/0 conn(0x7f210c09b4e0 legacy=0x7f210c068430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:55.065 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.043+0000 7f2109ffb700 1 -- 172.21.15.102:0/1923650733 >> v1:172.21.15.181:6789/0 conn(0x7f210c068ff0 legacy=0x7f210c134000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:55.065 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.043+0000 7f2109ffb700 10 monclient: _finish_hunting 0 2021-05-25T11:59:55.065 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.043+0000 7f2109ffb700 1 monclient: found mon.noname-a 2021-05-25T11:59:55.066 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.043+0000 7f2109ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T11:59:55.066 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.043+0000 7f2109ffb700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T11:59:55.066 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.043+0000 7f2109ffb700 1 -- 172.21.15.102:0/1923650733 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f210c090990 con 0x7f210c068b40 2021-05-25T11:59:55.066 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.043+0000 7f2109ffb700 10 monclient: _finish_auth 0 2021-05-25T11:59:55.066 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.043+0000 7f2109ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T11:59:55.067 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.043+0000 7f2109ffb700 1 -- 172.21.15.102:0/1923650733 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f2100001610 con 0x7f210c068b40 2021-05-25T11:59:55.067 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.043+0000 7f210bfff700 5 monclient: authenticate success, global_id 4278 2021-05-25T11:59:55.067 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.043+0000 7f210bfff700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T11:59:55.067 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.043+0000 7f2109ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:55.068 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.043+0000 7f2109ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T11:59:55.068 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.043+0000 7f2109ffb700 10 monclient: dump: 2021-05-25T11:59:55.068 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:55.068 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:55.068 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:55.069 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:55.069 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:55.069 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:55.069 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:55.069 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:55.070 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:55.071 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:55.071 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.043+0000 7f210bfff700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T11:59:55.071 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.043+0000 7f2109ffb700 1 -- 172.21.15.102:0/1923650733 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2100001800 con 0x7f210c068b40 2021-05-25T11:59:55.071 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.043+0000 7f2109ffb700 10 monclient: handle_config config(0 keys) v1 2021-05-25T11:59:55.072 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.043+0000 7f210bfff700 10 monclient: get_monmap_and_config success 2021-05-25T11:59:55.072 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.043+0000 7f2109ffb700 1 -- 172.21.15.102:0/1923650733 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f2100001b90 con 0x7f210c068b40 2021-05-25T11:59:55.072 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.043+0000 7f210bfff700 4 set_mon_vals no callback set 2021-05-25T11:59:55.072 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.043+0000 7f2109ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:55.073 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.043+0000 7f2109ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T11:59:55.073 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.043+0000 7f2109ffb700 10 monclient: dump: 2021-05-25T11:59:55.073 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:55.073 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:55.073 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:55.074 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:55.074 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:55.074 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:55.074 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:55.075 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:55.075 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:55.075 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:55.075 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.043+0000 7f210bfff700 10 monclient: shutdown 2021-05-25T11:59:55.075 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.043+0000 7f210bfff700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T11:59:55.076 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.043+0000 7f210bfff700 1 -- 172.21.15.102:0/1923650733 >> v1:172.21.15.102:6789/0 conn(0x7f210c068b40 legacy=0x7f210c1308d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:55.076 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.043+0000 7f210bfff700 1 -- 172.21.15.102:0/1923650733 shutdown_connections 2021-05-25T11:59:55.076 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.043+0000 7f210bfff700 1 -- 172.21.15.102:0/1923650733 >> 172.21.15.102:0/1923650733 conn(0x7f210c097f90 msgr2=0x7f210c09a9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T11:59:55.076 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.043+0000 7f210bfff700 1 -- 172.21.15.102:0/1923650733 shutdown_connections 2021-05-25T11:59:55.077 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.043+0000 7f210bfff700 1 -- 172.21.15.102:0/1923650733 wait complete. 2021-05-25T11:59:55.077 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.044+0000 7f210bfff700 10 monclient: build_initial_monmap 2021-05-25T11:59:55.077 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.044+0000 7f210bfff700 10 monclient: monmap: 2021-05-25T11:59:55.077 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T11:59:55.077 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T11:59:55.078 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T11:59:55.078 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T11:59:55.078 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T11:59:55.078 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T11:59:55.079 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T11:59:55.079 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T11:59:55.079 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T11:59:55.079 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:55.079 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.044+0000 7f210bfff700 1 librados: starting msgr at 2021-05-25T11:59:55.080 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.044+0000 7f210bfff700 1 librados: starting objecter 2021-05-25T11:59:55.080 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.044+0000 7f210bfff700 1 Processor -- start 2021-05-25T11:59:55.080 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.044+0000 7f210bfff700 1 -- start start 2021-05-25T11:59:55.080 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.044+0000 7f210bfff700 1 librados: setting wanted keys 2021-05-25T11:59:55.080 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.044+0000 7f210bfff700 1 librados: calling monclient init 2021-05-25T11:59:55.081 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.044+0000 7f210bfff700 10 monclient: init 2021-05-25T11:59:55.081 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.045+0000 7f210bfff700 10 monclient: _reopen_session rank -1 2021-05-25T11:59:55.081 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.045+0000 7f210bfff700 10 monclient: _add_conns ranks=[1,2,0] 2021-05-25T11:59:55.081 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.045+0000 7f210bfff700 10 monclient(hunting): picked mon.noname-b con 0x7f210c068b40 addr v1:172.21.15.181:6789/0 2021-05-25T11:59:55.082 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.045+0000 7f210bfff700 10 monclient(hunting): picked mon.noname-c con 0x7f210c068ff0 addr v1:172.21.15.102:6790/0 2021-05-25T11:59:55.082 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.045+0000 7f210bfff700 10 monclient(hunting): picked mon.noname-a con 0x7f210c09b4e0 addr v1:172.21.15.102:6789/0 2021-05-25T11:59:55.082 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.045+0000 7f210bfff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f210c11bbf0 con 0x7f210c09b4e0 2021-05-25T11:59:55.082 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.045+0000 7f210bfff700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f210c11b380 con 0x7f210c068b40 2021-05-25T11:59:55.082 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.045+0000 7f210bfff700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f210c11be20 con 0x7f210c068ff0 2021-05-25T11:59:55.083 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.045+0000 7f210bfff700 10 monclient(hunting): _renew_subs 2021-05-25T11:59:55.083 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.045+0000 7f210bfff700 10 monclient(hunting): authenticate will time out at 795.642273s 2021-05-25T11:59:55.083 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.045+0000 7f210a7fc700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f210c068ff0 0x7f210c13aa50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:53712/0 (socket says 172.21.15.102:53712) 2021-05-25T11:59:55.083 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.045+0000 7f210a7fc700 1 -- 172.21.15.102:0/1309835282 learned_addr learned my addr 172.21.15.102:0/1309835282 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T11:59:55.084 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.045+0000 7f210b7fe700 1 --1- 172.21.15.102:0/1309835282 >> v1:172.21.15.102:6789/0 conn(0x7f210c09b4e0 0x7f210c142da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:58836/0 (socket says 172.21.15.102:58836) 2021-05-25T11:59:55.084 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.045+0000 7f20f2ffd700 10 client.?.objecter ms_handle_connect 0x7f210c068ff0 2021-05-25T11:59:55.084 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.045+0000 7f20f2ffd700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:55.084 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.045+0000 7f20f2ffd700 10 client.?.objecter ms_handle_connect 0x7f210c09b4e0 2021-05-25T11:59:55.084 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.045+0000 7f20f2ffd700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:55.085 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.045+0000 7f20f2ffd700 10 client.?.objecter ms_handle_connect 0x7f210c068b40 2021-05-25T11:59:55.085 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.045+0000 7f20f2ffd700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:55.085 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.045+0000 7f20f2ffd700 1 -- 172.21.15.102:0/1309835282 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4112975694 0 0) 0x7f210c11be20 con 0x7f210c068ff0 2021-05-25T11:59:55.085 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.045+0000 7f20f2ffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:55.085 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.045+0000 7f20f2ffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:55.086 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.045+0000 7f20f2ffd700 10 monclient(hunting): my global_id is 4274 2021-05-25T11:59:55.086 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.045+0000 7f20f2ffd700 1 -- 172.21.15.102:0/1309835282 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20ec004c10 con 0x7f210c068ff0 2021-05-25T11:59:55.086 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.045+0000 7f20f2ffd700 1 -- 172.21.15.102:0/1309835282 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3516120743 0 0) 0x7f210c11bbf0 con 0x7f210c09b4e0 2021-05-25T11:59:55.086 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.045+0000 7f20f2ffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:55.087 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.045+0000 7f20f2ffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:55.087 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.045+0000 7f20f2ffd700 10 monclient(hunting): my global_id is 4281 2021-05-25T11:59:55.087 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.045+0000 7f20f2ffd700 1 -- 172.21.15.102:0/1309835282 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f210c11be20 con 0x7f210c09b4e0 2021-05-25T11:59:55.087 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.045+0000 7f20f2ffd700 1 -- 172.21.15.102:0/1309835282 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 373212749 0 0) 0x7f210c11b380 con 0x7f210c068b40 2021-05-25T11:59:55.088 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.045+0000 7f20f2ffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:55.088 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.045+0000 7f20f2ffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:55.088 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.045+0000 7f20f2ffd700 10 monclient(hunting): my global_id is 4276 2021-05-25T11:59:55.088 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.045+0000 7f20f2ffd700 1 -- 172.21.15.102:0/1309835282 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f210c11bbf0 con 0x7f210c068b40 2021-05-25T11:59:55.089 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.045+0000 7f20f2ffd700 1 -- 172.21.15.102:0/1309835282 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 660334763 0 0) 0x7f20ec004c10 con 0x7f210c068ff0 2021-05-25T11:59:55.089 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.045+0000 7f20f2ffd700 1 -- 172.21.15.102:0/1309835282 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f210c11b380 con 0x7f210c068ff0 2021-05-25T11:59:55.089 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.045+0000 7f20f2ffd700 1 -- 172.21.15.102:0/1309835282 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f2100001b10 con 0x7f210c068ff0 2021-05-25T11:59:55.089 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.045+0000 7f20f2ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:55.089 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.045+0000 7f20f2ffd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T11:59:55.090 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.045+0000 7f20f2ffd700 10 monclient(hunting): dump: 2021-05-25T11:59:55.090 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:55.090 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:55.090 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:55.090 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:55.091 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:55.091 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:55.091 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:55.091 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:55.092 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:55.092 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:55.092 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.045+0000 7f20f2ffd700 10 monclient(hunting): _finish_auth 0 2021-05-25T11:59:55.092 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.045+0000 7f20f2ffd700 1 -- 172.21.15.102:0/1309835282 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4219698251 0 0) 0x7f210c11be20 con 0x7f210c09b4e0 2021-05-25T11:59:55.092 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.045+0000 7f20f2ffd700 1 -- 172.21.15.102:0/1309835282 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f20ec004c10 con 0x7f210c09b4e0 2021-05-25T11:59:55.093 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.045+0000 7f20f2ffd700 1 -- 172.21.15.102:0/1309835282 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f20fc001c40 con 0x7f210c09b4e0 2021-05-25T11:59:55.093 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.045+0000 7f20f2ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:55.093 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.045+0000 7f20f2ffd700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2021-05-25T11:59:55.093 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.045+0000 7f20f2ffd700 10 monclient(hunting): dump: 2021-05-25T11:59:55.094 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:55.094 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:55.094 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:55.094 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:55.094 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:55.095 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:55.095 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:55.095 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:55.095 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:55.096 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:55.096 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.045+0000 7f20f2ffd700 1 -- 172.21.15.102:0/1309835282 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2991771115 0 0) 0x7f210c11bbf0 con 0x7f210c068b40 2021-05-25T11:59:55.096 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.045+0000 7f20f2ffd700 1 -- 172.21.15.102:0/1309835282 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f210c11be20 con 0x7f210c068b40 2021-05-25T11:59:55.096 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.045+0000 7f20f2ffd700 1 -- 172.21.15.102:0/1309835282 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f20f4001980 con 0x7f210c068b40 2021-05-25T11:59:55.096 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.045+0000 7f20f2ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:55.097 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.045+0000 7f20f2ffd700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2021-05-25T11:59:55.097 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.045+0000 7f20f2ffd700 10 monclient(hunting): dump: 2021-05-25T11:59:55.097 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:55.097 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:55.098 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:55.098 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:55.098 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:55.098 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:55.098 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:55.099 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:55.099 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:55.099 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:55.099 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.045+0000 7f20f2ffd700 1 -- 172.21.15.102:0/1309835282 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3802352723 0 0) 0x7f210c11b380 con 0x7f210c068ff0 2021-05-25T11:59:55.099 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.045+0000 7f20f2ffd700 1 -- 172.21.15.102:0/1309835282 >> v1:172.21.15.181:6789/0 conn(0x7f210c068b40 legacy=0x7f210c121310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:55.100 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.045+0000 7f20f2ffd700 1 -- 172.21.15.102:0/1309835282 >> v1:172.21.15.102:6789/0 conn(0x7f210c09b4e0 legacy=0x7f210c142da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:55.100 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.046+0000 7f20f2ffd700 10 monclient: _finish_hunting 0 2021-05-25T11:59:55.100 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.046+0000 7f20f2ffd700 1 monclient: found mon.c 2021-05-25T11:59:55.100 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.046+0000 7f20f2ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T11:59:55.101 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.046+0000 7f20f2ffd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T11:59:55.101 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.046+0000 7f20f2ffd700 1 -- 172.21.15.102:0/1309835282 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f210c094720 con 0x7f210c068ff0 2021-05-25T11:59:55.101 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.046+0000 7f20f2ffd700 10 monclient: _finish_auth 0 2021-05-25T11:59:55.101 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.046+0000 7f20f2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T11:59:55.101 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.046+0000 7f210bfff700 5 monclient: authenticate success, global_id 4274 2021-05-25T11:59:55.102 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.046+0000 7f210bfff700 10 monclient: _renew_subs 2021-05-25T11:59:55.102 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.046+0000 7f210bfff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T11:59:55.102 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.046+0000 7f210bfff700 1 -- 172.21.15.102:0/1309835282 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f210c119d10 con 0x7f210c068ff0 2021-05-25T11:59:55.102 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.046+0000 7f210bfff700 10 client.4274.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T11:59:55.102 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.046+0000 7f210bfff700 10 monclient: _renew_subs 2021-05-25T11:59:55.103 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.046+0000 7f210bfff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T11:59:55.103 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.046+0000 7f210bfff700 1 -- 172.21.15.102:0/1309835282 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f210c13b160 con 0x7f210c068ff0 2021-05-25T11:59:55.103 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.046+0000 7f210bfff700 1 librados: init done 2021-05-25T11:59:55.103 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.046+0000 7f20f2ffd700 1 -- 172.21.15.102:0/1309835282 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2100004830 con 0x7f210c068ff0 2021-05-25T11:59:55.104 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.046+0000 7f20f2ffd700 10 monclient: handle_config config(0 keys) v1 2021-05-25T11:59:55.104 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.046+0000 7f20f2ffd700 1 -- 172.21.15.102:0/1309835282 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f2100004bc0 con 0x7f210c068ff0 2021-05-25T11:59:55.104 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.046+0000 7f20f2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:55.104 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.046+0000 7f21097fa700 4 set_mon_vals no callback set 2021-05-25T11:59:55.104 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.046+0000 7f20f2ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T11:59:55.105 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.046+0000 7f20f2ffd700 10 monclient: dump: 2021-05-25T11:59:55.105 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:55.105 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:55.105 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:55.106 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:55.106 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:55.106 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:55.106 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:55.106 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:55.107 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:55.107 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:55.107 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.046+0000 7f210bfff700 10 client.4274.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:55.107 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.046+0000 7f210bfff700 15 client.4274.objecter _session_command_op_assign -1 1 2021-05-25T11:59:55.108 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.046+0000 7f210bfff700 20 client.4274.objecter _get_session osd=-1 returning homeless 2021-05-25T11:59:55.108 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.046+0000 7f210bfff700 10 client.4274.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T11:59:55.108 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.046+0000 7f210bfff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T11:59:55.108 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.046+0000 7f210bfff700 1 -- 172.21.15.102:0/1309835282 --> v1:172.21.15.102:6790/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f20e40028c0 con 0x7f210c068ff0 2021-05-25T11:59:55.108 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.046+0000 7f20f2ffd700 1 -- 172.21.15.102:0/1309835282 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f2100004da0 con 0x7f210c068ff0 2021-05-25T11:59:55.109 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.047+0000 7f20f2ffd700 1 -- 172.21.15.102:0/1309835282 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(15..15 src has 1..15) v4 ==== 3549+0+0 (unknown 3948819498 0 0) 0x7f210003cfe0 con 0x7f210c068ff0 2021-05-25T11:59:55.109 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.047+0000 7f20f2ffd700 10 client.4274.objecter ms_dispatch 0x7f210c001550 osd_map(15..15 src has 1..15) v4 2021-05-25T11:59:55.109 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.047+0000 7f20f2ffd700 3 client.4274.objecter handle_osd_map got epochs [15,15] > 0 2021-05-25T11:59:55.109 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.047+0000 7f20f2ffd700 3 client.4274.objecter handle_osd_map decoding full epoch 15 2021-05-25T11:59:55.109 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.047+0000 7f20f2ffd700 10 client.4274.objecter checking command 1 2021-05-25T11:59:55.110 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.047+0000 7f20f2ffd700 20 client.4274.objecter _get_session s=0x7f20ec039c20 osd=3 3 2021-05-25T11:59:55.110 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.047+0000 7f20f2ffd700 20 client.4274.objecter put_session s=0x7f20ec039c20 osd=3 3 2021-05-25T11:59:55.110 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.047+0000 7f20f2ffd700 15 client.4274.objecter _session_command_op_remove -1 1 2021-05-25T11:59:55.110 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.047+0000 7f20f2ffd700 20 client.4274.objecter _get_session s=0x7f20ec039c20 osd=3 3 2021-05-25T11:59:55.111 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.047+0000 7f20f2ffd700 20 client.4274.objecter get_session s=0x7f20ec039c20 osd=3 3 2021-05-25T11:59:55.111 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.047+0000 7f20f2ffd700 15 client.4274.objecter _session_command_op_assign 3 1 2021-05-25T11:59:55.111 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.047+0000 7f20f2ffd700 20 client.4274.objecter put_session s=0x7f20ec039c20 osd=3 4 2021-05-25T11:59:55.111 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.047+0000 7f20f2ffd700 10 client.4274.objecter _send_command 1 2021-05-25T11:59:55.111 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.047+0000 7f20f2ffd700 1 -- 172.21.15.102:0/1309835282 --> v1:172.21.15.181:6804/30005 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f20ec03dfa0 con 0x7f20ec03a2d0 2021-05-25T11:59:55.112 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.047+0000 7f20f2ffd700 20 client.4274.objecter dump_active .. 0 homeless 2021-05-25T11:59:55.112 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.047+0000 7f20f2ffd700 1 -- 172.21.15.102:0/1309835282 <== mon.2 v1:172.21.15.102:6790/0 9 ==== mon_get_version_reply(handle=1 version=15) v2 ==== 24+0+0 (unknown 2942983318 0 0) 0x7f210003d410 con 0x7f210c068ff0 2021-05-25T11:59:55.112 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.047+0000 7f20f2ffd700 10 monclient: handle_get_version_reply finishing 1 version 15 2021-05-25T11:59:55.112 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.047+0000 7f210affd700 10 monclient: get_auth_request con 0x7f20ec0334b0 auth_method 0 2021-05-25T11:59:55.113 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.047+0000 7f210b7fe700 10 monclient: get_auth_request con 0x7f20ec03a2d0 auth_method 0 2021-05-25T11:59:55.113 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.047+0000 7f20f2ffd700 10 client.4274.objecter ms_handle_connect 0x7f20ec0334b0 2021-05-25T11:59:55.113 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.047+0000 7f20f2ffd700 10 client.4274.objecter ms_handle_connect 0x7f20ec03a2d0 2021-05-25T11:59:55.113 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.048+0000 7f20f2ffd700 1 -- 172.21.15.102:0/1309835282 <== osd.3 v1:172.21.15.181:6804/30005 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23336 (unknown 0 0 3933607479) 0x7f20ec03dfa0 con 0x7f20ec03a2d0 2021-05-25T11:59:55.113 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.048+0000 7f20f2ffd700 10 client.4274.objecter ms_dispatch 0x7f210c001550 command_reply(tid 1: 0 ) v1 2021-05-25T11:59:55.114 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.048+0000 7f20f2ffd700 10 client.4274.objecter _finish_command 1 = system:0 2021-05-25T11:59:55.114 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.048+0000 7f20f2ffd700 20 client.4274.objecter put_session s=0x7f20ec039c20 osd=3 4 2021-05-25T11:59:55.114 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.048+0000 7f20f2ffd700 15 client.4274.objecter _session_command_op_remove 3 1 2021-05-25T11:59:55.115 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.062+0000 7f210bfff700 10 client.4274.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2021-05-25T11:59:55.116 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.062+0000 7f210bfff700 15 client.4274.objecter _session_command_op_assign -1 2 2021-05-25T11:59:55.116 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.062+0000 7f210bfff700 20 client.4274.objecter _get_session s=0x7f20ec039c20 osd=3 3 2021-05-25T11:59:55.116 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.062+0000 7f210bfff700 20 client.4274.objecter put_session s=0x7f20ec039c20 osd=3 3 2021-05-25T11:59:55.116 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.062+0000 7f210bfff700 20 client.4274.objecter _get_session s=0x7f20ec039c20 osd=3 3 2021-05-25T11:59:55.117 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.062+0000 7f210bfff700 15 client.4274.objecter _session_command_op_remove -1 2 2021-05-25T11:59:55.117 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.062+0000 7f210bfff700 20 client.4274.objecter get_session s=0x7f20ec039c20 osd=3 3 2021-05-25T11:59:55.117 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.062+0000 7f210bfff700 15 client.4274.objecter _session_command_op_assign 3 2 2021-05-25T11:59:55.117 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.062+0000 7f210bfff700 20 client.4274.objecter put_session s=0x7f20ec039c20 osd=3 4 2021-05-25T11:59:55.117 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.062+0000 7f210bfff700 10 client.4274.objecter _send_command 2 2021-05-25T11:59:55.118 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.062+0000 7f210bfff700 1 -- 172.21.15.102:0/1309835282 --> v1:172.21.15.181:6804/30005 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f20e4001150 con 0x7f20ec03a2d0 2021-05-25T11:59:55.118 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.063+0000 7f20f2ffd700 1 -- 172.21.15.102:0/1309835282 <== osd.3 v1:172.21.15.181:6804/30005 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (unknown 0 0 2570787547) 0x7f20e4001150 con 0x7f20ec03a2d0 2021-05-25T11:59:55.118 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.063+0000 7f20f2ffd700 10 client.4274.objecter ms_dispatch 0x7f210c001550 command_reply(tid 2: 0 ) v1 2021-05-25T11:59:55.118 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.063+0000 7f20f2ffd700 10 client.4274.objecter _finish_command 2 = system:0 2021-05-25T11:59:55.119 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.063+0000 7f20f2ffd700 20 client.4274.objecter put_session s=0x7f20ec039c20 osd=3 4 2021-05-25T11:59:55.119 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.063+0000 7f20f2ffd700 15 client.4274.objecter _session_command_op_remove 3 2 2021-05-25T11:59:55.121 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.063+0000 7f210bfff700 10 librados: watch_flush enter 2021-05-25T11:59:55.121 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.063+0000 7f210bfff700 10 librados: watch_flush exit 2021-05-25T11:59:55.122 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.063+0000 7f210bfff700 10 client.4274.objecter close_session for osd.3 2021-05-25T11:59:55.122 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.063+0000 7f210bfff700 1 -- 172.21.15.102:0/1309835282 >> v1:172.21.15.181:6804/30005 conn(0x7f20ec03a2d0 legacy=0x7f20ec03c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:55.122 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.063+0000 7f210bfff700 20 client.4274.objecter put_session s=0x7f20ec039c20 osd=3 1 2021-05-25T11:59:55.122 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.063+0000 7f210bfff700 20 client.4274.objecter shutdown clearing up homeless session... 2021-05-25T11:59:55.123 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.063+0000 7f210bfff700 10 client.4274.objecter successfully canceled tick 2021-05-25T11:59:55.123 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.063+0000 7f210bfff700 1 -- 172.21.15.102:0/1309835282 >> v1:172.21.15.181:6812/29582 conn(0x7f20ec0334b0 legacy=0x7f20ec035970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:55.123 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.063+0000 7f210bfff700 10 monclient: shutdown 2021-05-25T11:59:55.123 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.063+0000 7f210bfff700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T11:59:55.123 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.063+0000 7f210bfff700 1 -- 172.21.15.102:0/1309835282 >> v1:172.21.15.102:6790/0 conn(0x7f210c068ff0 legacy=0x7f210c13aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:55.124 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.063+0000 7f210b7fe700 1 -- 172.21.15.102:0/1309835282 reap_dead start 2021-05-25T11:59:55.124 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.063+0000 7f210bfff700 1 -- 172.21.15.102:0/1309835282 shutdown_connections 2021-05-25T11:59:55.124 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.063+0000 7f210bfff700 1 -- 172.21.15.102:0/1309835282 >> 172.21.15.102:0/1309835282 conn(0x7f210c097f90 msgr2=0x7f210c092410 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T11:59:55.124 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.063+0000 7f210bfff700 1 -- 172.21.15.102:0/1309835282 shutdown_connections 2021-05-25T11:59:55.124 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.063+0000 7f210bfff700 1 -- 172.21.15.102:0/1309835282 wait complete. 2021-05-25T11:59:55.125 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.063+0000 7f210bfff700 1 librados: shutdown 2021-05-25T11:59:55.125 INFO:teuthology.orchestra.run.smithi102.stdout:47244640258 2021-05-25T11:59:55.125 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2021-05-25T11:59:55.235 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.233+0000 7fbd7c5c3700 10 monclient: get_monmap_and_config 2021-05-25T11:59:55.235 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.233+0000 7fbd7c5c3700 10 monclient: build_initial_monmap 2021-05-25T11:59:55.235 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.233+0000 7fbd7c5c3700 10 monclient: monmap: 2021-05-25T11:59:55.235 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T11:59:55.236 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:55.236 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:55.234566+0000 2021-05-25T11:59:55.236 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:55.234566+0000 2021-05-25T11:59:55.236 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T11:59:55.237 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:55.237 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T11:59:55.237 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T11:59:55.237 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T11:59:55.237 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:55.249 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.234+0000 7fbd7c5c3700 1 Processor -- start 2021-05-25T11:59:55.249 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.234+0000 7fbd7c5c3700 1 -- start start 2021-05-25T11:59:55.249 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.234+0000 7fbd7c5c3700 10 monclient: init 2021-05-25T11:59:55.250 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.234+0000 7fbd7c5c3700 10 monclient: _reopen_session rank -1 2021-05-25T11:59:55.250 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.234+0000 7fbd7c5c3700 10 monclient: _add_conns ranks=[1,0,2] 2021-05-25T11:59:55.250 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.234+0000 7fbd7c5c3700 10 monclient(hunting): picked mon.noname-c con 0x7fbd740966d0 addr v1:172.21.15.181:6789/0 2021-05-25T11:59:55.250 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.234+0000 7fbd7c5c3700 10 monclient(hunting): picked mon.noname-a con 0x7fbd74096ab0 addr v1:172.21.15.102:6789/0 2021-05-25T11:59:55.251 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.234+0000 7fbd7c5c3700 10 monclient(hunting): picked mon.noname-b con 0x7fbd74132010 addr v1:172.21.15.102:6790/0 2021-05-25T11:59:55.251 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.234+0000 7fbd7c5c3700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd74064610 con 0x7fbd74096ab0 2021-05-25T11:59:55.251 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.234+0000 7fbd7c5c3700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd74064840 con 0x7fbd740966d0 2021-05-25T11:59:55.251 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.234+0000 7fbd7c5c3700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd74064ae0 con 0x7fbd74132010 2021-05-25T11:59:55.251 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.234+0000 7fbd7c5c3700 10 monclient(hunting): _renew_subs 2021-05-25T11:59:55.252 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.234+0000 7fbd7c5c3700 10 monclient(hunting): authenticate will time out at 795.831904s 2021-05-25T11:59:55.252 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.235+0000 7fbd79b5e700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fbd74096ab0 0x7fbd741308f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:58844/0 (socket says 172.21.15.102:58844) 2021-05-25T11:59:55.252 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.235+0000 7fbd79b5e700 1 -- 172.21.15.102:0/1542933089 learned_addr learned my addr 172.21.15.102:0/1542933089 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T11:59:55.252 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.235+0000 7fbd7ab60700 1 --1- 172.21.15.102:0/1542933089 >> v1:172.21.15.102:6790/0 conn(0x7fbd74132010 0x7fbd74134400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:53724/0 (socket says 172.21.15.102:53724) 2021-05-25T11:59:55.253 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.235+0000 7fbd7935d700 1 -- 172.21.15.102:0/1542933089 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1919970893 0 0) 0x7fbd74064610 con 0x7fbd74096ab0 2021-05-25T11:59:55.253 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.235+0000 7fbd7935d700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:55.253 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.235+0000 7fbd7935d700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:55.253 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.235+0000 7fbd7935d700 10 monclient(hunting): my global_id is 4284 2021-05-25T11:59:55.253 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.235+0000 7fbd7935d700 1 -- 172.21.15.102:0/1542933089 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd5c003270 con 0x7fbd74096ab0 2021-05-25T11:59:55.254 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.235+0000 7fbd7935d700 1 -- 172.21.15.102:0/1542933089 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1333101250 0 0) 0x7fbd74064ae0 con 0x7fbd74132010 2021-05-25T11:59:55.254 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.235+0000 7fbd7935d700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:55.254 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.235+0000 7fbd7935d700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:55.254 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.235+0000 7fbd7935d700 10 monclient(hunting): my global_id is 4277 2021-05-25T11:59:55.254 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.235+0000 7fbd7935d700 1 -- 172.21.15.102:0/1542933089 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd74064610 con 0x7fbd74132010 2021-05-25T11:59:55.255 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.235+0000 7fbd7935d700 1 -- 172.21.15.102:0/1542933089 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4041503195 0 0) 0x7fbd74064610 con 0x7fbd74132010 2021-05-25T11:59:55.255 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.235+0000 7fbd7935d700 1 -- 172.21.15.102:0/1542933089 >> v1:172.21.15.181:6789/0 conn(0x7fbd740966d0 legacy=0x7fbd740697d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:55.255 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.235+0000 7fbd7935d700 1 -- 172.21.15.102:0/1542933089 >> v1:172.21.15.102:6789/0 conn(0x7fbd74096ab0 legacy=0x7fbd741308f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:55.255 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.235+0000 7fbd7935d700 10 monclient: _finish_hunting 0 2021-05-25T11:59:55.256 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.235+0000 7fbd7935d700 1 monclient: found mon.noname-b 2021-05-25T11:59:55.256 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.235+0000 7fbd7935d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T11:59:55.256 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.235+0000 7fbd7935d700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T11:59:55.256 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.235+0000 7fbd7935d700 1 -- 172.21.15.102:0/1542933089 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd74069ee0 con 0x7fbd74132010 2021-05-25T11:59:55.256 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.235+0000 7fbd7935d700 10 monclient: _finish_auth 0 2021-05-25T11:59:55.257 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.235+0000 7fbd7935d700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T11:59:55.257 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.235+0000 7fbd7935d700 1 -- 172.21.15.102:0/1542933089 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fbd70001990 con 0x7fbd74132010 2021-05-25T11:59:55.257 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.235+0000 7fbd7c5c3700 5 monclient: authenticate success, global_id 4277 2021-05-25T11:59:55.257 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.235+0000 7fbd7c5c3700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T11:59:55.257 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.235+0000 7fbd7935d700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:55.258 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.235+0000 7fbd7935d700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T11:59:55.258 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.235+0000 7fbd7935d700 10 monclient: dump: 2021-05-25T11:59:55.258 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:55.258 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:55.259 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:55.259 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:55.259 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:55.259 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:55.259 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:55.260 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:55.260 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:55.260 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:55.260 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.235+0000 7fbd7c5c3700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T11:59:55.260 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.235+0000 7fbd7935d700 1 -- 172.21.15.102:0/1542933089 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbd70001bb0 con 0x7fbd74132010 2021-05-25T11:59:55.261 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.235+0000 7fbd7935d700 10 monclient: handle_config config(0 keys) v1 2021-05-25T11:59:55.261 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.235+0000 7fbd7c5c3700 10 monclient: get_monmap_and_config success 2021-05-25T11:59:55.261 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.235+0000 7fbd7935d700 1 -- 172.21.15.102:0/1542933089 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fbd70001f40 con 0x7fbd74132010 2021-05-25T11:59:55.261 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.235+0000 7fbd7935d700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:55.261 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.235+0000 7fbd7c5c3700 4 set_mon_vals no callback set 2021-05-25T11:59:55.262 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.235+0000 7fbd7935d700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T11:59:55.262 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.235+0000 7fbd7935d700 10 monclient: dump: 2021-05-25T11:59:55.262 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:55.262 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:55.263 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:55.263 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:55.263 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:55.263 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:55.263 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:55.264 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:55.264 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:55.264 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:55.264 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.236+0000 7fbd7c5c3700 10 monclient: shutdown 2021-05-25T11:59:55.265 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.236+0000 7fbd7c5c3700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T11:59:55.265 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.236+0000 7fbd7c5c3700 1 -- 172.21.15.102:0/1542933089 >> v1:172.21.15.102:6790/0 conn(0x7fbd74132010 legacy=0x7fbd74134400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:55.265 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.236+0000 7fbd7c5c3700 1 -- 172.21.15.102:0/1542933089 shutdown_connections 2021-05-25T11:59:55.265 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.236+0000 7fbd7c5c3700 1 -- 172.21.15.102:0/1542933089 >> 172.21.15.102:0/1542933089 conn(0x7fbd7409cd10 msgr2=0x7fbd7409d120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T11:59:55.265 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.236+0000 7fbd7c5c3700 1 -- 172.21.15.102:0/1542933089 shutdown_connections 2021-05-25T11:59:55.266 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.236+0000 7fbd7c5c3700 1 -- 172.21.15.102:0/1542933089 wait complete. 2021-05-25T11:59:55.266 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.236+0000 7fbd7c5c3700 10 monclient: build_initial_monmap 2021-05-25T11:59:55.266 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.236+0000 7fbd7c5c3700 10 monclient: monmap: 2021-05-25T11:59:55.266 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T11:59:55.267 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T11:59:55.267 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T11:59:55.267 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T11:59:55.267 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T11:59:55.267 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T11:59:55.268 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T11:59:55.268 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T11:59:55.268 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T11:59:55.268 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:55.268 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.236+0000 7fbd7c5c3700 1 librados: starting msgr at 2021-05-25T11:59:55.269 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.236+0000 7fbd7c5c3700 1 librados: starting objecter 2021-05-25T11:59:55.269 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.236+0000 7fbd7c5c3700 1 Processor -- start 2021-05-25T11:59:55.269 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.236+0000 7fbd7c5c3700 1 -- start start 2021-05-25T11:59:55.269 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.236+0000 7fbd7c5c3700 1 librados: setting wanted keys 2021-05-25T11:59:55.270 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.236+0000 7fbd7c5c3700 1 librados: calling monclient init 2021-05-25T11:59:55.270 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.236+0000 7fbd7c5c3700 10 monclient: init 2021-05-25T11:59:55.270 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.237+0000 7fbd7c5c3700 10 monclient: _reopen_session rank -1 2021-05-25T11:59:55.270 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.237+0000 7fbd7c5c3700 10 monclient: _add_conns ranks=[0,1,2] 2021-05-25T11:59:55.270 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.237+0000 7fbd7c5c3700 10 monclient(hunting): picked mon.noname-a con 0x7fbd740966d0 addr v1:172.21.15.102:6789/0 2021-05-25T11:59:55.271 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.237+0000 7fbd7c5c3700 10 monclient(hunting): picked mon.noname-b con 0x7fbd74096ab0 addr v1:172.21.15.181:6789/0 2021-05-25T11:59:55.271 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.237+0000 7fbd7c5c3700 10 monclient(hunting): picked mon.noname-c con 0x7fbd74132010 addr v1:172.21.15.102:6790/0 2021-05-25T11:59:55.271 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.237+0000 7fbd7c5c3700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd74095780 con 0x7fbd740966d0 2021-05-25T11:59:55.271 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.237+0000 7fbd7c5c3700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd74131820 con 0x7fbd74096ab0 2021-05-25T11:59:55.272 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.237+0000 7fbd7c5c3700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd74118af0 con 0x7fbd74132010 2021-05-25T11:59:55.272 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.237+0000 7fbd7c5c3700 10 monclient(hunting): _renew_subs 2021-05-25T11:59:55.272 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.237+0000 7fbd7c5c3700 10 monclient(hunting): authenticate will time out at 795.834378s 2021-05-25T11:59:55.272 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.237+0000 7fbd7a35f700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fbd740966d0 0x7fbd7413ad30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:58848/0 (socket says 172.21.15.102:58848) 2021-05-25T11:59:55.272 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.237+0000 7fbd7a35f700 1 -- 172.21.15.102:0/1229234488 learned_addr learned my addr 172.21.15.102:0/1229234488 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T11:59:55.273 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.237+0000 7fbd7ab60700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fbd74132010 0x7fbd74143080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:53730/0 (socket says 172.21.15.102:53730) 2021-05-25T11:59:55.273 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.237+0000 7fbd627fc700 10 client.?.objecter ms_handle_connect 0x7fbd740966d0 2021-05-25T11:59:55.273 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.237+0000 7fbd627fc700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:55.273 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.237+0000 7fbd627fc700 10 client.?.objecter ms_handle_connect 0x7fbd74132010 2021-05-25T11:59:55.273 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.237+0000 7fbd627fc700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:55.274 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.237+0000 7fbd627fc700 1 -- 172.21.15.102:0/1229234488 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2708634419 0 0) 0x7fbd74095780 con 0x7fbd740966d0 2021-05-25T11:59:55.274 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.237+0000 7fbd627fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:55.274 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.237+0000 7fbd627fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:55.274 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.237+0000 7fbd627fc700 10 monclient(hunting): my global_id is 4287 2021-05-25T11:59:55.275 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.237+0000 7fbd627fc700 1 -- 172.21.15.102:0/1229234488 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd54004c10 con 0x7fbd740966d0 2021-05-25T11:59:55.275 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.237+0000 7fbd627fc700 10 client.?.objecter ms_handle_connect 0x7fbd74096ab0 2021-05-25T11:59:55.275 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.237+0000 7fbd627fc700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:55.275 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.237+0000 7fbd627fc700 1 -- 172.21.15.102:0/1229234488 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3541669130 0 0) 0x7fbd74118af0 con 0x7fbd74132010 2021-05-25T11:59:55.275 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.237+0000 7fbd627fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:55.276 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.237+0000 7fbd627fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:55.276 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.237+0000 7fbd627fc700 10 monclient(hunting): my global_id is 4280 2021-05-25T11:59:55.276 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.237+0000 7fbd627fc700 1 -- 172.21.15.102:0/1229234488 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd74095780 con 0x7fbd74132010 2021-05-25T11:59:55.276 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.237+0000 7fbd627fc700 1 -- 172.21.15.102:0/1229234488 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3499504234 0 0) 0x7fbd74131820 con 0x7fbd74096ab0 2021-05-25T11:59:55.276 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.237+0000 7fbd627fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:55.277 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.237+0000 7fbd627fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:55.277 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.237+0000 7fbd627fc700 10 monclient(hunting): my global_id is 4282 2021-05-25T11:59:55.277 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.237+0000 7fbd627fc700 1 -- 172.21.15.102:0/1229234488 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd74118af0 con 0x7fbd74096ab0 2021-05-25T11:59:55.277 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.237+0000 7fbd627fc700 1 -- 172.21.15.102:0/1229234488 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3286579353 0 0) 0x7fbd54004c10 con 0x7fbd740966d0 2021-05-25T11:59:55.278 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.237+0000 7fbd627fc700 1 -- 172.21.15.102:0/1229234488 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbd74131820 con 0x7fbd740966d0 2021-05-25T11:59:55.278 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.237+0000 7fbd627fc700 1 -- 172.21.15.102:0/1229234488 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fbd68001940 con 0x7fbd740966d0 2021-05-25T11:59:55.278 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.237+0000 7fbd627fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:55.278 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.237+0000 7fbd627fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T11:59:55.278 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.237+0000 7fbd627fc700 10 monclient(hunting): dump: 2021-05-25T11:59:55.279 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:55.282 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:55.282 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:55.282 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:55.283 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:55.283 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:55.283 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:55.283 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:55.283 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:55.284 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:55.284 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.237+0000 7fbd627fc700 10 monclient(hunting): _finish_auth 0 2021-05-25T11:59:55.284 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.237+0000 7fbd627fc700 1 -- 172.21.15.102:0/1229234488 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3004428708 0 0) 0x7fbd74095780 con 0x7fbd74132010 2021-05-25T11:59:55.284 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.237+0000 7fbd627fc700 1 -- 172.21.15.102:0/1229234488 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbd54004c10 con 0x7fbd74132010 2021-05-25T11:59:55.285 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.237+0000 7fbd627fc700 1 -- 172.21.15.102:0/1229234488 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fbd70001950 con 0x7fbd74132010 2021-05-25T11:59:55.285 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.237+0000 7fbd627fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:55.285 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.238+0000 7fbd627fc700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2021-05-25T11:59:55.285 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.238+0000 7fbd627fc700 10 monclient(hunting): dump: 2021-05-25T11:59:55.285 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:55.286 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:55.286 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:55.286 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:55.286 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:55.287 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:55.287 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:55.287 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:55.287 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:55.287 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:55.288 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.238+0000 7fbd627fc700 1 -- 172.21.15.102:0/1229234488 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1991077776 0 0) 0x7fbd74131820 con 0x7fbd740966d0 2021-05-25T11:59:55.288 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.238+0000 7fbd627fc700 1 -- 172.21.15.102:0/1229234488 >> v1:172.21.15.102:6790/0 conn(0x7fbd74132010 legacy=0x7fbd74143080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:55.288 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.238+0000 7fbd627fc700 1 -- 172.21.15.102:0/1229234488 >> v1:172.21.15.181:6789/0 conn(0x7fbd74096ab0 legacy=0x7fbd74131110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:55.288 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.238+0000 7fbd627fc700 10 monclient: _finish_hunting 0 2021-05-25T11:59:55.289 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.238+0000 7fbd627fc700 1 monclient: found mon.a 2021-05-25T11:59:55.289 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.238+0000 7fbd627fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T11:59:55.289 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.238+0000 7fbd627fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:55.289 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.238+0000 7fbd627fc700 1 -- 172.21.15.102:0/1229234488 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd74117780 con 0x7fbd740966d0 2021-05-25T11:59:55.289 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.238+0000 7fbd627fc700 10 monclient: _finish_auth 0 2021-05-25T11:59:55.290 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.238+0000 7fbd627fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T11:59:55.290 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.238+0000 7fbd7c5c3700 5 monclient: authenticate success, global_id 4287 2021-05-25T11:59:55.290 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.238+0000 7fbd7c5c3700 10 monclient: _renew_subs 2021-05-25T11:59:55.290 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.238+0000 7fbd7c5c3700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:55.290 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.238+0000 7fbd7c5c3700 1 -- 172.21.15.102:0/1229234488 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd74068480 con 0x7fbd740966d0 2021-05-25T11:59:55.291 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.238+0000 7fbd7c5c3700 10 client.4287.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T11:59:55.291 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.238+0000 7fbd7c5c3700 10 monclient: _renew_subs 2021-05-25T11:59:55.291 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.238+0000 7fbd7c5c3700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:55.291 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.238+0000 7fbd7c5c3700 1 -- 172.21.15.102:0/1229234488 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd7413ba20 con 0x7fbd740966d0 2021-05-25T11:59:55.292 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.238+0000 7fbd7c5c3700 1 librados: init done 2021-05-25T11:59:55.292 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.238+0000 7fbd627fc700 1 -- 172.21.15.102:0/1229234488 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbd68001cd0 con 0x7fbd740966d0 2021-05-25T11:59:55.292 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.238+0000 7fbd627fc700 10 monclient: handle_config config(0 keys) v1 2021-05-25T11:59:55.292 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.238+0000 7fbd627fc700 1 -- 172.21.15.102:0/1229234488 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fbd68002410 con 0x7fbd740966d0 2021-05-25T11:59:55.292 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.238+0000 7fbd627fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:55.293 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.238+0000 7fbd78b5c700 4 set_mon_vals no callback set 2021-05-25T11:59:55.293 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.238+0000 7fbd627fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T11:59:55.293 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.238+0000 7fbd627fc700 10 monclient: dump: 2021-05-25T11:59:55.293 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:55.293 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:55.294 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:55.294 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:55.294 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:55.294 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:55.294 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:55.295 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:55.295 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:55.295 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:55.295 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.238+0000 7fbd7c5c3700 10 client.4287.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:55.295 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.238+0000 7fbd7c5c3700 15 client.4287.objecter _session_command_op_assign -1 1 2021-05-25T11:59:55.296 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.238+0000 7fbd7c5c3700 20 client.4287.objecter _get_session osd=-1 returning homeless 2021-05-25T11:59:55.296 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.238+0000 7fbd7c5c3700 10 client.4287.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T11:59:55.296 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.238+0000 7fbd7c5c3700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:55.296 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.238+0000 7fbd7c5c3700 1 -- 172.21.15.102:0/1229234488 --> v1:172.21.15.102:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fbd580028c0 con 0x7fbd740966d0 2021-05-25T11:59:55.296 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.238+0000 7fbd627fc700 1 -- 172.21.15.102:0/1229234488 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7fbd680025f0 con 0x7fbd740966d0 2021-05-25T11:59:55.297 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.239+0000 7fbd627fc700 1 -- 172.21.15.102:0/1229234488 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(15..15 src has 1..15) v4 ==== 3549+0+0 (unknown 3948819498 0 0) 0x7fbd6803ce80 con 0x7fbd740966d0 2021-05-25T11:59:55.297 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.239+0000 7fbd627fc700 10 client.4287.objecter ms_dispatch 0x7fbd74001550 osd_map(15..15 src has 1..15) v4 2021-05-25T11:59:55.297 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.239+0000 7fbd627fc700 3 client.4287.objecter handle_osd_map got epochs [15,15] > 0 2021-05-25T11:59:55.297 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.239+0000 7fbd627fc700 3 client.4287.objecter handle_osd_map decoding full epoch 15 2021-05-25T11:59:55.298 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.239+0000 7fbd627fc700 10 client.4287.objecter checking command 1 2021-05-25T11:59:55.298 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.239+0000 7fbd627fc700 20 client.4287.objecter _get_session s=0x7fbd54039a30 osd=4 3 2021-05-25T11:59:55.298 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.239+0000 7fbd627fc700 20 client.4287.objecter put_session s=0x7fbd54039a30 osd=4 3 2021-05-25T11:59:55.298 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.239+0000 7fbd627fc700 15 client.4287.objecter _session_command_op_remove -1 1 2021-05-25T11:59:55.298 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.239+0000 7fbd627fc700 20 client.4287.objecter _get_session s=0x7fbd54039a30 osd=4 3 2021-05-25T11:59:55.299 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.239+0000 7fbd627fc700 20 client.4287.objecter get_session s=0x7fbd54039a30 osd=4 3 2021-05-25T11:59:55.299 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.239+0000 7fbd627fc700 15 client.4287.objecter _session_command_op_assign 4 1 2021-05-25T11:59:55.299 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.239+0000 7fbd627fc700 20 client.4287.objecter put_session s=0x7fbd54039a30 osd=4 4 2021-05-25T11:59:55.299 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.239+0000 7fbd627fc700 10 client.4287.objecter _send_command 1 2021-05-25T11:59:55.299 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.239+0000 7fbd627fc700 1 -- 172.21.15.102:0/1229234488 --> v1:172.21.15.181:6800/30006 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbd5403ddb0 con 0x7fbd5403a0e0 2021-05-25T11:59:55.300 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.239+0000 7fbd627fc700 20 client.4287.objecter dump_active .. 0 homeless 2021-05-25T11:59:55.300 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.239+0000 7fbd627fc700 1 -- 172.21.15.102:0/1229234488 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_get_version_reply(handle=1 version=15) v2 ==== 24+0+0 (unknown 2942983318 0 0) 0x7fbd6803d2b0 con 0x7fbd740966d0 2021-05-25T11:59:55.300 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.239+0000 7fbd627fc700 10 monclient: handle_get_version_reply finishing 1 version 15 2021-05-25T11:59:55.300 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.239+0000 7fbd79b5e700 10 monclient: get_auth_request con 0x7fbd54033220 auth_method 0 2021-05-25T11:59:55.300 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.239+0000 7fbd7ab60700 10 monclient: get_auth_request con 0x7fbd5403a0e0 auth_method 0 2021-05-25T11:59:55.301 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.239+0000 7fbd627fc700 10 client.4287.objecter ms_handle_connect 0x7fbd54033220 2021-05-25T11:59:55.301 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.239+0000 7fbd627fc700 10 client.4287.objecter ms_handle_connect 0x7fbd5403a0e0 2021-05-25T11:59:55.301 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.240+0000 7fbd627fc700 1 -- 172.21.15.102:0/1229234488 <== osd.4 v1:172.21.15.181:6800/30006 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23336 (unknown 0 0 3933607479) 0x7fbd5403ddb0 con 0x7fbd5403a0e0 2021-05-25T11:59:55.301 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.240+0000 7fbd627fc700 10 client.4287.objecter ms_dispatch 0x7fbd74001550 command_reply(tid 1: 0 ) v1 2021-05-25T11:59:55.302 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.240+0000 7fbd627fc700 10 client.4287.objecter _finish_command 1 = system:0 2021-05-25T11:59:55.302 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.240+0000 7fbd627fc700 20 client.4287.objecter put_session s=0x7fbd54039a30 osd=4 4 2021-05-25T11:59:55.302 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.240+0000 7fbd627fc700 15 client.4287.objecter _session_command_op_remove 4 1 2021-05-25T11:59:55.303 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.254+0000 7fbd7c5c3700 10 client.4287.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2021-05-25T11:59:55.304 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.254+0000 7fbd7c5c3700 15 client.4287.objecter _session_command_op_assign -1 2 2021-05-25T11:59:55.304 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.254+0000 7fbd7c5c3700 20 client.4287.objecter _get_session s=0x7fbd54039a30 osd=4 3 2021-05-25T11:59:55.304 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.254+0000 7fbd7c5c3700 20 client.4287.objecter put_session s=0x7fbd54039a30 osd=4 3 2021-05-25T11:59:55.304 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.254+0000 7fbd7c5c3700 20 client.4287.objecter _get_session s=0x7fbd54039a30 osd=4 3 2021-05-25T11:59:55.304 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.254+0000 7fbd7c5c3700 15 client.4287.objecter _session_command_op_remove -1 2 2021-05-25T11:59:55.305 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.254+0000 7fbd7c5c3700 20 client.4287.objecter get_session s=0x7fbd54039a30 osd=4 3 2021-05-25T11:59:55.305 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.254+0000 7fbd7c5c3700 15 client.4287.objecter _session_command_op_assign 4 2 2021-05-25T11:59:55.305 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.254+0000 7fbd7c5c3700 20 client.4287.objecter put_session s=0x7fbd54039a30 osd=4 4 2021-05-25T11:59:55.305 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.254+0000 7fbd7c5c3700 10 client.4287.objecter _send_command 2 2021-05-25T11:59:55.306 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.254+0000 7fbd7c5c3700 1 -- 172.21.15.102:0/1229234488 --> v1:172.21.15.181:6800/30006 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fbd58001150 con 0x7fbd5403a0e0 2021-05-25T11:59:55.306 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.255+0000 7fbd627fc700 1 -- 172.21.15.102:0/1229234488 <== osd.4 v1:172.21.15.181:6800/30006 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (unknown 0 0 2570787547) 0x7fbd58001150 con 0x7fbd5403a0e0 2021-05-25T11:59:55.306 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.255+0000 7fbd627fc700 10 client.4287.objecter ms_dispatch 0x7fbd74001550 command_reply(tid 2: 0 ) v1 2021-05-25T11:59:55.306 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.255+0000 7fbd627fc700 10 client.4287.objecter _finish_command 2 = system:0 2021-05-25T11:59:55.306 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.255+0000 7fbd627fc700 20 client.4287.objecter put_session s=0x7fbd54039a30 osd=4 4 2021-05-25T11:59:55.307 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.255+0000 7fbd627fc700 15 client.4287.objecter _session_command_op_remove 4 2 2021-05-25T11:59:55.309 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.255+0000 7fbd7c5c3700 10 librados: watch_flush enter 2021-05-25T11:59:55.309 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.255+0000 7fbd7c5c3700 10 librados: watch_flush exit 2021-05-25T11:59:55.309 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.255+0000 7fbd7c5c3700 10 client.4287.objecter close_session for osd.4 2021-05-25T11:59:55.309 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.255+0000 7fbd7c5c3700 1 -- 172.21.15.102:0/1229234488 >> v1:172.21.15.181:6800/30006 conn(0x7fbd5403a0e0 legacy=0x7fbd5403c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:55.310 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.255+0000 7fbd7c5c3700 20 client.4287.objecter put_session s=0x7fbd54039a30 osd=4 1 2021-05-25T11:59:55.310 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.255+0000 7fbd7c5c3700 20 client.4287.objecter shutdown clearing up homeless session... 2021-05-25T11:59:55.310 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.255+0000 7fbd7c5c3700 10 client.4287.objecter successfully canceled tick 2021-05-25T11:59:55.310 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.255+0000 7fbd7c5c3700 1 -- 172.21.15.102:0/1229234488 >> v1:172.21.15.181:6812/29582 conn(0x7fbd54033220 legacy=0x7fbd540356e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:55.311 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.255+0000 7fbd7c5c3700 10 monclient: shutdown 2021-05-25T11:59:55.311 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.255+0000 7fbd7c5c3700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T11:59:55.311 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.255+0000 7fbd7c5c3700 1 -- 172.21.15.102:0/1229234488 >> v1:172.21.15.102:6789/0 conn(0x7fbd740966d0 legacy=0x7fbd7413ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:55.311 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.255+0000 7fbd7ab60700 1 -- 172.21.15.102:0/1229234488 reap_dead start 2021-05-25T11:59:55.311 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.255+0000 7fbd7c5c3700 1 -- 172.21.15.102:0/1229234488 shutdown_connections 2021-05-25T11:59:55.312 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.255+0000 7fbd7c5c3700 1 -- 172.21.15.102:0/1229234488 >> 172.21.15.102:0/1229234488 conn(0x7fbd7409cd10 msgr2=0x7fbd74098180 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T11:59:55.312 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.255+0000 7fbd7c5c3700 1 -- 172.21.15.102:0/1229234488 shutdown_connections 2021-05-25T11:59:55.312 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.255+0000 7fbd7c5c3700 1 -- 172.21.15.102:0/1229234488 wait complete. 2021-05-25T11:59:55.312 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.255+0000 7fbd7c5c3700 1 librados: shutdown 2021-05-25T11:59:55.312 INFO:teuthology.orchestra.run.smithi102.stdout:47244640258 2021-05-25T11:59:55.313 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2021-05-25T11:59:55.422 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.421+0000 7f494d6c6700 10 monclient: get_monmap_and_config 2021-05-25T11:59:55.423 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.421+0000 7f494d6c6700 10 monclient: build_initial_monmap 2021-05-25T11:59:55.423 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.421+0000 7f494d6c6700 10 monclient: monmap: 2021-05-25T11:59:55.423 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T11:59:55.423 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:55.424 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:55.422227+0000 2021-05-25T11:59:55.424 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:55.422227+0000 2021-05-25T11:59:55.424 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T11:59:55.424 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:55.424 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T11:59:55.425 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T11:59:55.425 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T11:59:55.425 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:55.435 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.421+0000 7f494d6c6700 1 Processor -- start 2021-05-25T11:59:55.435 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.421+0000 7f494d6c6700 1 -- start start 2021-05-25T11:59:55.435 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.421+0000 7f494d6c6700 10 monclient: init 2021-05-25T11:59:55.436 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.422+0000 7f494d6c6700 10 monclient: _reopen_session rank -1 2021-05-25T11:59:55.436 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.422+0000 7f494d6c6700 10 monclient: _add_conns ranks=[2,1,0] 2021-05-25T11:59:55.436 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.422+0000 7f494d6c6700 10 monclient(hunting): picked mon.noname-b con 0x7f4948090990 addr v1:172.21.15.102:6790/0 2021-05-25T11:59:55.436 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.422+0000 7f494d6c6700 10 monclient(hunting): picked mon.noname-c con 0x7f4948090d70 addr v1:172.21.15.181:6789/0 2021-05-25T11:59:55.436 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.422+0000 7f494d6c6700 10 monclient(hunting): picked mon.noname-a con 0x7f494812e330 addr v1:172.21.15.102:6789/0 2021-05-25T11:59:55.437 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.422+0000 7f494d6c6700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4948096fa0 con 0x7f494812e330 2021-05-25T11:59:55.437 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.422+0000 7f494d6c6700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49480926a0 con 0x7f4948090d70 2021-05-25T11:59:55.437 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.422+0000 7f494d6c6700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4948092940 con 0x7f4948090990 2021-05-25T11:59:55.437 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.422+0000 7f494d6c6700 10 monclient(hunting): _renew_subs 2021-05-25T11:59:55.438 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.422+0000 7f494d6c6700 10 monclient(hunting): authenticate will time out at 796.019430s 2021-05-25T11:59:55.438 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.422+0000 7f49477fe700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f494812e330 0x7f4948130720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:58860/0 (socket says 172.21.15.102:58860) 2021-05-25T11:59:55.438 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.422+0000 7f4946ffd700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f4948090990 0x7f4948093240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:53738/0 (socket says 172.21.15.102:53738) 2021-05-25T11:59:55.438 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.422+0000 7f49477fe700 1 -- 172.21.15.102:0/1774433653 learned_addr learned my addr 172.21.15.102:0/1774433653 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T11:59:55.438 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.422+0000 7f4945ffb700 1 -- 172.21.15.102:0/1774433653 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2261727711 0 0) 0x7f4948096fa0 con 0x7f494812e330 2021-05-25T11:59:55.439 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.422+0000 7f4945ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:55.439 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.422+0000 7f4945ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:55.439 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.422+0000 7f4945ffb700 10 monclient(hunting): my global_id is 4290 2021-05-25T11:59:55.439 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.422+0000 7f4945ffb700 1 -- 172.21.15.102:0/1774433653 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4934003270 con 0x7f494812e330 2021-05-25T11:59:55.440 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.422+0000 7f4945ffb700 1 -- 172.21.15.102:0/1774433653 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3056832119 0 0) 0x7f4948092940 con 0x7f4948090990 2021-05-25T11:59:55.440 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.422+0000 7f4945ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:55.440 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.422+0000 7f4945ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:55.440 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.422+0000 7f4945ffb700 10 monclient(hunting): my global_id is 4283 2021-05-25T11:59:55.440 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.422+0000 7f4945ffb700 1 -- 172.21.15.102:0/1774433653 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4948096fa0 con 0x7f4948090990 2021-05-25T11:59:55.441 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.423+0000 7f4945ffb700 1 -- 172.21.15.102:0/1774433653 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3608693382 0 0) 0x7f4934003270 con 0x7f494812e330 2021-05-25T11:59:55.441 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.423+0000 7f4945ffb700 1 -- 172.21.15.102:0/1774433653 >> v1:172.21.15.102:6790/0 conn(0x7f4948090990 legacy=0x7f4948093240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:55.441 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.423+0000 7f4945ffb700 1 -- 172.21.15.102:0/1774433653 >> v1:172.21.15.181:6789/0 conn(0x7f4948090d70 legacy=0x7f4948091200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:55.441 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.423+0000 7f4945ffb700 10 monclient: _finish_hunting 0 2021-05-25T11:59:55.441 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.423+0000 7f4945ffb700 1 monclient: found mon.noname-a 2021-05-25T11:59:55.442 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.423+0000 7f4945ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T11:59:55.442 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.423+0000 7f4945ffb700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T11:59:55.442 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.423+0000 7f4945ffb700 1 -- 172.21.15.102:0/1774433653 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4948092be0 con 0x7f494812e330 2021-05-25T11:59:55.442 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.423+0000 7f4945ffb700 10 monclient: _finish_auth 0 2021-05-25T11:59:55.443 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.423+0000 7f4945ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T11:59:55.443 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.423+0000 7f4945ffb700 1 -- 172.21.15.102:0/1774433653 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f4938001990 con 0x7f494812e330 2021-05-25T11:59:55.443 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.423+0000 7f494d6c6700 5 monclient: authenticate success, global_id 4290 2021-05-25T11:59:55.443 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.423+0000 7f494d6c6700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T11:59:55.443 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.423+0000 7f4945ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:55.444 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.423+0000 7f4945ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T11:59:55.444 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.423+0000 7f4945ffb700 10 monclient: dump: 2021-05-25T11:59:55.444 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:55.444 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:55.444 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:55.445 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:55.445 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:55.445 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:55.445 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:55.446 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:55.446 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:55.446 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:55.446 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.423+0000 7f494d6c6700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T11:59:55.446 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.423+0000 7f4945ffb700 1 -- 172.21.15.102:0/1774433653 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4938001bc0 con 0x7f494812e330 2021-05-25T11:59:55.447 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.423+0000 7f4945ffb700 10 monclient: handle_config config(0 keys) v1 2021-05-25T11:59:55.447 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.423+0000 7f4945ffb700 1 -- 172.21.15.102:0/1774433653 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f4938001f50 con 0x7f494812e330 2021-05-25T11:59:55.447 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.423+0000 7f494d6c6700 10 monclient: get_monmap_and_config success 2021-05-25T11:59:55.447 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.423+0000 7f494d6c6700 4 set_mon_vals no callback set 2021-05-25T11:59:55.448 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.423+0000 7f4945ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:55.448 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.423+0000 7f4945ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T11:59:55.448 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.423+0000 7f4945ffb700 10 monclient: dump: 2021-05-25T11:59:55.448 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:55.448 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:55.449 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:55.449 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:55.449 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:55.449 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:55.449 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:55.450 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:55.450 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:55.450 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:55.450 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.423+0000 7f494d6c6700 10 monclient: shutdown 2021-05-25T11:59:55.450 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.423+0000 7f494d6c6700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T11:59:55.451 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.423+0000 7f494d6c6700 1 -- 172.21.15.102:0/1774433653 >> v1:172.21.15.102:6789/0 conn(0x7f494812e330 legacy=0x7f4948130720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:55.451 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.423+0000 7f494d6c6700 1 -- 172.21.15.102:0/1774433653 shutdown_connections 2021-05-25T11:59:55.451 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.423+0000 7f494d6c6700 1 -- 172.21.15.102:0/1774433653 >> 172.21.15.102:0/1774433653 conn(0x7f494809b370 msgr2=0x7f4948097430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T11:59:55.451 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.423+0000 7f494d6c6700 1 -- 172.21.15.102:0/1774433653 shutdown_connections 2021-05-25T11:59:55.452 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.423+0000 7f494d6c6700 1 -- 172.21.15.102:0/1774433653 wait complete. 2021-05-25T11:59:55.452 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.424+0000 7f494d6c6700 10 monclient: build_initial_monmap 2021-05-25T11:59:55.452 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.424+0000 7f494d6c6700 10 monclient: monmap: 2021-05-25T11:59:55.452 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T11:59:55.452 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T11:59:55.453 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T11:59:55.453 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T11:59:55.453 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T11:59:55.453 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T11:59:55.453 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T11:59:55.454 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T11:59:55.454 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T11:59:55.454 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:55.454 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.424+0000 7f494d6c6700 1 librados: starting msgr at 2021-05-25T11:59:55.454 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.424+0000 7f494d6c6700 1 librados: starting objecter 2021-05-25T11:59:55.455 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.424+0000 7f494d6c6700 1 Processor -- start 2021-05-25T11:59:55.455 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.424+0000 7f494d6c6700 1 -- start start 2021-05-25T11:59:55.455 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.424+0000 7f494d6c6700 1 librados: setting wanted keys 2021-05-25T11:59:55.455 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.424+0000 7f494d6c6700 1 librados: calling monclient init 2021-05-25T11:59:55.456 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.424+0000 7f494d6c6700 10 monclient: init 2021-05-25T11:59:55.456 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.424+0000 7f494d6c6700 10 monclient: _reopen_session rank -1 2021-05-25T11:59:55.456 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.424+0000 7f494d6c6700 10 monclient: _add_conns ranks=[1,2,0] 2021-05-25T11:59:55.456 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.424+0000 7f494d6c6700 10 monclient(hunting): picked mon.noname-b con 0x7f4948090990 addr v1:172.21.15.181:6789/0 2021-05-25T11:59:55.456 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.424+0000 7f494d6c6700 10 monclient(hunting): picked mon.noname-c con 0x7f4948090d70 addr v1:172.21.15.102:6790/0 2021-05-25T11:59:55.457 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.425+0000 7f494d6c6700 10 monclient(hunting): picked mon.noname-a con 0x7f494812e330 addr v1:172.21.15.102:6789/0 2021-05-25T11:59:55.457 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.425+0000 7f494d6c6700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4948091fe0 con 0x7f494812e330 2021-05-25T11:59:55.457 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.425+0000 7f494d6c6700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49481177c0 con 0x7f4948090990 2021-05-25T11:59:55.457 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.425+0000 7f494d6c6700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4948117a60 con 0x7f4948090d70 2021-05-25T11:59:55.457 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.425+0000 7f494d6c6700 10 monclient(hunting): _renew_subs 2021-05-25T11:59:55.458 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.425+0000 7f494d6c6700 10 monclient(hunting): authenticate will time out at 796.022032s 2021-05-25T11:59:55.458 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.425+0000 7f49467fc700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f4948090d70 0x7f494813fd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:53742/0 (socket says 172.21.15.102:53742) 2021-05-25T11:59:55.458 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.425+0000 7f49467fc700 1 -- 172.21.15.102:0/2862959632 learned_addr learned my addr 172.21.15.102:0/2862959632 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T11:59:55.458 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.425+0000 7f492effd700 10 client.?.objecter ms_handle_connect 0x7f494812e330 2021-05-25T11:59:55.458 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.425+0000 7f492effd700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:55.459 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.425+0000 7f492effd700 10 client.?.objecter ms_handle_connect 0x7f4948090d70 2021-05-25T11:59:55.459 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.425+0000 7f492effd700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:55.459 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.425+0000 7f492effd700 1 -- 172.21.15.102:0/2862959632 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2134880898 0 0) 0x7f4948091fe0 con 0x7f494812e330 2021-05-25T11:59:55.459 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.425+0000 7f492effd700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:55.459 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.425+0000 7f492effd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:55.460 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.425+0000 7f492effd700 10 monclient(hunting): my global_id is 4293 2021-05-25T11:59:55.460 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.425+0000 7f492effd700 1 -- 172.21.15.102:0/2862959632 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4928004c10 con 0x7f494812e330 2021-05-25T11:59:55.460 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.425+0000 7f492effd700 10 client.?.objecter ms_handle_connect 0x7f4948090990 2021-05-25T11:59:55.460 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.425+0000 7f492effd700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:55.460 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.425+0000 7f492effd700 1 -- 172.21.15.102:0/2862959632 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3073423841 0 0) 0x7f4948117a60 con 0x7f4948090d70 2021-05-25T11:59:55.461 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.425+0000 7f492effd700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:55.461 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.425+0000 7f492effd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:55.461 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.425+0000 7f492effd700 10 monclient(hunting): my global_id is 4286 2021-05-25T11:59:55.461 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.425+0000 7f492effd700 1 -- 172.21.15.102:0/2862959632 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4948091fe0 con 0x7f4948090d70 2021-05-25T11:59:55.462 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.425+0000 7f492effd700 1 -- 172.21.15.102:0/2862959632 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1551000610 0 0) 0x7f49481177c0 con 0x7f4948090990 2021-05-25T11:59:55.462 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.425+0000 7f492effd700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:55.462 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.425+0000 7f492effd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:55.462 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.425+0000 7f492effd700 10 monclient(hunting): my global_id is 4288 2021-05-25T11:59:55.462 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.425+0000 7f492effd700 1 -- 172.21.15.102:0/2862959632 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4948117a60 con 0x7f4948090990 2021-05-25T11:59:55.463 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.425+0000 7f492effd700 1 -- 172.21.15.102:0/2862959632 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3012975068 0 0) 0x7f4928004c10 con 0x7f494812e330 2021-05-25T11:59:55.463 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.425+0000 7f492effd700 1 -- 172.21.15.102:0/2862959632 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f49481177c0 con 0x7f494812e330 2021-05-25T11:59:55.463 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.425+0000 7f492effd700 1 -- 172.21.15.102:0/2862959632 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f4938001ed0 con 0x7f494812e330 2021-05-25T11:59:55.463 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.425+0000 7f492effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:55.463 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.425+0000 7f492effd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T11:59:55.464 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.425+0000 7f492effd700 10 monclient(hunting): dump: 2021-05-25T11:59:55.464 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:55.464 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:55.464 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:55.464 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:55.465 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:55.465 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:55.465 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:55.465 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:55.465 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:55.466 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:55.466 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.425+0000 7f492effd700 10 monclient(hunting): _finish_auth 0 2021-05-25T11:59:55.466 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.425+0000 7f492effd700 1 -- 172.21.15.102:0/2862959632 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 134719530 0 0) 0x7f4948091fe0 con 0x7f4948090d70 2021-05-25T11:59:55.466 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.425+0000 7f492effd700 1 -- 172.21.15.102:0/2862959632 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4928004c10 con 0x7f4948090d70 2021-05-25T11:59:55.467 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.425+0000 7f492effd700 1 -- 172.21.15.102:0/2862959632 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f493c0018d0 con 0x7f4948090d70 2021-05-25T11:59:55.467 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.425+0000 7f492effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:55.467 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.425+0000 7f492effd700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2021-05-25T11:59:55.467 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.425+0000 7f492effd700 10 monclient(hunting): dump: 2021-05-25T11:59:55.467 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:55.468 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:55.468 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:55.468 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:55.468 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:55.468 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:55.469 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:55.469 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:55.469 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:55.469 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:55.469 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.425+0000 7f492effd700 1 -- 172.21.15.102:0/2862959632 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4015849081 0 0) 0x7f4948117a60 con 0x7f4948090990 2021-05-25T11:59:55.470 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.425+0000 7f492effd700 1 -- 172.21.15.102:0/2862959632 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4948091fe0 con 0x7f4948090990 2021-05-25T11:59:55.470 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.425+0000 7f492effd700 1 -- 172.21.15.102:0/2862959632 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f4930001970 con 0x7f4948090990 2021-05-25T11:59:55.470 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.425+0000 7f492effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:55.470 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.425+0000 7f492effd700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2021-05-25T11:59:55.471 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.425+0000 7f492effd700 10 monclient(hunting): dump: 2021-05-25T11:59:55.471 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:55.471 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:55.471 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:55.471 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:55.472 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:55.472 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:55.472 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:55.472 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:55.472 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:55.473 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:55.473 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.425+0000 7f492effd700 1 -- 172.21.15.102:0/2862959632 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3879932480 0 0) 0x7f49481177c0 con 0x7f494812e330 2021-05-25T11:59:55.473 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.425+0000 7f492effd700 1 -- 172.21.15.102:0/2862959632 >> v1:172.21.15.102:6790/0 conn(0x7f4948090d70 legacy=0x7f494813fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:55.473 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.425+0000 7f492effd700 1 -- 172.21.15.102:0/2862959632 >> v1:172.21.15.181:6789/0 conn(0x7f4948090990 legacy=0x7f49481292b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:55.473 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.425+0000 7f492effd700 10 monclient: _finish_hunting 0 2021-05-25T11:59:55.474 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.425+0000 7f492effd700 1 monclient: found mon.a 2021-05-25T11:59:55.474 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.425+0000 7f492effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T11:59:55.474 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.425+0000 7f492effd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:55.474 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.425+0000 7f492effd700 1 -- 172.21.15.102:0/2862959632 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4948116670 con 0x7f494812e330 2021-05-25T11:59:55.474 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.425+0000 7f492effd700 10 monclient: _finish_auth 0 2021-05-25T11:59:55.475 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.425+0000 7f492effd700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T11:59:55.475 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.425+0000 7f494d6c6700 5 monclient: authenticate success, global_id 4293 2021-05-25T11:59:55.475 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.425+0000 7f494d6c6700 10 monclient: _renew_subs 2021-05-25T11:59:55.475 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.425+0000 7f494d6c6700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:55.475 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.425+0000 7f494d6c6700 1 -- 172.21.15.102:0/2862959632 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f494812a0a0 con 0x7f494812e330 2021-05-25T11:59:55.476 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.425+0000 7f494d6c6700 10 client.4293.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T11:59:55.476 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.425+0000 7f494d6c6700 10 monclient: _renew_subs 2021-05-25T11:59:55.476 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.425+0000 7f494d6c6700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:55.476 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.425+0000 7f494d6c6700 1 -- 172.21.15.102:0/2862959632 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4948129c40 con 0x7f494812e330 2021-05-25T11:59:55.477 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.425+0000 7f494d6c6700 1 librados: init done 2021-05-25T11:59:55.477 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.425+0000 7f492effd700 1 -- 172.21.15.102:0/2862959632 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4938004bb0 con 0x7f494812e330 2021-05-25T11:59:55.477 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.425+0000 7f492effd700 10 monclient: handle_config config(0 keys) v1 2021-05-25T11:59:55.477 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.425+0000 7f492effd700 1 -- 172.21.15.102:0/2862959632 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f4938004ec0 con 0x7f494812e330 2021-05-25T11:59:55.477 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.425+0000 7f492effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:55.478 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.425+0000 7f492effd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T11:59:55.478 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.425+0000 7f49457fa700 4 set_mon_vals no callback set 2021-05-25T11:59:55.478 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.425+0000 7f492effd700 10 monclient: dump: 2021-05-25T11:59:55.478 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:55.479 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:55.479 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:55.479 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:55.479 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:55.479 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:55.480 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:55.480 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:55.480 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:55.480 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:55.480 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.426+0000 7f492effd700 1 -- 172.21.15.102:0/2862959632 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f49380050a0 con 0x7f494812e330 2021-05-25T11:59:55.481 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.426+0000 7f494d6c6700 10 client.4293.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:55.481 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.426+0000 7f492effd700 1 -- 172.21.15.102:0/2862959632 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(15..15 src has 1..15) v4 ==== 3549+0+0 (unknown 3948819498 0 0) 0x7f493803d2c0 con 0x7f494812e330 2021-05-25T11:59:55.481 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.426+0000 7f494d6c6700 15 client.4293.objecter _session_command_op_assign -1 1 2021-05-25T11:59:55.481 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.426+0000 7f494d6c6700 20 client.4293.objecter _get_session osd=-1 returning homeless 2021-05-25T11:59:55.482 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.426+0000 7f492effd700 10 client.4293.objecter ms_dispatch 0x7f4948001550 osd_map(15..15 src has 1..15) v4 2021-05-25T11:59:55.482 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.426+0000 7f494d6c6700 10 client.4293.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T11:59:55.482 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.426+0000 7f494d6c6700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:55.482 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.426+0000 7f494d6c6700 1 -- 172.21.15.102:0/2862959632 --> v1:172.21.15.102:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f49200028c0 con 0x7f494812e330 2021-05-25T11:59:55.482 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.426+0000 7f492effd700 3 client.4293.objecter handle_osd_map got epochs [15,15] > 0 2021-05-25T11:59:55.483 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.426+0000 7f492effd700 3 client.4293.objecter handle_osd_map decoding full epoch 15 2021-05-25T11:59:55.483 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.426+0000 7f492effd700 10 client.4293.objecter checking command 1 2021-05-25T11:59:55.483 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.426+0000 7f492effd700 20 client.4293.objecter _get_session s=0x7f4928039d10 osd=5 3 2021-05-25T11:59:55.483 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.426+0000 7f492effd700 20 client.4293.objecter put_session s=0x7f4928039d10 osd=5 3 2021-05-25T11:59:55.483 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.426+0000 7f492effd700 15 client.4293.objecter _session_command_op_remove -1 1 2021-05-25T11:59:55.484 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.426+0000 7f492effd700 20 client.4293.objecter _get_session s=0x7f4928039d10 osd=5 3 2021-05-25T11:59:55.484 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.426+0000 7f492effd700 20 client.4293.objecter get_session s=0x7f4928039d10 osd=5 3 2021-05-25T11:59:55.484 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.426+0000 7f492effd700 15 client.4293.objecter _session_command_op_assign 5 1 2021-05-25T11:59:55.484 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.426+0000 7f492effd700 20 client.4293.objecter put_session s=0x7f4928039d10 osd=5 4 2021-05-25T11:59:55.485 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.426+0000 7f492effd700 10 client.4293.objecter _send_command 1 2021-05-25T11:59:55.485 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.426+0000 7f492effd700 1 -- 172.21.15.102:0/2862959632 --> v1:172.21.15.181:6808/30008 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f492803e090 con 0x7f492803a3c0 2021-05-25T11:59:55.485 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.426+0000 7f492effd700 20 client.4293.objecter dump_active .. 0 homeless 2021-05-25T11:59:55.485 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.426+0000 7f492effd700 1 -- 172.21.15.102:0/2862959632 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_get_version_reply(handle=1 version=15) v2 ==== 24+0+0 (unknown 2942983318 0 0) 0x7f49380132c0 con 0x7f494812e330 2021-05-25T11:59:55.485 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.426+0000 7f492effd700 10 monclient: handle_get_version_reply finishing 1 version 15 2021-05-25T11:59:55.486 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.426+0000 7f4946ffd700 10 monclient: get_auth_request con 0x7f4928033520 auth_method 0 2021-05-25T11:59:55.486 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.427+0000 7f49467fc700 10 monclient: get_auth_request con 0x7f492803a3c0 auth_method 0 2021-05-25T11:59:55.486 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.427+0000 7f492effd700 10 client.4293.objecter ms_handle_connect 0x7f4928033520 2021-05-25T11:59:55.486 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.427+0000 7f492effd700 10 client.4293.objecter ms_handle_connect 0x7f492803a3c0 2021-05-25T11:59:55.486 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.427+0000 7f492effd700 1 -- 172.21.15.102:0/2862959632 <== osd.5 v1:172.21.15.181:6808/30008 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23336 (unknown 0 0 3933607479) 0x7f492803e090 con 0x7f492803a3c0 2021-05-25T11:59:55.487 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.427+0000 7f492effd700 10 client.4293.objecter ms_dispatch 0x7f4948001550 command_reply(tid 1: 0 ) v1 2021-05-25T11:59:55.487 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.427+0000 7f492effd700 10 client.4293.objecter _finish_command 1 = system:0 2021-05-25T11:59:55.487 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.427+0000 7f492effd700 20 client.4293.objecter put_session s=0x7f4928039d10 osd=5 4 2021-05-25T11:59:55.487 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.427+0000 7f492effd700 15 client.4293.objecter _session_command_op_remove 5 1 2021-05-25T11:59:55.489 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.441+0000 7f494d6c6700 10 client.4293.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2021-05-25T11:59:55.489 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.441+0000 7f494d6c6700 15 client.4293.objecter _session_command_op_assign -1 2 2021-05-25T11:59:55.489 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.441+0000 7f494d6c6700 20 client.4293.objecter _get_session s=0x7f4928039d10 osd=5 3 2021-05-25T11:59:55.489 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.441+0000 7f494d6c6700 20 client.4293.objecter put_session s=0x7f4928039d10 osd=5 3 2021-05-25T11:59:55.490 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.441+0000 7f494d6c6700 20 client.4293.objecter _get_session s=0x7f4928039d10 osd=5 3 2021-05-25T11:59:55.490 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.441+0000 7f494d6c6700 15 client.4293.objecter _session_command_op_remove -1 2 2021-05-25T11:59:55.490 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.441+0000 7f494d6c6700 20 client.4293.objecter get_session s=0x7f4928039d10 osd=5 3 2021-05-25T11:59:55.490 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.441+0000 7f494d6c6700 15 client.4293.objecter _session_command_op_assign 5 2 2021-05-25T11:59:55.490 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.441+0000 7f494d6c6700 20 client.4293.objecter put_session s=0x7f4928039d10 osd=5 4 2021-05-25T11:59:55.491 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.441+0000 7f494d6c6700 10 client.4293.objecter _send_command 2 2021-05-25T11:59:55.491 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.441+0000 7f494d6c6700 1 -- 172.21.15.102:0/2862959632 --> v1:172.21.15.181:6808/30008 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f4920001150 con 0x7f492803a3c0 2021-05-25T11:59:55.491 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.442+0000 7f492effd700 1 -- 172.21.15.102:0/2862959632 <== osd.5 v1:172.21.15.181:6808/30008 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (unknown 0 0 1800444376) 0x7f4920001150 con 0x7f492803a3c0 2021-05-25T11:59:55.491 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.442+0000 7f492effd700 10 client.4293.objecter ms_dispatch 0x7f4948001550 command_reply(tid 2: 0 ) v1 2021-05-25T11:59:55.492 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.442+0000 7f492effd700 10 client.4293.objecter _finish_command 2 = system:0 2021-05-25T11:59:55.492 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.442+0000 7f492effd700 20 client.4293.objecter put_session s=0x7f4928039d10 osd=5 4 2021-05-25T11:59:55.492 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.442+0000 7f492effd700 15 client.4293.objecter _session_command_op_remove 5 2 2021-05-25T11:59:55.494 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.442+0000 7f494d6c6700 10 librados: watch_flush enter 2021-05-25T11:59:55.494 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.442+0000 7f494d6c6700 10 librados: watch_flush exit 2021-05-25T11:59:55.495 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.442+0000 7f494d6c6700 10 client.4293.objecter close_session for osd.5 2021-05-25T11:59:55.495 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.442+0000 7f494d6c6700 1 -- 172.21.15.102:0/2862959632 >> v1:172.21.15.181:6808/30008 conn(0x7f492803a3c0 legacy=0x7f492803c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:55.495 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.442+0000 7f494d6c6700 20 client.4293.objecter put_session s=0x7f4928039d10 osd=5 1 2021-05-25T11:59:55.495 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.442+0000 7f494d6c6700 20 client.4293.objecter shutdown clearing up homeless session... 2021-05-25T11:59:55.495 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.442+0000 7f494d6c6700 10 client.4293.objecter successfully canceled tick 2021-05-25T11:59:55.496 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.442+0000 7f494d6c6700 1 -- 172.21.15.102:0/2862959632 >> v1:172.21.15.181:6812/29582 conn(0x7f4928033520 legacy=0x7f49280359c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:55.496 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.442+0000 7f494d6c6700 10 monclient: shutdown 2021-05-25T11:59:55.496 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.442+0000 7f494d6c6700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T11:59:55.496 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.442+0000 7f494d6c6700 1 -- 172.21.15.102:0/2862959632 >> v1:172.21.15.102:6789/0 conn(0x7f494812e330 legacy=0x7f4948143460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:55.497 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.442+0000 7f49477fe700 1 -- 172.21.15.102:0/2862959632 reap_dead start 2021-05-25T11:59:55.497 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.442+0000 7f494d6c6700 1 -- 172.21.15.102:0/2862959632 shutdown_connections 2021-05-25T11:59:55.497 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.442+0000 7f494d6c6700 1 -- 172.21.15.102:0/2862959632 >> 172.21.15.102:0/2862959632 conn(0x7f494809b370 msgr2=0x7f4948068110 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T11:59:55.497 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.442+0000 7f494d6c6700 1 -- 172.21.15.102:0/2862959632 shutdown_connections 2021-05-25T11:59:55.497 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.442+0000 7f494d6c6700 1 -- 172.21.15.102:0/2862959632 wait complete. 2021-05-25T11:59:55.498 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.442+0000 7f494d6c6700 1 librados: shutdown 2021-05-25T11:59:55.498 INFO:teuthology.orchestra.run.smithi102.stdout:47244640259 2021-05-25T11:59:55.498 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2021-05-25T11:59:55.618 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.607+0000 7fcc496e3700 10 monclient: get_monmap_and_config 2021-05-25T11:59:55.618 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.607+0000 7fcc496e3700 10 monclient: build_initial_monmap 2021-05-25T11:59:55.618 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.607+0000 7fcc496e3700 10 monclient: monmap: 2021-05-25T11:59:55.619 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T11:59:55.619 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:55.619 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:55.608126+0000 2021-05-25T11:59:55.619 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:55.608126+0000 2021-05-25T11:59:55.619 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T11:59:55.620 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:55.620 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T11:59:55.620 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T11:59:55.620 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T11:59:55.621 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:55.621 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.607+0000 7fcc496e3700 1 Processor -- start 2021-05-25T11:59:55.621 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.607+0000 7fcc496e3700 1 -- start start 2021-05-25T11:59:55.621 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.607+0000 7fcc496e3700 10 monclient: init 2021-05-25T11:59:55.621 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.608+0000 7fcc496e3700 10 monclient: _reopen_session rank -1 2021-05-25T11:59:55.622 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.608+0000 7fcc496e3700 10 monclient: _add_conns ranks=[0,2,1] 2021-05-25T11:59:55.622 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.608+0000 7fcc496e3700 10 monclient(hunting): picked mon.noname-a con 0x7fcc44097fa0 addr v1:172.21.15.102:6789/0 2021-05-25T11:59:55.622 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.608+0000 7fcc496e3700 10 monclient(hunting): picked mon.noname-b con 0x7fcc44063b70 addr v1:172.21.15.102:6790/0 2021-05-25T11:59:55.622 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.608+0000 7fcc496e3700 10 monclient(hunting): picked mon.noname-c con 0x7fcc441325c0 addr v1:172.21.15.181:6789/0 2021-05-25T11:59:55.623 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.608+0000 7fcc496e3700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc44096b20 con 0x7fcc44097fa0 2021-05-25T11:59:55.623 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.608+0000 7fcc496e3700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc4409b460 con 0x7fcc441325c0 2021-05-25T11:59:55.623 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.608+0000 7fcc496e3700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc440964c0 con 0x7fcc44063b70 2021-05-25T11:59:55.623 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.608+0000 7fcc496e3700 10 monclient(hunting): _renew_subs 2021-05-25T11:59:55.624 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.608+0000 7fcc496e3700 10 monclient(hunting): authenticate will time out at 796.205308s 2021-05-25T11:59:55.624 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.608+0000 7fcc427fc700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fcc44063b70 0x7fcc44064020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:53754/0 (socket says 172.21.15.102:53754) 2021-05-25T11:59:55.624 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.608+0000 7fcc42ffd700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fcc44097fa0 0x7fcc44063460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:58874/0 (socket says 172.21.15.102:58874) 2021-05-25T11:59:55.624 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.608+0000 7fcc427fc700 1 -- 172.21.15.102:0/2902529137 learned_addr learned my addr 172.21.15.102:0/2902529137 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T11:59:55.624 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.608+0000 7fcc41ffb700 1 -- 172.21.15.102:0/2902529137 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2116116962 0 0) 0x7fcc44096b20 con 0x7fcc44097fa0 2021-05-25T11:59:55.625 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.608+0000 7fcc41ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:55.625 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.608+0000 7fcc41ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:55.625 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.608+0000 7fcc41ffb700 10 monclient(hunting): my global_id is 4296 2021-05-25T11:59:55.625 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.608+0000 7fcc41ffb700 1 -- 172.21.15.102:0/2902529137 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc28003270 con 0x7fcc44097fa0 2021-05-25T11:59:55.626 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.608+0000 7fcc41ffb700 1 -- 172.21.15.102:0/2902529137 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 151572305 0 0) 0x7fcc440964c0 con 0x7fcc44063b70 2021-05-25T11:59:55.626 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.608+0000 7fcc41ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:55.626 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.608+0000 7fcc41ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:55.626 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.608+0000 7fcc41ffb700 10 monclient(hunting): my global_id is 4289 2021-05-25T11:59:55.627 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.608+0000 7fcc41ffb700 1 -- 172.21.15.102:0/2902529137 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc44096b20 con 0x7fcc44063b70 2021-05-25T11:59:55.627 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.608+0000 7fcc41ffb700 1 -- 172.21.15.102:0/2902529137 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 244218358 0 0) 0x7fcc28003270 con 0x7fcc44097fa0 2021-05-25T11:59:55.627 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.608+0000 7fcc41ffb700 1 -- 172.21.15.102:0/2902529137 >> v1:172.21.15.102:6790/0 conn(0x7fcc44063b70 legacy=0x7fcc44064020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:55.627 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.608+0000 7fcc41ffb700 1 -- 172.21.15.102:0/2902529137 >> v1:172.21.15.181:6789/0 conn(0x7fcc441325c0 legacy=0x7fcc441349b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:55.627 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.608+0000 7fcc41ffb700 10 monclient: _finish_hunting 0 2021-05-25T11:59:55.628 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.608+0000 7fcc41ffb700 1 monclient: found mon.noname-a 2021-05-25T11:59:55.628 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.608+0000 7fcc41ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T11:59:55.628 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.608+0000 7fcc41ffb700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T11:59:55.628 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.608+0000 7fcc41ffb700 1 -- 172.21.15.102:0/2902529137 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc44096760 con 0x7fcc44097fa0 2021-05-25T11:59:55.629 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.608+0000 7fcc41ffb700 10 monclient: _finish_auth 0 2021-05-25T11:59:55.629 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.608+0000 7fcc41ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T11:59:55.629 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.608+0000 7fcc41ffb700 1 -- 172.21.15.102:0/2902529137 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fcc38001610 con 0x7fcc44097fa0 2021-05-25T11:59:55.629 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.608+0000 7fcc41ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:55.629 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.608+0000 7fcc41ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T11:59:55.630 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.608+0000 7fcc41ffb700 10 monclient: dump: 2021-05-25T11:59:55.630 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:55.630 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:55.630 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:55.630 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:55.631 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:55.631 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:55.631 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:55.631 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:55.632 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:55.632 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:55.632 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.608+0000 7fcc496e3700 5 monclient: authenticate success, global_id 4296 2021-05-25T11:59:55.632 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.608+0000 7fcc496e3700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T11:59:55.632 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.609+0000 7fcc41ffb700 1 -- 172.21.15.102:0/2902529137 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcc38001840 con 0x7fcc44097fa0 2021-05-25T11:59:55.633 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.609+0000 7fcc41ffb700 10 monclient: handle_config config(0 keys) v1 2021-05-25T11:59:55.633 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.609+0000 7fcc496e3700 10 monclient: get_monmap_and_config success 2021-05-25T11:59:55.633 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.609+0000 7fcc41ffb700 1 -- 172.21.15.102:0/2902529137 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fcc38001bd0 con 0x7fcc44097fa0 2021-05-25T11:59:55.633 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.609+0000 7fcc41ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:55.633 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.609+0000 7fcc496e3700 4 set_mon_vals no callback set 2021-05-25T11:59:55.634 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.609+0000 7fcc41ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T11:59:55.634 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.609+0000 7fcc41ffb700 10 monclient: dump: 2021-05-25T11:59:55.634 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:55.634 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:55.635 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:55.635 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:55.635 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:55.635 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:55.635 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:55.636 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:55.636 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:55.636 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:55.636 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.609+0000 7fcc496e3700 10 monclient: shutdown 2021-05-25T11:59:55.636 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.609+0000 7fcc496e3700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T11:59:55.637 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.609+0000 7fcc496e3700 1 -- 172.21.15.102:0/2902529137 >> v1:172.21.15.102:6789/0 conn(0x7fcc44097fa0 legacy=0x7fcc44063460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:55.637 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.609+0000 7fcc496e3700 1 -- 172.21.15.102:0/2902529137 shutdown_connections 2021-05-25T11:59:55.637 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.609+0000 7fcc496e3700 1 -- 172.21.15.102:0/2902529137 >> 172.21.15.102:0/2902529137 conn(0x7fcc4409ef50 msgr2=0x7fcc4409f360 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T11:59:55.637 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.609+0000 7fcc496e3700 1 -- 172.21.15.102:0/2902529137 shutdown_connections 2021-05-25T11:59:55.638 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.609+0000 7fcc496e3700 1 -- 172.21.15.102:0/2902529137 wait complete. 2021-05-25T11:59:55.638 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.610+0000 7fcc496e3700 10 monclient: build_initial_monmap 2021-05-25T11:59:55.638 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.610+0000 7fcc496e3700 10 monclient: monmap: 2021-05-25T11:59:55.638 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T11:59:55.638 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T11:59:55.639 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T11:59:55.639 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T11:59:55.639 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T11:59:55.639 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T11:59:55.639 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T11:59:55.640 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T11:59:55.640 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T11:59:55.640 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:55.640 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.610+0000 7fcc496e3700 1 librados: starting msgr at 2021-05-25T11:59:55.641 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.610+0000 7fcc496e3700 1 librados: starting objecter 2021-05-25T11:59:55.641 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.610+0000 7fcc496e3700 1 Processor -- start 2021-05-25T11:59:55.641 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.610+0000 7fcc496e3700 1 -- start start 2021-05-25T11:59:55.641 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.610+0000 7fcc496e3700 1 librados: setting wanted keys 2021-05-25T11:59:55.641 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.610+0000 7fcc496e3700 1 librados: calling monclient init 2021-05-25T11:59:55.642 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.610+0000 7fcc496e3700 10 monclient: init 2021-05-25T11:59:55.642 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.610+0000 7fcc496e3700 10 monclient: _reopen_session rank -1 2021-05-25T11:59:55.642 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.610+0000 7fcc496e3700 10 monclient: _add_conns ranks=[1,2,0] 2021-05-25T11:59:55.642 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.610+0000 7fcc496e3700 10 monclient(hunting): picked mon.noname-b con 0x7fcc44063b70 addr v1:172.21.15.181:6789/0 2021-05-25T11:59:55.642 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.610+0000 7fcc496e3700 10 monclient(hunting): picked mon.noname-c con 0x7fcc44097fa0 addr v1:172.21.15.102:6790/0 2021-05-25T11:59:55.643 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.610+0000 7fcc496e3700 10 monclient(hunting): picked mon.noname-a con 0x7fcc441325c0 addr v1:172.21.15.102:6789/0 2021-05-25T11:59:55.643 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.610+0000 7fcc496e3700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc440950b0 con 0x7fcc441325c0 2021-05-25T11:59:55.643 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.610+0000 7fcc496e3700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc4413b640 con 0x7fcc44063b70 2021-05-25T11:59:55.643 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.610+0000 7fcc496e3700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc4412c250 con 0x7fcc44097fa0 2021-05-25T11:59:55.644 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.610+0000 7fcc496e3700 10 monclient(hunting): _renew_subs 2021-05-25T11:59:55.644 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.610+0000 7fcc496e3700 10 monclient(hunting): authenticate will time out at 796.207902s 2021-05-25T11:59:55.644 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.610+0000 7fcc427fc700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fcc44097fa0 0x7fcc4412bb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:53760/0 (socket says 172.21.15.102:53760) 2021-05-25T11:59:55.644 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.610+0000 7fcc427fc700 1 -- 172.21.15.102:0/3170847177 learned_addr learned my addr 172.21.15.102:0/3170847177 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T11:59:55.645 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.611+0000 7fcc2effd700 10 client.?.objecter ms_handle_connect 0x7fcc44097fa0 2021-05-25T11:59:55.645 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.611+0000 7fcc2effd700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:55.645 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.611+0000 7fcc2effd700 10 client.?.objecter ms_handle_connect 0x7fcc441325c0 2021-05-25T11:59:55.645 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.611+0000 7fcc2effd700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:55.645 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.611+0000 7fcc2effd700 1 -- 172.21.15.102:0/3170847177 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3852946944 0 0) 0x7fcc4412c250 con 0x7fcc44097fa0 2021-05-25T11:59:55.646 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.611+0000 7fcc2effd700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:55.646 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.611+0000 7fcc2effd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:55.646 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.611+0000 7fcc2effd700 10 monclient(hunting): my global_id is 4292 2021-05-25T11:59:55.646 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.611+0000 7fcc2effd700 1 -- 172.21.15.102:0/3170847177 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc24004c10 con 0x7fcc44097fa0 2021-05-25T11:59:55.647 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.611+0000 7fcc2effd700 1 -- 172.21.15.102:0/3170847177 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 526363880 0 0) 0x7fcc440950b0 con 0x7fcc441325c0 2021-05-25T11:59:55.647 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.611+0000 7fcc2effd700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:55.647 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.611+0000 7fcc2effd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:55.647 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.611+0000 7fcc2effd700 10 monclient(hunting): my global_id is 4299 2021-05-25T11:59:55.647 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.611+0000 7fcc2effd700 1 -- 172.21.15.102:0/3170847177 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc4412c250 con 0x7fcc441325c0 2021-05-25T11:59:55.648 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.611+0000 7fcc2effd700 10 client.?.objecter ms_handle_connect 0x7fcc44063b70 2021-05-25T11:59:55.648 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.611+0000 7fcc2effd700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:55.648 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.611+0000 7fcc2effd700 1 -- 172.21.15.102:0/3170847177 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 531405274 0 0) 0x7fcc44096760 con 0x7fcc44063b70 2021-05-25T11:59:55.648 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.611+0000 7fcc2effd700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:55.648 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.611+0000 7fcc2effd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:55.649 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.611+0000 7fcc2effd700 10 monclient(hunting): my global_id is 4294 2021-05-25T11:59:55.649 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.611+0000 7fcc2effd700 1 -- 172.21.15.102:0/3170847177 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc440950b0 con 0x7fcc44063b70 2021-05-25T11:59:55.649 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.611+0000 7fcc2effd700 1 -- 172.21.15.102:0/3170847177 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3831888768 0 0) 0x7fcc24004c10 con 0x7fcc44097fa0 2021-05-25T11:59:55.649 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.611+0000 7fcc2effd700 1 -- 172.21.15.102:0/3170847177 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcc44096760 con 0x7fcc44097fa0 2021-05-25T11:59:55.650 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.611+0000 7fcc2effd700 1 -- 172.21.15.102:0/3170847177 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1549532298 0 0) 0x7fcc4412c250 con 0x7fcc441325c0 2021-05-25T11:59:55.650 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.611+0000 7fcc2effd700 1 -- 172.21.15.102:0/3170847177 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcc24004c10 con 0x7fcc441325c0 2021-05-25T11:59:55.650 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.611+0000 7fcc2effd700 1 -- 172.21.15.102:0/3170847177 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fcc300019f0 con 0x7fcc44097fa0 2021-05-25T11:59:55.650 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.611+0000 7fcc2effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:55.651 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.611+0000 7fcc2effd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T11:59:55.651 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.611+0000 7fcc2effd700 10 monclient(hunting): dump: 2021-05-25T11:59:55.651 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:55.651 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:55.651 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:55.652 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:55.652 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:55.652 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:55.652 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:55.652 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:55.653 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:55.653 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:55.653 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.611+0000 7fcc2effd700 10 monclient(hunting): _finish_auth 0 2021-05-25T11:59:55.653 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.611+0000 7fcc2effd700 1 -- 172.21.15.102:0/3170847177 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fcc34001b00 con 0x7fcc441325c0 2021-05-25T11:59:55.653 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.611+0000 7fcc2effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:55.654 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.611+0000 7fcc2effd700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2021-05-25T11:59:55.654 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.611+0000 7fcc2effd700 10 monclient(hunting): dump: 2021-05-25T11:59:55.654 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:55.654 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:55.655 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:55.655 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:55.655 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:55.655 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:55.655 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:55.656 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:55.656 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:55.656 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:55.656 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.611+0000 7fcc2effd700 1 -- 172.21.15.102:0/3170847177 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1530013550 0 0) 0x7fcc24004c10 con 0x7fcc441325c0 2021-05-25T11:59:55.656 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.611+0000 7fcc2effd700 1 -- 172.21.15.102:0/3170847177 >> v1:172.21.15.102:6790/0 conn(0x7fcc44097fa0 legacy=0x7fcc4412bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:55.657 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.611+0000 7fcc2effd700 1 -- 172.21.15.102:0/3170847177 >> v1:172.21.15.181:6789/0 conn(0x7fcc44063b70 legacy=0x7fcc4413af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:55.657 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.611+0000 7fcc2effd700 10 monclient: _finish_hunting 0 2021-05-25T11:59:55.657 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.611+0000 7fcc2effd700 1 monclient: found mon.a 2021-05-25T11:59:55.657 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.611+0000 7fcc2effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T11:59:55.658 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.611+0000 7fcc2effd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:55.658 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.611+0000 7fcc2effd700 1 -- 172.21.15.102:0/3170847177 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc441162b0 con 0x7fcc441325c0 2021-05-25T11:59:55.658 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.611+0000 7fcc2effd700 10 monclient: _finish_auth 0 2021-05-25T11:59:55.658 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.611+0000 7fcc2effd700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T11:59:55.658 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.611+0000 7fcc496e3700 5 monclient: authenticate success, global_id 4299 2021-05-25T11:59:55.659 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.611+0000 7fcc496e3700 10 monclient: _renew_subs 2021-05-25T11:59:55.659 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.611+0000 7fcc496e3700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:55.659 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.611+0000 7fcc496e3700 1 -- 172.21.15.102:0/3170847177 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc4413bc70 con 0x7fcc441325c0 2021-05-25T11:59:55.659 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.611+0000 7fcc2effd700 1 -- 172.21.15.102:0/3170847177 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcc34001ca0 con 0x7fcc441325c0 2021-05-25T11:59:55.659 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.611+0000 7fcc2effd700 10 monclient: handle_config config(0 keys) v1 2021-05-25T11:59:55.660 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.611+0000 7fcc496e3700 10 client.4299.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T11:59:55.660 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.611+0000 7fcc496e3700 10 monclient: _renew_subs 2021-05-25T11:59:55.660 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.611+0000 7fcc2effd700 1 -- 172.21.15.102:0/3170847177 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fcc34001fb0 con 0x7fcc441325c0 2021-05-25T11:59:55.660 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.611+0000 7fcc496e3700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:55.660 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.611+0000 7fcc496e3700 1 -- 172.21.15.102:0/3170847177 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc44094990 con 0x7fcc441325c0 2021-05-25T11:59:55.661 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.611+0000 7fcc417fa700 4 set_mon_vals no callback set 2021-05-25T11:59:55.661 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.611+0000 7fcc496e3700 1 librados: init done 2021-05-25T11:59:55.661 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.611+0000 7fcc2effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:55.661 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.611+0000 7fcc2effd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T11:59:55.662 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.611+0000 7fcc2effd700 10 monclient: dump: 2021-05-25T11:59:55.662 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:55.662 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:55.662 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:55.662 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:55.663 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:55.663 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:55.663 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:55.663 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:55.663 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:55.664 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:55.664 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.612+0000 7fcc2effd700 1 -- 172.21.15.102:0/3170847177 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7fcc340021d0 con 0x7fcc441325c0 2021-05-25T11:59:55.664 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.612+0000 7fcc2effd700 1 -- 172.21.15.102:0/3170847177 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(15..15 src has 1..15) v4 ==== 3549+0+0 (unknown 3948819498 0 0) 0x7fcc3403a8f0 con 0x7fcc441325c0 2021-05-25T11:59:55.664 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.612+0000 7fcc2effd700 10 client.4299.objecter ms_dispatch 0x7fcc44001550 osd_map(15..15 src has 1..15) v4 2021-05-25T11:59:55.664 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.612+0000 7fcc2effd700 3 client.4299.objecter handle_osd_map got epochs [15,15] > 0 2021-05-25T11:59:55.665 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.612+0000 7fcc2effd700 3 client.4299.objecter handle_osd_map decoding full epoch 15 2021-05-25T11:59:55.665 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.612+0000 7fcc2effd700 20 client.4299.objecter dump_active .. 0 homeless 2021-05-25T11:59:55.665 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.612+0000 7fcc496e3700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:55.665 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.612+0000 7fcc42ffd700 10 monclient: get_auth_request con 0x7fcc24033260 auth_method 0 2021-05-25T11:59:55.665 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.612+0000 7fcc496e3700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:55.666 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.612+0000 7fcc496e3700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:55.666 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.612+0000 7fcc496e3700 1 -- 172.21.15.102:0/3170847177 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc44059730 con 0x7fcc441325c0 2021-05-25T11:59:55.666 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.612+0000 7fcc2effd700 10 client.4299.objecter ms_handle_connect 0x7fcc24033260 2021-05-25T11:59:55.666 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.616+0000 7fcc2effd700 1 -- 172.21.15.102:0/3170847177 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7fcc34010830 con 0x7fcc441325c0 2021-05-25T11:59:55.667 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.616+0000 7fcc2effd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:55.667 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.616+0000 7fcc2effd700 10 monclient: _finish_command 1 = system:0 2021-05-25T11:59:55.767 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.765+0000 7fcc496e3700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2021-05-25T11:59:55.767 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.765+0000 7fcc496e3700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2021-05-25T11:59:55.768 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.765+0000 7fcc496e3700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:55.768 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.765+0000 7fcc496e3700 1 -- 172.21.15.102:0/3170847177 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fcc4406c9a0 con 0x7fcc441325c0 2021-05-25T11:59:55.768 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.765+0000 7fcc2effd700 1 -- 172.21.15.102:0/3170847177 <== mon.0 v1:172.21.15.102:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (unknown 574334944 0 555833778) 0x7fcc34010420 con 0x7fcc441325c0 2021-05-25T11:59:55.768 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.765+0000 7fcc2effd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2021-05-25T11:59:55.768 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.765+0000 7fcc2effd700 10 monclient: _finish_command 2 = system:0 2021-05-25T11:59:55.769 INFO:teuthology.orchestra.run.smithi102.stdout:47244640258 2021-05-25T11:59:55.770 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.767+0000 7fcc496e3700 10 librados: watch_flush enter 2021-05-25T11:59:55.771 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.767+0000 7fcc496e3700 10 librados: watch_flush exit 2021-05-25T11:59:55.771 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.767+0000 7fcc496e3700 20 client.4299.objecter shutdown clearing up homeless session... 2021-05-25T11:59:55.771 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.767+0000 7fcc496e3700 10 client.4299.objecter successfully canceled tick 2021-05-25T11:59:55.771 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.767+0000 7fcc496e3700 1 -- 172.21.15.102:0/3170847177 >> v1:172.21.15.181:6812/29582 conn(0x7fcc24033260 legacy=0x7fcc24035720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:55.771 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.767+0000 7fcc496e3700 10 monclient: shutdown 2021-05-25T11:59:55.772 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.767+0000 7fcc496e3700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T11:59:55.772 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.767+0000 7fcc496e3700 1 -- 172.21.15.102:0/3170847177 >> v1:172.21.15.102:6789/0 conn(0x7fcc441325c0 legacy=0x7fcc441432e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:55.772 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.767+0000 7fcc496e3700 1 -- 172.21.15.102:0/3170847177 shutdown_connections 2021-05-25T11:59:55.772 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.767+0000 7fcc496e3700 1 -- 172.21.15.102:0/3170847177 >> 172.21.15.102:0/3170847177 conn(0x7fcc4409ef50 msgr2=0x7fcc44096a00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T11:59:55.772 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.767+0000 7fcc496e3700 1 -- 172.21.15.102:0/3170847177 shutdown_connections 2021-05-25T11:59:55.773 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.767+0000 7fcc496e3700 1 -- 172.21.15.102:0/3170847177 wait complete. 2021-05-25T11:59:55.773 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:55.767+0000 7fcc496e3700 1 librados: shutdown 2021-05-25T11:59:55.779 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640258 for osd.0 2021-05-25T11:59:56.780 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2021-05-25T11:59:56.905 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.903+0000 7ffb55923700 10 monclient: get_monmap_and_config 2021-05-25T11:59:56.906 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.903+0000 7ffb55923700 10 monclient: build_initial_monmap 2021-05-25T11:59:56.906 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.903+0000 7ffb55923700 10 monclient: monmap: 2021-05-25T11:59:56.906 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T11:59:56.906 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:56.907 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:56.905144+0000 2021-05-25T11:59:56.907 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:56.905144+0000 2021-05-25T11:59:56.907 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T11:59:56.907 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:56.907 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T11:59:56.908 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T11:59:56.908 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T11:59:56.908 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:56.918 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.903+0000 7ffb55923700 1 Processor -- start 2021-05-25T11:59:56.919 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.903+0000 7ffb55923700 1 -- start start 2021-05-25T11:59:56.919 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.903+0000 7ffb55923700 10 monclient: init 2021-05-25T11:59:56.919 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.904+0000 7ffb55923700 10 monclient: _reopen_session rank -1 2021-05-25T11:59:56.919 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.904+0000 7ffb55923700 10 monclient: _add_conns ranks=[0,1,2] 2021-05-25T11:59:56.920 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.904+0000 7ffb55923700 10 monclient(hunting): picked mon.noname-a con 0x7ffb50090990 addr v1:172.21.15.102:6789/0 2021-05-25T11:59:56.920 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.904+0000 7ffb55923700 10 monclient(hunting): picked mon.noname-c con 0x7ffb50090d70 addr v1:172.21.15.181:6789/0 2021-05-25T11:59:56.920 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.904+0000 7ffb55923700 10 monclient(hunting): picked mon.noname-b con 0x7ffb5012e330 addr v1:172.21.15.102:6790/0 2021-05-25T11:59:56.920 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.904+0000 7ffb55923700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb50096fa0 con 0x7ffb50090990 2021-05-25T11:59:56.920 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.904+0000 7ffb55923700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb500926a0 con 0x7ffb50090d70 2021-05-25T11:59:56.921 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.904+0000 7ffb55923700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb50092940 con 0x7ffb5012e330 2021-05-25T11:59:56.921 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.904+0000 7ffb55923700 10 monclient(hunting): _renew_subs 2021-05-25T11:59:56.921 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.904+0000 7ffb55923700 10 monclient(hunting): authenticate will time out at 797.502426s 2021-05-25T11:59:56.921 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.904+0000 7ffb4f7fe700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7ffb5012e330 0x7ffb50130720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:53770/0 (socket says 172.21.15.102:53770) 2021-05-25T11:59:56.922 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.904+0000 7ffb4effd700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7ffb50090990 0x7ffb50093240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:58888/0 (socket says 172.21.15.102:58888) 2021-05-25T11:59:56.922 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.904+0000 7ffb4f7fe700 1 -- 172.21.15.102:0/896660587 learned_addr learned my addr 172.21.15.102:0/896660587 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T11:59:56.922 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.904+0000 7ffb4dffb700 1 -- 172.21.15.102:0/896660587 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1381458319 0 0) 0x7ffb50092940 con 0x7ffb5012e330 2021-05-25T11:59:56.922 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.904+0000 7ffb4dffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:56.923 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.904+0000 7ffb4dffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:56.923 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.904+0000 7ffb4dffb700 10 monclient(hunting): my global_id is 4295 2021-05-25T11:59:56.923 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.904+0000 7ffb4dffb700 1 -- 172.21.15.102:0/896660587 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb34003270 con 0x7ffb5012e330 2021-05-25T11:59:56.923 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.904+0000 7ffb4dffb700 1 -- 172.21.15.102:0/896660587 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1200763645 0 0) 0x7ffb50096fa0 con 0x7ffb50090990 2021-05-25T11:59:56.923 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.904+0000 7ffb4dffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:56.924 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.904+0000 7ffb4dffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:56.924 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.904+0000 7ffb4dffb700 10 monclient(hunting): my global_id is 4302 2021-05-25T11:59:56.924 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.904+0000 7ffb4dffb700 1 -- 172.21.15.102:0/896660587 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb50092940 con 0x7ffb50090990 2021-05-25T11:59:56.924 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.905+0000 7ffb4dffb700 1 -- 172.21.15.102:0/896660587 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2155830820 0 0) 0x7ffb50092940 con 0x7ffb50090990 2021-05-25T11:59:56.925 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.905+0000 7ffb4dffb700 1 -- 172.21.15.102:0/896660587 >> v1:172.21.15.102:6790/0 conn(0x7ffb5012e330 legacy=0x7ffb50130720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:56.925 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.905+0000 7ffb4dffb700 1 -- 172.21.15.102:0/896660587 >> v1:172.21.15.181:6789/0 conn(0x7ffb50090d70 legacy=0x7ffb50091200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:56.925 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.905+0000 7ffb4dffb700 10 monclient: _finish_hunting 0 2021-05-25T11:59:56.925 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.905+0000 7ffb4dffb700 1 monclient: found mon.noname-a 2021-05-25T11:59:56.925 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.905+0000 7ffb4dffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T11:59:56.926 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.905+0000 7ffb4dffb700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T11:59:56.926 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.905+0000 7ffb4dffb700 1 -- 172.21.15.102:0/896660587 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb50092be0 con 0x7ffb50090990 2021-05-25T11:59:56.926 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.905+0000 7ffb4dffb700 10 monclient: _finish_auth 0 2021-05-25T11:59:56.926 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.905+0000 7ffb4dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T11:59:56.927 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.905+0000 7ffb4dffb700 1 -- 172.21.15.102:0/896660587 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7ffb3c001610 con 0x7ffb50090990 2021-05-25T11:59:56.927 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.905+0000 7ffb55923700 5 monclient: authenticate success, global_id 4302 2021-05-25T11:59:56.927 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.905+0000 7ffb55923700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T11:59:56.927 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.905+0000 7ffb4dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:56.928 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.905+0000 7ffb4dffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T11:59:56.928 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.905+0000 7ffb4dffb700 10 monclient: dump: 2021-05-25T11:59:56.928 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:56.928 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:56.928 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:56.929 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:56.929 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:56.929 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:56.929 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:56.930 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:56.930 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:56.930 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:56.930 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.905+0000 7ffb55923700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T11:59:56.930 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.905+0000 7ffb4dffb700 1 -- 172.21.15.102:0/896660587 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffb3c001830 con 0x7ffb50090990 2021-05-25T11:59:56.931 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.905+0000 7ffb4dffb700 10 monclient: handle_config config(0 keys) v1 2021-05-25T11:59:56.931 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.905+0000 7ffb55923700 10 monclient: get_monmap_and_config success 2021-05-25T11:59:56.931 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.905+0000 7ffb4dffb700 1 -- 172.21.15.102:0/896660587 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7ffb3c001b40 con 0x7ffb50090990 2021-05-25T11:59:56.931 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.905+0000 7ffb4dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:56.932 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.905+0000 7ffb55923700 4 set_mon_vals no callback set 2021-05-25T11:59:56.932 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.905+0000 7ffb4dffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T11:59:56.932 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.905+0000 7ffb4dffb700 10 monclient: dump: 2021-05-25T11:59:56.932 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:56.932 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:56.933 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:56.933 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:56.933 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:56.933 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:56.934 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:56.934 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:56.934 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:56.934 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:56.934 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.905+0000 7ffb55923700 10 monclient: shutdown 2021-05-25T11:59:56.935 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.905+0000 7ffb55923700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T11:59:56.935 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.905+0000 7ffb55923700 1 -- 172.21.15.102:0/896660587 >> v1:172.21.15.102:6789/0 conn(0x7ffb50090990 legacy=0x7ffb50093240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:56.935 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.905+0000 7ffb55923700 1 -- 172.21.15.102:0/896660587 shutdown_connections 2021-05-25T11:59:56.935 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.905+0000 7ffb55923700 1 -- 172.21.15.102:0/896660587 >> 172.21.15.102:0/896660587 conn(0x7ffb5009b370 msgr2=0x7ffb50097430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T11:59:56.936 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.905+0000 7ffb55923700 1 -- 172.21.15.102:0/896660587 shutdown_connections 2021-05-25T11:59:56.936 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.905+0000 7ffb55923700 1 -- 172.21.15.102:0/896660587 wait complete. 2021-05-25T11:59:56.936 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.906+0000 7ffb55923700 10 monclient: build_initial_monmap 2021-05-25T11:59:56.936 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.906+0000 7ffb55923700 10 monclient: monmap: 2021-05-25T11:59:56.936 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T11:59:56.937 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T11:59:56.937 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T11:59:56.937 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T11:59:56.937 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T11:59:56.937 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T11:59:56.938 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T11:59:56.938 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T11:59:56.938 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T11:59:56.938 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:56.939 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.906+0000 7ffb55923700 1 librados: starting msgr at 2021-05-25T11:59:56.939 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.906+0000 7ffb55923700 1 librados: starting objecter 2021-05-25T11:59:56.939 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.906+0000 7ffb55923700 1 Processor -- start 2021-05-25T11:59:56.939 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.906+0000 7ffb55923700 1 -- start start 2021-05-25T11:59:56.939 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.906+0000 7ffb55923700 1 librados: setting wanted keys 2021-05-25T11:59:56.940 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.906+0000 7ffb55923700 1 librados: calling monclient init 2021-05-25T11:59:56.940 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.906+0000 7ffb55923700 10 monclient: init 2021-05-25T11:59:56.940 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.907+0000 7ffb55923700 10 monclient: _reopen_session rank -1 2021-05-25T11:59:56.940 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.907+0000 7ffb55923700 10 monclient: _add_conns ranks=[0,1,2] 2021-05-25T11:59:56.941 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.907+0000 7ffb55923700 10 monclient(hunting): picked mon.noname-a con 0x7ffb50090990 addr v1:172.21.15.102:6789/0 2021-05-25T11:59:56.941 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.907+0000 7ffb55923700 10 monclient(hunting): picked mon.noname-b con 0x7ffb50090d70 addr v1:172.21.15.181:6789/0 2021-05-25T11:59:56.941 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.907+0000 7ffb55923700 10 monclient(hunting): picked mon.noname-c con 0x7ffb5012e330 addr v1:172.21.15.102:6790/0 2021-05-25T11:59:56.941 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.907+0000 7ffb55923700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb50112700 con 0x7ffb50090990 2021-05-25T11:59:56.941 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.907+0000 7ffb55923700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb501129a0 con 0x7ffb50090d70 2021-05-25T11:59:56.942 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.907+0000 7ffb55923700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb50112c40 con 0x7ffb5012e330 2021-05-25T11:59:56.942 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.907+0000 7ffb55923700 10 monclient(hunting): _renew_subs 2021-05-25T11:59:56.942 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.907+0000 7ffb55923700 10 monclient(hunting): authenticate will time out at 797.505330s 2021-05-25T11:59:56.942 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.907+0000 7ffb4effd700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7ffb50090990 0x7ffb50126210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:58894/0 (socket says 172.21.15.102:58894) 2021-05-25T11:59:56.942 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.907+0000 7ffb4effd700 1 -- 172.21.15.102:0/802348627 learned_addr learned my addr 172.21.15.102:0/802348627 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T11:59:56.943 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.907+0000 7ffb3affd700 10 client.?.objecter ms_handle_connect 0x7ffb5012e330 2021-05-25T11:59:56.943 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.907+0000 7ffb3affd700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:56.943 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.907+0000 7ffb3affd700 10 client.?.objecter ms_handle_connect 0x7ffb50090990 2021-05-25T11:59:56.943 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.907+0000 7ffb3affd700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:56.944 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.907+0000 7ffb3affd700 1 -- 172.21.15.102:0/802348627 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2654656219 0 0) 0x7ffb50112c40 con 0x7ffb5012e330 2021-05-25T11:59:56.944 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.907+0000 7ffb3affd700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:56.944 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.907+0000 7ffb3affd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:56.944 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.907+0000 7ffb3affd700 10 monclient(hunting): my global_id is 4298 2021-05-25T11:59:56.944 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.907+0000 7ffb3affd700 1 -- 172.21.15.102:0/802348627 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb30004c10 con 0x7ffb5012e330 2021-05-25T11:59:56.945 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.907+0000 7ffb3affd700 10 client.?.objecter ms_handle_connect 0x7ffb50090d70 2021-05-25T11:59:56.945 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.907+0000 7ffb3affd700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:56.945 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.907+0000 7ffb3affd700 1 -- 172.21.15.102:0/802348627 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3067780311 0 0) 0x7ffb50112700 con 0x7ffb50090990 2021-05-25T11:59:56.945 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.907+0000 7ffb3affd700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:56.946 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.907+0000 7ffb3affd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:56.946 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.907+0000 7ffb3affd700 10 monclient(hunting): my global_id is 4305 2021-05-25T11:59:56.946 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.907+0000 7ffb3affd700 1 -- 172.21.15.102:0/802348627 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb50112c40 con 0x7ffb50090990 2021-05-25T11:59:56.946 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.907+0000 7ffb3affd700 1 -- 172.21.15.102:0/802348627 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2826402749 0 0) 0x7ffb501129a0 con 0x7ffb50090d70 2021-05-25T11:59:56.946 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.907+0000 7ffb3affd700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:56.947 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.907+0000 7ffb3affd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:56.947 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.907+0000 7ffb3affd700 10 monclient(hunting): my global_id is 4300 2021-05-25T11:59:56.947 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.907+0000 7ffb3affd700 1 -- 172.21.15.102:0/802348627 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb50112700 con 0x7ffb50090d70 2021-05-25T11:59:56.947 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.907+0000 7ffb3affd700 1 -- 172.21.15.102:0/802348627 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3574426916 0 0) 0x7ffb30004c10 con 0x7ffb5012e330 2021-05-25T11:59:56.948 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.907+0000 7ffb3affd700 1 -- 172.21.15.102:0/802348627 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffb501129a0 con 0x7ffb5012e330 2021-05-25T11:59:56.948 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.907+0000 7ffb3affd700 1 -- 172.21.15.102:0/802348627 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7ffb40001d70 con 0x7ffb5012e330 2021-05-25T11:59:56.948 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.907+0000 7ffb3affd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:56.948 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.907+0000 7ffb3affd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T11:59:56.948 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.907+0000 7ffb3affd700 10 monclient(hunting): dump: 2021-05-25T11:59:56.949 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:56.949 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:56.949 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:56.949 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:56.950 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:56.950 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:56.950 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:56.950 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:56.950 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:56.951 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:56.951 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.907+0000 7ffb3affd700 10 monclient(hunting): _finish_auth 0 2021-05-25T11:59:56.951 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.907+0000 7ffb3affd700 1 -- 172.21.15.102:0/802348627 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2775814671 0 0) 0x7ffb50112c40 con 0x7ffb50090990 2021-05-25T11:59:56.951 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.907+0000 7ffb3affd700 1 -- 172.21.15.102:0/802348627 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffb30004c10 con 0x7ffb50090990 2021-05-25T11:59:56.951 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.907+0000 7ffb3affd700 1 -- 172.21.15.102:0/802348627 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7ffb3c001510 con 0x7ffb50090990 2021-05-25T11:59:56.952 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.907+0000 7ffb3affd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:56.952 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.907+0000 7ffb3affd700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2021-05-25T11:59:56.952 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.907+0000 7ffb3affd700 10 monclient(hunting): dump: 2021-05-25T11:59:56.952 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:56.953 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:56.953 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:56.953 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:56.953 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:56.953 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:56.954 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:56.954 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:56.954 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:56.954 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:56.955 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.908+0000 7ffb3affd700 1 -- 172.21.15.102:0/802348627 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1155160143 0 0) 0x7ffb50112700 con 0x7ffb50090d70 2021-05-25T11:59:56.955 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.908+0000 7ffb3affd700 1 -- 172.21.15.102:0/802348627 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffb50112c40 con 0x7ffb50090d70 2021-05-25T11:59:56.955 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.908+0000 7ffb3affd700 1 -- 172.21.15.102:0/802348627 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7ffb44001880 con 0x7ffb50090d70 2021-05-25T11:59:56.955 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.908+0000 7ffb3affd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:56.956 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.908+0000 7ffb3affd700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2021-05-25T11:59:56.956 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.908+0000 7ffb3affd700 10 monclient(hunting): dump: 2021-05-25T11:59:56.956 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:56.956 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:56.956 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:56.957 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:56.957 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:56.957 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:56.957 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:56.957 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:56.958 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:56.958 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:56.958 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.908+0000 7ffb3affd700 1 -- 172.21.15.102:0/802348627 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3116643998 0 0) 0x7ffb501129a0 con 0x7ffb5012e330 2021-05-25T11:59:56.958 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.908+0000 7ffb3affd700 1 -- 172.21.15.102:0/802348627 >> v1:172.21.15.181:6789/0 conn(0x7ffb50090d70 legacy=0x7ffb50138960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:56.959 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.908+0000 7ffb3affd700 1 -- 172.21.15.102:0/802348627 >> v1:172.21.15.102:6789/0 conn(0x7ffb50090990 legacy=0x7ffb50126210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:56.959 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.908+0000 7ffb3affd700 10 monclient: _finish_hunting 0 2021-05-25T11:59:56.959 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.908+0000 7ffb3affd700 1 monclient: found mon.c 2021-05-25T11:59:56.959 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.908+0000 7ffb3affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T11:59:56.959 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.908+0000 7ffb3affd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T11:59:56.960 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.908+0000 7ffb3affd700 1 -- 172.21.15.102:0/802348627 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb50126920 con 0x7ffb5012e330 2021-05-25T11:59:56.960 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.908+0000 7ffb3affd700 10 monclient: _finish_auth 0 2021-05-25T11:59:56.960 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.908+0000 7ffb3affd700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T11:59:56.960 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.908+0000 7ffb55923700 5 monclient: authenticate success, global_id 4298 2021-05-25T11:59:56.961 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.908+0000 7ffb55923700 10 monclient: _renew_subs 2021-05-25T11:59:56.961 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.908+0000 7ffb55923700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T11:59:56.961 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.908+0000 7ffb55923700 1 -- 172.21.15.102:0/802348627 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb50068920 con 0x7ffb5012e330 2021-05-25T11:59:56.961 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.908+0000 7ffb55923700 10 client.4298.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T11:59:56.961 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.908+0000 7ffb55923700 10 monclient: _renew_subs 2021-05-25T11:59:56.962 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.908+0000 7ffb55923700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T11:59:56.962 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.908+0000 7ffb55923700 1 -- 172.21.15.102:0/802348627 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb50139230 con 0x7ffb5012e330 2021-05-25T11:59:56.962 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.908+0000 7ffb55923700 1 librados: init done 2021-05-25T11:59:56.962 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.908+0000 7ffb3affd700 1 -- 172.21.15.102:0/802348627 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffb40001fe0 con 0x7ffb5012e330 2021-05-25T11:59:56.962 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.908+0000 7ffb3affd700 10 monclient: handle_config config(0 keys) v1 2021-05-25T11:59:56.963 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.908+0000 7ffb3affd700 1 -- 172.21.15.102:0/802348627 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7ffb40004d00 con 0x7ffb5012e330 2021-05-25T11:59:56.963 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.908+0000 7ffb3affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:56.963 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.908+0000 7ffb3affd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T11:59:56.963 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.908+0000 7ffb4d7fa700 4 set_mon_vals no callback set 2021-05-25T11:59:56.964 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.908+0000 7ffb3affd700 10 monclient: dump: 2021-05-25T11:59:56.964 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:56.964 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:56.964 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:56.964 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:56.965 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:56.965 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:56.965 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:56.965 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:56.965 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:56.966 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:56.966 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.909+0000 7ffb3affd700 1 -- 172.21.15.102:0/802348627 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7ffb4000e020 con 0x7ffb5012e330 2021-05-25T11:59:56.966 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.909+0000 7ffb55923700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:56.966 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.909+0000 7ffb55923700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:56.967 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.909+0000 7ffb55923700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T11:59:56.967 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.909+0000 7ffb55923700 1 -- 172.21.15.102:0/802348627 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb28004db0 con 0x7ffb5012e330 2021-05-25T11:59:56.967 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.909+0000 7ffb3affd700 1 -- 172.21.15.102:0/802348627 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(15..15 src has 1..15) v4 ==== 3549+0+0 (unknown 3948819498 0 0) 0x7ffb4003d040 con 0x7ffb5012e330 2021-05-25T11:59:56.967 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.909+0000 7ffb3affd700 10 client.4298.objecter ms_dispatch 0x7ffb50001550 osd_map(15..15 src has 1..15) v4 2021-05-25T11:59:56.967 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.909+0000 7ffb3affd700 3 client.4298.objecter handle_osd_map got epochs [15,15] > 0 2021-05-25T11:59:56.968 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.909+0000 7ffb3affd700 3 client.4298.objecter handle_osd_map decoding full epoch 15 2021-05-25T11:59:56.968 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.909+0000 7ffb3affd700 20 client.4298.objecter dump_active .. 0 homeless 2021-05-25T11:59:56.968 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.909+0000 7ffb4effd700 10 monclient: get_auth_request con 0x7ffb30033510 auth_method 0 2021-05-25T11:59:56.968 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.909+0000 7ffb3affd700 10 client.4298.objecter ms_handle_connect 0x7ffb30033510 2021-05-25T11:59:56.969 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.913+0000 7ffb3affd700 1 -- 172.21.15.102:0/802348627 <== mon.2 v1:172.21.15.102:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7ffb40012d00 con 0x7ffb5012e330 2021-05-25T11:59:56.969 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.913+0000 7ffb3affd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:56.969 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:56.913+0000 7ffb3affd700 10 monclient: _finish_command 1 = system:0 2021-05-25T11:59:57.068 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.065+0000 7ffb55923700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2021-05-25T11:59:57.068 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.065+0000 7ffb55923700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2021-05-25T11:59:57.068 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.065+0000 7ffb55923700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T11:59:57.069 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.065+0000 7ffb55923700 1 -- 172.21.15.102:0/802348627 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7ffb28005060 con 0x7ffb5012e330 2021-05-25T11:59:57.069 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.065+0000 7ffb3affd700 1 -- 172.21.15.102:0/802348627 <== mon.2 v1:172.21.15.102:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (unknown 574334944 0 847495621) 0x7ffb40012b20 con 0x7ffb5012e330 2021-05-25T11:59:57.069 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.065+0000 7ffb3affd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2021-05-25T11:59:57.069 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.065+0000 7ffb3affd700 10 monclient: _finish_command 2 = system:0 2021-05-25T11:59:57.070 INFO:teuthology.orchestra.run.smithi102.stdout:47244640259 2021-05-25T11:59:57.071 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.066+0000 7ffb55923700 10 librados: watch_flush enter 2021-05-25T11:59:57.071 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.067+0000 7ffb55923700 10 librados: watch_flush exit 2021-05-25T11:59:57.072 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.067+0000 7ffb55923700 20 client.4298.objecter shutdown clearing up homeless session... 2021-05-25T11:59:57.072 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.067+0000 7ffb55923700 10 client.4298.objecter successfully canceled tick 2021-05-25T11:59:57.072 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.067+0000 7ffb55923700 1 -- 172.21.15.102:0/802348627 >> v1:172.21.15.181:6812/29582 conn(0x7ffb30033510 legacy=0x7ffb300359d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:57.072 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.067+0000 7ffb55923700 10 monclient: shutdown 2021-05-25T11:59:57.072 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.067+0000 7ffb55923700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T11:59:57.073 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.067+0000 7ffb55923700 1 -- 172.21.15.102:0/802348627 >> v1:172.21.15.102:6790/0 conn(0x7ffb5012e330 legacy=0x7ffb50140cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:57.073 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.067+0000 7ffb55923700 1 -- 172.21.15.102:0/802348627 shutdown_connections 2021-05-25T11:59:57.073 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.067+0000 7ffb55923700 1 -- 172.21.15.102:0/802348627 >> 172.21.15.102:0/802348627 conn(0x7ffb5009b370 msgr2=0x7ffb501292f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T11:59:57.073 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.067+0000 7ffb55923700 1 -- 172.21.15.102:0/802348627 shutdown_connections 2021-05-25T11:59:57.074 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.067+0000 7ffb55923700 1 -- 172.21.15.102:0/802348627 wait complete. 2021-05-25T11:59:57.074 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.067+0000 7ffb55923700 1 librados: shutdown 2021-05-25T11:59:57.080 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.0 2021-05-25T11:59:57.081 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2021-05-25T11:59:57.190 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.187+0000 7f3cd4d66700 10 monclient: get_monmap_and_config 2021-05-25T11:59:57.191 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.187+0000 7f3cd4d66700 10 monclient: build_initial_monmap 2021-05-25T11:59:57.191 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.188+0000 7f3cd4d66700 10 monclient: monmap: 2021-05-25T11:59:57.191 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T11:59:57.191 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:57.192 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:57.189888+0000 2021-05-25T11:59:57.192 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:57.189888+0000 2021-05-25T11:59:57.192 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T11:59:57.192 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:57.192 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T11:59:57.193 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T11:59:57.193 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T11:59:57.193 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:57.193 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.188+0000 7f3cd4d66700 1 Processor -- start 2021-05-25T11:59:57.193 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.188+0000 7f3cd4d66700 1 -- start start 2021-05-25T11:59:57.194 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.188+0000 7f3cd4d66700 10 monclient: init 2021-05-25T11:59:57.203 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.189+0000 7f3cd4d66700 10 monclient: _reopen_session rank -1 2021-05-25T11:59:57.204 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.189+0000 7f3cd4d66700 10 monclient: _add_conns ranks=[0,2,1] 2021-05-25T11:59:57.204 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.189+0000 7f3cd4d66700 10 monclient(hunting): picked mon.noname-a con 0x7f3cd00966d0 addr v1:172.21.15.102:6789/0 2021-05-25T11:59:57.204 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.189+0000 7f3cd4d66700 10 monclient(hunting): picked mon.noname-b con 0x7f3cd0096ab0 addr v1:172.21.15.102:6790/0 2021-05-25T11:59:57.204 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.189+0000 7f3cd4d66700 10 monclient(hunting): picked mon.noname-c con 0x7f3cd0132010 addr v1:172.21.15.181:6789/0 2021-05-25T11:59:57.205 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.189+0000 7f3cd4d66700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cd0064610 con 0x7f3cd00966d0 2021-05-25T11:59:57.205 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.189+0000 7f3cd4d66700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cd0064840 con 0x7f3cd0132010 2021-05-25T11:59:57.205 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.189+0000 7f3cd4d66700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cd0064ae0 con 0x7f3cd0096ab0 2021-05-25T11:59:57.205 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.189+0000 7f3cd4d66700 10 monclient(hunting): _renew_subs 2021-05-25T11:59:57.205 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.189+0000 7f3cd4d66700 10 monclient(hunting): authenticate will time out at 797.787244s 2021-05-25T11:59:57.206 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.189+0000 7f3ccdd9b700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f3cd0096ab0 0x7f3cd01308f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:53782/0 (socket says 172.21.15.102:53782) 2021-05-25T11:59:57.206 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.189+0000 7f3ccdd9b700 1 -- 172.21.15.102:0/42468042 learned_addr learned my addr 172.21.15.102:0/42468042 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T11:59:57.206 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.189+0000 7f3cce59c700 1 --1- 172.21.15.102:0/42468042 >> v1:172.21.15.102:6789/0 conn(0x7f3cd00966d0 0x7f3cd00697d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:58902/0 (socket says 172.21.15.102:58902) 2021-05-25T11:59:57.206 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.189+0000 7f3ccd59a700 1 -- 172.21.15.102:0/42468042 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1713450971 0 0) 0x7f3cd0064610 con 0x7f3cd00966d0 2021-05-25T11:59:57.207 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.189+0000 7f3ccd59a700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:57.207 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.189+0000 7f3ccd59a700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:57.207 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.189+0000 7f3ccd59a700 10 monclient(hunting): my global_id is 4308 2021-05-25T11:59:57.207 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.189+0000 7f3ccd59a700 1 -- 172.21.15.102:0/42468042 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3cb4003270 con 0x7f3cd00966d0 2021-05-25T11:59:57.207 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.189+0000 7f3ccd59a700 1 -- 172.21.15.102:0/42468042 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2701259467 0 0) 0x7f3cd0064ae0 con 0x7f3cd0096ab0 2021-05-25T11:59:57.208 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.189+0000 7f3ccd59a700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:57.208 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.189+0000 7f3ccd59a700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:57.208 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.189+0000 7f3ccd59a700 10 monclient(hunting): my global_id is 4301 2021-05-25T11:59:57.208 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.189+0000 7f3ccd59a700 1 -- 172.21.15.102:0/42468042 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3cd0064610 con 0x7f3cd0096ab0 2021-05-25T11:59:57.208 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.189+0000 7f3ccd59a700 1 -- 172.21.15.102:0/42468042 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3106560351 0 0) 0x7f3cb4003270 con 0x7f3cd00966d0 2021-05-25T11:59:57.209 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.189+0000 7f3ccd59a700 1 -- 172.21.15.102:0/42468042 >> v1:172.21.15.102:6790/0 conn(0x7f3cd0096ab0 legacy=0x7f3cd01308f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:57.209 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.189+0000 7f3ccd59a700 1 -- 172.21.15.102:0/42468042 >> v1:172.21.15.181:6789/0 conn(0x7f3cd0132010 legacy=0x7f3cd0134400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:57.209 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.189+0000 7f3ccd59a700 10 monclient: _finish_hunting 0 2021-05-25T11:59:57.209 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.189+0000 7f3ccd59a700 1 monclient: found mon.noname-a 2021-05-25T11:59:57.209 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.189+0000 7f3ccd59a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T11:59:57.210 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.189+0000 7f3ccd59a700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T11:59:57.210 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.189+0000 7f3ccd59a700 1 -- 172.21.15.102:0/42468042 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cd0069ee0 con 0x7f3cd00966d0 2021-05-25T11:59:57.210 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.189+0000 7f3ccd59a700 10 monclient: _finish_auth 0 2021-05-25T11:59:57.210 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.189+0000 7f3ccd59a700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T11:59:57.211 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.189+0000 7f3cd4d66700 5 monclient: authenticate success, global_id 4308 2021-05-25T11:59:57.211 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.189+0000 7f3cd4d66700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T11:59:57.211 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.189+0000 7f3ccd59a700 1 -- 172.21.15.102:0/42468042 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f3cbc001610 con 0x7f3cd00966d0 2021-05-25T11:59:57.211 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.189+0000 7f3ccd59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:57.212 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.189+0000 7f3ccd59a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T11:59:57.212 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.189+0000 7f3ccd59a700 10 monclient: dump: 2021-05-25T11:59:57.212 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:57.212 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:57.212 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:57.213 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:57.213 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:57.213 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:57.213 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:57.213 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:57.214 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:57.214 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:57.214 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.189+0000 7f3cd4d66700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T11:59:57.215 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.189+0000 7f3ccd59a700 1 -- 172.21.15.102:0/42468042 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3cbc0018b0 con 0x7f3cd00966d0 2021-05-25T11:59:57.215 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.189+0000 7f3ccd59a700 10 monclient: handle_config config(0 keys) v1 2021-05-25T11:59:57.215 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.190+0000 7f3cd4d66700 10 monclient: get_monmap_and_config success 2021-05-25T11:59:57.215 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.190+0000 7f3cd4d66700 4 set_mon_vals no callback set 2021-05-25T11:59:57.215 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.190+0000 7f3ccd59a700 1 -- 172.21.15.102:0/42468042 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f3cbc001c40 con 0x7f3cd00966d0 2021-05-25T11:59:57.216 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.190+0000 7f3ccd59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:57.216 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.190+0000 7f3ccd59a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T11:59:57.216 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.190+0000 7f3ccd59a700 10 monclient: dump: 2021-05-25T11:59:57.216 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:57.216 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:57.217 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:57.217 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:57.217 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:57.217 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:57.217 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:57.218 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:57.218 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:57.218 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:57.218 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.190+0000 7f3cd4d66700 10 monclient: shutdown 2021-05-25T11:59:57.219 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.190+0000 7f3cd4d66700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T11:59:57.219 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.190+0000 7f3cd4d66700 1 -- 172.21.15.102:0/42468042 >> v1:172.21.15.102:6789/0 conn(0x7f3cd00966d0 legacy=0x7f3cd00697d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:57.219 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.190+0000 7f3cd4d66700 1 -- 172.21.15.102:0/42468042 shutdown_connections 2021-05-25T11:59:57.219 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.190+0000 7f3cd4d66700 1 -- 172.21.15.102:0/42468042 >> 172.21.15.102:0/42468042 conn(0x7f3cd009cd10 msgr2=0x7f3cd009d120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T11:59:57.219 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.190+0000 7f3cd4d66700 1 -- 172.21.15.102:0/42468042 shutdown_connections 2021-05-25T11:59:57.220 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.190+0000 7f3cd4d66700 1 -- 172.21.15.102:0/42468042 wait complete. 2021-05-25T11:59:57.220 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.190+0000 7f3cd4d66700 10 monclient: build_initial_monmap 2021-05-25T11:59:57.220 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.191+0000 7f3cd4d66700 10 monclient: monmap: 2021-05-25T11:59:57.220 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T11:59:57.220 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T11:59:57.221 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T11:59:57.221 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T11:59:57.221 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T11:59:57.221 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T11:59:57.221 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T11:59:57.222 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T11:59:57.222 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T11:59:57.222 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:57.222 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.191+0000 7f3cd4d66700 1 librados: starting msgr at 2021-05-25T11:59:57.222 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.191+0000 7f3cd4d66700 1 librados: starting objecter 2021-05-25T11:59:57.223 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.191+0000 7f3cd4d66700 1 Processor -- start 2021-05-25T11:59:57.223 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.191+0000 7f3cd4d66700 1 -- start start 2021-05-25T11:59:57.223 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.191+0000 7f3cd4d66700 1 librados: setting wanted keys 2021-05-25T11:59:57.223 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.191+0000 7f3cd4d66700 1 librados: calling monclient init 2021-05-25T11:59:57.224 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.191+0000 7f3cd4d66700 10 monclient: init 2021-05-25T11:59:57.224 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.191+0000 7f3cd4d66700 10 monclient: _reopen_session rank -1 2021-05-25T11:59:57.224 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.191+0000 7f3cd4d66700 10 monclient: _add_conns ranks=[2,1,0] 2021-05-25T11:59:57.224 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.191+0000 7f3cd4d66700 10 monclient(hunting): picked mon.noname-c con 0x7f3cd00966d0 addr v1:172.21.15.102:6790/0 2021-05-25T11:59:57.224 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.191+0000 7f3cd4d66700 10 monclient(hunting): picked mon.noname-b con 0x7f3cd0096ab0 addr v1:172.21.15.181:6789/0 2021-05-25T11:59:57.225 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.191+0000 7f3cd4d66700 10 monclient(hunting): picked mon.noname-a con 0x7f3cd0132010 addr v1:172.21.15.102:6789/0 2021-05-25T11:59:57.225 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.191+0000 7f3cd4d66700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cd0095890 con 0x7f3cd0132010 2021-05-25T11:59:57.225 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.191+0000 7f3cd4d66700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cd0118c00 con 0x7f3cd0096ab0 2021-05-25T11:59:57.225 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.191+0000 7f3cd4d66700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cd0118ea0 con 0x7f3cd00966d0 2021-05-25T11:59:57.225 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.191+0000 7f3cd4d66700 10 monclient(hunting): _renew_subs 2021-05-25T11:59:57.226 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.191+0000 7f3cd4d66700 10 monclient(hunting): authenticate will time out at 797.789759s 2021-05-25T11:59:57.226 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.191+0000 7f3cce59c700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f3cd00966d0 0x7f3cd0067180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:53788/0 (socket says 172.21.15.102:53788) 2021-05-25T11:59:57.226 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.191+0000 7f3cce59c700 1 -- 172.21.15.102:0/2104378710 learned_addr learned my addr 172.21.15.102:0/2104378710 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T11:59:57.226 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.191+0000 7f3cba7fc700 10 client.?.objecter ms_handle_connect 0x7f3cd00966d0 2021-05-25T11:59:57.226 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.191+0000 7f3cba7fc700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:57.227 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.191+0000 7f3cba7fc700 10 client.?.objecter ms_handle_connect 0x7f3cd0132010 2021-05-25T11:59:57.227 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.191+0000 7f3cba7fc700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:57.227 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.192+0000 7f3cba7fc700 1 -- 172.21.15.102:0/2104378710 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1722132402 0 0) 0x7f3cd0118ea0 con 0x7f3cd00966d0 2021-05-25T11:59:57.228 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.192+0000 7f3cba7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:57.228 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.192+0000 7f3cba7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:57.228 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.192+0000 7f3cba7fc700 10 monclient(hunting): my global_id is 4304 2021-05-25T11:59:57.228 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.192+0000 7f3cba7fc700 1 -- 172.21.15.102:0/2104378710 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3cac004c10 con 0x7f3cd00966d0 2021-05-25T11:59:57.228 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.192+0000 7f3cba7fc700 10 client.?.objecter ms_handle_connect 0x7f3cd0096ab0 2021-05-25T11:59:57.229 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.192+0000 7f3cba7fc700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:57.229 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.192+0000 7f3cba7fc700 1 -- 172.21.15.102:0/2104378710 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 337737605 0 0) 0x7f3cd0095890 con 0x7f3cd0132010 2021-05-25T11:59:57.229 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.192+0000 7f3cba7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:57.229 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.192+0000 7f3cba7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:57.229 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.192+0000 7f3cba7fc700 10 monclient(hunting): my global_id is 4311 2021-05-25T11:59:57.230 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.192+0000 7f3cba7fc700 1 -- 172.21.15.102:0/2104378710 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3cd0118ea0 con 0x7f3cd0132010 2021-05-25T11:59:57.230 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.192+0000 7f3cba7fc700 1 -- 172.21.15.102:0/2104378710 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2918574418 0 0) 0x7f3cd0118c00 con 0x7f3cd0096ab0 2021-05-25T11:59:57.230 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.192+0000 7f3cba7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:57.230 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.192+0000 7f3cba7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:57.230 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.192+0000 7f3cba7fc700 10 monclient(hunting): my global_id is 4306 2021-05-25T11:59:57.231 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.192+0000 7f3cba7fc700 1 -- 172.21.15.102:0/2104378710 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3cd0095890 con 0x7f3cd0096ab0 2021-05-25T11:59:57.231 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.192+0000 7f3cba7fc700 1 -- 172.21.15.102:0/2104378710 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2053900032 0 0) 0x7f3cd0118ea0 con 0x7f3cd0132010 2021-05-25T11:59:57.231 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.192+0000 7f3cba7fc700 1 -- 172.21.15.102:0/2104378710 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3cd0118c00 con 0x7f3cd0132010 2021-05-25T11:59:57.231 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.192+0000 7f3cba7fc700 1 -- 172.21.15.102:0/2104378710 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3905197975 0 0) 0x7f3cac004c10 con 0x7f3cd00966d0 2021-05-25T11:59:57.231 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.192+0000 7f3cba7fc700 1 -- 172.21.15.102:0/2104378710 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3cd0118ea0 con 0x7f3cd00966d0 2021-05-25T11:59:57.232 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.192+0000 7f3cba7fc700 1 -- 172.21.15.102:0/2104378710 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f3cc0001b80 con 0x7f3cd0132010 2021-05-25T11:59:57.232 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.192+0000 7f3cba7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:57.232 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.192+0000 7f3cba7fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T11:59:57.232 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.192+0000 7f3cba7fc700 10 monclient(hunting): dump: 2021-05-25T11:59:57.233 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:57.233 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:57.233 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:57.233 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:57.233 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:57.234 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:57.234 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:57.234 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:57.234 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:57.234 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:57.235 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.192+0000 7f3cba7fc700 10 monclient(hunting): _finish_auth 0 2021-05-25T11:59:57.235 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.192+0000 7f3cba7fc700 1 -- 172.21.15.102:0/2104378710 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f3cbc001bc0 con 0x7f3cd00966d0 2021-05-25T11:59:57.235 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.192+0000 7f3cba7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:57.235 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.192+0000 7f3cba7fc700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2021-05-25T11:59:57.235 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.192+0000 7f3cba7fc700 10 monclient(hunting): dump: 2021-05-25T11:59:57.236 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:57.236 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:57.236 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:57.236 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:57.236 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:57.237 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:57.237 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:57.237 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:57.237 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:57.237 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:57.238 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.192+0000 7f3cba7fc700 1 -- 172.21.15.102:0/2104378710 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4060803127 0 0) 0x7f3cd0095890 con 0x7f3cd0096ab0 2021-05-25T11:59:57.238 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.192+0000 7f3cba7fc700 1 -- 172.21.15.102:0/2104378710 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3cac004c10 con 0x7f3cd0096ab0 2021-05-25T11:59:57.238 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.192+0000 7f3cba7fc700 1 -- 172.21.15.102:0/2104378710 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f3cc40018e0 con 0x7f3cd0096ab0 2021-05-25T11:59:57.238 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.192+0000 7f3cba7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:57.238 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.192+0000 7f3cba7fc700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2021-05-25T11:59:57.239 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.192+0000 7f3cba7fc700 10 monclient(hunting): dump: 2021-05-25T11:59:57.239 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:57.239 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:57.239 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:57.239 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:57.240 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:57.240 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:57.240 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:57.241 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:57.241 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:57.241 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:57.241 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.192+0000 7f3cba7fc700 1 -- 172.21.15.102:0/2104378710 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3569075527 0 0) 0x7f3cd0118c00 con 0x7f3cd0132010 2021-05-25T11:59:57.242 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.192+0000 7f3cba7fc700 1 -- 172.21.15.102:0/2104378710 >> v1:172.21.15.102:6790/0 conn(0x7f3cd00966d0 legacy=0x7f3cd0067180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:57.242 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.192+0000 7f3cba7fc700 1 -- 172.21.15.102:0/2104378710 >> v1:172.21.15.181:6789/0 conn(0x7f3cd0096ab0 legacy=0x7f3cd01313e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:57.242 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.192+0000 7f3cba7fc700 10 monclient: _finish_hunting 0 2021-05-25T11:59:57.242 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.192+0000 7f3cba7fc700 1 monclient: found mon.a 2021-05-25T11:59:57.242 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.192+0000 7f3cba7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T11:59:57.243 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.192+0000 7f3cba7fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:57.243 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.192+0000 7f3cba7fc700 1 -- 172.21.15.102:0/2104378710 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cd0131af0 con 0x7f3cd0132010 2021-05-25T11:59:57.243 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.192+0000 7f3cba7fc700 10 monclient: _finish_auth 0 2021-05-25T11:59:57.243 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.192+0000 7f3cba7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T11:59:57.243 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.192+0000 7f3cd4d66700 5 monclient: authenticate success, global_id 4311 2021-05-25T11:59:57.244 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.192+0000 7f3cd4d66700 10 monclient: _renew_subs 2021-05-25T11:59:57.244 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.192+0000 7f3cd4d66700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:57.244 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.192+0000 7f3cd4d66700 1 -- 172.21.15.102:0/2104378710 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cd0067f50 con 0x7f3cd0132010 2021-05-25T11:59:57.244 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.192+0000 7f3cd4d66700 10 client.4311.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T11:59:57.244 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.192+0000 7f3cd4d66700 10 monclient: _renew_subs 2021-05-25T11:59:57.245 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.192+0000 7f3cd4d66700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:57.245 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.192+0000 7f3cd4d66700 1 -- 172.21.15.102:0/2104378710 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cd01179f0 con 0x7f3cd0132010 2021-05-25T11:59:57.245 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.192+0000 7f3cba7fc700 1 -- 172.21.15.102:0/2104378710 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3cc0001db0 con 0x7f3cd0132010 2021-05-25T11:59:57.245 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.192+0000 7f3cd4d66700 1 librados: init done 2021-05-25T11:59:57.245 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.192+0000 7f3cba7fc700 10 monclient: handle_config config(0 keys) v1 2021-05-25T11:59:57.246 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.192+0000 7f3cba7fc700 1 -- 172.21.15.102:0/2104378710 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f3cc0002140 con 0x7f3cd0132010 2021-05-25T11:59:57.246 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.192+0000 7f3cba7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:57.246 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.192+0000 7f3cba7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T11:59:57.246 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.192+0000 7f3cccd99700 4 set_mon_vals no callback set 2021-05-25T11:59:57.247 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.192+0000 7f3cba7fc700 10 monclient: dump: 2021-05-25T11:59:57.247 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:57.247 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:57.247 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:57.247 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:57.248 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:57.248 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:57.248 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:57.248 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:57.248 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:57.249 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:57.249 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.193+0000 7f3cba7fc700 1 -- 172.21.15.102:0/2104378710 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f3cc0002320 con 0x7f3cd0132010 2021-05-25T11:59:57.249 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.193+0000 7f3cd4d66700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:57.250 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.193+0000 7f3cd4d66700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:57.250 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.193+0000 7f3cd4d66700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:57.250 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.193+0000 7f3cd4d66700 1 -- 172.21.15.102:0/2104378710 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cd00987e0 con 0x7f3cd0132010 2021-05-25T11:59:57.251 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.193+0000 7f3cba7fc700 1 -- 172.21.15.102:0/2104378710 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(15..15 src has 1..15) v4 ==== 3549+0+0 (unknown 3948819498 0 0) 0x7f3cc003aa30 con 0x7f3cd0132010 2021-05-25T11:59:57.251 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.193+0000 7f3cba7fc700 10 client.4311.objecter ms_dispatch 0x7f3cd0001550 osd_map(15..15 src has 1..15) v4 2021-05-25T11:59:57.251 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.193+0000 7f3cba7fc700 3 client.4311.objecter handle_osd_map got epochs [15,15] > 0 2021-05-25T11:59:57.251 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.193+0000 7f3cba7fc700 3 client.4311.objecter handle_osd_map decoding full epoch 15 2021-05-25T11:59:57.251 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.193+0000 7f3cba7fc700 20 client.4311.objecter dump_active .. 0 homeless 2021-05-25T11:59:57.252 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.193+0000 7f3cce59c700 10 monclient: get_auth_request con 0x7f3cac033520 auth_method 0 2021-05-25T11:59:57.252 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.193+0000 7f3cba7fc700 10 client.4311.objecter ms_handle_connect 0x7f3cac033520 2021-05-25T11:59:57.252 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.197+0000 7f3cba7fc700 1 -- 172.21.15.102:0/2104378710 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7f3cc0002cd0 con 0x7f3cd0132010 2021-05-25T11:59:57.252 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.197+0000 7f3cba7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:57.253 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.197+0000 7f3cba7fc700 10 monclient: _finish_command 1 = system:0 2021-05-25T11:59:57.349 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.346+0000 7f3cd4d66700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2021-05-25T11:59:57.349 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.346+0000 7f3cd4d66700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2021-05-25T11:59:57.349 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.346+0000 7f3cd4d66700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:57.350 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.346+0000 7f3cd4d66700 1 -- 172.21.15.102:0/2104378710 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f3cd00962c0 con 0x7f3cd0132010 2021-05-25T11:59:57.350 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.346+0000 7f3cba7fc700 1 -- 172.21.15.102:0/2104378710 <== mon.0 v1:172.21.15.102:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (unknown 832126871 0 847495621) 0x7f3cc00107a0 con 0x7f3cd0132010 2021-05-25T11:59:57.350 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.346+0000 7f3cba7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2021-05-25T11:59:57.350 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.346+0000 7f3cba7fc700 10 monclient: _finish_command 2 = system:0 2021-05-25T11:59:57.351 INFO:teuthology.orchestra.run.smithi102.stdout:47244640259 2021-05-25T11:59:57.353 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.347+0000 7f3cd4d66700 10 librados: watch_flush enter 2021-05-25T11:59:57.353 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.347+0000 7f3cd4d66700 10 librados: watch_flush exit 2021-05-25T11:59:57.353 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.347+0000 7f3cd4d66700 20 client.4311.objecter shutdown clearing up homeless session... 2021-05-25T11:59:57.353 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.347+0000 7f3cd4d66700 10 client.4311.objecter successfully canceled tick 2021-05-25T11:59:57.354 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.348+0000 7f3cd4d66700 1 -- 172.21.15.102:0/2104378710 >> v1:172.21.15.181:6812/29582 conn(0x7f3cac033520 legacy=0x7f3cac0359e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:57.354 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.348+0000 7f3cd4d66700 10 monclient: shutdown 2021-05-25T11:59:57.354 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.348+0000 7f3cd4d66700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T11:59:57.354 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.348+0000 7f3cd4d66700 1 -- 172.21.15.102:0/2104378710 >> v1:172.21.15.102:6789/0 conn(0x7f3cd0132010 legacy=0x7f3cd0143430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:57.355 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.348+0000 7f3cd4d66700 1 -- 172.21.15.102:0/2104378710 shutdown_connections 2021-05-25T11:59:57.355 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.348+0000 7f3cd4d66700 1 -- 172.21.15.102:0/2104378710 >> 172.21.15.102:0/2104378710 conn(0x7f3cd009cd10 msgr2=0x7f3cd009bc10 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T11:59:57.355 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.348+0000 7f3cd4d66700 1 -- 172.21.15.102:0/2104378710 shutdown_connections 2021-05-25T11:59:57.355 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.348+0000 7f3cd4d66700 1 -- 172.21.15.102:0/2104378710 wait complete. 2021-05-25T11:59:57.355 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.348+0000 7f3cd4d66700 1 librados: shutdown 2021-05-25T11:59:57.361 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.1 2021-05-25T11:59:57.362 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2021-05-25T11:59:57.483 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.469+0000 7f38099d4700 10 monclient: get_monmap_and_config 2021-05-25T11:59:57.483 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.469+0000 7f38099d4700 10 monclient: build_initial_monmap 2021-05-25T11:59:57.483 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.469+0000 7f38099d4700 10 monclient: monmap: 2021-05-25T11:59:57.484 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T11:59:57.484 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:57.484 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:57.470990+0000 2021-05-25T11:59:57.484 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:57.470990+0000 2021-05-25T11:59:57.485 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T11:59:57.485 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:57.485 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T11:59:57.485 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T11:59:57.486 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T11:59:57.486 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:57.486 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.469+0000 7f38099d4700 1 Processor -- start 2021-05-25T11:59:57.486 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.469+0000 7f38099d4700 1 -- start start 2021-05-25T11:59:57.486 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.469+0000 7f38099d4700 10 monclient: init 2021-05-25T11:59:57.487 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.470+0000 7f38099d4700 10 monclient: _reopen_session rank -1 2021-05-25T11:59:57.487 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.470+0000 7f38099d4700 10 monclient: _add_conns ranks=[2,1,0] 2021-05-25T11:59:57.487 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.470+0000 7f38099d4700 10 monclient(hunting): picked mon.noname-b con 0x7f380409a5e0 addr v1:172.21.15.102:6790/0 2021-05-25T11:59:57.487 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.470+0000 7f38099d4700 10 monclient(hunting): picked mon.noname-c con 0x7f3804063de0 addr v1:172.21.15.181:6789/0 2021-05-25T11:59:57.488 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.470+0000 7f38099d4700 10 monclient(hunting): picked mon.noname-a con 0x7f38041325c0 addr v1:172.21.15.102:6789/0 2021-05-25T11:59:57.488 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.470+0000 7f38099d4700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f380409a3a0 con 0x7f38041325c0 2021-05-25T11:59:57.488 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.470+0000 7f38099d4700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f380409b400 con 0x7f3804063de0 2021-05-25T11:59:57.488 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.470+0000 7f38099d4700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38040963f0 con 0x7f380409a5e0 2021-05-25T11:59:57.488 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.470+0000 7f38099d4700 10 monclient(hunting): _renew_subs 2021-05-25T11:59:57.489 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.470+0000 7f38099d4700 10 monclient(hunting): authenticate will time out at 798.068178s 2021-05-25T11:59:57.489 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.470+0000 7f38037fe700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f38041325c0 0x7f38041349b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:58920/0 (socket says 172.21.15.102:58920) 2021-05-25T11:59:57.489 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.470+0000 7f3802ffd700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f380409a5e0 0x7f38040636d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:53794/0 (socket says 172.21.15.102:53794) 2021-05-25T11:59:57.489 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.470+0000 7f38037fe700 1 -- 172.21.15.102:0/2244135564 learned_addr learned my addr 172.21.15.102:0/2244135564 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T11:59:57.489 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.470+0000 7f3801ffb700 1 -- 172.21.15.102:0/2244135564 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 695274282 0 0) 0x7f380409a3a0 con 0x7f38041325c0 2021-05-25T11:59:57.490 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.470+0000 7f3801ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:57.490 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.470+0000 7f3801ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:57.490 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.470+0000 7f3801ffb700 10 monclient(hunting): my global_id is 4314 2021-05-25T11:59:57.490 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.470+0000 7f3801ffb700 1 -- 172.21.15.102:0/2244135564 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37f0003270 con 0x7f38041325c0 2021-05-25T11:59:57.491 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.470+0000 7f3801ffb700 1 -- 172.21.15.102:0/2244135564 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 590822865 0 0) 0x7f38040963f0 con 0x7f380409a5e0 2021-05-25T11:59:57.491 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.470+0000 7f3801ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:57.491 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.470+0000 7f3801ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:57.491 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.470+0000 7f3801ffb700 10 monclient(hunting): my global_id is 4307 2021-05-25T11:59:57.492 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.470+0000 7f3801ffb700 1 -- 172.21.15.102:0/2244135564 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f380409a3a0 con 0x7f380409a5e0 2021-05-25T11:59:57.492 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.470+0000 7f3801ffb700 1 -- 172.21.15.102:0/2244135564 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3627146155 0 0) 0x7f380409b400 con 0x7f3804063de0 2021-05-25T11:59:57.492 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.470+0000 7f3801ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:57.492 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.470+0000 7f3801ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:57.492 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.470+0000 7f3801ffb700 10 monclient(hunting): my global_id is 4309 2021-05-25T11:59:57.493 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.470+0000 7f3801ffb700 1 -- 172.21.15.102:0/2244135564 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38040963f0 con 0x7f3804063de0 2021-05-25T11:59:57.493 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.470+0000 7f3801ffb700 1 -- 172.21.15.102:0/2244135564 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1740575041 0 0) 0x7f380409a3a0 con 0x7f380409a5e0 2021-05-25T11:59:57.493 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.470+0000 7f3801ffb700 1 -- 172.21.15.102:0/2244135564 >> v1:172.21.15.181:6789/0 conn(0x7f3804063de0 legacy=0x7f3804064260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:57.493 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.470+0000 7f3801ffb700 1 -- 172.21.15.102:0/2244135564 >> v1:172.21.15.102:6789/0 conn(0x7f38041325c0 legacy=0x7f38041349b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:57.493 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.470+0000 7f3801ffb700 10 monclient: _finish_hunting 0 2021-05-25T11:59:57.494 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.470+0000 7f3801ffb700 1 monclient: found mon.noname-b 2021-05-25T11:59:57.494 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.470+0000 7f3801ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T11:59:57.494 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.470+0000 7f3801ffb700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T11:59:57.494 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.470+0000 7f3801ffb700 1 -- 172.21.15.102:0/2244135564 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3804096690 con 0x7f380409a5e0 2021-05-25T11:59:57.495 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.470+0000 7f3801ffb700 10 monclient: _finish_auth 0 2021-05-25T11:59:57.495 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.470+0000 7f3801ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T11:59:57.495 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.470+0000 7f3801ffb700 1 -- 172.21.15.102:0/2244135564 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f37ec001560 con 0x7f380409a5e0 2021-05-25T11:59:57.495 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.470+0000 7f38099d4700 5 monclient: authenticate success, global_id 4307 2021-05-25T11:59:57.495 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.470+0000 7f38099d4700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T11:59:57.496 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.470+0000 7f3801ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:57.496 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.470+0000 7f3801ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T11:59:57.496 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.470+0000 7f3801ffb700 10 monclient: dump: 2021-05-25T11:59:57.496 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:57.496 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:57.497 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:57.497 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:57.497 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:57.497 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:57.498 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:57.498 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:57.498 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:57.498 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:57.499 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.470+0000 7f38099d4700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T11:59:57.499 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.470+0000 7f3801ffb700 1 -- 172.21.15.102:0/2244135564 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f37ec001700 con 0x7f380409a5e0 2021-05-25T11:59:57.499 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.470+0000 7f3801ffb700 10 monclient: handle_config config(0 keys) v1 2021-05-25T11:59:57.499 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.470+0000 7f3801ffb700 1 -- 172.21.15.102:0/2244135564 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f37ec001a90 con 0x7f380409a5e0 2021-05-25T11:59:57.499 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.470+0000 7f38099d4700 10 monclient: get_monmap_and_config success 2021-05-25T11:59:57.500 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.470+0000 7f38099d4700 4 set_mon_vals no callback set 2021-05-25T11:59:57.500 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.470+0000 7f3801ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:57.500 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.470+0000 7f3801ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T11:59:57.500 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.470+0000 7f3801ffb700 10 monclient: dump: 2021-05-25T11:59:57.501 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:57.501 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:57.501 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:57.501 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:57.501 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:57.502 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:57.502 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:57.502 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:57.502 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:57.503 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:57.503 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.471+0000 7f38099d4700 10 monclient: shutdown 2021-05-25T11:59:57.503 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.471+0000 7f38099d4700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T11:59:57.503 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.471+0000 7f38099d4700 1 -- 172.21.15.102:0/2244135564 >> v1:172.21.15.102:6790/0 conn(0x7f380409a5e0 legacy=0x7f38040636d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:57.504 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.471+0000 7f38099d4700 1 -- 172.21.15.102:0/2244135564 shutdown_connections 2021-05-25T11:59:57.504 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.471+0000 7f38099d4700 1 -- 172.21.15.102:0/2244135564 >> 172.21.15.102:0/2244135564 conn(0x7f380409bc30 msgr2=0x7f380409c040 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T11:59:57.504 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.471+0000 7f38099d4700 1 -- 172.21.15.102:0/2244135564 shutdown_connections 2021-05-25T11:59:57.504 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.471+0000 7f38099d4700 1 -- 172.21.15.102:0/2244135564 wait complete. 2021-05-25T11:59:57.504 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.472+0000 7f38099d4700 10 monclient: build_initial_monmap 2021-05-25T11:59:57.505 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.472+0000 7f38099d4700 10 monclient: monmap: 2021-05-25T11:59:57.505 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T11:59:57.505 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T11:59:57.505 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T11:59:57.506 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T11:59:57.506 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T11:59:57.506 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T11:59:57.506 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T11:59:57.506 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T11:59:57.507 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T11:59:57.507 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:57.507 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.472+0000 7f38099d4700 1 librados: starting msgr at 2021-05-25T11:59:57.507 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.472+0000 7f38099d4700 1 librados: starting objecter 2021-05-25T11:59:57.508 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.472+0000 7f38099d4700 1 Processor -- start 2021-05-25T11:59:57.508 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.472+0000 7f38099d4700 1 -- start start 2021-05-25T11:59:57.508 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.472+0000 7f38099d4700 1 librados: setting wanted keys 2021-05-25T11:59:57.508 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.472+0000 7f38099d4700 1 librados: calling monclient init 2021-05-25T11:59:57.508 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.472+0000 7f38099d4700 10 monclient: init 2021-05-25T11:59:57.509 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.472+0000 7f38099d4700 10 monclient: _reopen_session rank -1 2021-05-25T11:59:57.509 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.472+0000 7f38099d4700 10 monclient: _add_conns ranks=[1,0,2] 2021-05-25T11:59:57.509 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.472+0000 7f38099d4700 10 monclient(hunting): picked mon.noname-b con 0x7f3804063de0 addr v1:172.21.15.181:6789/0 2021-05-25T11:59:57.509 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.472+0000 7f38099d4700 10 monclient(hunting): picked mon.noname-a con 0x7f380409a5e0 addr v1:172.21.15.102:6789/0 2021-05-25T11:59:57.509 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.472+0000 7f38099d4700 10 monclient(hunting): picked mon.noname-c con 0x7f38041325c0 addr v1:172.21.15.102:6790/0 2021-05-25T11:59:57.510 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.472+0000 7f38099d4700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f380412c250 con 0x7f380409a5e0 2021-05-25T11:59:57.510 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.472+0000 7f38099d4700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38041162b0 con 0x7f3804063de0 2021-05-25T11:59:57.510 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.472+0000 7f38099d4700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3804116550 con 0x7f38041325c0 2021-05-25T11:59:57.510 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.472+0000 7f38099d4700 10 monclient(hunting): _renew_subs 2021-05-25T11:59:57.511 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.472+0000 7f38099d4700 10 monclient(hunting): authenticate will time out at 798.070898s 2021-05-25T11:59:57.511 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.472+0000 7f38037fe700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f38041325c0 0x7f3804143a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:53802/0 (socket says 172.21.15.102:53802) 2021-05-25T11:59:57.511 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.472+0000 7f38037fe700 1 -- 172.21.15.102:0/2855078203 learned_addr learned my addr 172.21.15.102:0/2855078203 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T11:59:57.511 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.472+0000 7f38027fc700 1 --1- 172.21.15.102:0/2855078203 >> v1:172.21.15.102:6789/0 conn(0x7f380409a5e0 0x7f3804140340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:58926/0 (socket says 172.21.15.102:58926) 2021-05-25T11:59:57.511 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.473+0000 7f37eaffd700 10 client.?.objecter ms_handle_connect 0x7f38041325c0 2021-05-25T11:59:57.512 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.473+0000 7f37eaffd700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:57.512 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.473+0000 7f37eaffd700 10 client.?.objecter ms_handle_connect 0x7f380409a5e0 2021-05-25T11:59:57.512 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.473+0000 7f37eaffd700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:57.512 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.473+0000 7f37eaffd700 10 client.?.objecter ms_handle_connect 0x7f3804063de0 2021-05-25T11:59:57.512 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.473+0000 7f37eaffd700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:57.513 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.473+0000 7f37eaffd700 1 -- 172.21.15.102:0/2855078203 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2715450331 0 0) 0x7f3804116550 con 0x7f38041325c0 2021-05-25T11:59:57.513 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.473+0000 7f37eaffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:57.513 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.473+0000 7f37eaffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:57.513 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.473+0000 7f37eaffd700 10 monclient(hunting): my global_id is 4310 2021-05-25T11:59:57.514 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.473+0000 7f37eaffd700 1 -- 172.21.15.102:0/2855078203 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37e4004c10 con 0x7f38041325c0 2021-05-25T11:59:57.514 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.473+0000 7f37eaffd700 1 -- 172.21.15.102:0/2855078203 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1030052662 0 0) 0x7f380412c250 con 0x7f380409a5e0 2021-05-25T11:59:57.514 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.473+0000 7f37eaffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:57.514 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.473+0000 7f37eaffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:57.514 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.473+0000 7f37eaffd700 10 monclient(hunting): my global_id is 4317 2021-05-25T11:59:57.515 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.473+0000 7f37eaffd700 1 -- 172.21.15.102:0/2855078203 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3804116550 con 0x7f380409a5e0 2021-05-25T11:59:57.515 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.473+0000 7f37eaffd700 1 -- 172.21.15.102:0/2855078203 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2087576813 0 0) 0x7f38041162b0 con 0x7f3804063de0 2021-05-25T11:59:57.515 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.473+0000 7f37eaffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:57.515 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.473+0000 7f37eaffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:57.516 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.473+0000 7f37eaffd700 10 monclient(hunting): my global_id is 4312 2021-05-25T11:59:57.516 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.473+0000 7f37eaffd700 1 -- 172.21.15.102:0/2855078203 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f380412c250 con 0x7f3804063de0 2021-05-25T11:59:57.516 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.473+0000 7f37eaffd700 1 -- 172.21.15.102:0/2855078203 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2314023148 0 0) 0x7f37e4004c10 con 0x7f38041325c0 2021-05-25T11:59:57.516 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.473+0000 7f37eaffd700 1 -- 172.21.15.102:0/2855078203 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f38041162b0 con 0x7f38041325c0 2021-05-25T11:59:57.516 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.473+0000 7f37eaffd700 1 -- 172.21.15.102:0/2855078203 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2298254659 0 0) 0x7f3804116550 con 0x7f380409a5e0 2021-05-25T11:59:57.517 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.473+0000 7f37eaffd700 1 -- 172.21.15.102:0/2855078203 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f37e4004c10 con 0x7f380409a5e0 2021-05-25T11:59:57.517 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.473+0000 7f37eaffd700 1 -- 172.21.15.102:0/2855078203 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f37f4001d70 con 0x7f38041325c0 2021-05-25T11:59:57.517 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.473+0000 7f37eaffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:57.517 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.473+0000 7f37eaffd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T11:59:57.518 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.473+0000 7f37eaffd700 10 monclient(hunting): dump: 2021-05-25T11:59:57.518 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:57.518 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:57.518 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:57.518 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:57.519 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:57.519 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:57.519 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:57.519 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:57.519 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:57.520 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:57.520 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.473+0000 7f37eaffd700 10 monclient(hunting): _finish_auth 0 2021-05-25T11:59:57.520 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.473+0000 7f37eaffd700 1 -- 172.21.15.102:0/2855078203 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f37f80018c0 con 0x7f380409a5e0 2021-05-25T11:59:57.520 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.473+0000 7f37eaffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:57.520 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.473+0000 7f37eaffd700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2021-05-25T11:59:57.521 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.473+0000 7f37eaffd700 10 monclient(hunting): dump: 2021-05-25T11:59:57.521 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:57.521 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:57.521 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:57.521 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:57.522 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:57.522 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:57.522 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:57.522 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:57.523 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:57.523 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:57.523 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.473+0000 7f37eaffd700 1 -- 172.21.15.102:0/2855078203 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1401414657 0 0) 0x7f380412c250 con 0x7f3804063de0 2021-05-25T11:59:57.523 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.473+0000 7f37eaffd700 1 -- 172.21.15.102:0/2855078203 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3804116550 con 0x7f3804063de0 2021-05-25T11:59:57.523 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.473+0000 7f37eaffd700 1 -- 172.21.15.102:0/2855078203 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f37ec001da0 con 0x7f3804063de0 2021-05-25T11:59:57.524 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.473+0000 7f37eaffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:57.524 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.473+0000 7f37eaffd700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2021-05-25T11:59:57.524 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.473+0000 7f37eaffd700 10 monclient(hunting): dump: 2021-05-25T11:59:57.524 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:57.525 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:57.525 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:57.525 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:57.525 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:57.525 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:57.526 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:57.526 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:57.526 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:57.526 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:57.527 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.473+0000 7f37eaffd700 1 -- 172.21.15.102:0/2855078203 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 59389442 0 0) 0x7f37e4004c10 con 0x7f380409a5e0 2021-05-25T11:59:57.527 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.473+0000 7f37eaffd700 1 -- 172.21.15.102:0/2855078203 >> v1:172.21.15.102:6790/0 conn(0x7f38041325c0 legacy=0x7f3804143a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:57.527 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.473+0000 7f37eaffd700 1 -- 172.21.15.102:0/2855078203 >> v1:172.21.15.181:6789/0 conn(0x7f3804063de0 legacy=0x7f380412bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:57.527 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.473+0000 7f37eaffd700 10 monclient: _finish_hunting 0 2021-05-25T11:59:57.527 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.473+0000 7f37eaffd700 1 monclient: found mon.a 2021-05-25T11:59:57.528 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.473+0000 7f37eaffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T11:59:57.528 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.473+0000 7f37eaffd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:57.528 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.473+0000 7f37eaffd700 1 -- 172.21.15.102:0/2855078203 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3804137c90 con 0x7f380409a5e0 2021-05-25T11:59:57.528 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.473+0000 7f37eaffd700 10 monclient: _finish_auth 0 2021-05-25T11:59:57.529 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.473+0000 7f37eaffd700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T11:59:57.529 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.473+0000 7f38099d4700 5 monclient: authenticate success, global_id 4317 2021-05-25T11:59:57.529 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.473+0000 7f38099d4700 10 monclient: _renew_subs 2021-05-25T11:59:57.529 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.473+0000 7f38099d4700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:57.529 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.473+0000 7f38099d4700 1 -- 172.21.15.102:0/2855078203 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3804094840 con 0x7f380409a5e0 2021-05-25T11:59:57.530 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.473+0000 7f38099d4700 10 client.4317.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T11:59:57.530 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.473+0000 7f38099d4700 10 monclient: _renew_subs 2021-05-25T11:59:57.530 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.473+0000 7f38099d4700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:57.530 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.473+0000 7f38099d4700 1 -- 172.21.15.102:0/2855078203 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f38041386d0 con 0x7f380409a5e0 2021-05-25T11:59:57.530 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.473+0000 7f38099d4700 1 librados: init done 2021-05-25T11:59:57.531 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.473+0000 7f37eaffd700 1 -- 172.21.15.102:0/2855078203 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f37f8001a60 con 0x7f380409a5e0 2021-05-25T11:59:57.531 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.473+0000 7f37eaffd700 10 monclient: handle_config config(0 keys) v1 2021-05-25T11:59:57.531 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.473+0000 7f37eaffd700 1 -- 172.21.15.102:0/2855078203 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f37f8001df0 con 0x7f380409a5e0 2021-05-25T11:59:57.531 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.473+0000 7f37eaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:57.532 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.473+0000 7f37eaffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T11:59:57.532 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.473+0000 7f38017fa700 4 set_mon_vals no callback set 2021-05-25T11:59:57.532 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.473+0000 7f37eaffd700 10 monclient: dump: 2021-05-25T11:59:57.532 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:57.532 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:57.533 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:57.533 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:57.533 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:57.533 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:57.534 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:57.534 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:57.534 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:57.534 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:57.534 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.474+0000 7f37eaffd700 1 -- 172.21.15.102:0/2855078203 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f37f8001fd0 con 0x7f380409a5e0 2021-05-25T11:59:57.535 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.474+0000 7f38099d4700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:57.535 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.474+0000 7f38099d4700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:57.535 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.474+0000 7f38099d4700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:57.535 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.474+0000 7f38099d4700 1 -- 172.21.15.102:0/2855078203 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37dc004db0 con 0x7f380409a5e0 2021-05-25T11:59:57.535 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.474+0000 7f37eaffd700 1 -- 172.21.15.102:0/2855078203 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(15..15 src has 1..15) v4 ==== 3549+0+0 (unknown 3948819498 0 0) 0x7f37f803cc00 con 0x7f380409a5e0 2021-05-25T11:59:57.536 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.474+0000 7f37eaffd700 10 client.4317.objecter ms_dispatch 0x7f3804001550 osd_map(15..15 src has 1..15) v4 2021-05-25T11:59:57.536 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.474+0000 7f37eaffd700 3 client.4317.objecter handle_osd_map got epochs [15,15] > 0 2021-05-25T11:59:57.536 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.474+0000 7f37eaffd700 3 client.4317.objecter handle_osd_map decoding full epoch 15 2021-05-25T11:59:57.536 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.474+0000 7f37eaffd700 20 client.4317.objecter dump_active .. 0 homeless 2021-05-25T11:59:57.536 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.475+0000 7f3802ffd700 10 monclient: get_auth_request con 0x7f37e40334a0 auth_method 0 2021-05-25T11:59:57.537 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.475+0000 7f37eaffd700 10 client.4317.objecter ms_handle_connect 0x7f37e40334a0 2021-05-25T11:59:57.537 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.478+0000 7f37eaffd700 1 -- 172.21.15.102:0/2855078203 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7f37f8012ba0 con 0x7f380409a5e0 2021-05-25T11:59:57.537 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.478+0000 7f37eaffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:57.537 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.478+0000 7f37eaffd700 10 monclient: _finish_command 1 = system:0 2021-05-25T11:59:57.631 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.628+0000 7f38099d4700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2021-05-25T11:59:57.631 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.628+0000 7f38099d4700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2021-05-25T11:59:57.631 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.628+0000 7f38099d4700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:57.632 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.628+0000 7f38099d4700 1 -- 172.21.15.102:0/2855078203 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f37dc005060 con 0x7f380409a5e0 2021-05-25T11:59:57.632 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.628+0000 7f37eaffd700 1 -- 172.21.15.102:0/2855078203 <== mon.0 v1:172.21.15.102:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (unknown 92182286 0 847495621) 0x7f37f8012790 con 0x7f380409a5e0 2021-05-25T11:59:57.632 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.628+0000 7f37eaffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2021-05-25T11:59:57.632 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.628+0000 7f37eaffd700 10 monclient: _finish_command 2 = system:0 2021-05-25T11:59:57.633 INFO:teuthology.orchestra.run.smithi102.stdout:47244640259 2021-05-25T11:59:57.634 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.630+0000 7f38099d4700 10 librados: watch_flush enter 2021-05-25T11:59:57.635 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.630+0000 7f38099d4700 10 librados: watch_flush exit 2021-05-25T11:59:57.635 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.630+0000 7f38099d4700 20 client.4317.objecter shutdown clearing up homeless session... 2021-05-25T11:59:57.635 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.630+0000 7f38099d4700 10 client.4317.objecter successfully canceled tick 2021-05-25T11:59:57.635 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.630+0000 7f38099d4700 1 -- 172.21.15.102:0/2855078203 >> v1:172.21.15.181:6812/29582 conn(0x7f37e40334a0 legacy=0x7f37e4035960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:57.635 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.630+0000 7f38099d4700 10 monclient: shutdown 2021-05-25T11:59:57.636 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.630+0000 7f38099d4700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T11:59:57.636 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.630+0000 7f38099d4700 1 -- 172.21.15.102:0/2855078203 >> v1:172.21.15.102:6789/0 conn(0x7f380409a5e0 legacy=0x7f3804140340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:57.636 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.630+0000 7f38099d4700 1 -- 172.21.15.102:0/2855078203 shutdown_connections 2021-05-25T11:59:57.636 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.630+0000 7f38099d4700 1 -- 172.21.15.102:0/2855078203 >> 172.21.15.102:0/2855078203 conn(0x7f380409bc30 msgr2=0x7f380412e910 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T11:59:57.637 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.630+0000 7f38099d4700 1 -- 172.21.15.102:0/2855078203 shutdown_connections 2021-05-25T11:59:57.637 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.630+0000 7f38099d4700 1 -- 172.21.15.102:0/2855078203 wait complete. 2021-05-25T11:59:57.637 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.630+0000 7f38099d4700 1 librados: shutdown 2021-05-25T11:59:57.644 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.2 2021-05-25T11:59:57.644 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2021-05-25T11:59:57.753 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.751+0000 7f890dc68700 10 monclient: get_monmap_and_config 2021-05-25T11:59:57.754 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.751+0000 7f890dc68700 10 monclient: build_initial_monmap 2021-05-25T11:59:57.754 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.751+0000 7f890dc68700 10 monclient: monmap: 2021-05-25T11:59:57.754 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T11:59:57.754 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:57.755 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:57.753261+0000 2021-05-25T11:59:57.755 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:57.753261+0000 2021-05-25T11:59:57.755 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T11:59:57.755 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:57.756 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T11:59:57.756 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T11:59:57.756 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T11:59:57.756 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:57.769 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.751+0000 7f890dc68700 1 Processor -- start 2021-05-25T11:59:57.769 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.752+0000 7f890dc68700 1 -- start start 2021-05-25T11:59:57.769 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.752+0000 7f890dc68700 10 monclient: init 2021-05-25T11:59:57.770 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.752+0000 7f890dc68700 10 monclient: _reopen_session rank -1 2021-05-25T11:59:57.770 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.752+0000 7f890dc68700 10 monclient: _add_conns ranks=[0,2,1] 2021-05-25T11:59:57.770 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.752+0000 7f890dc68700 10 monclient(hunting): picked mon.noname-a con 0x7f89080975f0 addr v1:172.21.15.102:6789/0 2021-05-25T11:59:57.771 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.752+0000 7f890dc68700 10 monclient(hunting): picked mon.noname-b con 0x7f89080980e0 addr v1:172.21.15.102:6790/0 2021-05-25T11:59:57.771 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.752+0000 7f890dc68700 10 monclient(hunting): picked mon.noname-c con 0x7f8908094f60 addr v1:172.21.15.181:6789/0 2021-05-25T11:59:57.771 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.752+0000 7f890dc68700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8908064400 con 0x7f89080975f0 2021-05-25T11:59:57.771 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.752+0000 7f890dc68700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89080646a0 con 0x7f8908094f60 2021-05-25T11:59:57.771 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.752+0000 7f890dc68700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8908064940 con 0x7f89080980e0 2021-05-25T11:59:57.772 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.752+0000 7f890dc68700 10 monclient(hunting): _renew_subs 2021-05-25T11:59:57.772 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.752+0000 7f890dc68700 10 monclient(hunting): authenticate will time out at 798.350514s 2021-05-25T11:59:57.772 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.752+0000 7f88fffff700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f89080980e0 0x7f8908131150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:53808/0 (socket says 172.21.15.102:53808) 2021-05-25T11:59:57.772 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.752+0000 7f88fffff700 1 -- 172.21.15.102:0/4270488987 learned_addr learned my addr 172.21.15.102:0/4270488987 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T11:59:57.773 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.752+0000 7f890cc66700 1 --1- 172.21.15.102:0/4270488987 >> v1:172.21.15.102:6789/0 conn(0x7f89080975f0 0x7f89080979d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:58932/0 (socket says 172.21.15.102:58932) 2021-05-25T11:59:57.773 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.752+0000 7f88ff7fe700 1 -- 172.21.15.102:0/4270488987 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1067375560 0 0) 0x7f8908064940 con 0x7f89080980e0 2021-05-25T11:59:57.773 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.752+0000 7f88ff7fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:57.773 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.752+0000 7f88ff7fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:57.774 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.752+0000 7f88ff7fe700 10 monclient(hunting): my global_id is 4313 2021-05-25T11:59:57.774 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.752+0000 7f88ff7fe700 1 -- 172.21.15.102:0/4270488987 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88f4003270 con 0x7f89080980e0 2021-05-25T11:59:57.774 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.752+0000 7f88ff7fe700 1 -- 172.21.15.102:0/4270488987 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2955542898 0 0) 0x7f8908064400 con 0x7f89080975f0 2021-05-25T11:59:57.774 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.752+0000 7f88ff7fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:57.775 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.752+0000 7f88ff7fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:57.775 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.752+0000 7f88ff7fe700 10 monclient(hunting): my global_id is 4320 2021-05-25T11:59:57.775 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.752+0000 7f88ff7fe700 1 -- 172.21.15.102:0/4270488987 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8908064940 con 0x7f89080975f0 2021-05-25T11:59:57.775 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.753+0000 7f88ff7fe700 1 -- 172.21.15.102:0/4270488987 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 457672475 0 0) 0x7f89080646a0 con 0x7f8908094f60 2021-05-25T11:59:57.775 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.753+0000 7f88ff7fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:57.776 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.753+0000 7f88ff7fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:57.776 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.753+0000 7f88ff7fe700 10 monclient(hunting): my global_id is 4315 2021-05-25T11:59:57.776 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.753+0000 7f88ff7fe700 1 -- 172.21.15.102:0/4270488987 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8908064400 con 0x7f8908094f60 2021-05-25T11:59:57.776 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.753+0000 7f88ff7fe700 1 -- 172.21.15.102:0/4270488987 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4266942806 0 0) 0x7f88f4003270 con 0x7f89080980e0 2021-05-25T11:59:57.776 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.753+0000 7f88ff7fe700 1 -- 172.21.15.102:0/4270488987 >> v1:172.21.15.181:6789/0 conn(0x7f8908094f60 legacy=0x7f8908134880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:57.777 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.753+0000 7f88ff7fe700 1 -- 172.21.15.102:0/4270488987 >> v1:172.21.15.102:6789/0 conn(0x7f89080975f0 legacy=0x7f89080979d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:57.777 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.753+0000 7f88ff7fe700 10 monclient: _finish_hunting 0 2021-05-25T11:59:57.777 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.753+0000 7f88ff7fe700 1 monclient: found mon.noname-b 2021-05-25T11:59:57.777 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.753+0000 7f88ff7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T11:59:57.778 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.753+0000 7f88ff7fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T11:59:57.778 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.753+0000 7f88ff7fe700 1 -- 172.21.15.102:0/4270488987 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8908064be0 con 0x7f89080980e0 2021-05-25T11:59:57.778 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.753+0000 7f88ff7fe700 10 monclient: _finish_auth 0 2021-05-25T11:59:57.778 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.753+0000 7f88ff7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T11:59:57.778 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.753+0000 7f88ff7fe700 1 -- 172.21.15.102:0/4270488987 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f88f0001610 con 0x7f89080980e0 2021-05-25T11:59:57.779 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.753+0000 7f890dc68700 5 monclient: authenticate success, global_id 4313 2021-05-25T11:59:57.779 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.753+0000 7f890dc68700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T11:59:57.779 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.753+0000 7f88ff7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:57.779 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.753+0000 7f88ff7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T11:59:57.780 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.753+0000 7f88ff7fe700 10 monclient: dump: 2021-05-25T11:59:57.780 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:57.780 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:57.780 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:57.780 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:57.781 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:57.781 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:57.781 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:57.781 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:57.781 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:57.782 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:57.782 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.753+0000 7f890dc68700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T11:59:57.782 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.753+0000 7f88ff7fe700 1 -- 172.21.15.102:0/4270488987 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f88f0001830 con 0x7f89080980e0 2021-05-25T11:59:57.782 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.753+0000 7f88ff7fe700 10 monclient: handle_config config(0 keys) v1 2021-05-25T11:59:57.783 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.753+0000 7f890dc68700 10 monclient: get_monmap_and_config success 2021-05-25T11:59:57.783 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.753+0000 7f88ff7fe700 1 -- 172.21.15.102:0/4270488987 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f88f0001b40 con 0x7f89080980e0 2021-05-25T11:59:57.783 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.753+0000 7f88ff7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:57.783 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.753+0000 7f890dc68700 4 set_mon_vals no callback set 2021-05-25T11:59:57.783 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.753+0000 7f88ff7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T11:59:57.784 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.753+0000 7f88ff7fe700 10 monclient: dump: 2021-05-25T11:59:57.784 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:57.784 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:57.784 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:57.784 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:57.785 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:57.785 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:57.785 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:57.785 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:57.785 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:57.786 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:57.786 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.753+0000 7f890dc68700 10 monclient: shutdown 2021-05-25T11:59:57.786 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.753+0000 7f890dc68700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T11:59:57.786 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.753+0000 7f890dc68700 1 -- 172.21.15.102:0/4270488987 >> v1:172.21.15.102:6790/0 conn(0x7f89080980e0 legacy=0x7f8908131150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:57.787 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.753+0000 7f890dc68700 1 -- 172.21.15.102:0/4270488987 shutdown_connections 2021-05-25T11:59:57.787 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.753+0000 7f890dc68700 1 -- 172.21.15.102:0/4270488987 >> 172.21.15.102:0/4270488987 conn(0x7f890809c930 msgr2=0x7f890809cd40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T11:59:57.787 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.753+0000 7f890dc68700 1 -- 172.21.15.102:0/4270488987 shutdown_connections 2021-05-25T11:59:57.787 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.753+0000 7f890dc68700 1 -- 172.21.15.102:0/4270488987 wait complete. 2021-05-25T11:59:57.787 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.754+0000 7f890dc68700 10 monclient: build_initial_monmap 2021-05-25T11:59:57.788 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.754+0000 7f890dc68700 10 monclient: monmap: 2021-05-25T11:59:57.788 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T11:59:57.788 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T11:59:57.788 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T11:59:57.789 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T11:59:57.789 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T11:59:57.789 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T11:59:57.789 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T11:59:57.789 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T11:59:57.790 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T11:59:57.790 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:57.790 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.754+0000 7f890dc68700 1 librados: starting msgr at 2021-05-25T11:59:57.790 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.754+0000 7f890dc68700 1 librados: starting objecter 2021-05-25T11:59:57.790 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.754+0000 7f890dc68700 1 Processor -- start 2021-05-25T11:59:57.791 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.754+0000 7f890dc68700 1 -- start start 2021-05-25T11:59:57.791 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.754+0000 7f890dc68700 1 librados: setting wanted keys 2021-05-25T11:59:57.791 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.754+0000 7f890dc68700 1 librados: calling monclient init 2021-05-25T11:59:57.791 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.754+0000 7f890dc68700 10 monclient: init 2021-05-25T11:59:57.791 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.755+0000 7f890dc68700 10 monclient: _reopen_session rank -1 2021-05-25T11:59:57.792 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.755+0000 7f890dc68700 10 monclient: _add_conns ranks=[1,0,2] 2021-05-25T11:59:57.792 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.755+0000 7f890dc68700 10 monclient(hunting): picked mon.noname-b con 0x7f8908094f60 addr v1:172.21.15.181:6789/0 2021-05-25T11:59:57.792 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.755+0000 7f890dc68700 10 monclient(hunting): picked mon.noname-a con 0x7f89080975f0 addr v1:172.21.15.102:6789/0 2021-05-25T11:59:57.792 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.755+0000 7f890dc68700 10 monclient(hunting): picked mon.noname-c con 0x7f89080980e0 addr v1:172.21.15.102:6790/0 2021-05-25T11:59:57.793 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.755+0000 7f890dc68700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89080639f0 con 0x7f89080975f0 2021-05-25T11:59:57.793 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.755+0000 7f890dc68700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f890811bfb0 con 0x7f8908094f60 2021-05-25T11:59:57.793 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.755+0000 7f890dc68700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89080689b0 con 0x7f89080980e0 2021-05-25T11:59:57.793 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.755+0000 7f890dc68700 10 monclient(hunting): _renew_subs 2021-05-25T11:59:57.794 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.755+0000 7f890dc68700 10 monclient(hunting): authenticate will time out at 798.353227s 2021-05-25T11:59:57.794 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.755+0000 7f88fffff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f89080975f0 0x7f89081350a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:58938/0 (socket says 172.21.15.102:58938) 2021-05-25T11:59:57.794 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.755+0000 7f88fffff700 1 -- 172.21.15.102:0/4096611726 learned_addr learned my addr 172.21.15.102:0/4096611726 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T11:59:57.794 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.755+0000 7f88fcff9700 10 client.?.objecter ms_handle_connect 0x7f89080975f0 2021-05-25T11:59:57.795 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.755+0000 7f88fcff9700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:57.795 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.755+0000 7f88fcff9700 10 client.?.objecter ms_handle_connect 0x7f89080980e0 2021-05-25T11:59:57.795 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.755+0000 7f88fcff9700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:57.795 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.755+0000 7f88fcff9700 10 client.?.objecter ms_handle_connect 0x7f8908094f60 2021-05-25T11:59:57.795 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.755+0000 7f88fcff9700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:57.796 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.755+0000 7f88fcff9700 1 -- 172.21.15.102:0/4096611726 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 853232305 0 0) 0x7f89080639f0 con 0x7f89080975f0 2021-05-25T11:59:57.796 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.755+0000 7f88fcff9700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:57.796 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.755+0000 7f88fcff9700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:57.796 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.755+0000 7f88fcff9700 10 monclient(hunting): my global_id is 4323 2021-05-25T11:59:57.796 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.755+0000 7f88fcff9700 1 -- 172.21.15.102:0/4096611726 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88e8004c10 con 0x7f89080975f0 2021-05-25T11:59:57.797 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.755+0000 7f88fcff9700 1 -- 172.21.15.102:0/4096611726 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2033510675 0 0) 0x7f89080689b0 con 0x7f89080980e0 2021-05-25T11:59:57.797 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.755+0000 7f88fcff9700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:57.797 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.755+0000 7f88fcff9700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:57.797 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.755+0000 7f88fcff9700 10 monclient(hunting): my global_id is 4316 2021-05-25T11:59:57.798 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.755+0000 7f88fcff9700 1 -- 172.21.15.102:0/4096611726 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89080639f0 con 0x7f89080980e0 2021-05-25T11:59:57.798 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.755+0000 7f88fcff9700 1 -- 172.21.15.102:0/4096611726 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2406487565 0 0) 0x7f890811bfb0 con 0x7f8908094f60 2021-05-25T11:59:57.798 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.755+0000 7f88fcff9700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:57.798 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.755+0000 7f88fcff9700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:57.799 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.755+0000 7f88fcff9700 10 monclient(hunting): my global_id is 4318 2021-05-25T11:59:57.799 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.755+0000 7f88fcff9700 1 -- 172.21.15.102:0/4096611726 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89080689b0 con 0x7f8908094f60 2021-05-25T11:59:57.799 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.755+0000 7f88fcff9700 1 -- 172.21.15.102:0/4096611726 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4279875350 0 0) 0x7f88e8004c10 con 0x7f89080975f0 2021-05-25T11:59:57.799 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.755+0000 7f88fcff9700 1 -- 172.21.15.102:0/4096611726 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f890811bfb0 con 0x7f89080975f0 2021-05-25T11:59:57.799 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.755+0000 7f88fcff9700 1 -- 172.21.15.102:0/4096611726 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f88f0001510 con 0x7f89080975f0 2021-05-25T11:59:57.800 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.755+0000 7f88fcff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:57.800 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.755+0000 7f88fcff9700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T11:59:57.800 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.755+0000 7f88fcff9700 10 monclient(hunting): dump: 2021-05-25T11:59:57.800 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:57.801 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:57.801 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:57.801 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:57.801 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:57.801 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:57.802 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:57.802 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:57.802 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:57.802 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:57.803 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.755+0000 7f88fcff9700 10 monclient(hunting): _finish_auth 0 2021-05-25T11:59:57.803 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.755+0000 7f88fcff9700 1 -- 172.21.15.102:0/4096611726 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 675958612 0 0) 0x7f89080639f0 con 0x7f89080980e0 2021-05-25T11:59:57.803 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.755+0000 7f88fcff9700 1 -- 172.21.15.102:0/4096611726 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f88e8004c10 con 0x7f89080980e0 2021-05-25T11:59:57.803 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.755+0000 7f88fcff9700 1 -- 172.21.15.102:0/4096611726 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f8904001b80 con 0x7f89080980e0 2021-05-25T11:59:57.803 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.755+0000 7f88fcff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:57.804 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.755+0000 7f88fcff9700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2021-05-25T11:59:57.804 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.755+0000 7f88fcff9700 10 monclient(hunting): dump: 2021-05-25T11:59:57.804 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:57.804 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:57.804 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:57.805 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:57.805 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:57.805 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:57.805 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:57.806 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:57.806 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:57.806 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:57.806 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.755+0000 7f88fcff9700 1 -- 172.21.15.102:0/4096611726 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 328939944 0 0) 0x7f89080689b0 con 0x7f8908094f60 2021-05-25T11:59:57.806 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.755+0000 7f88fcff9700 1 -- 172.21.15.102:0/4096611726 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f89080639f0 con 0x7f8908094f60 2021-05-25T11:59:57.807 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.755+0000 7f88fcff9700 1 -- 172.21.15.102:0/4096611726 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f88f80019f0 con 0x7f8908094f60 2021-05-25T11:59:57.807 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.755+0000 7f88fcff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:57.807 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.755+0000 7f88fcff9700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2021-05-25T11:59:57.807 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.755+0000 7f88fcff9700 10 monclient(hunting): dump: 2021-05-25T11:59:57.808 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:57.808 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:57.808 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:57.808 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:57.808 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:57.809 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:57.809 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:57.809 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:57.809 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:57.809 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:57.810 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.755+0000 7f88fcff9700 1 -- 172.21.15.102:0/4096611726 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2287860289 0 0) 0x7f890811bfb0 con 0x7f89080975f0 2021-05-25T11:59:57.810 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.755+0000 7f88fcff9700 1 -- 172.21.15.102:0/4096611726 >> v1:172.21.15.102:6790/0 conn(0x7f89080980e0 legacy=0x7f89081434b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:57.810 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.755+0000 7f88fcff9700 1 -- 172.21.15.102:0/4096611726 >> v1:172.21.15.181:6789/0 conn(0x7f8908094f60 legacy=0x7f890813b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:57.810 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.755+0000 7f88fcff9700 10 monclient: _finish_hunting 0 2021-05-25T11:59:57.811 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.755+0000 7f88fcff9700 1 monclient: found mon.a 2021-05-25T11:59:57.811 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.755+0000 7f88fcff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T11:59:57.811 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.755+0000 7f88fcff9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:57.811 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.755+0000 7f88fcff9700 1 -- 172.21.15.102:0/4096611726 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89081357b0 con 0x7f89080975f0 2021-05-25T11:59:57.811 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.756+0000 7f88fcff9700 10 monclient: _finish_auth 0 2021-05-25T11:59:57.812 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.756+0000 7f88fcff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T11:59:57.812 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.756+0000 7f890dc68700 5 monclient: authenticate success, global_id 4323 2021-05-25T11:59:57.812 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.756+0000 7f890dc68700 10 monclient: _renew_subs 2021-05-25T11:59:57.812 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.756+0000 7f890dc68700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:57.813 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.756+0000 7f890dc68700 1 -- 172.21.15.102:0/4096611726 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f890811b5b0 con 0x7f89080975f0 2021-05-25T11:59:57.813 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.756+0000 7f890dc68700 10 client.4323.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T11:59:57.813 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.756+0000 7f890dc68700 10 monclient: _renew_subs 2021-05-25T11:59:57.813 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.756+0000 7f890dc68700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:57.813 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.756+0000 7f890dc68700 1 -- 172.21.15.102:0/4096611726 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8908119f70 con 0x7f89080975f0 2021-05-25T11:59:57.814 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.756+0000 7f890dc68700 1 librados: init done 2021-05-25T11:59:57.814 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.756+0000 7f88fcff9700 1 -- 172.21.15.102:0/4096611726 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f88f00047e0 con 0x7f89080975f0 2021-05-25T11:59:57.814 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.756+0000 7f88fcff9700 10 monclient: handle_config config(0 keys) v1 2021-05-25T11:59:57.814 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.756+0000 7f88fcff9700 1 -- 172.21.15.102:0/4096611726 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f88f0004af0 con 0x7f89080975f0 2021-05-25T11:59:57.815 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.756+0000 7f88fcff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:57.815 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.756+0000 7f88fcff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T11:59:57.815 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.756+0000 7f88feffd700 4 set_mon_vals no callback set 2021-05-25T11:59:57.815 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.756+0000 7f88fcff9700 10 monclient: dump: 2021-05-25T11:59:57.815 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:57.816 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:57.816 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:57.816 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:57.816 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:57.817 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:57.817 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:57.817 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:57.817 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:57.817 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:57.818 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.756+0000 7f88fcff9700 1 -- 172.21.15.102:0/4096611726 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f88f0004c50 con 0x7f89080975f0 2021-05-25T11:59:57.818 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.757+0000 7f890dc68700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:57.818 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.757+0000 7f890dc68700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:57.818 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.757+0000 7f890dc68700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:57.819 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.757+0000 7f890dc68700 1 -- 172.21.15.102:0/4096611726 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88e0004db0 con 0x7f89080975f0 2021-05-25T11:59:57.819 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.757+0000 7f88fcff9700 1 -- 172.21.15.102:0/4096611726 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(15..15 src has 1..15) v4 ==== 3549+0+0 (unknown 3948819498 0 0) 0x7f88f003ce80 con 0x7f89080975f0 2021-05-25T11:59:57.819 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.757+0000 7f88fcff9700 10 client.4323.objecter ms_dispatch 0x7f8908001550 osd_map(15..15 src has 1..15) v4 2021-05-25T11:59:57.819 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.757+0000 7f88fcff9700 3 client.4323.objecter handle_osd_map got epochs [15,15] > 0 2021-05-25T11:59:57.820 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.757+0000 7f88fcff9700 3 client.4323.objecter handle_osd_map decoding full epoch 15 2021-05-25T11:59:57.820 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.757+0000 7f88fcff9700 20 client.4323.objecter dump_active .. 0 homeless 2021-05-25T11:59:57.820 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.757+0000 7f890cc66700 10 monclient: get_auth_request con 0x7f88e8033550 auth_method 0 2021-05-25T11:59:57.820 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.757+0000 7f88fcff9700 10 client.4323.objecter ms_handle_connect 0x7f88e8033550 2021-05-25T11:59:57.820 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.761+0000 7f88fcff9700 1 -- 172.21.15.102:0/4096611726 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7f88f0012dd0 con 0x7f89080975f0 2021-05-25T11:59:57.821 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.761+0000 7f88fcff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:57.821 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.761+0000 7f88fcff9700 10 monclient: _finish_command 1 = system:0 2021-05-25T11:59:57.911 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.908+0000 7f890dc68700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2021-05-25T11:59:57.911 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.908+0000 7f890dc68700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2021-05-25T11:59:57.912 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.908+0000 7f890dc68700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:57.912 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.908+0000 7f890dc68700 1 -- 172.21.15.102:0/4096611726 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f88e0005060 con 0x7f89080975f0 2021-05-25T11:59:57.912 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.909+0000 7f88fcff9700 1 -- 172.21.15.102:0/4096611726 <== mon.0 v1:172.21.15.102:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (unknown 383520633 0 555833778) 0x7f88f00129c0 con 0x7f89080975f0 2021-05-25T11:59:57.912 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.909+0000 7f88fcff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2021-05-25T11:59:57.913 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.909+0000 7f88fcff9700 10 monclient: _finish_command 2 = system:0 2021-05-25T11:59:57.913 INFO:teuthology.orchestra.run.smithi102.stdout:47244640258 2021-05-25T11:59:57.915 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.910+0000 7f890dc68700 10 librados: watch_flush enter 2021-05-25T11:59:57.915 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.910+0000 7f890dc68700 10 librados: watch_flush exit 2021-05-25T11:59:57.915 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.910+0000 7f890dc68700 20 client.4323.objecter shutdown clearing up homeless session... 2021-05-25T11:59:57.915 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.910+0000 7f890dc68700 10 client.4323.objecter successfully canceled tick 2021-05-25T11:59:57.916 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.910+0000 7f890dc68700 1 -- 172.21.15.102:0/4096611726 >> v1:172.21.15.181:6812/29582 conn(0x7f88e8033550 legacy=0x7f88e8035a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:57.916 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.910+0000 7f890dc68700 10 monclient: shutdown 2021-05-25T11:59:57.916 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.910+0000 7f890dc68700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T11:59:57.916 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.910+0000 7f890dc68700 1 -- 172.21.15.102:0/4096611726 >> v1:172.21.15.102:6789/0 conn(0x7f89080975f0 legacy=0x7f89081350a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:57.916 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.910+0000 7f890dc68700 1 -- 172.21.15.102:0/4096611726 shutdown_connections 2021-05-25T11:59:57.917 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.910+0000 7f890dc68700 1 -- 172.21.15.102:0/4096611726 >> 172.21.15.102:0/4096611726 conn(0x7f890809c930 msgr2=0x7f890809af80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T11:59:57.917 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.910+0000 7f890dc68700 1 -- 172.21.15.102:0/4096611726 shutdown_connections 2021-05-25T11:59:57.917 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.910+0000 7f890dc68700 1 -- 172.21.15.102:0/4096611726 wait complete. 2021-05-25T11:59:57.917 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:57.910+0000 7f890dc68700 1 librados: shutdown 2021-05-25T11:59:57.924 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640258 got 47244640258 for osd.3 2021-05-25T11:59:57.924 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2021-05-25T11:59:58.035 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.032+0000 7f0080017700 10 monclient: get_monmap_and_config 2021-05-25T11:59:58.035 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.032+0000 7f0080017700 10 monclient: build_initial_monmap 2021-05-25T11:59:58.036 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.032+0000 7f0080017700 10 monclient: monmap: 2021-05-25T11:59:58.036 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T11:59:58.036 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:58.036 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:58.034797+0000 2021-05-25T11:59:58.036 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:58.034797+0000 2021-05-25T11:59:58.037 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T11:59:58.037 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:58.037 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T11:59:58.037 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T11:59:58.037 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T11:59:58.038 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:58.049 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.033+0000 7f0080017700 1 Processor -- start 2021-05-25T11:59:58.049 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.033+0000 7f0080017700 1 -- start start 2021-05-25T11:59:58.049 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.033+0000 7f0080017700 10 monclient: init 2021-05-25T11:59:58.050 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.034+0000 7f0080017700 10 monclient: _reopen_session rank -1 2021-05-25T11:59:58.050 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.034+0000 7f0080017700 10 monclient: _add_conns ranks=[2,0,1] 2021-05-25T11:59:58.050 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.034+0000 7f0080017700 10 monclient(hunting): picked mon.noname-b con 0x7f0078093e10 addr v1:172.21.15.102:6790/0 2021-05-25T11:59:58.050 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.034+0000 7f0080017700 10 monclient(hunting): picked mon.noname-a con 0x7f0078091b00 addr v1:172.21.15.102:6789/0 2021-05-25T11:59:58.050 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.034+0000 7f0080017700 10 monclient(hunting): picked mon.noname-c con 0x7f0078094900 addr v1:172.21.15.181:6789/0 2021-05-25T11:59:58.051 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.034+0000 7f0080017700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0078092a90 con 0x7f0078091b00 2021-05-25T11:59:58.051 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.034+0000 7f0080017700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0078092d30 con 0x7f0078094900 2021-05-25T11:59:58.051 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.034+0000 7f0080017700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0078092fd0 con 0x7f0078093e10 2021-05-25T11:59:58.051 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.034+0000 7f0080017700 10 monclient(hunting): _renew_subs 2021-05-25T11:59:58.052 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.034+0000 7f0080017700 10 monclient(hunting): authenticate will time out at 798.632129s 2021-05-25T11:59:58.052 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.034+0000 7f007ddb3700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f0078093e10 0x7f0078093550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:53822/0 (socket says 172.21.15.102:53822) 2021-05-25T11:59:58.052 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.034+0000 7f007d5b2700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f0078091b00 0x7f00780941f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:58946/0 (socket says 172.21.15.102:58946) 2021-05-25T11:59:58.052 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.034+0000 7f007ddb3700 1 -- 172.21.15.102:0/3335304796 learned_addr learned my addr 172.21.15.102:0/3335304796 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T11:59:58.053 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.034+0000 7f007cdb1700 1 -- 172.21.15.102:0/3335304796 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4087759408 0 0) 0x7f0078092a90 con 0x7f0078091b00 2021-05-25T11:59:58.053 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.034+0000 7f007cdb1700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:58.053 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.034+0000 7f007cdb1700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:58.053 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.034+0000 7f007cdb1700 10 monclient(hunting): my global_id is 4326 2021-05-25T11:59:58.053 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.034+0000 7f007cdb1700 1 -- 172.21.15.102:0/3335304796 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0064004380 con 0x7f0078091b00 2021-05-25T11:59:58.054 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.034+0000 7f007cdb1700 1 -- 172.21.15.102:0/3335304796 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 577484938 0 0) 0x7f0078092fd0 con 0x7f0078093e10 2021-05-25T11:59:58.054 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.034+0000 7f007cdb1700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:58.054 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.034+0000 7f007cdb1700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:58.054 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.034+0000 7f007cdb1700 10 monclient(hunting): my global_id is 4319 2021-05-25T11:59:58.055 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.034+0000 7f007cdb1700 1 -- 172.21.15.102:0/3335304796 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0078092a90 con 0x7f0078093e10 2021-05-25T11:59:58.055 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.034+0000 7f007cdb1700 1 -- 172.21.15.102:0/3335304796 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1302374728 0 0) 0x7f0078092d30 con 0x7f0078094900 2021-05-25T11:59:58.055 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.034+0000 7f007cdb1700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:58.055 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.034+0000 7f007cdb1700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:58.055 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.034+0000 7f007cdb1700 10 monclient(hunting): my global_id is 4321 2021-05-25T11:59:58.056 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.034+0000 7f007cdb1700 1 -- 172.21.15.102:0/3335304796 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0078092fd0 con 0x7f0078094900 2021-05-25T11:59:58.056 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.034+0000 7f007cdb1700 1 -- 172.21.15.102:0/3335304796 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 72782357 0 0) 0x7f0078092a90 con 0x7f0078093e10 2021-05-25T11:59:58.056 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.034+0000 7f007cdb1700 1 -- 172.21.15.102:0/3335304796 >> v1:172.21.15.181:6789/0 conn(0x7f0078094900 legacy=0x7f0078130130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:58.056 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.034+0000 7f007cdb1700 1 -- 172.21.15.102:0/3335304796 >> v1:172.21.15.102:6789/0 conn(0x7f0078091b00 legacy=0x7f00780941f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:58.056 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.034+0000 7f007cdb1700 10 monclient: _finish_hunting 0 2021-05-25T11:59:58.057 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.034+0000 7f007cdb1700 1 monclient: found mon.noname-b 2021-05-25T11:59:58.057 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.034+0000 7f007cdb1700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T11:59:58.057 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.034+0000 7f007cdb1700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T11:59:58.057 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.034+0000 7f007cdb1700 1 -- 172.21.15.102:0/3335304796 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0078134880 con 0x7f0078093e10 2021-05-25T11:59:58.058 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.034+0000 7f007cdb1700 10 monclient: _finish_auth 0 2021-05-25T11:59:58.058 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.034+0000 7f007cdb1700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T11:59:58.058 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.034+0000 7f0080017700 5 monclient: authenticate success, global_id 4319 2021-05-25T11:59:58.058 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.034+0000 7f007cdb1700 1 -- 172.21.15.102:0/3335304796 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f00680015f0 con 0x7f0078093e10 2021-05-25T11:59:58.058 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.034+0000 7f007cdb1700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:58.059 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.034+0000 7f007cdb1700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T11:59:58.059 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.034+0000 7f007cdb1700 10 monclient: dump: 2021-05-25T11:59:58.059 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:58.059 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:58.059 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:58.060 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:58.060 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:58.060 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:58.060 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:58.060 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:58.061 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:58.061 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:58.061 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.034+0000 7f0080017700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T11:59:58.061 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.034+0000 7f007cdb1700 1 -- 172.21.15.102:0/3335304796 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0068001810 con 0x7f0078093e10 2021-05-25T11:59:58.061 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.034+0000 7f007cdb1700 10 monclient: handle_config config(0 keys) v1 2021-05-25T11:59:58.062 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.034+0000 7f007cdb1700 1 -- 172.21.15.102:0/3335304796 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f0068001b60 con 0x7f0078093e10 2021-05-25T11:59:58.062 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.034+0000 7f0080017700 10 monclient: get_monmap_and_config success 2021-05-25T11:59:58.062 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.034+0000 7f0080017700 4 set_mon_vals no callback set 2021-05-25T11:59:58.062 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.034+0000 7f007cdb1700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:58.063 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.034+0000 7f007cdb1700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T11:59:58.063 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.034+0000 7f007cdb1700 10 monclient: dump: 2021-05-25T11:59:58.063 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:58.063 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:58.063 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:58.064 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:58.064 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:58.064 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:58.064 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:58.064 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:58.065 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:58.065 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:58.065 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.035+0000 7f0080017700 10 monclient: shutdown 2021-05-25T11:59:58.065 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.035+0000 7f0080017700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T11:59:58.065 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.035+0000 7f0080017700 1 -- 172.21.15.102:0/3335304796 >> v1:172.21.15.102:6790/0 conn(0x7f0078093e10 legacy=0x7f0078093550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:58.066 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.035+0000 7f0080017700 1 -- 172.21.15.102:0/3335304796 shutdown_connections 2021-05-25T11:59:58.066 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.035+0000 7f0080017700 1 -- 172.21.15.102:0/3335304796 >> 172.21.15.102:0/3335304796 conn(0x7f007809a910 msgr2=0x7f007809ad20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T11:59:58.066 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.035+0000 7f0080017700 1 -- 172.21.15.102:0/3335304796 shutdown_connections 2021-05-25T11:59:58.066 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.035+0000 7f0080017700 1 -- 172.21.15.102:0/3335304796 wait complete. 2021-05-25T11:59:58.067 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.035+0000 7f0080017700 10 monclient: build_initial_monmap 2021-05-25T11:59:58.067 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.035+0000 7f0080017700 10 monclient: monmap: 2021-05-25T11:59:58.067 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T11:59:58.067 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T11:59:58.067 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T11:59:58.068 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T11:59:58.068 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T11:59:58.068 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T11:59:58.068 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T11:59:58.069 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T11:59:58.069 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T11:59:58.069 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:58.069 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.036+0000 7f0080017700 1 librados: starting msgr at 2021-05-25T11:59:58.069 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.036+0000 7f0080017700 1 librados: starting objecter 2021-05-25T11:59:58.070 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.036+0000 7f0080017700 1 Processor -- start 2021-05-25T11:59:58.070 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.036+0000 7f0080017700 1 -- start start 2021-05-25T11:59:58.070 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.036+0000 7f0080017700 1 librados: setting wanted keys 2021-05-25T11:59:58.070 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.036+0000 7f0080017700 1 librados: calling monclient init 2021-05-25T11:59:58.070 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.036+0000 7f0080017700 10 monclient: init 2021-05-25T11:59:58.071 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.036+0000 7f0080017700 10 monclient: _reopen_session rank -1 2021-05-25T11:59:58.071 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.036+0000 7f0080017700 10 monclient: _add_conns ranks=[2,0,1] 2021-05-25T11:59:58.071 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.036+0000 7f0080017700 10 monclient(hunting): picked mon.noname-c con 0x7f0078091b00 addr v1:172.21.15.102:6790/0 2021-05-25T11:59:58.071 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.036+0000 7f0080017700 10 monclient(hunting): picked mon.noname-a con 0x7f0078093e10 addr v1:172.21.15.102:6789/0 2021-05-25T11:59:58.071 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.036+0000 7f0080017700 10 monclient(hunting): picked mon.noname-b con 0x7f0078094900 addr v1:172.21.15.181:6789/0 2021-05-25T11:59:58.072 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.036+0000 7f0080017700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0078132dd0 con 0x7f0078093e10 2021-05-25T11:59:58.072 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.036+0000 7f0080017700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0078133070 con 0x7f0078094900 2021-05-25T11:59:58.072 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.036+0000 7f0080017700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0078133310 con 0x7f0078091b00 2021-05-25T11:59:58.072 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.036+0000 7f0080017700 10 monclient(hunting): _renew_subs 2021-05-25T11:59:58.073 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.036+0000 7f0080017700 10 monclient(hunting): authenticate will time out at 798.634632s 2021-05-25T11:59:58.073 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.036+0000 7f007d5b2700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f0078093e10 0x7f007813ff90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:58950/0 (socket says 172.21.15.102:58950) 2021-05-25T11:59:58.073 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.036+0000 7f007d5b2700 1 -- 172.21.15.102:0/1120169458 learned_addr learned my addr 172.21.15.102:0/1120169458 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T11:59:58.073 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.036+0000 7f007ddb3700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f0078091b00 0x7f0078130350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:53830/0 (socket says 172.21.15.102:53830) 2021-05-25T11:59:58.073 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.036+0000 7f006dffb700 10 client.?.objecter ms_handle_connect 0x7f0078093e10 2021-05-25T11:59:58.074 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.036+0000 7f006dffb700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:58.074 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.036+0000 7f006dffb700 10 client.?.objecter ms_handle_connect 0x7f0078091b00 2021-05-25T11:59:58.074 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.036+0000 7f006dffb700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:58.074 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.036+0000 7f006dffb700 1 -- 172.21.15.102:0/1120169458 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 398472917 0 0) 0x7f0078132dd0 con 0x7f0078093e10 2021-05-25T11:59:58.075 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.036+0000 7f006dffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:58.075 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.036+0000 7f006dffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:58.075 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.036+0000 7f006dffb700 10 monclient(hunting): my global_id is 4329 2021-05-25T11:59:58.075 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.036+0000 7f006dffb700 1 -- 172.21.15.102:0/1120169458 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0058004c10 con 0x7f0078093e10 2021-05-25T11:59:58.075 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.036+0000 7f006dffb700 10 client.?.objecter ms_handle_connect 0x7f0078094900 2021-05-25T11:59:58.076 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.036+0000 7f006dffb700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:58.076 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.036+0000 7f006dffb700 1 -- 172.21.15.102:0/1120169458 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1413566439 0 0) 0x7f0078133310 con 0x7f0078091b00 2021-05-25T11:59:58.076 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.036+0000 7f006dffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:58.076 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.036+0000 7f006dffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:58.076 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.036+0000 7f006dffb700 10 monclient(hunting): my global_id is 4322 2021-05-25T11:59:58.077 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.036+0000 7f006dffb700 1 -- 172.21.15.102:0/1120169458 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0078132dd0 con 0x7f0078091b00 2021-05-25T11:59:58.077 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.037+0000 7f006dffb700 1 -- 172.21.15.102:0/1120169458 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2557321647 0 0) 0x7f0078133070 con 0x7f0078094900 2021-05-25T11:59:58.077 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.037+0000 7f006dffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:58.077 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.037+0000 7f006dffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:58.078 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.037+0000 7f006dffb700 10 monclient(hunting): my global_id is 4324 2021-05-25T11:59:58.078 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.037+0000 7f006dffb700 1 -- 172.21.15.102:0/1120169458 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0078133310 con 0x7f0078094900 2021-05-25T11:59:58.078 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.037+0000 7f006dffb700 1 -- 172.21.15.102:0/1120169458 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1952184145 0 0) 0x7f0058004c10 con 0x7f0078093e10 2021-05-25T11:59:58.078 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.037+0000 7f006dffb700 1 -- 172.21.15.102:0/1120169458 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0078133070 con 0x7f0078093e10 2021-05-25T11:59:58.079 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.037+0000 7f006dffb700 1 -- 172.21.15.102:0/1120169458 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f00600019f0 con 0x7f0078093e10 2021-05-25T11:59:58.079 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.037+0000 7f006dffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:58.079 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.037+0000 7f006dffb700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T11:59:58.079 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.037+0000 7f006dffb700 10 monclient(hunting): dump: 2021-05-25T11:59:58.079 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:58.080 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:58.080 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:58.080 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:58.080 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:58.081 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:58.081 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:58.081 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:58.081 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:58.081 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:58.082 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.037+0000 7f006dffb700 10 monclient(hunting): _finish_auth 0 2021-05-25T11:59:58.082 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.037+0000 7f006dffb700 1 -- 172.21.15.102:0/1120169458 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3938404632 0 0) 0x7f0078132dd0 con 0x7f0078091b00 2021-05-25T11:59:58.082 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.037+0000 7f006dffb700 1 -- 172.21.15.102:0/1120169458 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0058004c10 con 0x7f0078091b00 2021-05-25T11:59:58.082 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.037+0000 7f006dffb700 1 -- 172.21.15.102:0/1120169458 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f00680014f0 con 0x7f0078091b00 2021-05-25T11:59:58.083 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.037+0000 7f006dffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:58.083 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.037+0000 7f006dffb700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2021-05-25T11:59:58.083 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.037+0000 7f006dffb700 10 monclient(hunting): dump: 2021-05-25T11:59:58.083 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:58.083 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:58.084 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:58.084 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:58.084 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:58.084 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:58.084 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:58.085 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:58.085 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:58.085 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:58.085 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.037+0000 7f006dffb700 1 -- 172.21.15.102:0/1120169458 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 840048803 0 0) 0x7f0078133310 con 0x7f0078094900 2021-05-25T11:59:58.086 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.037+0000 7f006dffb700 1 -- 172.21.15.102:0/1120169458 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0078132dd0 con 0x7f0078094900 2021-05-25T11:59:58.086 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.037+0000 7f006dffb700 1 -- 172.21.15.102:0/1120169458 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f0074001c00 con 0x7f0078094900 2021-05-25T11:59:58.086 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.037+0000 7f006dffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:58.086 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.037+0000 7f006dffb700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2021-05-25T11:59:58.086 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.037+0000 7f006dffb700 10 monclient(hunting): dump: 2021-05-25T11:59:58.087 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:58.087 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:58.087 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:58.087 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:58.088 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:58.088 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:58.088 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:58.088 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:58.088 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:58.089 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:58.089 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.037+0000 7f006dffb700 1 -- 172.21.15.102:0/1120169458 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1461848036 0 0) 0x7f0078133070 con 0x7f0078093e10 2021-05-25T11:59:58.089 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.037+0000 7f006dffb700 1 -- 172.21.15.102:0/1120169458 >> v1:172.21.15.102:6790/0 conn(0x7f0078091b00 legacy=0x7f0078130350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:58.089 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.037+0000 7f006dffb700 1 -- 172.21.15.102:0/1120169458 >> v1:172.21.15.181:6789/0 conn(0x7f0078094900 legacy=0x7f00781436c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:58.089 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.037+0000 7f006dffb700 10 monclient: _finish_hunting 0 2021-05-25T11:59:58.090 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.037+0000 7f006dffb700 1 monclient: found mon.a 2021-05-25T11:59:58.090 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.037+0000 7f006dffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T11:59:58.090 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.037+0000 7f006dffb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:58.090 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.037+0000 7f006dffb700 1 -- 172.21.15.102:0/1120169458 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00781313f0 con 0x7f0078093e10 2021-05-25T11:59:58.091 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.037+0000 7f006dffb700 10 monclient: _finish_auth 0 2021-05-25T11:59:58.091 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.037+0000 7f006dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T11:59:58.091 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.037+0000 7f0080017700 5 monclient: authenticate success, global_id 4329 2021-05-25T11:59:58.091 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.037+0000 7f0080017700 10 monclient: _renew_subs 2021-05-25T11:59:58.091 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.037+0000 7f0080017700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:58.092 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.037+0000 7f0080017700 1 -- 172.21.15.102:0/1120169458 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0078130a60 con 0x7f0078093e10 2021-05-25T11:59:58.092 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.037+0000 7f006dffb700 1 -- 172.21.15.102:0/1120169458 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0060001cc0 con 0x7f0078093e10 2021-05-25T11:59:58.092 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.037+0000 7f006dffb700 10 monclient: handle_config config(0 keys) v1 2021-05-25T11:59:58.092 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.037+0000 7f006dffb700 1 -- 172.21.15.102:0/1120169458 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f0060004980 con 0x7f0078093e10 2021-05-25T11:59:58.093 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.037+0000 7f006dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:58.093 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.037+0000 7f006dffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T11:59:58.093 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.037+0000 7f006dffb700 10 monclient: dump: 2021-05-25T11:59:58.093 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:58.093 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:58.094 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:58.094 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:58.094 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:58.094 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:58.095 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:58.095 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:58.095 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:58.095 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:58.095 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.037+0000 7f0080017700 10 client.4329.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T11:59:58.096 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.037+0000 7f0080017700 10 monclient: _renew_subs 2021-05-25T11:59:58.096 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.037+0000 7f006ffff700 4 set_mon_vals no callback set 2021-05-25T11:59:58.096 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.037+0000 7f0080017700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:58.096 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.037+0000 7f0080017700 1 -- 172.21.15.102:0/1120169458 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0078133c10 con 0x7f0078093e10 2021-05-25T11:59:58.097 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.037+0000 7f0080017700 1 librados: init done 2021-05-25T11:59:58.097 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.038+0000 7f006dffb700 1 -- 172.21.15.102:0/1120169458 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f006000dca0 con 0x7f0078093e10 2021-05-25T11:59:58.097 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.038+0000 7f006dffb700 1 -- 172.21.15.102:0/1120169458 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(15..15 src has 1..15) v4 ==== 3549+0+0 (unknown 3948819498 0 0) 0x7f006003ce70 con 0x7f0078093e10 2021-05-25T11:59:58.097 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.038+0000 7f006dffb700 10 client.4329.objecter ms_dispatch 0x7f0078001550 osd_map(15..15 src has 1..15) v4 2021-05-25T11:59:58.097 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.038+0000 7f006dffb700 3 client.4329.objecter handle_osd_map got epochs [15,15] > 0 2021-05-25T11:59:58.098 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.038+0000 7f006dffb700 3 client.4329.objecter handle_osd_map decoding full epoch 15 2021-05-25T11:59:58.098 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.038+0000 7f006dffb700 20 client.4329.objecter dump_active .. 0 homeless 2021-05-25T11:59:58.098 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.038+0000 7f0080017700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:58.098 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.038+0000 7f0080017700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:58.099 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.038+0000 7f0080017700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:58.099 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.038+0000 7f0080017700 1 -- 172.21.15.102:0/1120169458 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0078068210 con 0x7f0078093e10 2021-05-25T11:59:58.099 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.038+0000 7f007ddb3700 10 monclient: get_auth_request con 0x7f0058033540 auth_method 0 2021-05-25T11:59:58.099 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.038+0000 7f006dffb700 10 client.4329.objecter ms_handle_connect 0x7f0058033540 2021-05-25T11:59:58.099 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.042+0000 7f006dffb700 1 -- 172.21.15.102:0/1120169458 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7f0060012d60 con 0x7f0078093e10 2021-05-25T11:59:58.100 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.042+0000 7f006dffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:58.100 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.042+0000 7f006dffb700 10 monclient: _finish_command 1 = system:0 2021-05-25T11:59:58.193 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.190+0000 7f0080017700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2021-05-25T11:59:58.193 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.190+0000 7f0080017700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2021-05-25T11:59:58.194 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.190+0000 7f0080017700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:58.194 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.190+0000 7f0080017700 1 -- 172.21.15.102:0/1120169458 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f0078059730 con 0x7f0078093e10 2021-05-25T11:59:58.194 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.190+0000 7f006dffb700 1 -- 172.21.15.102:0/1120169458 <== mon.0 v1:172.21.15.102:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (unknown 1823589948 0 847495621) 0x7f0060012950 con 0x7f0078093e10 2021-05-25T11:59:58.194 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.190+0000 7f006dffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2021-05-25T11:59:58.195 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.190+0000 7f006dffb700 10 monclient: _finish_command 2 = system:0 2021-05-25T11:59:58.195 INFO:teuthology.orchestra.run.smithi102.stdout:47244640259 2021-05-25T11:59:58.197 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.192+0000 7f0080017700 10 librados: watch_flush enter 2021-05-25T11:59:58.197 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.192+0000 7f0080017700 10 librados: watch_flush exit 2021-05-25T11:59:58.198 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.192+0000 7f0080017700 20 client.4329.objecter shutdown clearing up homeless session... 2021-05-25T11:59:58.198 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.192+0000 7f0080017700 10 client.4329.objecter successfully canceled tick 2021-05-25T11:59:58.198 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.192+0000 7f0080017700 1 -- 172.21.15.102:0/1120169458 >> v1:172.21.15.181:6812/29582 conn(0x7f0058033540 legacy=0x7f0058035a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:58.199 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.192+0000 7f0080017700 10 monclient: shutdown 2021-05-25T11:59:58.199 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.192+0000 7f0080017700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T11:59:58.199 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.192+0000 7f0080017700 1 -- 172.21.15.102:0/1120169458 >> v1:172.21.15.102:6789/0 conn(0x7f0078093e10 legacy=0x7f007813ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:58.200 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.192+0000 7f0080017700 1 -- 172.21.15.102:0/1120169458 shutdown_connections 2021-05-25T11:59:58.200 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.192+0000 7f0080017700 1 -- 172.21.15.102:0/1120169458 >> 172.21.15.102:0/1120169458 conn(0x7f007809a910 msgr2=0x7f007812e710 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T11:59:58.200 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.192+0000 7f0080017700 1 -- 172.21.15.102:0/1120169458 shutdown_connections 2021-05-25T11:59:58.201 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.192+0000 7f0080017700 1 -- 172.21.15.102:0/1120169458 wait complete. 2021-05-25T11:59:58.201 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.192+0000 7f0080017700 1 librados: shutdown 2021-05-25T11:59:58.206 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640258 got 47244640259 for osd.4 2021-05-25T11:59:58.207 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2021-05-25T11:59:58.317 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.315+0000 7f53562c2700 10 monclient: get_monmap_and_config 2021-05-25T11:59:58.317 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.315+0000 7f53562c2700 10 monclient: build_initial_monmap 2021-05-25T11:59:58.318 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.315+0000 7f53562c2700 10 monclient: monmap: 2021-05-25T11:59:58.318 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T11:59:58.318 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:58.318 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:58.316951+0000 2021-05-25T11:59:58.318 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:58.316951+0000 2021-05-25T11:59:58.319 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T11:59:58.319 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:58.320 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T11:59:58.320 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T11:59:58.320 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T11:59:58.321 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:58.331 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.315+0000 7f53562c2700 1 Processor -- start 2021-05-25T11:59:58.332 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.315+0000 7f53562c2700 1 -- start start 2021-05-25T11:59:58.332 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.315+0000 7f53562c2700 10 monclient: init 2021-05-25T11:59:58.332 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.316+0000 7f53562c2700 10 monclient: _reopen_session rank -1 2021-05-25T11:59:58.332 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.316+0000 7f53562c2700 10 monclient: _add_conns ranks=[0,2,1] 2021-05-25T11:59:58.333 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.316+0000 7f53562c2700 10 monclient(hunting): picked mon.noname-a con 0x7f535009a5e0 addr v1:172.21.15.102:6789/0 2021-05-25T11:59:58.333 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.316+0000 7f53562c2700 10 monclient(hunting): picked mon.noname-b con 0x7f5350063dc0 addr v1:172.21.15.102:6790/0 2021-05-25T11:59:58.333 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.316+0000 7f53562c2700 10 monclient(hunting): picked mon.noname-c con 0x7f53501325c0 addr v1:172.21.15.181:6789/0 2021-05-25T11:59:58.333 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.316+0000 7f53562c2700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5350099fe0 con 0x7f535009a5e0 2021-05-25T11:59:58.333 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.316+0000 7f53562c2700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f535009a1e0 con 0x7f53501325c0 2021-05-25T11:59:58.334 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.316+0000 7f53562c2700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f535009b400 con 0x7f5350063dc0 2021-05-25T11:59:58.334 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.316+0000 7f53562c2700 10 monclient(hunting): _renew_subs 2021-05-25T11:59:58.335 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.316+0000 7f53562c2700 10 monclient(hunting): authenticate will time out at 798.914205s 2021-05-25T11:59:58.336 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.316+0000 7f534f7fe700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f5350063dc0 0x7f5350064240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:53836/0 (socket says 172.21.15.102:53836) 2021-05-25T11:59:58.336 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.316+0000 7f534ffff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f535009a5e0 0x7f53500636b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:58958/0 (socket says 172.21.15.102:58958) 2021-05-25T11:59:58.336 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.316+0000 7f534f7fe700 1 -- 172.21.15.102:0/1473203035 learned_addr learned my addr 172.21.15.102:0/1473203035 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T11:59:58.336 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.316+0000 7f534effd700 1 -- 172.21.15.102:0/1473203035 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2661026700 0 0) 0x7f5350099fe0 con 0x7f535009a5e0 2021-05-25T11:59:58.337 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.316+0000 7f534effd700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:58.337 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.316+0000 7f534effd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:58.337 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.316+0000 7f534effd700 10 monclient(hunting): my global_id is 4332 2021-05-25T11:59:58.337 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.316+0000 7f534effd700 1 -- 172.21.15.102:0/1473203035 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5338003270 con 0x7f535009a5e0 2021-05-25T11:59:58.337 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.316+0000 7f534effd700 1 -- 172.21.15.102:0/1473203035 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3933341879 0 0) 0x7f535009b400 con 0x7f5350063dc0 2021-05-25T11:59:58.338 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.316+0000 7f534effd700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:58.338 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.316+0000 7f534effd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:58.338 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.316+0000 7f534effd700 10 monclient(hunting): my global_id is 4325 2021-05-25T11:59:58.339 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.316+0000 7f534effd700 1 -- 172.21.15.102:0/1473203035 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5350099fe0 con 0x7f5350063dc0 2021-05-25T11:59:58.339 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.316+0000 7f534effd700 1 -- 172.21.15.102:0/1473203035 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1090014513 0 0) 0x7f5350099fe0 con 0x7f5350063dc0 2021-05-25T11:59:58.339 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.316+0000 7f534effd700 1 -- 172.21.15.102:0/1473203035 >> v1:172.21.15.181:6789/0 conn(0x7f53501325c0 legacy=0x7f53501349b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:58.339 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.316+0000 7f534effd700 1 -- 172.21.15.102:0/1473203035 >> v1:172.21.15.102:6789/0 conn(0x7f535009a5e0 legacy=0x7f53500636b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:58.339 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.316+0000 7f534effd700 10 monclient: _finish_hunting 0 2021-05-25T11:59:58.340 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.316+0000 7f534effd700 1 monclient: found mon.noname-b 2021-05-25T11:59:58.340 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.316+0000 7f534effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T11:59:58.340 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.316+0000 7f534effd700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T11:59:58.340 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.316+0000 7f534effd700 1 -- 172.21.15.102:0/1473203035 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53500963f0 con 0x7f5350063dc0 2021-05-25T11:59:58.340 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.316+0000 7f534effd700 10 monclient: _finish_auth 0 2021-05-25T11:59:58.341 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.316+0000 7f534effd700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T11:59:58.341 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.316+0000 7f534effd700 1 -- 172.21.15.102:0/1473203035 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f53340015d0 con 0x7f5350063dc0 2021-05-25T11:59:58.341 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.316+0000 7f53562c2700 5 monclient: authenticate success, global_id 4325 2021-05-25T11:59:58.341 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.316+0000 7f53562c2700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T11:59:58.341 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.316+0000 7f534effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:58.342 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.316+0000 7f534effd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T11:59:58.342 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.316+0000 7f534effd700 10 monclient: dump: 2021-05-25T11:59:58.342 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:58.342 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:58.343 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:58.343 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:58.343 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:58.343 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:58.343 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:58.344 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:58.344 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:58.344 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:58.344 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.316+0000 7f53562c2700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T11:59:58.344 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.316+0000 7f534effd700 1 -- 172.21.15.102:0/1473203035 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5334001730 con 0x7f5350063dc0 2021-05-25T11:59:58.345 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.316+0000 7f534effd700 10 monclient: handle_config config(0 keys) v1 2021-05-25T11:59:58.345 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.316+0000 7f534effd700 1 -- 172.21.15.102:0/1473203035 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f5334001ac0 con 0x7f5350063dc0 2021-05-25T11:59:58.345 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.316+0000 7f53562c2700 10 monclient: get_monmap_and_config success 2021-05-25T11:59:58.345 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.316+0000 7f53562c2700 4 set_mon_vals no callback set 2021-05-25T11:59:58.345 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.316+0000 7f534effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:58.346 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.316+0000 7f534effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T11:59:58.346 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.316+0000 7f534effd700 10 monclient: dump: 2021-05-25T11:59:58.346 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:58.346 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:58.346 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:58.347 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:58.347 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:58.347 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:58.347 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:58.347 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:58.348 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:58.348 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:58.348 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.317+0000 7f53562c2700 10 monclient: shutdown 2021-05-25T11:59:58.348 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.317+0000 7f53562c2700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T11:59:58.349 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.317+0000 7f53562c2700 1 -- 172.21.15.102:0/1473203035 >> v1:172.21.15.102:6790/0 conn(0x7f5350063dc0 legacy=0x7f5350064240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:58.349 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.317+0000 7f53562c2700 1 -- 172.21.15.102:0/1473203035 shutdown_connections 2021-05-25T11:59:58.349 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.317+0000 7f53562c2700 1 -- 172.21.15.102:0/1473203035 >> 172.21.15.102:0/1473203035 conn(0x7f535009bc30 msgr2=0x7f535009c040 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T11:59:58.349 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.317+0000 7f53562c2700 1 -- 172.21.15.102:0/1473203035 shutdown_connections 2021-05-25T11:59:58.349 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.317+0000 7f53562c2700 1 -- 172.21.15.102:0/1473203035 wait complete. 2021-05-25T11:59:58.350 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.317+0000 7f53562c2700 10 monclient: build_initial_monmap 2021-05-25T11:59:58.350 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.317+0000 7f53562c2700 10 monclient: monmap: 2021-05-25T11:59:58.350 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T11:59:58.350 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T11:59:58.350 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T11:59:58.351 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T11:59:58.351 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T11:59:58.351 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T11:59:58.351 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T11:59:58.352 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T11:59:58.352 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T11:59:58.352 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:58.352 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.318+0000 7f53562c2700 1 librados: starting msgr at 2021-05-25T11:59:58.352 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.318+0000 7f53562c2700 1 librados: starting objecter 2021-05-25T11:59:58.353 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.318+0000 7f53562c2700 1 Processor -- start 2021-05-25T11:59:58.353 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.318+0000 7f53562c2700 1 -- start start 2021-05-25T11:59:58.353 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.318+0000 7f53562c2700 1 librados: setting wanted keys 2021-05-25T11:59:58.353 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.318+0000 7f53562c2700 1 librados: calling monclient init 2021-05-25T11:59:58.353 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.318+0000 7f53562c2700 10 monclient: init 2021-05-25T11:59:58.354 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.318+0000 7f53562c2700 10 monclient: _reopen_session rank -1 2021-05-25T11:59:58.354 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.318+0000 7f53562c2700 10 monclient: _add_conns ranks=[2,1,0] 2021-05-25T11:59:58.354 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.318+0000 7f53562c2700 10 monclient(hunting): picked mon.noname-c con 0x7f5350063dc0 addr v1:172.21.15.102:6790/0 2021-05-25T11:59:58.354 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.318+0000 7f53562c2700 10 monclient(hunting): picked mon.noname-b con 0x7f535009a5e0 addr v1:172.21.15.181:6789/0 2021-05-25T11:59:58.355 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.318+0000 7f53562c2700 10 monclient(hunting): picked mon.noname-a con 0x7f53501325c0 addr v1:172.21.15.102:6789/0 2021-05-25T11:59:58.355 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.318+0000 7f53562c2700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f535009ab10 con 0x7f53501325c0 2021-05-25T11:59:58.355 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.318+0000 7f53562c2700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f535013bc80 con 0x7f535009a5e0 2021-05-25T11:59:58.355 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.318+0000 7f53562c2700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5350117200 con 0x7f5350063dc0 2021-05-25T11:59:58.355 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.318+0000 7f53562c2700 10 monclient(hunting): _renew_subs 2021-05-25T11:59:58.356 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.318+0000 7f53562c2700 10 monclient(hunting): authenticate will time out at 798.916920s 2021-05-25T11:59:58.356 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.318+0000 7f534ffff700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f5350063dc0 0x7f535013ae60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:53842/0 (socket says 172.21.15.102:53842) 2021-05-25T11:59:58.356 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.318+0000 7f534ffff700 1 -- 172.21.15.102:0/3025085415 learned_addr learned my addr 172.21.15.102:0/3025085415 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T11:59:58.356 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.319+0000 7f5343fff700 10 client.?.objecter ms_handle_connect 0x7f5350063dc0 2021-05-25T11:59:58.357 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.319+0000 7f5343fff700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:58.357 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.319+0000 7f5343fff700 10 client.?.objecter ms_handle_connect 0x7f53501325c0 2021-05-25T11:59:58.357 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.319+0000 7f5343fff700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:58.357 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.319+0000 7f5343fff700 1 -- 172.21.15.102:0/3025085415 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3269438787 0 0) 0x7f5350117200 con 0x7f5350063dc0 2021-05-25T11:59:58.357 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.319+0000 7f5343fff700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:58.358 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.319+0000 7f5343fff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:58.358 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.319+0000 7f5343fff700 10 monclient(hunting): my global_id is 4328 2021-05-25T11:59:58.358 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.319+0000 7f5343fff700 1 -- 172.21.15.102:0/3025085415 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f532c004c10 con 0x7f5350063dc0 2021-05-25T11:59:58.358 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.319+0000 7f5343fff700 10 client.?.objecter ms_handle_connect 0x7f535009a5e0 2021-05-25T11:59:58.358 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.319+0000 7f5343fff700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:58.359 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.319+0000 7f5343fff700 1 -- 172.21.15.102:0/3025085415 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1152408392 0 0) 0x7f535009ab10 con 0x7f53501325c0 2021-05-25T11:59:58.359 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.319+0000 7f5343fff700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:58.359 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.319+0000 7f5343fff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:58.359 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.319+0000 7f5343fff700 10 monclient(hunting): my global_id is 4335 2021-05-25T11:59:58.360 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.319+0000 7f5343fff700 1 -- 172.21.15.102:0/3025085415 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5350117200 con 0x7f53501325c0 2021-05-25T11:59:58.360 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.319+0000 7f5343fff700 1 -- 172.21.15.102:0/3025085415 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3604196917 0 0) 0x7f535013bc80 con 0x7f535009a5e0 2021-05-25T11:59:58.360 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.319+0000 7f5343fff700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:58.360 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.319+0000 7f5343fff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:58.360 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.319+0000 7f5343fff700 10 monclient(hunting): my global_id is 4330 2021-05-25T11:59:58.361 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.319+0000 7f5343fff700 1 -- 172.21.15.102:0/3025085415 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f535009ab10 con 0x7f535009a5e0 2021-05-25T11:59:58.361 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.319+0000 7f5343fff700 1 -- 172.21.15.102:0/3025085415 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3785618045 0 0) 0x7f532c004c10 con 0x7f5350063dc0 2021-05-25T11:59:58.361 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.319+0000 7f5343fff700 1 -- 172.21.15.102:0/3025085415 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f535013bc80 con 0x7f5350063dc0 2021-05-25T11:59:58.361 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.319+0000 7f5343fff700 1 -- 172.21.15.102:0/3025085415 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f533c001a30 con 0x7f5350063dc0 2021-05-25T11:59:58.361 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.319+0000 7f5343fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:58.362 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.319+0000 7f5343fff700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T11:59:58.362 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.319+0000 7f5343fff700 10 monclient(hunting): dump: 2021-05-25T11:59:58.362 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:58.362 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:58.362 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:58.363 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:58.363 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:58.363 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:58.363 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:58.363 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:58.364 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:58.364 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:58.364 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.319+0000 7f5343fff700 10 monclient(hunting): _finish_auth 0 2021-05-25T11:59:58.364 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.319+0000 7f5343fff700 1 -- 172.21.15.102:0/3025085415 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1716363856 0 0) 0x7f5350117200 con 0x7f53501325c0 2021-05-25T11:59:58.365 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.319+0000 7f5343fff700 1 -- 172.21.15.102:0/3025085415 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f532c004c10 con 0x7f53501325c0 2021-05-25T11:59:58.365 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.319+0000 7f5343fff700 1 -- 172.21.15.102:0/3025085415 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f5344001b80 con 0x7f53501325c0 2021-05-25T11:59:58.365 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.319+0000 7f5343fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:58.365 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.319+0000 7f5343fff700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2021-05-25T11:59:58.365 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.319+0000 7f5343fff700 10 monclient(hunting): dump: 2021-05-25T11:59:58.366 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:58.366 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:58.366 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:58.366 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:58.366 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:58.367 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:58.367 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:58.367 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:58.367 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:58.367 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:58.368 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.319+0000 7f5343fff700 1 -- 172.21.15.102:0/3025085415 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 305502946 0 0) 0x7f535013bc80 con 0x7f5350063dc0 2021-05-25T11:59:58.368 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.319+0000 7f5343fff700 1 -- 172.21.15.102:0/3025085415 >> v1:172.21.15.181:6789/0 conn(0x7f535009a5e0 legacy=0x7f535013b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:58.368 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.319+0000 7f5343fff700 1 -- 172.21.15.102:0/3025085415 >> v1:172.21.15.102:6789/0 conn(0x7f53501325c0 legacy=0x7f53501431f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:58.368 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.319+0000 7f5343fff700 10 monclient: _finish_hunting 0 2021-05-25T11:59:58.368 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.319+0000 7f5343fff700 1 monclient: found mon.c 2021-05-25T11:59:58.369 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.319+0000 7f5343fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T11:59:58.369 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.319+0000 7f5343fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T11:59:58.369 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.319+0000 7f5343fff700 1 -- 172.21.15.102:0/3025085415 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53501174a0 con 0x7f5350063dc0 2021-05-25T11:59:58.369 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.319+0000 7f5343fff700 10 monclient: _finish_auth 0 2021-05-25T11:59:58.369 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.319+0000 7f5343fff700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T11:59:58.370 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.319+0000 7f53562c2700 5 monclient: authenticate success, global_id 4328 2021-05-25T11:59:58.370 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.319+0000 7f53562c2700 10 monclient: _renew_subs 2021-05-25T11:59:58.370 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.319+0000 7f53562c2700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T11:59:58.370 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.319+0000 7f53562c2700 1 -- 172.21.15.102:0/3025085415 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5350116040 con 0x7f5350063dc0 2021-05-25T11:59:58.370 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.319+0000 7f53562c2700 10 client.4328.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T11:59:58.371 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.319+0000 7f53562c2700 10 monclient: _renew_subs 2021-05-25T11:59:58.371 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.319+0000 7f53562c2700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T11:59:58.371 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.319+0000 7f53562c2700 1 -- 172.21.15.102:0/3025085415 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f535012cfb0 con 0x7f5350063dc0 2021-05-25T11:59:58.371 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.319+0000 7f53562c2700 1 librados: init done 2021-05-25T11:59:58.372 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.319+0000 7f5343fff700 1 -- 172.21.15.102:0/3025085415 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f533c0047e0 con 0x7f5350063dc0 2021-05-25T11:59:58.372 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.319+0000 7f5343fff700 10 monclient: handle_config config(0 keys) v1 2021-05-25T11:59:58.372 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.319+0000 7f5343fff700 1 -- 172.21.15.102:0/3025085415 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f533c004af0 con 0x7f5350063dc0 2021-05-25T11:59:58.372 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.319+0000 7f5343fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:58.373 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.319+0000 7f5343fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T11:59:58.373 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.319+0000 7f534e7fc700 4 set_mon_vals no callback set 2021-05-25T11:59:58.373 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.319+0000 7f5343fff700 10 monclient: dump: 2021-05-25T11:59:58.373 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:58.373 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:58.374 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:58.374 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:58.374 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:58.374 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:58.375 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:58.375 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:58.375 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:58.375 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:58.375 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.320+0000 7f5343fff700 1 -- 172.21.15.102:0/3025085415 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f533c00de10 con 0x7f5350063dc0 2021-05-25T11:59:58.376 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.320+0000 7f53562c2700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:58.376 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.320+0000 7f53562c2700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:58.376 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.320+0000 7f53562c2700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T11:59:58.376 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.320+0000 7f53562c2700 1 -- 172.21.15.102:0/3025085415 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5330004db0 con 0x7f5350063dc0 2021-05-25T11:59:58.376 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.320+0000 7f5343fff700 1 -- 172.21.15.102:0/3025085415 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(15..15 src has 1..15) v4 ==== 3549+0+0 (unknown 3948819498 0 0) 0x7f533c03cfe0 con 0x7f5350063dc0 2021-05-25T11:59:58.377 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.320+0000 7f5343fff700 10 client.4328.objecter ms_dispatch 0x7f5350001550 osd_map(15..15 src has 1..15) v4 2021-05-25T11:59:58.377 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.320+0000 7f5343fff700 3 client.4328.objecter handle_osd_map got epochs [15,15] > 0 2021-05-25T11:59:58.377 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.320+0000 7f5343fff700 3 client.4328.objecter handle_osd_map decoding full epoch 15 2021-05-25T11:59:58.377 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.320+0000 7f5343fff700 20 client.4328.objecter dump_active .. 0 homeless 2021-05-25T11:59:58.377 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.320+0000 7f534f7fe700 10 monclient: get_auth_request con 0x7f532c0333c0 auth_method 0 2021-05-25T11:59:58.378 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.321+0000 7f5343fff700 10 client.4328.objecter ms_handle_connect 0x7f532c0333c0 2021-05-25T11:59:58.378 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.324+0000 7f5343fff700 1 -- 172.21.15.102:0/3025085415 <== mon.2 v1:172.21.15.102:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7f533c00f360 con 0x7f5350063dc0 2021-05-25T11:59:58.378 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.324+0000 7f5343fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:58.378 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.324+0000 7f5343fff700 10 monclient: _finish_command 1 = system:0 2021-05-25T11:59:58.475 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.471+0000 7f53562c2700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2021-05-25T11:59:58.476 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.471+0000 7f53562c2700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2021-05-25T11:59:58.476 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.471+0000 7f53562c2700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T11:59:58.476 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.472+0000 7f53562c2700 1 -- 172.21.15.102:0/3025085415 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f5330005060 con 0x7f5350063dc0 2021-05-25T11:59:58.477 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.472+0000 7f5343fff700 1 -- 172.21.15.102:0/3025085415 <== mon.2 v1:172.21.15.102:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (unknown 2131975755 0 1444633209) 0x7f533c012ad0 con 0x7f5350063dc0 2021-05-25T11:59:58.477 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.472+0000 7f5343fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2021-05-25T11:59:58.477 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.472+0000 7f5343fff700 10 monclient: _finish_command 2 = system:0 2021-05-25T11:59:58.478 INFO:teuthology.orchestra.run.smithi102.stdout:47244640260 2021-05-25T11:59:58.479 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.473+0000 7f53562c2700 10 librados: watch_flush enter 2021-05-25T11:59:58.480 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.473+0000 7f53562c2700 10 librados: watch_flush exit 2021-05-25T11:59:58.480 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.473+0000 7f53562c2700 20 client.4328.objecter shutdown clearing up homeless session... 2021-05-25T11:59:58.480 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.473+0000 7f53562c2700 10 client.4328.objecter successfully canceled tick 2021-05-25T11:59:58.481 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.473+0000 7f53562c2700 1 -- 172.21.15.102:0/3025085415 >> v1:172.21.15.181:6812/29582 conn(0x7f532c0333c0 legacy=0x7f532c035880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:58.481 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.473+0000 7f53562c2700 10 monclient: shutdown 2021-05-25T11:59:58.481 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.473+0000 7f53562c2700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T11:59:58.481 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.473+0000 7f53562c2700 1 -- 172.21.15.102:0/3025085415 >> v1:172.21.15.102:6790/0 conn(0x7f5350063dc0 legacy=0x7f535013ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:58.482 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.473+0000 7f53562c2700 1 -- 172.21.15.102:0/3025085415 shutdown_connections 2021-05-25T11:59:58.482 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.473+0000 7f53562c2700 1 -- 172.21.15.102:0/3025085415 >> 172.21.15.102:0/3025085415 conn(0x7f535009bc30 msgr2=0x7f5350131c20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T11:59:58.482 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.473+0000 7f53562c2700 1 -- 172.21.15.102:0/3025085415 shutdown_connections 2021-05-25T11:59:58.482 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.473+0000 7f53562c2700 1 -- 172.21.15.102:0/3025085415 wait complete. 2021-05-25T11:59:58.483 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.474+0000 7f53562c2700 1 librados: shutdown 2021-05-25T11:59:58.487 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640260 for osd.5 2021-05-25T11:59:58.487 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2021-05-25T11:59:58.488 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-05-25T11:59:58.597 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.594+0000 7fa87d737700 10 monclient: get_monmap_and_config 2021-05-25T11:59:58.597 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.594+0000 7fa87d737700 10 monclient: build_initial_monmap 2021-05-25T11:59:58.598 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.594+0000 7fa87d737700 10 monclient: monmap: 2021-05-25T11:59:58.598 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T11:59:58.598 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:58.599 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:58.596781+0000 2021-05-25T11:59:58.599 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:58.596781+0000 2021-05-25T11:59:58.599 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T11:59:58.599 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:58.599 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T11:59:58.600 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T11:59:58.600 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T11:59:58.600 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:58.612 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.595+0000 7fa87d737700 1 Processor -- start 2021-05-25T11:59:58.612 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.595+0000 7fa87d737700 1 -- start start 2021-05-25T11:59:58.613 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.595+0000 7fa87d737700 10 monclient: init 2021-05-25T11:59:58.613 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.595+0000 7fa87d737700 10 monclient: _reopen_session rank -1 2021-05-25T11:59:58.613 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.595+0000 7fa87d737700 10 monclient: _add_conns ranks=[1,0,2] 2021-05-25T11:59:58.614 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.595+0000 7fa87d737700 10 monclient(hunting): picked mon.noname-c con 0x7fa8780989a0 addr v1:172.21.15.181:6789/0 2021-05-25T11:59:58.614 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.595+0000 7fa87d737700 10 monclient(hunting): picked mon.noname-a con 0x7fa878097ed0 addr v1:172.21.15.102:6789/0 2021-05-25T11:59:58.614 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.595+0000 7fa87d737700 10 monclient(hunting): picked mon.noname-b con 0x7fa878098350 addr v1:172.21.15.102:6790/0 2021-05-25T11:59:58.614 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.595+0000 7fa87d737700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa878063aa0 con 0x7fa878097ed0 2021-05-25T11:59:58.615 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.595+0000 7fa87d737700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa878099ae0 con 0x7fa8780989a0 2021-05-25T11:59:58.615 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.595+0000 7fa87d737700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa87809acd0 con 0x7fa878098350 2021-05-25T11:59:58.615 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.595+0000 7fa87d737700 10 monclient(hunting): _renew_subs 2021-05-25T11:59:58.615 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.595+0000 7fa87d737700 10 monclient(hunting): authenticate will time out at 799.193934s 2021-05-25T11:59:58.616 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.596+0000 7fa8767fc700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fa878097ed0 0x7fa878130900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:58974/0 (socket says 172.21.15.102:58974) 2021-05-25T11:59:58.616 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.596+0000 7fa8767fc700 1 -- 172.21.15.102:0/2128861945 learned_addr learned my addr 172.21.15.102:0/2128861945 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T11:59:58.616 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.596+0000 7fa8777fe700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fa878098350 0x7fa878134030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:53854/0 (socket says 172.21.15.102:53854) 2021-05-25T11:59:58.617 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.596+0000 7fa875ffb700 1 -- 172.21.15.102:0/2128861945 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1755042569 0 0) 0x7fa87809acd0 con 0x7fa878098350 2021-05-25T11:59:58.617 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.596+0000 7fa875ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:58.617 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.596+0000 7fa875ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:58.617 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.596+0000 7fa875ffb700 10 monclient(hunting): my global_id is 4331 2021-05-25T11:59:58.618 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.596+0000 7fa875ffb700 1 -- 172.21.15.102:0/2128861945 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa864003270 con 0x7fa878098350 2021-05-25T11:59:58.618 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.596+0000 7fa875ffb700 1 -- 172.21.15.102:0/2128861945 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1060784687 0 0) 0x7fa878063aa0 con 0x7fa878097ed0 2021-05-25T11:59:58.618 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.596+0000 7fa875ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:58.618 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.596+0000 7fa875ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:58.619 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.596+0000 7fa875ffb700 10 monclient(hunting): my global_id is 4338 2021-05-25T11:59:58.619 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.596+0000 7fa875ffb700 1 -- 172.21.15.102:0/2128861945 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa87809acd0 con 0x7fa878097ed0 2021-05-25T11:59:58.619 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.596+0000 7fa875ffb700 1 -- 172.21.15.102:0/2128861945 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 185521984 0 0) 0x7fa878099ae0 con 0x7fa8780989a0 2021-05-25T11:59:58.620 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.596+0000 7fa875ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:58.620 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.596+0000 7fa875ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:58.621 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.596+0000 7fa875ffb700 10 monclient(hunting): my global_id is 4333 2021-05-25T11:59:58.621 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.596+0000 7fa875ffb700 1 -- 172.21.15.102:0/2128861945 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa878063aa0 con 0x7fa8780989a0 2021-05-25T11:59:58.622 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.596+0000 7fa875ffb700 1 -- 172.21.15.102:0/2128861945 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 543784341 0 0) 0x7fa87809acd0 con 0x7fa878097ed0 2021-05-25T11:59:58.622 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.596+0000 7fa875ffb700 1 -- 172.21.15.102:0/2128861945 >> v1:172.21.15.102:6790/0 conn(0x7fa878098350 legacy=0x7fa878134030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:58.623 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.596+0000 7fa875ffb700 1 -- 172.21.15.102:0/2128861945 >> v1:172.21.15.181:6789/0 conn(0x7fa8780989a0 legacy=0x7fa878063390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:58.623 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.596+0000 7fa875ffb700 10 monclient: _finish_hunting 0 2021-05-25T11:59:58.623 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.596+0000 7fa875ffb700 1 monclient: found mon.noname-a 2021-05-25T11:59:58.624 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.596+0000 7fa875ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T11:59:58.624 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.596+0000 7fa875ffb700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T11:59:58.625 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.596+0000 7fa875ffb700 1 -- 172.21.15.102:0/2128861945 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa87809af70 con 0x7fa878097ed0 2021-05-25T11:59:58.625 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.596+0000 7fa875ffb700 10 monclient: _finish_auth 0 2021-05-25T11:59:58.625 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.596+0000 7fa875ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T11:59:58.626 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.596+0000 7fa875ffb700 1 -- 172.21.15.102:0/2128861945 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fa8600015f0 con 0x7fa878097ed0 2021-05-25T11:59:58.626 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.596+0000 7fa87d737700 5 monclient: authenticate success, global_id 4338 2021-05-25T11:59:58.627 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.596+0000 7fa87d737700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T11:59:58.627 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.596+0000 7fa875ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:58.628 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.596+0000 7fa875ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T11:59:58.628 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.596+0000 7fa875ffb700 10 monclient: dump: 2021-05-25T11:59:58.628 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:58.629 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:58.629 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:58.630 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:58.630 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:58.631 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:58.631 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:58.631 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:58.632 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:58.632 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:58.633 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.596+0000 7fa87d737700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T11:59:58.633 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.596+0000 7fa875ffb700 1 -- 172.21.15.102:0/2128861945 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa860001810 con 0x7fa878097ed0 2021-05-25T11:59:58.634 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.596+0000 7fa875ffb700 10 monclient: handle_config config(0 keys) v1 2021-05-25T11:59:58.634 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.596+0000 7fa87d737700 10 monclient: get_monmap_and_config success 2021-05-25T11:59:58.635 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.596+0000 7fa875ffb700 1 -- 172.21.15.102:0/2128861945 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fa860001b60 con 0x7fa878097ed0 2021-05-25T11:59:58.635 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.596+0000 7fa875ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:58.635 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.596+0000 7fa87d737700 4 set_mon_vals no callback set 2021-05-25T11:59:58.636 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.596+0000 7fa875ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T11:59:58.636 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.596+0000 7fa875ffb700 10 monclient: dump: 2021-05-25T11:59:58.637 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:58.637 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:58.637 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:58.638 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:58.638 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:58.639 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:58.639 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:58.640 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:58.640 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:58.640 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:58.641 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.597+0000 7fa87d737700 10 monclient: shutdown 2021-05-25T11:59:58.641 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.597+0000 7fa87d737700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T11:59:58.641 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.597+0000 7fa87d737700 1 -- 172.21.15.102:0/2128861945 >> v1:172.21.15.102:6789/0 conn(0x7fa878097ed0 legacy=0x7fa878130900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:58.642 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.597+0000 7fa87d737700 1 -- 172.21.15.102:0/2128861945 shutdown_connections 2021-05-25T11:59:58.642 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.597+0000 7fa87d737700 1 -- 172.21.15.102:0/2128861945 >> 172.21.15.102:0/2128861945 conn(0x7fa87809e9d0 msgr2=0x7fa87809ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T11:59:58.642 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.597+0000 7fa87d737700 1 -- 172.21.15.102:0/2128861945 shutdown_connections 2021-05-25T11:59:58.642 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.597+0000 7fa87d737700 1 -- 172.21.15.102:0/2128861945 wait complete. 2021-05-25T11:59:58.643 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.597+0000 7fa87d737700 10 monclient: build_initial_monmap 2021-05-25T11:59:58.643 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.597+0000 7fa87d737700 10 monclient: monmap: 2021-05-25T11:59:58.643 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T11:59:58.643 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T11:59:58.643 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T11:59:58.644 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T11:59:58.644 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T11:59:58.644 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T11:59:58.644 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T11:59:58.644 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T11:59:58.645 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T11:59:58.645 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:58.645 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.598+0000 7fa87d737700 1 librados: starting msgr at 2021-05-25T11:59:58.645 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.598+0000 7fa87d737700 1 librados: starting objecter 2021-05-25T11:59:58.645 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.598+0000 7fa87d737700 1 Processor -- start 2021-05-25T11:59:58.646 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.598+0000 7fa87d737700 1 -- start start 2021-05-25T11:59:58.646 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.598+0000 7fa87d737700 1 librados: setting wanted keys 2021-05-25T11:59:58.646 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.598+0000 7fa87d737700 1 librados: calling monclient init 2021-05-25T11:59:58.646 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.598+0000 7fa87d737700 10 monclient: init 2021-05-25T11:59:58.647 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.598+0000 7fa87d737700 10 monclient: _reopen_session rank -1 2021-05-25T11:59:58.647 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.598+0000 7fa87d737700 10 monclient: _add_conns ranks=[2,1,0] 2021-05-25T11:59:58.647 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.598+0000 7fa87d737700 10 monclient(hunting): picked mon.noname-c con 0x7fa878097ed0 addr v1:172.21.15.102:6790/0 2021-05-25T11:59:58.647 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.598+0000 7fa87d737700 10 monclient(hunting): picked mon.noname-b con 0x7fa878098350 addr v1:172.21.15.181:6789/0 2021-05-25T11:59:58.647 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.598+0000 7fa87d737700 10 monclient(hunting): picked mon.noname-a con 0x7fa8780989a0 addr v1:172.21.15.102:6789/0 2021-05-25T11:59:58.648 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.598+0000 7fa87d737700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa87809a190 con 0x7fa8780989a0 2021-05-25T11:59:58.648 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.598+0000 7fa87d737700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa87811ba40 con 0x7fa878098350 2021-05-25T11:59:58.648 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.598+0000 7fa87d737700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa87811bce0 con 0x7fa878097ed0 2021-05-25T11:59:58.648 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.598+0000 7fa87d737700 10 monclient(hunting): _renew_subs 2021-05-25T11:59:58.648 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.598+0000 7fa87d737700 10 monclient(hunting): authenticate will time out at 799.196659s 2021-05-25T11:59:58.649 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.598+0000 7fa876ffd700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fa878097ed0 0x7fa8781211d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:53856/0 (socket says 172.21.15.102:53856) 2021-05-25T11:59:58.649 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.598+0000 7fa876ffd700 1 -- 172.21.15.102:0/750028983 learned_addr learned my addr 172.21.15.102:0/750028983 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T11:59:58.649 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.598+0000 7fa85effd700 10 client.?.objecter ms_handle_connect 0x7fa878097ed0 2021-05-25T11:59:58.649 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.598+0000 7fa85effd700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:58.649 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.598+0000 7fa85effd700 10 client.?.objecter ms_handle_connect 0x7fa8780989a0 2021-05-25T11:59:58.650 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.598+0000 7fa85effd700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:58.650 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.598+0000 7fa85effd700 10 client.?.objecter ms_handle_connect 0x7fa878098350 2021-05-25T11:59:58.650 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.598+0000 7fa85effd700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:58.650 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.598+0000 7fa85effd700 1 -- 172.21.15.102:0/750028983 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3826363737 0 0) 0x7fa87809a190 con 0x7fa8780989a0 2021-05-25T11:59:58.651 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.598+0000 7fa85effd700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:58.651 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.598+0000 7fa85effd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:58.651 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.598+0000 7fa85effd700 10 monclient(hunting): my global_id is 4341 2021-05-25T11:59:58.651 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.598+0000 7fa85effd700 1 -- 172.21.15.102:0/750028983 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa858004c10 con 0x7fa8780989a0 2021-05-25T11:59:58.652 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.599+0000 7fa85effd700 1 -- 172.21.15.102:0/750028983 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1046774944 0 0) 0x7fa87811bce0 con 0x7fa878097ed0 2021-05-25T11:59:58.652 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.599+0000 7fa85effd700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:58.652 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.599+0000 7fa85effd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:58.652 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.599+0000 7fa85effd700 10 monclient(hunting): my global_id is 4334 2021-05-25T11:59:58.652 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.599+0000 7fa85effd700 1 -- 172.21.15.102:0/750028983 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa87809a190 con 0x7fa878097ed0 2021-05-25T11:59:58.653 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.599+0000 7fa85effd700 1 -- 172.21.15.102:0/750028983 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3975643166 0 0) 0x7fa87811ba40 con 0x7fa878098350 2021-05-25T11:59:58.653 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.599+0000 7fa85effd700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:58.653 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.599+0000 7fa85effd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:58.653 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.599+0000 7fa85effd700 10 monclient(hunting): my global_id is 4336 2021-05-25T11:59:58.654 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.599+0000 7fa85effd700 1 -- 172.21.15.102:0/750028983 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa87811bce0 con 0x7fa878098350 2021-05-25T11:59:58.654 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.599+0000 7fa85effd700 1 -- 172.21.15.102:0/750028983 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1469611202 0 0) 0x7fa858004c10 con 0x7fa8780989a0 2021-05-25T11:59:58.654 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.599+0000 7fa85effd700 1 -- 172.21.15.102:0/750028983 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa87811ba40 con 0x7fa8780989a0 2021-05-25T11:59:58.654 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.599+0000 7fa85effd700 1 -- 172.21.15.102:0/750028983 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fa868001540 con 0x7fa8780989a0 2021-05-25T11:59:58.655 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.599+0000 7fa85effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:58.655 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.599+0000 7fa85effd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T11:59:58.655 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.599+0000 7fa85effd700 10 monclient(hunting): dump: 2021-05-25T11:59:58.655 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:58.655 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:58.656 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:58.656 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:58.656 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:58.656 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:58.656 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:58.657 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:58.657 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:58.657 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:58.657 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.599+0000 7fa85effd700 10 monclient(hunting): _finish_auth 0 2021-05-25T11:59:58.657 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.599+0000 7fa85effd700 1 -- 172.21.15.102:0/750028983 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3999076665 0 0) 0x7fa87809a190 con 0x7fa878097ed0 2021-05-25T11:59:58.658 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.599+0000 7fa85effd700 1 -- 172.21.15.102:0/750028983 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa858004c10 con 0x7fa878097ed0 2021-05-25T11:59:58.658 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.599+0000 7fa85effd700 1 -- 172.21.15.102:0/750028983 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fa86c001990 con 0x7fa878097ed0 2021-05-25T11:59:58.658 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.599+0000 7fa85effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:58.658 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.599+0000 7fa85effd700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2021-05-25T11:59:58.659 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.599+0000 7fa85effd700 10 monclient(hunting): dump: 2021-05-25T11:59:58.659 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:58.659 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:58.659 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:58.659 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:58.660 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:58.660 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:58.660 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:58.660 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:58.660 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:58.661 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:58.661 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.599+0000 7fa85effd700 1 -- 172.21.15.102:0/750028983 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1116461855 0 0) 0x7fa87811bce0 con 0x7fa878098350 2021-05-25T11:59:58.661 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.599+0000 7fa85effd700 1 -- 172.21.15.102:0/750028983 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa87809a190 con 0x7fa878098350 2021-05-25T11:59:58.661 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.599+0000 7fa85effd700 1 -- 172.21.15.102:0/750028983 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fa8600014f0 con 0x7fa878098350 2021-05-25T11:59:58.662 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.599+0000 7fa85effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:58.662 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.599+0000 7fa85effd700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2021-05-25T11:59:58.662 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.599+0000 7fa85effd700 10 monclient(hunting): dump: 2021-05-25T11:59:58.662 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:58.662 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:58.663 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:58.663 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:58.663 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:58.663 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:58.663 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:58.664 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:58.664 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:58.664 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:58.664 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.599+0000 7fa85effd700 1 -- 172.21.15.102:0/750028983 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2000258299 0 0) 0x7fa87811ba40 con 0x7fa8780989a0 2021-05-25T11:59:58.664 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.599+0000 7fa85effd700 1 -- 172.21.15.102:0/750028983 >> v1:172.21.15.102:6790/0 conn(0x7fa878097ed0 legacy=0x7fa8781211d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:58.665 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.599+0000 7fa85effd700 1 -- 172.21.15.102:0/750028983 >> v1:172.21.15.181:6789/0 conn(0x7fa878098350 legacy=0x7fa87813aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:58.665 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.599+0000 7fa85effd700 10 monclient: _finish_hunting 0 2021-05-25T11:59:58.665 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.599+0000 7fa85effd700 1 monclient: found mon.a 2021-05-25T11:59:58.665 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.599+0000 7fa85effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T11:59:58.666 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.599+0000 7fa85effd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:58.666 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.599+0000 7fa85effd700 1 -- 172.21.15.102:0/750028983 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa878119b70 con 0x7fa8780989a0 2021-05-25T11:59:58.666 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.599+0000 7fa85effd700 10 monclient: _finish_auth 0 2021-05-25T11:59:58.666 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.599+0000 7fa85effd700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T11:59:58.666 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.599+0000 7fa87d737700 5 monclient: authenticate success, global_id 4341 2021-05-25T11:59:58.667 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.599+0000 7fa87d737700 10 monclient: _renew_subs 2021-05-25T11:59:58.667 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.599+0000 7fa87d737700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:58.667 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.599+0000 7fa87d737700 1 -- 172.21.15.102:0/750028983 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8781218e0 con 0x7fa8780989a0 2021-05-25T11:59:58.667 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.599+0000 7fa87d737700 10 client.4341.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T11:59:58.668 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.599+0000 7fa87d737700 10 monclient: _renew_subs 2021-05-25T11:59:58.668 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.599+0000 7fa87d737700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:58.668 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.599+0000 7fa87d737700 1 -- 172.21.15.102:0/750028983 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa87813b200 con 0x7fa8780989a0 2021-05-25T11:59:58.668 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.599+0000 7fa87d737700 1 librados: init done 2021-05-25T11:59:58.668 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.599+0000 7fa85effd700 1 -- 172.21.15.102:0/750028983 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa8680048b0 con 0x7fa8780989a0 2021-05-25T11:59:58.669 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.599+0000 7fa85effd700 10 monclient: handle_config config(0 keys) v1 2021-05-25T11:59:58.669 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.599+0000 7fa85effd700 1 -- 172.21.15.102:0/750028983 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fa868004bc0 con 0x7fa8780989a0 2021-05-25T11:59:58.669 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.599+0000 7fa85effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:58.669 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.599+0000 7fa8757fa700 4 set_mon_vals no callback set 2021-05-25T11:59:58.670 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.599+0000 7fa85effd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T11:59:58.670 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.599+0000 7fa85effd700 10 monclient: dump: 2021-05-25T11:59:58.670 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:58.670 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:58.671 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:58.671 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:58.671 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:58.671 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:58.671 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:58.672 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:58.672 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:58.672 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:58.672 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.600+0000 7fa85effd700 1 -- 172.21.15.102:0/750028983 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7fa86800dee0 con 0x7fa8780989a0 2021-05-25T11:59:58.672 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.600+0000 7fa87d737700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:58.673 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.600+0000 7fa87d737700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:58.673 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.600+0000 7fa87d737700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:58.673 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.600+0000 7fa85effd700 1 -- 172.21.15.102:0/750028983 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(15..15 src has 1..15) v4 ==== 3549+0+0 (unknown 3948819498 0 0) 0x7fa86803cf30 con 0x7fa8780989a0 2021-05-25T11:59:58.673 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.600+0000 7fa85effd700 10 client.4341.objecter ms_dispatch 0x7fa878001550 osd_map(15..15 src has 1..15) v4 2021-05-25T11:59:58.673 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.600+0000 7fa85effd700 3 client.4341.objecter handle_osd_map got epochs [15,15] > 0 2021-05-25T11:59:58.674 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.600+0000 7fa85effd700 3 client.4341.objecter handle_osd_map decoding full epoch 15 2021-05-25T11:59:58.674 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.600+0000 7fa87d737700 1 -- 172.21.15.102:0/750028983 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa850004db0 con 0x7fa8780989a0 2021-05-25T11:59:58.674 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.600+0000 7fa85effd700 20 client.4341.objecter dump_active .. 0 homeless 2021-05-25T11:59:58.674 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.600+0000 7fa876ffd700 10 monclient: get_auth_request con 0x7fa858033520 auth_method 0 2021-05-25T11:59:58.675 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.600+0000 7fa85effd700 10 client.4341.objecter ms_handle_connect 0x7fa858033520 2021-05-25T11:59:58.675 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.604+0000 7fa85effd700 1 -- 172.21.15.102:0/750028983 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7fa868012e70 con 0x7fa8780989a0 2021-05-25T11:59:58.675 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.604+0000 7fa85effd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:58.675 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.604+0000 7fa85effd700 10 monclient: _finish_command 1 = system:0 2021-05-25T11:59:58.751 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.748+0000 7fa87d737700 1 -- 172.21.15.102:0/750028983 --> v1:172.21.15.181:6812/29582 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa850000b90 con 0x7fa858033520 2021-05-25T11:59:58.753 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.751+0000 7fa85effd700 1 -- 172.21.15.102:0/750028983 <== mgr.4105 v1:172.21.15.181:6812/29582 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+38639 (unknown 2965378022 0 1705510130) 0x7fa850000b90 con 0x7fa858033520 2021-05-25T11:59:58.754 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:59:58.755 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.752+0000 7fa87d737700 10 librados: watch_flush enter 2021-05-25T11:59:58.755 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.752+0000 7fa87d737700 10 librados: watch_flush exit 2021-05-25T11:59:58.756 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.752+0000 7fa87d737700 20 client.4341.objecter shutdown clearing up homeless session... 2021-05-25T11:59:58.756 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.752+0000 7fa87d737700 10 client.4341.objecter successfully canceled tick 2021-05-25T11:59:58.756 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.752+0000 7fa87d737700 1 -- 172.21.15.102:0/750028983 >> v1:172.21.15.181:6812/29582 conn(0x7fa858033520 legacy=0x7fa8580359e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:58.756 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.752+0000 7fa87d737700 10 monclient: shutdown 2021-05-25T11:59:58.757 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.752+0000 7fa87d737700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T11:59:58.757 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.752+0000 7fa87d737700 1 -- 172.21.15.102:0/750028983 >> v1:172.21.15.102:6789/0 conn(0x7fa8780989a0 legacy=0x7fa878142e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:58.757 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.752+0000 7fa87d737700 1 -- 172.21.15.102:0/750028983 shutdown_connections 2021-05-25T11:59:58.757 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.752+0000 7fa87d737700 1 -- 172.21.15.102:0/750028983 >> 172.21.15.102:0/750028983 conn(0x7fa87809e9d0 msgr2=0x7fa87809c800 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T11:59:58.757 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.752+0000 7fa87d737700 1 -- 172.21.15.102:0/750028983 shutdown_connections 2021-05-25T11:59:58.758 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.752+0000 7fa87d737700 1 -- 172.21.15.102:0/750028983 wait complete. 2021-05-25T11:59:58.758 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.752+0000 7fa87d737700 1 librados: shutdown 2021-05-25T11:59:58.758 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2021-05-25T11:59:58.765 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2021-05-25T11:59:57.556206+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":11,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":29544,"kb_used_data":760,"kb_used_omap":0,"kb_used_meta":28736,"kb_avail":566201496,"statfs":{"total":579820584960,"available":579790331904,"internally_reserved":0,"allocated":778240,"data_stored":230516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29425664},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2021-05-25T11:59:52.780051+0000","last_change":"2021-05-25T11:59:52.779532+0000","last_active":"2021-05-25T11:59:52.780051+0000","last_peered":"2021-05-25T11:59:52.780051+0000","last_clean":"2021-05-25T11:59:52.780051+0000","last_became_active":"2021-05-25T11:59:52.779240+0000","last_became_peered":"2021-05-25T11:59:52.779240+0000","last_unstale":"2021-05-25T11:59:52.780051+0000","last_undegraded":"2021-05-25T11:59:52.780051+0000","last_fullsized":"2021-05-25T11:59:52.780051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-05-25T11:59:51.583021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-05-25T11:59:51.583021+0000","last_clean_scrub_stamp":"2021-05-25T11:59:51.583021+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2021-05-25T11:59:52.600778+0000","last_change":"2021-05-25T11:59:52.599864+0000","last_active":"2021-05-25T11:59:52.600778+0000","last_peered":"2021-05-25T11:59:52.600778+0000","last_clean":"2021-05-25T11:59:52.600778+0000","last_became_active":"2021-05-25T11:59:52.599550+0000","last_became_peered":"2021-05-25T11:59:52.599550+0000","last_unstale":"2021-05-25T11:59:52.600778+0000","last_undegraded":"2021-05-25T11:59:52.600778+0000","last_fullsized":"2021-05-25T11:59:52.600778+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-05-25T11:59:51.583021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-05-25T11:59:51.583021+0000","last_clean_scrub_stamp":"2021-05-25T11:59:51.583021+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2021-05-25T11:59:52.599789+0000","last_change":"2021-05-25T11:59:52.599209+0000","last_active":"2021-05-25T11:59:52.599789+0000","last_peered":"2021-05-25T11:59:52.599789+0000","last_clean":"2021-05-25T11:59:52.599789+0000","last_became_active":"2021-05-25T11:59:52.598818+0000","last_became_peered":"2021-05-25T11:59:52.598818+0000","last_unstale":"2021-05-25T11:59:52.599789+0000","last_undegraded":"2021-05-25T11:59:52.599789+0000","last_fullsized":"2021-05-25T11:59:52.599789+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-05-25T11:59:51.583021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-05-25T11:59:51.583021+0000","last_clean_scrub_stamp":"2021-05-25T11:59:51.583021+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2021-05-25T11:59:52.599030+0000","last_change":"2021-05-25T11:59:52.596805+0000","last_active":"2021-05-25T11:59:52.599030+0000","last_peered":"2021-05-25T11:59:52.599030+0000","last_clean":"2021-05-25T11:59:52.599030+0000","last_became_active":"2021-05-25T11:59:52.596538+0000","last_became_peered":"2021-05-25T11:59:52.596538+0000","last_unstale":"2021-05-25T11:59:52.599030+0000","last_undegraded":"2021-05-25T11:59:52.599030+0000","last_fullsized":"2021-05-25T11:59:52.599030+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-05-25T11:59:51.583021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-05-25T11:59:51.583021+0000","last_clean_scrub_stamp":"2021-05-25T11:59:51.583021+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"10","reported_epoch":"14","state":"active+clean","last_fresh":"2021-05-25T11:59:52.593134+0000","last_change":"2021-05-25T11:59:51.592476+0000","last_active":"2021-05-25T11:59:52.593134+0000","last_peered":"2021-05-25T11:59:52.593134+0000","last_clean":"2021-05-25T11:59:52.593134+0000","last_became_active":"2021-05-25T11:59:51.592157+0000","last_became_peered":"2021-05-25T11:59:51.592157+0000","last_unstale":"2021-05-25T11:59:52.593134+0000","last_undegraded":"2021-05-25T11:59:52.593134+0000","last_fullsized":"2021-05-25T11:59:52.593134+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-05-25T11:59:50.578675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-05-25T11:59:50.578675+0000","last_clean_scrub_stamp":"2021-05-25T11:59:50.578675+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2021-05-25T11:59:52.607326+0000","last_change":"2021-05-25T11:59:52.602323+0000","last_active":"2021-05-25T11:59:52.607326+0000","last_peered":"2021-05-25T11:59:52.607326+0000","last_clean":"2021-05-25T11:59:52.607326+0000","last_became_active":"2021-05-25T11:59:52.602047+0000","last_became_peered":"2021-05-25T11:59:52.602047+0000","last_unstale":"2021-05-25T11:59:52.607326+0000","last_undegraded":"2021-05-25T11:59:52.607326+0000","last_fullsized":"2021-05-25T11:59:52.607326+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-05-25T11:59:51.583021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-05-25T11:59:51.583021+0000","last_clean_scrub_stamp":"2021-05-25T11:59:51.583021+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2021-05-25T11:59:52.600845+0000","last_change":"2021-05-25T11:59:52.598411+0000","last_active":"2021-05-25T11:59:52.600845+0000","last_peered":"2021-05-25T11:59:52.600845+0000","last_clean":"2021-05-25T11:59:52.600845+0000","last_became_active":"2021-05-25T11:59:52.598090+0000","last_became_peered":"2021-05-25T11:59:52.598090+0000","last_unstale":"2021-05-25T11:59:52.600845+0000","last_undegraded":"2021-05-25T11:59:52.600845+0000","last_fullsized":"2021-05-25T11:59:52.600845+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-05-25T11:59:51.583021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-05-25T11:59:51.583021+0000","last_clean_scrub_stamp":"2021-05-25T11:59:51.583021+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2021-05-25T11:59:52.607316+0000","last_change":"2021-05-25T11:59:52.602212+0000","last_active":"2021-05-25T11:59:52.607316+0000","last_peered":"2021-05-25T11:59:52.607316+0000","last_clean":"2021-05-25T11:59:52.607316+0000","last_became_active":"2021-05-25T11:59:52.601888+0000","last_became_peered":"2021-05-25T11:59:52.601888+0000","last_unstale":"2021-05-25T11:59:52.607316+0000","last_undegraded":"2021-05-25T11:59:52.607316+0000","last_fullsized":"2021-05-25T11:59:52.607316+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-05-25T11:59:51.583021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-05-25T11:59:51.583021+0000","last_clean_scrub_stamp":"2021-05-25T11:59:51.583021+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2021-05-25T11:59:52.598621+0000","last_change":"2021-05-25T11:59:52.597495+0000","last_active":"2021-05-25T11:59:52.598621+0000","last_peered":"2021-05-25T11:59:52.598621+0000","last_clean":"2021-05-25T11:59:52.598621+0000","last_became_active":"2021-05-25T11:59:52.597224+0000","last_became_peered":"2021-05-25T11:59:52.597224+0000","last_unstale":"2021-05-25T11:59:52.598621+0000","last_undegraded":"2021-05-25T11:59:52.598621+0000","last_fullsized":"2021-05-25T11:59:52.598621+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-05-25T11:59:51.583021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-05-25T11:59:51.583021+0000","last_clean_scrub_stamp":"2021-05-25T11:59:51.583021+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4936,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366904,"statfs":{"total":96636764160,"available":96631709696,"internally_reserved":0,"allocated":131072,"data_stored":38737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40500000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.309}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44600000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35699999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]}]},{"osd":4,"up_from":11,"seq":47244640259,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4868,"kb_used_data":124,"kb_used_omap":0,"kb_used_meta":4736,"kb_avail":94366972,"statfs":{"total":96636764160,"available":96631779328,"internally_reserved":0,"allocated":126976,"data_stored":37784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4849664},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29899999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55100000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47499999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46200000000000002}]}]},{"osd":3,"up_from":11,"seq":47244640259,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4936,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366904,"statfs":{"total":96636764160,"available":96631709696,"internally_reserved":0,"allocated":131072,"data_stored":38737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64700000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51200000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68799999999999994}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33800000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.377}]}]},{"osd":2,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4936,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366904,"statfs":{"total":96636764160,"available":96631709696,"internally_reserved":0,"allocated":131072,"data_stored":38737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.216}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.22600000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.5}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49099999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51100000000000001}]}]},{"osd":0,"up_from":11,"seq":47244640260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4936,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366904,"statfs":{"total":96636764160,"available":96631709696,"internally_reserved":0,"allocated":131072,"data_stored":38737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28299999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53500000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52400000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73699999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42899999999999999}]}]},{"osd":1,"up_from":11,"seq":47244640261,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4932,"kb_used_data":124,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366908,"statfs":{"total":96636764160,"available":96631713792,"internally_reserved":0,"allocated":126976,"data_stored":37784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.216}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.254}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47899999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66600000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-05-25T11:59:58.767 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-05-25T11:59:58.875 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.872+0000 7fb7da826700 10 monclient: get_monmap_and_config 2021-05-25T11:59:58.875 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.872+0000 7fb7da826700 10 monclient: build_initial_monmap 2021-05-25T11:59:58.875 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.872+0000 7fb7da826700 10 monclient: monmap: 2021-05-25T11:59:58.876 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T11:59:58.876 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:58.876 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:58.874647+0000 2021-05-25T11:59:58.876 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:58.874647+0000 2021-05-25T11:59:58.877 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T11:59:58.877 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:58.877 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T11:59:58.877 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T11:59:58.878 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T11:59:58.878 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:58.888 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.873+0000 7fb7da826700 1 Processor -- start 2021-05-25T11:59:58.889 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.873+0000 7fb7da826700 1 -- start start 2021-05-25T11:59:58.889 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.873+0000 7fb7da826700 10 monclient: init 2021-05-25T11:59:58.889 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.873+0000 7fb7da826700 10 monclient: _reopen_session rank -1 2021-05-25T11:59:58.889 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.873+0000 7fb7da826700 10 monclient: _add_conns ranks=[2,1,0] 2021-05-25T11:59:58.890 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.873+0000 7fb7da826700 10 monclient(hunting): picked mon.noname-b con 0x7fb7d4097fa0 addr v1:172.21.15.102:6790/0 2021-05-25T11:59:58.890 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.873+0000 7fb7da826700 10 monclient(hunting): picked mon.noname-c con 0x7fb7d4063b70 addr v1:172.21.15.181:6789/0 2021-05-25T11:59:58.890 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.873+0000 7fb7da826700 10 monclient(hunting): picked mon.noname-a con 0x7fb7d41325c0 addr v1:172.21.15.102:6789/0 2021-05-25T11:59:58.890 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.873+0000 7fb7da826700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7d4096b20 con 0x7fb7d41325c0 2021-05-25T11:59:58.890 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.874+0000 7fb7da826700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7d409b460 con 0x7fb7d4063b70 2021-05-25T11:59:58.891 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.874+0000 7fb7da826700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7d40964c0 con 0x7fb7d4097fa0 2021-05-25T11:59:58.891 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.874+0000 7fb7da826700 10 monclient(hunting): _renew_subs 2021-05-25T11:59:58.891 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.874+0000 7fb7da826700 10 monclient(hunting): authenticate will time out at 799.472028s 2021-05-25T11:59:58.891 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.874+0000 7fb7d3fff700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fb7d4097fa0 0x7fb7d4063460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:53864/0 (socket says 172.21.15.102:53864) 2021-05-25T11:59:58.891 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.874+0000 7fb7d3fff700 1 -- 172.21.15.102:0/3829370137 learned_addr learned my addr 172.21.15.102:0/3829370137 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T11:59:58.892 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.874+0000 7fb7d2ffd700 1 -- 172.21.15.102:0/3829370137 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2948375201 0 0) 0x7fb7d40964c0 con 0x7fb7d4097fa0 2021-05-25T11:59:58.892 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.874+0000 7fb7d2ffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:58.892 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.874+0000 7fb7d2ffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:58.892 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.874+0000 7fb7d2ffd700 10 monclient(hunting): my global_id is 4337 2021-05-25T11:59:58.893 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.874+0000 7fb7d2ffd700 1 -- 172.21.15.102:0/3829370137 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7bc003270 con 0x7fb7d4097fa0 2021-05-25T11:59:58.893 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.874+0000 7fb7d2ffd700 1 -- 172.21.15.102:0/3829370137 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 441208248 0 0) 0x7fb7d4096b20 con 0x7fb7d41325c0 2021-05-25T11:59:58.893 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.874+0000 7fb7d2ffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:58.893 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.874+0000 7fb7d2ffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:58.894 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.874+0000 7fb7d2ffd700 10 monclient(hunting): my global_id is 4344 2021-05-25T11:59:58.894 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.874+0000 7fb7d2ffd700 1 -- 172.21.15.102:0/3829370137 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7d40964c0 con 0x7fb7d41325c0 2021-05-25T11:59:58.894 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.874+0000 7fb7d2ffd700 1 -- 172.21.15.102:0/3829370137 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 809585539 0 0) 0x7fb7bc003270 con 0x7fb7d4097fa0 2021-05-25T11:59:58.894 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.874+0000 7fb7d2ffd700 1 -- 172.21.15.102:0/3829370137 >> v1:172.21.15.181:6789/0 conn(0x7fb7d4063b70 legacy=0x7fb7d4064020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:58.895 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.874+0000 7fb7d2ffd700 1 -- 172.21.15.102:0/3829370137 >> v1:172.21.15.102:6789/0 conn(0x7fb7d41325c0 legacy=0x7fb7d41349b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:58.895 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.874+0000 7fb7d2ffd700 10 monclient: _finish_hunting 0 2021-05-25T11:59:58.895 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.874+0000 7fb7d2ffd700 1 monclient: found mon.noname-b 2021-05-25T11:59:58.895 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.874+0000 7fb7d2ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T11:59:58.896 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.874+0000 7fb7d2ffd700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T11:59:58.896 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.874+0000 7fb7d2ffd700 1 -- 172.21.15.102:0/3829370137 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7d4096760 con 0x7fb7d4097fa0 2021-05-25T11:59:58.896 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.874+0000 7fb7d2ffd700 10 monclient: _finish_auth 0 2021-05-25T11:59:58.896 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.874+0000 7fb7d2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T11:59:58.897 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.874+0000 7fb7d2ffd700 1 -- 172.21.15.102:0/3829370137 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fb7b8001610 con 0x7fb7d4097fa0 2021-05-25T11:59:58.897 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.874+0000 7fb7da826700 5 monclient: authenticate success, global_id 4337 2021-05-25T11:59:58.897 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.874+0000 7fb7d2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:58.897 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.874+0000 7fb7d2ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T11:59:58.897 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.874+0000 7fb7d2ffd700 10 monclient: dump: 2021-05-25T11:59:58.898 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:58.898 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:58.898 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:58.898 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:58.899 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:58.899 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:58.899 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:58.899 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:58.900 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:58.900 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:58.900 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.874+0000 7fb7da826700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T11:59:58.900 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.874+0000 7fb7d2ffd700 1 -- 172.21.15.102:0/3829370137 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb7b8001840 con 0x7fb7d4097fa0 2021-05-25T11:59:58.900 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.874+0000 7fb7d2ffd700 10 monclient: handle_config config(0 keys) v1 2021-05-25T11:59:58.901 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.874+0000 7fb7da826700 10 monclient: get_monmap_and_config success 2021-05-25T11:59:58.901 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.874+0000 7fb7d2ffd700 1 -- 172.21.15.102:0/3829370137 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fb7b8001bd0 con 0x7fb7d4097fa0 2021-05-25T11:59:58.901 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.874+0000 7fb7d2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:58.901 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.874+0000 7fb7da826700 4 set_mon_vals no callback set 2021-05-25T11:59:58.902 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.874+0000 7fb7d2ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T11:59:58.902 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.874+0000 7fb7d2ffd700 10 monclient: dump: 2021-05-25T11:59:58.902 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:58.902 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:58.903 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:58.903 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:58.903 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:58.903 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:58.903 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:58.904 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:58.904 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:58.904 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:58.904 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.875+0000 7fb7da826700 10 monclient: shutdown 2021-05-25T11:59:58.905 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.875+0000 7fb7da826700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T11:59:58.905 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.875+0000 7fb7da826700 1 -- 172.21.15.102:0/3829370137 >> v1:172.21.15.102:6790/0 conn(0x7fb7d4097fa0 legacy=0x7fb7d4063460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:58.905 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.875+0000 7fb7da826700 1 -- 172.21.15.102:0/3829370137 shutdown_connections 2021-05-25T11:59:58.905 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.875+0000 7fb7da826700 1 -- 172.21.15.102:0/3829370137 >> 172.21.15.102:0/3829370137 conn(0x7fb7d409ef50 msgr2=0x7fb7d409f360 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T11:59:58.905 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.875+0000 7fb7da826700 1 -- 172.21.15.102:0/3829370137 shutdown_connections 2021-05-25T11:59:58.906 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.875+0000 7fb7da826700 1 -- 172.21.15.102:0/3829370137 wait complete. 2021-05-25T11:59:58.906 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.875+0000 7fb7da826700 10 monclient: build_initial_monmap 2021-05-25T11:59:58.906 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.875+0000 7fb7da826700 10 monclient: monmap: 2021-05-25T11:59:58.906 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T11:59:58.907 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T11:59:58.907 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T11:59:58.907 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T11:59:58.907 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T11:59:58.907 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T11:59:58.908 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T11:59:58.908 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T11:59:58.908 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T11:59:58.908 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:58.908 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.875+0000 7fb7da826700 1 librados: starting msgr at 2021-05-25T11:59:58.909 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.875+0000 7fb7da826700 1 librados: starting objecter 2021-05-25T11:59:58.909 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.876+0000 7fb7da826700 1 Processor -- start 2021-05-25T11:59:58.909 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.876+0000 7fb7da826700 1 -- start start 2021-05-25T11:59:58.909 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.876+0000 7fb7da826700 1 librados: setting wanted keys 2021-05-25T11:59:58.910 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.876+0000 7fb7da826700 1 librados: calling monclient init 2021-05-25T11:59:58.910 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.876+0000 7fb7da826700 10 monclient: init 2021-05-25T11:59:58.910 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.876+0000 7fb7da826700 10 monclient: _reopen_session rank -1 2021-05-25T11:59:58.910 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.876+0000 7fb7da826700 10 monclient: _add_conns ranks=[1,0,2] 2021-05-25T11:59:58.911 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.876+0000 7fb7da826700 10 monclient(hunting): picked mon.noname-b con 0x7fb7d4063b70 addr v1:172.21.15.181:6789/0 2021-05-25T11:59:58.911 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.876+0000 7fb7da826700 10 monclient(hunting): picked mon.noname-a con 0x7fb7d4097fa0 addr v1:172.21.15.102:6789/0 2021-05-25T11:59:58.911 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.876+0000 7fb7da826700 10 monclient(hunting): picked mon.noname-c con 0x7fb7d41325c0 addr v1:172.21.15.102:6790/0 2021-05-25T11:59:58.911 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.876+0000 7fb7da826700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7d409ab70 con 0x7fb7d4097fa0 2021-05-25T11:59:58.911 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.876+0000 7fb7da826700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7d4132400 con 0x7fb7d4063b70 2021-05-25T11:59:58.912 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.876+0000 7fb7da826700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7d411c460 con 0x7fb7d41325c0 2021-05-25T11:59:58.912 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.876+0000 7fb7da826700 10 monclient(hunting): _renew_subs 2021-05-25T11:59:58.912 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.876+0000 7fb7da826700 10 monclient(hunting): authenticate will time out at 799.474716s 2021-05-25T11:59:58.912 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.876+0000 7fb7d8dc3700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fb7d41325c0 0x7fb7d4143520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:53874/0 (socket says 172.21.15.102:53874) 2021-05-25T11:59:58.913 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.876+0000 7fb7d8dc3700 1 -- 172.21.15.102:0/321053627 learned_addr learned my addr 172.21.15.102:0/321053627 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T11:59:58.913 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.876+0000 7fb7d37fe700 1 --1- 172.21.15.102:0/321053627 >> v1:172.21.15.102:6789/0 conn(0x7fb7d4097fa0 0x7fb7d4131cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:58994/0 (socket says 172.21.15.102:58994) 2021-05-25T11:59:58.913 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.876+0000 7fb7c7fff700 10 client.?.objecter ms_handle_connect 0x7fb7d41325c0 2021-05-25T11:59:58.913 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.876+0000 7fb7c7fff700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:58.913 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.876+0000 7fb7c7fff700 10 client.?.objecter ms_handle_connect 0x7fb7d4097fa0 2021-05-25T11:59:58.914 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.876+0000 7fb7c7fff700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:58.914 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.877+0000 7fb7c7fff700 1 -- 172.21.15.102:0/321053627 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2212808690 0 0) 0x7fb7d411c460 con 0x7fb7d41325c0 2021-05-25T11:59:58.914 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.877+0000 7fb7c7fff700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:58.914 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.877+0000 7fb7c7fff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:58.915 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.877+0000 7fb7c7fff700 10 monclient(hunting): my global_id is 4340 2021-05-25T11:59:58.915 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.877+0000 7fb7c7fff700 1 -- 172.21.15.102:0/321053627 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7b0004c10 con 0x7fb7d41325c0 2021-05-25T11:59:58.915 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.877+0000 7fb7c7fff700 10 client.?.objecter ms_handle_connect 0x7fb7d4063b70 2021-05-25T11:59:58.915 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.877+0000 7fb7c7fff700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:58.916 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.877+0000 7fb7c7fff700 1 -- 172.21.15.102:0/321053627 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2044978356 0 0) 0x7fb7d409ab70 con 0x7fb7d4097fa0 2021-05-25T11:59:58.916 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.877+0000 7fb7c7fff700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:58.916 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.877+0000 7fb7c7fff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:58.916 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.877+0000 7fb7c7fff700 10 monclient(hunting): my global_id is 4347 2021-05-25T11:59:58.917 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.877+0000 7fb7c7fff700 1 -- 172.21.15.102:0/321053627 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7d411c460 con 0x7fb7d4097fa0 2021-05-25T11:59:58.917 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.877+0000 7fb7c7fff700 1 -- 172.21.15.102:0/321053627 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2827555198 0 0) 0x7fb7d4132400 con 0x7fb7d4063b70 2021-05-25T11:59:58.917 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.877+0000 7fb7c7fff700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:58.917 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.877+0000 7fb7c7fff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:58.917 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.877+0000 7fb7c7fff700 10 monclient(hunting): my global_id is 4342 2021-05-25T11:59:58.918 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.877+0000 7fb7c7fff700 1 -- 172.21.15.102:0/321053627 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7d409ab70 con 0x7fb7d4063b70 2021-05-25T11:59:58.918 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.877+0000 7fb7c7fff700 1 -- 172.21.15.102:0/321053627 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2192330602 0 0) 0x7fb7b0004c10 con 0x7fb7d41325c0 2021-05-25T11:59:58.918 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.877+0000 7fb7c7fff700 1 -- 172.21.15.102:0/321053627 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb7d4132400 con 0x7fb7d41325c0 2021-05-25T11:59:58.918 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.877+0000 7fb7c7fff700 1 -- 172.21.15.102:0/321053627 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fb7c8001e20 con 0x7fb7d41325c0 2021-05-25T11:59:58.919 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.877+0000 7fb7c7fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:58.919 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.877+0000 7fb7c7fff700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T11:59:58.919 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.877+0000 7fb7c7fff700 10 monclient(hunting): dump: 2021-05-25T11:59:58.919 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:58.920 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:58.920 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:58.920 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:58.920 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:58.920 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:58.921 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:58.921 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:58.921 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:58.921 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:58.922 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.877+0000 7fb7c7fff700 10 monclient(hunting): _finish_auth 0 2021-05-25T11:59:58.922 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.877+0000 7fb7c7fff700 1 -- 172.21.15.102:0/321053627 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3429610656 0 0) 0x7fb7d411c460 con 0x7fb7d4097fa0 2021-05-25T11:59:58.922 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.877+0000 7fb7c7fff700 1 -- 172.21.15.102:0/321053627 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb7b0004c10 con 0x7fb7d4097fa0 2021-05-25T11:59:58.922 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.877+0000 7fb7c7fff700 1 -- 172.21.15.102:0/321053627 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fb7c0001840 con 0x7fb7d4097fa0 2021-05-25T11:59:58.923 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.877+0000 7fb7c7fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:58.923 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.877+0000 7fb7c7fff700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2021-05-25T11:59:58.923 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.877+0000 7fb7c7fff700 10 monclient(hunting): dump: 2021-05-25T11:59:58.923 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:58.923 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:58.924 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:58.924 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:58.924 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:58.924 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:58.925 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:58.925 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:58.925 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:58.925 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:58.926 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.877+0000 7fb7c7fff700 1 -- 172.21.15.102:0/321053627 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1508375573 0 0) 0x7fb7d409ab70 con 0x7fb7d4063b70 2021-05-25T11:59:58.926 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.877+0000 7fb7c7fff700 1 -- 172.21.15.102:0/321053627 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb7d411c460 con 0x7fb7d4063b70 2021-05-25T11:59:58.926 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.877+0000 7fb7c7fff700 1 -- 172.21.15.102:0/321053627 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4256459033 0 0) 0x7fb7b0004c10 con 0x7fb7d4097fa0 2021-05-25T11:59:58.926 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.877+0000 7fb7c7fff700 1 -- 172.21.15.102:0/321053627 >> v1:172.21.15.102:6790/0 conn(0x7fb7d41325c0 legacy=0x7fb7d4143520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:58.926 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.877+0000 7fb7c7fff700 1 -- 172.21.15.102:0/321053627 >> v1:172.21.15.181:6789/0 conn(0x7fb7d4063b70 legacy=0x7fb7d413b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:58.927 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.877+0000 7fb7c7fff700 10 monclient: _finish_hunting 0 2021-05-25T11:59:58.927 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.877+0000 7fb7c7fff700 1 monclient: found mon.a 2021-05-25T11:59:58.927 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.877+0000 7fb7c7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T11:59:58.927 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.877+0000 7fb7c7fff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:58.928 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.877+0000 7fb7c7fff700 1 -- 172.21.15.102:0/321053627 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7d411c700 con 0x7fb7d4097fa0 2021-05-25T11:59:58.928 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.877+0000 7fb7c7fff700 10 monclient: _finish_auth 0 2021-05-25T11:59:58.928 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.877+0000 7fb7c7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T11:59:58.928 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.877+0000 7fb7da826700 5 monclient: authenticate success, global_id 4347 2021-05-25T11:59:58.929 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.877+0000 7fb7da826700 10 monclient: _renew_subs 2021-05-25T11:59:58.929 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.877+0000 7fb7da826700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:58.930 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.877+0000 7fb7da826700 1 -- 172.21.15.102:0/321053627 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7d411b230 con 0x7fb7d4097fa0 2021-05-25T11:59:58.930 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.877+0000 7fb7da826700 10 client.4347.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T11:59:58.931 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.877+0000 7fb7da826700 10 monclient: _renew_subs 2021-05-25T11:59:58.931 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.877+0000 7fb7da826700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:58.931 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.877+0000 7fb7da826700 1 -- 172.21.15.102:0/321053627 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7d4119bf0 con 0x7fb7d4097fa0 2021-05-25T11:59:58.931 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.877+0000 7fb7da826700 1 librados: init done 2021-05-25T11:59:58.932 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.877+0000 7fb7c7fff700 1 -- 172.21.15.102:0/321053627 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb7c0001e30 con 0x7fb7d4097fa0 2021-05-25T11:59:58.932 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.877+0000 7fb7c7fff700 10 monclient: handle_config config(0 keys) v1 2021-05-25T11:59:58.932 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.877+0000 7fb7c7fff700 1 -- 172.21.15.102:0/321053627 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fb7c0002140 con 0x7fb7d4097fa0 2021-05-25T11:59:58.932 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.877+0000 7fb7c7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:58.933 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.877+0000 7fb7c7fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T11:59:58.933 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.877+0000 7fb7d27fc700 4 set_mon_vals no callback set 2021-05-25T11:59:58.933 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.877+0000 7fb7c7fff700 10 monclient: dump: 2021-05-25T11:59:58.933 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:58.933 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:58.934 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:58.934 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:58.934 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:58.934 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:58.935 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:58.935 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:58.935 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:58.935 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:58.935 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.878+0000 7fb7c7fff700 1 -- 172.21.15.102:0/321053627 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7fb7c0002320 con 0x7fb7d4097fa0 2021-05-25T11:59:58.936 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.878+0000 7fb7da826700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:58.936 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.878+0000 7fb7c7fff700 1 -- 172.21.15.102:0/321053627 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(15..15 src has 1..15) v4 ==== 3549+0+0 (unknown 3948819498 0 0) 0x7fb7c003cc60 con 0x7fb7d4097fa0 2021-05-25T11:59:58.936 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.878+0000 7fb7c7fff700 10 client.4347.objecter ms_dispatch 0x7fb7d4001550 osd_map(15..15 src has 1..15) v4 2021-05-25T11:59:58.936 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.878+0000 7fb7c7fff700 3 client.4347.objecter handle_osd_map got epochs [15,15] > 0 2021-05-25T11:59:58.937 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.878+0000 7fb7da826700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:58.937 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.878+0000 7fb7c7fff700 3 client.4347.objecter handle_osd_map decoding full epoch 15 2021-05-25T11:59:58.937 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.878+0000 7fb7da826700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:58.937 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.878+0000 7fb7da826700 1 -- 172.21.15.102:0/321053627 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7b4004db0 con 0x7fb7d4097fa0 2021-05-25T11:59:58.937 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.878+0000 7fb7c7fff700 20 client.4347.objecter dump_active .. 0 homeless 2021-05-25T11:59:58.938 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.878+0000 7fb7d3fff700 10 monclient: get_auth_request con 0x7fb7b00334d0 auth_method 0 2021-05-25T11:59:58.938 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.878+0000 7fb7c7fff700 10 client.4347.objecter ms_handle_connect 0x7fb7b00334d0 2021-05-25T11:59:58.938 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.882+0000 7fb7c7fff700 1 -- 172.21.15.102:0/321053627 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7fb7c0002ca0 con 0x7fb7d4097fa0 2021-05-25T11:59:58.938 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.882+0000 7fb7c7fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:58.939 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:58.882+0000 7fb7c7fff700 10 monclient: _finish_command 1 = system:0 2021-05-25T11:59:59.029 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.027+0000 7fb7da826700 1 -- 172.21.15.102:0/321053627 --> v1:172.21.15.181:6812/29582 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb7b4000b90 con 0x7fb7b00334d0 2021-05-25T11:59:59.031 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.029+0000 7fb7c7fff700 1 -- 172.21.15.102:0/321053627 <== mgr.4105 v1:172.21.15.181:6812/29582 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+38639 (unknown 2965378022 0 1705510130) 0x7fb7b4000b90 con 0x7fb7b00334d0 2021-05-25T11:59:59.032 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:59:59.033 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.030+0000 7fb7da826700 10 librados: watch_flush enter 2021-05-25T11:59:59.034 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.030+0000 7fb7da826700 10 librados: watch_flush exit 2021-05-25T11:59:59.034 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.030+0000 7fb7da826700 20 client.4347.objecter shutdown clearing up homeless session... 2021-05-25T11:59:59.034 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.030+0000 7fb7da826700 10 client.4347.objecter successfully canceled tick 2021-05-25T11:59:59.034 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.030+0000 7fb7da826700 1 -- 172.21.15.102:0/321053627 >> v1:172.21.15.181:6812/29582 conn(0x7fb7b00334d0 legacy=0x7fb7b0035990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:59.035 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.030+0000 7fb7da826700 10 monclient: shutdown 2021-05-25T11:59:59.035 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.030+0000 7fb7da826700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T11:59:59.035 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.030+0000 7fb7da826700 1 -- 172.21.15.102:0/321053627 >> v1:172.21.15.102:6789/0 conn(0x7fb7d4097fa0 legacy=0x7fb7d4131cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:59.035 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.030+0000 7fb7da826700 1 -- 172.21.15.102:0/321053627 shutdown_connections 2021-05-25T11:59:59.036 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.030+0000 7fb7da826700 1 -- 172.21.15.102:0/321053627 >> 172.21.15.102:0/321053627 conn(0x7fb7d409ef50 msgr2=0x7fb7d4096aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T11:59:59.036 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.031+0000 7fb7da826700 1 -- 172.21.15.102:0/321053627 shutdown_connections 2021-05-25T11:59:59.036 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.031+0000 7fb7da826700 1 -- 172.21.15.102:0/321053627 wait complete. 2021-05-25T11:59:59.036 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.031+0000 7fb7da826700 1 librados: shutdown 2021-05-25T11:59:59.036 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2021-05-25T11:59:59.044 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2021-05-25T11:59:57.556206+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":11,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":29544,"kb_used_data":760,"kb_used_omap":0,"kb_used_meta":28736,"kb_avail":566201496,"statfs":{"total":579820584960,"available":579790331904,"internally_reserved":0,"allocated":778240,"data_stored":230516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29425664},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2021-05-25T11:59:52.780051+0000","last_change":"2021-05-25T11:59:52.779532+0000","last_active":"2021-05-25T11:59:52.780051+0000","last_peered":"2021-05-25T11:59:52.780051+0000","last_clean":"2021-05-25T11:59:52.780051+0000","last_became_active":"2021-05-25T11:59:52.779240+0000","last_became_peered":"2021-05-25T11:59:52.779240+0000","last_unstale":"2021-05-25T11:59:52.780051+0000","last_undegraded":"2021-05-25T11:59:52.780051+0000","last_fullsized":"2021-05-25T11:59:52.780051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-05-25T11:59:51.583021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-05-25T11:59:51.583021+0000","last_clean_scrub_stamp":"2021-05-25T11:59:51.583021+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2021-05-25T11:59:52.600778+0000","last_change":"2021-05-25T11:59:52.599864+0000","last_active":"2021-05-25T11:59:52.600778+0000","last_peered":"2021-05-25T11:59:52.600778+0000","last_clean":"2021-05-25T11:59:52.600778+0000","last_became_active":"2021-05-25T11:59:52.599550+0000","last_became_peered":"2021-05-25T11:59:52.599550+0000","last_unstale":"2021-05-25T11:59:52.600778+0000","last_undegraded":"2021-05-25T11:59:52.600778+0000","last_fullsized":"2021-05-25T11:59:52.600778+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-05-25T11:59:51.583021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-05-25T11:59:51.583021+0000","last_clean_scrub_stamp":"2021-05-25T11:59:51.583021+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2021-05-25T11:59:52.599789+0000","last_change":"2021-05-25T11:59:52.599209+0000","last_active":"2021-05-25T11:59:52.599789+0000","last_peered":"2021-05-25T11:59:52.599789+0000","last_clean":"2021-05-25T11:59:52.599789+0000","last_became_active":"2021-05-25T11:59:52.598818+0000","last_became_peered":"2021-05-25T11:59:52.598818+0000","last_unstale":"2021-05-25T11:59:52.599789+0000","last_undegraded":"2021-05-25T11:59:52.599789+0000","last_fullsized":"2021-05-25T11:59:52.599789+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-05-25T11:59:51.583021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-05-25T11:59:51.583021+0000","last_clean_scrub_stamp":"2021-05-25T11:59:51.583021+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2021-05-25T11:59:52.599030+0000","last_change":"2021-05-25T11:59:52.596805+0000","last_active":"2021-05-25T11:59:52.599030+0000","last_peered":"2021-05-25T11:59:52.599030+0000","last_clean":"2021-05-25T11:59:52.599030+0000","last_became_active":"2021-05-25T11:59:52.596538+0000","last_became_peered":"2021-05-25T11:59:52.596538+0000","last_unstale":"2021-05-25T11:59:52.599030+0000","last_undegraded":"2021-05-25T11:59:52.599030+0000","last_fullsized":"2021-05-25T11:59:52.599030+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-05-25T11:59:51.583021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-05-25T11:59:51.583021+0000","last_clean_scrub_stamp":"2021-05-25T11:59:51.583021+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"10","reported_epoch":"14","state":"active+clean","last_fresh":"2021-05-25T11:59:52.593134+0000","last_change":"2021-05-25T11:59:51.592476+0000","last_active":"2021-05-25T11:59:52.593134+0000","last_peered":"2021-05-25T11:59:52.593134+0000","last_clean":"2021-05-25T11:59:52.593134+0000","last_became_active":"2021-05-25T11:59:51.592157+0000","last_became_peered":"2021-05-25T11:59:51.592157+0000","last_unstale":"2021-05-25T11:59:52.593134+0000","last_undegraded":"2021-05-25T11:59:52.593134+0000","last_fullsized":"2021-05-25T11:59:52.593134+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-05-25T11:59:50.578675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-05-25T11:59:50.578675+0000","last_clean_scrub_stamp":"2021-05-25T11:59:50.578675+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2021-05-25T11:59:52.607326+0000","last_change":"2021-05-25T11:59:52.602323+0000","last_active":"2021-05-25T11:59:52.607326+0000","last_peered":"2021-05-25T11:59:52.607326+0000","last_clean":"2021-05-25T11:59:52.607326+0000","last_became_active":"2021-05-25T11:59:52.602047+0000","last_became_peered":"2021-05-25T11:59:52.602047+0000","last_unstale":"2021-05-25T11:59:52.607326+0000","last_undegraded":"2021-05-25T11:59:52.607326+0000","last_fullsized":"2021-05-25T11:59:52.607326+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-05-25T11:59:51.583021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-05-25T11:59:51.583021+0000","last_clean_scrub_stamp":"2021-05-25T11:59:51.583021+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2021-05-25T11:59:52.600845+0000","last_change":"2021-05-25T11:59:52.598411+0000","last_active":"2021-05-25T11:59:52.600845+0000","last_peered":"2021-05-25T11:59:52.600845+0000","last_clean":"2021-05-25T11:59:52.600845+0000","last_became_active":"2021-05-25T11:59:52.598090+0000","last_became_peered":"2021-05-25T11:59:52.598090+0000","last_unstale":"2021-05-25T11:59:52.600845+0000","last_undegraded":"2021-05-25T11:59:52.600845+0000","last_fullsized":"2021-05-25T11:59:52.600845+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-05-25T11:59:51.583021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-05-25T11:59:51.583021+0000","last_clean_scrub_stamp":"2021-05-25T11:59:51.583021+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2021-05-25T11:59:52.607316+0000","last_change":"2021-05-25T11:59:52.602212+0000","last_active":"2021-05-25T11:59:52.607316+0000","last_peered":"2021-05-25T11:59:52.607316+0000","last_clean":"2021-05-25T11:59:52.607316+0000","last_became_active":"2021-05-25T11:59:52.601888+0000","last_became_peered":"2021-05-25T11:59:52.601888+0000","last_unstale":"2021-05-25T11:59:52.607316+0000","last_undegraded":"2021-05-25T11:59:52.607316+0000","last_fullsized":"2021-05-25T11:59:52.607316+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-05-25T11:59:51.583021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-05-25T11:59:51.583021+0000","last_clean_scrub_stamp":"2021-05-25T11:59:51.583021+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2021-05-25T11:59:52.598621+0000","last_change":"2021-05-25T11:59:52.597495+0000","last_active":"2021-05-25T11:59:52.598621+0000","last_peered":"2021-05-25T11:59:52.598621+0000","last_clean":"2021-05-25T11:59:52.598621+0000","last_became_active":"2021-05-25T11:59:52.597224+0000","last_became_peered":"2021-05-25T11:59:52.597224+0000","last_unstale":"2021-05-25T11:59:52.598621+0000","last_undegraded":"2021-05-25T11:59:52.598621+0000","last_fullsized":"2021-05-25T11:59:52.598621+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-05-25T11:59:51.583021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-05-25T11:59:51.583021+0000","last_clean_scrub_stamp":"2021-05-25T11:59:51.583021+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4936,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366904,"statfs":{"total":96636764160,"available":96631709696,"internally_reserved":0,"allocated":131072,"data_stored":38737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40500000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.309}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44600000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35699999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]}]},{"osd":4,"up_from":11,"seq":47244640259,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4868,"kb_used_data":124,"kb_used_omap":0,"kb_used_meta":4736,"kb_avail":94366972,"statfs":{"total":96636764160,"available":96631779328,"internally_reserved":0,"allocated":126976,"data_stored":37784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4849664},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29899999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55100000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47499999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46200000000000002}]}]},{"osd":3,"up_from":11,"seq":47244640259,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4936,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366904,"statfs":{"total":96636764160,"available":96631709696,"internally_reserved":0,"allocated":131072,"data_stored":38737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64700000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51200000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68799999999999994}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33800000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.377}]}]},{"osd":2,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4936,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366904,"statfs":{"total":96636764160,"available":96631709696,"internally_reserved":0,"allocated":131072,"data_stored":38737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.216}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.22600000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.5}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49099999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51100000000000001}]}]},{"osd":0,"up_from":11,"seq":47244640260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4936,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366904,"statfs":{"total":96636764160,"available":96631709696,"internally_reserved":0,"allocated":131072,"data_stored":38737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28299999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53500000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52400000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73699999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42899999999999999}]}]},{"osd":1,"up_from":11,"seq":47244640261,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4932,"kb_used_data":124,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366908,"statfs":{"total":96636764160,"available":96631713792,"internally_reserved":0,"allocated":126976,"data_stored":37784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.216}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.254}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47899999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66600000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-05-25T11:59:59.045 INFO:tasks.ceph.ceph_manager.ceph:clean! 2021-05-25T11:59:59.046 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2021-05-25T11:59:59.046 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2021-05-25T11:59:59.047 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2021-05-25T11:59:59.154 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.152+0000 7f5e1b7eb700 10 monclient: get_monmap_and_config 2021-05-25T11:59:59.155 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.152+0000 7f5e1b7eb700 10 monclient: build_initial_monmap 2021-05-25T11:59:59.155 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.152+0000 7f5e1b7eb700 10 monclient: monmap: 2021-05-25T11:59:59.155 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T11:59:59.156 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:59.156 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:59.154000+0000 2021-05-25T11:59:59.156 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:59.154000+0000 2021-05-25T11:59:59.156 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T11:59:59.157 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:59.157 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T11:59:59.157 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T11:59:59.157 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T11:59:59.157 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:59.158 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.152+0000 7f5e1b7eb700 1 Processor -- start 2021-05-25T11:59:59.158 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.152+0000 7f5e1b7eb700 1 -- start start 2021-05-25T11:59:59.158 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.152+0000 7f5e1b7eb700 10 monclient: init 2021-05-25T11:59:59.169 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.153+0000 7f5e1b7eb700 10 monclient: _reopen_session rank -1 2021-05-25T11:59:59.169 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.153+0000 7f5e1b7eb700 10 monclient: _add_conns ranks=[0,1,2] 2021-05-25T11:59:59.169 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.153+0000 7f5e1b7eb700 10 monclient(hunting): picked mon.noname-a con 0x7f5e14063840 addr v1:172.21.15.102:6789/0 2021-05-25T11:59:59.169 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.153+0000 7f5e1b7eb700 10 monclient(hunting): picked mon.noname-c con 0x7f5e14094da0 addr v1:172.21.15.181:6789/0 2021-05-25T11:59:59.170 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.153+0000 7f5e1b7eb700 10 monclient(hunting): picked mon.noname-b con 0x7f5e141323c0 addr v1:172.21.15.102:6790/0 2021-05-25T11:59:59.170 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.153+0000 7f5e1b7eb700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e14063fb0 con 0x7f5e14063840 2021-05-25T11:59:59.170 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.153+0000 7f5e1b7eb700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e140982f0 con 0x7f5e14094da0 2021-05-25T11:59:59.170 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.153+0000 7f5e1b7eb700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e14064510 con 0x7f5e141323c0 2021-05-25T11:59:59.171 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.153+0000 7f5e1b7eb700 10 monclient(hunting): _renew_subs 2021-05-25T11:59:59.171 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.153+0000 7f5e1b7eb700 10 monclient(hunting): authenticate will time out at 799.751343s 2021-05-25T11:59:59.171 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.153+0000 7f5e19587700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f5e14063840 0x7f5e140974d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:59000/0 (socket says 172.21.15.102:59000) 2021-05-25T11:59:59.171 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.153+0000 7f5e19d88700 1 --1- 172.21.15.102:0/374421632 >> v1:172.21.15.102:6790/0 conn(0x7f5e141323c0 0x7f5e141347b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:53882/0 (socket says 172.21.15.102:53882) 2021-05-25T11:59:59.172 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.153+0000 7f5e19587700 1 -- 172.21.15.102:0/374421632 learned_addr learned my addr 172.21.15.102:0/374421632 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T11:59:59.172 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.153+0000 7f5e03fff700 1 -- 172.21.15.102:0/374421632 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3528510574 0 0) 0x7f5e14064510 con 0x7f5e141323c0 2021-05-25T11:59:59.172 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.153+0000 7f5e03fff700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:59.172 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.153+0000 7f5e03fff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:59.172 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.153+0000 7f5e03fff700 10 monclient(hunting): my global_id is 4343 2021-05-25T11:59:59.173 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.153+0000 7f5e03fff700 1 -- 172.21.15.102:0/374421632 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5dfc003270 con 0x7f5e141323c0 2021-05-25T11:59:59.173 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.153+0000 7f5e03fff700 1 -- 172.21.15.102:0/374421632 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 278917477 0 0) 0x7f5e14063fb0 con 0x7f5e14063840 2021-05-25T11:59:59.173 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.153+0000 7f5e03fff700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:59.173 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.153+0000 7f5e03fff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:59.174 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.153+0000 7f5e03fff700 10 monclient(hunting): my global_id is 4350 2021-05-25T11:59:59.174 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.153+0000 7f5e03fff700 1 -- 172.21.15.102:0/374421632 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e14064510 con 0x7f5e14063840 2021-05-25T11:59:59.174 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.153+0000 7f5e03fff700 1 -- 172.21.15.102:0/374421632 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1061730238 0 0) 0x7f5dfc003270 con 0x7f5e141323c0 2021-05-25T11:59:59.174 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.153+0000 7f5e03fff700 1 -- 172.21.15.102:0/374421632 >> v1:172.21.15.181:6789/0 conn(0x7f5e14094da0 legacy=0x7f5e14097be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:59.174 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.153+0000 7f5e03fff700 1 -- 172.21.15.102:0/374421632 >> v1:172.21.15.102:6789/0 conn(0x7f5e14063840 legacy=0x7f5e140974d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:59.175 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.153+0000 7f5e03fff700 10 monclient: _finish_hunting 0 2021-05-25T11:59:59.175 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.153+0000 7f5e03fff700 1 monclient: found mon.noname-b 2021-05-25T11:59:59.175 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.153+0000 7f5e03fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T11:59:59.175 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.153+0000 7f5e03fff700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T11:59:59.176 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.153+0000 7f5e03fff700 1 -- 172.21.15.102:0/374421632 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e140647b0 con 0x7f5e141323c0 2021-05-25T11:59:59.176 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.153+0000 7f5e03fff700 10 monclient: _finish_auth 0 2021-05-25T11:59:59.176 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.153+0000 7f5e03fff700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T11:59:59.176 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.153+0000 7f5e03fff700 1 -- 172.21.15.102:0/374421632 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f5e10001990 con 0x7f5e141323c0 2021-05-25T11:59:59.177 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.154+0000 7f5e1b7eb700 5 monclient: authenticate success, global_id 4343 2021-05-25T11:59:59.178 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.154+0000 7f5e1b7eb700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T11:59:59.178 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.154+0000 7f5e03fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:59.178 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.154+0000 7f5e03fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T11:59:59.178 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.154+0000 7f5e03fff700 10 monclient: dump: 2021-05-25T11:59:59.179 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:59.179 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:59.179 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:59.179 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:59.180 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:59.180 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:59.180 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:59.180 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:59.180 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:59.181 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:59.181 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.154+0000 7f5e1b7eb700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T11:59:59.181 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.154+0000 7f5e03fff700 1 -- 172.21.15.102:0/374421632 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5e10001b40 con 0x7f5e141323c0 2021-05-25T11:59:59.181 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.154+0000 7f5e03fff700 10 monclient: handle_config config(0 keys) v1 2021-05-25T11:59:59.182 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.154+0000 7f5e03fff700 1 -- 172.21.15.102:0/374421632 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f5e10001ed0 con 0x7f5e141323c0 2021-05-25T11:59:59.182 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.154+0000 7f5e1b7eb700 10 monclient: get_monmap_and_config success 2021-05-25T11:59:59.182 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.154+0000 7f5e1b7eb700 4 set_mon_vals no callback set 2021-05-25T11:59:59.182 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.154+0000 7f5e03fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:59.183 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.154+0000 7f5e03fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T11:59:59.183 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.154+0000 7f5e03fff700 10 monclient: dump: 2021-05-25T11:59:59.183 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:59.183 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:59.184 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:59.184 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:59.184 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:59.184 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:59.184 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:59.185 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:59.185 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:59.185 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:59.185 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.154+0000 7f5e1b7eb700 10 monclient: shutdown 2021-05-25T11:59:59.185 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.154+0000 7f5e1b7eb700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T11:59:59.186 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.154+0000 7f5e1b7eb700 1 -- 172.21.15.102:0/374421632 >> v1:172.21.15.102:6790/0 conn(0x7f5e141323c0 legacy=0x7f5e141347b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:59.186 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.154+0000 7f5e1b7eb700 1 -- 172.21.15.102:0/374421632 shutdown_connections 2021-05-25T11:59:59.186 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.154+0000 7f5e1b7eb700 1 -- 172.21.15.102:0/374421632 >> 172.21.15.102:0/374421632 conn(0x7f5e1409cb30 msgr2=0x7f5e1409cf40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T11:59:59.186 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.154+0000 7f5e1b7eb700 1 -- 172.21.15.102:0/374421632 shutdown_connections 2021-05-25T11:59:59.187 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.154+0000 7f5e1b7eb700 1 -- 172.21.15.102:0/374421632 wait complete. 2021-05-25T11:59:59.187 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.155+0000 7f5e1b7eb700 10 monclient: build_initial_monmap 2021-05-25T11:59:59.187 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.155+0000 7f5e1b7eb700 10 monclient: monmap: 2021-05-25T11:59:59.187 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T11:59:59.187 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T11:59:59.188 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T11:59:59.188 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T11:59:59.188 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T11:59:59.188 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T11:59:59.188 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T11:59:59.189 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T11:59:59.189 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T11:59:59.189 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:59.189 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.155+0000 7f5e1b7eb700 1 librados: starting msgr at 2021-05-25T11:59:59.190 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.155+0000 7f5e1b7eb700 1 librados: starting objecter 2021-05-25T11:59:59.191 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.155+0000 7f5e1b7eb700 1 Processor -- start 2021-05-25T11:59:59.191 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.155+0000 7f5e1b7eb700 1 -- start start 2021-05-25T11:59:59.191 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.155+0000 7f5e1b7eb700 1 librados: setting wanted keys 2021-05-25T11:59:59.191 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.155+0000 7f5e1b7eb700 1 librados: calling monclient init 2021-05-25T11:59:59.191 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.155+0000 7f5e1b7eb700 10 monclient: init 2021-05-25T11:59:59.192 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.155+0000 7f5e1b7eb700 10 monclient: _reopen_session rank -1 2021-05-25T11:59:59.192 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.155+0000 7f5e1b7eb700 10 monclient: _add_conns ranks=[0,1,2] 2021-05-25T11:59:59.192 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.156+0000 7f5e1b7eb700 10 monclient(hunting): picked mon.noname-a con 0x7f5e14063840 addr v1:172.21.15.102:6789/0 2021-05-25T11:59:59.192 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.156+0000 7f5e1b7eb700 10 monclient(hunting): picked mon.noname-b con 0x7f5e14094da0 addr v1:172.21.15.181:6789/0 2021-05-25T11:59:59.192 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.156+0000 7f5e1b7eb700 10 monclient(hunting): picked mon.noname-c con 0x7f5e141323c0 addr v1:172.21.15.102:6790/0 2021-05-25T11:59:59.193 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.156+0000 7f5e1b7eb700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e1411c140 con 0x7f5e14063840 2021-05-25T11:59:59.193 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.156+0000 7f5e1b7eb700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e1411c3e0 con 0x7f5e14094da0 2021-05-25T11:59:59.193 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.156+0000 7f5e1b7eb700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e1411c680 con 0x7f5e141323c0 2021-05-25T11:59:59.193 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.156+0000 7f5e1b7eb700 10 monclient(hunting): _renew_subs 2021-05-25T11:59:59.194 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.156+0000 7f5e1b7eb700 10 monclient(hunting): authenticate will time out at 799.754044s 2021-05-25T11:59:59.194 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.156+0000 7f5e19587700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f5e14063840 0x7f5e14118960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:59006/0 (socket says 172.21.15.102:59006) 2021-05-25T11:59:59.194 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.156+0000 7f5e19587700 1 -- 172.21.15.102:0/950178583 learned_addr learned my addr 172.21.15.102:0/950178583 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T11:59:59.194 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.156+0000 7f5e017fa700 10 client.?.objecter ms_handle_connect 0x7f5e141323c0 2021-05-25T11:59:59.194 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.156+0000 7f5e017fa700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:59.195 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.156+0000 7f5e017fa700 10 client.?.objecter ms_handle_connect 0x7f5e14063840 2021-05-25T11:59:59.195 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.156+0000 7f5e017fa700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:59.195 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.156+0000 7f5e017fa700 1 -- 172.21.15.102:0/950178583 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1955578361 0 0) 0x7f5e1411c680 con 0x7f5e141323c0 2021-05-25T11:59:59.196 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.156+0000 7f5e017fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:59.196 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.156+0000 7f5e017fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:59.196 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.156+0000 7f5e017fa700 10 monclient(hunting): my global_id is 4346 2021-05-25T11:59:59.196 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.156+0000 7f5e017fa700 1 -- 172.21.15.102:0/950178583 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5df4004c10 con 0x7f5e141323c0 2021-05-25T11:59:59.196 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.156+0000 7f5e017fa700 1 -- 172.21.15.102:0/950178583 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2273137558 0 0) 0x7f5e1411c140 con 0x7f5e14063840 2021-05-25T11:59:59.197 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.156+0000 7f5e017fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:59.197 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.156+0000 7f5e017fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:59.197 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.156+0000 7f5e017fa700 10 monclient(hunting): my global_id is 4353 2021-05-25T11:59:59.197 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.156+0000 7f5e017fa700 1 -- 172.21.15.102:0/950178583 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e1411c680 con 0x7f5e14063840 2021-05-25T11:59:59.198 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.156+0000 7f5e017fa700 10 client.?.objecter ms_handle_connect 0x7f5e14094da0 2021-05-25T11:59:59.198 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.156+0000 7f5e017fa700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:59.198 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.156+0000 7f5e017fa700 1 -- 172.21.15.102:0/950178583 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 615944770 0 0) 0x7f5df4004c10 con 0x7f5e141323c0 2021-05-25T11:59:59.198 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.156+0000 7f5e017fa700 1 -- 172.21.15.102:0/950178583 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5e1411c140 con 0x7f5e141323c0 2021-05-25T11:59:59.198 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.156+0000 7f5e017fa700 1 -- 172.21.15.102:0/950178583 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f5e10001950 con 0x7f5e141323c0 2021-05-25T11:59:59.199 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.156+0000 7f5e017fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:59.199 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.156+0000 7f5e017fa700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T11:59:59.199 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.156+0000 7f5e017fa700 10 monclient(hunting): dump: 2021-05-25T11:59:59.199 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:59.200 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:59.200 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:59.200 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:59.200 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:59.201 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:59.201 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:59.201 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:59.201 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:59.201 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:59.202 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.156+0000 7f5e017fa700 10 monclient(hunting): _finish_auth 0 2021-05-25T11:59:59.202 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.156+0000 7f5e017fa700 1 -- 172.21.15.102:0/950178583 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4166925827 0 0) 0x7f5e1411c3e0 con 0x7f5e14094da0 2021-05-25T11:59:59.202 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.156+0000 7f5e017fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:59.202 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.156+0000 7f5e017fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:59.203 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.156+0000 7f5e017fa700 10 monclient(hunting): my global_id is 4348 2021-05-25T11:59:59.203 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.156+0000 7f5e017fa700 1 -- 172.21.15.102:0/950178583 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5df4004c10 con 0x7f5e14094da0 2021-05-25T11:59:59.203 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.156+0000 7f5e017fa700 1 -- 172.21.15.102:0/950178583 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 211928409 0 0) 0x7f5e1411c680 con 0x7f5e14063840 2021-05-25T11:59:59.203 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.156+0000 7f5e017fa700 1 -- 172.21.15.102:0/950178583 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5e1411c3e0 con 0x7f5e14063840 2021-05-25T11:59:59.203 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.156+0000 7f5e017fa700 1 -- 172.21.15.102:0/950178583 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f5e08001780 con 0x7f5e14063840 2021-05-25T11:59:59.204 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.156+0000 7f5e017fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:59.204 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.156+0000 7f5e017fa700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2021-05-25T11:59:59.204 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.156+0000 7f5e017fa700 10 monclient(hunting): dump: 2021-05-25T11:59:59.204 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:59.204 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:59.205 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:59.205 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:59.205 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:59.205 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:59.206 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:59.206 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:59.206 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:59.206 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:59.206 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.156+0000 7f5e017fa700 1 -- 172.21.15.102:0/950178583 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1390714439 0 0) 0x7f5e1411c140 con 0x7f5e141323c0 2021-05-25T11:59:59.207 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.156+0000 7f5e017fa700 1 -- 172.21.15.102:0/950178583 >> v1:172.21.15.181:6789/0 conn(0x7f5e14094da0 legacy=0x7f5e14119070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:59.207 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.156+0000 7f5e017fa700 1 -- 172.21.15.102:0/950178583 >> v1:172.21.15.102:6789/0 conn(0x7f5e14063840 legacy=0x7f5e14118960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:59.207 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.156+0000 7f5e017fa700 10 monclient: _finish_hunting 0 2021-05-25T11:59:59.207 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.156+0000 7f5e017fa700 1 monclient: found mon.c 2021-05-25T11:59:59.208 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.156+0000 7f5e017fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T11:59:59.208 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.156+0000 7f5e017fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T11:59:59.208 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.156+0000 7f5e017fa700 1 -- 172.21.15.102:0/950178583 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e1411a510 con 0x7f5e141323c0 2021-05-25T11:59:59.208 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.156+0000 7f5e017fa700 10 monclient: _finish_auth 0 2021-05-25T11:59:59.208 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.156+0000 7f5e017fa700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T11:59:59.209 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.156+0000 7f5e1b7eb700 5 monclient: authenticate success, global_id 4346 2021-05-25T11:59:59.209 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.156+0000 7f5e1b7eb700 10 monclient: _renew_subs 2021-05-25T11:59:59.209 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.156+0000 7f5e1b7eb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T11:59:59.209 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.156+0000 7f5e1b7eb700 1 -- 172.21.15.102:0/950178583 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e14132050 con 0x7f5e141323c0 2021-05-25T11:59:59.210 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.156+0000 7f5e1b7eb700 10 client.4346.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T11:59:59.210 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.156+0000 7f5e1b7eb700 10 monclient: _renew_subs 2021-05-25T11:59:59.210 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.156+0000 7f5e1b7eb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T11:59:59.210 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.156+0000 7f5e1b7eb700 1 -- 172.21.15.102:0/950178583 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e1411cc00 con 0x7f5e141323c0 2021-05-25T11:59:59.210 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.156+0000 7f5e1b7eb700 1 librados: init done 2021-05-25T11:59:59.211 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.156+0000 7f5e017fa700 1 -- 172.21.15.102:0/950178583 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5e10004c20 con 0x7f5e141323c0 2021-05-25T11:59:59.211 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.156+0000 7f5e017fa700 10 monclient: handle_config config(0 keys) v1 2021-05-25T11:59:59.211 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.156+0000 7f5e017fa700 1 -- 172.21.15.102:0/950178583 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f5e10004f30 con 0x7f5e141323c0 2021-05-25T11:59:59.211 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.156+0000 7f5e017fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:59.212 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.156+0000 7f5e037fe700 4 set_mon_vals no callback set 2021-05-25T11:59:59.212 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.156+0000 7f5e017fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T11:59:59.212 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.156+0000 7f5e017fa700 10 monclient: dump: 2021-05-25T11:59:59.212 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:59.212 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:59.213 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:59.213 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:59.213 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:59.213 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:59.213 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:59.214 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:59.214 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:59.214 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:59.214 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.157+0000 7f5e017fa700 1 -- 172.21.15.102:0/950178583 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f5e10005110 con 0x7f5e141323c0 2021-05-25T11:59:59.214 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.157+0000 7f5e1b7eb700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:59.215 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.157+0000 7f5e1b7eb700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:59.215 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.157+0000 7f5e1b7eb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T11:59:59.215 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.157+0000 7f5e1b7eb700 1 -- 172.21.15.102:0/950178583 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dec004db0 con 0x7f5e141323c0 2021-05-25T11:59:59.215 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.157+0000 7f5e017fa700 1 -- 172.21.15.102:0/950178583 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(15..15 src has 1..15) v4 ==== 3549+0+0 (unknown 3948819498 0 0) 0x7f5e1003d330 con 0x7f5e141323c0 2021-05-25T11:59:59.216 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.157+0000 7f5e017fa700 10 client.4346.objecter ms_dispatch 0x7f5e14001550 osd_map(15..15 src has 1..15) v4 2021-05-25T11:59:59.216 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.157+0000 7f5e017fa700 3 client.4346.objecter handle_osd_map got epochs [15,15] > 0 2021-05-25T11:59:59.216 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.157+0000 7f5e017fa700 3 client.4346.objecter handle_osd_map decoding full epoch 15 2021-05-25T11:59:59.216 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.157+0000 7f5e017fa700 20 client.4346.objecter dump_active .. 0 homeless 2021-05-25T11:59:59.216 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.157+0000 7f5e19587700 10 monclient: get_auth_request con 0x7f5df4033390 auth_method 0 2021-05-25T11:59:59.217 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.158+0000 7f5e017fa700 10 client.4346.objecter ms_handle_connect 0x7f5df4033390 2021-05-25T11:59:59.217 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.161+0000 7f5e017fa700 1 -- 172.21.15.102:0/950178583 <== mon.2 v1:172.21.15.102:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7f5e100130b0 con 0x7f5e141323c0 2021-05-25T11:59:59.217 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.161+0000 7f5e017fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:59.217 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.161+0000 7f5e017fa700 10 monclient: _finish_command 1 = system:0 2021-05-25T11:59:59.371 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.368+0000 7f5e1b7eb700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2021-05-25T11:59:59.371 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.368+0000 7f5e1b7eb700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2021-05-25T11:59:59.371 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.368+0000 7f5e1b7eb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T11:59:59.371 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.368+0000 7f5e1b7eb700 1 -- 172.21.15.102:0/950178583 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f5dec0055e0 con 0x7f5e141323c0 2021-05-25T11:59:59.372 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.368+0000 7f5e017fa700 1 -- 172.21.15.102:0/950178583 <== mon.2 v1:172.21.15.102:6790/0 10 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (unknown 3730786198 0 4185958460) 0x7f5e10012ed0 con 0x7f5e141323c0 2021-05-25T11:59:59.372 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.368+0000 7f5e017fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2021-05-25T11:59:59.372 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.368+0000 7f5e017fa700 10 monclient: _finish_command 2 = system:0 2021-05-25T11:59:59.373 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:59:59.373 INFO:teuthology.orchestra.run.smithi102.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2021-05-25T11:59:59.375 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.369+0000 7f5e1b7eb700 10 librados: watch_flush enter 2021-05-25T11:59:59.375 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.369+0000 7f5e1b7eb700 10 librados: watch_flush exit 2021-05-25T11:59:59.375 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.369+0000 7f5e1b7eb700 20 client.4346.objecter shutdown clearing up homeless session... 2021-05-25T11:59:59.375 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.369+0000 7f5e1b7eb700 10 client.4346.objecter successfully canceled tick 2021-05-25T11:59:59.376 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.370+0000 7f5e1b7eb700 1 -- 172.21.15.102:0/950178583 >> v1:172.21.15.181:6812/29582 conn(0x7f5df4033390 legacy=0x7f5df4035850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:59.376 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.370+0000 7f5e1b7eb700 10 monclient: shutdown 2021-05-25T11:59:59.376 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.370+0000 7f5e1b7eb700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T11:59:59.376 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.370+0000 7f5e1b7eb700 1 -- 172.21.15.102:0/950178583 >> v1:172.21.15.102:6790/0 conn(0x7f5e141323c0 legacy=0x7f5e14143880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:59.377 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.370+0000 7f5e1b7eb700 1 -- 172.21.15.102:0/950178583 shutdown_connections 2021-05-25T11:59:59.377 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.370+0000 7f5e1b7eb700 1 -- 172.21.15.102:0/950178583 >> 172.21.15.102:0/950178583 conn(0x7f5e1409cb30 msgr2=0x7f5e14067d30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T11:59:59.377 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.370+0000 7f5e1b7eb700 1 -- 172.21.15.102:0/950178583 shutdown_connections 2021-05-25T11:59:59.377 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.370+0000 7f5e1b7eb700 1 -- 172.21.15.102:0/950178583 wait complete. 2021-05-25T11:59:59.377 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.370+0000 7f5e1b7eb700 1 librados: shutdown 2021-05-25T11:59:59.383 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2021-05-25T11:59:59.384 INFO:teuthology.run_tasks:Running task mon_thrash... 2021-05-25T11:59:59.398 INFO:tasks.mon_thrash:Beginning mon_thrash... 2021-05-25T11:59:59.398 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-05-25T11:59:59.507 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.505+0000 7fa3b0b45700 10 monclient: get_monmap_and_config 2021-05-25T11:59:59.507 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.505+0000 7fa3b0b45700 10 monclient: build_initial_monmap 2021-05-25T11:59:59.508 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.505+0000 7fa3b0b45700 10 monclient: monmap: 2021-05-25T11:59:59.508 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T11:59:59.508 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:59.508 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:59.507071+0000 2021-05-25T11:59:59.508 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:59.507071+0000 2021-05-25T11:59:59.509 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T11:59:59.509 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:59.509 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T11:59:59.509 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T11:59:59.509 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T11:59:59.510 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:59.519 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.505+0000 7fa3b0b45700 1 Processor -- start 2021-05-25T11:59:59.519 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.505+0000 7fa3b0b45700 1 -- start start 2021-05-25T11:59:59.520 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.505+0000 7fa3b0b45700 10 monclient: init 2021-05-25T11:59:59.520 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.506+0000 7fa3b0b45700 10 monclient: _reopen_session rank -1 2021-05-25T11:59:59.520 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.506+0000 7fa3b0b45700 10 monclient: _add_conns ranks=[0,1,2] 2021-05-25T11:59:59.520 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.506+0000 7fa3b0b45700 10 monclient(hunting): picked mon.noname-a con 0x7fa3ac097fa0 addr v1:172.21.15.102:6789/0 2021-05-25T11:59:59.521 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.506+0000 7fa3b0b45700 10 monclient(hunting): picked mon.noname-c con 0x7fa3ac063b70 addr v1:172.21.15.181:6789/0 2021-05-25T11:59:59.521 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.506+0000 7fa3b0b45700 10 monclient(hunting): picked mon.noname-b con 0x7fa3ac1325c0 addr v1:172.21.15.102:6790/0 2021-05-25T11:59:59.522 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.506+0000 7fa3b0b45700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3ac096b20 con 0x7fa3ac097fa0 2021-05-25T11:59:59.522 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.506+0000 7fa3b0b45700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3ac09b460 con 0x7fa3ac063b70 2021-05-25T11:59:59.522 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.506+0000 7fa3b0b45700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3ac0964c0 con 0x7fa3ac1325c0 2021-05-25T11:59:59.523 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.506+0000 7fa3b0b45700 10 monclient(hunting): _renew_subs 2021-05-25T11:59:59.523 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.506+0000 7fa3b0b45700 10 monclient(hunting): authenticate will time out at 800.104326s 2021-05-25T11:59:59.523 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.506+0000 7fa3aad9d700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fa3ac1325c0 0x7fa3ac1349b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:53896/0 (socket says 172.21.15.102:53896) 2021-05-25T11:59:59.523 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.506+0000 7fa3aad9d700 1 -- 172.21.15.102:0/1600820923 learned_addr learned my addr 172.21.15.102:0/1600820923 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T11:59:59.523 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.506+0000 7fa3aa59c700 1 --1- 172.21.15.102:0/1600820923 >> v1:172.21.15.102:6789/0 conn(0x7fa3ac097fa0 0x7fa3ac063460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:59014/0 (socket says 172.21.15.102:59014) 2021-05-25T11:59:59.524 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.506+0000 7fa3a959a700 1 -- 172.21.15.102:0/1600820923 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3963783451 0 0) 0x7fa3ac0964c0 con 0x7fa3ac1325c0 2021-05-25T11:59:59.524 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.506+0000 7fa3a959a700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:59.524 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.506+0000 7fa3a959a700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:59.524 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.506+0000 7fa3a959a700 10 monclient(hunting): my global_id is 4349 2021-05-25T11:59:59.525 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.506+0000 7fa3a959a700 1 -- 172.21.15.102:0/1600820923 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa390003270 con 0x7fa3ac1325c0 2021-05-25T11:59:59.525 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.506+0000 7fa3a959a700 1 -- 172.21.15.102:0/1600820923 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 656795607 0 0) 0x7fa3ac096b20 con 0x7fa3ac097fa0 2021-05-25T11:59:59.525 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.506+0000 7fa3a959a700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:59.525 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.506+0000 7fa3a959a700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:59.525 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.506+0000 7fa3a959a700 10 monclient(hunting): my global_id is 4356 2021-05-25T11:59:59.526 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.506+0000 7fa3a959a700 1 -- 172.21.15.102:0/1600820923 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3ac0964c0 con 0x7fa3ac097fa0 2021-05-25T11:59:59.526 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.506+0000 7fa3a959a700 1 -- 172.21.15.102:0/1600820923 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1749469832 0 0) 0x7fa3ac0964c0 con 0x7fa3ac097fa0 2021-05-25T11:59:59.526 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.506+0000 7fa3a959a700 1 -- 172.21.15.102:0/1600820923 >> v1:172.21.15.102:6790/0 conn(0x7fa3ac1325c0 legacy=0x7fa3ac1349b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:59.526 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.506+0000 7fa3a959a700 1 -- 172.21.15.102:0/1600820923 >> v1:172.21.15.181:6789/0 conn(0x7fa3ac063b70 legacy=0x7fa3ac064020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:59.527 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.506+0000 7fa3a959a700 10 monclient: _finish_hunting 0 2021-05-25T11:59:59.527 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.506+0000 7fa3a959a700 1 monclient: found mon.noname-a 2021-05-25T11:59:59.527 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.506+0000 7fa3a959a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T11:59:59.527 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.506+0000 7fa3a959a700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T11:59:59.527 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.506+0000 7fa3a959a700 1 -- 172.21.15.102:0/1600820923 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3ac096760 con 0x7fa3ac097fa0 2021-05-25T11:59:59.528 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.506+0000 7fa3a959a700 10 monclient: _finish_auth 0 2021-05-25T11:59:59.528 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.506+0000 7fa3a959a700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T11:59:59.528 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.506+0000 7fa3b0b45700 5 monclient: authenticate success, global_id 4356 2021-05-25T11:59:59.528 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.506+0000 7fa3a959a700 1 -- 172.21.15.102:0/1600820923 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fa394001610 con 0x7fa3ac097fa0 2021-05-25T11:59:59.528 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.506+0000 7fa3b0b45700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T11:59:59.529 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.506+0000 7fa3a959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:59.529 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.506+0000 7fa3a959a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T11:59:59.529 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.506+0000 7fa3a959a700 10 monclient: dump: 2021-05-25T11:59:59.529 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:59.530 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:59.530 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:59.530 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:59.530 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:59.530 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:59.531 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:59.531 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:59.531 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:59.531 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:59.531 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.507+0000 7fa3b0b45700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T11:59:59.532 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.507+0000 7fa3a959a700 1 -- 172.21.15.102:0/1600820923 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa394001840 con 0x7fa3ac097fa0 2021-05-25T11:59:59.532 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.507+0000 7fa3a959a700 10 monclient: handle_config config(0 keys) v1 2021-05-25T11:59:59.532 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.507+0000 7fa3b0b45700 10 monclient: get_monmap_and_config success 2021-05-25T11:59:59.532 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.507+0000 7fa3a959a700 1 -- 172.21.15.102:0/1600820923 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fa394001bd0 con 0x7fa3ac097fa0 2021-05-25T11:59:59.533 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.507+0000 7fa3b0b45700 4 set_mon_vals no callback set 2021-05-25T11:59:59.533 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.507+0000 7fa3a959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:59.533 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.507+0000 7fa3a959a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T11:59:59.533 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.507+0000 7fa3a959a700 10 monclient: dump: 2021-05-25T11:59:59.533 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:59.534 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:59.534 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:59.534 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:59.534 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:59.534 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:59.535 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:59.535 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:59.535 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:59.535 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:59.536 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.507+0000 7fa3b0b45700 10 monclient: shutdown 2021-05-25T11:59:59.536 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.507+0000 7fa3b0b45700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T11:59:59.536 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.507+0000 7fa3b0b45700 1 -- 172.21.15.102:0/1600820923 >> v1:172.21.15.102:6789/0 conn(0x7fa3ac097fa0 legacy=0x7fa3ac063460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:59.536 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.507+0000 7fa3b0b45700 1 -- 172.21.15.102:0/1600820923 shutdown_connections 2021-05-25T11:59:59.536 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.507+0000 7fa3b0b45700 1 -- 172.21.15.102:0/1600820923 >> 172.21.15.102:0/1600820923 conn(0x7fa3ac09ef50 msgr2=0x7fa3ac09f360 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T11:59:59.537 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.507+0000 7fa3b0b45700 1 -- 172.21.15.102:0/1600820923 shutdown_connections 2021-05-25T11:59:59.537 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.507+0000 7fa3b0b45700 1 -- 172.21.15.102:0/1600820923 wait complete. 2021-05-25T11:59:59.537 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.508+0000 7fa3b0b45700 10 monclient: build_initial_monmap 2021-05-25T11:59:59.537 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.508+0000 7fa3b0b45700 10 monclient: monmap: 2021-05-25T11:59:59.537 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T11:59:59.538 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T11:59:59.538 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T11:59:59.538 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T11:59:59.538 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T11:59:59.539 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T11:59:59.539 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T11:59:59.539 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T11:59:59.539 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T11:59:59.539 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:59.540 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.508+0000 7fa3b0b45700 1 librados: starting msgr at 2021-05-25T11:59:59.540 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.508+0000 7fa3b0b45700 1 librados: starting objecter 2021-05-25T11:59:59.540 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.508+0000 7fa3b0b45700 1 Processor -- start 2021-05-25T11:59:59.540 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.508+0000 7fa3b0b45700 1 -- start start 2021-05-25T11:59:59.541 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.508+0000 7fa3b0b45700 1 librados: setting wanted keys 2021-05-25T11:59:59.541 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.508+0000 7fa3b0b45700 1 librados: calling monclient init 2021-05-25T11:59:59.541 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.508+0000 7fa3b0b45700 10 monclient: init 2021-05-25T11:59:59.541 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.508+0000 7fa3b0b45700 10 monclient: _reopen_session rank -1 2021-05-25T11:59:59.541 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.508+0000 7fa3b0b45700 10 monclient: _add_conns ranks=[0,2,1] 2021-05-25T11:59:59.542 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.508+0000 7fa3b0b45700 10 monclient(hunting): picked mon.noname-a con 0x7fa3ac063b70 addr v1:172.21.15.102:6789/0 2021-05-25T11:59:59.542 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.508+0000 7fa3b0b45700 10 monclient(hunting): picked mon.noname-c con 0x7fa3ac097fa0 addr v1:172.21.15.102:6790/0 2021-05-25T11:59:59.542 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.508+0000 7fa3b0b45700 10 monclient(hunting): picked mon.noname-b con 0x7fa3ac1325c0 addr v1:172.21.15.181:6789/0 2021-05-25T11:59:59.542 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.508+0000 7fa3b0b45700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3ac12c230 con 0x7fa3ac063b70 2021-05-25T11:59:59.543 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.508+0000 7fa3b0b45700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3ac116290 con 0x7fa3ac1325c0 2021-05-25T11:59:59.543 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.508+0000 7fa3b0b45700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3ac116530 con 0x7fa3ac097fa0 2021-05-25T11:59:59.543 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.508+0000 7fa3b0b45700 10 monclient(hunting): _renew_subs 2021-05-25T11:59:59.543 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.508+0000 7fa3b0b45700 10 monclient(hunting): authenticate will time out at 800.106955s 2021-05-25T11:59:59.543 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.509+0000 7fa3aa59c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fa3ac063b70 0x7fa3ac12bb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:59020/0 (socket says 172.21.15.102:59020) 2021-05-25T11:59:59.544 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.509+0000 7fa3aa59c700 1 -- 172.21.15.102:0/3696873702 learned_addr learned my addr 172.21.15.102:0/3696873702 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T11:59:59.544 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.509+0000 7fa39a7fc700 10 client.?.objecter ms_handle_connect 0x7fa3ac063b70 2021-05-25T11:59:59.544 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.509+0000 7fa39a7fc700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:59.544 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.509+0000 7fa39a7fc700 10 client.?.objecter ms_handle_connect 0x7fa3ac097fa0 2021-05-25T11:59:59.545 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.509+0000 7fa39a7fc700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:59.545 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.509+0000 7fa39a7fc700 1 -- 172.21.15.102:0/3696873702 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2824994351 0 0) 0x7fa3ac12c230 con 0x7fa3ac063b70 2021-05-25T11:59:59.545 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.509+0000 7fa39a7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:59.545 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.509+0000 7fa39a7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:59.545 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.509+0000 7fa39a7fc700 10 monclient(hunting): my global_id is 4359 2021-05-25T11:59:59.546 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.509+0000 7fa39a7fc700 1 -- 172.21.15.102:0/3696873702 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa388004c10 con 0x7fa3ac063b70 2021-05-25T11:59:59.546 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.509+0000 7fa39a7fc700 1 -- 172.21.15.102:0/3696873702 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4284948554 0 0) 0x7fa3ac116530 con 0x7fa3ac097fa0 2021-05-25T11:59:59.546 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.509+0000 7fa39a7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:59.546 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.509+0000 7fa39a7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:59.547 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.509+0000 7fa39a7fc700 10 monclient(hunting): my global_id is 4352 2021-05-25T11:59:59.547 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.509+0000 7fa39a7fc700 1 -- 172.21.15.102:0/3696873702 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3ac12c230 con 0x7fa3ac097fa0 2021-05-25T11:59:59.547 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.509+0000 7fa39a7fc700 10 client.?.objecter ms_handle_connect 0x7fa3ac1325c0 2021-05-25T11:59:59.547 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.509+0000 7fa39a7fc700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:59.548 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.509+0000 7fa39a7fc700 1 -- 172.21.15.102:0/3696873702 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1224424333 0 0) 0x7fa3ac116290 con 0x7fa3ac1325c0 2021-05-25T11:59:59.548 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.509+0000 7fa39a7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:59.548 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.509+0000 7fa39a7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:59.548 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.509+0000 7fa39a7fc700 10 monclient(hunting): my global_id is 4354 2021-05-25T11:59:59.548 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.509+0000 7fa39a7fc700 1 -- 172.21.15.102:0/3696873702 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3ac116530 con 0x7fa3ac1325c0 2021-05-25T11:59:59.549 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.509+0000 7fa39a7fc700 1 -- 172.21.15.102:0/3696873702 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3847532804 0 0) 0x7fa388004c10 con 0x7fa3ac063b70 2021-05-25T11:59:59.549 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.509+0000 7fa39a7fc700 1 -- 172.21.15.102:0/3696873702 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa3ac116290 con 0x7fa3ac063b70 2021-05-25T11:59:59.549 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.509+0000 7fa39a7fc700 1 -- 172.21.15.102:0/3696873702 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2616643947 0 0) 0x7fa3ac12c230 con 0x7fa3ac097fa0 2021-05-25T11:59:59.549 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.509+0000 7fa39a7fc700 1 -- 172.21.15.102:0/3696873702 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa388004c10 con 0x7fa3ac097fa0 2021-05-25T11:59:59.550 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.509+0000 7fa39a7fc700 1 -- 172.21.15.102:0/3696873702 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fa394001c10 con 0x7fa3ac063b70 2021-05-25T11:59:59.550 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.509+0000 7fa39a7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:59.550 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.509+0000 7fa39a7fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T11:59:59.550 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.509+0000 7fa39a7fc700 10 monclient(hunting): dump: 2021-05-25T11:59:59.550 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:59.551 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:59.551 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:59.551 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:59.551 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:59.552 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:59.552 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:59.552 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:59.552 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:59.552 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:59.553 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.509+0000 7fa39a7fc700 10 monclient(hunting): _finish_auth 0 2021-05-25T11:59:59.553 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.509+0000 7fa39a7fc700 1 -- 172.21.15.102:0/3696873702 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fa3a0001840 con 0x7fa3ac097fa0 2021-05-25T11:59:59.553 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.509+0000 7fa39a7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:59.553 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.509+0000 7fa39a7fc700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2021-05-25T11:59:59.553 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.509+0000 7fa39a7fc700 10 monclient(hunting): dump: 2021-05-25T11:59:59.554 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:59.554 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:59.554 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:59.554 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:59.555 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:59.555 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:59.555 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:59.555 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:59.555 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:59.556 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:59.556 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.509+0000 7fa39a7fc700 1 -- 172.21.15.102:0/3696873702 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1392544757 0 0) 0x7fa3ac116290 con 0x7fa3ac063b70 2021-05-25T11:59:59.556 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.509+0000 7fa39a7fc700 1 -- 172.21.15.102:0/3696873702 >> v1:172.21.15.102:6790/0 conn(0x7fa3ac097fa0 legacy=0x7fa3ac0925d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:59.556 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.509+0000 7fa39a7fc700 1 -- 172.21.15.102:0/3696873702 >> v1:172.21.15.181:6789/0 conn(0x7fa3ac1325c0 legacy=0x7fa3ac1432c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:59.556 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.509+0000 7fa39a7fc700 10 monclient: _finish_hunting 0 2021-05-25T11:59:59.557 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.509+0000 7fa39a7fc700 1 monclient: found mon.a 2021-05-25T11:59:59.557 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.509+0000 7fa39a7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T11:59:59.557 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.509+0000 7fa39a7fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:59.557 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.509+0000 7fa39a7fc700 1 -- 172.21.15.102:0/3696873702 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3ac092ce0 con 0x7fa3ac063b70 2021-05-25T11:59:59.558 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.509+0000 7fa39a7fc700 10 monclient: _finish_auth 0 2021-05-25T11:59:59.558 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.509+0000 7fa39a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T11:59:59.558 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.509+0000 7fa3b0b45700 5 monclient: authenticate success, global_id 4359 2021-05-25T11:59:59.558 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.509+0000 7fa3b0b45700 10 monclient: _renew_subs 2021-05-25T11:59:59.558 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.509+0000 7fa3b0b45700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:59.559 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.509+0000 7fa3b0b45700 1 -- 172.21.15.102:0/3696873702 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3ac0949a0 con 0x7fa3ac063b70 2021-05-25T11:59:59.559 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.509+0000 7fa3b0b45700 10 client.4359.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T11:59:59.559 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.509+0000 7fa3b0b45700 10 monclient: _renew_subs 2021-05-25T11:59:59.559 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.509+0000 7fa3b0b45700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:59.559 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.509+0000 7fa3b0b45700 1 -- 172.21.15.102:0/3696873702 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3ac093360 con 0x7fa3ac063b70 2021-05-25T11:59:59.560 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.509+0000 7fa3b0b45700 1 librados: init done 2021-05-25T11:59:59.560 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.509+0000 7fa39a7fc700 1 -- 172.21.15.102:0/3696873702 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa3940048a0 con 0x7fa3ac063b70 2021-05-25T11:59:59.560 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.509+0000 7fa39a7fc700 10 monclient: handle_config config(0 keys) v1 2021-05-25T11:59:59.560 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.509+0000 7fa39a7fc700 1 -- 172.21.15.102:0/3696873702 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fa394004bb0 con 0x7fa3ac063b70 2021-05-25T11:59:59.560 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.509+0000 7fa39a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:59.561 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.509+0000 7fa3a8d99700 4 set_mon_vals no callback set 2021-05-25T11:59:59.561 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.509+0000 7fa39a7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T11:59:59.561 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.509+0000 7fa39a7fc700 10 monclient: dump: 2021-05-25T11:59:59.561 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:59.562 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:59.562 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:59.562 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:59.562 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:59.562 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:59.563 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:59.563 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:59.563 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:59.563 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:59.563 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.510+0000 7fa39a7fc700 1 -- 172.21.15.102:0/3696873702 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7fa394004d10 con 0x7fa3ac063b70 2021-05-25T11:59:59.564 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.510+0000 7fa3b0b45700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:59.564 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.510+0000 7fa3b0b45700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:59.564 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.510+0000 7fa3b0b45700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:59.564 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.510+0000 7fa3b0b45700 1 -- 172.21.15.102:0/3696873702 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa38c004db0 con 0x7fa3ac063b70 2021-05-25T11:59:59.565 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.510+0000 7fa39a7fc700 1 -- 172.21.15.102:0/3696873702 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(15..15 src has 1..15) v4 ==== 3549+0+0 (unknown 3948819498 0 0) 0x7fa39403cff0 con 0x7fa3ac063b70 2021-05-25T11:59:59.565 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.510+0000 7fa39a7fc700 10 client.4359.objecter ms_dispatch 0x7fa3ac001550 osd_map(15..15 src has 1..15) v4 2021-05-25T11:59:59.565 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.510+0000 7fa39a7fc700 3 client.4359.objecter handle_osd_map got epochs [15,15] > 0 2021-05-25T11:59:59.565 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.510+0000 7fa39a7fc700 3 client.4359.objecter handle_osd_map decoding full epoch 15 2021-05-25T11:59:59.565 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.510+0000 7fa39a7fc700 20 client.4359.objecter dump_active .. 0 homeless 2021-05-25T11:59:59.566 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.510+0000 7fa3a9d9b700 10 monclient: get_auth_request con 0x7fa3880334c0 auth_method 0 2021-05-25T11:59:59.566 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.511+0000 7fa39a7fc700 10 client.4359.objecter ms_handle_connect 0x7fa3880334c0 2021-05-25T11:59:59.566 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.514+0000 7fa39a7fc700 1 -- 172.21.15.102:0/3696873702 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7fa394012f40 con 0x7fa3ac063b70 2021-05-25T11:59:59.566 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.514+0000 7fa39a7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:59.566 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.514+0000 7fa39a7fc700 10 monclient: _finish_command 1 = system:0 2021-05-25T11:59:59.666 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.664+0000 7fa3b0b45700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2021-05-25T11:59:59.667 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.664+0000 7fa3b0b45700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2021-05-25T11:59:59.667 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.664+0000 7fa3b0b45700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T11:59:59.667 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.664+0000 7fa3b0b45700 1 -- 172.21.15.102:0/3696873702 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa38c005060 con 0x7fa3ac063b70 2021-05-25T11:59:59.668 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.664+0000 7fa39a7fc700 1 -- 172.21.15.102:0/3696873702 <== mon.0 v1:172.21.15.102:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) v1 ==== 74+0+10024 (unknown 4221056075 0 1374345216) 0x7fa394012b30 con 0x7fa3ac063b70 2021-05-25T11:59:59.668 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.664+0000 7fa39a7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2021-05-25T11:59:59.668 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.664+0000 7fa39a7fc700 10 monclient: _finish_command 2 = system:0 2021-05-25T11:59:59.669 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T11:59:59.669 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":15,"fsid":"20ee91b6-8935-4b98-b868-85c03789ccc4","created":"2021-05-25T11:59:41.238908+0000","modified":"2021-05-25T11:59:53.591171+0000","last_up_change":"2021-05-25T11:59:49.572449+0000","last_in_change":"2021-05-25T11:59:42.972556+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-05-25T11:59:49.609415+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-05-25T11:59:50.997242+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"8f1f285f-7041-4c53-ab09-f638ab670036","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6807","nonce":29973}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6809","nonce":29973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6811","nonce":29973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6810","nonce":29973}]},"public_addr":"172.21.15.102:6807/29973","cluster_addr":"172.21.15.102:6809/29973","heartbeat_back_addr":"172.21.15.102:6811/29973","heartbeat_front_addr":"172.21.15.102:6810/29973","state":["exists","up"]},{"osd":1,"uuid":"d0d92491-3d97-4e4e-acf4-9dc7c0e4701f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6802","nonce":29974}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6804","nonce":29974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6808","nonce":29974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6806","nonce":29974}]},"public_addr":"172.21.15.102:6802/29974","cluster_addr":"172.21.15.102:6804/29974","heartbeat_back_addr":"172.21.15.102:6808/29974","heartbeat_front_addr":"172.21.15.102:6806/29974","state":["exists","up"]},{"osd":2,"uuid":"80a3eb90-b36e-4035-af91-3917fbf8dc2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6800","nonce":29975}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6801","nonce":29975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6805","nonce":29975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6803","nonce":29975}]},"public_addr":"172.21.15.102:6800/29975","cluster_addr":"172.21.15.102:6801/29975","heartbeat_back_addr":"172.21.15.102:6805/29975","heartbeat_front_addr":"172.21.15.102:6803/29975","state":["exists","up"]},{"osd":3,"uuid":"45889d6a-dcfb-4670-afef-cee9bbb0476a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6804","nonce":30005}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6805","nonce":30005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6807","nonce":30005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6806","nonce":30005}]},"public_addr":"172.21.15.181:6804/30005","cluster_addr":"172.21.15.181:6805/30005","heartbeat_back_addr":"172.21.15.181:6807/30005","heartbeat_front_addr":"172.21.15.181:6806/30005","state":["exists","up"]},{"osd":4,"uuid":"69e097b7-1472-442f-af42-225c0e40ad5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6800","nonce":30006}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6801","nonce":30006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6803","nonce":30006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6802","nonce":30006}]},"public_addr":"172.21.15.181:6800/30006","cluster_addr":"172.21.15.181:6801/30006","heartbeat_back_addr":"172.21.15.181:6803/30006","heartbeat_front_addr":"172.21.15.181:6802/30006","state":["exists","up"]},{"osd":5,"uuid":"f1045691-db58-471e-943a-e4d99129a917","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6808","nonce":30008}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6809","nonce":30008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6811","nonce":30008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6810","nonce":30008}]},"public_addr":"172.21.15.181:6808/30008","cluster_addr":"172.21.15.181:6809/30008","heartbeat_back_addr":"172.21.15.181:6811/30008","heartbeat_front_addr":"172.21.15.181:6810/30008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-05-25T11:59:48.129055+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-05-25T11:59:48.090298+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-05-25T11:59:48.049016+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-05-25T11:59:48.069527+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-05-25T11:59:48.002930+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-05-25T11:59:48.052367+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-05-25T11:59:59.671 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.666+0000 7fa3b0b45700 10 librados: watch_flush enter 2021-05-25T11:59:59.671 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.666+0000 7fa3b0b45700 10 librados: watch_flush exit 2021-05-25T11:59:59.671 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.666+0000 7fa3b0b45700 20 client.4359.objecter shutdown clearing up homeless session... 2021-05-25T11:59:59.671 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.666+0000 7fa3b0b45700 10 client.4359.objecter successfully canceled tick 2021-05-25T11:59:59.672 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.666+0000 7fa3b0b45700 1 -- 172.21.15.102:0/3696873702 >> v1:172.21.15.181:6812/29582 conn(0x7fa3880334c0 legacy=0x7fa388035980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:59.672 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.666+0000 7fa3b0b45700 10 monclient: shutdown 2021-05-25T11:59:59.672 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.666+0000 7fa3b0b45700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T11:59:59.672 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.666+0000 7fa3b0b45700 1 -- 172.21.15.102:0/3696873702 >> v1:172.21.15.102:6789/0 conn(0x7fa3ac063b70 legacy=0x7fa3ac12bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:59.672 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.666+0000 7fa3b0b45700 1 -- 172.21.15.102:0/3696873702 shutdown_connections 2021-05-25T11:59:59.673 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.666+0000 7fa3b0b45700 1 -- 172.21.15.102:0/3696873702 >> 172.21.15.102:0/3696873702 conn(0x7fa3ac09ef50 msgr2=0x7fa3ac096a00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T11:59:59.673 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.666+0000 7fa3b0b45700 1 -- 172.21.15.102:0/3696873702 shutdown_connections 2021-05-25T11:59:59.673 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.666+0000 7fa3b0b45700 1 -- 172.21.15.102:0/3696873702 wait complete. 2021-05-25T11:59:59.673 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.666+0000 7fa3b0b45700 1 librados: shutdown 2021-05-25T11:59:59.679 INFO:tasks.mon_thrash.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2021-05-25T11:59:49.609415+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '14', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2021-05-25T11:59:50.997242+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-05-25T11:59:59.680 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2021-05-25T11:59:59.792 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.789+0000 7f498aa41700 10 monclient: get_monmap_and_config 2021-05-25T11:59:59.792 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.789+0000 7f498aa41700 10 monclient: build_initial_monmap 2021-05-25T11:59:59.792 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.789+0000 7f498aa41700 10 monclient: monmap: 2021-05-25T11:59:59.793 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T11:59:59.793 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:59.793 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:59.791775+0000 2021-05-25T11:59:59.793 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:59.791775+0000 2021-05-25T11:59:59.794 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T11:59:59.794 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:59.794 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T11:59:59.794 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T11:59:59.794 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T11:59:59.795 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:59.805 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.790+0000 7f498aa41700 1 Processor -- start 2021-05-25T11:59:59.805 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.790+0000 7f498aa41700 1 -- start start 2021-05-25T11:59:59.805 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.790+0000 7f498aa41700 10 monclient: init 2021-05-25T11:59:59.806 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.790+0000 7f498aa41700 10 monclient: _reopen_session rank -1 2021-05-25T11:59:59.806 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.791+0000 7f498aa41700 10 monclient: _add_conns ranks=[2,0,1] 2021-05-25T11:59:59.806 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.791+0000 7f498aa41700 10 monclient(hunting): picked mon.noname-b con 0x7f498412b8e0 addr v1:172.21.15.102:6790/0 2021-05-25T11:59:59.806 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.791+0000 7f498aa41700 10 monclient(hunting): picked mon.noname-a con 0x7f498412bcc0 addr v1:172.21.15.102:6789/0 2021-05-25T11:59:59.806 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.791+0000 7f498aa41700 10 monclient(hunting): picked mon.noname-c con 0x7f4984131e30 addr v1:172.21.15.181:6789/0 2021-05-25T11:59:59.807 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.791+0000 7f498aa41700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4984091480 con 0x7f498412bcc0 2021-05-25T11:59:59.807 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.791+0000 7f498aa41700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4984091e60 con 0x7f4984131e30 2021-05-25T11:59:59.807 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.791+0000 7f498aa41700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49840960f0 con 0x7f498412b8e0 2021-05-25T11:59:59.807 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.791+0000 7f498aa41700 10 monclient(hunting): _renew_subs 2021-05-25T11:59:59.807 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.791+0000 7f498aa41700 10 monclient(hunting): authenticate will time out at 800.389092s 2021-05-25T11:59:59.808 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.791+0000 7f49837fe700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f498412bcc0 0x7f498412c170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:59030/0 (socket says 172.21.15.102:59030) 2021-05-25T11:59:59.808 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.791+0000 7f49837fe700 1 -- 172.21.15.102:0/3740989614 learned_addr learned my addr 172.21.15.102:0/3740989614 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T11:59:59.808 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.791+0000 7f4982ffd700 1 -- 172.21.15.102:0/3740989614 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1975221308 0 0) 0x7f4984091480 con 0x7f498412bcc0 2021-05-25T11:59:59.808 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.791+0000 7f4982ffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:59.808 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.791+0000 7f4982ffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:59.809 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.791+0000 7f4982ffd700 10 monclient(hunting): my global_id is 4362 2021-05-25T11:59:59.809 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.791+0000 7f4982ffd700 1 -- 172.21.15.102:0/3740989614 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f496c003270 con 0x7f498412bcc0 2021-05-25T11:59:59.809 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.791+0000 7f4982ffd700 1 -- 172.21.15.102:0/3740989614 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1714081241 0 0) 0x7f49840960f0 con 0x7f498412b8e0 2021-05-25T11:59:59.809 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.791+0000 7f4982ffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:59.809 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.791+0000 7f4982ffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:59.810 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.791+0000 7f4982ffd700 10 monclient(hunting): my global_id is 4355 2021-05-25T11:59:59.810 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.791+0000 7f4982ffd700 1 -- 172.21.15.102:0/3740989614 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4984091480 con 0x7f498412b8e0 2021-05-25T11:59:59.810 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.791+0000 7f4982ffd700 1 -- 172.21.15.102:0/3740989614 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 342869123 0 0) 0x7f4984091e60 con 0x7f4984131e30 2021-05-25T11:59:59.810 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.791+0000 7f4982ffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:59.810 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.791+0000 7f4982ffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:59.811 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.791+0000 7f4982ffd700 10 monclient(hunting): my global_id is 4357 2021-05-25T11:59:59.811 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.791+0000 7f4982ffd700 1 -- 172.21.15.102:0/3740989614 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49840960f0 con 0x7f4984131e30 2021-05-25T11:59:59.811 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.791+0000 7f4982ffd700 1 -- 172.21.15.102:0/3740989614 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2209123649 0 0) 0x7f4984091480 con 0x7f498412b8e0 2021-05-25T11:59:59.811 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.791+0000 7f4982ffd700 1 -- 172.21.15.102:0/3740989614 >> v1:172.21.15.181:6789/0 conn(0x7f4984131e30 legacy=0x7f4984134220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:59.811 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.791+0000 7f4982ffd700 1 -- 172.21.15.102:0/3740989614 >> v1:172.21.15.102:6789/0 conn(0x7f498412bcc0 legacy=0x7f498412c170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:59.812 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.791+0000 7f4982ffd700 10 monclient: _finish_hunting 0 2021-05-25T11:59:59.812 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.791+0000 7f4982ffd700 1 monclient: found mon.noname-b 2021-05-25T11:59:59.812 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.791+0000 7f4982ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T11:59:59.812 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.791+0000 7f4982ffd700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T11:59:59.813 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.791+0000 7f4982ffd700 1 -- 172.21.15.102:0/3740989614 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4984095e10 con 0x7f498412b8e0 2021-05-25T11:59:59.813 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.791+0000 7f4982ffd700 10 monclient: _finish_auth 0 2021-05-25T11:59:59.813 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.791+0000 7f4982ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T11:59:59.813 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.791+0000 7f4982ffd700 1 -- 172.21.15.102:0/3740989614 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f49680015f0 con 0x7f498412b8e0 2021-05-25T11:59:59.813 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.791+0000 7f4982ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:59.814 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.791+0000 7f4982ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T11:59:59.814 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.791+0000 7f4982ffd700 10 monclient: dump: 2021-05-25T11:59:59.814 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:59.814 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:59.814 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:59.815 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:59.815 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:59.815 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:59.815 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:59.816 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:59.816 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:59.816 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:59.816 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.791+0000 7f498aa41700 5 monclient: authenticate success, global_id 4355 2021-05-25T11:59:59.816 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.791+0000 7f498aa41700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T11:59:59.817 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.791+0000 7f4982ffd700 1 -- 172.21.15.102:0/3740989614 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4968001910 con 0x7f498412b8e0 2021-05-25T11:59:59.817 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.791+0000 7f4982ffd700 10 monclient: handle_config config(0 keys) v1 2021-05-25T11:59:59.817 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.791+0000 7f498aa41700 10 monclient: get_monmap_and_config success 2021-05-25T11:59:59.817 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.791+0000 7f498aa41700 4 set_mon_vals no callback set 2021-05-25T11:59:59.817 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.791+0000 7f4982ffd700 1 -- 172.21.15.102:0/3740989614 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f4968001ca0 con 0x7f498412b8e0 2021-05-25T11:59:59.818 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.791+0000 7f4982ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:59.818 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.791+0000 7f4982ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T11:59:59.818 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.791+0000 7f4982ffd700 10 monclient: dump: 2021-05-25T11:59:59.818 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:59.819 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:59.819 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:59.819 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:59.819 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:59.819 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:59.820 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:59.820 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:59.820 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:59.820 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:59.820 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.792+0000 7f498aa41700 10 monclient: shutdown 2021-05-25T11:59:59.821 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.792+0000 7f498aa41700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T11:59:59.821 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.792+0000 7f498aa41700 1 -- 172.21.15.102:0/3740989614 >> v1:172.21.15.102:6790/0 conn(0x7f498412b8e0 legacy=0x7f4984096870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:59.821 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.792+0000 7f498aa41700 1 -- 172.21.15.102:0/3740989614 shutdown_connections 2021-05-25T11:59:59.821 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.792+0000 7f498aa41700 1 -- 172.21.15.102:0/3740989614 >> 172.21.15.102:0/3740989614 conn(0x7f4984097c00 msgr2=0x7f4984098010 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T11:59:59.822 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.792+0000 7f498aa41700 1 -- 172.21.15.102:0/3740989614 shutdown_connections 2021-05-25T11:59:59.822 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.792+0000 7f498aa41700 1 -- 172.21.15.102:0/3740989614 wait complete. 2021-05-25T11:59:59.822 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.792+0000 7f498aa41700 10 monclient: build_initial_monmap 2021-05-25T11:59:59.822 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.792+0000 7f498aa41700 10 monclient: monmap: 2021-05-25T11:59:59.823 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T11:59:59.823 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T11:59:59.823 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T11:59:59.823 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T11:59:59.823 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T11:59:59.824 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T11:59:59.824 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T11:59:59.824 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T11:59:59.824 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T11:59:59.824 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:59.825 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.793+0000 7f498aa41700 1 librados: starting msgr at 2021-05-25T11:59:59.825 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.793+0000 7f498aa41700 1 librados: starting objecter 2021-05-25T11:59:59.825 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.793+0000 7f498aa41700 1 Processor -- start 2021-05-25T11:59:59.826 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.793+0000 7f498aa41700 1 -- start start 2021-05-25T11:59:59.826 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.793+0000 7f498aa41700 1 librados: setting wanted keys 2021-05-25T11:59:59.826 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.793+0000 7f498aa41700 1 librados: calling monclient init 2021-05-25T11:59:59.826 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.793+0000 7f498aa41700 10 monclient: init 2021-05-25T11:59:59.826 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.793+0000 7f498aa41700 10 monclient: _reopen_session rank -1 2021-05-25T11:59:59.827 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.793+0000 7f498aa41700 10 monclient: _add_conns ranks=[1,2,0] 2021-05-25T11:59:59.827 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.793+0000 7f498aa41700 10 monclient(hunting): picked mon.noname-b con 0x7f498412b8e0 addr v1:172.21.15.181:6789/0 2021-05-25T11:59:59.827 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.793+0000 7f498aa41700 10 monclient(hunting): picked mon.noname-c con 0x7f498412bcc0 addr v1:172.21.15.102:6790/0 2021-05-25T11:59:59.827 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.793+0000 7f498aa41700 10 monclient(hunting): picked mon.noname-a con 0x7f4984131e30 addr v1:172.21.15.102:6789/0 2021-05-25T11:59:59.828 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.793+0000 7f498aa41700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4984097060 con 0x7f4984131e30 2021-05-25T11:59:59.828 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.793+0000 7f498aa41700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4984134000 con 0x7f498412b8e0 2021-05-25T11:59:59.828 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.793+0000 7f498aa41700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f498412e4f0 con 0x7f498412bcc0 2021-05-25T11:59:59.828 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.793+0000 7f498aa41700 10 monclient(hunting): _renew_subs 2021-05-25T11:59:59.828 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.793+0000 7f498aa41700 10 monclient(hunting): authenticate will time out at 800.391836s 2021-05-25T11:59:59.829 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.793+0000 7f49837fe700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f498412bcc0 0x7f49841338f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:53916/0 (socket says 172.21.15.102:53916) 2021-05-25T11:59:59.829 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.793+0000 7f49837fe700 1 -- 172.21.15.102:0/1223851815 learned_addr learned my addr 172.21.15.102:0/1223851815 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T11:59:59.829 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.793+0000 7f4988fde700 1 --1- 172.21.15.102:0/1223851815 >> v1:172.21.15.102:6789/0 conn(0x7f4984131e30 0x7f4984143a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:59040/0 (socket says 172.21.15.102:59040) 2021-05-25T11:59:59.829 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.794+0000 7f4977fff700 10 client.?.objecter ms_handle_connect 0x7f498412bcc0 2021-05-25T11:59:59.829 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.794+0000 7f4977fff700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:59.830 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.794+0000 7f4977fff700 10 client.?.objecter ms_handle_connect 0x7f4984131e30 2021-05-25T11:59:59.830 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.794+0000 7f4977fff700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:59.830 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.794+0000 7f4977fff700 10 client.?.objecter ms_handle_connect 0x7f498412b8e0 2021-05-25T11:59:59.830 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.794+0000 7f4977fff700 10 client.?.objecter resend_mon_ops 2021-05-25T11:59:59.831 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.794+0000 7f4977fff700 1 -- 172.21.15.102:0/1223851815 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 967556008 0 0) 0x7f498412e4f0 con 0x7f498412bcc0 2021-05-25T11:59:59.831 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.794+0000 7f4977fff700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:59.831 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.794+0000 7f4977fff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:59.831 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.794+0000 7f4977fff700 10 monclient(hunting): my global_id is 4358 2021-05-25T11:59:59.831 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.794+0000 7f4977fff700 1 -- 172.21.15.102:0/1223851815 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4960004c10 con 0x7f498412bcc0 2021-05-25T11:59:59.832 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.794+0000 7f4977fff700 1 -- 172.21.15.102:0/1223851815 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3810567998 0 0) 0x7f4984097060 con 0x7f4984131e30 2021-05-25T11:59:59.832 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.794+0000 7f4977fff700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:59.832 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.794+0000 7f4977fff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:59.832 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.794+0000 7f4977fff700 10 monclient(hunting): my global_id is 4365 2021-05-25T11:59:59.833 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.794+0000 7f4977fff700 1 -- 172.21.15.102:0/1223851815 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f498412e4f0 con 0x7f4984131e30 2021-05-25T11:59:59.833 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.794+0000 7f4977fff700 1 -- 172.21.15.102:0/1223851815 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3212112742 0 0) 0x7f4984134000 con 0x7f498412b8e0 2021-05-25T11:59:59.833 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.794+0000 7f4977fff700 10 monclient(hunting): _init_auth method 2 2021-05-25T11:59:59.833 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.794+0000 7f4977fff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T11:59:59.833 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.794+0000 7f4977fff700 10 monclient(hunting): my global_id is 4360 2021-05-25T11:59:59.834 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.794+0000 7f4977fff700 1 -- 172.21.15.102:0/1223851815 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4984097060 con 0x7f498412b8e0 2021-05-25T11:59:59.834 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.794+0000 7f4977fff700 1 -- 172.21.15.102:0/1223851815 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 71301412 0 0) 0x7f4960004c10 con 0x7f498412bcc0 2021-05-25T11:59:59.834 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.794+0000 7f4977fff700 1 -- 172.21.15.102:0/1223851815 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4984134000 con 0x7f498412bcc0 2021-05-25T11:59:59.834 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.794+0000 7f4977fff700 1 -- 172.21.15.102:0/1223851815 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f49700019f0 con 0x7f498412bcc0 2021-05-25T11:59:59.834 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.794+0000 7f4977fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:59.835 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.794+0000 7f4977fff700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T11:59:59.835 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.794+0000 7f4977fff700 10 monclient(hunting): dump: 2021-05-25T11:59:59.835 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:59.835 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:59.836 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:59.836 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:59.836 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:59.836 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:59.836 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:59.837 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:59.837 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:59.837 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:59.837 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.794+0000 7f4977fff700 10 monclient(hunting): _finish_auth 0 2021-05-25T11:59:59.838 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.794+0000 7f4977fff700 1 -- 172.21.15.102:0/1223851815 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1782200636 0 0) 0x7f498412e4f0 con 0x7f4984131e30 2021-05-25T11:59:59.838 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.794+0000 7f4977fff700 1 -- 172.21.15.102:0/1223851815 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4960004c10 con 0x7f4984131e30 2021-05-25T11:59:59.838 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.794+0000 7f4977fff700 1 -- 172.21.15.102:0/1223851815 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f4978001bc0 con 0x7f4984131e30 2021-05-25T11:59:59.838 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.794+0000 7f4977fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:59.838 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.794+0000 7f4977fff700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2021-05-25T11:59:59.839 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.794+0000 7f4977fff700 10 monclient(hunting): dump: 2021-05-25T11:59:59.839 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:59.839 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:59.839 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:59.839 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:59.840 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:59.840 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:59.840 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:59.840 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:59.840 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:59.841 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:59.841 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.794+0000 7f4977fff700 1 -- 172.21.15.102:0/1223851815 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2176291349 0 0) 0x7f4984097060 con 0x7f498412b8e0 2021-05-25T11:59:59.841 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.794+0000 7f4977fff700 1 -- 172.21.15.102:0/1223851815 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f498412e4f0 con 0x7f498412b8e0 2021-05-25T11:59:59.841 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.794+0000 7f4977fff700 1 -- 172.21.15.102:0/1223851815 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1145724627 0 0) 0x7f4984134000 con 0x7f498412bcc0 2021-05-25T11:59:59.841 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.794+0000 7f4977fff700 1 -- 172.21.15.102:0/1223851815 >> v1:172.21.15.181:6789/0 conn(0x7f498412b8e0 legacy=0x7f49841331e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:59.842 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.794+0000 7f4977fff700 1 -- 172.21.15.102:0/1223851815 >> v1:172.21.15.102:6789/0 conn(0x7f4984131e30 legacy=0x7f4984143a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:59.842 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.794+0000 7f4977fff700 10 monclient: _finish_hunting 0 2021-05-25T11:59:59.842 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.794+0000 7f4977fff700 1 monclient: found mon.c 2021-05-25T11:59:59.842 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.794+0000 7f4977fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T11:59:59.842 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.794+0000 7f4977fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T11:59:59.843 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.794+0000 7f4977fff700 1 -- 172.21.15.102:0/1223851815 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4984134c30 con 0x7f498412bcc0 2021-05-25T11:59:59.843 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.794+0000 7f4977fff700 10 monclient: _finish_auth 0 2021-05-25T11:59:59.843 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.794+0000 7f4977fff700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T11:59:59.843 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.794+0000 7f498aa41700 5 monclient: authenticate success, global_id 4358 2021-05-25T11:59:59.843 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.794+0000 7f498aa41700 10 monclient: _renew_subs 2021-05-25T11:59:59.844 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.794+0000 7f498aa41700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T11:59:59.844 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.794+0000 7f498aa41700 1 -- 172.21.15.102:0/1223851815 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4984131350 con 0x7f498412bcc0 2021-05-25T11:59:59.844 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.794+0000 7f4977fff700 1 -- 172.21.15.102:0/1223851815 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4970001ca0 con 0x7f498412bcc0 2021-05-25T11:59:59.844 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.794+0000 7f4977fff700 10 monclient: handle_config config(0 keys) v1 2021-05-25T11:59:59.844 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.794+0000 7f4977fff700 1 -- 172.21.15.102:0/1223851815 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f4970004980 con 0x7f498412bcc0 2021-05-25T11:59:59.845 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.794+0000 7f4977fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T11:59:59.845 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.794+0000 7f4977fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T11:59:59.845 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.794+0000 7f4977fff700 10 monclient: dump: 2021-05-25T11:59:59.845 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T11:59:59.845 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T11:59:59.846 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:59.846 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T11:59:59.846 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T11:59:59.846 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T11:59:59.846 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T11:59:59.847 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T11:59:59.847 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T11:59:59.847 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T11:59:59.847 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.794+0000 7f49827fc700 4 set_mon_vals no callback set 2021-05-25T11:59:59.848 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.794+0000 7f498aa41700 10 client.4358.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T11:59:59.848 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.794+0000 7f498aa41700 10 monclient: _renew_subs 2021-05-25T11:59:59.848 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.794+0000 7f498aa41700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T11:59:59.848 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.794+0000 7f498aa41700 1 -- 172.21.15.102:0/1223851815 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f498412d430 con 0x7f498412bcc0 2021-05-25T11:59:59.848 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.794+0000 7f498aa41700 1 librados: init done 2021-05-25T11:59:59.849 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.795+0000 7f4977fff700 1 -- 172.21.15.102:0/1223851815 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f497000dd20 con 0x7f498412bcc0 2021-05-25T11:59:59.849 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.795+0000 7f4977fff700 1 -- 172.21.15.102:0/1223851815 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(15..15 src has 1..15) v4 ==== 3549+0+0 (unknown 3948819498 0 0) 0x7f497003ced0 con 0x7f498412bcc0 2021-05-25T11:59:59.849 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.795+0000 7f4977fff700 10 client.4358.objecter ms_dispatch 0x7f4984001550 osd_map(15..15 src has 1..15) v4 2021-05-25T11:59:59.849 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.795+0000 7f4977fff700 3 client.4358.objecter handle_osd_map got epochs [15,15] > 0 2021-05-25T11:59:59.849 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.795+0000 7f4977fff700 3 client.4358.objecter handle_osd_map decoding full epoch 15 2021-05-25T11:59:59.850 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.795+0000 7f4977fff700 20 client.4358.objecter dump_active .. 0 homeless 2021-05-25T11:59:59.850 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.795+0000 7f4983fff700 10 monclient: get_auth_request con 0x7f4960033540 auth_method 0 2021-05-25T11:59:59.850 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.795+0000 7f498aa41700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:59.850 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.795+0000 7f498aa41700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:59.850 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.795+0000 7f498aa41700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T11:59:59.851 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.795+0000 7f498aa41700 1 -- 172.21.15.102:0/1223851815 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f498412e350 con 0x7f498412bcc0 2021-05-25T11:59:59.851 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.795+0000 7f4977fff700 10 client.4358.objecter ms_handle_connect 0x7f4960033540 2021-05-25T11:59:59.851 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.799+0000 7f4977fff700 1 -- 172.21.15.102:0/1223851815 <== mon.2 v1:172.21.15.102:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7f498412e350 con 0x7f498412bcc0 2021-05-25T11:59:59.851 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.799+0000 7f4977fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T11:59:59.851 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.799+0000 7f4977fff700 10 monclient: _finish_command 1 = system:0 2021-05-25T11:59:59.952 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.949+0000 7f498aa41700 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}] 2021-05-25T11:59:59.952 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.949+0000 7f498aa41700 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}] 2021-05-25T11:59:59.953 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.949+0000 7f498aa41700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T11:59:59.953 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.949+0000 7f498aa41700 1 -- 172.21.15.102:0/1223851815 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f498412e350 con 0x7f498412bcc0 2021-05-25T11:59:59.953 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.950+0000 7f4977fff700 1 -- 172.21.15.102:0/1223851815 <== mon.2 v1:172.21.15.102:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v15) v1 ==== 110+0+10 (unknown 1134451309 0 2170607528) 0x7f498412e350 con 0x7f498412bcc0 2021-05-25T11:59:59.953 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.950+0000 7f4977fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}] 2021-05-25T11:59:59.954 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.950+0000 7f4977fff700 10 monclient: _finish_command 2 = system:0 2021-05-25T11:59:59.954 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2021-05-25T11:59:59.956 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.951+0000 7f498aa41700 10 librados: watch_flush enter 2021-05-25T11:59:59.956 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.951+0000 7f498aa41700 10 librados: watch_flush exit 2021-05-25T11:59:59.956 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.951+0000 7f498aa41700 20 client.4358.objecter shutdown clearing up homeless session... 2021-05-25T11:59:59.956 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.951+0000 7f498aa41700 10 client.4358.objecter successfully canceled tick 2021-05-25T11:59:59.957 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.951+0000 7f498aa41700 1 -- 172.21.15.102:0/1223851815 >> v1:172.21.15.181:6812/29582 conn(0x7f4960033540 legacy=0x7f4960035a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:59.957 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.951+0000 7f498aa41700 10 monclient: shutdown 2021-05-25T11:59:59.957 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.951+0000 7f498aa41700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T11:59:59.957 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.951+0000 7f498aa41700 1 -- 172.21.15.102:0/1223851815 >> v1:172.21.15.102:6790/0 conn(0x7f498412bcc0 legacy=0x7f49841338f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T11:59:59.957 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.951+0000 7f498aa41700 1 -- 172.21.15.102:0/1223851815 shutdown_connections 2021-05-25T11:59:59.958 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.951+0000 7f498aa41700 1 -- 172.21.15.102:0/1223851815 >> 172.21.15.102:0/1223851815 conn(0x7f4984097c00 msgr2=0x7f4984123a90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T11:59:59.958 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.951+0000 7f498aa41700 1 -- 172.21.15.102:0/1223851815 shutdown_connections 2021-05-25T11:59:59.958 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.951+0000 7f498aa41700 1 -- 172.21.15.102:0/1223851815 wait complete. 2021-05-25T11:59:59.958 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T11:59:59.951+0000 7f498aa41700 1 librados: shutdown 2021-05-25T11:59:59.965 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2021-05-25T12:00:00.082 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.079+0000 7fb871e46700 10 monclient: get_monmap_and_config 2021-05-25T12:00:00.082 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.079+0000 7fb871e46700 10 monclient: build_initial_monmap 2021-05-25T12:00:00.082 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.079+0000 7fb871e46700 10 monclient: monmap: 2021-05-25T12:00:00.083 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:00:00.083 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:00.083 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:00:00.081507+0000 2021-05-25T12:00:00.083 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:00:00.081507+0000 2021-05-25T12:00:00.084 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:00:00.084 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:00.084 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:00:00.084 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:00:00.084 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:00:00.085 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:00.094 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.080+0000 7fb871e46700 1 Processor -- start 2021-05-25T12:00:00.095 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.080+0000 7fb871e46700 1 -- start start 2021-05-25T12:00:00.095 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.080+0000 7fb871e46700 10 monclient: init 2021-05-25T12:00:00.095 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.080+0000 7fb871e46700 10 monclient: _reopen_session rank -1 2021-05-25T12:00:00.095 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.080+0000 7fb871e46700 10 monclient: _add_conns ranks=[0,1,2] 2021-05-25T12:00:00.096 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.080+0000 7fb871e46700 10 monclient(hunting): picked mon.noname-a con 0x7fb86c128c90 addr v1:172.21.15.102:6789/0 2021-05-25T12:00:00.096 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.080+0000 7fb871e46700 10 monclient(hunting): picked mon.noname-c con 0x7fb86c068da0 addr v1:172.21.15.181:6789/0 2021-05-25T12:00:00.096 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.080+0000 7fb871e46700 10 monclient(hunting): picked mon.noname-b con 0x7fb86c069250 addr v1:172.21.15.102:6790/0 2021-05-25T12:00:00.096 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.080+0000 7fb871e46700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb86c0976b0 con 0x7fb86c128c90 2021-05-25T12:00:00.096 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.080+0000 7fb871e46700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb86c097950 con 0x7fb86c068da0 2021-05-25T12:00:00.097 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.080+0000 7fb871e46700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb86c097bf0 con 0x7fb86c069250 2021-05-25T12:00:00.097 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.080+0000 7fb871e46700 10 monclient(hunting): _renew_subs 2021-05-25T12:00:00.097 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.080+0000 7fb871e46700 10 monclient(hunting): authenticate will time out at 800.678732s 2021-05-25T12:00:00.097 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.080+0000 7fb86b7fe700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fb86c128c90 0x7fb86c128240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:59044/0 (socket says 172.21.15.102:59044) 2021-05-25T12:00:00.097 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.080+0000 7fb86b7fe700 1 -- 172.21.15.102:0/407111478 learned_addr learned my addr 172.21.15.102:0/407111478 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:00:00.098 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.080+0000 7fb86bfff700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fb86c069250 0x7fb86c134200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:53926/0 (socket says 172.21.15.102:53926) 2021-05-25T12:00:00.098 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.081+0000 7fb86a7fc700 1 -- 172.21.15.102:0/407111478 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3027648122 0 0) 0x7fb86c097bf0 con 0x7fb86c069250 2021-05-25T12:00:00.098 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.081+0000 7fb86a7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:00.098 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.081+0000 7fb86a7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:00.099 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.081+0000 7fb86a7fc700 10 monclient(hunting): my global_id is 4361 2021-05-25T12:00:00.099 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.081+0000 7fb86a7fc700 1 -- 172.21.15.102:0/407111478 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb850003270 con 0x7fb86c069250 2021-05-25T12:00:00.099 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.081+0000 7fb86a7fc700 1 -- 172.21.15.102:0/407111478 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2429944440 0 0) 0x7fb86c0976b0 con 0x7fb86c128c90 2021-05-25T12:00:00.099 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.081+0000 7fb86a7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:00.099 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.081+0000 7fb86a7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:00.100 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.081+0000 7fb86a7fc700 10 monclient(hunting): my global_id is 4368 2021-05-25T12:00:00.100 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.081+0000 7fb86a7fc700 1 -- 172.21.15.102:0/407111478 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb86c097bf0 con 0x7fb86c128c90 2021-05-25T12:00:00.100 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.081+0000 7fb86a7fc700 1 -- 172.21.15.102:0/407111478 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1208426297 0 0) 0x7fb86c097950 con 0x7fb86c068da0 2021-05-25T12:00:00.100 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.081+0000 7fb86a7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:00.100 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.081+0000 7fb86a7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:00.101 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.081+0000 7fb86a7fc700 10 monclient(hunting): my global_id is 4363 2021-05-25T12:00:00.101 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.081+0000 7fb86a7fc700 1 -- 172.21.15.102:0/407111478 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb86c0976b0 con 0x7fb86c068da0 2021-05-25T12:00:00.101 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.081+0000 7fb86a7fc700 1 -- 172.21.15.102:0/407111478 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3884676763 0 0) 0x7fb86c097bf0 con 0x7fb86c128c90 2021-05-25T12:00:00.101 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.081+0000 7fb86a7fc700 1 -- 172.21.15.102:0/407111478 >> v1:172.21.15.102:6790/0 conn(0x7fb86c069250 legacy=0x7fb86c134200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:00.101 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.081+0000 7fb86a7fc700 1 -- 172.21.15.102:0/407111478 >> v1:172.21.15.181:6789/0 conn(0x7fb86c068da0 legacy=0x7fb86c130ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:00.102 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.081+0000 7fb86a7fc700 10 monclient: _finish_hunting 0 2021-05-25T12:00:00.102 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.081+0000 7fb86a7fc700 1 monclient: found mon.noname-a 2021-05-25T12:00:00.102 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.081+0000 7fb86a7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:00:00.102 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.081+0000 7fb86a7fc700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T12:00:00.103 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.081+0000 7fb86a7fc700 1 -- 172.21.15.102:0/407111478 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb86c097e90 con 0x7fb86c128c90 2021-05-25T12:00:00.103 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.081+0000 7fb86a7fc700 10 monclient: _finish_auth 0 2021-05-25T12:00:00.103 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.081+0000 7fb86a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:00:00.103 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.081+0000 7fb86a7fc700 1 -- 172.21.15.102:0/407111478 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fb858001790 con 0x7fb86c128c90 2021-05-25T12:00:00.103 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.081+0000 7fb871e46700 5 monclient: authenticate success, global_id 4368 2021-05-25T12:00:00.104 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.081+0000 7fb871e46700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:00:00.104 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.081+0000 7fb86a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:00.104 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.081+0000 7fb86a7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:00:00.104 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.081+0000 7fb86a7fc700 10 monclient: dump: 2021-05-25T12:00:00.105 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:00.105 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:00.105 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:00.105 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:00.105 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:00.106 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:00.106 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:00.106 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:00.106 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:00.106 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:00.107 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.081+0000 7fb871e46700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:00:00.107 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.081+0000 7fb86a7fc700 1 -- 172.21.15.102:0/407111478 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb8580019b0 con 0x7fb86c128c90 2021-05-25T12:00:00.107 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.081+0000 7fb86a7fc700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:00:00.107 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.081+0000 7fb86a7fc700 1 -- 172.21.15.102:0/407111478 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fb858001cc0 con 0x7fb86c128c90 2021-05-25T12:00:00.108 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.081+0000 7fb871e46700 10 monclient: get_monmap_and_config success 2021-05-25T12:00:00.108 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.081+0000 7fb871e46700 4 set_mon_vals no callback set 2021-05-25T12:00:00.108 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.081+0000 7fb86a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:00.108 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.081+0000 7fb86a7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:00:00.108 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.081+0000 7fb86a7fc700 10 monclient: dump: 2021-05-25T12:00:00.109 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:00.109 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:00.109 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:00.109 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:00.109 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:00.110 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:00.110 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:00.110 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:00.110 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:00.111 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:00.111 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.081+0000 7fb871e46700 10 monclient: shutdown 2021-05-25T12:00:00.111 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.081+0000 7fb871e46700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:00:00.111 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.081+0000 7fb871e46700 1 -- 172.21.15.102:0/407111478 >> v1:172.21.15.102:6789/0 conn(0x7fb86c128c90 legacy=0x7fb86c128240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:00.111 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.082+0000 7fb871e46700 1 -- 172.21.15.102:0/407111478 shutdown_connections 2021-05-25T12:00:00.112 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.082+0000 7fb871e46700 1 -- 172.21.15.102:0/407111478 >> 172.21.15.102:0/407111478 conn(0x7fb86c0984c0 msgr2=0x7fb86c0988d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:00:00.112 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.082+0000 7fb871e46700 1 -- 172.21.15.102:0/407111478 shutdown_connections 2021-05-25T12:00:00.112 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.082+0000 7fb871e46700 1 -- 172.21.15.102:0/407111478 wait complete. 2021-05-25T12:00:00.112 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.082+0000 7fb871e46700 10 monclient: build_initial_monmap 2021-05-25T12:00:00.113 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.082+0000 7fb871e46700 10 monclient: monmap: 2021-05-25T12:00:00.113 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:00:00.113 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:00:00.113 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:00:00.114 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:00:00.114 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:00:00.114 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:00:00.114 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:00:00.115 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:00:00.115 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:00:00.115 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:00.115 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.082+0000 7fb871e46700 1 librados: starting msgr at 2021-05-25T12:00:00.115 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.082+0000 7fb871e46700 1 librados: starting objecter 2021-05-25T12:00:00.116 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.083+0000 7fb871e46700 1 Processor -- start 2021-05-25T12:00:00.116 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.083+0000 7fb871e46700 1 -- start start 2021-05-25T12:00:00.116 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.083+0000 7fb871e46700 1 librados: setting wanted keys 2021-05-25T12:00:00.116 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.083+0000 7fb871e46700 1 librados: calling monclient init 2021-05-25T12:00:00.117 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.083+0000 7fb871e46700 10 monclient: init 2021-05-25T12:00:00.117 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.083+0000 7fb871e46700 10 monclient: _reopen_session rank -1 2021-05-25T12:00:00.117 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.083+0000 7fb871e46700 10 monclient: _add_conns ranks=[0,1,2] 2021-05-25T12:00:00.117 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.083+0000 7fb871e46700 10 monclient(hunting): picked mon.noname-a con 0x7fb86c068da0 addr v1:172.21.15.102:6789/0 2021-05-25T12:00:00.117 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.083+0000 7fb871e46700 10 monclient(hunting): picked mon.noname-b con 0x7fb86c069250 addr v1:172.21.15.181:6789/0 2021-05-25T12:00:00.118 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.083+0000 7fb871e46700 10 monclient(hunting): picked mon.noname-c con 0x7fb86c128c90 addr v1:172.21.15.102:6790/0 2021-05-25T12:00:00.118 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.083+0000 7fb871e46700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb86c096d30 con 0x7fb86c068da0 2021-05-25T12:00:00.118 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.083+0000 7fb871e46700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb86c065350 con 0x7fb86c069250 2021-05-25T12:00:00.118 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.083+0000 7fb871e46700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb86c114bc0 con 0x7fb86c128c90 2021-05-25T12:00:00.118 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.083+0000 7fb871e46700 10 monclient(hunting): _renew_subs 2021-05-25T12:00:00.119 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.083+0000 7fb871e46700 10 monclient(hunting): authenticate will time out at 800.681462s 2021-05-25T12:00:00.119 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.083+0000 7fb86b7fe700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fb86c068da0 0x7fb86c127180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:59050/0 (socket says 172.21.15.102:59050) 2021-05-25T12:00:00.119 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.083+0000 7fb86b7fe700 1 -- 172.21.15.102:0/4282305267 learned_addr learned my addr 172.21.15.102:0/4282305267 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:00:00.119 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.083+0000 7fb8577fe700 10 client.?.objecter ms_handle_connect 0x7fb86c068da0 2021-05-25T12:00:00.120 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.083+0000 7fb8577fe700 10 client.?.objecter resend_mon_ops 2021-05-25T12:00:00.120 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.083+0000 7fb8577fe700 10 client.?.objecter ms_handle_connect 0x7fb86c128c90 2021-05-25T12:00:00.120 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.083+0000 7fb8577fe700 10 client.?.objecter resend_mon_ops 2021-05-25T12:00:00.120 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.083+0000 7fb8577fe700 1 -- 172.21.15.102:0/4282305267 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3867380821 0 0) 0x7fb86c096d30 con 0x7fb86c068da0 2021-05-25T12:00:00.120 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.083+0000 7fb8577fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:00.121 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.083+0000 7fb8577fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:00.121 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.083+0000 7fb8577fe700 10 monclient(hunting): my global_id is 4371 2021-05-25T12:00:00.121 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.083+0000 7fb8577fe700 1 -- 172.21.15.102:0/4282305267 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb84c004c10 con 0x7fb86c068da0 2021-05-25T12:00:00.121 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.083+0000 7fb8577fe700 1 -- 172.21.15.102:0/4282305267 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4251627389 0 0) 0x7fb86c114bc0 con 0x7fb86c128c90 2021-05-25T12:00:00.121 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.083+0000 7fb8577fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:00.122 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.083+0000 7fb8577fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:00.122 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.083+0000 7fb8577fe700 10 monclient(hunting): my global_id is 4364 2021-05-25T12:00:00.122 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.083+0000 7fb8577fe700 1 -- 172.21.15.102:0/4282305267 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb86c096d30 con 0x7fb86c128c90 2021-05-25T12:00:00.122 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.083+0000 7fb8577fe700 10 client.?.objecter ms_handle_connect 0x7fb86c069250 2021-05-25T12:00:00.123 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.083+0000 7fb8577fe700 10 client.?.objecter resend_mon_ops 2021-05-25T12:00:00.123 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.083+0000 7fb8577fe700 1 -- 172.21.15.102:0/4282305267 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3452421982 0 0) 0x7fb86c065350 con 0x7fb86c069250 2021-05-25T12:00:00.123 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.083+0000 7fb8577fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:00.123 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.083+0000 7fb8577fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:00.123 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.083+0000 7fb8577fe700 10 monclient(hunting): my global_id is 4366 2021-05-25T12:00:00.124 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.083+0000 7fb8577fe700 1 -- 172.21.15.102:0/4282305267 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb86c114bc0 con 0x7fb86c069250 2021-05-25T12:00:00.124 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.083+0000 7fb8577fe700 1 -- 172.21.15.102:0/4282305267 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1058619456 0 0) 0x7fb86c096d30 con 0x7fb86c128c90 2021-05-25T12:00:00.124 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.084+0000 7fb8577fe700 1 -- 172.21.15.102:0/4282305267 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb86c065350 con 0x7fb86c128c90 2021-05-25T12:00:00.124 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.084+0000 7fb8577fe700 1 -- 172.21.15.102:0/4282305267 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4242867793 0 0) 0x7fb84c004c10 con 0x7fb86c068da0 2021-05-25T12:00:00.124 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.084+0000 7fb8577fe700 1 -- 172.21.15.102:0/4282305267 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb86c096d30 con 0x7fb86c068da0 2021-05-25T12:00:00.125 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.084+0000 7fb8577fe700 1 -- 172.21.15.102:0/4282305267 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fb860001d70 con 0x7fb86c128c90 2021-05-25T12:00:00.125 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.084+0000 7fb8577fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:00.125 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.084+0000 7fb8577fe700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:00:00.125 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.084+0000 7fb8577fe700 10 monclient(hunting): dump: 2021-05-25T12:00:00.126 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:00.126 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:00.126 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:00.126 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:00.126 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:00.127 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:00.127 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:00.127 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:00.127 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:00.128 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:00.128 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.084+0000 7fb8577fe700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:00:00.128 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.084+0000 7fb8577fe700 1 -- 172.21.15.102:0/4282305267 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fb858001690 con 0x7fb86c068da0 2021-05-25T12:00:00.128 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.084+0000 7fb8577fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:00.128 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.084+0000 7fb8577fe700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2021-05-25T12:00:00.129 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.084+0000 7fb8577fe700 10 monclient(hunting): dump: 2021-05-25T12:00:00.129 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:00.129 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:00.129 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:00.129 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:00.130 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:00.130 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:00.130 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:00.130 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:00.131 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:00.131 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:00.131 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.084+0000 7fb8577fe700 1 -- 172.21.15.102:0/4282305267 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1757155361 0 0) 0x7fb86c114bc0 con 0x7fb86c069250 2021-05-25T12:00:00.131 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.084+0000 7fb8577fe700 1 -- 172.21.15.102:0/4282305267 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb84c004c10 con 0x7fb86c069250 2021-05-25T12:00:00.132 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.084+0000 7fb8577fe700 1 -- 172.21.15.102:0/4282305267 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fb85c001900 con 0x7fb86c069250 2021-05-25T12:00:00.132 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.084+0000 7fb8577fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:00.132 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.084+0000 7fb8577fe700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2021-05-25T12:00:00.132 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.084+0000 7fb8577fe700 10 monclient(hunting): dump: 2021-05-25T12:00:00.133 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:00.133 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:00.133 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:00.133 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:00.133 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:00.134 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:00.134 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:00.134 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:00.134 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:00.135 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:00.135 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.084+0000 7fb8577fe700 1 -- 172.21.15.102:0/4282305267 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2655226926 0 0) 0x7fb86c096d30 con 0x7fb86c068da0 2021-05-25T12:00:00.135 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.084+0000 7fb8577fe700 1 -- 172.21.15.102:0/4282305267 >> v1:172.21.15.102:6790/0 conn(0x7fb86c128c90 legacy=0x7fb86c13f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:00.135 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.084+0000 7fb8577fe700 1 -- 172.21.15.102:0/4282305267 >> v1:172.21.15.181:6789/0 conn(0x7fb86c069250 legacy=0x7fb86c13bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:00.135 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.084+0000 7fb8577fe700 10 monclient: _finish_hunting 0 2021-05-25T12:00:00.136 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.084+0000 7fb8577fe700 1 monclient: found mon.a 2021-05-25T12:00:00.136 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.084+0000 7fb8577fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:00:00.136 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.084+0000 7fb8577fe700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:00:00.136 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.084+0000 7fb8577fe700 1 -- 172.21.15.102:0/4282305267 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb86c1140f0 con 0x7fb86c068da0 2021-05-25T12:00:00.136 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.084+0000 7fb8577fe700 10 monclient: _finish_auth 0 2021-05-25T12:00:00.137 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.084+0000 7fb8577fe700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:00:00.137 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.084+0000 7fb871e46700 5 monclient: authenticate success, global_id 4371 2021-05-25T12:00:00.137 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.084+0000 7fb871e46700 10 monclient: _renew_subs 2021-05-25T12:00:00.137 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.084+0000 7fb871e46700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:00:00.137 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.084+0000 7fb871e46700 1 -- 172.21.15.102:0/4282305267 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb86c0954e0 con 0x7fb86c068da0 2021-05-25T12:00:00.138 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.084+0000 7fb871e46700 10 client.4371.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:00:00.138 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.084+0000 7fb871e46700 10 monclient: _renew_subs 2021-05-25T12:00:00.138 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.084+0000 7fb871e46700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:00:00.138 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.084+0000 7fb871e46700 1 -- 172.21.15.102:0/4282305267 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb86c127a50 con 0x7fb86c068da0 2021-05-25T12:00:00.139 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.084+0000 7fb871e46700 1 librados: init done 2021-05-25T12:00:00.139 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.084+0000 7fb8577fe700 1 -- 172.21.15.102:0/4282305267 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb8580022a0 con 0x7fb86c068da0 2021-05-25T12:00:00.139 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.084+0000 7fb8577fe700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:00:00.139 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.084+0000 7fb8577fe700 1 -- 172.21.15.102:0/4282305267 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fb858002630 con 0x7fb86c068da0 2021-05-25T12:00:00.139 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.084+0000 7fb8577fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:00.140 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.084+0000 7fb869ffb700 4 set_mon_vals no callback set 2021-05-25T12:00:00.140 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.084+0000 7fb8577fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:00:00.140 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.084+0000 7fb8577fe700 10 monclient: dump: 2021-05-25T12:00:00.140 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:00.140 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:00.140 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:00.141 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:00.141 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:00.141 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:00.141 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:00.142 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:00.142 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:00.142 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:00.142 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.085+0000 7fb8577fe700 1 -- 172.21.15.102:0/4282305267 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7fb858002810 con 0x7fb86c068da0 2021-05-25T12:00:00.142 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.085+0000 7fb871e46700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:00:00.143 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.085+0000 7fb871e46700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:00:00.143 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.085+0000 7fb871e46700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:00:00.143 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.085+0000 7fb871e46700 1 -- 172.21.15.102:0/4282305267 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb844004db0 con 0x7fb86c068da0 2021-05-25T12:00:00.143 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.085+0000 7fb8577fe700 1 -- 172.21.15.102:0/4282305267 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(15..15 src has 1..15) v4 ==== 3549+0+0 (unknown 3948819498 0 0) 0x7fb85803cde0 con 0x7fb86c068da0 2021-05-25T12:00:00.143 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.085+0000 7fb8577fe700 10 client.4371.objecter ms_dispatch 0x7fb86c001550 osd_map(15..15 src has 1..15) v4 2021-05-25T12:00:00.144 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.085+0000 7fb8577fe700 3 client.4371.objecter handle_osd_map got epochs [15,15] > 0 2021-05-25T12:00:00.144 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.085+0000 7fb8577fe700 3 client.4371.objecter handle_osd_map decoding full epoch 15 2021-05-25T12:00:00.144 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.085+0000 7fb8577fe700 20 client.4371.objecter dump_active .. 0 homeless 2021-05-25T12:00:00.144 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.085+0000 7fb86affd700 10 monclient: get_auth_request con 0x7fb84c033550 auth_method 0 2021-05-25T12:00:00.144 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.085+0000 7fb8577fe700 10 client.4371.objecter ms_handle_connect 0x7fb84c033550 2021-05-25T12:00:00.145 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.089+0000 7fb8577fe700 1 -- 172.21.15.102:0/4282305267 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7fb858012b60 con 0x7fb86c068da0 2021-05-25T12:00:00.145 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.089+0000 7fb8577fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:00:00.145 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.089+0000 7fb8577fe700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:00:00.241 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.238+0000 7fb871e46700 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2021-05-25T12:00:00.241 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.238+0000 7fb871e46700 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2021-05-25T12:00:00.241 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.238+0000 7fb871e46700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:00:00.242 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.238+0000 7fb871e46700 1 -- 172.21.15.102:0/4282305267 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fb8440055e0 con 0x7fb86c068da0 2021-05-25T12:00:00.242 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.238+0000 7fb8577fe700 1 -- 172.21.15.102:0/4282305267 <== mon.0 v1:172.21.15.102:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v15) v1 ==== 92+0+10 (unknown 946793192 0 265718887) 0x7fb858012980 con 0x7fb86c068da0 2021-05-25T12:00:00.242 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.238+0000 7fb8577fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2021-05-25T12:00:00.242 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.238+0000 7fb8577fe700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:00:00.243 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 8 2021-05-25T12:00:00.244 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.240+0000 7fb871e46700 10 librados: watch_flush enter 2021-05-25T12:00:00.244 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.240+0000 7fb871e46700 10 librados: watch_flush exit 2021-05-25T12:00:00.245 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.240+0000 7fb871e46700 20 client.4371.objecter shutdown clearing up homeless session... 2021-05-25T12:00:00.245 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.240+0000 7fb871e46700 10 client.4371.objecter successfully canceled tick 2021-05-25T12:00:00.245 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.240+0000 7fb871e46700 1 -- 172.21.15.102:0/4282305267 >> v1:172.21.15.181:6812/29582 conn(0x7fb84c033550 legacy=0x7fb84c035a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:00.245 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.240+0000 7fb871e46700 10 monclient: shutdown 2021-05-25T12:00:00.246 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.240+0000 7fb871e46700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:00:00.246 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.240+0000 7fb871e46700 1 -- 172.21.15.102:0/4282305267 >> v1:172.21.15.102:6789/0 conn(0x7fb86c068da0 legacy=0x7fb86c127180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:00.246 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.240+0000 7fb871e46700 1 -- 172.21.15.102:0/4282305267 shutdown_connections 2021-05-25T12:00:00.246 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.240+0000 7fb871e46700 1 -- 172.21.15.102:0/4282305267 >> 172.21.15.102:0/4282305267 conn(0x7fb86c0984c0 msgr2=0x7fb86c0661b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:00:00.247 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.240+0000 7fb871e46700 1 -- 172.21.15.102:0/4282305267 shutdown_connections 2021-05-25T12:00:00.247 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.240+0000 7fb871e46700 1 -- 172.21.15.102:0/4282305267 wait complete. 2021-05-25T12:00:00.247 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.240+0000 7fb871e46700 1 librados: shutdown 2021-05-25T12:00:00.253 INFO:tasks.mon_thrash.ceph_manager:waiting for clean 2021-05-25T12:00:00.254 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-05-25T12:00:00.366 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.364+0000 7f3982c77700 10 monclient: get_monmap_and_config 2021-05-25T12:00:00.367 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.364+0000 7f3982c77700 10 monclient: build_initial_monmap 2021-05-25T12:00:00.367 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.364+0000 7f3982c77700 10 monclient: monmap: 2021-05-25T12:00:00.367 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:00:00.367 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:00.368 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:00:00.365937+0000 2021-05-25T12:00:00.368 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:00:00.365937+0000 2021-05-25T12:00:00.368 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:00:00.368 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:00.369 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:00:00.369 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:00:00.369 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:00:00.369 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:00.380 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.364+0000 7f3982c77700 1 Processor -- start 2021-05-25T12:00:00.380 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.364+0000 7f3982c77700 1 -- start start 2021-05-25T12:00:00.380 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.364+0000 7f3982c77700 10 monclient: init 2021-05-25T12:00:00.380 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.365+0000 7f3982c77700 10 monclient: _reopen_session rank -1 2021-05-25T12:00:00.381 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.365+0000 7f3982c77700 10 monclient: _add_conns ranks=[1,0,2] 2021-05-25T12:00:00.381 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.365+0000 7f3982c77700 10 monclient(hunting): picked mon.noname-c con 0x7f397c093d90 addr v1:172.21.15.181:6789/0 2021-05-25T12:00:00.381 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.365+0000 7f3982c77700 10 monclient(hunting): picked mon.noname-a con 0x7f397c132900 addr v1:172.21.15.102:6789/0 2021-05-25T12:00:00.381 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.365+0000 7f3982c77700 10 monclient(hunting): picked mon.noname-b con 0x7f397c136410 addr v1:172.21.15.102:6790/0 2021-05-25T12:00:00.382 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.365+0000 7f3982c77700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f397c091ed0 con 0x7f397c132900 2021-05-25T12:00:00.382 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.365+0000 7f3982c77700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f397c092170 con 0x7f397c093d90 2021-05-25T12:00:00.382 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.365+0000 7f3982c77700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f397c092410 con 0x7f397c136410 2021-05-25T12:00:00.382 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.365+0000 7f3982c77700 10 monclient(hunting): _renew_subs 2021-05-25T12:00:00.382 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.365+0000 7f3982c77700 10 monclient(hunting): authenticate will time out at 800.963178s 2021-05-25T12:00:00.383 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.365+0000 7f3973fff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f397c132900 0x7f397c134cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:59058/0 (socket says 172.21.15.102:59058) 2021-05-25T12:00:00.383 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.365+0000 7f3973fff700 1 -- 172.21.15.102:0/4079165398 learned_addr learned my addr 172.21.15.102:0/4079165398 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:00:00.383 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.365+0000 7f3981214700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f397c136410 0x7f397c138800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:53940/0 (socket says 172.21.15.102:53940) 2021-05-25T12:00:00.383 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.365+0000 7f39737fe700 1 -- 172.21.15.102:0/4079165398 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4010297065 0 0) 0x7f397c092410 con 0x7f397c136410 2021-05-25T12:00:00.384 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.365+0000 7f39737fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:00.384 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.365+0000 7f39737fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:00.384 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.365+0000 7f39737fe700 10 monclient(hunting): my global_id is 4367 2021-05-25T12:00:00.384 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.365+0000 7f39737fe700 1 -- 172.21.15.102:0/4079165398 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3968003270 con 0x7f397c136410 2021-05-25T12:00:00.384 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.365+0000 7f39737fe700 1 -- 172.21.15.102:0/4079165398 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1080274883 0 0) 0x7f397c091ed0 con 0x7f397c132900 2021-05-25T12:00:00.385 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.365+0000 7f39737fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:00.385 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.365+0000 7f39737fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:00.385 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.365+0000 7f39737fe700 10 monclient(hunting): my global_id is 4374 2021-05-25T12:00:00.385 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.365+0000 7f39737fe700 1 -- 172.21.15.102:0/4079165398 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f397c092410 con 0x7f397c132900 2021-05-25T12:00:00.385 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.365+0000 7f39737fe700 1 -- 172.21.15.102:0/4079165398 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2194227660 0 0) 0x7f3968003270 con 0x7f397c136410 2021-05-25T12:00:00.386 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.365+0000 7f39737fe700 1 -- 172.21.15.102:0/4079165398 >> v1:172.21.15.181:6789/0 conn(0x7f397c093d90 legacy=0x7f397c0691c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:00.386 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.365+0000 7f39737fe700 1 -- 172.21.15.102:0/4079165398 >> v1:172.21.15.102:6789/0 conn(0x7f397c132900 legacy=0x7f397c134cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:00.386 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.365+0000 7f39737fe700 10 monclient: _finish_hunting 0 2021-05-25T12:00:00.386 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.365+0000 7f39737fe700 1 monclient: found mon.noname-b 2021-05-25T12:00:00.386 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.365+0000 7f39737fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:00:00.387 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.365+0000 7f39737fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T12:00:00.387 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.365+0000 7f39737fe700 1 -- 172.21.15.102:0/4079165398 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f397c094170 con 0x7f397c136410 2021-05-25T12:00:00.387 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.365+0000 7f39737fe700 10 monclient: _finish_auth 0 2021-05-25T12:00:00.387 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.365+0000 7f39737fe700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:00:00.387 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.365+0000 7f3982c77700 5 monclient: authenticate success, global_id 4367 2021-05-25T12:00:00.388 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.365+0000 7f39737fe700 1 -- 172.21.15.102:0/4079165398 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f3978001990 con 0x7f397c136410 2021-05-25T12:00:00.388 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.365+0000 7f3982c77700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:00:00.388 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.365+0000 7f39737fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:00.388 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.365+0000 7f39737fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:00:00.388 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.365+0000 7f39737fe700 10 monclient: dump: 2021-05-25T12:00:00.389 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:00.389 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:00.389 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:00.389 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:00.389 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:00.389 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:00.390 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:00.390 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:00.390 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:00.390 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:00.391 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.365+0000 7f3982c77700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:00:00.391 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.365+0000 7f39737fe700 1 -- 172.21.15.102:0/4079165398 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3978001c30 con 0x7f397c136410 2021-05-25T12:00:00.391 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.365+0000 7f39737fe700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:00:00.391 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.365+0000 7f39737fe700 1 -- 172.21.15.102:0/4079165398 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f3978001fc0 con 0x7f397c136410 2021-05-25T12:00:00.391 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.365+0000 7f3982c77700 10 monclient: get_monmap_and_config success 2021-05-25T12:00:00.392 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.365+0000 7f3982c77700 4 set_mon_vals no callback set 2021-05-25T12:00:00.392 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.365+0000 7f39737fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:00.392 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.365+0000 7f39737fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:00:00.392 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.365+0000 7f39737fe700 10 monclient: dump: 2021-05-25T12:00:00.393 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:00.393 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:00.393 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:00.393 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:00.393 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:00.394 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:00.394 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:00.394 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:00.394 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:00.395 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:00.395 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.366+0000 7f3982c77700 10 monclient: shutdown 2021-05-25T12:00:00.395 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.366+0000 7f3982c77700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:00:00.395 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.366+0000 7f3982c77700 1 -- 172.21.15.102:0/4079165398 >> v1:172.21.15.102:6790/0 conn(0x7f397c136410 legacy=0x7f397c138800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:00.396 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.366+0000 7f3982c77700 1 -- 172.21.15.102:0/4079165398 shutdown_connections 2021-05-25T12:00:00.396 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.366+0000 7f3982c77700 1 -- 172.21.15.102:0/4079165398 >> 172.21.15.102:0/4079165398 conn(0x7f397c09b370 msgr2=0x7f397c097430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:00:00.396 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.366+0000 7f3982c77700 1 -- 172.21.15.102:0/4079165398 shutdown_connections 2021-05-25T12:00:00.396 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.366+0000 7f3982c77700 1 -- 172.21.15.102:0/4079165398 wait complete. 2021-05-25T12:00:00.396 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.366+0000 7f3982c77700 10 monclient: build_initial_monmap 2021-05-25T12:00:00.397 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.366+0000 7f3982c77700 10 monclient: monmap: 2021-05-25T12:00:00.397 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:00:00.397 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:00:00.397 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:00:00.398 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:00:00.398 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:00:00.398 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:00:00.398 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:00:00.399 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:00:00.399 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:00:00.399 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:00.399 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.367+0000 7f3982c77700 1 librados: starting msgr at 2021-05-25T12:00:00.399 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.367+0000 7f3982c77700 1 librados: starting objecter 2021-05-25T12:00:00.400 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.367+0000 7f3982c77700 1 Processor -- start 2021-05-25T12:00:00.400 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.367+0000 7f3982c77700 1 -- start start 2021-05-25T12:00:00.400 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.367+0000 7f3982c77700 1 librados: setting wanted keys 2021-05-25T12:00:00.400 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.367+0000 7f3982c77700 1 librados: calling monclient init 2021-05-25T12:00:00.401 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.367+0000 7f3982c77700 10 monclient: init 2021-05-25T12:00:00.401 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.368+0000 7f3982c77700 10 monclient: _reopen_session rank -1 2021-05-25T12:00:00.401 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.368+0000 7f3982c77700 10 monclient: _add_conns ranks=[0,1,2] 2021-05-25T12:00:00.401 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.368+0000 7f3982c77700 10 monclient(hunting): picked mon.noname-a con 0x7f397c093d90 addr v1:172.21.15.102:6789/0 2021-05-25T12:00:00.401 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.368+0000 7f3982c77700 10 monclient(hunting): picked mon.noname-b con 0x7f397c132900 addr v1:172.21.15.181:6789/0 2021-05-25T12:00:00.402 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.368+0000 7f3982c77700 10 monclient(hunting): picked mon.noname-c con 0x7f397c136410 addr v1:172.21.15.102:6790/0 2021-05-25T12:00:00.402 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.368+0000 7f3982c77700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f397c0685d0 con 0x7f397c093d90 2021-05-25T12:00:00.402 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.368+0000 7f3982c77700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f397c068870 con 0x7f397c132900 2021-05-25T12:00:00.402 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.368+0000 7f3982c77700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f397c0669c0 con 0x7f397c136410 2021-05-25T12:00:00.402 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.368+0000 7f3982c77700 10 monclient(hunting): _renew_subs 2021-05-25T12:00:00.403 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.368+0000 7f3982c77700 10 monclient(hunting): authenticate will time out at 800.966209s 2021-05-25T12:00:00.403 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.368+0000 7f3980a13700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f397c093d90 0x7f397c131b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:59064/0 (socket says 172.21.15.102:59064) 2021-05-25T12:00:00.403 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.368+0000 7f3981214700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f397c136410 0x7f397c147d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:53946/0 (socket says 172.21.15.102:53946) 2021-05-25T12:00:00.403 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.368+0000 7f3980a13700 1 -- 172.21.15.102:0/3335866079 learned_addr learned my addr 172.21.15.102:0/3335866079 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:00:00.404 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.368+0000 7f3970ff9700 10 client.?.objecter ms_handle_connect 0x7f397c136410 2021-05-25T12:00:00.404 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.368+0000 7f3970ff9700 10 client.?.objecter resend_mon_ops 2021-05-25T12:00:00.404 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.368+0000 7f3970ff9700 10 client.?.objecter ms_handle_connect 0x7f397c093d90 2021-05-25T12:00:00.404 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.368+0000 7f3970ff9700 10 client.?.objecter resend_mon_ops 2021-05-25T12:00:00.404 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.368+0000 7f3970ff9700 10 client.?.objecter ms_handle_connect 0x7f397c132900 2021-05-25T12:00:00.405 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.368+0000 7f3970ff9700 10 client.?.objecter resend_mon_ops 2021-05-25T12:00:00.405 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.368+0000 7f3970ff9700 1 -- 172.21.15.102:0/3335866079 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1861959925 0 0) 0x7f397c0669c0 con 0x7f397c136410 2021-05-25T12:00:00.405 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.368+0000 7f3970ff9700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:00.405 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.368+0000 7f3970ff9700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:00.405 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.368+0000 7f3970ff9700 10 monclient(hunting): my global_id is 4370 2021-05-25T12:00:00.406 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.368+0000 7f3970ff9700 1 -- 172.21.15.102:0/3335866079 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f395c004c10 con 0x7f397c136410 2021-05-25T12:00:00.406 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.368+0000 7f3970ff9700 1 -- 172.21.15.102:0/3335866079 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3970372270 0 0) 0x7f397c0685d0 con 0x7f397c093d90 2021-05-25T12:00:00.406 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.368+0000 7f3970ff9700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:00.406 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.368+0000 7f3970ff9700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:00.406 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.368+0000 7f3970ff9700 10 monclient(hunting): my global_id is 4377 2021-05-25T12:00:00.407 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.368+0000 7f3970ff9700 1 -- 172.21.15.102:0/3335866079 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f397c0669c0 con 0x7f397c093d90 2021-05-25T12:00:00.407 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.368+0000 7f3970ff9700 1 -- 172.21.15.102:0/3335866079 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1245131474 0 0) 0x7f397c068870 con 0x7f397c132900 2021-05-25T12:00:00.407 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.368+0000 7f3970ff9700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:00.407 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.368+0000 7f3970ff9700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:00.408 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.368+0000 7f3970ff9700 10 monclient(hunting): my global_id is 4372 2021-05-25T12:00:00.408 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.368+0000 7f3970ff9700 1 -- 172.21.15.102:0/3335866079 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f397c0685d0 con 0x7f397c132900 2021-05-25T12:00:00.408 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.368+0000 7f3970ff9700 1 -- 172.21.15.102:0/3335866079 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1317868360 0 0) 0x7f395c004c10 con 0x7f397c136410 2021-05-25T12:00:00.408 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.368+0000 7f3970ff9700 1 -- 172.21.15.102:0/3335866079 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f397c068870 con 0x7f397c136410 2021-05-25T12:00:00.408 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.368+0000 7f3970ff9700 1 -- 172.21.15.102:0/3335866079 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f3978001f40 con 0x7f397c136410 2021-05-25T12:00:00.409 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.368+0000 7f3970ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:00.409 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.368+0000 7f3970ff9700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:00:00.409 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.368+0000 7f3970ff9700 10 monclient(hunting): dump: 2021-05-25T12:00:00.409 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:00.410 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:00.410 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:00.410 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:00.410 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:00.410 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:00.411 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:00.411 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:00.411 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:00.411 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:00.411 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.368+0000 7f3970ff9700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:00:00.412 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.368+0000 7f3970ff9700 1 -- 172.21.15.102:0/3335866079 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1405538937 0 0) 0x7f397c0669c0 con 0x7f397c093d90 2021-05-25T12:00:00.412 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.368+0000 7f3970ff9700 1 -- 172.21.15.102:0/3335866079 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f395c004c10 con 0x7f397c093d90 2021-05-25T12:00:00.412 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.368+0000 7f3970ff9700 1 -- 172.21.15.102:0/3335866079 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f396c0017f0 con 0x7f397c093d90 2021-05-25T12:00:00.412 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.368+0000 7f3970ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:00.412 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.368+0000 7f3970ff9700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2021-05-25T12:00:00.413 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.368+0000 7f3970ff9700 10 monclient(hunting): dump: 2021-05-25T12:00:00.413 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:00.413 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:00.413 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:00.413 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:00.414 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:00.414 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:00.414 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:00.414 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:00.414 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:00.415 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:00.415 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.368+0000 7f3970ff9700 1 -- 172.21.15.102:0/3335866079 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2876689399 0 0) 0x7f397c0685d0 con 0x7f397c132900 2021-05-25T12:00:00.415 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.368+0000 7f3970ff9700 1 -- 172.21.15.102:0/3335866079 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f397c0669c0 con 0x7f397c132900 2021-05-25T12:00:00.415 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.368+0000 7f3970ff9700 1 -- 172.21.15.102:0/3335866079 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f39640019f0 con 0x7f397c132900 2021-05-25T12:00:00.416 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.368+0000 7f3970ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:00.416 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.368+0000 7f3970ff9700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2021-05-25T12:00:00.416 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.368+0000 7f3970ff9700 10 monclient(hunting): dump: 2021-05-25T12:00:00.416 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:00.416 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:00.417 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:00.417 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:00.417 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:00.417 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:00.418 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:00.418 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:00.418 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:00.418 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:00.418 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.368+0000 7f3970ff9700 1 -- 172.21.15.102:0/3335866079 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3286053213 0 0) 0x7f397c068870 con 0x7f397c136410 2021-05-25T12:00:00.419 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.368+0000 7f3970ff9700 1 -- 172.21.15.102:0/3335866079 >> v1:172.21.15.181:6789/0 conn(0x7f397c132900 legacy=0x7f397c144620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:00.419 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.368+0000 7f3970ff9700 1 -- 172.21.15.102:0/3335866079 >> v1:172.21.15.102:6789/0 conn(0x7f397c093d90 legacy=0x7f397c131b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:00.419 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.368+0000 7f3970ff9700 10 monclient: _finish_hunting 0 2021-05-25T12:00:00.419 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.368+0000 7f3970ff9700 1 monclient: found mon.c 2021-05-25T12:00:00.419 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.368+0000 7f3970ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:00:00.420 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.368+0000 7f3970ff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:00:00.420 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.368+0000 7f3970ff9700 1 -- 172.21.15.102:0/3335866079 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f397c13a540 con 0x7f397c136410 2021-05-25T12:00:00.420 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.368+0000 7f3970ff9700 10 monclient: _finish_auth 0 2021-05-25T12:00:00.420 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.368+0000 7f3970ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:00:00.421 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.369+0000 7f3982c77700 5 monclient: authenticate success, global_id 4370 2021-05-25T12:00:00.421 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.369+0000 7f3982c77700 10 monclient: _renew_subs 2021-05-25T12:00:00.421 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.369+0000 7f3982c77700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:00:00.421 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.369+0000 7f3982c77700 1 -- 172.21.15.102:0/3335866079 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f397c13cc70 con 0x7f397c136410 2021-05-25T12:00:00.421 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.369+0000 7f3982c77700 10 client.4370.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:00:00.422 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.369+0000 7f3982c77700 10 monclient: _renew_subs 2021-05-25T12:00:00.422 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.369+0000 7f3982c77700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:00:00.422 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.369+0000 7f3982c77700 1 -- 172.21.15.102:0/3335866079 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f397c0677f0 con 0x7f397c136410 2021-05-25T12:00:00.422 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.369+0000 7f3982c77700 1 librados: init done 2021-05-25T12:00:00.422 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.369+0000 7f3970ff9700 1 -- 172.21.15.102:0/3335866079 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3978001500 con 0x7f397c136410 2021-05-25T12:00:00.423 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.369+0000 7f3970ff9700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:00:00.423 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.369+0000 7f3970ff9700 1 -- 172.21.15.102:0/3335866079 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f3978004da0 con 0x7f397c136410 2021-05-25T12:00:00.423 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.369+0000 7f3970ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:00.423 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.369+0000 7f3972ffd700 4 set_mon_vals no callback set 2021-05-25T12:00:00.424 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.369+0000 7f3970ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:00:00.424 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.369+0000 7f3970ff9700 10 monclient: dump: 2021-05-25T12:00:00.424 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:00.424 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:00.424 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:00.425 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:00.425 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:00.425 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:00.425 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:00.425 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:00.426 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:00.426 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:00.426 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.369+0000 7f3970ff9700 1 -- 172.21.15.102:0/3335866079 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f397800e0c0 con 0x7f397c136410 2021-05-25T12:00:00.426 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.369+0000 7f3982c77700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:00:00.426 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.369+0000 7f3982c77700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:00:00.427 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.369+0000 7f3982c77700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:00:00.427 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.369+0000 7f3982c77700 1 -- 172.21.15.102:0/3335866079 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3954004db0 con 0x7f397c136410 2021-05-25T12:00:00.427 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.370+0000 7f3970ff9700 1 -- 172.21.15.102:0/3335866079 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(15..15 src has 1..15) v4 ==== 3549+0+0 (unknown 3948819498 0 0) 0x7f397803d200 con 0x7f397c136410 2021-05-25T12:00:00.427 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.370+0000 7f3970ff9700 10 client.4370.objecter ms_dispatch 0x7f397c001550 osd_map(15..15 src has 1..15) v4 2021-05-25T12:00:00.427 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.370+0000 7f3970ff9700 3 client.4370.objecter handle_osd_map got epochs [15,15] > 0 2021-05-25T12:00:00.428 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.370+0000 7f3970ff9700 3 client.4370.objecter handle_osd_map decoding full epoch 15 2021-05-25T12:00:00.428 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.370+0000 7f3970ff9700 20 client.4370.objecter dump_active .. 0 homeless 2021-05-25T12:00:00.428 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.370+0000 7f3980a13700 10 monclient: get_auth_request con 0x7f395c033550 auth_method 0 2021-05-25T12:00:00.428 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.370+0000 7f3970ff9700 10 client.4370.objecter ms_handle_connect 0x7f395c033550 2021-05-25T12:00:00.428 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.374+0000 7f3970ff9700 1 -- 172.21.15.102:0/3335866079 <== mon.2 v1:172.21.15.102:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7f3978013030 con 0x7f397c136410 2021-05-25T12:00:00.429 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.374+0000 7f3970ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:00:00.429 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.374+0000 7f3970ff9700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:00:00.521 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.519+0000 7f3982c77700 1 -- 172.21.15.102:0/3335866079 --> v1:172.21.15.181:6812/29582 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3954000b90 con 0x7f395c033550 2021-05-25T12:00:00.523 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.521+0000 7f3970ff9700 1 -- 172.21.15.102:0/3335866079 <== mgr.4105 v1:172.21.15.181:6812/29582 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+38639 (unknown 2965378022 0 3736317541) 0x7f3954000b90 con 0x7f395c033550 2021-05-25T12:00:00.524 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T12:00:00.525 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.522+0000 7f3982c77700 10 librados: watch_flush enter 2021-05-25T12:00:00.526 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.522+0000 7f3982c77700 10 librados: watch_flush exit 2021-05-25T12:00:00.526 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.522+0000 7f3982c77700 20 client.4370.objecter shutdown clearing up homeless session... 2021-05-25T12:00:00.526 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.522+0000 7f3982c77700 10 client.4370.objecter successfully canceled tick 2021-05-25T12:00:00.526 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.522+0000 7f3982c77700 1 -- 172.21.15.102:0/3335866079 >> v1:172.21.15.181:6812/29582 conn(0x7f395c033550 legacy=0x7f395c035a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:00.527 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.522+0000 7f3982c77700 10 monclient: shutdown 2021-05-25T12:00:00.527 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.522+0000 7f3982c77700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:00:00.527 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.522+0000 7f3982c77700 1 -- 172.21.15.102:0/3335866079 >> v1:172.21.15.102:6790/0 conn(0x7f397c136410 legacy=0x7f397c147d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:00.527 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.522+0000 7f3982c77700 1 -- 172.21.15.102:0/3335866079 shutdown_connections 2021-05-25T12:00:00.527 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.522+0000 7f3982c77700 1 -- 172.21.15.102:0/3335866079 >> 172.21.15.102:0/3335866079 conn(0x7f397c09b370 msgr2=0x7f397c138ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:00:00.528 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.522+0000 7f3982c77700 1 -- 172.21.15.102:0/3335866079 shutdown_connections 2021-05-25T12:00:00.528 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.522+0000 7f3982c77700 1 -- 172.21.15.102:0/3335866079 wait complete. 2021-05-25T12:00:00.528 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.522+0000 7f3982c77700 1 librados: shutdown 2021-05-25T12:00:00.528 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2021-05-25T12:00:00.536 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2021-05-25T11:59:59.556595+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":11,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":29544,"kb_used_data":760,"kb_used_omap":0,"kb_used_meta":28736,"kb_avail":566201496,"statfs":{"total":579820584960,"available":579790331904,"internally_reserved":0,"allocated":778240,"data_stored":230516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29425664},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2021-05-25T11:59:52.780051+0000","last_change":"2021-05-25T11:59:52.779532+0000","last_active":"2021-05-25T11:59:52.780051+0000","last_peered":"2021-05-25T11:59:52.780051+0000","last_clean":"2021-05-25T11:59:52.780051+0000","last_became_active":"2021-05-25T11:59:52.779240+0000","last_became_peered":"2021-05-25T11:59:52.779240+0000","last_unstale":"2021-05-25T11:59:52.780051+0000","last_undegraded":"2021-05-25T11:59:52.780051+0000","last_fullsized":"2021-05-25T11:59:52.780051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-05-25T11:59:51.583021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-05-25T11:59:51.583021+0000","last_clean_scrub_stamp":"2021-05-25T11:59:51.583021+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2021-05-25T11:59:52.600778+0000","last_change":"2021-05-25T11:59:52.599864+0000","last_active":"2021-05-25T11:59:52.600778+0000","last_peered":"2021-05-25T11:59:52.600778+0000","last_clean":"2021-05-25T11:59:52.600778+0000","last_became_active":"2021-05-25T11:59:52.599550+0000","last_became_peered":"2021-05-25T11:59:52.599550+0000","last_unstale":"2021-05-25T11:59:52.600778+0000","last_undegraded":"2021-05-25T11:59:52.600778+0000","last_fullsized":"2021-05-25T11:59:52.600778+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-05-25T11:59:51.583021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-05-25T11:59:51.583021+0000","last_clean_scrub_stamp":"2021-05-25T11:59:51.583021+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2021-05-25T11:59:52.599789+0000","last_change":"2021-05-25T11:59:52.599209+0000","last_active":"2021-05-25T11:59:52.599789+0000","last_peered":"2021-05-25T11:59:52.599789+0000","last_clean":"2021-05-25T11:59:52.599789+0000","last_became_active":"2021-05-25T11:59:52.598818+0000","last_became_peered":"2021-05-25T11:59:52.598818+0000","last_unstale":"2021-05-25T11:59:52.599789+0000","last_undegraded":"2021-05-25T11:59:52.599789+0000","last_fullsized":"2021-05-25T11:59:52.599789+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-05-25T11:59:51.583021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-05-25T11:59:51.583021+0000","last_clean_scrub_stamp":"2021-05-25T11:59:51.583021+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2021-05-25T11:59:52.599030+0000","last_change":"2021-05-25T11:59:52.596805+0000","last_active":"2021-05-25T11:59:52.599030+0000","last_peered":"2021-05-25T11:59:52.599030+0000","last_clean":"2021-05-25T11:59:52.599030+0000","last_became_active":"2021-05-25T11:59:52.596538+0000","last_became_peered":"2021-05-25T11:59:52.596538+0000","last_unstale":"2021-05-25T11:59:52.599030+0000","last_undegraded":"2021-05-25T11:59:52.599030+0000","last_fullsized":"2021-05-25T11:59:52.599030+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-05-25T11:59:51.583021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-05-25T11:59:51.583021+0000","last_clean_scrub_stamp":"2021-05-25T11:59:51.583021+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"10","reported_epoch":"14","state":"active+clean","last_fresh":"2021-05-25T11:59:52.593134+0000","last_change":"2021-05-25T11:59:51.592476+0000","last_active":"2021-05-25T11:59:52.593134+0000","last_peered":"2021-05-25T11:59:52.593134+0000","last_clean":"2021-05-25T11:59:52.593134+0000","last_became_active":"2021-05-25T11:59:51.592157+0000","last_became_peered":"2021-05-25T11:59:51.592157+0000","last_unstale":"2021-05-25T11:59:52.593134+0000","last_undegraded":"2021-05-25T11:59:52.593134+0000","last_fullsized":"2021-05-25T11:59:52.593134+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-05-25T11:59:50.578675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-05-25T11:59:50.578675+0000","last_clean_scrub_stamp":"2021-05-25T11:59:50.578675+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2021-05-25T11:59:52.607326+0000","last_change":"2021-05-25T11:59:52.602323+0000","last_active":"2021-05-25T11:59:52.607326+0000","last_peered":"2021-05-25T11:59:52.607326+0000","last_clean":"2021-05-25T11:59:52.607326+0000","last_became_active":"2021-05-25T11:59:52.602047+0000","last_became_peered":"2021-05-25T11:59:52.602047+0000","last_unstale":"2021-05-25T11:59:52.607326+0000","last_undegraded":"2021-05-25T11:59:52.607326+0000","last_fullsized":"2021-05-25T11:59:52.607326+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-05-25T11:59:51.583021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-05-25T11:59:51.583021+0000","last_clean_scrub_stamp":"2021-05-25T11:59:51.583021+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2021-05-25T11:59:52.600845+0000","last_change":"2021-05-25T11:59:52.598411+0000","last_active":"2021-05-25T11:59:52.600845+0000","last_peered":"2021-05-25T11:59:52.600845+0000","last_clean":"2021-05-25T11:59:52.600845+0000","last_became_active":"2021-05-25T11:59:52.598090+0000","last_became_peered":"2021-05-25T11:59:52.598090+0000","last_unstale":"2021-05-25T11:59:52.600845+0000","last_undegraded":"2021-05-25T11:59:52.600845+0000","last_fullsized":"2021-05-25T11:59:52.600845+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-05-25T11:59:51.583021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-05-25T11:59:51.583021+0000","last_clean_scrub_stamp":"2021-05-25T11:59:51.583021+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2021-05-25T11:59:52.607316+0000","last_change":"2021-05-25T11:59:52.602212+0000","last_active":"2021-05-25T11:59:52.607316+0000","last_peered":"2021-05-25T11:59:52.607316+0000","last_clean":"2021-05-25T11:59:52.607316+0000","last_became_active":"2021-05-25T11:59:52.601888+0000","last_became_peered":"2021-05-25T11:59:52.601888+0000","last_unstale":"2021-05-25T11:59:52.607316+0000","last_undegraded":"2021-05-25T11:59:52.607316+0000","last_fullsized":"2021-05-25T11:59:52.607316+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-05-25T11:59:51.583021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-05-25T11:59:51.583021+0000","last_clean_scrub_stamp":"2021-05-25T11:59:51.583021+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2021-05-25T11:59:52.598621+0000","last_change":"2021-05-25T11:59:52.597495+0000","last_active":"2021-05-25T11:59:52.598621+0000","last_peered":"2021-05-25T11:59:52.598621+0000","last_clean":"2021-05-25T11:59:52.598621+0000","last_became_active":"2021-05-25T11:59:52.597224+0000","last_became_peered":"2021-05-25T11:59:52.597224+0000","last_unstale":"2021-05-25T11:59:52.598621+0000","last_undegraded":"2021-05-25T11:59:52.598621+0000","last_fullsized":"2021-05-25T11:59:52.598621+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-05-25T11:59:51.583021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-05-25T11:59:51.583021+0000","last_clean_scrub_stamp":"2021-05-25T11:59:51.583021+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4936,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366904,"statfs":{"total":96636764160,"available":96631709696,"internally_reserved":0,"allocated":131072,"data_stored":38737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40500000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.309}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44600000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35699999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]}]},{"osd":4,"up_from":11,"seq":47244640259,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4868,"kb_used_data":124,"kb_used_omap":0,"kb_used_meta":4736,"kb_avail":94366972,"statfs":{"total":96636764160,"available":96631779328,"internally_reserved":0,"allocated":126976,"data_stored":37784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4849664},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29899999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55100000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47499999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46200000000000002}]}]},{"osd":3,"up_from":11,"seq":47244640259,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4936,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366904,"statfs":{"total":96636764160,"available":96631709696,"internally_reserved":0,"allocated":131072,"data_stored":38737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64700000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51200000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68799999999999994}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33800000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.377}]}]},{"osd":2,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4936,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366904,"statfs":{"total":96636764160,"available":96631709696,"internally_reserved":0,"allocated":131072,"data_stored":38737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.216}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.22600000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.5}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49099999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51100000000000001}]}]},{"osd":0,"up_from":11,"seq":47244640260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4936,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366904,"statfs":{"total":96636764160,"available":96631709696,"internally_reserved":0,"allocated":131072,"data_stored":38737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28299999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53500000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52400000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73699999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42899999999999999}]}]},{"osd":1,"up_from":11,"seq":47244640261,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4932,"kb_used_data":124,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366908,"statfs":{"total":96636764160,"available":96631713792,"internally_reserved":0,"allocated":126976,"data_stored":37784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.216}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.254}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47899999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66600000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-05-25T12:00:00.538 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-05-25T12:00:00.648 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.645+0000 7f74625f1700 10 monclient: get_monmap_and_config 2021-05-25T12:00:00.648 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.645+0000 7f74625f1700 10 monclient: build_initial_monmap 2021-05-25T12:00:00.648 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.645+0000 7f74625f1700 10 monclient: monmap: 2021-05-25T12:00:00.649 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:00:00.649 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:00.649 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:00:00.647551+0000 2021-05-25T12:00:00.649 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:00:00.647551+0000 2021-05-25T12:00:00.649 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:00:00.650 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:00.650 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:00:00.650 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:00:00.650 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:00:00.651 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:00.661 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.646+0000 7f74625f1700 1 Processor -- start 2021-05-25T12:00:00.661 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.646+0000 7f74625f1700 1 -- start start 2021-05-25T12:00:00.661 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.646+0000 7f74625f1700 10 monclient: init 2021-05-25T12:00:00.662 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.646+0000 7f74625f1700 10 monclient: _reopen_session rank -1 2021-05-25T12:00:00.662 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.646+0000 7f74625f1700 10 monclient: _add_conns ranks=[1,0,2] 2021-05-25T12:00:00.662 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.646+0000 7f74625f1700 10 monclient(hunting): picked mon.noname-c con 0x7f745c128c70 addr v1:172.21.15.181:6789/0 2021-05-25T12:00:00.662 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.646+0000 7f74625f1700 10 monclient(hunting): picked mon.noname-a con 0x7f745c129050 addr v1:172.21.15.102:6789/0 2021-05-25T12:00:00.663 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.646+0000 7f74625f1700 10 monclient(hunting): picked mon.noname-b con 0x7f745c132740 addr v1:172.21.15.102:6790/0 2021-05-25T12:00:00.663 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.646+0000 7f74625f1700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f745c0697c0 con 0x7f745c129050 2021-05-25T12:00:00.663 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.646+0000 7f74625f1700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f745c0960e0 con 0x7f745c128c70 2021-05-25T12:00:00.663 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.646+0000 7f74625f1700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f745c096380 con 0x7f745c132740 2021-05-25T12:00:00.663 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.646+0000 7f74625f1700 10 monclient(hunting): _renew_subs 2021-05-25T12:00:00.664 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.646+0000 7f74625f1700 10 monclient(hunting): authenticate will time out at 801.244840s 2021-05-25T12:00:00.664 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.646+0000 7f7460b8e700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f745c132740 0x7f745c134b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:53952/0 (socket says 172.21.15.102:53952) 2021-05-25T12:00:00.664 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.646+0000 7f745b7fe700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f745c129050 0x7f745c131020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:59072/0 (socket says 172.21.15.102:59072) 2021-05-25T12:00:00.664 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.647+0000 7f7460b8e700 1 -- 172.21.15.102:0/3599718527 learned_addr learned my addr 172.21.15.102:0/3599718527 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:00:00.665 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.647+0000 7f745affd700 1 -- 172.21.15.102:0/3599718527 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 955146248 0 0) 0x7f745c096380 con 0x7f745c132740 2021-05-25T12:00:00.665 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.647+0000 7f745affd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:00.665 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.647+0000 7f745affd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:00.665 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.647+0000 7f745affd700 10 monclient(hunting): my global_id is 4373 2021-05-25T12:00:00.665 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.647+0000 7f745affd700 1 -- 172.21.15.102:0/3599718527 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7440003270 con 0x7f745c132740 2021-05-25T12:00:00.666 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.647+0000 7f745affd700 1 -- 172.21.15.102:0/3599718527 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3191219492 0 0) 0x7f745c0697c0 con 0x7f745c129050 2021-05-25T12:00:00.666 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.647+0000 7f745affd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:00.666 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.647+0000 7f745affd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:00.666 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.647+0000 7f745affd700 10 monclient(hunting): my global_id is 4380 2021-05-25T12:00:00.666 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.647+0000 7f745affd700 1 -- 172.21.15.102:0/3599718527 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f745c096380 con 0x7f745c129050 2021-05-25T12:00:00.667 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.647+0000 7f745affd700 1 -- 172.21.15.102:0/3599718527 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 287967736 0 0) 0x7f7440003270 con 0x7f745c132740 2021-05-25T12:00:00.667 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.647+0000 7f745affd700 1 -- 172.21.15.102:0/3599718527 >> v1:172.21.15.181:6789/0 conn(0x7f745c128c70 legacy=0x7f745c096fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:00.667 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.647+0000 7f745affd700 1 -- 172.21.15.102:0/3599718527 >> v1:172.21.15.102:6789/0 conn(0x7f745c129050 legacy=0x7f745c131020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:00.667 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.647+0000 7f745affd700 10 monclient: _finish_hunting 0 2021-05-25T12:00:00.668 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.647+0000 7f745affd700 1 monclient: found mon.noname-b 2021-05-25T12:00:00.668 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.647+0000 7f745affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:00:00.668 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.647+0000 7f745affd700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T12:00:00.668 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.647+0000 7f745affd700 1 -- 172.21.15.102:0/3599718527 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f745c096620 con 0x7f745c132740 2021-05-25T12:00:00.668 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.647+0000 7f745affd700 10 monclient: _finish_auth 0 2021-05-25T12:00:00.669 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.647+0000 7f745affd700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:00:00.669 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.647+0000 7f74625f1700 5 monclient: authenticate success, global_id 4373 2021-05-25T12:00:00.669 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.647+0000 7f745affd700 1 -- 172.21.15.102:0/3599718527 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f7450001990 con 0x7f745c132740 2021-05-25T12:00:00.669 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.647+0000 7f745affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:00.670 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.647+0000 7f745affd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:00:00.670 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.647+0000 7f745affd700 10 monclient: dump: 2021-05-25T12:00:00.670 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:00.670 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:00.671 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:00.671 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:00.671 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:00.671 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:00.671 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:00.672 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:00.672 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:00.672 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:00.672 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.647+0000 7f74625f1700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:00:00.672 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.647+0000 7f745affd700 1 -- 172.21.15.102:0/3599718527 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7450001bc0 con 0x7f745c132740 2021-05-25T12:00:00.673 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.647+0000 7f745affd700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:00:00.673 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.647+0000 7f745affd700 1 -- 172.21.15.102:0/3599718527 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f7450001f50 con 0x7f745c132740 2021-05-25T12:00:00.673 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.647+0000 7f74625f1700 10 monclient: get_monmap_and_config success 2021-05-25T12:00:00.673 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.647+0000 7f74625f1700 4 set_mon_vals no callback set 2021-05-25T12:00:00.674 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.647+0000 7f745affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:00.674 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.647+0000 7f745affd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:00:00.674 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.647+0000 7f745affd700 10 monclient: dump: 2021-05-25T12:00:00.674 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:00.674 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:00.675 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:00.675 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:00.675 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:00.675 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:00.675 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:00.676 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:00.676 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:00.676 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:00.676 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.648+0000 7f74625f1700 10 monclient: shutdown 2021-05-25T12:00:00.676 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.648+0000 7f74625f1700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:00:00.677 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.648+0000 7f74625f1700 1 -- 172.21.15.102:0/3599718527 >> v1:172.21.15.102:6790/0 conn(0x7f745c132740 legacy=0x7f745c134b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:00.677 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.648+0000 7f74625f1700 1 -- 172.21.15.102:0/3599718527 shutdown_connections 2021-05-25T12:00:00.677 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.648+0000 7f74625f1700 1 -- 172.21.15.102:0/3599718527 >> 172.21.15.102:0/3599718527 conn(0x7f745c062810 msgr2=0x7f745c09aeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:00:00.677 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.648+0000 7f74625f1700 1 -- 172.21.15.102:0/3599718527 shutdown_connections 2021-05-25T12:00:00.678 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.648+0000 7f74625f1700 1 -- 172.21.15.102:0/3599718527 wait complete. 2021-05-25T12:00:00.678 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.648+0000 7f74625f1700 10 monclient: build_initial_monmap 2021-05-25T12:00:00.678 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.648+0000 7f74625f1700 10 monclient: monmap: 2021-05-25T12:00:00.678 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:00:00.679 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:00:00.679 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:00:00.679 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:00:00.679 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:00:00.679 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:00:00.680 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:00:00.680 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:00:00.680 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:00:00.680 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:00.681 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.648+0000 7f74625f1700 1 librados: starting msgr at 2021-05-25T12:00:00.681 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.648+0000 7f74625f1700 1 librados: starting objecter 2021-05-25T12:00:00.681 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.648+0000 7f74625f1700 1 Processor -- start 2021-05-25T12:00:00.681 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.649+0000 7f74625f1700 1 -- start start 2021-05-25T12:00:00.681 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.649+0000 7f74625f1700 1 librados: setting wanted keys 2021-05-25T12:00:00.682 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.649+0000 7f74625f1700 1 librados: calling monclient init 2021-05-25T12:00:00.682 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.649+0000 7f74625f1700 10 monclient: init 2021-05-25T12:00:00.682 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.649+0000 7f74625f1700 10 monclient: _reopen_session rank -1 2021-05-25T12:00:00.682 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.649+0000 7f74625f1700 10 monclient: _add_conns ranks=[2,1,0] 2021-05-25T12:00:00.682 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.649+0000 7f74625f1700 10 monclient(hunting): picked mon.noname-c con 0x7f745c128c70 addr v1:172.21.15.102:6790/0 2021-05-25T12:00:00.683 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.649+0000 7f74625f1700 10 monclient(hunting): picked mon.noname-b con 0x7f745c129050 addr v1:172.21.15.181:6789/0 2021-05-25T12:00:00.683 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.649+0000 7f74625f1700 10 monclient(hunting): picked mon.noname-a con 0x7f745c132740 addr v1:172.21.15.102:6789/0 2021-05-25T12:00:00.683 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.649+0000 7f74625f1700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f745c12e3a0 con 0x7f745c132740 2021-05-25T12:00:00.683 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.649+0000 7f74625f1700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f745c130d60 con 0x7f745c129050 2021-05-25T12:00:00.684 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.649+0000 7f74625f1700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f745c131000 con 0x7f745c128c70 2021-05-25T12:00:00.684 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.649+0000 7f74625f1700 10 monclient(hunting): _renew_subs 2021-05-25T12:00:00.684 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.649+0000 7f74625f1700 10 monclient(hunting): authenticate will time out at 801.247548s 2021-05-25T12:00:00.684 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.649+0000 7f745bfff700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f745c128c70 0x7f745c12d580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:53956/0 (socket says 172.21.15.102:53956) 2021-05-25T12:00:00.684 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.649+0000 7f745bfff700 1 -- 172.21.15.102:0/725852801 learned_addr learned my addr 172.21.15.102:0/725852801 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:00:00.685 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.649+0000 7f7447fff700 10 client.?.objecter ms_handle_connect 0x7f745c128c70 2021-05-25T12:00:00.685 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.649+0000 7f7447fff700 10 client.?.objecter resend_mon_ops 2021-05-25T12:00:00.685 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.649+0000 7f7447fff700 10 client.?.objecter ms_handle_connect 0x7f745c132740 2021-05-25T12:00:00.685 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.649+0000 7f7447fff700 10 client.?.objecter resend_mon_ops 2021-05-25T12:00:00.686 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.649+0000 7f7447fff700 1 -- 172.21.15.102:0/725852801 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1375667871 0 0) 0x7f745c131000 con 0x7f745c128c70 2021-05-25T12:00:00.686 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.649+0000 7f7447fff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:00.686 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.649+0000 7f7447fff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:00.686 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.649+0000 7f7447fff700 10 monclient(hunting): my global_id is 4376 2021-05-25T12:00:00.687 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.649+0000 7f7447fff700 1 -- 172.21.15.102:0/725852801 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f743c004c10 con 0x7f745c128c70 2021-05-25T12:00:00.687 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.649+0000 7f7447fff700 10 client.?.objecter ms_handle_connect 0x7f745c129050 2021-05-25T12:00:00.687 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.649+0000 7f7447fff700 10 client.?.objecter resend_mon_ops 2021-05-25T12:00:00.687 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.649+0000 7f7447fff700 1 -- 172.21.15.102:0/725852801 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3192777294 0 0) 0x7f745c12e3a0 con 0x7f745c132740 2021-05-25T12:00:00.687 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.649+0000 7f7447fff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:00.688 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.649+0000 7f7447fff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:00.688 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.649+0000 7f7447fff700 10 monclient(hunting): my global_id is 4383 2021-05-25T12:00:00.688 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.649+0000 7f7447fff700 1 -- 172.21.15.102:0/725852801 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f745c131000 con 0x7f745c132740 2021-05-25T12:00:00.689 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.649+0000 7f7447fff700 1 -- 172.21.15.102:0/725852801 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3918014471 0 0) 0x7f745c130d60 con 0x7f745c129050 2021-05-25T12:00:00.689 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.649+0000 7f7447fff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:00.689 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.649+0000 7f7447fff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:00.689 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.649+0000 7f7447fff700 10 monclient(hunting): my global_id is 4378 2021-05-25T12:00:00.690 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.649+0000 7f7447fff700 1 -- 172.21.15.102:0/725852801 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f745c12e3a0 con 0x7f745c129050 2021-05-25T12:00:00.690 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.650+0000 7f7447fff700 1 -- 172.21.15.102:0/725852801 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4289111306 0 0) 0x7f743c004c10 con 0x7f745c128c70 2021-05-25T12:00:00.690 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.650+0000 7f7447fff700 1 -- 172.21.15.102:0/725852801 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f745c130d60 con 0x7f745c128c70 2021-05-25T12:00:00.690 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.650+0000 7f7447fff700 1 -- 172.21.15.102:0/725852801 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f744c0019a0 con 0x7f745c128c70 2021-05-25T12:00:00.691 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.650+0000 7f7447fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:00.691 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.650+0000 7f7447fff700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:00:00.691 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.650+0000 7f7447fff700 10 monclient(hunting): dump: 2021-05-25T12:00:00.691 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:00.691 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:00.692 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:00.692 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:00.692 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:00.692 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:00.692 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:00.693 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:00.693 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:00.693 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:00.693 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.650+0000 7f7447fff700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:00:00.693 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.650+0000 7f7447fff700 1 -- 172.21.15.102:0/725852801 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4228854649 0 0) 0x7f745c131000 con 0x7f745c132740 2021-05-25T12:00:00.694 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.650+0000 7f7447fff700 1 -- 172.21.15.102:0/725852801 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f743c004c10 con 0x7f745c132740 2021-05-25T12:00:00.694 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.650+0000 7f7447fff700 1 -- 172.21.15.102:0/725852801 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f7450001ed0 con 0x7f745c132740 2021-05-25T12:00:00.694 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.650+0000 7f7447fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:00.695 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.650+0000 7f7447fff700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2021-05-25T12:00:00.695 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.650+0000 7f7447fff700 10 monclient(hunting): dump: 2021-05-25T12:00:00.695 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:00.695 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:00.696 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:00.696 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:00.696 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:00.696 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:00.697 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:00.697 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:00.698 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:00.698 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:00.698 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.650+0000 7f7447fff700 1 -- 172.21.15.102:0/725852801 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 159742444 0 0) 0x7f745c12e3a0 con 0x7f745c129050 2021-05-25T12:00:00.699 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.650+0000 7f7447fff700 1 -- 172.21.15.102:0/725852801 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f745c131000 con 0x7f745c129050 2021-05-25T12:00:00.699 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.650+0000 7f7447fff700 1 -- 172.21.15.102:0/725852801 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f74480019f0 con 0x7f745c129050 2021-05-25T12:00:00.699 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.650+0000 7f7447fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:00.699 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.650+0000 7f7447fff700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2021-05-25T12:00:00.699 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.650+0000 7f7447fff700 10 monclient(hunting): dump: 2021-05-25T12:00:00.700 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:00.700 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:00.700 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:00.700 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:00.700 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:00.701 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:00.701 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:00.701 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:00.701 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:00.701 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:00.702 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.650+0000 7f7447fff700 1 -- 172.21.15.102:0/725852801 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 154132215 0 0) 0x7f745c130d60 con 0x7f745c128c70 2021-05-25T12:00:00.702 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.650+0000 7f7447fff700 1 -- 172.21.15.102:0/725852801 >> v1:172.21.15.181:6789/0 conn(0x7f745c129050 legacy=0x7f745c12dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:00.702 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.650+0000 7f7447fff700 1 -- 172.21.15.102:0/725852801 >> v1:172.21.15.102:6789/0 conn(0x7f745c132740 legacy=0x7f745c143c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:00.702 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.650+0000 7f7447fff700 10 monclient: _finish_hunting 0 2021-05-25T12:00:00.703 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.650+0000 7f7447fff700 1 monclient: found mon.c 2021-05-25T12:00:00.703 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.650+0000 7f7447fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:00:00.703 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.650+0000 7f7447fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:00:00.703 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.650+0000 7f7447fff700 1 -- 172.21.15.102:0/725852801 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f745c1312a0 con 0x7f745c128c70 2021-05-25T12:00:00.704 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.650+0000 7f7447fff700 10 monclient: _finish_auth 0 2021-05-25T12:00:00.704 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.650+0000 7f7447fff700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:00:00.704 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.650+0000 7f74625f1700 5 monclient: authenticate success, global_id 4376 2021-05-25T12:00:00.704 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.650+0000 7f74625f1700 10 monclient: _renew_subs 2021-05-25T12:00:00.704 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.650+0000 7f74625f1700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:00:00.705 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.650+0000 7f74625f1700 1 -- 172.21.15.102:0/725852801 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f745c12f190 con 0x7f745c128c70 2021-05-25T12:00:00.705 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.650+0000 7f74625f1700 10 client.4376.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:00:00.705 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.650+0000 7f74625f1700 10 monclient: _renew_subs 2021-05-25T12:00:00.705 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.650+0000 7f74625f1700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:00:00.705 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.650+0000 7f74625f1700 1 -- 172.21.15.102:0/725852801 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f745c1263e0 con 0x7f745c128c70 2021-05-25T12:00:00.706 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.650+0000 7f74625f1700 1 librados: init done 2021-05-25T12:00:00.706 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.650+0000 7f7447fff700 1 -- 172.21.15.102:0/725852801 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f744c001ce0 con 0x7f745c128c70 2021-05-25T12:00:00.706 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.650+0000 7f7447fff700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:00:00.706 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.650+0000 7f7447fff700 1 -- 172.21.15.102:0/725852801 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f744c002350 con 0x7f745c128c70 2021-05-25T12:00:00.707 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.650+0000 7f7447fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:00.707 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.650+0000 7f7447fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:00:00.707 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.650+0000 7f745a7fc700 4 set_mon_vals no callback set 2021-05-25T12:00:00.707 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.650+0000 7f7447fff700 10 monclient: dump: 2021-05-25T12:00:00.707 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:00.708 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:00.708 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:00.708 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:00.708 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:00.708 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:00.709 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:00.709 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:00.709 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:00.709 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:00.709 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.651+0000 7f7447fff700 1 -- 172.21.15.102:0/725852801 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f744c002530 con 0x7f745c128c70 2021-05-25T12:00:00.710 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.651+0000 7f74625f1700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:00:00.710 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.651+0000 7f74625f1700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:00:00.710 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.651+0000 7f74625f1700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:00:00.710 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.651+0000 7f74625f1700 1 -- 172.21.15.102:0/725852801 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7434004db0 con 0x7f745c128c70 2021-05-25T12:00:00.710 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.651+0000 7f7447fff700 1 -- 172.21.15.102:0/725852801 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(15..15 src has 1..15) v4 ==== 3549+0+0 (unknown 3948819498 0 0) 0x7f744c03cdc0 con 0x7f745c128c70 2021-05-25T12:00:00.711 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.651+0000 7f7447fff700 10 client.4376.objecter ms_dispatch 0x7f745c001550 osd_map(15..15 src has 1..15) v4 2021-05-25T12:00:00.711 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.651+0000 7f7447fff700 3 client.4376.objecter handle_osd_map got epochs [15,15] > 0 2021-05-25T12:00:00.711 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.651+0000 7f7447fff700 3 client.4376.objecter handle_osd_map decoding full epoch 15 2021-05-25T12:00:00.711 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.651+0000 7f7447fff700 20 client.4376.objecter dump_active .. 0 homeless 2021-05-25T12:00:00.712 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.651+0000 7f745b7fe700 10 monclient: get_auth_request con 0x7f743c033520 auth_method 0 2021-05-25T12:00:00.712 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.651+0000 7f7447fff700 10 client.4376.objecter ms_handle_connect 0x7f743c033520 2021-05-25T12:00:00.712 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.655+0000 7f7447fff700 1 -- 172.21.15.102:0/725852801 <== mon.2 v1:172.21.15.102:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7f744c012d10 con 0x7f745c128c70 2021-05-25T12:00:00.712 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.655+0000 7f7447fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:00:00.712 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.655+0000 7f7447fff700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:00:00.802 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.800+0000 7f74625f1700 1 -- 172.21.15.102:0/725852801 --> v1:172.21.15.181:6812/29582 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7434000b90 con 0x7f743c033520 2021-05-25T12:00:00.804 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.802+0000 7f7447fff700 1 -- 172.21.15.102:0/725852801 <== mgr.4105 v1:172.21.15.181:6812/29582 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+38639 (unknown 2965378022 0 3736317541) 0x7f7434000b90 con 0x7f743c033520 2021-05-25T12:00:00.805 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-05-25T12:00:00.807 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.803+0000 7f74625f1700 10 librados: watch_flush enter 2021-05-25T12:00:00.807 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.803+0000 7f74625f1700 10 librados: watch_flush exit 2021-05-25T12:00:00.807 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.803+0000 7f74625f1700 20 client.4376.objecter shutdown clearing up homeless session... 2021-05-25T12:00:00.807 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.803+0000 7f74625f1700 10 client.4376.objecter successfully canceled tick 2021-05-25T12:00:00.808 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.803+0000 7f74625f1700 1 -- 172.21.15.102:0/725852801 >> v1:172.21.15.181:6812/29582 conn(0x7f743c033520 legacy=0x7f743c0359e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:00.808 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.803+0000 7f74625f1700 10 monclient: shutdown 2021-05-25T12:00:00.808 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.803+0000 7f74625f1700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:00:00.808 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.803+0000 7f74625f1700 1 -- 172.21.15.102:0/725852801 >> v1:172.21.15.102:6790/0 conn(0x7f745c128c70 legacy=0x7f745c12d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:00.809 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.803+0000 7f74625f1700 1 -- 172.21.15.102:0/725852801 shutdown_connections 2021-05-25T12:00:00.809 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.803+0000 7f74625f1700 1 -- 172.21.15.102:0/725852801 >> 172.21.15.102:0/725852801 conn(0x7f745c062810 msgr2=0x7f745c127380 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:00:00.809 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.803+0000 7f74625f1700 1 -- 172.21.15.102:0/725852801 shutdown_connections 2021-05-25T12:00:00.809 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.803+0000 7f74625f1700 1 -- 172.21.15.102:0/725852801 wait complete. 2021-05-25T12:00:00.809 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.803+0000 7f74625f1700 1 librados: shutdown 2021-05-25T12:00:00.810 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2021-05-25T12:00:00.817 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2021-05-25T11:59:59.556595+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":11,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":29544,"kb_used_data":760,"kb_used_omap":0,"kb_used_meta":28736,"kb_avail":566201496,"statfs":{"total":579820584960,"available":579790331904,"internally_reserved":0,"allocated":778240,"data_stored":230516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29425664},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2021-05-25T11:59:52.780051+0000","last_change":"2021-05-25T11:59:52.779532+0000","last_active":"2021-05-25T11:59:52.780051+0000","last_peered":"2021-05-25T11:59:52.780051+0000","last_clean":"2021-05-25T11:59:52.780051+0000","last_became_active":"2021-05-25T11:59:52.779240+0000","last_became_peered":"2021-05-25T11:59:52.779240+0000","last_unstale":"2021-05-25T11:59:52.780051+0000","last_undegraded":"2021-05-25T11:59:52.780051+0000","last_fullsized":"2021-05-25T11:59:52.780051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-05-25T11:59:51.583021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-05-25T11:59:51.583021+0000","last_clean_scrub_stamp":"2021-05-25T11:59:51.583021+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2021-05-25T11:59:52.600778+0000","last_change":"2021-05-25T11:59:52.599864+0000","last_active":"2021-05-25T11:59:52.600778+0000","last_peered":"2021-05-25T11:59:52.600778+0000","last_clean":"2021-05-25T11:59:52.600778+0000","last_became_active":"2021-05-25T11:59:52.599550+0000","last_became_peered":"2021-05-25T11:59:52.599550+0000","last_unstale":"2021-05-25T11:59:52.600778+0000","last_undegraded":"2021-05-25T11:59:52.600778+0000","last_fullsized":"2021-05-25T11:59:52.600778+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-05-25T11:59:51.583021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-05-25T11:59:51.583021+0000","last_clean_scrub_stamp":"2021-05-25T11:59:51.583021+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2021-05-25T11:59:52.599789+0000","last_change":"2021-05-25T11:59:52.599209+0000","last_active":"2021-05-25T11:59:52.599789+0000","last_peered":"2021-05-25T11:59:52.599789+0000","last_clean":"2021-05-25T11:59:52.599789+0000","last_became_active":"2021-05-25T11:59:52.598818+0000","last_became_peered":"2021-05-25T11:59:52.598818+0000","last_unstale":"2021-05-25T11:59:52.599789+0000","last_undegraded":"2021-05-25T11:59:52.599789+0000","last_fullsized":"2021-05-25T11:59:52.599789+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-05-25T11:59:51.583021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-05-25T11:59:51.583021+0000","last_clean_scrub_stamp":"2021-05-25T11:59:51.583021+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2021-05-25T11:59:52.599030+0000","last_change":"2021-05-25T11:59:52.596805+0000","last_active":"2021-05-25T11:59:52.599030+0000","last_peered":"2021-05-25T11:59:52.599030+0000","last_clean":"2021-05-25T11:59:52.599030+0000","last_became_active":"2021-05-25T11:59:52.596538+0000","last_became_peered":"2021-05-25T11:59:52.596538+0000","last_unstale":"2021-05-25T11:59:52.599030+0000","last_undegraded":"2021-05-25T11:59:52.599030+0000","last_fullsized":"2021-05-25T11:59:52.599030+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-05-25T11:59:51.583021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-05-25T11:59:51.583021+0000","last_clean_scrub_stamp":"2021-05-25T11:59:51.583021+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"10","reported_epoch":"14","state":"active+clean","last_fresh":"2021-05-25T11:59:52.593134+0000","last_change":"2021-05-25T11:59:51.592476+0000","last_active":"2021-05-25T11:59:52.593134+0000","last_peered":"2021-05-25T11:59:52.593134+0000","last_clean":"2021-05-25T11:59:52.593134+0000","last_became_active":"2021-05-25T11:59:51.592157+0000","last_became_peered":"2021-05-25T11:59:51.592157+0000","last_unstale":"2021-05-25T11:59:52.593134+0000","last_undegraded":"2021-05-25T11:59:52.593134+0000","last_fullsized":"2021-05-25T11:59:52.593134+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-05-25T11:59:50.578675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-05-25T11:59:50.578675+0000","last_clean_scrub_stamp":"2021-05-25T11:59:50.578675+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2021-05-25T11:59:52.607326+0000","last_change":"2021-05-25T11:59:52.602323+0000","last_active":"2021-05-25T11:59:52.607326+0000","last_peered":"2021-05-25T11:59:52.607326+0000","last_clean":"2021-05-25T11:59:52.607326+0000","last_became_active":"2021-05-25T11:59:52.602047+0000","last_became_peered":"2021-05-25T11:59:52.602047+0000","last_unstale":"2021-05-25T11:59:52.607326+0000","last_undegraded":"2021-05-25T11:59:52.607326+0000","last_fullsized":"2021-05-25T11:59:52.607326+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-05-25T11:59:51.583021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-05-25T11:59:51.583021+0000","last_clean_scrub_stamp":"2021-05-25T11:59:51.583021+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2021-05-25T11:59:52.600845+0000","last_change":"2021-05-25T11:59:52.598411+0000","last_active":"2021-05-25T11:59:52.600845+0000","last_peered":"2021-05-25T11:59:52.600845+0000","last_clean":"2021-05-25T11:59:52.600845+0000","last_became_active":"2021-05-25T11:59:52.598090+0000","last_became_peered":"2021-05-25T11:59:52.598090+0000","last_unstale":"2021-05-25T11:59:52.600845+0000","last_undegraded":"2021-05-25T11:59:52.600845+0000","last_fullsized":"2021-05-25T11:59:52.600845+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-05-25T11:59:51.583021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-05-25T11:59:51.583021+0000","last_clean_scrub_stamp":"2021-05-25T11:59:51.583021+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2021-05-25T11:59:52.607316+0000","last_change":"2021-05-25T11:59:52.602212+0000","last_active":"2021-05-25T11:59:52.607316+0000","last_peered":"2021-05-25T11:59:52.607316+0000","last_clean":"2021-05-25T11:59:52.607316+0000","last_became_active":"2021-05-25T11:59:52.601888+0000","last_became_peered":"2021-05-25T11:59:52.601888+0000","last_unstale":"2021-05-25T11:59:52.607316+0000","last_undegraded":"2021-05-25T11:59:52.607316+0000","last_fullsized":"2021-05-25T11:59:52.607316+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-05-25T11:59:51.583021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-05-25T11:59:51.583021+0000","last_clean_scrub_stamp":"2021-05-25T11:59:51.583021+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2021-05-25T11:59:52.598621+0000","last_change":"2021-05-25T11:59:52.597495+0000","last_active":"2021-05-25T11:59:52.598621+0000","last_peered":"2021-05-25T11:59:52.598621+0000","last_clean":"2021-05-25T11:59:52.598621+0000","last_became_active":"2021-05-25T11:59:52.597224+0000","last_became_peered":"2021-05-25T11:59:52.597224+0000","last_unstale":"2021-05-25T11:59:52.598621+0000","last_undegraded":"2021-05-25T11:59:52.598621+0000","last_fullsized":"2021-05-25T11:59:52.598621+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-05-25T11:59:51.583021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-05-25T11:59:51.583021+0000","last_clean_scrub_stamp":"2021-05-25T11:59:51.583021+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4936,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366904,"statfs":{"total":96636764160,"available":96631709696,"internally_reserved":0,"allocated":131072,"data_stored":38737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40500000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.309}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44600000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35699999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]}]},{"osd":4,"up_from":11,"seq":47244640259,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4868,"kb_used_data":124,"kb_used_omap":0,"kb_used_meta":4736,"kb_avail":94366972,"statfs":{"total":96636764160,"available":96631779328,"internally_reserved":0,"allocated":126976,"data_stored":37784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4849664},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29899999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55100000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47499999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46200000000000002}]}]},{"osd":3,"up_from":11,"seq":47244640259,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4936,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366904,"statfs":{"total":96636764160,"available":96631709696,"internally_reserved":0,"allocated":131072,"data_stored":38737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64700000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51200000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68799999999999994}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33800000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.377}]}]},{"osd":2,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4936,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366904,"statfs":{"total":96636764160,"available":96631709696,"internally_reserved":0,"allocated":131072,"data_stored":38737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.216}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.22600000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.5}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49099999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51100000000000001}]}]},{"osd":0,"up_from":11,"seq":47244640260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4936,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366904,"statfs":{"total":96636764160,"available":96631709696,"internally_reserved":0,"allocated":131072,"data_stored":38737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28299999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53500000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52400000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73699999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42899999999999999}]}]},{"osd":1,"up_from":11,"seq":47244640261,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4932,"kb_used_data":124,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366908,"statfs":{"total":96636764160,"available":96631713792,"internally_reserved":0,"allocated":126976,"data_stored":37784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.216}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.254}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47899999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66600000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-05-25T12:00:00.818 INFO:tasks.mon_thrash.ceph_manager:clean! 2021-05-25T12:00:00.819 DEBUG:tasks.mon_thrash:Yielding 2021-05-25T12:00:00.819 INFO:teuthology.run_tasks:Running task workunit... 2021-05-25T12:00:00.841 INFO:tasks.workunit:Pulling workunits from ref a16ab1caacd8b41a7fe74bc40a10ec74ee8f5802 2021-05-25T12:00:00.843 INFO:tasks.workunit:Making a separate scratch dir for every client... 2021-05-25T12:00:00.843 DEBUG:teuthology.orchestra.run.smithi181:> stat -- /home/ubuntu/cephtest/mnt.0 2021-05-25T12:00:00.844 INFO:tasks.mon_thrash.mon_thrasher:start thrashing 2021-05-25T12:00:00.844 INFO:tasks.mon_thrash.mon_thrasher:seed: 1621944000, revive delay: 20.0, thrash delay: 1.0 thrash many: True, maintain quorum: True store thrash: False, probability: 50 freeze mon: prob 10.0 duration 20.0 2021-05-25T12:00:00.844 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2021-05-25T12:00:00.845 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2021-05-25T12:00:00.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-05-25T12:00:00.863 INFO:teuthology.orchestra.run.smithi181.stderr:stat: cannot stat '/home/ubuntu/cephtest/mnt.0': No such file or directory 2021-05-25T12:00:00.863 DEBUG:teuthology.orchestra.run.smithi181:> mkdir -- /home/ubuntu/cephtest/mnt.0 2021-05-25T12:00:00.920 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2021-05-25T12:00:00.920 DEBUG:teuthology.orchestra.run.smithi181:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2021-05-25T12:00:00.952 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.950+0000 7f76f1270700 10 monclient: get_monmap_and_config 2021-05-25T12:00:00.952 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.950+0000 7f76f1270700 10 monclient: build_initial_monmap 2021-05-25T12:00:00.953 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.950+0000 7f76f1270700 10 monclient: monmap: 2021-05-25T12:00:00.953 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:00:00.953 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:00.953 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:00:00.952122+0000 2021-05-25T12:00:00.954 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:00:00.952122+0000 2021-05-25T12:00:00.954 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:00:00.954 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:00.954 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:00:00.954 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:00:00.955 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:00:00.955 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:00.965 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.950+0000 7f76f1270700 1 Processor -- start 2021-05-25T12:00:00.965 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.950+0000 7f76f1270700 1 -- start start 2021-05-25T12:00:00.966 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.950+0000 7f76f1270700 10 monclient: init 2021-05-25T12:00:00.966 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.951+0000 7f76f1270700 10 monclient: _reopen_session rank -1 2021-05-25T12:00:00.966 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.951+0000 7f76f1270700 10 monclient: _add_conns ranks=[0,1,2] 2021-05-25T12:00:00.966 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.951+0000 7f76f1270700 10 monclient(hunting): picked mon.noname-a con 0x7f76ec09a5e0 addr v1:172.21.15.102:6789/0 2021-05-25T12:00:00.967 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.951+0000 7f76f1270700 10 monclient(hunting): picked mon.noname-c con 0x7f76ec063dc0 addr v1:172.21.15.181:6789/0 2021-05-25T12:00:00.967 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.951+0000 7f76f1270700 10 monclient(hunting): picked mon.noname-b con 0x7f76ec1325c0 addr v1:172.21.15.102:6790/0 2021-05-25T12:00:00.967 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.951+0000 7f76f1270700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76ec099fe0 con 0x7f76ec09a5e0 2021-05-25T12:00:00.967 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.951+0000 7f76f1270700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76ec09a1e0 con 0x7f76ec063dc0 2021-05-25T12:00:00.968 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.951+0000 7f76f1270700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76ec09b400 con 0x7f76ec1325c0 2021-05-25T12:00:00.968 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.951+0000 7f76f1270700 10 monclient(hunting): _renew_subs 2021-05-25T12:00:00.968 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.951+0000 7f76f1270700 10 monclient(hunting): authenticate will time out at 801.549272s 2021-05-25T12:00:00.968 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.951+0000 7f76eaffd700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f76ec09a5e0 0x7f76ec0636b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:59086/0 (socket says 172.21.15.102:59086) 2021-05-25T12:00:00.968 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.951+0000 7f76eaffd700 1 -- 172.21.15.102:0/2673980019 learned_addr learned my addr 172.21.15.102:0/2673980019 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:00:00.969 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.951+0000 7f76eb7fe700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f76ec1325c0 0x7f76ec1349b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:53968/0 (socket says 172.21.15.102:53968) 2021-05-25T12:00:00.969 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.951+0000 7f76e9ffb700 1 -- 172.21.15.102:0/2673980019 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3299327755 0 0) 0x7f76ec099fe0 con 0x7f76ec09a5e0 2021-05-25T12:00:00.969 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.951+0000 7f76e9ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:00.969 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.951+0000 7f76e9ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:00.970 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.951+0000 7f76e9ffb700 10 monclient(hunting): my global_id is 4386 2021-05-25T12:00:00.970 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.951+0000 7f76e9ffb700 1 -- 172.21.15.102:0/2673980019 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76d0003270 con 0x7f76ec09a5e0 2021-05-25T12:00:00.970 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.951+0000 7f76e9ffb700 1 -- 172.21.15.102:0/2673980019 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4060089697 0 0) 0x7f76ec09b400 con 0x7f76ec1325c0 2021-05-25T12:00:00.970 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.951+0000 7f76e9ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:00.970 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.951+0000 7f76e9ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:00.971 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.951+0000 7f76e9ffb700 10 monclient(hunting): my global_id is 4379 2021-05-25T12:00:00.971 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.951+0000 7f76e9ffb700 1 -- 172.21.15.102:0/2673980019 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76ec099fe0 con 0x7f76ec1325c0 2021-05-25T12:00:00.971 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.951+0000 7f76e9ffb700 1 -- 172.21.15.102:0/2673980019 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3114583936 0 0) 0x7f76d0003270 con 0x7f76ec09a5e0 2021-05-25T12:00:00.971 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.951+0000 7f76e9ffb700 1 -- 172.21.15.102:0/2673980019 >> v1:172.21.15.102:6790/0 conn(0x7f76ec1325c0 legacy=0x7f76ec1349b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:00.972 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.951+0000 7f76e9ffb700 1 -- 172.21.15.102:0/2673980019 >> v1:172.21.15.181:6789/0 conn(0x7f76ec063dc0 legacy=0x7f76ec064240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:00.972 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.951+0000 7f76e9ffb700 10 monclient: _finish_hunting 0 2021-05-25T12:00:00.972 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.951+0000 7f76e9ffb700 1 monclient: found mon.noname-a 2021-05-25T12:00:00.972 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.951+0000 7f76e9ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:00:00.972 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.951+0000 7f76e9ffb700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T12:00:00.973 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.951+0000 7f76e9ffb700 1 -- 172.21.15.102:0/2673980019 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76ec0963f0 con 0x7f76ec09a5e0 2021-05-25T12:00:00.973 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.951+0000 7f76e9ffb700 10 monclient: _finish_auth 0 2021-05-25T12:00:00.973 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.951+0000 7f76e9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:00:00.973 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.951+0000 7f76f1270700 5 monclient: authenticate success, global_id 4386 2021-05-25T12:00:00.974 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.951+0000 7f76f1270700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:00:00.974 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.951+0000 7f76e9ffb700 1 -- 172.21.15.102:0/2673980019 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f76e0001610 con 0x7f76ec09a5e0 2021-05-25T12:00:00.974 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.951+0000 7f76e9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:00.974 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.951+0000 7f76e9ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:00:00.974 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.951+0000 7f76e9ffb700 10 monclient: dump: 2021-05-25T12:00:00.975 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:00.975 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:00.975 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:00.975 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:00.976 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:00.976 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:00.976 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:00.976 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:00.976 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:00.977 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:00.977 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.951+0000 7f76f1270700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:00:00.977 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.952+0000 7f76e9ffb700 1 -- 172.21.15.102:0/2673980019 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f76e00017c0 con 0x7f76ec09a5e0 2021-05-25T12:00:00.977 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.952+0000 7f76e9ffb700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:00:00.977 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.952+0000 7f76e9ffb700 1 -- 172.21.15.102:0/2673980019 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f76e0001b50 con 0x7f76ec09a5e0 2021-05-25T12:00:00.978 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.952+0000 7f76f1270700 10 monclient: get_monmap_and_config success 2021-05-25T12:00:00.978 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.952+0000 7f76f1270700 4 set_mon_vals no callback set 2021-05-25T12:00:00.978 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.952+0000 7f76e9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:00.978 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.952+0000 7f76e9ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:00:00.979 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.952+0000 7f76e9ffb700 10 monclient: dump: 2021-05-25T12:00:00.979 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:00.979 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:00.979 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:00.980 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:00.980 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:00.980 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:00.980 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:00.980 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:00.981 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:00.981 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:00.981 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.952+0000 7f76f1270700 10 monclient: shutdown 2021-05-25T12:00:00.981 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.952+0000 7f76f1270700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:00:00.981 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.952+0000 7f76f1270700 1 -- 172.21.15.102:0/2673980019 >> v1:172.21.15.102:6789/0 conn(0x7f76ec09a5e0 legacy=0x7f76ec0636b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:00.982 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.952+0000 7f76f1270700 1 -- 172.21.15.102:0/2673980019 shutdown_connections 2021-05-25T12:00:00.982 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.952+0000 7f76f1270700 1 -- 172.21.15.102:0/2673980019 >> 172.21.15.102:0/2673980019 conn(0x7f76ec09bc30 msgr2=0x7f76ec09c040 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:00:00.982 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.952+0000 7f76f1270700 1 -- 172.21.15.102:0/2673980019 shutdown_connections 2021-05-25T12:00:00.982 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.952+0000 7f76f1270700 1 -- 172.21.15.102:0/2673980019 wait complete. 2021-05-25T12:00:00.982 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.953+0000 7f76f1270700 10 monclient: build_initial_monmap 2021-05-25T12:00:00.983 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.953+0000 7f76f1270700 10 monclient: monmap: 2021-05-25T12:00:00.983 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:00:00.983 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:00:00.983 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:00:00.983 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:00:00.984 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:00:00.984 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:00:00.984 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:00:00.984 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:00:00.984 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:00:00.985 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:00.985 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.953+0000 7f76f1270700 1 librados: starting msgr at 2021-05-25T12:00:00.985 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.953+0000 7f76f1270700 1 librados: starting objecter 2021-05-25T12:00:00.985 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.953+0000 7f76f1270700 1 Processor -- start 2021-05-25T12:00:00.985 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.953+0000 7f76f1270700 1 -- start start 2021-05-25T12:00:00.986 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.953+0000 7f76f1270700 1 librados: setting wanted keys 2021-05-25T12:00:00.986 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.953+0000 7f76f1270700 1 librados: calling monclient init 2021-05-25T12:00:00.986 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.953+0000 7f76f1270700 10 monclient: init 2021-05-25T12:00:00.986 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.953+0000 7f76f1270700 10 monclient: _reopen_session rank -1 2021-05-25T12:00:00.986 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.953+0000 7f76f1270700 10 monclient: _add_conns ranks=[2,1,0] 2021-05-25T12:00:00.987 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.953+0000 7f76f1270700 10 monclient(hunting): picked mon.noname-c con 0x7f76ec063dc0 addr v1:172.21.15.102:6790/0 2021-05-25T12:00:00.987 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.953+0000 7f76f1270700 10 monclient(hunting): picked mon.noname-b con 0x7f76ec09a5e0 addr v1:172.21.15.181:6789/0 2021-05-25T12:00:00.987 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.953+0000 7f76f1270700 10 monclient(hunting): picked mon.noname-a con 0x7f76ec1325c0 addr v1:172.21.15.102:6789/0 2021-05-25T12:00:00.987 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.953+0000 7f76f1270700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76ec11c370 con 0x7f76ec1325c0 2021-05-25T12:00:00.988 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.953+0000 7f76f1270700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76ec11c610 con 0x7f76ec09a5e0 2021-05-25T12:00:00.988 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.953+0000 7f76f1270700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76ec11c8b0 con 0x7f76ec063dc0 2021-05-25T12:00:00.988 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.953+0000 7f76f1270700 10 monclient(hunting): _renew_subs 2021-05-25T12:00:00.988 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.953+0000 7f76f1270700 10 monclient(hunting): authenticate will time out at 801.551949s 2021-05-25T12:00:00.988 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.954+0000 7f76eaffd700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f76ec063dc0 0x7f76ec118c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:53970/0 (socket says 172.21.15.102:53970) 2021-05-25T12:00:00.989 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.954+0000 7f76eaffd700 1 -- 172.21.15.102:0/474811993 learned_addr learned my addr 172.21.15.102:0/474811993 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:00:00.989 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.954+0000 7f76eb7fe700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f76ec1325c0 0x7f76ec143a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:59096/0 (socket says 172.21.15.102:59096) 2021-05-25T12:00:00.989 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.954+0000 7f76d6ffd700 10 client.?.objecter ms_handle_connect 0x7f76ec063dc0 2021-05-25T12:00:00.989 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.954+0000 7f76d6ffd700 10 client.?.objecter resend_mon_ops 2021-05-25T12:00:00.990 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.954+0000 7f76d6ffd700 10 client.?.objecter ms_handle_connect 0x7f76ec1325c0 2021-05-25T12:00:00.990 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.954+0000 7f76d6ffd700 10 client.?.objecter resend_mon_ops 2021-05-25T12:00:00.990 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.954+0000 7f76d6ffd700 1 -- 172.21.15.102:0/474811993 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1954956940 0 0) 0x7f76ec11c8b0 con 0x7f76ec063dc0 2021-05-25T12:00:00.990 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.954+0000 7f76d6ffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:00.991 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.954+0000 7f76d6ffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:00.991 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.954+0000 7f76d6ffd700 10 monclient(hunting): my global_id is 4382 2021-05-25T12:00:00.991 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.954+0000 7f76d6ffd700 1 -- 172.21.15.102:0/474811993 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76cc004c10 con 0x7f76ec063dc0 2021-05-25T12:00:00.991 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.954+0000 7f76d6ffd700 10 client.?.objecter ms_handle_connect 0x7f76ec09a5e0 2021-05-25T12:00:00.992 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.954+0000 7f76d6ffd700 10 client.?.objecter resend_mon_ops 2021-05-25T12:00:00.992 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.954+0000 7f76d6ffd700 1 -- 172.21.15.102:0/474811993 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 895722570 0 0) 0x7f76ec11c370 con 0x7f76ec1325c0 2021-05-25T12:00:00.992 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.954+0000 7f76d6ffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:00.992 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.954+0000 7f76d6ffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:00.992 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.954+0000 7f76d6ffd700 10 monclient(hunting): my global_id is 4389 2021-05-25T12:00:00.993 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.954+0000 7f76d6ffd700 1 -- 172.21.15.102:0/474811993 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76ec11c8b0 con 0x7f76ec1325c0 2021-05-25T12:00:00.993 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.954+0000 7f76d6ffd700 1 -- 172.21.15.102:0/474811993 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2800602245 0 0) 0x7f76ec11c610 con 0x7f76ec09a5e0 2021-05-25T12:00:00.993 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.954+0000 7f76d6ffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:00.993 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.954+0000 7f76d6ffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:00.993 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.954+0000 7f76d6ffd700 10 monclient(hunting): my global_id is 4384 2021-05-25T12:00:00.994 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.954+0000 7f76d6ffd700 1 -- 172.21.15.102:0/474811993 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76ec11c370 con 0x7f76ec09a5e0 2021-05-25T12:00:00.994 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.954+0000 7f76d6ffd700 1 -- 172.21.15.102:0/474811993 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 474690286 0 0) 0x7f76cc004c10 con 0x7f76ec063dc0 2021-05-25T12:00:00.994 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.954+0000 7f76d6ffd700 1 -- 172.21.15.102:0/474811993 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f76ec11c610 con 0x7f76ec063dc0 2021-05-25T12:00:00.994 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.954+0000 7f76d6ffd700 1 -- 172.21.15.102:0/474811993 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f76e00015d0 con 0x7f76ec063dc0 2021-05-25T12:00:00.995 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.954+0000 7f76d6ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:00.995 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.954+0000 7f76d6ffd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:00:00.995 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.954+0000 7f76d6ffd700 10 monclient(hunting): dump: 2021-05-25T12:00:00.995 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:00.995 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:00.996 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:00.996 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:00.996 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:00.996 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:00.996 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:00.996 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:00.997 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:00.997 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:00.997 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.954+0000 7f76d6ffd700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:00:00.997 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.954+0000 7f76d6ffd700 1 -- 172.21.15.102:0/474811993 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 53606602 0 0) 0x7f76ec11c8b0 con 0x7f76ec1325c0 2021-05-25T12:00:00.998 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.954+0000 7f76d6ffd700 1 -- 172.21.15.102:0/474811993 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f76cc004c10 con 0x7f76ec1325c0 2021-05-25T12:00:00.998 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.954+0000 7f76d6ffd700 1 -- 172.21.15.102:0/474811993 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f76dc001c10 con 0x7f76ec1325c0 2021-05-25T12:00:00.998 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.954+0000 7f76d6ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:00.998 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.954+0000 7f76d6ffd700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2021-05-25T12:00:00.999 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.954+0000 7f76d6ffd700 10 monclient(hunting): dump: 2021-05-25T12:00:00.999 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:00.999 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:00.999 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:00.999 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:01.000 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:01.000 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:01.000 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:01.000 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:01.000 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:01.001 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:01.001 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.954+0000 7f76d6ffd700 1 -- 172.21.15.102:0/474811993 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 913139930 0 0) 0x7f76ec11c370 con 0x7f76ec09a5e0 2021-05-25T12:00:01.001 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.954+0000 7f76d6ffd700 1 -- 172.21.15.102:0/474811993 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f76ec11c8b0 con 0x7f76ec09a5e0 2021-05-25T12:00:01.001 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.954+0000 7f76d6ffd700 1 -- 172.21.15.102:0/474811993 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1906324166 0 0) 0x7f76ec11c610 con 0x7f76ec063dc0 2021-05-25T12:00:01.001 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.954+0000 7f76d6ffd700 1 -- 172.21.15.102:0/474811993 >> v1:172.21.15.181:6789/0 conn(0x7f76ec09a5e0 legacy=0x7f76ec119340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:01.002 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.954+0000 7f76d6ffd700 1 -- 172.21.15.102:0/474811993 >> v1:172.21.15.102:6789/0 conn(0x7f76ec1325c0 legacy=0x7f76ec143a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:01.002 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.954+0000 7f76d6ffd700 10 monclient: _finish_hunting 0 2021-05-25T12:00:01.002 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.954+0000 7f76d6ffd700 1 monclient: found mon.c 2021-05-25T12:00:01.002 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.954+0000 7f76d6ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:00:01.003 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.954+0000 7f76d6ffd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:00:01.003 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.954+0000 7f76d6ffd700 1 -- 172.21.15.102:0/474811993 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76ec11a7b0 con 0x7f76ec063dc0 2021-05-25T12:00:01.003 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.954+0000 7f76d6ffd700 10 monclient: _finish_auth 0 2021-05-25T12:00:01.003 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.954+0000 7f76d6ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:00:01.004 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.954+0000 7f76f1270700 5 monclient: authenticate success, global_id 4382 2021-05-25T12:00:01.004 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.954+0000 7f76f1270700 10 monclient: _renew_subs 2021-05-25T12:00:01.004 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.954+0000 7f76f1270700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:00:01.004 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.954+0000 7f76f1270700 1 -- 172.21.15.102:0/474811993 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76ec1322a0 con 0x7f76ec063dc0 2021-05-25T12:00:01.005 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.954+0000 7f76f1270700 10 client.4382.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:00:01.005 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.954+0000 7f76f1270700 10 monclient: _renew_subs 2021-05-25T12:00:01.005 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.954+0000 7f76f1270700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:00:01.005 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.954+0000 7f76f1270700 1 -- 172.21.15.102:0/474811993 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f76ec11ce30 con 0x7f76ec063dc0 2021-05-25T12:00:01.005 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.954+0000 7f76f1270700 1 librados: init done 2021-05-25T12:00:01.006 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.954+0000 7f76d6ffd700 1 -- 172.21.15.102:0/474811993 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f76e00049b0 con 0x7f76ec063dc0 2021-05-25T12:00:01.006 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.954+0000 7f76d6ffd700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:00:01.006 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.954+0000 7f76d6ffd700 1 -- 172.21.15.102:0/474811993 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f76e0004d40 con 0x7f76ec063dc0 2021-05-25T12:00:01.006 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.954+0000 7f76d6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:01.007 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.954+0000 7f76d6ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:00:01.007 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.954+0000 7f76e97fa700 4 set_mon_vals no callback set 2021-05-25T12:00:01.007 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.954+0000 7f76d6ffd700 10 monclient: dump: 2021-05-25T12:00:01.007 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:01.007 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:01.008 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:01.008 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:01.008 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:01.008 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:01.009 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:01.009 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:01.009 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:01.009 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:01.010 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.955+0000 7f76d6ffd700 1 -- 172.21.15.102:0/474811993 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f76e0004f20 con 0x7f76ec063dc0 2021-05-25T12:00:01.010 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.955+0000 7f76f1270700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:00:01.010 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.955+0000 7f76f1270700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:00:01.010 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.955+0000 7f76f1270700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:00:01.010 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.955+0000 7f76f1270700 1 -- 172.21.15.102:0/474811993 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76c4004db0 con 0x7f76ec063dc0 2021-05-25T12:00:01.011 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.955+0000 7f76d6ffd700 1 -- 172.21.15.102:0/474811993 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(15..15 src has 1..15) v4 ==== 3549+0+0 (unknown 3948819498 0 0) 0x7f76e003d1a0 con 0x7f76ec063dc0 2021-05-25T12:00:01.011 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.955+0000 7f76d6ffd700 10 client.4382.objecter ms_dispatch 0x7f76ec001550 osd_map(15..15 src has 1..15) v4 2021-05-25T12:00:01.011 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.955+0000 7f76d6ffd700 3 client.4382.objecter handle_osd_map got epochs [15,15] > 0 2021-05-25T12:00:01.011 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.955+0000 7f76d6ffd700 3 client.4382.objecter handle_osd_map decoding full epoch 15 2021-05-25T12:00:01.011 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.955+0000 7f76d6ffd700 20 client.4382.objecter dump_active .. 0 homeless 2021-05-25T12:00:01.012 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.955+0000 7f76ea7fc700 10 monclient: get_auth_request con 0x7f76cc0334a0 auth_method 0 2021-05-25T12:00:01.012 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.956+0000 7f76d6ffd700 10 client.4382.objecter ms_handle_connect 0x7f76cc0334a0 2021-05-25T12:00:01.012 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.959+0000 7f76d6ffd700 1 -- 172.21.15.102:0/474811993 <== mon.2 v1:172.21.15.102:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7f76e00130d0 con 0x7f76ec063dc0 2021-05-25T12:00:01.012 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.959+0000 7f76d6ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:00:01.012 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:00.959+0000 7f76d6ffd700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:00:01.014 INFO:tasks.workunit:timeout=3h 2021-05-25T12:00:01.014 INFO:tasks.workunit:cleanup=True 2021-05-25T12:00:01.015 DEBUG:teuthology.orchestra.run.smithi181:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout a16ab1caacd8b41a7fe74bc40a10ec74ee8f5802 2021-05-25T12:00:01.071 INFO:tasks.workunit.client.0.smithi181.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2021-05-25T12:00:01.170 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.167+0000 7f76f1270700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2021-05-25T12:00:01.171 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.168+0000 7f76f1270700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2021-05-25T12:00:01.171 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.168+0000 7f76f1270700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:00:01.171 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.168+0000 7f76f1270700 1 -- 172.21.15.102:0/474811993 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f76c40055e0 con 0x7f76ec063dc0 2021-05-25T12:00:01.172 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.168+0000 7f76d6ffd700 1 -- 172.21.15.102:0/474811993 <== mon.2 v1:172.21.15.102:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1342 (unknown 254746657 0 921647380) 0x7f76e0012ea0 con 0x7f76ec063dc0 2021-05-25T12:00:01.172 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.168+0000 7f76d6ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2021-05-25T12:00:01.172 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.168+0000 7f76d6ffd700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:00:01.172 INFO:teuthology.orchestra.run.smithi102.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":19,"features":{"quorum_con":"4540138303579357183","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"20ee91b6-8935-4b98-b868-85c03789ccc4","modified":"2021-05-25T11:59:11.011522Z","created":"2021-05-25T11:59:11.011522Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6790","nonce":0}]},"addr":"172.21.15.102:6790/0","public_addr":"172.21.15.102:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2021-05-25T12:00:01.174 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.169+0000 7f76f1270700 10 librados: watch_flush enter 2021-05-25T12:00:01.174 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.169+0000 7f76f1270700 10 librados: watch_flush exit 2021-05-25T12:00:01.174 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.169+0000 7f76f1270700 20 client.4382.objecter shutdown clearing up homeless session... 2021-05-25T12:00:01.175 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.169+0000 7f76f1270700 10 client.4382.objecter successfully canceled tick 2021-05-25T12:00:01.175 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.169+0000 7f76f1270700 1 -- 172.21.15.102:0/474811993 >> v1:172.21.15.181:6812/29582 conn(0x7f76cc0334a0 legacy=0x7f76cc035960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:01.175 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.169+0000 7f76f1270700 10 monclient: shutdown 2021-05-25T12:00:01.175 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.169+0000 7f76f1270700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:00:01.176 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.169+0000 7f76f1270700 1 -- 172.21.15.102:0/474811993 >> v1:172.21.15.102:6790/0 conn(0x7f76ec063dc0 legacy=0x7f76ec118c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:01.176 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.169+0000 7f76f1270700 1 -- 172.21.15.102:0/474811993 shutdown_connections 2021-05-25T12:00:01.176 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.170+0000 7f76f1270700 1 -- 172.21.15.102:0/474811993 >> 172.21.15.102:0/474811993 conn(0x7f76ec09bc30 msgr2=0x7f76ec09d9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:00:01.176 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.170+0000 7f76f1270700 1 -- 172.21.15.102:0/474811993 shutdown_connections 2021-05-25T12:00:01.176 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.170+0000 7f76f1270700 1 -- 172.21.15.102:0/474811993 wait complete. 2021-05-25T12:00:01.177 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.170+0000 7f76f1270700 1 librados: shutdown 2021-05-25T12:00:01.183 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2021-05-25T12:00:01.183 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2021-05-25T12:00:01.184 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2021-05-25T12:00:01.294 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.291+0000 7fe10b207700 10 monclient: get_monmap_and_config 2021-05-25T12:00:01.294 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.291+0000 7fe10b207700 10 monclient: build_initial_monmap 2021-05-25T12:00:01.294 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.291+0000 7fe10b207700 10 monclient: monmap: 2021-05-25T12:00:01.294 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:00:01.295 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:01.295 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:00:01.293497+0000 2021-05-25T12:00:01.295 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:00:01.293497+0000 2021-05-25T12:00:01.296 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:00:01.296 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:01.296 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:00:01.296 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:00:01.296 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:00:01.297 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:01.310 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.292+0000 7fe10b207700 1 Processor -- start 2021-05-25T12:00:01.310 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.292+0000 7fe10b207700 1 -- start start 2021-05-25T12:00:01.310 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.292+0000 7fe10b207700 10 monclient: init 2021-05-25T12:00:01.311 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.292+0000 7fe10b207700 10 monclient: _reopen_session rank -1 2021-05-25T12:00:01.311 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.292+0000 7fe10b207700 10 monclient: _add_conns ranks=[1,0,2] 2021-05-25T12:00:01.311 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.292+0000 7fe10b207700 10 monclient(hunting): picked mon.noname-c con 0x7fe104090990 addr v1:172.21.15.181:6789/0 2021-05-25T12:00:01.312 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.292+0000 7fe10b207700 10 monclient(hunting): picked mon.noname-a con 0x7fe104090d70 addr v1:172.21.15.102:6789/0 2021-05-25T12:00:01.312 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.292+0000 7fe10b207700 10 monclient(hunting): picked mon.noname-b con 0x7fe10412e330 addr v1:172.21.15.102:6790/0 2021-05-25T12:00:01.312 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.292+0000 7fe10b207700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe104096fa0 con 0x7fe104090d70 2021-05-25T12:00:01.312 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.292+0000 7fe10b207700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1040926a0 con 0x7fe104090990 2021-05-25T12:00:01.312 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.292+0000 7fe10b207700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe104092940 con 0x7fe10412e330 2021-05-25T12:00:01.313 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.292+0000 7fe10b207700 10 monclient(hunting): _renew_subs 2021-05-25T12:00:01.313 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.292+0000 7fe10b207700 10 monclient(hunting): authenticate will time out at 801.890802s 2021-05-25T12:00:01.313 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.292+0000 7fe0fbfff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fe104090d70 0x7fe104091200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:59102/0 (socket says 172.21.15.102:59102) 2021-05-25T12:00:01.313 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.292+0000 7fe0fbfff700 1 -- 172.21.15.102:0/3411662180 learned_addr learned my addr 172.21.15.102:0/3411662180 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:00:01.314 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.292+0000 7fe1097a4700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fe10412e330 0x7fe104130720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:53982/0 (socket says 172.21.15.102:53982) 2021-05-25T12:00:01.314 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.293+0000 7fe0fb7fe700 1 -- 172.21.15.102:0/3411662180 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2766948891 0 0) 0x7fe104092940 con 0x7fe10412e330 2021-05-25T12:00:01.314 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.293+0000 7fe0fb7fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:01.314 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.293+0000 7fe0fb7fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:01.314 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.293+0000 7fe0fb7fe700 10 monclient(hunting): my global_id is 4385 2021-05-25T12:00:01.315 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.293+0000 7fe0fb7fe700 1 -- 172.21.15.102:0/3411662180 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0f0003270 con 0x7fe10412e330 2021-05-25T12:00:01.315 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.293+0000 7fe0fb7fe700 1 -- 172.21.15.102:0/3411662180 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2240001477 0 0) 0x7fe104096fa0 con 0x7fe104090d70 2021-05-25T12:00:01.315 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.293+0000 7fe0fb7fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:01.315 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.293+0000 7fe0fb7fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:01.316 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.293+0000 7fe0fb7fe700 10 monclient(hunting): my global_id is 4392 2021-05-25T12:00:01.316 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.293+0000 7fe0fb7fe700 1 -- 172.21.15.102:0/3411662180 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe104092940 con 0x7fe104090d70 2021-05-25T12:00:01.316 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.293+0000 7fe0fb7fe700 1 -- 172.21.15.102:0/3411662180 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1286266439 0 0) 0x7fe1040926a0 con 0x7fe104090990 2021-05-25T12:00:01.316 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.293+0000 7fe0fb7fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:01.316 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.293+0000 7fe0fb7fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:01.317 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.293+0000 7fe0fb7fe700 10 monclient(hunting): my global_id is 4387 2021-05-25T12:00:01.317 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.293+0000 7fe0fb7fe700 1 -- 172.21.15.102:0/3411662180 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe104096fa0 con 0x7fe104090990 2021-05-25T12:00:01.317 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.293+0000 7fe0fb7fe700 1 -- 172.21.15.102:0/3411662180 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4168067359 0 0) 0x7fe104092940 con 0x7fe104090d70 2021-05-25T12:00:01.317 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.293+0000 7fe0fb7fe700 1 -- 172.21.15.102:0/3411662180 >> v1:172.21.15.102:6790/0 conn(0x7fe10412e330 legacy=0x7fe104130720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:01.317 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.293+0000 7fe0fb7fe700 1 -- 172.21.15.102:0/3411662180 >> v1:172.21.15.181:6789/0 conn(0x7fe104090990 legacy=0x7fe104093240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:01.318 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.293+0000 7fe0fb7fe700 10 monclient: _finish_hunting 0 2021-05-25T12:00:01.318 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.293+0000 7fe0fb7fe700 1 monclient: found mon.noname-a 2021-05-25T12:00:01.318 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.293+0000 7fe0fb7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:00:01.318 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.293+0000 7fe0fb7fe700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T12:00:01.319 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.293+0000 7fe0fb7fe700 1 -- 172.21.15.102:0/3411662180 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe104092be0 con 0x7fe104090d70 2021-05-25T12:00:01.319 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.293+0000 7fe0fb7fe700 10 monclient: _finish_auth 0 2021-05-25T12:00:01.319 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.293+0000 7fe0fb7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:00:01.319 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.293+0000 7fe0fb7fe700 1 -- 172.21.15.102:0/3411662180 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fe0ec0015f0 con 0x7fe104090d70 2021-05-25T12:00:01.320 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.293+0000 7fe10b207700 5 monclient: authenticate success, global_id 4392 2021-05-25T12:00:01.320 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.293+0000 7fe10b207700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:00:01.320 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.293+0000 7fe0fb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:01.320 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.293+0000 7fe0fb7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:00:01.320 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.293+0000 7fe0fb7fe700 10 monclient: dump: 2021-05-25T12:00:01.321 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:01.321 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:01.321 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:01.321 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:01.322 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:01.322 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:01.322 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:01.322 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:01.323 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:01.323 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:01.323 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.293+0000 7fe10b207700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:00:01.323 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.293+0000 7fe0fb7fe700 1 -- 172.21.15.102:0/3411662180 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe0ec0018a0 con 0x7fe104090d70 2021-05-25T12:00:01.324 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.293+0000 7fe0fb7fe700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:00:01.324 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.293+0000 7fe0fb7fe700 1 -- 172.21.15.102:0/3411662180 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fe0ec001c30 con 0x7fe104090d70 2021-05-25T12:00:01.324 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.293+0000 7fe10b207700 10 monclient: get_monmap_and_config success 2021-05-25T12:00:01.326 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.293+0000 7fe10b207700 4 set_mon_vals no callback set 2021-05-25T12:00:01.326 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.293+0000 7fe0fb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:01.326 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.293+0000 7fe0fb7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:00:01.327 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.293+0000 7fe0fb7fe700 10 monclient: dump: 2021-05-25T12:00:01.327 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:01.327 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:01.327 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:01.328 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:01.328 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:01.328 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:01.328 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:01.328 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:01.329 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:01.329 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:01.329 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.293+0000 7fe10b207700 10 monclient: shutdown 2021-05-25T12:00:01.329 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.293+0000 7fe10b207700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:00:01.330 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.293+0000 7fe10b207700 1 -- 172.21.15.102:0/3411662180 >> v1:172.21.15.102:6789/0 conn(0x7fe104090d70 legacy=0x7fe104091200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:01.330 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.294+0000 7fe10b207700 1 -- 172.21.15.102:0/3411662180 shutdown_connections 2021-05-25T12:00:01.330 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.294+0000 7fe10b207700 1 -- 172.21.15.102:0/3411662180 >> 172.21.15.102:0/3411662180 conn(0x7fe10409b370 msgr2=0x7fe104097430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:00:01.331 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.294+0000 7fe10b207700 1 -- 172.21.15.102:0/3411662180 shutdown_connections 2021-05-25T12:00:01.331 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.294+0000 7fe10b207700 1 -- 172.21.15.102:0/3411662180 wait complete. 2021-05-25T12:00:01.332 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.294+0000 7fe10b207700 10 monclient: build_initial_monmap 2021-05-25T12:00:01.332 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.294+0000 7fe10b207700 10 monclient: monmap: 2021-05-25T12:00:01.332 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:00:01.332 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:00:01.333 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:00:01.333 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:00:01.333 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:00:01.333 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:00:01.333 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:00:01.334 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:00:01.334 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:00:01.334 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:01.334 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.294+0000 7fe10b207700 1 librados: starting msgr at 2021-05-25T12:00:01.335 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.294+0000 7fe10b207700 1 librados: starting objecter 2021-05-25T12:00:01.335 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.294+0000 7fe10b207700 1 Processor -- start 2021-05-25T12:00:01.335 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.295+0000 7fe10b207700 1 -- start start 2021-05-25T12:00:01.335 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.295+0000 7fe10b207700 1 librados: setting wanted keys 2021-05-25T12:00:01.335 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.295+0000 7fe10b207700 1 librados: calling monclient init 2021-05-25T12:00:01.336 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.295+0000 7fe10b207700 10 monclient: init 2021-05-25T12:00:01.336 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.295+0000 7fe10b207700 10 monclient: _reopen_session rank -1 2021-05-25T12:00:01.336 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.295+0000 7fe10b207700 10 monclient: _add_conns ranks=[2,0,1] 2021-05-25T12:00:01.336 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.295+0000 7fe10b207700 10 monclient(hunting): picked mon.noname-c con 0x7fe104090990 addr v1:172.21.15.102:6790/0 2021-05-25T12:00:01.337 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.295+0000 7fe10b207700 10 monclient(hunting): picked mon.noname-a con 0x7fe104090d70 addr v1:172.21.15.102:6789/0 2021-05-25T12:00:01.337 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.295+0000 7fe10b207700 10 monclient(hunting): picked mon.noname-b con 0x7fe10412e330 addr v1:172.21.15.181:6789/0 2021-05-25T12:00:01.337 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.295+0000 7fe10b207700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe104118510 con 0x7fe104090d70 2021-05-25T12:00:01.337 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.295+0000 7fe10b207700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1041187b0 con 0x7fe10412e330 2021-05-25T12:00:01.337 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.295+0000 7fe10b207700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe104118a50 con 0x7fe104090990 2021-05-25T12:00:01.338 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.295+0000 7fe10b207700 10 monclient(hunting): _renew_subs 2021-05-25T12:00:01.338 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.295+0000 7fe10b207700 10 monclient(hunting): authenticate will time out at 801.893526s 2021-05-25T12:00:01.338 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.295+0000 7fe0fbfff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fe104090d70 0x7fe104140110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:59108/0 (socket says 172.21.15.102:59108) 2021-05-25T12:00:01.338 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.295+0000 7fe108fa3700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fe104090990 0x7fe10413b480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:53984/0 (socket says 172.21.15.102:53984) 2021-05-25T12:00:01.338 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.295+0000 7fe0fbfff700 1 -- 172.21.15.102:0/3687418747 learned_addr learned my addr 172.21.15.102:0/3687418747 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:00:01.339 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.295+0000 7fe0f8ff9700 10 client.?.objecter ms_handle_connect 0x7fe104090d70 2021-05-25T12:00:01.339 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.295+0000 7fe0f8ff9700 10 client.?.objecter resend_mon_ops 2021-05-25T12:00:01.339 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.295+0000 7fe0f8ff9700 10 client.?.objecter ms_handle_connect 0x7fe104090990 2021-05-25T12:00:01.339 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.295+0000 7fe0f8ff9700 10 client.?.objecter resend_mon_ops 2021-05-25T12:00:01.340 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.295+0000 7fe0f8ff9700 1 -- 172.21.15.102:0/3687418747 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3788257806 0 0) 0x7fe104118510 con 0x7fe104090d70 2021-05-25T12:00:01.340 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.295+0000 7fe0f8ff9700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:01.340 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.295+0000 7fe0f8ff9700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:01.340 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.295+0000 7fe0f8ff9700 10 monclient(hunting): my global_id is 4395 2021-05-25T12:00:01.341 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.295+0000 7fe0f8ff9700 1 -- 172.21.15.102:0/3687418747 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0e4004c10 con 0x7fe104090d70 2021-05-25T12:00:01.341 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.295+0000 7fe0f8ff9700 1 -- 172.21.15.102:0/3687418747 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3692951395 0 0) 0x7fe104118a50 con 0x7fe104090990 2021-05-25T12:00:01.341 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.295+0000 7fe0f8ff9700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:01.341 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.295+0000 7fe0f8ff9700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:01.341 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.295+0000 7fe0f8ff9700 10 monclient(hunting): my global_id is 4388 2021-05-25T12:00:01.342 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.295+0000 7fe0f8ff9700 1 -- 172.21.15.102:0/3687418747 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe104118510 con 0x7fe104090990 2021-05-25T12:00:01.342 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.295+0000 7fe0f8ff9700 10 client.?.objecter ms_handle_connect 0x7fe10412e330 2021-05-25T12:00:01.342 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.295+0000 7fe0f8ff9700 10 client.?.objecter resend_mon_ops 2021-05-25T12:00:01.342 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.296+0000 7fe0f8ff9700 1 -- 172.21.15.102:0/3687418747 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 873988413 0 0) 0x7fe0e4004c10 con 0x7fe104090d70 2021-05-25T12:00:01.343 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.296+0000 7fe0f8ff9700 1 -- 172.21.15.102:0/3687418747 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe104118a50 con 0x7fe104090d70 2021-05-25T12:00:01.343 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.296+0000 7fe0f8ff9700 1 -- 172.21.15.102:0/3687418747 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fe0ec0016b0 con 0x7fe104090d70 2021-05-25T12:00:01.343 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.296+0000 7fe0f8ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:01.343 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.296+0000 7fe0f8ff9700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:00:01.344 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.296+0000 7fe0f8ff9700 10 monclient(hunting): dump: 2021-05-25T12:00:01.344 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:01.344 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:01.344 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:01.344 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:01.345 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:01.345 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:01.345 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:01.345 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:01.346 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:01.346 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:01.346 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.296+0000 7fe0f8ff9700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:00:01.346 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.296+0000 7fe0f8ff9700 1 -- 172.21.15.102:0/3687418747 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1937932432 0 0) 0x7fe104118510 con 0x7fe104090990 2021-05-25T12:00:01.346 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.296+0000 7fe0f8ff9700 1 -- 172.21.15.102:0/3687418747 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe0e4004c10 con 0x7fe104090990 2021-05-25T12:00:01.347 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.296+0000 7fe0f8ff9700 1 -- 172.21.15.102:0/3687418747 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1764445614 0 0) 0x7fe1041187b0 con 0x7fe10412e330 2021-05-25T12:00:01.347 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.296+0000 7fe0f8ff9700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:01.347 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.296+0000 7fe0f8ff9700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:01.347 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.296+0000 7fe0f8ff9700 10 monclient(hunting): my global_id is 4390 2021-05-25T12:00:01.348 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.296+0000 7fe0f8ff9700 1 -- 172.21.15.102:0/3687418747 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe104118510 con 0x7fe10412e330 2021-05-25T12:00:01.348 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.296+0000 7fe0f8ff9700 1 -- 172.21.15.102:0/3687418747 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fe0f4001840 con 0x7fe104090990 2021-05-25T12:00:01.348 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.296+0000 7fe0f8ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:01.348 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.296+0000 7fe0f8ff9700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2021-05-25T12:00:01.348 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.296+0000 7fe0f8ff9700 10 monclient(hunting): dump: 2021-05-25T12:00:01.349 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:01.349 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:01.349 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:01.349 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:01.350 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:01.350 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:01.350 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:01.350 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:01.351 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:01.351 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:01.351 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.296+0000 7fe0f8ff9700 1 -- 172.21.15.102:0/3687418747 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3465924883 0 0) 0x7fe104118a50 con 0x7fe104090d70 2021-05-25T12:00:01.351 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.296+0000 7fe0f8ff9700 1 -- 172.21.15.102:0/3687418747 >> v1:172.21.15.102:6790/0 conn(0x7fe104090990 legacy=0x7fe10413b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:01.352 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.296+0000 7fe0f8ff9700 1 -- 172.21.15.102:0/3687418747 >> v1:172.21.15.181:6789/0 conn(0x7fe10412e330 legacy=0x7fe104143840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:01.352 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.296+0000 7fe0f8ff9700 10 monclient: _finish_hunting 0 2021-05-25T12:00:01.352 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.296+0000 7fe0f8ff9700 1 monclient: found mon.a 2021-05-25T12:00:01.352 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.296+0000 7fe0f8ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:00:01.352 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.296+0000 7fe0f8ff9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:00:01.353 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.296+0000 7fe0f8ff9700 1 -- 172.21.15.102:0/3687418747 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe104116900 con 0x7fe104090d70 2021-05-25T12:00:01.353 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.296+0000 7fe0f8ff9700 10 monclient: _finish_auth 0 2021-05-25T12:00:01.353 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.296+0000 7fe0f8ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:00:01.353 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.296+0000 7fe10b207700 5 monclient: authenticate success, global_id 4395 2021-05-25T12:00:01.354 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.296+0000 7fe10b207700 10 monclient: _renew_subs 2021-05-25T12:00:01.354 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.296+0000 7fe10b207700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:00:01.354 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.296+0000 7fe10b207700 1 -- 172.21.15.102:0/3687418747 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe10413bb90 con 0x7fe104090d70 2021-05-25T12:00:01.354 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.296+0000 7fe10b207700 10 client.4395.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:00:01.354 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.296+0000 7fe10b207700 10 monclient: _renew_subs 2021-05-25T12:00:01.355 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.296+0000 7fe10b207700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:00:01.355 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.296+0000 7fe10b207700 1 -- 172.21.15.102:0/3687418747 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe104068b10 con 0x7fe104090d70 2021-05-25T12:00:01.355 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.296+0000 7fe10b207700 1 librados: init done 2021-05-25T12:00:01.355 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.296+0000 7fe0f8ff9700 1 -- 172.21.15.102:0/3687418747 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe0ec0049e0 con 0x7fe104090d70 2021-05-25T12:00:01.356 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.296+0000 7fe0f8ff9700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:00:01.356 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.296+0000 7fe0f8ff9700 1 -- 172.21.15.102:0/3687418747 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fe0ec004d70 con 0x7fe104090d70 2021-05-25T12:00:01.356 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.296+0000 7fe0f8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:01.356 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.296+0000 7fe0f8ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:00:01.356 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.296+0000 7fe0faffd700 4 set_mon_vals no callback set 2021-05-25T12:00:01.357 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.296+0000 7fe0f8ff9700 10 monclient: dump: 2021-05-25T12:00:01.357 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:01.357 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:01.357 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:01.357 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:01.358 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:01.358 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:01.358 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:01.358 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:01.359 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:01.359 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:01.359 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.296+0000 7fe10b207700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:00:01.359 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.296+0000 7fe10b207700 1 -- 172.21.15.102:0/3687418747 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fe0dc0065a0 con 0x7fe0dc002840 2021-05-25T12:00:01.359 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.297+0000 7fe0f8ff9700 1 -- 172.21.15.102:0/3687418747 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7fe0ec004ed0 con 0x7fe104090d70 2021-05-25T12:00:01.360 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.297+0000 7fe0f8ff9700 10 client.4395.objecter ms_handle_connect 0x7fe0dc002840 2021-05-25T12:00:01.360 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.297+0000 7fe0f8ff9700 10 client.4395.objecter resend_mon_ops 2021-05-25T12:00:01.360 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.297+0000 7fe0f8ff9700 1 -- 172.21.15.102:0/3687418747 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(15..15 src has 1..15) v4 ==== 3549+0+0 (unknown 3948819498 0 0) 0x7fe0ec03d1a0 con 0x7fe104090d70 2021-05-25T12:00:01.360 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.297+0000 7fe0f8ff9700 10 client.4395.objecter ms_dispatch 0x7fe104001550 osd_map(15..15 src has 1..15) v4 2021-05-25T12:00:01.360 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.297+0000 7fe0f8ff9700 3 client.4395.objecter handle_osd_map got epochs [15,15] > 0 2021-05-25T12:00:01.361 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.297+0000 7fe0f8ff9700 3 client.4395.objecter handle_osd_map decoding full epoch 15 2021-05-25T12:00:01.361 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.297+0000 7fe0f8ff9700 20 client.4395.objecter dump_active .. 0 homeless 2021-05-25T12:00:01.361 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.297+0000 7fe0f8ff9700 1 -- 172.21.15.102:0/3687418747 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 726375486 0 0) 0x7fe0dc0065a0 con 0x7fe0dc002840 2021-05-25T12:00:01.361 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.297+0000 7fe0f8ff9700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:01.362 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.297+0000 7fe0f8ff9700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:01.362 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.297+0000 7fe0f8ff9700 10 monclient(hunting): my global_id is 4398 2021-05-25T12:00:01.362 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.297+0000 7fe0f8ff9700 1 -- 172.21.15.102:0/3687418747 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe104118a50 con 0x7fe0dc002840 2021-05-25T12:00:01.362 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.297+0000 7fe1097a4700 10 monclient: get_auth_request con 0x7fe0e40334b0 auth_method 0 2021-05-25T12:00:01.363 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.297+0000 7fe0f8ff9700 1 -- 172.21.15.102:0/3687418747 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3193056459 0 0) 0x7fe104118a50 con 0x7fe0dc002840 2021-05-25T12:00:01.363 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.297+0000 7fe0f8ff9700 1 -- 172.21.15.102:0/3687418747 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe0dc0065a0 con 0x7fe0dc002840 2021-05-25T12:00:01.363 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.297+0000 7fe0f8ff9700 1 -- 172.21.15.102:0/3687418747 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fe0f4001e70 con 0x7fe0dc002840 2021-05-25T12:00:01.363 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.297+0000 7fe0f8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:01.363 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.297+0000 7fe0f8ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:00:01.364 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.297+0000 7fe0f8ff9700 10 monclient: dump: 2021-05-25T12:00:01.364 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:01.364 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:01.364 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:01.365 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:01.365 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:01.365 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:01.365 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:01.365 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:01.366 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:01.366 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:01.366 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.297+0000 7fe0f8ff9700 1 -- 172.21.15.102:0/3687418747 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 461638302 0 0) 0x7fe0dc0065a0 con 0x7fe0dc002840 2021-05-25T12:00:01.366 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.297+0000 7fe0f8ff9700 1 -- 172.21.15.102:0/3687418747 --> v1:172.21.15.102:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe0dc009e40 con 0x7fe0dc002840 2021-05-25T12:00:01.367 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.297+0000 7fe0f8ff9700 10 client.4395.objecter ms_handle_connect 0x7fe0e40334b0 2021-05-25T12:00:01.367 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.297+0000 7fe0f8ff9700 1 -- 172.21.15.102:0/3687418747 <== mon.0 v1:172.21.15.102:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7fe0f4005510 con 0x7fe0dc002840 2021-05-25T12:00:01.367 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.297+0000 7fe0f8ff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:00:01.367 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.297+0000 7fe0f8ff9700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:00:01.367 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.297+0000 7fe0f8ff9700 1 -- 172.21.15.102:0/3687418747 >> v1:172.21.15.102:6789/0 conn(0x7fe0dc002840 legacy=0x7fe0dc004ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:01.368 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.297+0000 7fe0f8ff9700 1 -- 172.21.15.102:0/3687418747 >> v1:172.21.15.102:6789/0 conn(0x7fe0dc002840 legacy=0x7fe0dc004ce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:00:01.368 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.304+0000 7fe10b207700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:00:01.368 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.304+0000 7fe10b207700 1 -- 172.21.15.102:0/3687418747 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fe0dc0074f0 con 0x7fe0dc006890 2021-05-25T12:00:01.368 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.304+0000 7fe0f8ff9700 10 client.4395.objecter ms_handle_connect 0x7fe0dc006890 2021-05-25T12:00:01.369 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.304+0000 7fe0f8ff9700 10 client.4395.objecter resend_mon_ops 2021-05-25T12:00:01.369 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.304+0000 7fe0f8ff9700 1 -- 172.21.15.102:0/3687418747 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3776817051 0 0) 0x7fe0dc0074f0 con 0x7fe0dc006890 2021-05-25T12:00:01.370 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.304+0000 7fe0f8ff9700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:01.370 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.304+0000 7fe0f8ff9700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:01.370 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.304+0000 7fe0f8ff9700 10 monclient(hunting): my global_id is 4401 2021-05-25T12:00:01.371 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.304+0000 7fe0f8ff9700 1 -- 172.21.15.102:0/3687418747 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0dc0065a0 con 0x7fe0dc006890 2021-05-25T12:00:01.371 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.305+0000 7fe0f8ff9700 1 -- 172.21.15.102:0/3687418747 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3839952499 0 0) 0x7fe0dc0065a0 con 0x7fe0dc006890 2021-05-25T12:00:01.371 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.305+0000 7fe0f8ff9700 1 -- 172.21.15.102:0/3687418747 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe0dc0074f0 con 0x7fe0dc006890 2021-05-25T12:00:01.371 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.305+0000 7fe0f8ff9700 1 -- 172.21.15.102:0/3687418747 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fe0f4002460 con 0x7fe0dc006890 2021-05-25T12:00:01.372 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.305+0000 7fe0f8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:01.372 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.305+0000 7fe0f8ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:00:01.372 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.305+0000 7fe0f8ff9700 10 monclient: dump: 2021-05-25T12:00:01.372 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:01.373 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:01.373 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:01.373 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:01.373 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:01.374 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:01.374 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:01.374 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:01.374 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:01.375 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:01.375 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.305+0000 7fe0f8ff9700 1 -- 172.21.15.102:0/3687418747 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2154609312 0 0) 0x7fe0dc0074f0 con 0x7fe0dc006890 2021-05-25T12:00:01.375 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.305+0000 7fe0f8ff9700 1 -- 172.21.15.102:0/3687418747 --> v1:172.21.15.102:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe0dc008770 con 0x7fe0dc006890 2021-05-25T12:00:01.375 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.305+0000 7fe0f8ff9700 1 -- 172.21.15.102:0/3687418747 <== mon.0 v1:172.21.15.102:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3942 (unknown 0 0 2232538786) 0x7fe0f4002780 con 0x7fe0dc006890 2021-05-25T12:00:01.376 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.305+0000 7fe0f8ff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:00:01.376 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.305+0000 7fe0f8ff9700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:00:01.376 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.305+0000 7fe0f8ff9700 1 -- 172.21.15.102:0/3687418747 >> v1:172.21.15.102:6789/0 conn(0x7fe0dc006890 legacy=0x7fe0dc006d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:01.376 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.305+0000 7fe0f8ff9700 1 -- 172.21.15.102:0/3687418747 >> v1:172.21.15.102:6789/0 conn(0x7fe0dc006890 legacy=0x7fe0dc006d20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:00:01.377 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:00:01.377 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:00:01.377 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:00:01.378 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "leader", 2021-05-25T12:00:01.378 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 8, 2021-05-25T12:00:01.378 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:00:01.378 INFO:teuthology.orchestra.run.smithi102.stdout: 0, 2021-05-25T12:00:01.379 INFO:teuthology.orchestra.run.smithi102.stdout: 1, 2021-05-25T12:00:01.379 INFO:teuthology.orchestra.run.smithi102.stdout: 2 2021-05-25T12:00:01.379 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:00:01.379 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 20, 2021-05-25T12:00:01.380 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:00:01.380 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:00:01.380 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:00:01.380 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:00:01.381 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:00:01.381 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:00:01.381 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:00:01.381 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:00:01.381 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:00:01.382 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:00:01.382 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:00:01.382 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:00:01.382 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:00:01.383 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:00:01.383 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:00:01.383 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:00:01.383 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:00:01.384 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:00:01.384 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:00:01.384 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:00:01.384 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:00:01.384 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:00:01.385 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:00:01.385 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:00:01.385 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:00:01.385 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:01.386 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:00:01.386 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:00:01.386 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:00:01.386 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:00:01.387 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:00:01.387 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:00:01.387 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:00:01.387 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:00:01.388 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:00:01.388 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:00:01.388 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:00:01.388 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:00:01.389 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:00:01.389 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:00:01.389 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:00:01.389 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:00:01.389 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:00:01.390 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:00:01.390 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:00:01.390 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:00:01.390 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:00:01.391 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:00:01.391 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:00:01.391 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:00:01.391 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:00:01.392 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:00:01.392 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:01.392 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:00:01.392 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:01.393 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:00:01.393 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:00:01.393 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:00:01.393 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:00:01.393 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:01.394 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:00:01.394 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:00:01.394 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:00:01.394 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:01.395 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:00:01.395 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:01.395 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:00:01.395 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:00:01.396 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:00:01.396 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:00:01.396 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:00:01.396 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:01.397 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:01.397 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:00:01.397 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:00:01.397 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:00:01.397 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:00:01.398 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:01.398 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:00:01.398 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:00:01.398 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:00:01.399 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:01.399 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:00:01.399 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:01.399 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:00:01.400 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:00:01.400 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:00:01.400 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:00:01.400 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:00:01.401 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:01.401 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:01.401 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:00:01.401 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:00:01.402 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:00:01.402 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:00:01.402 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:01.402 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:00:01.402 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:00:01.403 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:00:01.403 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:01.403 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:00:01.403 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:01.404 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:00:01.404 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:00:01.404 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:00:01.404 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:00:01.405 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:00:01.405 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:01.405 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:00:01.405 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:01.406 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:00:01.406 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:00:01.406 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:01.406 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:00:01.406 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:00:01.407 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:00:01.407 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:01.407 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:00:01.407 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": [ 2021-05-25T12:00:01.408 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:01.408 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:00:01.408 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:00:01.408 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:00:01.409 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:01.409 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:00:01.409 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:00:01.409 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:01.410 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:00:01.410 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:00:01.410 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 4 2021-05-25T12:00:01.410 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:01.411 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:00:01.411 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:01.411 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:00:01.411 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:00:01.414 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.305+0000 7fe10b207700 10 librados: watch_flush enter 2021-05-25T12:00:01.414 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.305+0000 7fe10b207700 10 librados: watch_flush exit 2021-05-25T12:00:01.414 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.305+0000 7fe10b207700 20 client.4395.objecter shutdown clearing up homeless session... 2021-05-25T12:00:01.414 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.305+0000 7fe10b207700 10 client.4395.objecter successfully canceled tick 2021-05-25T12:00:01.415 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.305+0000 7fe10b207700 1 -- 172.21.15.102:0/3687418747 >> v1:172.21.15.181:6812/29582 conn(0x7fe0e40334b0 legacy=0x7fe0e4035970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:01.415 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.305+0000 7fe10b207700 10 monclient: shutdown 2021-05-25T12:00:01.415 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.305+0000 7fe10b207700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:00:01.415 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.305+0000 7fe10b207700 1 -- 172.21.15.102:0/3687418747 >> v1:172.21.15.102:6789/0 conn(0x7fe104090d70 legacy=0x7fe104140110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:01.416 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.305+0000 7fe1097a4700 1 -- 172.21.15.102:0/3687418747 reap_dead start 2021-05-25T12:00:01.416 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.305+0000 7fe1097a4700 1 -- 172.21.15.102:0/3687418747 reap_dead start 2021-05-25T12:00:01.416 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.305+0000 7fe10b207700 1 -- 172.21.15.102:0/3687418747 shutdown_connections 2021-05-25T12:00:01.416 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.305+0000 7fe10b207700 1 -- 172.21.15.102:0/3687418747 >> 172.21.15.102:0/3687418747 conn(0x7fe10409b370 msgr2=0x7fe104093a60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:00:01.417 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.305+0000 7fe10b207700 1 -- 172.21.15.102:0/3687418747 shutdown_connections 2021-05-25T12:00:01.417 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.305+0000 7fe10b207700 1 -- 172.21.15.102:0/3687418747 wait complete. 2021-05-25T12:00:01.417 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.306+0000 7fe10b207700 1 librados: shutdown 2021-05-25T12:00:01.418 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2021-05-25T12:00:01.528 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.525+0000 7f89ec740700 10 monclient: get_monmap_and_config 2021-05-25T12:00:01.529 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.525+0000 7f89ec740700 10 monclient: build_initial_monmap 2021-05-25T12:00:01.529 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.526+0000 7f89ec740700 10 monclient: monmap: 2021-05-25T12:00:01.529 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:00:01.530 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:01.530 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:00:01.527887+0000 2021-05-25T12:00:01.530 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:00:01.527887+0000 2021-05-25T12:00:01.531 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:00:01.531 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:01.531 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:00:01.532 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:00:01.532 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:00:01.532 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:01.532 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.526+0000 7f89ec740700 1 Processor -- start 2021-05-25T12:00:01.533 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.526+0000 7f89ec740700 1 -- start start 2021-05-25T12:00:01.533 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.526+0000 7f89ec740700 10 monclient: init 2021-05-25T12:00:01.547 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.527+0000 7f89ec740700 10 monclient: _reopen_session rank -1 2021-05-25T12:00:01.547 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.527+0000 7f89ec740700 10 monclient: _add_conns ranks=[1,2,0] 2021-05-25T12:00:01.547 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.527+0000 7f89ec740700 10 monclient(hunting): picked mon.noname-c con 0x7f89e4069190 addr v1:172.21.15.181:6789/0 2021-05-25T12:00:01.547 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.527+0000 7f89ec740700 10 monclient(hunting): picked mon.noname-b con 0x7f89e4069c80 addr v1:172.21.15.102:6790/0 2021-05-25T12:00:01.548 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.527+0000 7f89ec740700 10 monclient(hunting): picked mon.noname-a con 0x7f89e4096ab0 addr v1:172.21.15.102:6789/0 2021-05-25T12:00:01.548 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.527+0000 7f89ec740700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89e40959b0 con 0x7f89e4096ab0 2021-05-25T12:00:01.548 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.527+0000 7f89ec740700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89e40914e0 con 0x7f89e4069190 2021-05-25T12:00:01.548 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.527+0000 7f89ec740700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89e4095c10 con 0x7f89e4069c80 2021-05-25T12:00:01.548 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.527+0000 7f89ec740700 10 monclient(hunting): _renew_subs 2021-05-25T12:00:01.549 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.527+0000 7f89ec740700 10 monclient(hunting): authenticate will time out at 802.125211s 2021-05-25T12:00:01.549 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.527+0000 7f89e9cdb700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f89e4069c80 0x7f89e4130e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:53998/0 (socket says 172.21.15.102:53998) 2021-05-25T12:00:01.549 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.527+0000 7f89e9cdb700 1 -- 172.21.15.102:0/1308633590 learned_addr learned my addr 172.21.15.102:0/1308633590 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:00:01.549 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.527+0000 7f89e94da700 1 -- 172.21.15.102:0/1308633590 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3544372561 0 0) 0x7f89e4095c10 con 0x7f89e4069c80 2021-05-25T12:00:01.550 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.527+0000 7f89e94da700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:01.550 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.527+0000 7f89e94da700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:01.550 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.527+0000 7f89e94da700 10 monclient(hunting): my global_id is 4391 2021-05-25T12:00:01.550 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.527+0000 7f89e94da700 1 -- 172.21.15.102:0/1308633590 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89cc003270 con 0x7f89e4069c80 2021-05-25T12:00:01.550 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.527+0000 7f89e94da700 1 -- 172.21.15.102:0/1308633590 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2954204184 0 0) 0x7f89e40959b0 con 0x7f89e4096ab0 2021-05-25T12:00:01.551 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.527+0000 7f89e94da700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:01.551 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.527+0000 7f89e94da700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:01.551 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.527+0000 7f89e94da700 10 monclient(hunting): my global_id is 4404 2021-05-25T12:00:01.552 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.527+0000 7f89e94da700 1 -- 172.21.15.102:0/1308633590 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89e4095c10 con 0x7f89e4096ab0 2021-05-25T12:00:01.552 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.527+0000 7f89e94da700 1 -- 172.21.15.102:0/1308633590 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3814235949 0 0) 0x7f89cc003270 con 0x7f89e4069c80 2021-05-25T12:00:01.552 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.527+0000 7f89e94da700 1 -- 172.21.15.102:0/1308633590 >> v1:172.21.15.181:6789/0 conn(0x7f89e4069190 legacy=0x7f89e4069570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:01.552 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.527+0000 7f89e94da700 1 -- 172.21.15.102:0/1308633590 >> v1:172.21.15.102:6789/0 conn(0x7f89e4096ab0 legacy=0x7f89e4134560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:01.552 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.527+0000 7f89e94da700 10 monclient: _finish_hunting 0 2021-05-25T12:00:01.553 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.527+0000 7f89e94da700 1 monclient: found mon.noname-b 2021-05-25T12:00:01.553 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.527+0000 7f89e94da700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:00:01.553 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.527+0000 7f89e94da700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T12:00:01.553 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.527+0000 7f89e94da700 1 -- 172.21.15.102:0/1308633590 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89e4095eb0 con 0x7f89e4069c80 2021-05-25T12:00:01.554 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.527+0000 7f89e94da700 10 monclient: _finish_auth 0 2021-05-25T12:00:01.554 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.527+0000 7f89e94da700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:00:01.554 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.527+0000 7f89ec740700 5 monclient: authenticate success, global_id 4391 2021-05-25T12:00:01.554 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.527+0000 7f89e94da700 1 -- 172.21.15.102:0/1308633590 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f89d4001610 con 0x7f89e4069c80 2021-05-25T12:00:01.554 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.527+0000 7f89ec740700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:00:01.555 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.527+0000 7f89e94da700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:01.555 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.527+0000 7f89e94da700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:00:01.555 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.527+0000 7f89e94da700 10 monclient: dump: 2021-05-25T12:00:01.555 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:01.556 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:01.556 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:01.556 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:01.556 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:01.556 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:01.557 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:01.557 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:01.557 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:01.557 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:01.557 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.527+0000 7f89ec740700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:00:01.558 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.527+0000 7f89e94da700 1 -- 172.21.15.102:0/1308633590 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f89d40018b0 con 0x7f89e4069c80 2021-05-25T12:00:01.558 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.527+0000 7f89e94da700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:00:01.558 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.527+0000 7f89ec740700 10 monclient: get_monmap_and_config success 2021-05-25T12:00:01.558 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.527+0000 7f89e94da700 1 -- 172.21.15.102:0/1308633590 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f89d4001c40 con 0x7f89e4069c80 2021-05-25T12:00:01.558 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.527+0000 7f89e94da700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:01.559 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.527+0000 7f89ec740700 4 set_mon_vals no callback set 2021-05-25T12:00:01.559 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.527+0000 7f89e94da700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:00:01.559 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.527+0000 7f89e94da700 10 monclient: dump: 2021-05-25T12:00:01.559 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:01.560 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:01.560 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:01.560 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:01.560 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:01.560 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:01.561 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:01.561 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:01.561 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:01.561 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:01.562 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.528+0000 7f89ec740700 10 monclient: shutdown 2021-05-25T12:00:01.562 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.528+0000 7f89ec740700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:00:01.562 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.528+0000 7f89ec740700 1 -- 172.21.15.102:0/1308633590 >> v1:172.21.15.102:6790/0 conn(0x7f89e4069c80 legacy=0x7f89e4130e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:01.562 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.528+0000 7f89ec740700 1 -- 172.21.15.102:0/1308633590 shutdown_connections 2021-05-25T12:00:01.562 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.528+0000 7f89ec740700 1 -- 172.21.15.102:0/1308633590 >> 172.21.15.102:0/1308633590 conn(0x7f89e409af90 msgr2=0x7f89e409b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:00:01.563 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.528+0000 7f89ec740700 1 -- 172.21.15.102:0/1308633590 shutdown_connections 2021-05-25T12:00:01.563 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.528+0000 7f89ec740700 1 -- 172.21.15.102:0/1308633590 wait complete. 2021-05-25T12:00:01.563 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.528+0000 7f89ec740700 10 monclient: build_initial_monmap 2021-05-25T12:00:01.563 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.528+0000 7f89ec740700 10 monclient: monmap: 2021-05-25T12:00:01.564 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:00:01.564 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:00:01.564 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:00:01.564 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:00:01.564 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:00:01.565 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:00:01.565 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:00:01.565 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:00:01.565 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:00:01.565 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:01.566 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.529+0000 7f89ec740700 1 librados: starting msgr at 2021-05-25T12:00:01.566 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.529+0000 7f89ec740700 1 librados: starting objecter 2021-05-25T12:00:01.566 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.529+0000 7f89ec740700 1 Processor -- start 2021-05-25T12:00:01.566 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.529+0000 7f89ec740700 1 -- start start 2021-05-25T12:00:01.567 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.529+0000 7f89ec740700 1 librados: setting wanted keys 2021-05-25T12:00:01.567 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.529+0000 7f89ec740700 1 librados: calling monclient init 2021-05-25T12:00:01.567 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.529+0000 7f89ec740700 10 monclient: init 2021-05-25T12:00:01.567 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.529+0000 7f89ec740700 10 monclient: _reopen_session rank -1 2021-05-25T12:00:01.568 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.529+0000 7f89ec740700 10 monclient: _add_conns ranks=[0,1,2] 2021-05-25T12:00:01.568 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.529+0000 7f89ec740700 10 monclient(hunting): picked mon.noname-a con 0x7f89e4069190 addr v1:172.21.15.102:6789/0 2021-05-25T12:00:01.568 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.529+0000 7f89ec740700 10 monclient(hunting): picked mon.noname-b con 0x7f89e4069c80 addr v1:172.21.15.181:6789/0 2021-05-25T12:00:01.568 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.529+0000 7f89ec740700 10 monclient(hunting): picked mon.noname-c con 0x7f89e4096ab0 addr v1:172.21.15.102:6790/0 2021-05-25T12:00:01.568 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.529+0000 7f89ec740700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89e411aa30 con 0x7f89e4069190 2021-05-25T12:00:01.569 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.529+0000 7f89ec740700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89e411acd0 con 0x7f89e4069c80 2021-05-25T12:00:01.569 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.529+0000 7f89ec740700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89e411af70 con 0x7f89e4096ab0 2021-05-25T12:00:01.569 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.529+0000 7f89ec740700 10 monclient(hunting): _renew_subs 2021-05-25T12:00:01.569 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.529+0000 7f89ec740700 10 monclient(hunting): authenticate will time out at 802.127721s 2021-05-25T12:00:01.569 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.529+0000 7f89ea4dc700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f89e4069190 0x7f89e413b430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:59124/0 (socket says 172.21.15.102:59124) 2021-05-25T12:00:01.570 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.529+0000 7f89ea4dc700 1 -- 172.21.15.102:0/1998808220 learned_addr learned my addr 172.21.15.102:0/1998808220 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:00:01.570 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.529+0000 7f89d27fc700 10 client.?.objecter ms_handle_connect 0x7f89e4069190 2021-05-25T12:00:01.570 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.529+0000 7f89d27fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:00:01.570 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.529+0000 7f89d27fc700 10 client.?.objecter ms_handle_connect 0x7f89e4096ab0 2021-05-25T12:00:01.571 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.529+0000 7f89d27fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:00:01.571 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.529+0000 7f89d27fc700 10 client.?.objecter ms_handle_connect 0x7f89e4069c80 2021-05-25T12:00:01.571 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.529+0000 7f89d27fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:00:01.571 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.529+0000 7f89d27fc700 1 -- 172.21.15.102:0/1998808220 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 411206643 0 0) 0x7f89e411aa30 con 0x7f89e4069190 2021-05-25T12:00:01.571 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.530+0000 7f89d27fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:01.572 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.530+0000 7f89d27fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:01.572 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.530+0000 7f89d27fc700 10 monclient(hunting): my global_id is 4407 2021-05-25T12:00:01.572 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.530+0000 7f89d27fc700 1 -- 172.21.15.102:0/1998808220 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89c4004c10 con 0x7f89e4069190 2021-05-25T12:00:01.572 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.530+0000 7f89d27fc700 1 -- 172.21.15.102:0/1998808220 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 513055478 0 0) 0x7f89e411af70 con 0x7f89e4096ab0 2021-05-25T12:00:01.572 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.530+0000 7f89d27fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:01.573 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.530+0000 7f89d27fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:01.573 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.530+0000 7f89d27fc700 10 monclient(hunting): my global_id is 4394 2021-05-25T12:00:01.573 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.530+0000 7f89d27fc700 1 -- 172.21.15.102:0/1998808220 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89e411aa30 con 0x7f89e4096ab0 2021-05-25T12:00:01.573 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.530+0000 7f89d27fc700 1 -- 172.21.15.102:0/1998808220 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 794431507 0 0) 0x7f89e411acd0 con 0x7f89e4069c80 2021-05-25T12:00:01.574 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.530+0000 7f89d27fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:01.574 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.530+0000 7f89d27fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:01.574 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.530+0000 7f89d27fc700 10 monclient(hunting): my global_id is 4396 2021-05-25T12:00:01.574 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.530+0000 7f89d27fc700 1 -- 172.21.15.102:0/1998808220 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89e411af70 con 0x7f89e4069c80 2021-05-25T12:00:01.575 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.530+0000 7f89d27fc700 1 -- 172.21.15.102:0/1998808220 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4030796350 0 0) 0x7f89c4004c10 con 0x7f89e4069190 2021-05-25T12:00:01.575 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.530+0000 7f89d27fc700 1 -- 172.21.15.102:0/1998808220 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f89e411acd0 con 0x7f89e4069190 2021-05-25T12:00:01.575 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.530+0000 7f89d27fc700 1 -- 172.21.15.102:0/1998808220 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f89d8001940 con 0x7f89e4069190 2021-05-25T12:00:01.575 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.530+0000 7f89d27fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:01.576 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.530+0000 7f89d27fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:00:01.576 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.530+0000 7f89d27fc700 10 monclient(hunting): dump: 2021-05-25T12:00:01.576 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:01.576 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:01.576 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:01.577 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:01.577 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:01.577 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:01.577 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:01.578 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:01.578 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:01.578 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:01.578 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.530+0000 7f89d27fc700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:00:01.579 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.530+0000 7f89d27fc700 1 -- 172.21.15.102:0/1998808220 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3868488233 0 0) 0x7f89e411aa30 con 0x7f89e4096ab0 2021-05-25T12:00:01.579 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.530+0000 7f89d27fc700 1 -- 172.21.15.102:0/1998808220 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f89c4004c10 con 0x7f89e4096ab0 2021-05-25T12:00:01.579 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.530+0000 7f89d27fc700 1 -- 172.21.15.102:0/1998808220 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f89e0001d70 con 0x7f89e4096ab0 2021-05-25T12:00:01.579 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.530+0000 7f89d27fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:01.579 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.530+0000 7f89d27fc700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2021-05-25T12:00:01.580 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.530+0000 7f89d27fc700 10 monclient(hunting): dump: 2021-05-25T12:00:01.580 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:01.580 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:01.580 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:01.581 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:01.581 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:01.581 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:01.581 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:01.582 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:01.582 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:01.582 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:01.582 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.530+0000 7f89d27fc700 1 -- 172.21.15.102:0/1998808220 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3955699611 0 0) 0x7f89e411af70 con 0x7f89e4069c80 2021-05-25T12:00:01.583 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.530+0000 7f89d27fc700 1 -- 172.21.15.102:0/1998808220 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f89e411aa30 con 0x7f89e4069c80 2021-05-25T12:00:01.583 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.530+0000 7f89d27fc700 1 -- 172.21.15.102:0/1998808220 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f89d4001bc0 con 0x7f89e4069c80 2021-05-25T12:00:01.583 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.530+0000 7f89d27fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:01.583 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.530+0000 7f89d27fc700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2021-05-25T12:00:01.583 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.530+0000 7f89d27fc700 10 monclient(hunting): dump: 2021-05-25T12:00:01.584 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:01.584 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:01.584 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:01.584 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:01.584 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:01.585 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:01.585 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:01.585 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:01.585 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:01.585 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:01.586 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.530+0000 7f89d27fc700 1 -- 172.21.15.102:0/1998808220 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1750177637 0 0) 0x7f89e411acd0 con 0x7f89e4069190 2021-05-25T12:00:01.586 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.530+0000 7f89d27fc700 1 -- 172.21.15.102:0/1998808220 >> v1:172.21.15.102:6790/0 conn(0x7f89e4096ab0 legacy=0x7f89e41437f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:01.586 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.530+0000 7f89d27fc700 1 -- 172.21.15.102:0/1998808220 >> v1:172.21.15.181:6789/0 conn(0x7f89e4069c80 legacy=0x7f89e41400c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:01.586 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.530+0000 7f89d27fc700 10 monclient: _finish_hunting 0 2021-05-25T12:00:01.587 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.530+0000 7f89d27fc700 1 monclient: found mon.a 2021-05-25T12:00:01.587 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.530+0000 7f89d27fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:00:01.587 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.530+0000 7f89d27fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:00:01.587 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.530+0000 7f89d27fc700 1 -- 172.21.15.102:0/1998808220 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89e4118e00 con 0x7f89e4069190 2021-05-25T12:00:01.588 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.530+0000 7f89d27fc700 10 monclient: _finish_auth 0 2021-05-25T12:00:01.588 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.530+0000 7f89d27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:00:01.588 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.530+0000 7f89ec740700 5 monclient: authenticate success, global_id 4407 2021-05-25T12:00:01.588 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.530+0000 7f89ec740700 10 monclient: _renew_subs 2021-05-25T12:00:01.588 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.530+0000 7f89ec740700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:00:01.589 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.530+0000 7f89ec740700 1 -- 172.21.15.102:0/1998808220 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89e413bb40 con 0x7f89e4069190 2021-05-25T12:00:01.589 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.530+0000 7f89ec740700 10 client.4407.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:00:01.589 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.530+0000 7f89ec740700 10 monclient: _renew_subs 2021-05-25T12:00:01.589 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.530+0000 7f89ec740700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:00:01.590 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.530+0000 7f89ec740700 1 -- 172.21.15.102:0/1998808220 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f89e4119c60 con 0x7f89e4069190 2021-05-25T12:00:01.590 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.530+0000 7f89ec740700 1 librados: init done 2021-05-25T12:00:01.590 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.530+0000 7f89d27fc700 1 -- 172.21.15.102:0/1998808220 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f89d8001eb0 con 0x7f89e4069190 2021-05-25T12:00:01.590 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.530+0000 7f89d27fc700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:00:01.590 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.530+0000 7f89d27fc700 1 -- 172.21.15.102:0/1998808220 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f89d8002240 con 0x7f89e4069190 2021-05-25T12:00:01.591 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.530+0000 7f89d27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:01.591 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.530+0000 7f89d27fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:00:01.591 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.530+0000 7f89e8cd9700 4 set_mon_vals no callback set 2021-05-25T12:00:01.591 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.530+0000 7f89d27fc700 10 monclient: dump: 2021-05-25T12:00:01.591 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:01.592 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:01.592 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:01.592 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:01.592 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:01.592 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:01.593 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:01.593 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:01.593 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:01.593 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:01.594 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.531+0000 7f89d27fc700 1 -- 172.21.15.102:0/1998808220 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f89d8002420 con 0x7f89e4069190 2021-05-25T12:00:01.594 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.531+0000 7f89ec740700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:00:01.594 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.531+0000 7f89ec740700 1 -- 172.21.15.102:0/1998808220 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f89c80065a0 con 0x7f89c8002840 2021-05-25T12:00:01.594 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.531+0000 7f89d27fc700 1 -- 172.21.15.102:0/1998808220 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(15..15 src has 1..15) v4 ==== 3549+0+0 (unknown 3948819498 0 0) 0x7f89d803ccf0 con 0x7f89e4069190 2021-05-25T12:00:01.594 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.531+0000 7f89d27fc700 10 client.4407.objecter ms_dispatch 0x7f89e4001550 osd_map(15..15 src has 1..15) v4 2021-05-25T12:00:01.595 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.531+0000 7f89d27fc700 3 client.4407.objecter handle_osd_map got epochs [15,15] > 0 2021-05-25T12:00:01.595 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.531+0000 7f89d27fc700 3 client.4407.objecter handle_osd_map decoding full epoch 15 2021-05-25T12:00:01.595 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.531+0000 7f89d27fc700 20 client.4407.objecter dump_active .. 0 homeless 2021-05-25T12:00:01.596 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.531+0000 7f89d27fc700 10 client.4407.objecter ms_handle_connect 0x7f89c8002840 2021-05-25T12:00:01.596 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.531+0000 7f89d27fc700 10 client.4407.objecter resend_mon_ops 2021-05-25T12:00:01.596 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.531+0000 7f89d27fc700 1 -- 172.21.15.102:0/1998808220 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1560231254 0 0) 0x7f89c80065a0 con 0x7f89c8002840 2021-05-25T12:00:01.597 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.531+0000 7f89d27fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:01.597 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.531+0000 7f89d27fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:01.598 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.531+0000 7f89d27fc700 10 monclient(hunting): my global_id is 4397 2021-05-25T12:00:01.598 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.531+0000 7f89d27fc700 1 -- 172.21.15.102:0/1998808220 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89e411acd0 con 0x7f89c8002840 2021-05-25T12:00:01.598 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.531+0000 7f89eacdd700 10 monclient: get_auth_request con 0x7f89c4033500 auth_method 0 2021-05-25T12:00:01.599 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.531+0000 7f89d27fc700 1 -- 172.21.15.102:0/1998808220 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3775774154 0 0) 0x7f89e411acd0 con 0x7f89c8002840 2021-05-25T12:00:01.599 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.531+0000 7f89d27fc700 1 -- 172.21.15.102:0/1998808220 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f89c80065a0 con 0x7f89c8002840 2021-05-25T12:00:01.600 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.531+0000 7f89d27fc700 1 -- 172.21.15.102:0/1998808220 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f89d4004830 con 0x7f89c8002840 2021-05-25T12:00:01.600 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.531+0000 7f89d27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:01.601 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.531+0000 7f89d27fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:00:01.601 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.531+0000 7f89d27fc700 10 monclient: dump: 2021-05-25T12:00:01.601 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:01.602 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:01.602 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:01.602 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:01.603 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:01.603 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:01.604 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:01.604 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:01.604 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:01.605 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:01.605 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.531+0000 7f89d27fc700 1 -- 172.21.15.102:0/1998808220 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 939614133 0 0) 0x7f89c80065a0 con 0x7f89c8002840 2021-05-25T12:00:01.605 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.531+0000 7f89d27fc700 1 -- 172.21.15.102:0/1998808220 --> v1:172.21.15.102:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f89c8009e40 con 0x7f89c8002840 2021-05-25T12:00:01.605 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.532+0000 7f89d27fc700 10 client.4407.objecter ms_handle_connect 0x7f89c4033500 2021-05-25T12:00:01.605 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.532+0000 7f89d27fc700 1 -- 172.21.15.102:0/1998808220 <== mon.2 v1:172.21.15.102:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7f89d4006b60 con 0x7f89c8002840 2021-05-25T12:00:01.606 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.532+0000 7f89d27fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:00:01.606 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.532+0000 7f89d27fc700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:00:01.606 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.532+0000 7f89d27fc700 1 -- 172.21.15.102:0/1998808220 >> v1:172.21.15.102:6790/0 conn(0x7f89c8002840 legacy=0x7f89c8004ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:01.606 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.532+0000 7f89d27fc700 1 -- 172.21.15.102:0/1998808220 >> v1:172.21.15.102:6790/0 conn(0x7f89c8002840 legacy=0x7f89c8004ce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:00:01.607 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.538+0000 7f89ec740700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:00:01.607 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.538+0000 7f89ec740700 1 -- 172.21.15.102:0/1998808220 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f89c8007510 con 0x7f89c80068b0 2021-05-25T12:00:01.607 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.539+0000 7f89d27fc700 10 client.4407.objecter ms_handle_connect 0x7f89c80068b0 2021-05-25T12:00:01.607 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.539+0000 7f89d27fc700 10 client.4407.objecter resend_mon_ops 2021-05-25T12:00:01.607 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.539+0000 7f89d27fc700 1 -- 172.21.15.102:0/1998808220 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2930326330 0 0) 0x7f89c8007510 con 0x7f89c80068b0 2021-05-25T12:00:01.608 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.539+0000 7f89d27fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:01.608 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.539+0000 7f89d27fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:01.608 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.539+0000 7f89d27fc700 10 monclient(hunting): my global_id is 4400 2021-05-25T12:00:01.608 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.539+0000 7f89d27fc700 1 -- 172.21.15.102:0/1998808220 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89c80065a0 con 0x7f89c80068b0 2021-05-25T12:00:01.609 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.539+0000 7f89d27fc700 1 -- 172.21.15.102:0/1998808220 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1032573245 0 0) 0x7f89c80065a0 con 0x7f89c80068b0 2021-05-25T12:00:01.609 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.539+0000 7f89d27fc700 1 -- 172.21.15.102:0/1998808220 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f89c8007510 con 0x7f89c80068b0 2021-05-25T12:00:01.609 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.539+0000 7f89d27fc700 1 -- 172.21.15.102:0/1998808220 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f89d4004f20 con 0x7f89c80068b0 2021-05-25T12:00:01.609 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.539+0000 7f89d27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:01.610 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.539+0000 7f89d27fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:00:01.610 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.539+0000 7f89d27fc700 10 monclient: dump: 2021-05-25T12:00:01.610 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:01.610 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:01.611 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:01.611 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:01.611 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:01.611 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:01.611 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:01.612 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:01.612 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:01.612 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:01.612 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.539+0000 7f89d27fc700 1 -- 172.21.15.102:0/1998808220 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 985757528 0 0) 0x7f89c8007510 con 0x7f89c80068b0 2021-05-25T12:00:01.613 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.539+0000 7f89d27fc700 1 -- 172.21.15.102:0/1998808220 --> v1:172.21.15.102:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f89c8008770 con 0x7f89c80068b0 2021-05-25T12:00:01.613 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.539+0000 7f89d27fc700 1 -- 172.21.15.102:0/1998808220 <== mon.2 v1:172.21.15.102:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3940 (unknown 0 0 3227053038) 0x7f89d40051a0 con 0x7f89c80068b0 2021-05-25T12:00:01.613 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.539+0000 7f89d27fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:00:01.613 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.539+0000 7f89d27fc700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:00:01.613 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.539+0000 7f89d27fc700 1 -- 172.21.15.102:0/1998808220 >> v1:172.21.15.102:6790/0 conn(0x7f89c80068b0 legacy=0x7f89c8006d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:01.614 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.539+0000 7f89d27fc700 1 -- 172.21.15.102:0/1998808220 >> v1:172.21.15.102:6790/0 conn(0x7f89c80068b0 legacy=0x7f89c8006d40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:00:01.615 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:00:01.615 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:00:01.615 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:00:01.615 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "peon", 2021-05-25T12:00:01.616 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 8, 2021-05-25T12:00:01.616 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:00:01.616 INFO:teuthology.orchestra.run.smithi102.stdout: 0, 2021-05-25T12:00:01.616 INFO:teuthology.orchestra.run.smithi102.stdout: 1, 2021-05-25T12:00:01.616 INFO:teuthology.orchestra.run.smithi102.stdout: 2 2021-05-25T12:00:01.617 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:00:01.617 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 20, 2021-05-25T12:00:01.617 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:00:01.617 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:00:01.618 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:00:01.618 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:00:01.618 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:00:01.619 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:00:01.619 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:00:01.620 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:00:01.620 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:00:01.620 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:00:01.621 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:00:01.621 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:00:01.622 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:00:01.622 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:00:01.622 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:00:01.623 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:00:01.623 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:00:01.624 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:00:01.624 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:00:01.624 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:00:01.625 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:00:01.625 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:00:01.626 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:00:01.626 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:00:01.627 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:00:01.627 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:01.627 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:00:01.628 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:00:01.628 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:00:01.629 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:00:01.629 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:00:01.630 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:00:01.630 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:00:01.630 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:00:01.631 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:00:01.631 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:00:01.632 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:00:01.632 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:00:01.633 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:00:01.633 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:00:01.634 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:00:01.634 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:00:01.634 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:00:01.635 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:00:01.635 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:00:01.636 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:00:01.636 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:00:01.637 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:00:01.637 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:00:01.638 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:00:01.638 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:00:01.638 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:00:01.639 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:01.639 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:00:01.640 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:01.640 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:00:01.641 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:00:01.641 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:00:01.641 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:00:01.642 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:01.642 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:00:01.643 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:00:01.643 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:00:01.644 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:01.644 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:00:01.645 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:01.645 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:00:01.645 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:00:01.646 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:00:01.646 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:00:01.647 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:00:01.647 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:01.648 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:01.648 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:00:01.648 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:00:01.649 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:00:01.649 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:00:01.650 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:01.650 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:00:01.651 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:00:01.651 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:00:01.652 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:01.652 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:00:01.652 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:01.653 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:00:01.653 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:00:01.654 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:00:01.654 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:00:01.654 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:00:01.655 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:01.655 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:01.655 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:00:01.655 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:00:01.655 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:00:01.656 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:00:01.656 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:01.656 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:00:01.656 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:00:01.656 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:00:01.657 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:01.657 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:00:01.657 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:01.657 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:00:01.657 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:00:01.658 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:00:01.658 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:00:01.658 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:00:01.658 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:01.658 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:00:01.659 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:01.659 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:00:01.659 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:00:01.659 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:01.659 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:00:01.660 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:00:01.660 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:00:01.660 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:01.660 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:00:01.660 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": [ 2021-05-25T12:00:01.661 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:01.661 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:00:01.661 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:00:01.661 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 2 2021-05-25T12:00:01.661 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:01.662 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:00:01.662 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:00:01.662 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:01.662 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:00:01.663 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:00:01.663 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:00:01.663 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:01.663 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:00:01.663 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:01.664 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:00:01.664 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:00:01.666 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.539+0000 7f89ec740700 10 librados: watch_flush enter 2021-05-25T12:00:01.666 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.539+0000 7f89ec740700 10 librados: watch_flush exit 2021-05-25T12:00:01.667 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.539+0000 7f89ec740700 20 client.4407.objecter shutdown clearing up homeless session... 2021-05-25T12:00:01.667 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.539+0000 7f89ec740700 10 client.4407.objecter successfully canceled tick 2021-05-25T12:00:01.667 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.540+0000 7f89ec740700 1 -- 172.21.15.102:0/1998808220 >> v1:172.21.15.181:6812/29582 conn(0x7f89c4033500 legacy=0x7f89c40359c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:01.667 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.540+0000 7f89ec740700 10 monclient: shutdown 2021-05-25T12:00:01.667 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.540+0000 7f89ec740700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:00:01.668 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.540+0000 7f89ec740700 1 -- 172.21.15.102:0/1998808220 >> v1:172.21.15.102:6789/0 conn(0x7f89e4069190 legacy=0x7f89e413b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:01.668 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.540+0000 7f89eacdd700 1 -- 172.21.15.102:0/1998808220 reap_dead start 2021-05-25T12:00:01.668 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.540+0000 7f89eacdd700 1 -- 172.21.15.102:0/1998808220 reap_dead start 2021-05-25T12:00:01.668 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.540+0000 7f89ec740700 1 -- 172.21.15.102:0/1998808220 shutdown_connections 2021-05-25T12:00:01.669 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.540+0000 7f89ec740700 1 -- 172.21.15.102:0/1998808220 >> 172.21.15.102:0/1998808220 conn(0x7f89e409af90 msgr2=0x7f89e412eac0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:00:01.669 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.540+0000 7f89ec740700 1 -- 172.21.15.102:0/1998808220 shutdown_connections 2021-05-25T12:00:01.669 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.540+0000 7f89ec740700 1 -- 172.21.15.102:0/1998808220 wait complete. 2021-05-25T12:00:01.669 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.540+0000 7f89ec740700 1 librados: shutdown 2021-05-25T12:00:01.670 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2021-05-25T12:00:01.800 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.778+0000 7fb9854aa700 10 monclient: get_monmap_and_config 2021-05-25T12:00:01.800 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.778+0000 7fb9854aa700 10 monclient: build_initial_monmap 2021-05-25T12:00:01.800 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.778+0000 7fb9854aa700 10 monclient: monmap: 2021-05-25T12:00:01.801 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:00:01.801 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:01.801 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:00:01.780012+0000 2021-05-25T12:00:01.801 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:00:01.780012+0000 2021-05-25T12:00:01.802 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:00:01.802 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:01.802 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:00:01.802 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:00:01.803 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:00:01.803 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:01.803 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.778+0000 7fb9854aa700 1 Processor -- start 2021-05-25T12:00:01.804 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.778+0000 7fb9854aa700 1 -- start start 2021-05-25T12:00:01.804 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.778+0000 7fb9854aa700 10 monclient: init 2021-05-25T12:00:01.804 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.780+0000 7fb9854aa700 10 monclient: _reopen_session rank -1 2021-05-25T12:00:01.805 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.780+0000 7fb9854aa700 10 monclient: _add_conns ranks=[1,0,2] 2021-05-25T12:00:01.805 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.780+0000 7fb9854aa700 10 monclient(hunting): picked mon.noname-c con 0x7fb980063840 addr v1:172.21.15.181:6789/0 2021-05-25T12:00:01.805 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.780+0000 7fb9854aa700 10 monclient(hunting): picked mon.noname-a con 0x7fb980094da0 addr v1:172.21.15.102:6789/0 2021-05-25T12:00:01.805 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.780+0000 7fb9854aa700 10 monclient(hunting): picked mon.noname-b con 0x7fb9801323c0 addr v1:172.21.15.102:6790/0 2021-05-25T12:00:01.806 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.780+0000 7fb9854aa700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb980063fb0 con 0x7fb980094da0 2021-05-25T12:00:01.806 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.780+0000 7fb9854aa700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9800982f0 con 0x7fb980063840 2021-05-25T12:00:01.806 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.780+0000 7fb9854aa700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb980064510 con 0x7fb9801323c0 2021-05-25T12:00:01.806 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.780+0000 7fb9854aa700 10 monclient(hunting): _renew_subs 2021-05-25T12:00:01.806 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.780+0000 7fb9854aa700 10 monclient(hunting): authenticate will time out at 802.377125s 2021-05-25T12:00:01.807 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.780+0000 7fb97f7fe700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fb9801323c0 0x7fb9801347b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:54018/0 (socket says 172.21.15.102:54018) 2021-05-25T12:00:01.807 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.780+0000 7fb97f7fe700 1 -- 172.21.15.102:0/2317712019 learned_addr learned my addr 172.21.15.102:0/2317712019 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:00:01.807 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.780+0000 7fb97dffb700 1 -- 172.21.15.102:0/2317712019 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2183495059 0 0) 0x7fb980063fb0 con 0x7fb980094da0 2021-05-25T12:00:01.807 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.780+0000 7fb97dffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:01.808 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.780+0000 7fb97dffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:01.808 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.780+0000 7fb97dffb700 10 monclient(hunting): my global_id is 4410 2021-05-25T12:00:01.808 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.780+0000 7fb97dffb700 1 -- 172.21.15.102:0/2317712019 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb96c003270 con 0x7fb980094da0 2021-05-25T12:00:01.808 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.780+0000 7fb97dffb700 1 -- 172.21.15.102:0/2317712019 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4277579865 0 0) 0x7fb980064510 con 0x7fb9801323c0 2021-05-25T12:00:01.808 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.780+0000 7fb97dffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:01.809 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.780+0000 7fb97dffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:01.809 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.780+0000 7fb97dffb700 10 monclient(hunting): my global_id is 4403 2021-05-25T12:00:01.809 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.780+0000 7fb97dffb700 1 -- 172.21.15.102:0/2317712019 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb980063fb0 con 0x7fb9801323c0 2021-05-25T12:00:01.810 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.780+0000 7fb97dffb700 1 -- 172.21.15.102:0/2317712019 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3366502940 0 0) 0x7fb9800982f0 con 0x7fb980063840 2021-05-25T12:00:01.810 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.780+0000 7fb97dffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:01.810 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.780+0000 7fb97dffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:01.810 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.780+0000 7fb97dffb700 10 monclient(hunting): my global_id is 4399 2021-05-25T12:00:01.811 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.780+0000 7fb97dffb700 1 -- 172.21.15.102:0/2317712019 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb980064510 con 0x7fb980063840 2021-05-25T12:00:01.811 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.780+0000 7fb97dffb700 1 -- 172.21.15.102:0/2317712019 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4270649960 0 0) 0x7fb96c003270 con 0x7fb980094da0 2021-05-25T12:00:01.811 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.780+0000 7fb97dffb700 1 -- 172.21.15.102:0/2317712019 >> v1:172.21.15.102:6790/0 conn(0x7fb9801323c0 legacy=0x7fb9801347b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:01.811 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.780+0000 7fb97dffb700 1 -- 172.21.15.102:0/2317712019 >> v1:172.21.15.181:6789/0 conn(0x7fb980063840 legacy=0x7fb9800974d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:01.811 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.780+0000 7fb97dffb700 10 monclient: _finish_hunting 0 2021-05-25T12:00:01.812 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.780+0000 7fb97dffb700 1 monclient: found mon.noname-a 2021-05-25T12:00:01.812 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.780+0000 7fb97dffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:00:01.812 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.780+0000 7fb97dffb700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T12:00:01.812 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.780+0000 7fb97dffb700 1 -- 172.21.15.102:0/2317712019 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9800647b0 con 0x7fb980094da0 2021-05-25T12:00:01.812 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.780+0000 7fb97dffb700 10 monclient: _finish_auth 0 2021-05-25T12:00:01.813 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.780+0000 7fb97dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:00:01.813 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.780+0000 7fb9854aa700 5 monclient: authenticate success, global_id 4410 2021-05-25T12:00:01.813 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.780+0000 7fb97dffb700 1 -- 172.21.15.102:0/2317712019 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fb968001610 con 0x7fb980094da0 2021-05-25T12:00:01.814 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.780+0000 7fb9854aa700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:00:01.814 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.780+0000 7fb97dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:01.814 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.780+0000 7fb97dffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:00:01.814 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.780+0000 7fb97dffb700 10 monclient: dump: 2021-05-25T12:00:01.814 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:01.815 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:01.815 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:01.815 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:01.815 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:01.816 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:01.816 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:01.816 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:01.816 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:01.816 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:01.817 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.780+0000 7fb9854aa700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:00:01.817 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.780+0000 7fb97dffb700 1 -- 172.21.15.102:0/2317712019 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb968001800 con 0x7fb980094da0 2021-05-25T12:00:01.817 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.780+0000 7fb97dffb700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:00:01.817 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.780+0000 7fb9854aa700 10 monclient: get_monmap_and_config success 2021-05-25T12:00:01.818 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.780+0000 7fb97dffb700 1 -- 172.21.15.102:0/2317712019 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fb968001b90 con 0x7fb980094da0 2021-05-25T12:00:01.818 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.780+0000 7fb97dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:01.818 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.780+0000 7fb9854aa700 4 set_mon_vals no callback set 2021-05-25T12:00:01.818 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.780+0000 7fb97dffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:00:01.819 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.780+0000 7fb97dffb700 10 monclient: dump: 2021-05-25T12:00:01.819 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:01.819 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:01.819 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:01.820 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:01.820 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:01.820 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:01.820 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:01.820 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:01.821 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:01.821 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:01.821 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.780+0000 7fb9854aa700 10 monclient: shutdown 2021-05-25T12:00:01.821 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.780+0000 7fb9854aa700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:00:01.822 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.780+0000 7fb9854aa700 1 -- 172.21.15.102:0/2317712019 >> v1:172.21.15.102:6789/0 conn(0x7fb980094da0 legacy=0x7fb980097be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:01.822 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.780+0000 7fb9854aa700 1 -- 172.21.15.102:0/2317712019 shutdown_connections 2021-05-25T12:00:01.822 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.780+0000 7fb9854aa700 1 -- 172.21.15.102:0/2317712019 >> 172.21.15.102:0/2317712019 conn(0x7fb98009cb30 msgr2=0x7fb98009cf40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:00:01.822 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.780+0000 7fb9854aa700 1 -- 172.21.15.102:0/2317712019 shutdown_connections 2021-05-25T12:00:01.823 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.780+0000 7fb9854aa700 1 -- 172.21.15.102:0/2317712019 wait complete. 2021-05-25T12:00:01.823 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.781+0000 7fb9854aa700 10 monclient: build_initial_monmap 2021-05-25T12:00:01.823 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.781+0000 7fb9854aa700 10 monclient: monmap: 2021-05-25T12:00:01.823 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:00:01.823 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:00:01.824 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:00:01.824 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:00:01.824 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:00:01.824 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:00:01.825 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:00:01.825 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:00:01.825 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:00:01.825 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:01.825 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.781+0000 7fb9854aa700 1 librados: starting msgr at 2021-05-25T12:00:01.826 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.781+0000 7fb9854aa700 1 librados: starting objecter 2021-05-25T12:00:01.826 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.781+0000 7fb9854aa700 1 Processor -- start 2021-05-25T12:00:01.826 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.781+0000 7fb9854aa700 1 -- start start 2021-05-25T12:00:01.826 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.781+0000 7fb9854aa700 1 librados: setting wanted keys 2021-05-25T12:00:01.827 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.781+0000 7fb9854aa700 1 librados: calling monclient init 2021-05-25T12:00:01.827 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.781+0000 7fb9854aa700 10 monclient: init 2021-05-25T12:00:01.827 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.781+0000 7fb9854aa700 10 monclient: _reopen_session rank -1 2021-05-25T12:00:01.827 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.781+0000 7fb9854aa700 10 monclient: _add_conns ranks=[0,2,1] 2021-05-25T12:00:01.827 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.781+0000 7fb9854aa700 10 monclient(hunting): picked mon.noname-a con 0x7fb980063840 addr v1:172.21.15.102:6789/0 2021-05-25T12:00:01.828 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.781+0000 7fb9854aa700 10 monclient(hunting): picked mon.noname-c con 0x7fb980094da0 addr v1:172.21.15.102:6790/0 2021-05-25T12:00:01.828 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.781+0000 7fb9854aa700 10 monclient(hunting): picked mon.noname-b con 0x7fb9801323c0 addr v1:172.21.15.181:6789/0 2021-05-25T12:00:01.828 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.781+0000 7fb9854aa700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb98011a510 con 0x7fb980063840 2021-05-25T12:00:01.828 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.781+0000 7fb9854aa700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb98011c2e0 con 0x7fb9801323c0 2021-05-25T12:00:01.829 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.781+0000 7fb9854aa700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb98011c580 con 0x7fb980094da0 2021-05-25T12:00:01.829 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.781+0000 7fb9854aa700 10 monclient(hunting): _renew_subs 2021-05-25T12:00:01.829 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.781+0000 7fb9854aa700 10 monclient(hunting): authenticate will time out at 802.379844s 2021-05-25T12:00:01.829 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.781+0000 7fb97effd700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fb980063840 0x7fb980118990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:59142/0 (socket says 172.21.15.102:59142) 2021-05-25T12:00:01.830 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.781+0000 7fb97effd700 1 -- 172.21.15.102:0/629903046 learned_addr learned my addr 172.21.15.102:0/629903046 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:00:01.830 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.781+0000 7fb966ffd700 10 client.?.objecter ms_handle_connect 0x7fb980063840 2021-05-25T12:00:01.830 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.782+0000 7fb966ffd700 10 client.?.objecter resend_mon_ops 2021-05-25T12:00:01.830 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.782+0000 7fb966ffd700 10 client.?.objecter ms_handle_connect 0x7fb980094da0 2021-05-25T12:00:01.830 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.782+0000 7fb966ffd700 10 client.?.objecter resend_mon_ops 2021-05-25T12:00:01.831 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.782+0000 7fb966ffd700 1 -- 172.21.15.102:0/629903046 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 5404692 0 0) 0x7fb98011a510 con 0x7fb980063840 2021-05-25T12:00:01.831 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.782+0000 7fb966ffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:01.831 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.782+0000 7fb966ffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:01.831 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.782+0000 7fb966ffd700 10 monclient(hunting): my global_id is 4413 2021-05-25T12:00:01.831 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.782+0000 7fb966ffd700 1 -- 172.21.15.102:0/629903046 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb960004c10 con 0x7fb980063840 2021-05-25T12:00:01.832 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.782+0000 7fb966ffd700 10 client.?.objecter ms_handle_connect 0x7fb9801323c0 2021-05-25T12:00:01.832 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.782+0000 7fb966ffd700 10 client.?.objecter resend_mon_ops 2021-05-25T12:00:01.832 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.782+0000 7fb966ffd700 1 -- 172.21.15.102:0/629903046 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4264825440 0 0) 0x7fb98011c580 con 0x7fb980094da0 2021-05-25T12:00:01.832 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.782+0000 7fb966ffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:01.833 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.782+0000 7fb966ffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:01.833 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.782+0000 7fb966ffd700 10 monclient(hunting): my global_id is 4406 2021-05-25T12:00:01.833 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.782+0000 7fb966ffd700 1 -- 172.21.15.102:0/629903046 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb98011a510 con 0x7fb980094da0 2021-05-25T12:00:01.833 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.782+0000 7fb966ffd700 1 -- 172.21.15.102:0/629903046 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 318362896 0 0) 0x7fb98011c2e0 con 0x7fb9801323c0 2021-05-25T12:00:01.833 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.782+0000 7fb966ffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:01.834 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.782+0000 7fb966ffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:01.834 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.782+0000 7fb966ffd700 10 monclient(hunting): my global_id is 4402 2021-05-25T12:00:01.834 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.782+0000 7fb966ffd700 1 -- 172.21.15.102:0/629903046 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb98011c580 con 0x7fb9801323c0 2021-05-25T12:00:01.834 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.782+0000 7fb966ffd700 1 -- 172.21.15.102:0/629903046 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2249468402 0 0) 0x7fb98011a510 con 0x7fb980094da0 2021-05-25T12:00:01.835 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.782+0000 7fb966ffd700 1 -- 172.21.15.102:0/629903046 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb98011c2e0 con 0x7fb980094da0 2021-05-25T12:00:01.835 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.782+0000 7fb966ffd700 1 -- 172.21.15.102:0/629903046 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fb968001b50 con 0x7fb980094da0 2021-05-25T12:00:01.835 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.782+0000 7fb966ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:01.835 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.782+0000 7fb966ffd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:00:01.835 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.782+0000 7fb966ffd700 10 monclient(hunting): dump: 2021-05-25T12:00:01.836 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:01.836 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:01.836 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:01.836 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:01.837 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:01.837 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:01.837 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:01.837 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:01.837 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:01.838 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:01.838 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.782+0000 7fb966ffd700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:00:01.838 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.782+0000 7fb966ffd700 1 -- 172.21.15.102:0/629903046 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4107297693 0 0) 0x7fb960004c10 con 0x7fb980063840 2021-05-25T12:00:01.838 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.782+0000 7fb966ffd700 1 -- 172.21.15.102:0/629903046 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb98011a510 con 0x7fb980063840 2021-05-25T12:00:01.838 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.782+0000 7fb966ffd700 1 -- 172.21.15.102:0/629903046 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fb9740028f0 con 0x7fb980063840 2021-05-25T12:00:01.839 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.782+0000 7fb966ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:01.839 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.782+0000 7fb966ffd700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2021-05-25T12:00:01.839 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.782+0000 7fb966ffd700 10 monclient(hunting): dump: 2021-05-25T12:00:01.839 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:01.840 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:01.840 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:01.840 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:01.840 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:01.840 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:01.841 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:01.841 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:01.841 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:01.841 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:01.841 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.782+0000 7fb966ffd700 1 -- 172.21.15.102:0/629903046 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2455029357 0 0) 0x7fb98011c580 con 0x7fb9801323c0 2021-05-25T12:00:01.842 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.782+0000 7fb966ffd700 1 -- 172.21.15.102:0/629903046 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb960004c10 con 0x7fb9801323c0 2021-05-25T12:00:01.842 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.782+0000 7fb966ffd700 1 -- 172.21.15.102:0/629903046 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fb970001c30 con 0x7fb9801323c0 2021-05-25T12:00:01.842 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.782+0000 7fb966ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:01.842 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.782+0000 7fb966ffd700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2021-05-25T12:00:01.843 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.782+0000 7fb966ffd700 10 monclient(hunting): dump: 2021-05-25T12:00:01.843 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:01.843 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:01.843 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:01.843 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:01.844 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:01.844 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:01.844 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:01.844 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:01.845 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:01.845 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:01.845 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.782+0000 7fb966ffd700 1 -- 172.21.15.102:0/629903046 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2034610897 0 0) 0x7fb98011c2e0 con 0x7fb980094da0 2021-05-25T12:00:01.845 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.782+0000 7fb966ffd700 1 -- 172.21.15.102:0/629903046 >> v1:172.21.15.181:6789/0 conn(0x7fb9801323c0 legacy=0x7fb9801197b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:01.845 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.782+0000 7fb966ffd700 1 -- 172.21.15.102:0/629903046 >> v1:172.21.15.102:6789/0 conn(0x7fb980063840 legacy=0x7fb980118990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:01.846 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.782+0000 7fb966ffd700 10 monclient: _finish_hunting 0 2021-05-25T12:00:01.846 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.782+0000 7fb966ffd700 1 monclient: found mon.c 2021-05-25T12:00:01.846 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.782+0000 7fb966ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:00:01.846 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.782+0000 7fb966ffd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:00:01.846 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.782+0000 7fb966ffd700 1 -- 172.21.15.102:0/629903046 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9801311f0 con 0x7fb980094da0 2021-05-25T12:00:01.847 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.782+0000 7fb966ffd700 10 monclient: _finish_auth 0 2021-05-25T12:00:01.847 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.782+0000 7fb966ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:00:01.847 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.782+0000 7fb9854aa700 5 monclient: authenticate success, global_id 4406 2021-05-25T12:00:01.847 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.782+0000 7fb9854aa700 10 monclient: _renew_subs 2021-05-25T12:00:01.848 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.782+0000 7fb9854aa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:00:01.848 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.782+0000 7fb9854aa700 1 -- 172.21.15.102:0/629903046 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb98011c940 con 0x7fb980094da0 2021-05-25T12:00:01.848 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.782+0000 7fb9854aa700 10 client.4406.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:00:01.848 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.782+0000 7fb9854aa700 10 monclient: _renew_subs 2021-05-25T12:00:01.848 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.782+0000 7fb9854aa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:00:01.849 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.782+0000 7fb9854aa700 1 -- 172.21.15.102:0/629903046 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb98011b0d0 con 0x7fb980094da0 2021-05-25T12:00:01.849 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.782+0000 7fb9854aa700 1 librados: init done 2021-05-25T12:00:01.849 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.782+0000 7fb966ffd700 1 -- 172.21.15.102:0/629903046 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb968002110 con 0x7fb980094da0 2021-05-25T12:00:01.849 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.782+0000 7fb966ffd700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:00:01.849 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.782+0000 7fb966ffd700 1 -- 172.21.15.102:0/629903046 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fb968002420 con 0x7fb980094da0 2021-05-25T12:00:01.850 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.782+0000 7fb966ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:01.850 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.782+0000 7fb966ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:00:01.850 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.782+0000 7fb97d7fa700 4 set_mon_vals no callback set 2021-05-25T12:00:01.850 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.782+0000 7fb966ffd700 10 monclient: dump: 2021-05-25T12:00:01.851 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:01.851 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:01.851 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:01.851 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:01.851 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:01.852 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:01.852 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:01.852 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:01.852 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:01.853 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:01.853 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.783+0000 7fb9854aa700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:00:01.853 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.783+0000 7fb9854aa700 1 -- 172.21.15.102:0/629903046 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fb9580065a0 con 0x7fb958002840 2021-05-25T12:00:01.853 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.783+0000 7fb966ffd700 1 -- 172.21.15.102:0/629903046 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7fb968002600 con 0x7fb980094da0 2021-05-25T12:00:01.853 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.783+0000 7fb966ffd700 1 -- 172.21.15.102:0/629903046 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(15..15 src has 1..15) v4 ==== 3549+0+0 (unknown 3948819498 0 0) 0x7fb96803a8a0 con 0x7fb980094da0 2021-05-25T12:00:01.854 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.783+0000 7fb966ffd700 10 client.4406.objecter ms_dispatch 0x7fb980001550 osd_map(15..15 src has 1..15) v4 2021-05-25T12:00:01.854 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.783+0000 7fb966ffd700 3 client.4406.objecter handle_osd_map got epochs [15,15] > 0 2021-05-25T12:00:01.854 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.783+0000 7fb966ffd700 3 client.4406.objecter handle_osd_map decoding full epoch 15 2021-05-25T12:00:01.854 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.783+0000 7fb966ffd700 20 client.4406.objecter dump_active .. 0 homeless 2021-05-25T12:00:01.854 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.783+0000 7fb966ffd700 10 client.4406.objecter ms_handle_connect 0x7fb958002840 2021-05-25T12:00:01.855 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.783+0000 7fb966ffd700 10 client.4406.objecter resend_mon_ops 2021-05-25T12:00:01.855 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.783+0000 7fb97f7fe700 10 monclient: get_auth_request con 0x7fb960033500 auth_method 0 2021-05-25T12:00:01.855 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.783+0000 7fb966ffd700 1 -- 172.21.15.102:0/629903046 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4197256949 0 0) 0x7fb9580065a0 con 0x7fb958002840 2021-05-25T12:00:01.855 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.783+0000 7fb966ffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:01.856 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.783+0000 7fb966ffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:01.856 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.783+0000 7fb966ffd700 10 monclient(hunting): my global_id is 4405 2021-05-25T12:00:01.856 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.783+0000 7fb966ffd700 1 -- 172.21.15.102:0/629903046 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb98011c2e0 con 0x7fb958002840 2021-05-25T12:00:01.856 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.784+0000 7fb966ffd700 10 client.4406.objecter ms_handle_connect 0x7fb960033500 2021-05-25T12:00:01.856 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.784+0000 7fb966ffd700 1 -- 172.21.15.102:0/629903046 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3225659164 0 0) 0x7fb98011c2e0 con 0x7fb958002840 2021-05-25T12:00:01.857 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.784+0000 7fb966ffd700 1 -- 172.21.15.102:0/629903046 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb9580065a0 con 0x7fb958002840 2021-05-25T12:00:01.857 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.784+0000 7fb966ffd700 1 -- 172.21.15.102:0/629903046 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fb974001740 con 0x7fb958002840 2021-05-25T12:00:01.857 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.784+0000 7fb966ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:01.857 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.784+0000 7fb966ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T12:00:01.857 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.784+0000 7fb966ffd700 10 monclient: dump: 2021-05-25T12:00:01.858 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:01.858 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:01.858 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:01.858 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:01.858 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:01.859 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:01.859 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:01.859 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:01.859 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:01.859 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:01.860 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.784+0000 7fb966ffd700 1 -- 172.21.15.102:0/629903046 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2450559935 0 0) 0x7fb9580065a0 con 0x7fb958002840 2021-05-25T12:00:01.860 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.784+0000 7fb966ffd700 1 -- 172.21.15.102:0/629903046 --> v1:172.21.15.181:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb958009e40 con 0x7fb958002840 2021-05-25T12:00:01.860 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.784+0000 7fb966ffd700 1 -- 172.21.15.102:0/629903046 <== mon.1 v1:172.21.15.181:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7fb9740019f0 con 0x7fb958002840 2021-05-25T12:00:01.860 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.784+0000 7fb966ffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:00:01.861 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.784+0000 7fb966ffd700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:00:01.861 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.784+0000 7fb966ffd700 1 -- 172.21.15.102:0/629903046 >> v1:172.21.15.181:6789/0 conn(0x7fb958002840 legacy=0x7fb958004ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:01.861 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.784+0000 7fb966ffd700 1 -- 172.21.15.102:0/629903046 >> v1:172.21.15.181:6789/0 conn(0x7fb958002840 legacy=0x7fb958004ce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:00:01.861 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.791+0000 7fb9854aa700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:00:01.861 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.791+0000 7fb9854aa700 1 -- 172.21.15.102:0/629903046 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fb958007510 con 0x7fb9580068b0 2021-05-25T12:00:01.862 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.791+0000 7fb966ffd700 10 client.4406.objecter ms_handle_connect 0x7fb9580068b0 2021-05-25T12:00:01.862 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.791+0000 7fb966ffd700 10 client.4406.objecter resend_mon_ops 2021-05-25T12:00:01.862 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.792+0000 7fb966ffd700 1 -- 172.21.15.102:0/629903046 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1485423098 0 0) 0x7fb958007510 con 0x7fb9580068b0 2021-05-25T12:00:01.862 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.792+0000 7fb966ffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:01.863 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.792+0000 7fb966ffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:01.863 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.792+0000 7fb966ffd700 10 monclient(hunting): my global_id is 4408 2021-05-25T12:00:01.863 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.792+0000 7fb966ffd700 1 -- 172.21.15.102:0/629903046 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9580065a0 con 0x7fb9580068b0 2021-05-25T12:00:01.863 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.792+0000 7fb966ffd700 1 -- 172.21.15.102:0/629903046 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3512773913 0 0) 0x7fb9580065a0 con 0x7fb9580068b0 2021-05-25T12:00:01.863 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.792+0000 7fb966ffd700 1 -- 172.21.15.102:0/629903046 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb958007510 con 0x7fb9580068b0 2021-05-25T12:00:01.864 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.792+0000 7fb966ffd700 1 -- 172.21.15.102:0/629903046 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fb974001ff0 con 0x7fb9580068b0 2021-05-25T12:00:01.864 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.792+0000 7fb966ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:01.864 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.792+0000 7fb966ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T12:00:01.864 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.792+0000 7fb966ffd700 10 monclient: dump: 2021-05-25T12:00:01.864 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:01.865 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:01.865 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:01.865 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:01.865 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:01.865 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:01.866 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:01.866 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:01.866 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:01.866 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:01.867 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.792+0000 7fb966ffd700 1 -- 172.21.15.102:0/629903046 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 705087928 0 0) 0x7fb958007510 con 0x7fb9580068b0 2021-05-25T12:00:01.867 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.792+0000 7fb966ffd700 1 -- 172.21.15.102:0/629903046 --> v1:172.21.15.181:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fb958008770 con 0x7fb9580068b0 2021-05-25T12:00:01.867 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.792+0000 7fb966ffd700 1 -- 172.21.15.102:0/629903046 <== mon.1 v1:172.21.15.181:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4110 (unknown 0 0 2803202256) 0x7fb974001e10 con 0x7fb9580068b0 2021-05-25T12:00:01.867 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.792+0000 7fb966ffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:00:01.867 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.792+0000 7fb966ffd700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:00:01.868 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.792+0000 7fb966ffd700 1 -- 172.21.15.102:0/629903046 >> v1:172.21.15.181:6789/0 conn(0x7fb9580068b0 legacy=0x7fb958006d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:01.868 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.792+0000 7fb966ffd700 1 -- 172.21.15.102:0/629903046 >> v1:172.21.15.181:6789/0 conn(0x7fb9580068b0 legacy=0x7fb958006d40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:00:01.868 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:00:01.869 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:00:01.869 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:00:01.869 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "peon", 2021-05-25T12:00:01.869 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 8, 2021-05-25T12:00:01.870 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:00:01.870 INFO:teuthology.orchestra.run.smithi102.stdout: 0, 2021-05-25T12:00:01.870 INFO:teuthology.orchestra.run.smithi102.stdout: 1, 2021-05-25T12:00:01.870 INFO:teuthology.orchestra.run.smithi102.stdout: 2 2021-05-25T12:00:01.870 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:00:01.871 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 20, 2021-05-25T12:00:01.871 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:00:01.871 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:00:01.871 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:00:01.871 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:00:01.872 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:00:01.872 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:00:01.872 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:00:01.872 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:00:01.873 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:00:01.873 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:00:01.873 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:00:01.873 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:00:01.873 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:00:01.874 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:00:01.874 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:00:01.874 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:00:01.874 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:00:01.874 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:00:01.875 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:00:01.875 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:00:01.875 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:00:01.875 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:00:01.876 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:00:01.876 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:00:01.876 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:00:01.876 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:01.876 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:00:01.877 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:00:01.877 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:00:01.877 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:00:01.877 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:00:01.878 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:00:01.878 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:00:01.878 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:00:01.878 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:00:01.879 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:00:01.879 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:00:01.879 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:00:01.879 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:00:01.879 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:00:01.880 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:00:01.880 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:00:01.880 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:00:01.880 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:00:01.880 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:00:01.881 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:00:01.881 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:00:01.881 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:00:01.881 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:00:01.882 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:00:01.882 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:00:01.882 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:00:01.882 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:01.882 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:00:01.883 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:01.883 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:00:01.883 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:00:01.883 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:00:01.883 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:00:01.884 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:01.884 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:00:01.884 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:00:01.884 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:00:01.885 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:01.885 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:00:01.885 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:01.885 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:00:01.885 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:00:01.886 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:00:01.886 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:00:01.886 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:00:01.886 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:01.886 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:01.887 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:00:01.887 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:00:01.887 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:00:01.887 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:00:01.888 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:01.888 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:00:01.888 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:00:01.888 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:00:01.888 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:01.889 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:00:01.889 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:01.889 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:00:01.889 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:00:01.889 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:00:01.890 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:00:01.890 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:00:01.890 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:01.890 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:01.890 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:00:01.891 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:00:01.891 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:00:01.891 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:00:01.891 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:01.891 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:00:01.892 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:00:01.892 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:00:01.892 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:01.892 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:00:01.892 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:01.893 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:00:01.893 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:00:01.893 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:00:01.893 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:00:01.894 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:00:01.894 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:01.894 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:00:01.894 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:01.895 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:00:01.895 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:00:01.895 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:01.895 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:00:01.895 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:00:01.896 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:00:01.896 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:01.896 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:00:01.896 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": [ 2021-05-25T12:00:01.896 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:01.897 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:00:01.897 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:00:01.897 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 3 2021-05-25T12:00:01.897 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:01.898 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:00:01.898 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:00:01.898 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:01.898 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:00:01.899 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:00:01.899 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 2 2021-05-25T12:00:01.899 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:01.899 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:00:01.899 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": [ 2021-05-25T12:00:01.900 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:01.900 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:00:01.900 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:00:01.900 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:00:01.901 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:01.901 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:00:01.901 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:01.901 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:00:01.901 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:00:01.903 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.793+0000 7fb9854aa700 10 librados: watch_flush enter 2021-05-25T12:00:01.904 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.793+0000 7fb9854aa700 10 librados: watch_flush exit 2021-05-25T12:00:01.904 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.793+0000 7fb9854aa700 20 client.4406.objecter shutdown clearing up homeless session... 2021-05-25T12:00:01.904 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.793+0000 7fb9854aa700 10 client.4406.objecter successfully canceled tick 2021-05-25T12:00:01.904 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.793+0000 7fb9854aa700 1 -- 172.21.15.102:0/629903046 >> v1:172.21.15.181:6812/29582 conn(0x7fb960033500 legacy=0x7fb9600359c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:01.905 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.793+0000 7fb9854aa700 10 monclient: shutdown 2021-05-25T12:00:01.905 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.793+0000 7fb9854aa700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:00:01.905 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.793+0000 7fb9854aa700 1 -- 172.21.15.102:0/629903046 >> v1:172.21.15.102:6790/0 conn(0x7fb980094da0 legacy=0x7fb9801190a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:01.905 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.793+0000 7fb97f7fe700 1 -- 172.21.15.102:0/629903046 reap_dead start 2021-05-25T12:00:01.905 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.793+0000 7fb97f7fe700 1 -- 172.21.15.102:0/629903046 reap_dead start 2021-05-25T12:00:01.906 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.793+0000 7fb9854aa700 1 -- 172.21.15.102:0/629903046 shutdown_connections 2021-05-25T12:00:01.906 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.793+0000 7fb9854aa700 1 -- 172.21.15.102:0/629903046 >> 172.21.15.102:0/629903046 conn(0x7fb98009cb30 msgr2=0x7fb980068a50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:00:01.906 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.793+0000 7fb9854aa700 1 -- 172.21.15.102:0/629903046 shutdown_connections 2021-05-25T12:00:01.906 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.793+0000 7fb9854aa700 1 -- 172.21.15.102:0/629903046 wait complete. 2021-05-25T12:00:01.907 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:01.793+0000 7fb9854aa700 1 librados: shutdown 2021-05-25T12:00:01.908 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2021-05-25T12:00:01.908 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2021-05-25T12:00:01.908 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2021-05-25T12:00:01.909 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2021-05-25T12:00:01.909 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2021-05-25T12:00:01.909 INFO:teuthology.orchestra.run:waiting for 300 2021-05-25T12:00:01.922 INFO:tasks.ceph.mon.b:Stopped 2021-05-25T12:00:01.922 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2021-05-25T12:00:01.923 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2021-05-25T12:00:02.032 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.030+0000 7f4a18f8c700 10 monclient: get_monmap_and_config 2021-05-25T12:00:02.032 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.030+0000 7f4a18f8c700 10 monclient: build_initial_monmap 2021-05-25T12:00:02.032 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.030+0000 7f4a18f8c700 10 monclient: monmap: 2021-05-25T12:00:02.032 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:00:02.033 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:02.033 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:00:02.031525+0000 2021-05-25T12:00:02.033 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:00:02.031525+0000 2021-05-25T12:00:02.034 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:00:02.034 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:02.034 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:00:02.034 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:00:02.035 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:00:02.035 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:02.045 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.031+0000 7f4a18f8c700 1 Processor -- start 2021-05-25T12:00:02.045 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.031+0000 7f4a18f8c700 1 -- start start 2021-05-25T12:00:02.046 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.031+0000 7f4a18f8c700 10 monclient: init 2021-05-25T12:00:02.046 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.031+0000 7f4a18f8c700 10 monclient: _reopen_session rank -1 2021-05-25T12:00:02.046 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.031+0000 7f4a18f8c700 10 monclient: _add_conns ranks=[2,0,1] 2021-05-25T12:00:02.046 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.031+0000 7f4a18f8c700 10 monclient(hunting): picked mon.noname-b con 0x7f4a14090990 addr v1:172.21.15.102:6790/0 2021-05-25T12:00:02.046 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.031+0000 7f4a18f8c700 10 monclient(hunting): picked mon.noname-a con 0x7f4a14090d70 addr v1:172.21.15.102:6789/0 2021-05-25T12:00:02.047 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.031+0000 7f4a18f8c700 10 monclient(hunting): picked mon.noname-c con 0x7f4a1412e330 addr v1:172.21.15.181:6789/0 2021-05-25T12:00:02.047 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.031+0000 7f4a18f8c700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a14096fa0 con 0x7f4a14090d70 2021-05-25T12:00:02.047 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.031+0000 7f4a18f8c700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a140926a0 con 0x7f4a1412e330 2021-05-25T12:00:02.048 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.031+0000 7f4a18f8c700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a14092940 con 0x7f4a14090990 2021-05-25T12:00:02.048 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.031+0000 7f4a18f8c700 10 monclient(hunting): _renew_subs 2021-05-25T12:00:02.048 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.031+0000 7f4a18f8c700 10 monclient(hunting): authenticate will time out at 802.628859s 2021-05-25T12:00:02.048 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.031+0000 7f4a11d9b700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f4a14090d70 0x7f4a14091200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:59156/0 (socket says 172.21.15.102:59156) 2021-05-25T12:00:02.048 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.031+0000 7f4a1259c700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f4a14090990 0x7f4a14093240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:54032/0 (socket says 172.21.15.102:54032) 2021-05-25T12:00:02.049 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.031+0000 7f4a11d9b700 1 -- 172.21.15.102:0/90057780 learned_addr learned my addr 172.21.15.102:0/90057780 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:00:02.049 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.031+0000 7f4a12d9d700 1 -- 172.21.15.102:0/90057780 >> v1:172.21.15.181:6789/0 conn(0x7f4a1412e330 legacy=0x7f4a14130720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2021-05-25T12:00:02.049 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.031+0000 7f4a1159a700 1 -- 172.21.15.102:0/90057780 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1813183097 0 0) 0x7f4a14092940 con 0x7f4a14090990 2021-05-25T12:00:02.049 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.031+0000 7f4a1159a700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:02.050 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.031+0000 7f4a1159a700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:02.050 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.031+0000 7f4a1159a700 10 monclient(hunting): my global_id is 4409 2021-05-25T12:00:02.050 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.031+0000 7f4a1159a700 1 -- 172.21.15.102:0/90057780 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49f8004380 con 0x7f4a14090990 2021-05-25T12:00:02.050 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.031+0000 7f4a1159a700 1 -- 172.21.15.102:0/90057780 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3356252100 0 0) 0x7f4a14096fa0 con 0x7f4a14090d70 2021-05-25T12:00:02.050 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.031+0000 7f4a1159a700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:02.051 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.031+0000 7f4a1159a700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:02.051 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.031+0000 7f4a1159a700 10 monclient(hunting): my global_id is 4416 2021-05-25T12:00:02.051 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.031+0000 7f4a1159a700 1 -- 172.21.15.102:0/90057780 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a14092940 con 0x7f4a14090d70 2021-05-25T12:00:02.051 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.031+0000 7f4a1159a700 1 -- 172.21.15.102:0/90057780 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1919832816 0 0) 0x7f49f8004380 con 0x7f4a14090990 2021-05-25T12:00:02.051 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.031+0000 7f4a1159a700 1 -- 172.21.15.102:0/90057780 >> v1:172.21.15.181:6789/0 conn(0x7f4a1412e330 legacy=0x7f4a14130720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-05-25T12:00:02.052 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.031+0000 7f4a1159a700 1 -- 172.21.15.102:0/90057780 >> v1:172.21.15.102:6789/0 conn(0x7f4a14090d70 legacy=0x7f4a14091200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:02.052 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.031+0000 7f4a1159a700 10 monclient: _finish_hunting 0 2021-05-25T12:00:02.052 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.031+0000 7f4a1159a700 1 monclient: found mon.noname-b 2021-05-25T12:00:02.052 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.031+0000 7f4a1159a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:00:02.053 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.031+0000 7f4a1159a700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T12:00:02.053 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.031+0000 7f4a1159a700 1 -- 172.21.15.102:0/90057780 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a14092be0 con 0x7f4a14090990 2021-05-25T12:00:02.053 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.031+0000 7f4a1159a700 10 monclient: _finish_auth 0 2021-05-25T12:00:02.053 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.031+0000 7f4a1159a700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:00:02.053 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.031+0000 7f4a18f8c700 5 monclient: authenticate success, global_id 4409 2021-05-25T12:00:02.054 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.031+0000 7f4a1159a700 1 -- 172.21.15.102:0/90057780 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f49fc001610 con 0x7f4a14090990 2021-05-25T12:00:02.054 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.031+0000 7f4a1159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:02.054 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.031+0000 7f4a1159a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:00:02.054 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.031+0000 7f4a1159a700 10 monclient: dump: 2021-05-25T12:00:02.055 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:02.055 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:02.055 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:02.055 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:02.055 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:02.056 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:02.056 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:02.056 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:02.056 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:02.056 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:02.057 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.031+0000 7f4a18f8c700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:00:02.057 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.031+0000 7f4a1159a700 1 -- 172.21.15.102:0/90057780 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f49fc001880 con 0x7f4a14090990 2021-05-25T12:00:02.057 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.031+0000 7f4a1159a700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:00:02.057 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.031+0000 7f4a1159a700 1 -- 172.21.15.102:0/90057780 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f49fc001c10 con 0x7f4a14090990 2021-05-25T12:00:02.058 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.031+0000 7f4a18f8c700 10 monclient: get_monmap_and_config success 2021-05-25T12:00:02.058 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.031+0000 7f4a18f8c700 4 set_mon_vals no callback set 2021-05-25T12:00:02.058 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.031+0000 7f4a1159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:02.058 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.031+0000 7f4a1159a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:00:02.059 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.031+0000 7f4a1159a700 10 monclient: dump: 2021-05-25T12:00:02.059 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:02.059 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:02.059 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:02.059 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:02.060 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:02.060 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:02.060 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:02.060 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:02.061 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:02.061 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:02.061 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.032+0000 7f4a18f8c700 10 monclient: shutdown 2021-05-25T12:00:02.061 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.032+0000 7f4a18f8c700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:00:02.061 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.032+0000 7f4a18f8c700 1 -- 172.21.15.102:0/90057780 >> v1:172.21.15.102:6790/0 conn(0x7f4a14090990 legacy=0x7f4a14093240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:02.062 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.032+0000 7f4a18f8c700 1 -- 172.21.15.102:0/90057780 shutdown_connections 2021-05-25T12:00:02.062 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.032+0000 7f4a18f8c700 1 -- 172.21.15.102:0/90057780 >> 172.21.15.102:0/90057780 conn(0x7f4a1409b370 msgr2=0x7f4a14097430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:00:02.062 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.032+0000 7f4a18f8c700 1 -- 172.21.15.102:0/90057780 shutdown_connections 2021-05-25T12:00:02.062 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.032+0000 7f4a18f8c700 1 -- 172.21.15.102:0/90057780 wait complete. 2021-05-25T12:00:02.062 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.032+0000 7f4a18f8c700 10 monclient: build_initial_monmap 2021-05-25T12:00:02.063 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.032+0000 7f4a18f8c700 10 monclient: monmap: 2021-05-25T12:00:02.063 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:00:02.063 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:00:02.063 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:00:02.064 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:00:02.064 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:00:02.064 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:00:02.064 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:00:02.065 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:00:02.065 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:00:02.065 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:02.065 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.032+0000 7f4a18f8c700 1 librados: starting msgr at 2021-05-25T12:00:02.066 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.032+0000 7f4a18f8c700 1 librados: starting objecter 2021-05-25T12:00:02.066 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.032+0000 7f4a18f8c700 1 Processor -- start 2021-05-25T12:00:02.066 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.032+0000 7f4a18f8c700 1 -- start start 2021-05-25T12:00:02.066 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.032+0000 7f4a18f8c700 1 librados: setting wanted keys 2021-05-25T12:00:02.067 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.032+0000 7f4a18f8c700 1 librados: calling monclient init 2021-05-25T12:00:02.067 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.032+0000 7f4a18f8c700 10 monclient: init 2021-05-25T12:00:02.067 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.033+0000 7f4a18f8c700 10 monclient: _reopen_session rank -1 2021-05-25T12:00:02.068 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.033+0000 7f4a18f8c700 10 monclient: _add_conns ranks=[0,2,1] 2021-05-25T12:00:02.068 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.033+0000 7f4a18f8c700 10 monclient(hunting): picked mon.noname-a con 0x7f4a14090990 addr v1:172.21.15.102:6789/0 2021-05-25T12:00:02.068 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.033+0000 7f4a18f8c700 10 monclient(hunting): picked mon.noname-c con 0x7f4a14090d70 addr v1:172.21.15.102:6790/0 2021-05-25T12:00:02.069 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.033+0000 7f4a18f8c700 10 monclient(hunting): picked mon.noname-b con 0x7f4a1412e330 addr v1:172.21.15.181:6789/0 2021-05-25T12:00:02.069 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.033+0000 7f4a18f8c700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a14112cb0 con 0x7f4a14090990 2021-05-25T12:00:02.070 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.033+0000 7f4a18f8c700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a14091fe0 con 0x7f4a1412e330 2021-05-25T12:00:02.070 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.033+0000 7f4a18f8c700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a1412a100 con 0x7f4a14090d70 2021-05-25T12:00:02.070 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.033+0000 7f4a18f8c700 10 monclient(hunting): _renew_subs 2021-05-25T12:00:02.070 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.033+0000 7f4a18f8c700 10 monclient(hunting): authenticate will time out at 802.631394s 2021-05-25T12:00:02.071 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.033+0000 7f4a1259c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f4a14090990 0x7f4a14126280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:59160/0 (socket says 172.21.15.102:59160) 2021-05-25T12:00:02.071 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.033+0000 7f4a11d9b700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f4a14090d70 0x7f4a1413dcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:54040/0 (socket says 172.21.15.102:54040) 2021-05-25T12:00:02.071 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.033+0000 7f4a1259c700 1 -- 172.21.15.102:0/680787103 learned_addr learned my addr 172.21.15.102:0/680787103 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:00:02.071 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.033+0000 7f4a12d9d700 1 -- 172.21.15.102:0/680787103 >> v1:172.21.15.181:6789/0 conn(0x7f4a1412e330 legacy=0x7f4a14141420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.181:6789/0 2021-05-25T12:00:02.072 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.033+0000 7f4a027fc700 10 client.?.objecter ms_handle_connect 0x7f4a14090990 2021-05-25T12:00:02.072 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.033+0000 7f4a027fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:00:02.072 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.033+0000 7f4a027fc700 10 client.?.objecter ms_handle_connect 0x7f4a14090d70 2021-05-25T12:00:02.072 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.033+0000 7f4a027fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:00:02.073 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.033+0000 7f4a027fc700 1 -- 172.21.15.102:0/680787103 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2944483103 0 0) 0x7f4a14112cb0 con 0x7f4a14090990 2021-05-25T12:00:02.073 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.033+0000 7f4a027fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:02.073 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.033+0000 7f4a027fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:02.073 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.033+0000 7f4a027fc700 10 monclient(hunting): my global_id is 4419 2021-05-25T12:00:02.073 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.033+0000 7f4a027fc700 1 -- 172.21.15.102:0/680787103 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49f0004c10 con 0x7f4a14090990 2021-05-25T12:00:02.074 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.033+0000 7f4a027fc700 1 -- 172.21.15.102:0/680787103 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4263848034 0 0) 0x7f4a1412a100 con 0x7f4a14090d70 2021-05-25T12:00:02.074 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.033+0000 7f4a027fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:02.074 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.033+0000 7f4a027fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:02.074 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.033+0000 7f4a027fc700 10 monclient(hunting): my global_id is 4412 2021-05-25T12:00:02.075 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.033+0000 7f4a027fc700 1 -- 172.21.15.102:0/680787103 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a14112cb0 con 0x7f4a14090d70 2021-05-25T12:00:02.075 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.033+0000 7f4a027fc700 1 -- 172.21.15.102:0/680787103 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1361681755 0 0) 0x7f49f0004c10 con 0x7f4a14090990 2021-05-25T12:00:02.075 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.033+0000 7f4a027fc700 1 -- 172.21.15.102:0/680787103 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4a1412a100 con 0x7f4a14090990 2021-05-25T12:00:02.075 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.033+0000 7f4a027fc700 1 -- 172.21.15.102:0/680787103 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f49fc0015d0 con 0x7f4a14090990 2021-05-25T12:00:02.075 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.033+0000 7f4a027fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:02.076 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.033+0000 7f4a027fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:00:02.076 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.033+0000 7f4a027fc700 10 monclient(hunting): dump: 2021-05-25T12:00:02.076 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:02.076 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:02.076 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:02.077 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:02.077 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:02.077 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:02.077 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:02.078 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:02.078 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:02.078 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:02.078 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.033+0000 7f4a027fc700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:00:02.078 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.033+0000 7f4a027fc700 1 -- 172.21.15.102:0/680787103 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2562542616 0 0) 0x7f4a14112cb0 con 0x7f4a14090d70 2021-05-25T12:00:02.079 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.033+0000 7f4a027fc700 1 -- 172.21.15.102:0/680787103 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f49f0004c10 con 0x7f4a14090d70 2021-05-25T12:00:02.079 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.033+0000 7f4a027fc700 1 -- 172.21.15.102:0/680787103 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f4a080019d0 con 0x7f4a14090d70 2021-05-25T12:00:02.079 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.033+0000 7f4a027fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:02.079 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.033+0000 7f4a027fc700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2021-05-25T12:00:02.079 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.033+0000 7f4a027fc700 10 monclient(hunting): dump: 2021-05-25T12:00:02.080 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:02.080 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:02.080 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:02.080 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:02.081 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:02.081 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:02.081 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:02.081 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:02.081 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:02.082 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:02.082 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.034+0000 7f4a027fc700 1 -- 172.21.15.102:0/680787103 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3240684040 0 0) 0x7f4a1412a100 con 0x7f4a14090990 2021-05-25T12:00:02.082 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.034+0000 7f4a027fc700 1 -- 172.21.15.102:0/680787103 >> v1:172.21.15.102:6790/0 conn(0x7f4a14090d70 legacy=0x7f4a1413dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:02.082 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.034+0000 7f4a027fc700 1 -- 172.21.15.102:0/680787103 >> v1:172.21.15.181:6789/0 conn(0x7f4a1412e330 legacy=0x7f4a14141420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-05-25T12:00:02.082 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.034+0000 7f4a027fc700 10 monclient: _finish_hunting 0 2021-05-25T12:00:02.083 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.034+0000 7f4a027fc700 1 monclient: found mon.a 2021-05-25T12:00:02.083 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.034+0000 7f4a027fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:00:02.083 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.034+0000 7f4a027fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:00:02.084 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.034+0000 7f4a027fc700 1 -- 172.21.15.102:0/680787103 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a14126990 con 0x7f4a14090990 2021-05-25T12:00:02.084 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.034+0000 7f4a027fc700 10 monclient: _finish_auth 0 2021-05-25T12:00:02.084 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.034+0000 7f4a027fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:00:02.084 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.034+0000 7f4a18f8c700 5 monclient: authenticate success, global_id 4419 2021-05-25T12:00:02.085 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.034+0000 7f4a18f8c700 10 monclient: _renew_subs 2021-05-25T12:00:02.085 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.034+0000 7f4a18f8c700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:00:02.085 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.034+0000 7f4a18f8c700 1 -- 172.21.15.102:0/680787103 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a14136eb0 con 0x7f4a14090990 2021-05-25T12:00:02.085 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.034+0000 7f4a18f8c700 10 client.4419.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:00:02.086 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.034+0000 7f4a18f8c700 10 monclient: _renew_subs 2021-05-25T12:00:02.086 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.034+0000 7f4a18f8c700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:00:02.086 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.034+0000 7f4a18f8c700 1 -- 172.21.15.102:0/680787103 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a14139800 con 0x7f4a14090990 2021-05-25T12:00:02.086 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.034+0000 7f4a18f8c700 1 librados: init done 2021-05-25T12:00:02.086 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.034+0000 7f4a027fc700 1 -- 172.21.15.102:0/680787103 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f49fc004960 con 0x7f4a14090990 2021-05-25T12:00:02.087 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.034+0000 7f4a027fc700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:00:02.087 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.034+0000 7f4a027fc700 1 -- 172.21.15.102:0/680787103 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f49fc004cf0 con 0x7f4a14090990 2021-05-25T12:00:02.087 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.034+0000 7f4a027fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:02.088 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.034+0000 7f4a10d99700 4 set_mon_vals no callback set 2021-05-25T12:00:02.088 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.034+0000 7f4a027fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:00:02.088 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.034+0000 7f4a027fc700 10 monclient: dump: 2021-05-25T12:00:02.088 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:02.088 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:02.089 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:02.089 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:02.089 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:02.089 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:02.090 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:02.090 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:02.090 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:02.090 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:02.090 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.034+0000 7f4a027fc700 1 -- 172.21.15.102:0/680787103 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f49fc004ed0 con 0x7f4a14090990 2021-05-25T12:00:02.091 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.034+0000 7f4a18f8c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:00:02.091 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.034+0000 7f4a18f8c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:00:02.091 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.034+0000 7f4a18f8c700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:00:02.091 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.034+0000 7f4a18f8c700 1 -- 172.21.15.102:0/680787103 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49f4004db0 con 0x7f4a14090990 2021-05-25T12:00:02.092 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.035+0000 7f4a027fc700 1 -- 172.21.15.102:0/680787103 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(15..15 src has 1..15) v4 ==== 3549+0+0 (unknown 3948819498 0 0) 0x7f49fc03d0f0 con 0x7f4a14090990 2021-05-25T12:00:02.092 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.035+0000 7f4a027fc700 10 client.4419.objecter ms_dispatch 0x7f4a14001550 osd_map(15..15 src has 1..15) v4 2021-05-25T12:00:02.092 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.035+0000 7f4a027fc700 3 client.4419.objecter handle_osd_map got epochs [15,15] > 0 2021-05-25T12:00:02.092 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.035+0000 7f4a027fc700 3 client.4419.objecter handle_osd_map decoding full epoch 15 2021-05-25T12:00:02.092 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.035+0000 7f4a027fc700 20 client.4419.objecter dump_active .. 0 homeless 2021-05-25T12:00:02.093 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.035+0000 7f4a11d9b700 10 monclient: get_auth_request con 0x7f49f00332a0 auth_method 0 2021-05-25T12:00:02.093 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.035+0000 7f4a027fc700 10 client.4419.objecter ms_handle_connect 0x7f49f00332a0 2021-05-25T12:00:02.093 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.039+0000 7f4a027fc700 1 -- 172.21.15.102:0/680787103 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7f49fc0130f0 con 0x7f4a14090990 2021-05-25T12:00:02.093 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.039+0000 7f4a027fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:00:02.093 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.039+0000 7f4a027fc700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:00:02.248 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.245+0000 7f4a18f8c700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2021-05-25T12:00:02.249 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.245+0000 7f4a18f8c700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2021-05-25T12:00:02.249 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.245+0000 7f4a18f8c700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:00:02.249 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.245+0000 7f4a18f8c700 1 -- 172.21.15.102:0/680787103 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f49f40055e0 con 0x7f4a14090990 2021-05-25T12:00:02.250 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.245+0000 7f4a027fc700 1 -- 172.21.15.102:0/680787103 <== mon.0 v1:172.21.15.102:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1342 (unknown 254746657 0 4290852217) 0x7f49fc012ce0 con 0x7f4a14090990 2021-05-25T12:00:02.250 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.245+0000 7f4a027fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2021-05-25T12:00:02.250 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.245+0000 7f4a027fc700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:00:02.251 INFO:teuthology.orchestra.run.smithi102.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540138303579357183","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"20ee91b6-8935-4b98-b868-85c03789ccc4","modified":"2021-05-25T11:59:11.011522Z","created":"2021-05-25T11:59:11.011522Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6790","nonce":0}]},"addr":"172.21.15.102:6790/0","public_addr":"172.21.15.102:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2021-05-25T12:00:02.252 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.247+0000 7f4a18f8c700 10 librados: watch_flush enter 2021-05-25T12:00:02.253 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.247+0000 7f4a18f8c700 10 librados: watch_flush exit 2021-05-25T12:00:02.253 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.247+0000 7f4a18f8c700 20 client.4419.objecter shutdown clearing up homeless session... 2021-05-25T12:00:02.253 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.247+0000 7f4a18f8c700 10 client.4419.objecter successfully canceled tick 2021-05-25T12:00:02.253 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.247+0000 7f4a18f8c700 1 -- 172.21.15.102:0/680787103 >> v1:172.21.15.181:6812/29582 conn(0x7f49f00332a0 legacy=0x7f49f0035760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:02.254 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.247+0000 7f4a18f8c700 10 monclient: shutdown 2021-05-25T12:00:02.254 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.247+0000 7f4a18f8c700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:00:02.255 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.247+0000 7f4a18f8c700 1 -- 172.21.15.102:0/680787103 >> v1:172.21.15.102:6789/0 conn(0x7f4a14090990 legacy=0x7f4a14126280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:02.255 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.247+0000 7f4a18f8c700 1 -- 172.21.15.102:0/680787103 shutdown_connections 2021-05-25T12:00:02.255 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.247+0000 7f4a18f8c700 1 -- 172.21.15.102:0/680787103 >> 172.21.15.102:0/680787103 conn(0x7f4a1409b370 msgr2=0x7f4a141292f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:00:02.256 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.247+0000 7f4a18f8c700 1 -- 172.21.15.102:0/680787103 shutdown_connections 2021-05-25T12:00:02.256 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.247+0000 7f4a18f8c700 1 -- 172.21.15.102:0/680787103 wait complete. 2021-05-25T12:00:02.256 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:02.247+0000 7f4a18f8c700 1 librados: shutdown 2021-05-25T12:00:05.262 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2021-05-25T12:00:05.385 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.383+0000 7f5d72d44700 10 monclient: get_monmap_and_config 2021-05-25T12:00:05.386 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.383+0000 7f5d72d44700 10 monclient: build_initial_monmap 2021-05-25T12:00:05.386 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.383+0000 7f5d72d44700 10 monclient: monmap: 2021-05-25T12:00:05.386 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:00:05.387 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:05.387 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:00:05.384711+0000 2021-05-25T12:00:05.387 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:00:05.384711+0000 2021-05-25T12:00:05.387 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:00:05.387 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:05.388 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:00:05.388 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:00:05.388 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:00:05.388 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:05.389 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.384+0000 7f5d72d44700 1 Processor -- start 2021-05-25T12:00:05.389 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.384+0000 7f5d72d44700 1 -- start start 2021-05-25T12:00:05.389 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.384+0000 7f5d72d44700 10 monclient: init 2021-05-25T12:00:05.398 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.384+0000 7f5d72d44700 10 monclient: _reopen_session rank -1 2021-05-25T12:00:05.398 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.385+0000 7f5d72d44700 10 monclient: _add_conns ranks=[0,2,1] 2021-05-25T12:00:05.398 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.385+0000 7f5d72d44700 10 monclient(hunting): picked mon.noname-a con 0x7f5d6c09a350 addr v1:172.21.15.102:6789/0 2021-05-25T12:00:05.399 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.385+0000 7f5d72d44700 10 monclient(hunting): picked mon.noname-b con 0x7f5d6c064140 addr v1:172.21.15.102:6790/0 2021-05-25T12:00:05.399 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.385+0000 7f5d72d44700 10 monclient(hunting): picked mon.noname-c con 0x7f5d6c131d40 addr v1:172.21.15.181:6789/0 2021-05-25T12:00:05.399 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.385+0000 7f5d72d44700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d6c096240 con 0x7f5d6c09a350 2021-05-25T12:00:05.399 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.385+0000 7f5d72d44700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d6c096470 con 0x7f5d6c131d40 2021-05-25T12:00:05.400 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.385+0000 7f5d72d44700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d6c095bf0 con 0x7f5d6c064140 2021-05-25T12:00:05.400 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.385+0000 7f5d72d44700 10 monclient(hunting): _renew_subs 2021-05-25T12:00:05.400 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.385+0000 7f5d72d44700 10 monclient(hunting): authenticate will time out at 805.982103s 2021-05-25T12:00:05.400 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.385+0000 7f5d712e1700 1 -- >> v1:172.21.15.181:6789/0 conn(0x7f5d6c131d40 legacy=0x7f5d6c134130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2021-05-25T12:00:05.401 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.385+0000 7f5d70ae0700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f5d6c09a350 0x7f5d6c128ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:59186/0 (socket says 172.21.15.102:59186) 2021-05-25T12:00:05.401 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.385+0000 7f5d70ae0700 1 -- 172.21.15.102:0/2595063759 learned_addr learned my addr 172.21.15.102:0/2595063759 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:00:05.401 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.385+0000 7f5d63fff700 1 --1- 172.21.15.102:0/2595063759 >> v1:172.21.15.102:6790/0 conn(0x7f5d6c064140 0x7f5d6c098930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:54064/0 (socket says 172.21.15.102:54064) 2021-05-25T12:00:05.402 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.385+0000 7f5d637fe700 1 -- 172.21.15.102:0/2595063759 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3351101188 0 0) 0x7f5d6c096240 con 0x7f5d6c09a350 2021-05-25T12:00:05.402 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.385+0000 7f5d637fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:05.402 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.385+0000 7f5d637fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:05.402 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.385+0000 7f5d637fe700 10 monclient(hunting): my global_id is 4422 2021-05-25T12:00:05.402 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.385+0000 7f5d637fe700 1 -- 172.21.15.102:0/2595063759 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d58003270 con 0x7f5d6c09a350 2021-05-25T12:00:05.403 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.385+0000 7f5d637fe700 1 -- 172.21.15.102:0/2595063759 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1580403663 0 0) 0x7f5d58003270 con 0x7f5d6c09a350 2021-05-25T12:00:05.403 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.385+0000 7f5d637fe700 1 -- 172.21.15.102:0/2595063759 >> v1:172.21.15.102:6790/0 conn(0x7f5d6c064140 legacy=0x7f5d6c098930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:05.403 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.385+0000 7f5d637fe700 1 -- 172.21.15.102:0/2595063759 >> v1:172.21.15.181:6789/0 conn(0x7f5d6c131d40 legacy=0x7f5d6c134130 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-05-25T12:00:05.403 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.385+0000 7f5d637fe700 10 monclient: _finish_hunting 0 2021-05-25T12:00:05.404 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.385+0000 7f5d637fe700 1 monclient: found mon.noname-a 2021-05-25T12:00:05.404 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.385+0000 7f5d637fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:00:05.404 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.385+0000 7f5d637fe700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T12:00:05.404 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.385+0000 7f5d637fe700 1 -- 172.21.15.102:0/2595063759 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d6c099040 con 0x7f5d6c09a350 2021-05-25T12:00:05.404 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.385+0000 7f5d637fe700 10 monclient: _finish_auth 0 2021-05-25T12:00:05.405 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.385+0000 7f5d637fe700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:00:05.405 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.385+0000 7f5d72d44700 5 monclient: authenticate success, global_id 4422 2021-05-25T12:00:05.405 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.385+0000 7f5d637fe700 1 -- 172.21.15.102:0/2595063759 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f5d5c001610 con 0x7f5d6c09a350 2021-05-25T12:00:05.405 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.385+0000 7f5d72d44700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:00:05.406 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.385+0000 7f5d637fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:05.406 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.385+0000 7f5d637fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:00:05.406 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.385+0000 7f5d637fe700 10 monclient: dump: 2021-05-25T12:00:05.406 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:05.406 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:05.407 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:05.407 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:05.407 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:05.407 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:05.408 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:05.408 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:05.408 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:05.408 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:05.408 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.385+0000 7f5d72d44700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:00:05.409 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.385+0000 7f5d637fe700 1 -- 172.21.15.102:0/2595063759 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5d5c0018c0 con 0x7f5d6c09a350 2021-05-25T12:00:05.409 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.385+0000 7f5d637fe700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:00:05.409 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.385+0000 7f5d637fe700 1 -- 172.21.15.102:0/2595063759 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f5d5c001c90 con 0x7f5d6c09a350 2021-05-25T12:00:05.409 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.385+0000 7f5d72d44700 10 monclient: get_monmap_and_config success 2021-05-25T12:00:05.409 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.385+0000 7f5d72d44700 4 set_mon_vals no callback set 2021-05-25T12:00:05.410 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.385+0000 7f5d637fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:05.410 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.385+0000 7f5d637fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:00:05.410 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.385+0000 7f5d637fe700 10 monclient: dump: 2021-05-25T12:00:05.410 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:05.410 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:05.411 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:05.411 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:05.411 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:05.411 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:05.412 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:05.412 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:05.412 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:05.412 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:05.412 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.386+0000 7f5d72d44700 10 monclient: shutdown 2021-05-25T12:00:05.413 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.386+0000 7f5d72d44700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:00:05.413 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.386+0000 7f5d72d44700 1 -- 172.21.15.102:0/2595063759 >> v1:172.21.15.102:6789/0 conn(0x7f5d6c09a350 legacy=0x7f5d6c128ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:05.413 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.386+0000 7f5d72d44700 1 -- 172.21.15.102:0/2595063759 shutdown_connections 2021-05-25T12:00:05.413 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.386+0000 7f5d72d44700 1 -- 172.21.15.102:0/2595063759 >> 172.21.15.102:0/2595063759 conn(0x7f5d6c09b710 msgr2=0x7f5d6c09bb20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:00:05.413 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.386+0000 7f5d72d44700 1 -- 172.21.15.102:0/2595063759 shutdown_connections 2021-05-25T12:00:05.414 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.386+0000 7f5d72d44700 1 -- 172.21.15.102:0/2595063759 wait complete. 2021-05-25T12:00:05.414 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.386+0000 7f5d72d44700 10 monclient: build_initial_monmap 2021-05-25T12:00:05.414 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.386+0000 7f5d72d44700 10 monclient: monmap: 2021-05-25T12:00:05.414 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:00:05.415 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:00:05.415 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:00:05.415 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:00:05.415 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:00:05.415 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:00:05.416 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:00:05.416 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:00:05.416 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:00:05.416 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:05.416 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.387+0000 7f5d72d44700 1 librados: starting msgr at 2021-05-25T12:00:05.417 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.387+0000 7f5d72d44700 1 librados: starting objecter 2021-05-25T12:00:05.417 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.387+0000 7f5d72d44700 1 Processor -- start 2021-05-25T12:00:05.417 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.387+0000 7f5d72d44700 1 -- start start 2021-05-25T12:00:05.417 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.387+0000 7f5d72d44700 1 librados: setting wanted keys 2021-05-25T12:00:05.418 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.387+0000 7f5d72d44700 1 librados: calling monclient init 2021-05-25T12:00:05.418 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.387+0000 7f5d72d44700 10 monclient: init 2021-05-25T12:00:05.418 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.387+0000 7f5d72d44700 10 monclient: _reopen_session rank -1 2021-05-25T12:00:05.418 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.387+0000 7f5d72d44700 10 monclient: _add_conns ranks=[1,2,0] 2021-05-25T12:00:05.418 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.387+0000 7f5d72d44700 10 monclient(hunting): picked mon.noname-b con 0x7f5d6c064140 addr v1:172.21.15.181:6789/0 2021-05-25T12:00:05.419 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.387+0000 7f5d72d44700 10 monclient(hunting): picked mon.noname-c con 0x7f5d6c09a350 addr v1:172.21.15.102:6790/0 2021-05-25T12:00:05.419 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.387+0000 7f5d72d44700 10 monclient(hunting): picked mon.noname-a con 0x7f5d6c131d40 addr v1:172.21.15.102:6789/0 2021-05-25T12:00:05.419 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.387+0000 7f5d72d44700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d6c0931e0 con 0x7f5d6c131d40 2021-05-25T12:00:05.419 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.387+0000 7f5d72d44700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d6c0922c0 con 0x7f5d6c064140 2021-05-25T12:00:05.420 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.387+0000 7f5d72d44700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d6c093410 con 0x7f5d6c09a350 2021-05-25T12:00:05.420 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.387+0000 7f5d72d44700 10 monclient(hunting): _renew_subs 2021-05-25T12:00:05.420 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.387+0000 7f5d72d44700 10 monclient(hunting): authenticate will time out at 805.984998s 2021-05-25T12:00:05.420 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.388+0000 7f5d70ae0700 1 -- >> v1:172.21.15.181:6789/0 conn(0x7f5d6c064140 legacy=0x7f5d6c094e00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.181:6789/0 2021-05-25T12:00:05.420 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.388+0000 7f5d63fff700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f5d6c09a350 0x7f5d6c1401b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:54072/0 (socket says 172.21.15.102:54072) 2021-05-25T12:00:05.421 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.388+0000 7f5d63fff700 1 -- 172.21.15.102:0/959091548 learned_addr learned my addr 172.21.15.102:0/959091548 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:00:05.421 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.388+0000 7f5d60ff9700 10 client.?.objecter ms_handle_connect 0x7f5d6c131d40 2021-05-25T12:00:05.421 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.388+0000 7f5d60ff9700 10 client.?.objecter resend_mon_ops 2021-05-25T12:00:05.421 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.388+0000 7f5d60ff9700 10 client.?.objecter ms_handle_connect 0x7f5d6c09a350 2021-05-25T12:00:05.421 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.388+0000 7f5d60ff9700 10 client.?.objecter resend_mon_ops 2021-05-25T12:00:05.422 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.388+0000 7f5d60ff9700 1 -- 172.21.15.102:0/959091548 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3675548565 0 0) 0x7f5d6c0931e0 con 0x7f5d6c131d40 2021-05-25T12:00:05.422 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.388+0000 7f5d60ff9700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:05.422 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.388+0000 7f5d60ff9700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:05.422 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.388+0000 7f5d60ff9700 10 monclient(hunting): my global_id is 4425 2021-05-25T12:00:05.423 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.388+0000 7f5d60ff9700 1 -- 172.21.15.102:0/959091548 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d4c004c10 con 0x7f5d6c131d40 2021-05-25T12:00:05.423 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.388+0000 7f5d60ff9700 1 -- 172.21.15.102:0/959091548 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1811356369 0 0) 0x7f5d4c004c10 con 0x7f5d6c131d40 2021-05-25T12:00:05.423 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.388+0000 7f5d60ff9700 1 -- 172.21.15.102:0/959091548 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5d6c0931e0 con 0x7f5d6c131d40 2021-05-25T12:00:05.423 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.388+0000 7f5d60ff9700 1 -- 172.21.15.102:0/959091548 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f5d68004110 con 0x7f5d6c131d40 2021-05-25T12:00:05.423 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.388+0000 7f5d60ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:05.424 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.388+0000 7f5d60ff9700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:00:05.424 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.388+0000 7f5d60ff9700 10 monclient(hunting): dump: 2021-05-25T12:00:05.424 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:05.424 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:05.424 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:05.425 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:05.425 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:05.425 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:05.425 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:05.425 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:05.426 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:05.426 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:05.426 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.388+0000 7f5d60ff9700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:00:05.426 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.388+0000 7f5d60ff9700 1 -- 172.21.15.102:0/959091548 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1999436844 0 0) 0x7f5d6c0931e0 con 0x7f5d6c131d40 2021-05-25T12:00:05.427 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.388+0000 7f5d60ff9700 1 -- 172.21.15.102:0/959091548 >> v1:172.21.15.102:6790/0 conn(0x7f5d6c09a350 legacy=0x7f5d6c1401b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:05.427 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.388+0000 7f5d60ff9700 1 -- 172.21.15.102:0/959091548 >> v1:172.21.15.181:6789/0 conn(0x7f5d6c064140 legacy=0x7f5d6c094e00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-05-25T12:00:05.427 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.388+0000 7f5d60ff9700 10 monclient: _finish_hunting 0 2021-05-25T12:00:05.427 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.388+0000 7f5d60ff9700 1 monclient: found mon.a 2021-05-25T12:00:05.427 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.388+0000 7f5d60ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:00:05.428 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.388+0000 7f5d60ff9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:00:05.428 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.388+0000 7f5d60ff9700 1 -- 172.21.15.102:0/959091548 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d6c0936b0 con 0x7f5d6c131d40 2021-05-25T12:00:05.428 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.388+0000 7f5d60ff9700 10 monclient: _finish_auth 0 2021-05-25T12:00:05.428 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.388+0000 7f5d60ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:00:05.428 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.388+0000 7f5d72d44700 5 monclient: authenticate success, global_id 4425 2021-05-25T12:00:05.429 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.388+0000 7f5d72d44700 10 monclient: _renew_subs 2021-05-25T12:00:05.429 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.388+0000 7f5d72d44700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:00:05.429 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.388+0000 7f5d72d44700 1 -- 172.21.15.102:0/959091548 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d6c093950 con 0x7f5d6c131d40 2021-05-25T12:00:05.429 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.388+0000 7f5d72d44700 10 client.4425.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:00:05.430 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.388+0000 7f5d72d44700 10 monclient: _renew_subs 2021-05-25T12:00:05.430 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.388+0000 7f5d72d44700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:00:05.430 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.388+0000 7f5d72d44700 1 -- 172.21.15.102:0/959091548 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d6c095810 con 0x7f5d6c131d40 2021-05-25T12:00:05.430 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.388+0000 7f5d72d44700 1 librados: init done 2021-05-25T12:00:05.430 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.389+0000 7f5d60ff9700 1 -- 172.21.15.102:0/959091548 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5d68004330 con 0x7f5d6c131d40 2021-05-25T12:00:05.431 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.389+0000 7f5d60ff9700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:00:05.431 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.389+0000 7f5d60ff9700 1 -- 172.21.15.102:0/959091548 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f5d68004640 con 0x7f5d6c131d40 2021-05-25T12:00:05.431 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.389+0000 7f5d60ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:05.431 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.389+0000 7f5d62ffd700 4 set_mon_vals no callback set 2021-05-25T12:00:05.431 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.389+0000 7f5d60ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:00:05.432 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.389+0000 7f5d60ff9700 10 monclient: dump: 2021-05-25T12:00:05.432 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:05.432 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:05.432 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:05.432 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:05.433 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:05.433 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:05.433 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:05.433 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:05.434 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:05.434 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:05.434 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.389+0000 7f5d60ff9700 1 -- 172.21.15.102:0/959091548 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f5d68004820 con 0x7f5d6c131d40 2021-05-25T12:00:05.434 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.389+0000 7f5d72d44700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:00:05.434 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.389+0000 7f5d72d44700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:00:05.435 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.389+0000 7f5d72d44700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:00:05.435 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.389+0000 7f5d60ff9700 1 -- 172.21.15.102:0/959091548 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(15..15 src has 1..15) v4 ==== 3549+0+0 (unknown 3948819498 0 0) 0x7f5d6803cfe0 con 0x7f5d6c131d40 2021-05-25T12:00:05.435 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.389+0000 7f5d60ff9700 10 client.4425.objecter ms_dispatch 0x7f5d6c001550 osd_map(15..15 src has 1..15) v4 2021-05-25T12:00:05.435 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.389+0000 7f5d72d44700 1 -- 172.21.15.102:0/959091548 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d44004db0 con 0x7f5d6c131d40 2021-05-25T12:00:05.435 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.389+0000 7f5d60ff9700 3 client.4425.objecter handle_osd_map got epochs [15,15] > 0 2021-05-25T12:00:05.436 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.389+0000 7f5d60ff9700 3 client.4425.objecter handle_osd_map decoding full epoch 15 2021-05-25T12:00:05.436 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.389+0000 7f5d60ff9700 20 client.4425.objecter dump_active .. 0 homeless 2021-05-25T12:00:05.436 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.389+0000 7f5d70ae0700 10 monclient: get_auth_request con 0x7f5d4c033140 auth_method 0 2021-05-25T12:00:05.436 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.390+0000 7f5d60ff9700 10 client.4425.objecter ms_handle_connect 0x7f5d4c033140 2021-05-25T12:00:05.437 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.393+0000 7f5d60ff9700 1 -- 172.21.15.102:0/959091548 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7f5d68012cb0 con 0x7f5d6c131d40 2021-05-25T12:00:05.437 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.394+0000 7f5d60ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:00:05.437 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.394+0000 7f5d60ff9700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:00:05.603 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.601+0000 7f5d72d44700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2021-05-25T12:00:05.604 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.601+0000 7f5d72d44700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2021-05-25T12:00:05.604 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.601+0000 7f5d72d44700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:00:05.604 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.601+0000 7f5d72d44700 1 -- 172.21.15.102:0/959091548 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f5d440055e0 con 0x7f5d6c131d40 2021-05-25T12:00:05.604 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.602+0000 7f5d60ff9700 1 -- 172.21.15.102:0/959091548 <== mon.0 v1:172.21.15.102:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1342 (unknown 254746657 0 3376866232) 0x7f5d68012ad0 con 0x7f5d6c131d40 2021-05-25T12:00:05.605 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.602+0000 7f5d60ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2021-05-25T12:00:05.605 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.602+0000 7f5d60ff9700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:00:05.605 INFO:teuthology.orchestra.run.smithi102.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540138303579357183","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"20ee91b6-8935-4b98-b868-85c03789ccc4","modified":"2021-05-25T11:59:11.011522Z","created":"2021-05-25T11:59:11.011522Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6790","nonce":0}]},"addr":"172.21.15.102:6790/0","public_addr":"172.21.15.102:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2021-05-25T12:00:05.607 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.603+0000 7f5d72d44700 10 librados: watch_flush enter 2021-05-25T12:00:05.607 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.603+0000 7f5d72d44700 10 librados: watch_flush exit 2021-05-25T12:00:05.608 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.603+0000 7f5d72d44700 20 client.4425.objecter shutdown clearing up homeless session... 2021-05-25T12:00:05.608 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.603+0000 7f5d72d44700 10 client.4425.objecter successfully canceled tick 2021-05-25T12:00:05.608 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.603+0000 7f5d72d44700 1 -- 172.21.15.102:0/959091548 >> v1:172.21.15.181:6812/29582 conn(0x7f5d4c033140 legacy=0x7f5d4c035600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:05.608 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.603+0000 7f5d72d44700 10 monclient: shutdown 2021-05-25T12:00:05.608 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.603+0000 7f5d72d44700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:00:05.609 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.603+0000 7f5d72d44700 1 -- 172.21.15.102:0/959091548 >> v1:172.21.15.102:6789/0 conn(0x7f5d6c131d40 legacy=0x7f5d6c1438e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:05.609 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.603+0000 7f5d72d44700 1 -- 172.21.15.102:0/959091548 shutdown_connections 2021-05-25T12:00:05.609 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.603+0000 7f5d72d44700 1 -- 172.21.15.102:0/959091548 >> 172.21.15.102:0/959091548 conn(0x7f5d6c09b710 msgr2=0x7f5d6c09d2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:00:05.609 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.603+0000 7f5d72d44700 1 -- 172.21.15.102:0/959091548 shutdown_connections 2021-05-25T12:00:05.609 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.603+0000 7f5d72d44700 1 -- 172.21.15.102:0/959091548 wait complete. 2021-05-25T12:00:05.610 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:05.603+0000 7f5d72d44700 1 librados: shutdown 2021-05-25T12:00:08.617 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2021-05-25T12:00:08.735 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:08.733+0000 7f1adeb47700 10 monclient: get_monmap_and_config 2021-05-25T12:00:08.736 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:08.733+0000 7f1adeb47700 10 monclient: build_initial_monmap 2021-05-25T12:00:08.736 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:08.733+0000 7f1adeb47700 10 monclient: monmap: 2021-05-25T12:00:08.736 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:00:08.736 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:08.736 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:00:08.734606+0000 2021-05-25T12:00:08.737 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:00:08.734606+0000 2021-05-25T12:00:08.737 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:00:08.737 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:08.737 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:00:08.738 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:00:08.738 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:00:08.738 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:08.738 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:08.734+0000 7f1adeb47700 1 Processor -- start 2021-05-25T12:00:08.739 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:08.734+0000 7f1adeb47700 1 -- start start 2021-05-25T12:00:08.739 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:08.734+0000 7f1adeb47700 10 monclient: init 2021-05-25T12:00:08.741 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:08.734+0000 7f1adeb47700 10 monclient: _reopen_session rank -1 2021-05-25T12:00:08.741 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:08.734+0000 7f1adeb47700 10 monclient: _add_conns ranks=[0,1,2] 2021-05-25T12:00:08.741 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:08.734+0000 7f1adeb47700 10 monclient(hunting): picked mon.noname-a con 0x7f1ad80975f0 addr v1:172.21.15.102:6789/0 2021-05-25T12:00:08.742 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:08.734+0000 7f1adeb47700 10 monclient(hunting): picked mon.noname-c con 0x7f1ad80980e0 addr v1:172.21.15.181:6789/0 2021-05-25T12:00:08.742 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:08.734+0000 7f1adeb47700 10 monclient(hunting): picked mon.noname-b con 0x7f1ad8094f60 addr v1:172.21.15.102:6790/0 2021-05-25T12:00:08.742 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:08.734+0000 7f1adeb47700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ad8064400 con 0x7f1ad80975f0 2021-05-25T12:00:08.742 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:08.734+0000 7f1adeb47700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ad80646a0 con 0x7f1ad80980e0 2021-05-25T12:00:08.743 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:08.734+0000 7f1adeb47700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ad8064940 con 0x7f1ad8094f60 2021-05-25T12:00:08.743 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:08.734+0000 7f1adeb47700 10 monclient(hunting): _renew_subs 2021-05-25T12:00:08.743 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:08.734+0000 7f1adeb47700 10 monclient(hunting): authenticate will time out at 809.331923s 2021-05-25T12:00:08.743 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:08.735+0000 7f1add344700 1 -- >> v1:172.21.15.181:6789/0 conn(0x7f1ad80980e0 legacy=0x7f1ad8131150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2021-05-25T12:00:08.744 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:08.735+0000 7f1addb45700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f1ad80975f0 0x7f1ad80979d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:59204/0 (socket says 172.21.15.102:59204) 2021-05-25T12:00:08.744 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:08.735+0000 7f1addb45700 1 -- 172.21.15.102:0/448929997 learned_addr learned my addr 172.21.15.102:0/448929997 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:00:08.744 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:08.735+0000 7f1ade346700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f1ad8094f60 0x7f1ad8134880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:54086/0 (socket says 172.21.15.102:54086) 2021-05-25T12:00:08.937 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:08.936+0000 7f1add344700 1 -- 172.21.15.102:0/448929997 >> v1:172.21.15.181:6789/0 conn(0x7f1ad80980e0 legacy=0x7f1ad8131150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2021-05-25T12:00:09.338 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:09.337+0000 7f1add344700 1 -- 172.21.15.102:0/448929997 >> v1:172.21.15.181:6789/0 conn(0x7f1ad80980e0 legacy=0x7f1ad8131150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2021-05-25T12:00:10.141 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:10.138+0000 7f1add344700 1 -- 172.21.15.102:0/448929997 >> v1:172.21.15.181:6789/0 conn(0x7f1ad80980e0 legacy=0x7f1ad8131150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2021-05-25T12:00:11.737 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:11.735+0000 7f1ac77fe700 10 monclient(hunting): tick 2021-05-25T12:00:11.737 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:11.735+0000 7f1ac77fe700 1 monclient(hunting): continuing hunt 2021-05-25T12:00:11.737 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:11.735+0000 7f1ac77fe700 10 monclient(hunting): _reopen_session rank -1 2021-05-25T12:00:11.738 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:11.735+0000 7f1ac77fe700 1 -- 172.21.15.102:0/448929997 >> v1:172.21.15.102:6790/0 conn(0x7f1ad8094f60 legacy=0x7f1ad8134880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:11.738 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:11.735+0000 7f1ac77fe700 1 -- 172.21.15.102:0/448929997 >> v1:172.21.15.181:6789/0 conn(0x7f1ad80980e0 legacy=0x7f1ad8131150 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-05-25T12:00:11.738 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:11.735+0000 7f1ac77fe700 1 -- 172.21.15.102:0/448929997 >> v1:172.21.15.102:6789/0 conn(0x7f1ad80975f0 legacy=0x7f1ad80979d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:11.738 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:11.735+0000 7f1ac77fe700 10 monclient: _add_conns ranks=[1,0,2] 2021-05-25T12:00:11.739 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:11.735+0000 7f1ac77fe700 10 monclient(hunting): picked mon.noname-c con 0x7f1ad80980e0 addr v1:172.21.15.181:6789/0 2021-05-25T12:00:11.739 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:11.735+0000 7f1ac77fe700 10 monclient(hunting): picked mon.noname-a con 0x7f1ab8006b70 addr v1:172.21.15.102:6789/0 2021-05-25T12:00:11.739 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:11.735+0000 7f1ac77fe700 10 monclient(hunting): picked mon.noname-b con 0x7f1ad8094f60 addr v1:172.21.15.102:6790/0 2021-05-25T12:00:11.739 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:11.735+0000 7f1ac77fe700 1 -- 172.21.15.102:0/448929997 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ad8064be0 con 0x7f1ab8006b70 2021-05-25T12:00:11.739 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:11.735+0000 7f1ac77fe700 1 -- 172.21.15.102:0/448929997 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ad80646a0 con 0x7f1ad80980e0 2021-05-25T12:00:11.740 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:11.735+0000 7f1ac77fe700 1 -- 172.21.15.102:0/448929997 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ab8010140 con 0x7f1ad8094f60 2021-05-25T12:00:11.740 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:11.735+0000 7f1ac77fe700 10 monclient(hunting): _renew_subs 2021-05-25T12:00:11.740 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:11.735+0000 7f1addb45700 1 -- 172.21.15.102:0/448929997 >> v1:172.21.15.181:6789/0 conn(0x7f1ad80980e0 legacy=0x7f1ab8005430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2021-05-25T12:00:11.937 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:11.936+0000 7f1addb45700 1 -- 172.21.15.102:0/448929997 >> v1:172.21.15.181:6789/0 conn(0x7f1ad80980e0 legacy=0x7f1ab8005430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2021-05-25T12:00:12.338 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:12.337+0000 7f1addb45700 1 -- 172.21.15.102:0/448929997 >> v1:172.21.15.181:6789/0 conn(0x7f1ad80980e0 legacy=0x7f1ab8005430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2021-05-25T12:00:13.139 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:13.138+0000 7f1addb45700 1 -- 172.21.15.102:0/448929997 >> v1:172.21.15.181:6789/0 conn(0x7f1ad80980e0 legacy=0x7f1ab8005430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2021-05-25T12:00:14.737 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:14.735+0000 7f1ac77fe700 10 monclient(hunting): tick 2021-05-25T12:00:14.737 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:14.735+0000 7f1ac77fe700 1 monclient(hunting): continuing hunt 2021-05-25T12:00:14.738 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:14.735+0000 7f1ac77fe700 10 monclient(hunting): _reopen_session rank -1 2021-05-25T12:00:14.738 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:14.735+0000 7f1ac77fe700 1 -- 172.21.15.102:0/448929997 >> v1:172.21.15.102:6790/0 conn(0x7f1ad8094f60 legacy=0x7f1ab800c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:14.738 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:14.735+0000 7f1ac77fe700 1 -- 172.21.15.102:0/448929997 >> v1:172.21.15.181:6789/0 conn(0x7f1ad80980e0 legacy=0x7f1ab8005430 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-05-25T12:00:14.739 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:14.735+0000 7f1ac77fe700 1 -- 172.21.15.102:0/448929997 >> v1:172.21.15.102:6789/0 conn(0x7f1ab8006b70 legacy=0x7f1ab8009000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:14.739 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:14.735+0000 7f1ac77fe700 10 monclient: _add_conns ranks=[0,1,2] 2021-05-25T12:00:14.739 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:14.735+0000 7f1ac77fe700 10 monclient(hunting): picked mon.noname-a con 0x7f1ab8010480 addr v1:172.21.15.102:6789/0 2021-05-25T12:00:14.739 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:14.735+0000 7f1ac77fe700 10 monclient(hunting): picked mon.noname-c con 0x7f1ad80980e0 addr v1:172.21.15.181:6789/0 2021-05-25T12:00:14.740 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:14.735+0000 7f1ac77fe700 10 monclient(hunting): picked mon.noname-b con 0x7f1ab8012690 addr v1:172.21.15.102:6790/0 2021-05-25T12:00:14.740 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:14.735+0000 7f1ac77fe700 1 -- 172.21.15.102:0/448929997 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ab8011410 con 0x7f1ab8010480 2021-05-25T12:00:14.740 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:14.735+0000 7f1ac77fe700 1 -- 172.21.15.102:0/448929997 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ad80646a0 con 0x7f1ad80980e0 2021-05-25T12:00:14.740 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:14.735+0000 7f1ac77fe700 1 -- 172.21.15.102:0/448929997 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ab800a890 con 0x7f1ab8012690 2021-05-25T12:00:14.741 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:14.735+0000 7f1ac77fe700 10 monclient(hunting): _renew_subs 2021-05-25T12:00:14.741 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:14.735+0000 7f1add344700 1 -- 172.21.15.102:0/448929997 >> v1:172.21.15.181:6789/0 conn(0x7f1ad80980e0 legacy=0x7f1ab8005430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2021-05-25T12:00:14.937 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:14.936+0000 7f1add344700 1 -- 172.21.15.102:0/448929997 >> v1:172.21.15.181:6789/0 conn(0x7f1ad80980e0 legacy=0x7f1ab8005430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2021-05-25T12:00:15.338 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:15.337+0000 7f1add344700 1 -- 172.21.15.102:0/448929997 >> v1:172.21.15.181:6789/0 conn(0x7f1ad80980e0 legacy=0x7f1ab8005430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2021-05-25T12:00:16.053 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.050+0000 7f1adcb43700 1 -- 172.21.15.102:0/448929997 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3574518314 0 0) 0x7f1ab8011410 con 0x7f1ab8010480 2021-05-25T12:00:16.053 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.050+0000 7f1adcb43700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:16.053 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.050+0000 7f1adcb43700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:16.054 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.050+0000 7f1adcb43700 10 monclient(hunting): my global_id is 4428 2021-05-25T12:00:16.054 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.050+0000 7f1adcb43700 1 -- 172.21.15.102:0/448929997 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ac0003270 con 0x7f1ab8010480 2021-05-25T12:00:16.054 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.051+0000 7f1adcb43700 1 -- 172.21.15.102:0/448929997 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1873940100 0 0) 0x7f1ac0003270 con 0x7f1ab8010480 2021-05-25T12:00:16.054 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.051+0000 7f1adcb43700 1 -- 172.21.15.102:0/448929997 >> v1:172.21.15.102:6790/0 conn(0x7f1ab8012690 legacy=0x7f1ab8014a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:16.055 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.051+0000 7f1adcb43700 1 -- 172.21.15.102:0/448929997 >> v1:172.21.15.181:6789/0 conn(0x7f1ad80980e0 legacy=0x7f1ab8005430 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-05-25T12:00:16.055 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.051+0000 7f1adcb43700 10 monclient: _finish_hunting 0 2021-05-25T12:00:16.055 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.051+0000 7f1adcb43700 1 monclient: found mon.noname-a 2021-05-25T12:00:16.055 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.051+0000 7f1adcb43700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:00:16.055 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.051+0000 7f1adcb43700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T12:00:16.056 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.051+0000 7f1adcb43700 1 -- 172.21.15.102:0/448929997 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ab8009730 con 0x7f1ab8010480 2021-05-25T12:00:16.056 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.051+0000 7f1adcb43700 10 monclient: _finish_auth 0 2021-05-25T12:00:16.056 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.051+0000 7f1adcb43700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:00:16.056 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.051+0000 7f1adeb47700 5 monclient: authenticate success, global_id 4428 2021-05-25T12:00:16.056 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.051+0000 7f1adcb43700 1 -- 172.21.15.102:0/448929997 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f1ac80015d0 con 0x7f1ab8010480 2021-05-25T12:00:16.057 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.051+0000 7f1adcb43700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:16.057 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.051+0000 7f1adcb43700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:00:16.057 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.051+0000 7f1adcb43700 10 monclient: dump: 2021-05-25T12:00:16.057 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:16.058 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:16.058 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:16.058 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:16.058 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:16.058 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:16.059 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:16.059 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:16.059 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:16.059 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:16.060 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.051+0000 7f1adeb47700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:00:16.067 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.053+0000 7f1adcb43700 1 -- 172.21.15.102:0/448929997 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1ac8001800 con 0x7f1ab8010480 2021-05-25T12:00:16.067 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.053+0000 7f1adcb43700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:00:16.067 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.053+0000 7f1adcb43700 1 -- 172.21.15.102:0/448929997 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f1ac8001bd0 con 0x7f1ab8010480 2021-05-25T12:00:16.067 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.053+0000 7f1adeb47700 10 monclient: get_monmap_and_config success 2021-05-25T12:00:16.068 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.053+0000 7f1adeb47700 4 set_mon_vals no callback set 2021-05-25T12:00:16.068 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.053+0000 7f1adcb43700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:16.068 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.053+0000 7f1adcb43700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:00:16.068 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.053+0000 7f1adcb43700 10 monclient: dump: 2021-05-25T12:00:16.068 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:16.069 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:16.069 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:16.069 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:16.069 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:16.070 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:16.070 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:16.070 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:16.070 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:16.070 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:16.071 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.054+0000 7f1adeb47700 10 monclient: shutdown 2021-05-25T12:00:16.071 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.054+0000 7f1adeb47700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:00:16.071 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.054+0000 7f1adeb47700 1 -- 172.21.15.102:0/448929997 >> v1:172.21.15.102:6789/0 conn(0x7f1ab8010480 legacy=0x7f1ab80108c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:16.072 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.054+0000 7f1adeb47700 1 -- 172.21.15.102:0/448929997 shutdown_connections 2021-05-25T12:00:16.072 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.054+0000 7f1adeb47700 1 -- 172.21.15.102:0/448929997 >> 172.21.15.102:0/448929997 conn(0x7f1ad809c930 msgr2=0x7f1ad809cd40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:00:16.072 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.054+0000 7f1adeb47700 1 -- 172.21.15.102:0/448929997 shutdown_connections 2021-05-25T12:00:16.072 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.054+0000 7f1adeb47700 1 -- 172.21.15.102:0/448929997 wait complete. 2021-05-25T12:00:16.073 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.055+0000 7f1adeb47700 10 monclient: build_initial_monmap 2021-05-25T12:00:16.073 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.055+0000 7f1adeb47700 10 monclient: monmap: 2021-05-25T12:00:16.073 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:00:16.073 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:00:16.073 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:00:16.074 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:00:16.074 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:00:16.074 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:00:16.074 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:00:16.075 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:00:16.075 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:00:16.075 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:16.075 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.056+0000 7f1adeb47700 1 librados: starting msgr at 2021-05-25T12:00:16.075 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.056+0000 7f1adeb47700 1 librados: starting objecter 2021-05-25T12:00:16.076 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.056+0000 7f1adeb47700 1 Processor -- start 2021-05-25T12:00:16.076 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.056+0000 7f1adeb47700 1 -- start start 2021-05-25T12:00:16.076 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.056+0000 7f1adeb47700 1 librados: setting wanted keys 2021-05-25T12:00:16.076 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.056+0000 7f1adeb47700 1 librados: calling monclient init 2021-05-25T12:00:16.076 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.056+0000 7f1adeb47700 10 monclient: init 2021-05-25T12:00:16.077 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.056+0000 7f1adeb47700 10 monclient: _reopen_session rank -1 2021-05-25T12:00:16.077 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.056+0000 7f1adeb47700 10 monclient: _add_conns ranks=[2,1,0] 2021-05-25T12:00:16.077 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.056+0000 7f1adeb47700 10 monclient(hunting): picked mon.noname-c con 0x7f1ab8010480 addr v1:172.21.15.102:6790/0 2021-05-25T12:00:16.077 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.056+0000 7f1adeb47700 10 monclient(hunting): picked mon.noname-b con 0x7f1ab8012690 addr v1:172.21.15.181:6789/0 2021-05-25T12:00:16.078 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.057+0000 7f1adeb47700 10 monclient(hunting): picked mon.noname-a con 0x7f1ad80980e0 addr v1:172.21.15.102:6789/0 2021-05-25T12:00:16.078 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.057+0000 7f1adeb47700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ad81194a0 con 0x7f1ad80980e0 2021-05-25T12:00:16.078 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.057+0000 7f1adeb47700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ad8119740 con 0x7f1ab8012690 2021-05-25T12:00:16.078 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.057+0000 7f1adeb47700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ad81199e0 con 0x7f1ab8010480 2021-05-25T12:00:16.078 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.057+0000 7f1add344700 1 -- >> v1:172.21.15.181:6789/0 conn(0x7f1ab8012690 legacy=0x7f1ad812c160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.181:6789/0 2021-05-25T12:00:16.079 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.057+0000 7f1ade346700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f1ad80980e0 0x7f1ad81438b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:59246/0 (socket says 172.21.15.102:59246) 2021-05-25T12:00:16.079 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.057+0000 7f1addb45700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f1ab8010480 0x7f1ad8130900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:54120/0 (socket says 172.21.15.102:54120) 2021-05-25T12:00:16.079 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.057+0000 7f1ade346700 1 -- 172.21.15.102:0/1009633177 learned_addr learned my addr 172.21.15.102:0/1009633177 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:00:16.079 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.057+0000 7f1adeb47700 10 monclient(hunting): _renew_subs 2021-05-25T12:00:16.080 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.057+0000 7f1adeb47700 10 monclient(hunting): authenticate will time out at 816.654219s 2021-05-25T12:00:16.080 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.057+0000 7f1ac5ffb700 10 client.?.objecter ms_handle_connect 0x7f1ad80980e0 2021-05-25T12:00:16.080 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.057+0000 7f1ac5ffb700 10 client.?.objecter resend_mon_ops 2021-05-25T12:00:16.080 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.057+0000 7f1ac5ffb700 10 client.?.objecter ms_handle_connect 0x7f1ab8010480 2021-05-25T12:00:16.080 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.057+0000 7f1ac5ffb700 10 client.?.objecter resend_mon_ops 2021-05-25T12:00:16.081 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.057+0000 7f1ac5ffb700 1 -- 172.21.15.102:0/1009633177 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3501271495 0 0) 0x7f1ad81194a0 con 0x7f1ad80980e0 2021-05-25T12:00:16.081 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.057+0000 7f1ac5ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:16.081 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.057+0000 7f1ac5ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:16.081 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.057+0000 7f1ac5ffb700 10 monclient(hunting): my global_id is 4431 2021-05-25T12:00:16.081 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.057+0000 7f1ac5ffb700 1 -- 172.21.15.102:0/1009633177 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ab8009530 con 0x7f1ad80980e0 2021-05-25T12:00:16.082 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.057+0000 7f1ac5ffb700 1 -- 172.21.15.102:0/1009633177 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3125528270 0 0) 0x7f1ad81199e0 con 0x7f1ab8010480 2021-05-25T12:00:16.082 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.057+0000 7f1ac5ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:16.082 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.057+0000 7f1ac5ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:16.082 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.057+0000 7f1ac5ffb700 10 monclient(hunting): my global_id is 4415 2021-05-25T12:00:16.083 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.057+0000 7f1ac5ffb700 1 -- 172.21.15.102:0/1009633177 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ad81194a0 con 0x7f1ab8010480 2021-05-25T12:00:16.083 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.057+0000 7f1ac5ffb700 1 -- 172.21.15.102:0/1009633177 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1410269579 0 0) 0x7f1ab8009530 con 0x7f1ad80980e0 2021-05-25T12:00:16.083 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.057+0000 7f1ac5ffb700 1 -- 172.21.15.102:0/1009633177 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1ad81199e0 con 0x7f1ad80980e0 2021-05-25T12:00:16.083 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.057+0000 7f1ac5ffb700 1 -- 172.21.15.102:0/1009633177 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2959983429 0 0) 0x7f1ad81194a0 con 0x7f1ab8010480 2021-05-25T12:00:16.083 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.057+0000 7f1ac5ffb700 1 -- 172.21.15.102:0/1009633177 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1ab8009530 con 0x7f1ab8010480 2021-05-25T12:00:16.084 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.057+0000 7f1ac5ffb700 1 -- 172.21.15.102:0/1009633177 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f1ad4001ad0 con 0x7f1ad80980e0 2021-05-25T12:00:16.084 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.057+0000 7f1ac5ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:16.084 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.057+0000 7f1ac5ffb700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:00:16.084 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.057+0000 7f1ac5ffb700 10 monclient(hunting): dump: 2021-05-25T12:00:16.085 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:16.085 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:16.085 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:16.085 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:16.085 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:16.086 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:16.086 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:16.086 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:16.086 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:16.087 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:16.087 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.057+0000 7f1ac5ffb700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:00:16.087 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.057+0000 7f1ac5ffb700 1 -- 172.21.15.102:0/1009633177 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f1ac8001b10 con 0x7f1ab8010480 2021-05-25T12:00:16.087 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.057+0000 7f1ac5ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:16.087 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.057+0000 7f1ac5ffb700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2021-05-25T12:00:16.088 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.057+0000 7f1ac5ffb700 10 monclient(hunting): dump: 2021-05-25T12:00:16.088 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:16.088 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:16.088 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:16.089 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:16.089 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:16.089 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:16.089 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:16.089 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:16.090 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:16.090 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:16.090 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.058+0000 7f1ac5ffb700 1 -- 172.21.15.102:0/1009633177 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2018266786 0 0) 0x7f1ad81199e0 con 0x7f1ad80980e0 2021-05-25T12:00:16.090 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.058+0000 7f1ac5ffb700 1 -- 172.21.15.102:0/1009633177 >> v1:172.21.15.102:6790/0 conn(0x7f1ab8010480 legacy=0x7f1ad8130900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:16.090 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.058+0000 7f1ac5ffb700 1 -- 172.21.15.102:0/1009633177 >> v1:172.21.15.181:6789/0 conn(0x7f1ab8012690 legacy=0x7f1ad812c160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-05-25T12:00:16.091 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.058+0000 7f1ac5ffb700 10 monclient: _finish_hunting 0 2021-05-25T12:00:16.091 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.058+0000 7f1ac5ffb700 1 monclient: found mon.a 2021-05-25T12:00:16.091 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.058+0000 7f1ac5ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:00:16.091 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.058+0000 7f1ac5ffb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:00:16.092 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.058+0000 7f1ac5ffb700 1 -- 172.21.15.102:0/1009633177 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ad8117870 con 0x7f1ad80980e0 2021-05-25T12:00:16.092 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.058+0000 7f1ac5ffb700 10 monclient: _finish_auth 0 2021-05-25T12:00:16.092 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.058+0000 7f1ac5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:00:16.092 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.058+0000 7f1adeb47700 5 monclient: authenticate success, global_id 4431 2021-05-25T12:00:16.092 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.058+0000 7f1adeb47700 10 monclient: _renew_subs 2021-05-25T12:00:16.093 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.058+0000 7f1adeb47700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:00:16.093 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.058+0000 7f1adeb47700 1 -- 172.21.15.102:0/1009633177 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ad812c870 con 0x7f1ad80980e0 2021-05-25T12:00:16.093 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.058+0000 7f1ac5ffb700 1 -- 172.21.15.102:0/1009633177 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1ad4001cc0 con 0x7f1ad80980e0 2021-05-25T12:00:16.093 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.058+0000 7f1ac5ffb700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:00:16.094 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.058+0000 7f1ac5ffb700 1 -- 172.21.15.102:0/1009633177 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f1ad4004ad0 con 0x7f1ad80980e0 2021-05-25T12:00:16.094 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.058+0000 7f1ac5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:16.094 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.058+0000 7f1ac5ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:00:16.094 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.058+0000 7f1ac5ffb700 10 monclient: dump: 2021-05-25T12:00:16.094 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:16.095 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:16.095 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:16.095 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:16.095 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:16.096 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:16.096 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:16.096 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:16.096 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:16.096 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:16.097 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.058+0000 7f1ac7fff700 4 set_mon_vals no callback set 2021-05-25T12:00:16.097 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.058+0000 7f1adeb47700 10 client.4431.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:00:16.097 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.058+0000 7f1adeb47700 10 monclient: _renew_subs 2021-05-25T12:00:16.097 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.058+0000 7f1adeb47700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:00:16.098 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.058+0000 7f1adeb47700 1 -- 172.21.15.102:0/1009633177 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ad8116f30 con 0x7f1ad80980e0 2021-05-25T12:00:16.098 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.058+0000 7f1adeb47700 1 librados: init done 2021-05-25T12:00:16.098 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.059+0000 7f1ac5ffb700 1 -- 172.21.15.102:0/1009633177 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f1ad4004cb0 con 0x7f1ad80980e0 2021-05-25T12:00:16.098 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.059+0000 7f1ac5ffb700 1 -- 172.21.15.102:0/1009633177 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(15..15 src has 1..15) v4 ==== 3549+0+0 (unknown 3948819498 0 0) 0x7f1ad403cfd0 con 0x7f1ad80980e0 2021-05-25T12:00:16.099 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.059+0000 7f1ac5ffb700 10 client.4431.objecter ms_dispatch 0x7f1ad8001550 osd_map(15..15 src has 1..15) v4 2021-05-25T12:00:16.099 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.059+0000 7f1ac5ffb700 3 client.4431.objecter handle_osd_map got epochs [15,15] > 0 2021-05-25T12:00:16.099 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.059+0000 7f1ac5ffb700 3 client.4431.objecter handle_osd_map decoding full epoch 15 2021-05-25T12:00:16.099 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.059+0000 7f1ac5ffb700 20 client.4431.objecter dump_active .. 0 homeless 2021-05-25T12:00:16.099 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.059+0000 7f1addb45700 10 monclient: get_auth_request con 0x7f1ab80346b0 auth_method 0 2021-05-25T12:00:16.100 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.059+0000 7f1adeb47700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:00:16.100 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.059+0000 7f1ac5ffb700 10 client.4431.objecter ms_handle_connect 0x7f1ab80346b0 2021-05-25T12:00:16.100 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.059+0000 7f1adeb47700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:00:16.101 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.059+0000 7f1adeb47700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:00:16.101 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.059+0000 7f1adeb47700 1 -- 172.21.15.102:0/1009633177 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ad8119300 con 0x7f1ad80980e0 2021-05-25T12:00:16.101 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.064+0000 7f1ac5ffb700 1 -- 172.21.15.102:0/1009633177 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7f1ad8119300 con 0x7f1ad80980e0 2021-05-25T12:00:16.101 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.064+0000 7f1ac5ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:00:16.101 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.064+0000 7f1ac5ffb700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:00:16.275 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.273+0000 7f1adeb47700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2021-05-25T12:00:16.275 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.273+0000 7f1adeb47700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2021-05-25T12:00:16.276 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.273+0000 7f1adeb47700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:00:16.276 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.273+0000 7f1adeb47700 1 -- 172.21.15.102:0/1009633177 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f1ad8119300 con 0x7f1ad80980e0 2021-05-25T12:00:16.276 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.273+0000 7f1ac5ffb700 1 -- 172.21.15.102:0/1009633177 <== mon.0 v1:172.21.15.102:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1336 (unknown 254746657 0 164109427) 0x7f1ad8119300 con 0x7f1ad80980e0 2021-05-25T12:00:16.277 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.273+0000 7f1ac5ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2021-05-25T12:00:16.277 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.273+0000 7f1ac5ffb700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:00:16.278 INFO:teuthology.orchestra.run.smithi102.stdout:{"election_epoch":12,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138303579357183","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"20ee91b6-8935-4b98-b868-85c03789ccc4","modified":"2021-05-25T11:59:11.011522Z","created":"2021-05-25T11:59:11.011522Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6790","nonce":0}]},"addr":"172.21.15.102:6790/0","public_addr":"172.21.15.102:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2021-05-25T12:00:16.279 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.275+0000 7f1adeb47700 10 librados: watch_flush enter 2021-05-25T12:00:16.279 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.275+0000 7f1adeb47700 10 librados: watch_flush exit 2021-05-25T12:00:16.280 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.275+0000 7f1adeb47700 20 client.4431.objecter shutdown clearing up homeless session... 2021-05-25T12:00:16.280 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.275+0000 7f1adeb47700 10 client.4431.objecter successfully canceled tick 2021-05-25T12:00:16.280 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.275+0000 7f1adeb47700 1 -- 172.21.15.102:0/1009633177 >> v1:172.21.15.181:6812/29582 conn(0x7f1ab80346b0 legacy=0x7f1ab8036b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:16.280 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.275+0000 7f1adeb47700 10 monclient: shutdown 2021-05-25T12:00:16.281 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.275+0000 7f1adeb47700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:00:16.281 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.275+0000 7f1adeb47700 1 -- 172.21.15.102:0/1009633177 >> v1:172.21.15.102:6789/0 conn(0x7f1ad80980e0 legacy=0x7f1ad81438b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:16.281 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.275+0000 7f1adeb47700 1 -- 172.21.15.102:0/1009633177 shutdown_connections 2021-05-25T12:00:16.281 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.275+0000 7f1adeb47700 1 -- 172.21.15.102:0/1009633177 >> 172.21.15.102:0/1009633177 conn(0x7f1ad809c930 msgr2=0x7f1ad8066260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:00:16.282 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.275+0000 7f1adeb47700 1 -- 172.21.15.102:0/1009633177 shutdown_connections 2021-05-25T12:00:16.282 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.275+0000 7f1adeb47700 1 -- 172.21.15.102:0/1009633177 wait complete. 2021-05-25T12:00:16.282 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.275+0000 7f1adeb47700 1 librados: shutdown 2021-05-25T12:00:16.287 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2021-05-25T12:00:16.288 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2021-05-25T12:00:16.398 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.396+0000 7f8d0420d700 10 monclient: get_monmap_and_config 2021-05-25T12:00:16.398 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.396+0000 7f8d0420d700 10 monclient: build_initial_monmap 2021-05-25T12:00:16.399 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.396+0000 7f8d0420d700 10 monclient: monmap: 2021-05-25T12:00:16.399 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:00:16.399 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:16.399 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:00:16.397577+0000 2021-05-25T12:00:16.400 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:00:16.397577+0000 2021-05-25T12:00:16.400 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:00:16.401 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:16.401 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:00:16.401 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:00:16.401 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:00:16.401 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:16.402 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.397+0000 7f8d0420d700 1 Processor -- start 2021-05-25T12:00:16.402 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.397+0000 7f8d0420d700 1 -- start start 2021-05-25T12:00:16.402 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.397+0000 7f8d0420d700 10 monclient: init 2021-05-25T12:00:16.417 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.397+0000 7f8d0420d700 10 monclient: _reopen_session rank -1 2021-05-25T12:00:16.417 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.397+0000 7f8d0420d700 10 monclient: _add_conns ranks=[1,2,0] 2021-05-25T12:00:16.417 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.397+0000 7f8d0420d700 10 monclient(hunting): picked mon.noname-c con 0x7f8cfc068f60 addr v1:172.21.15.181:6789/0 2021-05-25T12:00:16.418 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.397+0000 7f8d0420d700 10 monclient(hunting): picked mon.noname-b con 0x7f8cfc096800 addr v1:172.21.15.102:6790/0 2021-05-25T12:00:16.418 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.397+0000 7f8d0420d700 10 monclient(hunting): picked mon.noname-a con 0x7f8cfc131ae0 addr v1:172.21.15.102:6789/0 2021-05-25T12:00:16.418 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.397+0000 7f8d0420d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cfc129040 con 0x7f8cfc131ae0 2021-05-25T12:00:16.418 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.397+0000 7f8d0420d700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cfc097360 con 0x7f8cfc068f60 2021-05-25T12:00:16.418 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.397+0000 7f8d0420d700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cfc069a50 con 0x7f8cfc096800 2021-05-25T12:00:16.419 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.397+0000 7f8d0420d700 10 monclient(hunting): _renew_subs 2021-05-25T12:00:16.419 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.397+0000 7f8d0420d700 10 monclient(hunting): authenticate will time out at 816.994818s 2021-05-25T12:00:16.419 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.397+0000 7f8d01fa9700 1 -- >> v1:172.21.15.181:6789/0 conn(0x7f8cfc068f60 legacy=0x7f8cfc128930 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2021-05-25T12:00:16.419 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.397+0000 7f8d027aa700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f8cfc131ae0 0x7f8cfc133ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:59252/0 (socket says 172.21.15.102:59252) 2021-05-25T12:00:16.420 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.397+0000 7f8d027aa700 1 -- 172.21.15.102:0/324254631 learned_addr learned my addr 172.21.15.102:0/324254631 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:00:16.420 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.397+0000 7f8d017a8700 1 --1- 172.21.15.102:0/324254631 >> v1:172.21.15.102:6790/0 conn(0x7f8cfc096800 0x7f8cfc069340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:54128/0 (socket says 172.21.15.102:54128) 2021-05-25T12:00:16.420 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.398+0000 7f8d00fa7700 1 -- 172.21.15.102:0/324254631 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4009045974 0 0) 0x7f8cfc069a50 con 0x7f8cfc096800 2021-05-25T12:00:16.420 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.398+0000 7f8d00fa7700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:16.421 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.398+0000 7f8d00fa7700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:16.421 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.398+0000 7f8d00fa7700 10 monclient(hunting): my global_id is 4418 2021-05-25T12:00:16.421 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.398+0000 7f8d00fa7700 1 -- 172.21.15.102:0/324254631 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ce4003270 con 0x7f8cfc096800 2021-05-25T12:00:16.421 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.398+0000 7f8d00fa7700 1 -- 172.21.15.102:0/324254631 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 375250873 0 0) 0x7f8cfc129040 con 0x7f8cfc131ae0 2021-05-25T12:00:16.422 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.398+0000 7f8d00fa7700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:16.422 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.398+0000 7f8d00fa7700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:16.422 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.398+0000 7f8d00fa7700 10 monclient(hunting): my global_id is 4434 2021-05-25T12:00:16.422 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.398+0000 7f8d00fa7700 1 -- 172.21.15.102:0/324254631 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cfc069a50 con 0x7f8cfc131ae0 2021-05-25T12:00:16.423 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.398+0000 7f8d00fa7700 1 -- 172.21.15.102:0/324254631 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3590954777 0 0) 0x7f8cfc069a50 con 0x7f8cfc131ae0 2021-05-25T12:00:16.423 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.398+0000 7f8d00fa7700 1 -- 172.21.15.102:0/324254631 >> v1:172.21.15.102:6790/0 conn(0x7f8cfc096800 legacy=0x7f8cfc069340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:16.423 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.398+0000 7f8d00fa7700 1 -- 172.21.15.102:0/324254631 >> v1:172.21.15.181:6789/0 conn(0x7f8cfc068f60 legacy=0x7f8cfc128930 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-05-25T12:00:16.423 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.398+0000 7f8d00fa7700 10 monclient: _finish_hunting 0 2021-05-25T12:00:16.424 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.398+0000 7f8d00fa7700 1 monclient: found mon.noname-a 2021-05-25T12:00:16.424 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.398+0000 7f8d00fa7700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:00:16.424 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.398+0000 7f8d00fa7700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T12:00:16.424 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.398+0000 7f8d00fa7700 1 -- 172.21.15.102:0/324254631 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cfc061ce0 con 0x7f8cfc131ae0 2021-05-25T12:00:16.425 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.398+0000 7f8d00fa7700 10 monclient: _finish_auth 0 2021-05-25T12:00:16.425 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.398+0000 7f8d00fa7700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:00:16.425 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.398+0000 7f8d0420d700 5 monclient: authenticate success, global_id 4434 2021-05-25T12:00:16.425 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.398+0000 7f8d00fa7700 1 -- 172.21.15.102:0/324254631 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f8cf8001970 con 0x7f8cfc131ae0 2021-05-25T12:00:16.426 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.398+0000 7f8d0420d700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:00:16.426 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.398+0000 7f8d00fa7700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:16.426 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.398+0000 7f8d00fa7700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:00:16.426 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.398+0000 7f8d00fa7700 10 monclient: dump: 2021-05-25T12:00:16.427 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:16.427 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:16.427 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:16.427 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:16.427 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:16.428 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:16.428 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:16.428 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:16.428 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:16.429 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:16.429 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.398+0000 7f8d0420d700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:00:16.429 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.398+0000 7f8d00fa7700 1 -- 172.21.15.102:0/324254631 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8cf8001c10 con 0x7f8cfc131ae0 2021-05-25T12:00:16.430 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.398+0000 7f8d00fa7700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:00:16.430 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.398+0000 7f8d00fa7700 1 -- 172.21.15.102:0/324254631 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f8cf8001fa0 con 0x7f8cfc131ae0 2021-05-25T12:00:16.430 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.398+0000 7f8d0420d700 10 monclient: get_monmap_and_config success 2021-05-25T12:00:16.431 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.398+0000 7f8d00fa7700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:16.431 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.398+0000 7f8d0420d700 4 set_mon_vals no callback set 2021-05-25T12:00:16.431 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.398+0000 7f8d00fa7700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:00:16.432 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.398+0000 7f8d00fa7700 10 monclient: dump: 2021-05-25T12:00:16.432 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:16.432 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:16.432 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:16.433 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:16.433 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:16.433 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:16.433 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:16.434 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:16.434 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:16.434 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:16.434 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.398+0000 7f8d0420d700 10 monclient: shutdown 2021-05-25T12:00:16.435 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.398+0000 7f8d0420d700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:00:16.435 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.398+0000 7f8d0420d700 1 -- 172.21.15.102:0/324254631 >> v1:172.21.15.102:6789/0 conn(0x7f8cfc131ae0 legacy=0x7f8cfc133ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:16.435 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.398+0000 7f8d0420d700 1 -- 172.21.15.102:0/324254631 shutdown_connections 2021-05-25T12:00:16.435 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.398+0000 7f8d0420d700 1 -- 172.21.15.102:0/324254631 >> 172.21.15.102:0/324254631 conn(0x7f8cfc09b920 msgr2=0x7f8cfc09bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:00:16.436 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.399+0000 7f8d0420d700 1 -- 172.21.15.102:0/324254631 shutdown_connections 2021-05-25T12:00:16.436 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.399+0000 7f8d0420d700 1 -- 172.21.15.102:0/324254631 wait complete. 2021-05-25T12:00:16.436 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.399+0000 7f8d0420d700 10 monclient: build_initial_monmap 2021-05-25T12:00:16.436 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.399+0000 7f8d0420d700 10 monclient: monmap: 2021-05-25T12:00:16.437 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:00:16.437 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:00:16.437 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:00:16.437 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:00:16.438 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:00:16.438 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:00:16.438 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:00:16.438 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:00:16.439 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:00:16.439 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:16.439 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.399+0000 7f8d0420d700 1 librados: starting msgr at 2021-05-25T12:00:16.439 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.399+0000 7f8d0420d700 1 librados: starting objecter 2021-05-25T12:00:16.440 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.399+0000 7f8d0420d700 1 Processor -- start 2021-05-25T12:00:16.440 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.399+0000 7f8d0420d700 1 -- start start 2021-05-25T12:00:16.440 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.399+0000 7f8d0420d700 1 librados: setting wanted keys 2021-05-25T12:00:16.440 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.399+0000 7f8d0420d700 1 librados: calling monclient init 2021-05-25T12:00:16.441 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.399+0000 7f8d0420d700 10 monclient: init 2021-05-25T12:00:16.441 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.400+0000 7f8d0420d700 10 monclient: _reopen_session rank -1 2021-05-25T12:00:16.441 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.400+0000 7f8d0420d700 10 monclient: _add_conns ranks=[2,0,1] 2021-05-25T12:00:16.441 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.400+0000 7f8d0420d700 10 monclient(hunting): picked mon.noname-c con 0x7f8cfc068f60 addr v1:172.21.15.102:6790/0 2021-05-25T12:00:16.441 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.400+0000 7f8d0420d700 10 monclient(hunting): picked mon.noname-a con 0x7f8cfc096800 addr v1:172.21.15.102:6789/0 2021-05-25T12:00:16.442 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.400+0000 7f8d0420d700 10 monclient(hunting): picked mon.noname-b con 0x7f8cfc131ae0 addr v1:172.21.15.181:6789/0 2021-05-25T12:00:16.442 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.400+0000 7f8d0420d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cfc096ca0 con 0x7f8cfc096800 2021-05-25T12:00:16.442 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.400+0000 7f8d0420d700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cfc127de0 con 0x7f8cfc131ae0 2021-05-25T12:00:16.442 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.400+0000 7f8d0420d700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cfc12c3a0 con 0x7f8cfc068f60 2021-05-25T12:00:16.443 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.400+0000 7f8d0420d700 10 monclient(hunting): _renew_subs 2021-05-25T12:00:16.443 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.400+0000 7f8d0420d700 10 monclient(hunting): authenticate will time out at 816.997341s 2021-05-25T12:00:16.443 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.400+0000 7f8d01fa9700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f8cfc068f60 0x7f8cfc130880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:54134/0 (socket says 172.21.15.102:54134) 2021-05-25T12:00:16.443 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.400+0000 7f8d01fa9700 1 -- 172.21.15.102:0/2902350138 learned_addr learned my addr 172.21.15.102:0/2902350138 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:00:16.444 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.400+0000 7f8d027aa700 1 -- 172.21.15.102:0/2902350138 >> v1:172.21.15.181:6789/0 conn(0x7f8cfc131ae0 legacy=0x7f8cfc143450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.181:6789/0 2021-05-25T12:00:16.444 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.400+0000 7f8ce9ffb700 10 client.?.objecter ms_handle_connect 0x7f8cfc096800 2021-05-25T12:00:16.444 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.400+0000 7f8ce9ffb700 10 client.?.objecter resend_mon_ops 2021-05-25T12:00:16.444 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.400+0000 7f8ce9ffb700 10 client.?.objecter ms_handle_connect 0x7f8cfc068f60 2021-05-25T12:00:16.445 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.400+0000 7f8ce9ffb700 10 client.?.objecter resend_mon_ops 2021-05-25T12:00:16.445 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.400+0000 7f8ce9ffb700 1 -- 172.21.15.102:0/2902350138 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1673835303 0 0) 0x7f8cfc12c3a0 con 0x7f8cfc068f60 2021-05-25T12:00:16.445 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.400+0000 7f8ce9ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:16.445 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.400+0000 7f8ce9ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:16.445 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.400+0000 7f8ce9ffb700 10 monclient(hunting): my global_id is 4421 2021-05-25T12:00:16.446 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.400+0000 7f8ce9ffb700 1 -- 172.21.15.102:0/2902350138 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cdc004c10 con 0x7f8cfc068f60 2021-05-25T12:00:16.446 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.400+0000 7f8ce9ffb700 1 -- 172.21.15.102:0/2902350138 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2498899971 0 0) 0x7f8cfc096ca0 con 0x7f8cfc096800 2021-05-25T12:00:16.446 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.400+0000 7f8ce9ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:16.446 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.400+0000 7f8ce9ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:16.447 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.400+0000 7f8ce9ffb700 10 monclient(hunting): my global_id is 4437 2021-05-25T12:00:16.447 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.400+0000 7f8ce9ffb700 1 -- 172.21.15.102:0/2902350138 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cfc12c3a0 con 0x7f8cfc096800 2021-05-25T12:00:16.447 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.400+0000 7f8ce9ffb700 1 -- 172.21.15.102:0/2902350138 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1113419459 0 0) 0x7f8cdc004c10 con 0x7f8cfc068f60 2021-05-25T12:00:16.447 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.400+0000 7f8ce9ffb700 1 -- 172.21.15.102:0/2902350138 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8cfc096ca0 con 0x7f8cfc068f60 2021-05-25T12:00:16.448 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.400+0000 7f8ce9ffb700 1 -- 172.21.15.102:0/2902350138 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f8cec003f20 con 0x7f8cfc068f60 2021-05-25T12:00:16.448 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.400+0000 7f8ce9ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:16.448 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.400+0000 7f8ce9ffb700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:00:16.448 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.400+0000 7f8ce9ffb700 10 monclient(hunting): dump: 2021-05-25T12:00:16.449 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:16.449 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:16.449 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:16.449 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:16.449 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:16.450 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:16.450 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:16.450 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:16.451 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:16.451 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:16.452 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.400+0000 7f8ce9ffb700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:00:16.452 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.400+0000 7f8ce9ffb700 1 -- 172.21.15.102:0/2902350138 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2299234780 0 0) 0x7f8cfc12c3a0 con 0x7f8cfc096800 2021-05-25T12:00:16.453 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.400+0000 7f8ce9ffb700 1 -- 172.21.15.102:0/2902350138 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8cdc004c10 con 0x7f8cfc096800 2021-05-25T12:00:16.453 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.400+0000 7f8ce9ffb700 1 -- 172.21.15.102:0/2902350138 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f8cf0001a70 con 0x7f8cfc096800 2021-05-25T12:00:16.453 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.400+0000 7f8ce9ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:16.454 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.400+0000 7f8ce9ffb700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2021-05-25T12:00:16.454 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.400+0000 7f8ce9ffb700 10 monclient(hunting): dump: 2021-05-25T12:00:16.455 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:16.455 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:16.456 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:16.456 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:16.457 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:16.457 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:16.457 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:16.458 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:16.458 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:16.459 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:16.459 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.400+0000 7f8ce9ffb700 1 -- 172.21.15.102:0/2902350138 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3433897905 0 0) 0x7f8cfc096ca0 con 0x7f8cfc068f60 2021-05-25T12:00:16.460 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.400+0000 7f8ce9ffb700 1 -- 172.21.15.102:0/2902350138 >> v1:172.21.15.181:6789/0 conn(0x7f8cfc131ae0 legacy=0x7f8cfc143450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-05-25T12:00:16.460 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.401+0000 7f8ce9ffb700 1 -- 172.21.15.102:0/2902350138 >> v1:172.21.15.102:6789/0 conn(0x7f8cfc096800 legacy=0x7f8cfc13fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:16.461 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.401+0000 7f8ce9ffb700 10 monclient: _finish_hunting 0 2021-05-25T12:00:16.461 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.401+0000 7f8ce9ffb700 1 monclient: found mon.c 2021-05-25T12:00:16.461 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.401+0000 7f8ce9ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:00:16.462 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.401+0000 7f8ce9ffb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:00:16.462 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.401+0000 7f8ce9ffb700 1 -- 172.21.15.102:0/2902350138 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cfc068ad0 con 0x7f8cfc068f60 2021-05-25T12:00:16.463 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.401+0000 7f8ce9ffb700 10 monclient: _finish_auth 0 2021-05-25T12:00:16.463 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.401+0000 7f8ce9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:00:16.464 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.401+0000 7f8d0420d700 5 monclient: authenticate success, global_id 4421 2021-05-25T12:00:16.464 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.401+0000 7f8d0420d700 10 monclient: _renew_subs 2021-05-25T12:00:16.465 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.401+0000 7f8d0420d700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:00:16.465 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.401+0000 7f8d0420d700 1 -- 172.21.15.102:0/2902350138 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cfc12ad10 con 0x7f8cfc068f60 2021-05-25T12:00:16.466 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.401+0000 7f8ce9ffb700 1 -- 172.21.15.102:0/2902350138 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8cec004140 con 0x7f8cfc068f60 2021-05-25T12:00:16.466 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.401+0000 7f8ce9ffb700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:00:16.467 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.401+0000 7f8ce9ffb700 1 -- 172.21.15.102:0/2902350138 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f8cec004450 con 0x7f8cfc068f60 2021-05-25T12:00:16.467 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.401+0000 7f8ce9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:16.468 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.401+0000 7f8ce9ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:00:16.468 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.401+0000 7f8ce9ffb700 10 monclient: dump: 2021-05-25T12:00:16.469 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:16.469 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:16.470 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:16.470 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:16.470 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:16.471 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:16.471 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:16.472 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:16.472 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:16.473 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:16.473 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.401+0000 7f8cebfff700 4 set_mon_vals no callback set 2021-05-25T12:00:16.473 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.401+0000 7f8d0420d700 10 client.4421.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:00:16.474 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.401+0000 7f8d0420d700 10 monclient: _renew_subs 2021-05-25T12:00:16.474 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.401+0000 7f8d0420d700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:00:16.474 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.401+0000 7f8d0420d700 1 -- 172.21.15.102:0/2902350138 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cfc131150 con 0x7f8cfc068f60 2021-05-25T12:00:16.474 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.401+0000 7f8d0420d700 1 librados: init done 2021-05-25T12:00:16.475 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.401+0000 7f8ce9ffb700 1 -- 172.21.15.102:0/2902350138 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f8cec004630 con 0x7f8cfc068f60 2021-05-25T12:00:16.475 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.401+0000 7f8d0420d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:00:16.475 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.401+0000 7f8d0420d700 1 -- 172.21.15.102:0/2902350138 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f8cfc12b840 con 0x7f8cfc067f70 2021-05-25T12:00:16.475 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.402+0000 7f8ce9ffb700 1 -- 172.21.15.102:0/2902350138 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(15..15 src has 1..15) v4 ==== 3549+0+0 (unknown 3948819498 0 0) 0x7f8cec03cda0 con 0x7f8cfc068f60 2021-05-25T12:00:16.476 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.402+0000 7f8ce9ffb700 10 client.4421.objecter ms_dispatch 0x7f8cfc001550 osd_map(15..15 src has 1..15) v4 2021-05-25T12:00:16.476 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.402+0000 7f8ce9ffb700 3 client.4421.objecter handle_osd_map got epochs [15,15] > 0 2021-05-25T12:00:16.476 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.402+0000 7f8ce9ffb700 3 client.4421.objecter handle_osd_map decoding full epoch 15 2021-05-25T12:00:16.476 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.402+0000 7f8ce9ffb700 20 client.4421.objecter dump_active .. 0 homeless 2021-05-25T12:00:16.477 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.402+0000 7f8ce9ffb700 10 client.4421.objecter ms_handle_connect 0x7f8cfc067f70 2021-05-25T12:00:16.477 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.402+0000 7f8ce9ffb700 10 client.4421.objecter resend_mon_ops 2021-05-25T12:00:16.477 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.402+0000 7f8ce9ffb700 1 -- 172.21.15.102:0/2902350138 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3425746035 0 0) 0x7f8cfc12b840 con 0x7f8cfc067f70 2021-05-25T12:00:16.477 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.402+0000 7f8ce9ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:16.478 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.402+0000 7f8ce9ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:16.478 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.402+0000 7f8ce9ffb700 10 monclient(hunting): my global_id is 4440 2021-05-25T12:00:16.478 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.402+0000 7f8ce9ffb700 1 -- 172.21.15.102:0/2902350138 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cfc096ca0 con 0x7f8cfc067f70 2021-05-25T12:00:16.478 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.402+0000 7f8d027aa700 10 monclient: get_auth_request con 0x7f8cdc0332a0 auth_method 0 2021-05-25T12:00:16.479 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.402+0000 7f8ce9ffb700 1 -- 172.21.15.102:0/2902350138 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3574042298 0 0) 0x7f8cfc096ca0 con 0x7f8cfc067f70 2021-05-25T12:00:16.479 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.402+0000 7f8ce9ffb700 1 -- 172.21.15.102:0/2902350138 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8cfc12b840 con 0x7f8cfc067f70 2021-05-25T12:00:16.479 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.402+0000 7f8ce9ffb700 1 -- 172.21.15.102:0/2902350138 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f8cf00046d0 con 0x7f8cfc067f70 2021-05-25T12:00:16.479 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.402+0000 7f8ce9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:16.480 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.402+0000 7f8ce9ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:00:16.480 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.402+0000 7f8ce9ffb700 10 monclient: dump: 2021-05-25T12:00:16.480 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:16.480 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:16.481 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:16.481 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:16.481 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:16.481 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:16.481 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:16.482 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:16.482 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:16.482 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:16.482 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.402+0000 7f8ce9ffb700 1 -- 172.21.15.102:0/2902350138 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3048643492 0 0) 0x7f8cfc12b840 con 0x7f8cfc067f70 2021-05-25T12:00:16.483 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.402+0000 7f8ce9ffb700 1 -- 172.21.15.102:0/2902350138 --> v1:172.21.15.102:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8cfc068350 con 0x7f8cfc067f70 2021-05-25T12:00:16.483 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.402+0000 7f8ce9ffb700 10 client.4421.objecter ms_handle_connect 0x7f8cdc0332a0 2021-05-25T12:00:16.483 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.402+0000 7f8ce9ffb700 1 -- 172.21.15.102:0/2902350138 <== mon.0 v1:172.21.15.102:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7f8cf00069a0 con 0x7f8cfc067f70 2021-05-25T12:00:16.483 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.402+0000 7f8ce9ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:00:16.484 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.402+0000 7f8ce9ffb700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:00:16.484 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.402+0000 7f8ce9ffb700 1 -- 172.21.15.102:0/2902350138 >> v1:172.21.15.102:6789/0 conn(0x7f8cfc067f70 legacy=0x7f8cfc126350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:16.484 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.402+0000 7f8ce9ffb700 1 -- 172.21.15.102:0/2902350138 >> v1:172.21.15.102:6789/0 conn(0x7f8cfc067f70 legacy=0x7f8cfc126350 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:00:16.484 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.409+0000 7f8d0420d700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:00:16.484 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.409+0000 7f8d0420d700 1 -- 172.21.15.102:0/2902350138 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f8cfc059730 con 0x7f8cfc027ce0 2021-05-25T12:00:16.485 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.409+0000 7f8ce9ffb700 10 client.4421.objecter ms_handle_connect 0x7f8cfc027ce0 2021-05-25T12:00:16.485 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.409+0000 7f8ce9ffb700 10 client.4421.objecter resend_mon_ops 2021-05-25T12:00:16.485 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.409+0000 7f8ce9ffb700 1 -- 172.21.15.102:0/2902350138 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3442377109 0 0) 0x7f8cfc059730 con 0x7f8cfc027ce0 2021-05-25T12:00:16.485 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.409+0000 7f8ce9ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:16.486 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.409+0000 7f8ce9ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:16.486 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.409+0000 7f8ce9ffb700 10 monclient(hunting): my global_id is 4443 2021-05-25T12:00:16.486 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.409+0000 7f8ce9ffb700 1 -- 172.21.15.102:0/2902350138 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cfc12b840 con 0x7f8cfc027ce0 2021-05-25T12:00:16.486 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.409+0000 7f8ce9ffb700 1 -- 172.21.15.102:0/2902350138 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 189814813 0 0) 0x7f8cfc12b840 con 0x7f8cfc027ce0 2021-05-25T12:00:16.487 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.409+0000 7f8ce9ffb700 1 -- 172.21.15.102:0/2902350138 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8cfc059730 con 0x7f8cfc027ce0 2021-05-25T12:00:16.487 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.409+0000 7f8ce9ffb700 1 -- 172.21.15.102:0/2902350138 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f8cf0004dc0 con 0x7f8cfc027ce0 2021-05-25T12:00:16.487 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.409+0000 7f8ce9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:16.487 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.409+0000 7f8ce9ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:00:16.487 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.409+0000 7f8ce9ffb700 10 monclient: dump: 2021-05-25T12:00:16.488 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:16.488 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:16.488 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:16.488 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:16.489 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:16.489 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:16.489 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:16.489 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:16.490 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:16.490 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:16.490 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.410+0000 7f8ce9ffb700 1 -- 172.21.15.102:0/2902350138 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1151068511 0 0) 0x7f8cfc059730 con 0x7f8cfc027ce0 2021-05-25T12:00:16.490 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.410+0000 7f8ce9ffb700 1 -- 172.21.15.102:0/2902350138 --> v1:172.21.15.102:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f8cfc05d790 con 0x7f8cfc027ce0 2021-05-25T12:00:16.490 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.410+0000 7f8ce9ffb700 1 -- 172.21.15.102:0/2902350138 <== mon.0 v1:172.21.15.102:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3931 (unknown 0 0 3970686215) 0x7f8cf0006080 con 0x7f8cfc027ce0 2021-05-25T12:00:16.491 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.410+0000 7f8ce9ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:00:16.491 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.410+0000 7f8ce9ffb700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:00:16.491 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.410+0000 7f8ce9ffb700 1 -- 172.21.15.102:0/2902350138 >> v1:172.21.15.102:6789/0 conn(0x7f8cfc027ce0 legacy=0x7f8cfc126a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:16.491 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.410+0000 7f8ce9ffb700 1 -- 172.21.15.102:0/2902350138 >> v1:172.21.15.102:6789/0 conn(0x7f8cfc027ce0 legacy=0x7f8cfc126a60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:00:16.492 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:00:16.492 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:00:16.492 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:00:16.493 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "leader", 2021-05-25T12:00:16.493 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 12, 2021-05-25T12:00:16.493 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:00:16.493 INFO:teuthology.orchestra.run.smithi102.stdout: 0, 2021-05-25T12:00:16.494 INFO:teuthology.orchestra.run.smithi102.stdout: 2 2021-05-25T12:00:16.494 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:00:16.494 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 0, 2021-05-25T12:00:16.494 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:00:16.495 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:00:16.495 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:00:16.495 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:00:16.495 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:00:16.495 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:00:16.496 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:00:16.496 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:00:16.496 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:00:16.496 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:00:16.497 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:00:16.497 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:00:16.497 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:00:16.497 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:00:16.498 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:00:16.498 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:00:16.498 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:00:16.498 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:00:16.499 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:00:16.499 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:00:16.499 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:00:16.499 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:00:16.499 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:00:16.500 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:00:16.500 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:00:16.500 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:16.500 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:00:16.501 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:00:16.501 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:00:16.501 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:00:16.501 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:00:16.502 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:00:16.502 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:00:16.502 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:00:16.502 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:00:16.503 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:00:16.503 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:00:16.503 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:00:16.503 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:00:16.503 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:00:16.504 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:00:16.504 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:00:16.504 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:00:16.504 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:00:16.505 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:00:16.505 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:00:16.505 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:00:16.505 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:00:16.505 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:00:16.506 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:00:16.506 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:00:16.506 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:00:16.506 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:16.507 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:00:16.507 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:16.507 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:00:16.507 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:00:16.507 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:00:16.508 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:00:16.508 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:16.508 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:00:16.508 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:00:16.509 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:00:16.509 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:16.509 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:00:16.509 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:16.509 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:00:16.510 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:00:16.510 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:00:16.510 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:00:16.510 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:00:16.511 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:16.511 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:16.511 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:00:16.511 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:00:16.511 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:00:16.512 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:00:16.512 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:16.512 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:00:16.513 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:00:16.513 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:00:16.513 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:16.514 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:00:16.514 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:16.515 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:00:16.515 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:00:16.515 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:00:16.515 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:00:16.515 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:00:16.516 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:16.516 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:16.516 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:00:16.516 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:00:16.517 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:00:16.517 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:00:16.517 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:16.517 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:00:16.517 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:00:16.518 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:00:16.518 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:16.518 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:00:16.518 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:16.519 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:00:16.519 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:00:16.519 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:00:16.519 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:00:16.519 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:00:16.520 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:16.520 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:00:16.520 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:16.520 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:00:16.521 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:00:16.521 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:16.521 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:00:16.521 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:00:16.521 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:00:16.522 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:16.522 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:00:16.522 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": [ 2021-05-25T12:00:16.522 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:16.523 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:00:16.523 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:00:16.523 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:00:16.523 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:16.524 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:00:16.524 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:00:16.524 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:16.524 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:00:16.524 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:00:16.525 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 4 2021-05-25T12:00:16.525 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:16.525 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:00:16.525 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:16.526 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:00:16.526 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:00:16.528 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.410+0000 7f8d0420d700 10 librados: watch_flush enter 2021-05-25T12:00:16.528 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.410+0000 7f8d0420d700 10 librados: watch_flush exit 2021-05-25T12:00:16.529 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.410+0000 7f8d0420d700 20 client.4421.objecter shutdown clearing up homeless session... 2021-05-25T12:00:16.529 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.410+0000 7f8d0420d700 10 client.4421.objecter successfully canceled tick 2021-05-25T12:00:16.529 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.410+0000 7f8d0420d700 1 -- 172.21.15.102:0/2902350138 >> v1:172.21.15.181:6812/29582 conn(0x7f8cdc0332a0 legacy=0x7f8cdc035760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:16.529 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.410+0000 7f8d0420d700 10 monclient: shutdown 2021-05-25T12:00:16.530 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.410+0000 7f8d0420d700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:00:16.530 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.410+0000 7f8d0420d700 1 -- 172.21.15.102:0/2902350138 >> v1:172.21.15.102:6790/0 conn(0x7f8cfc068f60 legacy=0x7f8cfc130880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:16.530 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.410+0000 7f8d027aa700 1 -- 172.21.15.102:0/2902350138 reap_dead start 2021-05-25T12:00:16.530 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.410+0000 7f8d027aa700 1 -- 172.21.15.102:0/2902350138 reap_dead start 2021-05-25T12:00:16.530 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.410+0000 7f8d0420d700 1 -- 172.21.15.102:0/2902350138 shutdown_connections 2021-05-25T12:00:16.531 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.410+0000 7f8d0420d700 1 -- 172.21.15.102:0/2902350138 >> 172.21.15.102:0/2902350138 conn(0x7f8cfc09b920 msgr2=0x7f8cfc133ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:00:16.531 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.410+0000 7f8d0420d700 1 -- 172.21.15.102:0/2902350138 shutdown_connections 2021-05-25T12:00:16.531 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.410+0000 7f8d0420d700 1 -- 172.21.15.102:0/2902350138 wait complete. 2021-05-25T12:00:16.531 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.410+0000 7f8d0420d700 1 librados: shutdown 2021-05-25T12:00:16.532 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2021-05-25T12:00:16.642 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.640+0000 7f83dcb65700 10 monclient: get_monmap_and_config 2021-05-25T12:00:16.643 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.640+0000 7f83dcb65700 10 monclient: build_initial_monmap 2021-05-25T12:00:16.643 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.641+0000 7f83dcb65700 10 monclient: monmap: 2021-05-25T12:00:16.643 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:00:16.644 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:16.644 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:00:16.641883+0000 2021-05-25T12:00:16.644 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:00:16.641883+0000 2021-05-25T12:00:16.644 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:00:16.644 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:16.645 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:00:16.645 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:00:16.645 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:00:16.645 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:16.646 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.641+0000 7f83dcb65700 1 Processor -- start 2021-05-25T12:00:16.646 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.641+0000 7f83dcb65700 1 -- start start 2021-05-25T12:00:16.646 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.641+0000 7f83dcb65700 10 monclient: init 2021-05-25T12:00:16.661 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.642+0000 7f83dcb65700 10 monclient: _reopen_session rank -1 2021-05-25T12:00:16.661 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.642+0000 7f83dcb65700 10 monclient: _add_conns ranks=[2,1,0] 2021-05-25T12:00:16.661 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.642+0000 7f83dcb65700 10 monclient(hunting): picked mon.noname-b con 0x7f83d80699f0 addr v1:172.21.15.102:6790/0 2021-05-25T12:00:16.662 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.642+0000 7f83dcb65700 10 monclient(hunting): picked mon.noname-c con 0x7f83d8096790 addr v1:172.21.15.181:6789/0 2021-05-25T12:00:16.662 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.642+0000 7f83dcb65700 10 monclient(hunting): picked mon.noname-a con 0x7f83d8130520 addr v1:172.21.15.102:6789/0 2021-05-25T12:00:16.662 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.642+0000 7f83dcb65700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83d8094690 con 0x7f83d8130520 2021-05-25T12:00:16.662 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.642+0000 7f83dcb65700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83d8096c40 con 0x7f83d8096790 2021-05-25T12:00:16.663 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.642+0000 7f83dcb65700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83d8097180 con 0x7f83d80699f0 2021-05-25T12:00:16.663 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.642+0000 7f83dcb65700 10 monclient(hunting): _renew_subs 2021-05-25T12:00:16.663 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.642+0000 7f83dcb65700 10 monclient(hunting): authenticate will time out at 817.239195s 2021-05-25T12:00:16.664 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.642+0000 7f83d5d9b700 1 -- >> v1:172.21.15.181:6789/0 conn(0x7f83d8096790 legacy=0x7f83d8098290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2021-05-25T12:00:16.664 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.642+0000 7f83d6d9d700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f83d8130520 0x7f83d8132910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:59270/0 (socket says 172.21.15.102:59270) 2021-05-25T12:00:16.664 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.642+0000 7f83d6d9d700 1 -- 172.21.15.102:0/3697921716 learned_addr learned my addr 172.21.15.102:0/3697921716 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:00:16.664 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.642+0000 7f83d659c700 1 --1- 172.21.15.102:0/3697921716 >> v1:172.21.15.102:6790/0 conn(0x7f83d80699f0 0x7f83d8097b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:54146/0 (socket says 172.21.15.102:54146) 2021-05-25T12:00:16.665 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.642+0000 7f83d559a700 1 -- 172.21.15.102:0/3697921716 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 804464979 0 0) 0x7f83d8097180 con 0x7f83d80699f0 2021-05-25T12:00:16.665 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.642+0000 7f83d559a700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:16.665 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.642+0000 7f83d559a700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:16.665 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.642+0000 7f83d559a700 10 monclient(hunting): my global_id is 4424 2021-05-25T12:00:16.665 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.642+0000 7f83d559a700 1 -- 172.21.15.102:0/3697921716 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83bc003270 con 0x7f83d80699f0 2021-05-25T12:00:16.666 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.642+0000 7f83d559a700 1 -- 172.21.15.102:0/3697921716 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 675522203 0 0) 0x7f83d8094690 con 0x7f83d8130520 2021-05-25T12:00:16.666 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.642+0000 7f83d559a700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:16.666 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.642+0000 7f83d559a700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:16.666 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.642+0000 7f83d559a700 10 monclient(hunting): my global_id is 4446 2021-05-25T12:00:16.667 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.642+0000 7f83d559a700 1 -- 172.21.15.102:0/3697921716 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83d8097180 con 0x7f83d8130520 2021-05-25T12:00:16.667 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.642+0000 7f83d559a700 1 -- 172.21.15.102:0/3697921716 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 639092787 0 0) 0x7f83d8097180 con 0x7f83d8130520 2021-05-25T12:00:16.667 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.642+0000 7f83d559a700 1 -- 172.21.15.102:0/3697921716 >> v1:172.21.15.102:6790/0 conn(0x7f83d80699f0 legacy=0x7f83d8097b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:16.667 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.642+0000 7f83d559a700 1 -- 172.21.15.102:0/3697921716 >> v1:172.21.15.181:6789/0 conn(0x7f83d8096790 legacy=0x7f83d8098290 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-05-25T12:00:16.668 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.642+0000 7f83d559a700 10 monclient: _finish_hunting 0 2021-05-25T12:00:16.668 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.642+0000 7f83d559a700 1 monclient: found mon.noname-a 2021-05-25T12:00:16.668 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.642+0000 7f83d559a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:00:16.668 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.642+0000 7f83d559a700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T12:00:16.669 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.642+0000 7f83d559a700 1 -- 172.21.15.102:0/3697921716 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83d80989a0 con 0x7f83d8130520 2021-05-25T12:00:16.669 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.642+0000 7f83d559a700 10 monclient: _finish_auth 0 2021-05-25T12:00:16.669 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.642+0000 7f83d559a700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:00:16.669 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.642+0000 7f83d559a700 1 -- 172.21.15.102:0/3697921716 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f83c80018f0 con 0x7f83d8130520 2021-05-25T12:00:16.670 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.642+0000 7f83dcb65700 5 monclient: authenticate success, global_id 4446 2021-05-25T12:00:16.670 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.642+0000 7f83d559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:16.670 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.642+0000 7f83d559a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:00:16.671 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.642+0000 7f83d559a700 10 monclient: dump: 2021-05-25T12:00:16.671 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:16.671 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:16.671 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:16.672 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:16.672 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:16.672 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:16.672 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:16.673 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:16.673 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:16.673 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:16.673 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.642+0000 7f83dcb65700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:00:16.674 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.642+0000 7f83d559a700 1 -- 172.21.15.102:0/3697921716 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f83c8001b90 con 0x7f83d8130520 2021-05-25T12:00:16.674 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.642+0000 7f83d559a700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:00:16.674 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.642+0000 7f83dcb65700 10 monclient: get_monmap_and_config success 2021-05-25T12:00:16.674 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.642+0000 7f83d559a700 1 -- 172.21.15.102:0/3697921716 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f83c8001f20 con 0x7f83d8130520 2021-05-25T12:00:16.675 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.642+0000 7f83dcb65700 4 set_mon_vals no callback set 2021-05-25T12:00:16.675 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.642+0000 7f83d559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:16.675 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.642+0000 7f83d559a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:00:16.676 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.642+0000 7f83d559a700 10 monclient: dump: 2021-05-25T12:00:16.676 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:16.676 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:16.676 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:16.677 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:16.677 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:16.677 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:16.677 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:16.677 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:16.678 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:16.678 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:16.678 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.643+0000 7f83dcb65700 10 monclient: shutdown 2021-05-25T12:00:16.678 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.643+0000 7f83dcb65700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:00:16.679 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.643+0000 7f83dcb65700 1 -- 172.21.15.102:0/3697921716 >> v1:172.21.15.102:6789/0 conn(0x7f83d8130520 legacy=0x7f83d8132910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:16.679 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.643+0000 7f83dcb65700 1 -- 172.21.15.102:0/3697921716 shutdown_connections 2021-05-25T12:00:16.679 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.643+0000 7f83dcb65700 1 -- 172.21.15.102:0/3697921716 >> 172.21.15.102:0/3697921716 conn(0x7f83d809cd10 msgr2=0x7f83d809d120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:00:16.679 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.643+0000 7f83dcb65700 1 -- 172.21.15.102:0/3697921716 shutdown_connections 2021-05-25T12:00:16.680 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.643+0000 7f83dcb65700 1 -- 172.21.15.102:0/3697921716 wait complete. 2021-05-25T12:00:16.680 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.643+0000 7f83dcb65700 10 monclient: build_initial_monmap 2021-05-25T12:00:16.680 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.643+0000 7f83dcb65700 10 monclient: monmap: 2021-05-25T12:00:16.681 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:00:16.681 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:00:16.682 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:00:16.682 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:00:16.683 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:00:16.683 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:00:16.684 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:00:16.684 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:00:16.684 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:00:16.685 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:16.685 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.644+0000 7f83dcb65700 1 librados: starting msgr at 2021-05-25T12:00:16.686 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.644+0000 7f83dcb65700 1 librados: starting objecter 2021-05-25T12:00:16.686 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.644+0000 7f83dcb65700 1 Processor -- start 2021-05-25T12:00:16.686 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.644+0000 7f83dcb65700 1 -- start start 2021-05-25T12:00:16.687 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.644+0000 7f83dcb65700 1 librados: setting wanted keys 2021-05-25T12:00:16.687 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.644+0000 7f83dcb65700 1 librados: calling monclient init 2021-05-25T12:00:16.688 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.644+0000 7f83dcb65700 10 monclient: init 2021-05-25T12:00:16.688 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.644+0000 7f83dcb65700 10 monclient: _reopen_session rank -1 2021-05-25T12:00:16.689 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.644+0000 7f83dcb65700 10 monclient: _add_conns ranks=[0,1,2] 2021-05-25T12:00:16.689 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.644+0000 7f83dcb65700 10 monclient(hunting): picked mon.noname-a con 0x7f83d80699f0 addr v1:172.21.15.102:6789/0 2021-05-25T12:00:16.689 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.644+0000 7f83dcb65700 10 monclient(hunting): picked mon.noname-b con 0x7f83d8096790 addr v1:172.21.15.181:6789/0 2021-05-25T12:00:16.690 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.644+0000 7f83dcb65700 10 monclient(hunting): picked mon.noname-c con 0x7f83d8130520 addr v1:172.21.15.102:6790/0 2021-05-25T12:00:16.690 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.644+0000 7f83dcb65700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83d812ffe0 con 0x7f83d80699f0 2021-05-25T12:00:16.691 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.644+0000 7f83dcb65700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83d8115490 con 0x7f83d8096790 2021-05-25T12:00:16.691 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.644+0000 7f83dcb65700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83d8115c90 con 0x7f83d8130520 2021-05-25T12:00:16.692 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.644+0000 7f83dcb65700 10 monclient(hunting): _renew_subs 2021-05-25T12:00:16.692 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.644+0000 7f83dcb65700 10 monclient(hunting): authenticate will time out at 817.241687s 2021-05-25T12:00:16.692 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.644+0000 7f83d659c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f83d80699f0 0x7f83d81390f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:59274/0 (socket says 172.21.15.102:59274) 2021-05-25T12:00:16.692 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.644+0000 7f83d659c700 1 -- 172.21.15.102:0/2965573786 learned_addr learned my addr 172.21.15.102:0/2965573786 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:00:16.693 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.644+0000 7f83d5d9b700 1 -- 172.21.15.102:0/2965573786 >> v1:172.21.15.181:6789/0 conn(0x7f83d8096790 legacy=0x7f83d813dcf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.181:6789/0 2021-05-25T12:00:16.693 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.644+0000 7f83c67fc700 10 client.?.objecter ms_handle_connect 0x7f83d80699f0 2021-05-25T12:00:16.693 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.644+0000 7f83c67fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:00:16.693 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.644+0000 7f83c67fc700 10 client.?.objecter ms_handle_connect 0x7f83d8130520 2021-05-25T12:00:16.694 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.644+0000 7f83c67fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:00:16.694 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.644+0000 7f83c67fc700 1 -- 172.21.15.102:0/2965573786 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1927409896 0 0) 0x7f83d812ffe0 con 0x7f83d80699f0 2021-05-25T12:00:16.694 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.644+0000 7f83c67fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:16.694 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.644+0000 7f83c67fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:16.695 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.644+0000 7f83c67fc700 10 monclient(hunting): my global_id is 4449 2021-05-25T12:00:16.695 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.644+0000 7f83c67fc700 1 -- 172.21.15.102:0/2965573786 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83b4004c10 con 0x7f83d80699f0 2021-05-25T12:00:16.695 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.644+0000 7f83c67fc700 1 -- 172.21.15.102:0/2965573786 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1724964344 0 0) 0x7f83d8115c90 con 0x7f83d8130520 2021-05-25T12:00:16.695 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.644+0000 7f83c67fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:16.696 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.644+0000 7f83c67fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:16.696 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.644+0000 7f83c67fc700 10 monclient(hunting): my global_id is 4427 2021-05-25T12:00:16.696 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.644+0000 7f83c67fc700 1 -- 172.21.15.102:0/2965573786 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83d812ffe0 con 0x7f83d8130520 2021-05-25T12:00:16.696 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.645+0000 7f83c67fc700 1 -- 172.21.15.102:0/2965573786 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2415768215 0 0) 0x7f83b4004c10 con 0x7f83d80699f0 2021-05-25T12:00:16.697 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.645+0000 7f83c67fc700 1 -- 172.21.15.102:0/2965573786 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f83d8115c90 con 0x7f83d80699f0 2021-05-25T12:00:16.697 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.645+0000 7f83c67fc700 1 -- 172.21.15.102:0/2965573786 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f83c0001a70 con 0x7f83d80699f0 2021-05-25T12:00:16.697 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.645+0000 7f83c67fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:16.697 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.645+0000 7f83c67fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:00:16.698 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.645+0000 7f83c67fc700 10 monclient(hunting): dump: 2021-05-25T12:00:16.698 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:16.698 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:16.698 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:16.698 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:16.699 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:16.699 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:16.699 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:16.699 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:16.700 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:16.700 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:16.700 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.645+0000 7f83c67fc700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:00:16.700 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.645+0000 7f83c67fc700 1 -- 172.21.15.102:0/2965573786 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3065283016 0 0) 0x7f83d812ffe0 con 0x7f83d8130520 2021-05-25T12:00:16.700 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.645+0000 7f83c67fc700 1 -- 172.21.15.102:0/2965573786 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f83b4004c10 con 0x7f83d8130520 2021-05-25T12:00:16.701 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.645+0000 7f83c67fc700 1 -- 172.21.15.102:0/2965573786 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f83c8001ea0 con 0x7f83d8130520 2021-05-25T12:00:16.701 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.645+0000 7f83c67fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:16.701 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.645+0000 7f83c67fc700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2021-05-25T12:00:16.701 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.645+0000 7f83c67fc700 10 monclient(hunting): dump: 2021-05-25T12:00:16.702 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:16.702 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:16.702 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:16.702 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:16.703 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:16.703 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:16.703 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:16.703 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:16.703 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:16.704 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:16.704 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.645+0000 7f83c67fc700 1 -- 172.21.15.102:0/2965573786 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2383787955 0 0) 0x7f83b4004c10 con 0x7f83d8130520 2021-05-25T12:00:16.704 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.645+0000 7f83c67fc700 1 -- 172.21.15.102:0/2965573786 >> v1:172.21.15.181:6789/0 conn(0x7f83d8096790 legacy=0x7f83d813dcf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-05-25T12:00:16.704 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.645+0000 7f83c67fc700 1 -- 172.21.15.102:0/2965573786 >> v1:172.21.15.102:6789/0 conn(0x7f83d80699f0 legacy=0x7f83d81390f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:16.705 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.645+0000 7f83c67fc700 10 monclient: _finish_hunting 0 2021-05-25T12:00:16.705 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.645+0000 7f83c67fc700 1 monclient: found mon.c 2021-05-25T12:00:16.705 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.645+0000 7f83c67fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:00:16.705 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.645+0000 7f83c67fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:00:16.706 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.645+0000 7f83c67fc700 1 -- 172.21.15.102:0/2965573786 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83d8139800 con 0x7f83d8130520 2021-05-25T12:00:16.706 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.645+0000 7f83c67fc700 10 monclient: _finish_auth 0 2021-05-25T12:00:16.706 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.645+0000 7f83c67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:00:16.706 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.645+0000 7f83dcb65700 5 monclient: authenticate success, global_id 4427 2021-05-25T12:00:16.707 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.645+0000 7f83dcb65700 10 monclient: _renew_subs 2021-05-25T12:00:16.707 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.645+0000 7f83dcb65700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:00:16.707 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.645+0000 7f83dcb65700 1 -- 172.21.15.102:0/2965573786 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83d8094cc0 con 0x7f83d8130520 2021-05-25T12:00:16.707 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.645+0000 7f83dcb65700 10 client.4427.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:00:16.707 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.645+0000 7f83dcb65700 10 monclient: _renew_subs 2021-05-25T12:00:16.708 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.645+0000 7f83dcb65700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:00:16.708 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.645+0000 7f83dcb65700 1 -- 172.21.15.102:0/2965573786 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f83d812bb50 con 0x7f83d8130520 2021-05-25T12:00:16.708 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.645+0000 7f83dcb65700 1 librados: init done 2021-05-25T12:00:16.708 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.645+0000 7f83c67fc700 1 -- 172.21.15.102:0/2965573786 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f83c8001460 con 0x7f83d8130520 2021-05-25T12:00:16.709 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.645+0000 7f83c67fc700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:00:16.709 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.645+0000 7f83c67fc700 1 -- 172.21.15.102:0/2965573786 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f83c8004d00 con 0x7f83d8130520 2021-05-25T12:00:16.709 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.645+0000 7f83c67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:16.709 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.645+0000 7f83d4d99700 4 set_mon_vals no callback set 2021-05-25T12:00:16.710 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.645+0000 7f83c67fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:00:16.710 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.645+0000 7f83c67fc700 10 monclient: dump: 2021-05-25T12:00:16.710 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:16.710 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:16.711 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:16.711 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:16.711 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:16.711 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:16.712 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:16.712 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:16.712 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:16.712 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:16.712 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.646+0000 7f83dcb65700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:00:16.713 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.646+0000 7f83dcb65700 1 -- 172.21.15.102:0/2965573786 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f83b80065a0 con 0x7f83b8002840 2021-05-25T12:00:16.713 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.646+0000 7f83c67fc700 1 -- 172.21.15.102:0/2965573786 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f83c800e020 con 0x7f83d8130520 2021-05-25T12:00:16.713 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.646+0000 7f83c67fc700 10 client.4427.objecter ms_handle_connect 0x7f83b8002840 2021-05-25T12:00:16.713 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.646+0000 7f83c67fc700 10 client.4427.objecter resend_mon_ops 2021-05-25T12:00:16.714 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.646+0000 7f83c67fc700 1 -- 172.21.15.102:0/2965573786 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(15..15 src has 1..15) v4 ==== 3549+0+0 (unknown 3948819498 0 0) 0x7f83c803d160 con 0x7f83d8130520 2021-05-25T12:00:16.714 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.646+0000 7f83c67fc700 10 client.4427.objecter ms_dispatch 0x7f83d8001550 osd_map(15..15 src has 1..15) v4 2021-05-25T12:00:16.714 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.646+0000 7f83c67fc700 3 client.4427.objecter handle_osd_map got epochs [15,15] > 0 2021-05-25T12:00:16.714 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.646+0000 7f83c67fc700 3 client.4427.objecter handle_osd_map decoding full epoch 15 2021-05-25T12:00:16.714 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.646+0000 7f83c67fc700 20 client.4427.objecter dump_active .. 0 homeless 2021-05-25T12:00:16.715 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.646+0000 7f83c67fc700 1 -- 172.21.15.102:0/2965573786 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 551673313 0 0) 0x7f83b80065a0 con 0x7f83b8002840 2021-05-25T12:00:16.715 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.646+0000 7f83c67fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:16.715 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.646+0000 7f83c67fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:16.715 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.646+0000 7f83c67fc700 10 monclient(hunting): my global_id is 4430 2021-05-25T12:00:16.716 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.646+0000 7f83c67fc700 1 -- 172.21.15.102:0/2965573786 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83b4004c10 con 0x7f83b8002840 2021-05-25T12:00:16.716 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.646+0000 7f83d5d9b700 10 monclient: get_auth_request con 0x7f83b4033260 auth_method 0 2021-05-25T12:00:16.716 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.646+0000 7f83c67fc700 1 -- 172.21.15.102:0/2965573786 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2550349232 0 0) 0x7f83b4004c10 con 0x7f83b8002840 2021-05-25T12:00:16.716 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.646+0000 7f83c67fc700 1 -- 172.21.15.102:0/2965573786 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f83b80065a0 con 0x7f83b8002840 2021-05-25T12:00:16.717 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.646+0000 7f83c67fc700 1 -- 172.21.15.102:0/2965573786 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f83c0001d90 con 0x7f83b8002840 2021-05-25T12:00:16.717 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.646+0000 7f83c67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:16.717 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.646+0000 7f83c67fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:00:16.717 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.646+0000 7f83c67fc700 10 monclient: dump: 2021-05-25T12:00:16.718 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:16.718 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:16.718 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:16.718 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:16.719 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:16.719 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:16.719 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:16.719 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:16.719 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:16.720 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:16.720 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.646+0000 7f83c67fc700 1 -- 172.21.15.102:0/2965573786 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1156664393 0 0) 0x7f83b80065a0 con 0x7f83b8002840 2021-05-25T12:00:16.720 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.646+0000 7f83c67fc700 1 -- 172.21.15.102:0/2965573786 --> v1:172.21.15.102:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f83b8009e40 con 0x7f83b8002840 2021-05-25T12:00:16.721 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.646+0000 7f83c67fc700 10 client.4427.objecter ms_handle_connect 0x7f83b4033260 2021-05-25T12:00:16.721 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.647+0000 7f83c67fc700 1 -- 172.21.15.102:0/2965573786 <== mon.2 v1:172.21.15.102:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7f83c0004df0 con 0x7f83b8002840 2021-05-25T12:00:16.721 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.647+0000 7f83c67fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:00:16.721 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.647+0000 7f83c67fc700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:00:16.721 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.647+0000 7f83c67fc700 1 -- 172.21.15.102:0/2965573786 >> v1:172.21.15.102:6790/0 conn(0x7f83b8002840 legacy=0x7f83b8004ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:16.722 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.647+0000 7f83c67fc700 1 -- 172.21.15.102:0/2965573786 >> v1:172.21.15.102:6790/0 conn(0x7f83b8002840 legacy=0x7f83b8004ce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:00:16.722 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.653+0000 7f83dcb65700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:00:16.722 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.653+0000 7f83dcb65700 1 -- 172.21.15.102:0/2965573786 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f83b8007510 con 0x7f83b80068b0 2021-05-25T12:00:16.722 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.653+0000 7f83c67fc700 10 client.4427.objecter ms_handle_connect 0x7f83b80068b0 2021-05-25T12:00:16.723 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.653+0000 7f83c67fc700 10 client.4427.objecter resend_mon_ops 2021-05-25T12:00:16.723 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.653+0000 7f83c67fc700 1 -- 172.21.15.102:0/2965573786 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3749703907 0 0) 0x7f83b8007510 con 0x7f83b80068b0 2021-05-25T12:00:16.723 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.653+0000 7f83c67fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:16.723 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.653+0000 7f83c67fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:16.724 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.653+0000 7f83c67fc700 10 monclient(hunting): my global_id is 4433 2021-05-25T12:00:16.724 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.653+0000 7f83c67fc700 1 -- 172.21.15.102:0/2965573786 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83b80065a0 con 0x7f83b80068b0 2021-05-25T12:00:16.724 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.654+0000 7f83c67fc700 1 -- 172.21.15.102:0/2965573786 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1966174490 0 0) 0x7f83b80065a0 con 0x7f83b80068b0 2021-05-25T12:00:16.724 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.654+0000 7f83c67fc700 1 -- 172.21.15.102:0/2965573786 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f83b8007510 con 0x7f83b80068b0 2021-05-25T12:00:16.724 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.654+0000 7f83c67fc700 1 -- 172.21.15.102:0/2965573786 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f83c0005500 con 0x7f83b80068b0 2021-05-25T12:00:16.725 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.654+0000 7f83c67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:16.725 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.654+0000 7f83c67fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:00:16.725 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.654+0000 7f83c67fc700 10 monclient: dump: 2021-05-25T12:00:16.725 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:16.726 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:16.726 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:16.726 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:16.726 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:16.727 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:16.727 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:16.727 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:16.727 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:16.727 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:16.728 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.654+0000 7f83c67fc700 1 -- 172.21.15.102:0/2965573786 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3710314295 0 0) 0x7f83b8007510 con 0x7f83b80068b0 2021-05-25T12:00:16.728 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.654+0000 7f83c67fc700 1 -- 172.21.15.102:0/2965573786 --> v1:172.21.15.102:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f83b8008770 con 0x7f83b80068b0 2021-05-25T12:00:16.728 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.654+0000 7f83c67fc700 1 -- 172.21.15.102:0/2965573786 <== mon.2 v1:172.21.15.102:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4099 (unknown 0 0 2390279424) 0x7f83c00056b0 con 0x7f83b80068b0 2021-05-25T12:00:16.728 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.654+0000 7f83c67fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:00:16.729 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.654+0000 7f83c67fc700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:00:16.729 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.654+0000 7f83c67fc700 1 -- 172.21.15.102:0/2965573786 >> v1:172.21.15.102:6790/0 conn(0x7f83b80068b0 legacy=0x7f83b8006d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:16.729 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.654+0000 7f83c67fc700 1 -- 172.21.15.102:0/2965573786 >> v1:172.21.15.102:6790/0 conn(0x7f83b80068b0 legacy=0x7f83b8006d40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:00:16.730 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:00:16.730 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:00:16.730 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:00:16.730 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "peon", 2021-05-25T12:00:16.731 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 12, 2021-05-25T12:00:16.731 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:00:16.731 INFO:teuthology.orchestra.run.smithi102.stdout: 0, 2021-05-25T12:00:16.731 INFO:teuthology.orchestra.run.smithi102.stdout: 2 2021-05-25T12:00:16.732 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:00:16.732 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 0, 2021-05-25T12:00:16.732 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:00:16.732 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:00:16.733 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:00:16.733 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:00:16.734 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:00:16.734 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:00:16.734 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:00:16.735 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:00:16.735 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:00:16.736 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:00:16.736 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:00:16.736 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:00:16.737 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:00:16.737 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:00:16.737 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:00:16.737 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:00:16.738 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:00:16.738 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:00:16.738 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:00:16.738 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:00:16.739 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:00:16.739 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:00:16.739 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:00:16.739 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:00:16.740 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:00:16.740 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:16.740 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:00:16.740 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:00:16.740 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:00:16.741 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:00:16.741 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:00:16.741 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:00:16.741 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:00:16.742 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:00:16.742 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:00:16.742 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:00:16.742 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:00:16.743 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:00:16.743 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:00:16.743 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:00:16.743 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:00:16.744 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:00:16.744 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:00:16.744 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:00:16.744 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:00:16.745 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:00:16.745 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:00:16.745 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:00:16.745 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:00:16.745 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:00:16.746 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:00:16.746 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:00:16.746 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:16.746 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:00:16.747 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:16.747 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:00:16.747 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:00:16.747 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:00:16.747 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:00:16.748 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:16.748 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:00:16.748 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:00:16.748 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:00:16.749 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:16.749 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:00:16.749 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:16.749 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:00:16.750 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:00:16.750 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:00:16.750 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:00:16.750 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:00:16.751 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:16.751 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:16.751 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:00:16.751 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:00:16.752 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:00:16.752 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:00:16.752 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:16.752 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:00:16.753 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:00:16.753 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:00:16.753 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:16.753 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:00:16.754 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:16.754 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:00:16.754 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:00:16.754 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:00:16.755 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:00:16.755 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:00:16.755 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:16.755 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:16.756 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:00:16.756 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:00:16.756 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:00:16.756 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:00:16.757 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:16.757 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:00:16.757 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:00:16.757 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:00:16.758 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:16.758 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:00:16.758 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:16.758 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:00:16.759 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:00:16.759 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:00:16.759 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:00:16.759 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:00:16.760 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:16.760 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:00:16.760 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:16.760 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:00:16.761 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:00:16.761 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:16.761 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:00:16.761 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:00:16.761 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:00:16.762 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:16.762 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:00:16.762 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": [ 2021-05-25T12:00:16.762 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:16.763 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:00:16.763 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:00:16.763 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 5 2021-05-25T12:00:16.763 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:16.764 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:00:16.764 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:00:16.764 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:16.764 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:00:16.764 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:00:16.765 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 2 2021-05-25T12:00:16.765 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:16.765 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:00:16.765 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": [ 2021-05-25T12:00:16.765 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:16.766 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:00:16.766 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:00:16.766 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:00:16.766 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:16.767 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:00:16.767 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:16.767 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:00:16.767 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:00:16.770 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.654+0000 7f83dcb65700 10 librados: watch_flush enter 2021-05-25T12:00:16.770 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.654+0000 7f83dcb65700 10 librados: watch_flush exit 2021-05-25T12:00:16.770 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.654+0000 7f83dcb65700 20 client.4427.objecter shutdown clearing up homeless session... 2021-05-25T12:00:16.771 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.654+0000 7f83dcb65700 10 client.4427.objecter successfully canceled tick 2021-05-25T12:00:16.771 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.654+0000 7f83dcb65700 1 -- 172.21.15.102:0/2965573786 >> v1:172.21.15.181:6812/29582 conn(0x7f83b4033260 legacy=0x7f83b4035720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:16.771 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.654+0000 7f83dcb65700 10 monclient: shutdown 2021-05-25T12:00:16.771 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.654+0000 7f83dcb65700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:00:16.772 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.654+0000 7f83dcb65700 1 -- 172.21.15.102:0/2965573786 >> v1:172.21.15.102:6790/0 conn(0x7f83d8130520 legacy=0x7f83d8141420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:16.772 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.654+0000 7f83d6d9d700 1 -- 172.21.15.102:0/2965573786 reap_dead start 2021-05-25T12:00:16.772 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.654+0000 7f83d6d9d700 1 -- 172.21.15.102:0/2965573786 reap_dead start 2021-05-25T12:00:16.772 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.654+0000 7f83dcb65700 1 -- 172.21.15.102:0/2965573786 shutdown_connections 2021-05-25T12:00:16.773 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.654+0000 7f83dcb65700 1 -- 172.21.15.102:0/2965573786 >> 172.21.15.102:0/2965573786 conn(0x7f83d809cd10 msgr2=0x7f83d8092d20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:00:16.773 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.654+0000 7f83dcb65700 1 -- 172.21.15.102:0/2965573786 shutdown_connections 2021-05-25T12:00:16.773 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.654+0000 7f83dcb65700 1 -- 172.21.15.102:0/2965573786 wait complete. 2021-05-25T12:00:16.773 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:16.654+0000 7f83dcb65700 1 librados: shutdown 2021-05-25T12:00:16.774 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2021-05-25T12:00:36.775 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2021-05-25T12:00:36.775 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2021-05-25T12:00:36.776 INFO:tasks.ceph.mon.b:Restarting daemon 2021-05-25T12:00:36.776 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2021-05-25T12:00:36.779 INFO:tasks.ceph.mon.b:Started 2021-05-25T12:00:36.780 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2021-05-25T12:00:36.780 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2021-05-25T12:00:36.900 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.899+0000 7f3d1b59e700 10 monclient: get_monmap_and_config 2021-05-25T12:00:36.901 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.899+0000 7f3d1b59e700 10 monclient: build_initial_monmap 2021-05-25T12:00:36.901 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.899+0000 7f3d1b59e700 10 monclient: monmap: 2021-05-25T12:00:36.901 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:00:36.902 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:36.902 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:00:36.900267+0000 2021-05-25T12:00:36.902 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:00:36.900267+0000 2021-05-25T12:00:36.902 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:00:36.903 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:36.903 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:00:36.903 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:00:36.903 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:00:36.903 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:36.913 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.899+0000 7f3d1b59e700 1 Processor -- start 2021-05-25T12:00:36.913 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.899+0000 7f3d1b59e700 1 -- start start 2021-05-25T12:00:36.913 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.899+0000 7f3d1b59e700 10 monclient: init 2021-05-25T12:00:36.914 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.900+0000 7f3d1b59e700 10 monclient: _reopen_session rank -1 2021-05-25T12:00:36.914 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.900+0000 7f3d1b59e700 10 monclient: _add_conns ranks=[0,2,1] 2021-05-25T12:00:36.914 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.900+0000 7f3d1b59e700 10 monclient(hunting): picked mon.noname-a con 0x7f3d1c098930 addr v1:172.21.15.102:6789/0 2021-05-25T12:00:36.914 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.900+0000 7f3d1b59e700 10 monclient(hunting): picked mon.noname-b con 0x7f3d1c12bca0 addr v1:172.21.15.102:6790/0 2021-05-25T12:00:36.915 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.900+0000 7f3d1b59e700 10 monclient(hunting): picked mon.noname-c con 0x7f3d1c12c150 addr v1:172.21.15.181:6789/0 2021-05-25T12:00:36.915 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.900+0000 7f3d1b59e700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d1c099b20 con 0x7f3d1c098930 2021-05-25T12:00:36.915 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.900+0000 7f3d1b59e700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d1c09a3a0 con 0x7f3d1c12c150 2021-05-25T12:00:36.915 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.900+0000 7f3d1b59e700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d1c09ac60 con 0x7f3d1c12bca0 2021-05-25T12:00:36.915 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.900+0000 7f3d1b59e700 10 monclient(hunting): _renew_subs 2021-05-25T12:00:36.916 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.900+0000 7f3d1b59e700 10 monclient(hunting): authenticate will time out at 837.497571s 2021-05-25T12:00:36.916 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.900+0000 7f3d19d9b700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f3d1c12bca0 0x7f3d1c1308c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:54168/0 (socket says 172.21.15.102:54168) 2021-05-25T12:00:36.916 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.900+0000 7f3d1a59c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f3d1c098930 0x7f3d1c12b590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:59290/0 (socket says 172.21.15.102:59290) 2021-05-25T12:00:36.916 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.900+0000 7f3d19d9b700 1 -- 172.21.15.102:0/146319030 learned_addr learned my addr 172.21.15.102:0/146319030 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:00:36.916 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.900+0000 7f3d1959a700 1 -- 172.21.15.102:0/146319030 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 265172648 0 0) 0x7f3d1c099b20 con 0x7f3d1c098930 2021-05-25T12:00:36.917 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.900+0000 7f3d1959a700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:36.917 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.900+0000 7f3d1959a700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:36.917 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.900+0000 7f3d1959a700 10 monclient(hunting): my global_id is 4452 2021-05-25T12:00:36.917 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.900+0000 7f3d1959a700 1 -- 172.21.15.102:0/146319030 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d00003270 con 0x7f3d1c098930 2021-05-25T12:00:36.918 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.900+0000 7f3d1959a700 1 -- 172.21.15.102:0/146319030 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2173040401 0 0) 0x7f3d1c09ac60 con 0x7f3d1c12bca0 2021-05-25T12:00:36.918 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.900+0000 7f3d1959a700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:36.918 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.900+0000 7f3d1959a700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:36.918 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.900+0000 7f3d1959a700 10 monclient(hunting): my global_id is 4436 2021-05-25T12:00:36.918 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.901+0000 7f3d1959a700 1 -- 172.21.15.102:0/146319030 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d1c099b20 con 0x7f3d1c12bca0 2021-05-25T12:00:36.919 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.901+0000 7f3d1959a700 1 -- 172.21.15.102:0/146319030 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4023742219 0 0) 0x7f3d1c099b20 con 0x7f3d1c12bca0 2021-05-25T12:00:36.919 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.901+0000 7f3d1959a700 1 -- 172.21.15.102:0/146319030 >> v1:172.21.15.181:6789/0 conn(0x7f3d1c12c150 legacy=0x7f3d1c133ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:36.919 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.901+0000 7f3d1959a700 1 -- 172.21.15.102:0/146319030 >> v1:172.21.15.102:6789/0 conn(0x7f3d1c098930 legacy=0x7f3d1c12b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:36.919 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.901+0000 7f3d1959a700 10 monclient: _finish_hunting 0 2021-05-25T12:00:36.919 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.901+0000 7f3d1959a700 1 monclient: found mon.noname-b 2021-05-25T12:00:36.920 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.901+0000 7f3d1959a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:00:36.920 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.901+0000 7f3d1959a700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T12:00:36.920 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.901+0000 7f3d1959a700 1 -- 172.21.15.102:0/146319030 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d1c09af00 con 0x7f3d1c12bca0 2021-05-25T12:00:36.920 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.901+0000 7f3d1959a700 10 monclient: _finish_auth 0 2021-05-25T12:00:36.920 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.901+0000 7f3d1959a700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:00:36.921 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.901+0000 7f3d1b59e700 5 monclient: authenticate success, global_id 4436 2021-05-25T12:00:36.921 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.901+0000 7f3d1959a700 1 -- 172.21.15.102:0/146319030 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f3d080015f0 con 0x7f3d1c12bca0 2021-05-25T12:00:36.921 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.901+0000 7f3d1959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:36.921 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.901+0000 7f3d1959a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:00:36.921 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.901+0000 7f3d1959a700 10 monclient: dump: 2021-05-25T12:00:36.922 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:36.922 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:36.922 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:36.922 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:36.923 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:36.923 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:36.923 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:36.923 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:36.923 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:36.924 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:36.924 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.901+0000 7f3d1b59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:00:36.924 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.901+0000 7f3d1959a700 1 -- 172.21.15.102:0/146319030 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3d08001820 con 0x7f3d1c12bca0 2021-05-25T12:00:36.924 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.901+0000 7f3d1959a700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:00:36.924 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.901+0000 7f3d1959a700 1 -- 172.21.15.102:0/146319030 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f3d08001bf0 con 0x7f3d1c12bca0 2021-05-25T12:00:36.925 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.901+0000 7f3d1b59e700 10 monclient: get_monmap_and_config success 2021-05-25T12:00:36.925 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.901+0000 7f3d1b59e700 4 set_mon_vals no callback set 2021-05-25T12:00:36.925 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.901+0000 7f3d1959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:36.925 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.901+0000 7f3d1959a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:00:36.925 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.901+0000 7f3d1959a700 10 monclient: dump: 2021-05-25T12:00:36.926 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:36.926 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:36.926 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:36.926 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:36.926 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:36.927 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:36.927 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:36.927 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:36.927 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:36.927 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:36.928 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.901+0000 7f3d1b59e700 10 monclient: shutdown 2021-05-25T12:00:36.928 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.901+0000 7f3d1b59e700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:00:36.928 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.901+0000 7f3d1b59e700 1 -- 172.21.15.102:0/146319030 >> v1:172.21.15.102:6790/0 conn(0x7f3d1c12bca0 legacy=0x7f3d1c1308c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:36.928 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.901+0000 7f3d1b59e700 1 -- 172.21.15.102:0/146319030 shutdown_connections 2021-05-25T12:00:36.928 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.901+0000 7f3d1b59e700 1 -- 172.21.15.102:0/146319030 >> 172.21.15.102:0/146319030 conn(0x7f3d1c09e9d0 msgr2=0x7f3d1c09ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:00:36.929 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.901+0000 7f3d1b59e700 1 -- 172.21.15.102:0/146319030 shutdown_connections 2021-05-25T12:00:36.929 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.902+0000 7f3d1b59e700 1 -- 172.21.15.102:0/146319030 wait complete. 2021-05-25T12:00:36.929 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.902+0000 7f3d1b59e700 10 monclient: build_initial_monmap 2021-05-25T12:00:36.929 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.902+0000 7f3d1b59e700 10 monclient: monmap: 2021-05-25T12:00:36.930 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:00:36.930 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:00:36.930 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:00:36.930 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:00:36.930 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:00:36.931 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:00:36.931 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:00:36.931 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:00:36.931 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:00:36.931 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:36.932 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.902+0000 7f3d1b59e700 1 librados: starting msgr at 2021-05-25T12:00:36.932 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.902+0000 7f3d1b59e700 1 librados: starting objecter 2021-05-25T12:00:36.932 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.902+0000 7f3d1b59e700 1 Processor -- start 2021-05-25T12:00:36.932 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.902+0000 7f3d1b59e700 1 -- start start 2021-05-25T12:00:36.932 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.902+0000 7f3d1b59e700 1 librados: setting wanted keys 2021-05-25T12:00:36.933 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.902+0000 7f3d1b59e700 1 librados: calling monclient init 2021-05-25T12:00:36.933 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.902+0000 7f3d1b59e700 10 monclient: init 2021-05-25T12:00:36.933 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.903+0000 7f3d1b59e700 10 monclient: _reopen_session rank -1 2021-05-25T12:00:36.933 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.903+0000 7f3d1b59e700 10 monclient: _add_conns ranks=[2,1,0] 2021-05-25T12:00:36.933 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.903+0000 7f3d1b59e700 10 monclient(hunting): picked mon.noname-c con 0x7f3d1c098930 addr v1:172.21.15.102:6790/0 2021-05-25T12:00:36.934 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.903+0000 7f3d1b59e700 10 monclient(hunting): picked mon.noname-b con 0x7f3d1c12bca0 addr v1:172.21.15.181:6789/0 2021-05-25T12:00:36.934 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.903+0000 7f3d1b59e700 10 monclient(hunting): picked mon.noname-a con 0x7f3d1c12c150 addr v1:172.21.15.102:6789/0 2021-05-25T12:00:36.934 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.903+0000 7f3d1b59e700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d1c092c50 con 0x7f3d1c12c150 2021-05-25T12:00:36.934 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.903+0000 7f3d1b59e700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d1c0948e0 con 0x7f3d1c12bca0 2021-05-25T12:00:36.934 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.903+0000 7f3d1b59e700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d1c094b80 con 0x7f3d1c098930 2021-05-25T12:00:36.935 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.903+0000 7f3d1b59e700 10 monclient(hunting): _renew_subs 2021-05-25T12:00:36.935 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.903+0000 7f3d1b59e700 10 monclient(hunting): authenticate will time out at 837.500338s 2021-05-25T12:00:36.935 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.903+0000 7f3d1a59c700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f3d1c098930 0x7f3d1c092540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:54174/0 (socket says 172.21.15.102:54174) 2021-05-25T12:00:36.935 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.903+0000 7f3d1a59c700 1 -- 172.21.15.102:0/537647259 learned_addr learned my addr 172.21.15.102:0/537647259 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:00:36.936 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.903+0000 7f3d067fc700 10 client.?.objecter ms_handle_connect 0x7f3d1c098930 2021-05-25T12:00:36.936 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.903+0000 7f3d067fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:00:36.936 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.903+0000 7f3d067fc700 10 client.?.objecter ms_handle_connect 0x7f3d1c12c150 2021-05-25T12:00:36.936 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.903+0000 7f3d067fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:00:36.936 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.903+0000 7f3d067fc700 10 client.?.objecter ms_handle_connect 0x7f3d1c12bca0 2021-05-25T12:00:36.937 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.903+0000 7f3d067fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:00:36.937 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.904+0000 7f3d067fc700 1 -- 172.21.15.102:0/537647259 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3129466064 0 0) 0x7f3d1c092c50 con 0x7f3d1c12c150 2021-05-25T12:00:36.937 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.904+0000 7f3d067fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:36.937 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.904+0000 7f3d067fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:36.937 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.904+0000 7f3d067fc700 10 monclient(hunting): my global_id is 4455 2021-05-25T12:00:36.938 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.904+0000 7f3d067fc700 1 -- 172.21.15.102:0/537647259 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3cf8004c10 con 0x7f3d1c12c150 2021-05-25T12:00:36.938 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.904+0000 7f3d067fc700 1 -- 172.21.15.102:0/537647259 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1737162528 0 0) 0x7f3d1c094b80 con 0x7f3d1c098930 2021-05-25T12:00:36.938 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.904+0000 7f3d067fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:36.938 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.904+0000 7f3d067fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:36.938 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.904+0000 7f3d067fc700 10 monclient(hunting): my global_id is 4439 2021-05-25T12:00:36.939 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.904+0000 7f3d067fc700 1 -- 172.21.15.102:0/537647259 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d1c092c50 con 0x7f3d1c098930 2021-05-25T12:00:36.939 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.904+0000 7f3d067fc700 1 -- 172.21.15.102:0/537647259 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1862861902 0 0) 0x7f3cf8004c10 con 0x7f3d1c12c150 2021-05-25T12:00:36.939 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.904+0000 7f3d067fc700 1 -- 172.21.15.102:0/537647259 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3d1c094b80 con 0x7f3d1c12c150 2021-05-25T12:00:36.939 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.904+0000 7f3d067fc700 1 -- 172.21.15.102:0/537647259 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f3d0c002b60 con 0x7f3d1c12c150 2021-05-25T12:00:36.940 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.904+0000 7f3d067fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:36.940 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.904+0000 7f3d067fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:00:36.940 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.904+0000 7f3d067fc700 10 monclient(hunting): dump: 2021-05-25T12:00:36.940 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:36.940 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:36.941 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:36.941 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:36.941 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:36.941 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:36.941 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:36.942 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:36.942 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:36.942 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:36.942 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.904+0000 7f3d067fc700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:00:36.942 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.904+0000 7f3d067fc700 1 -- 172.21.15.102:0/537647259 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 201587570 0 0) 0x7f3d1c094b80 con 0x7f3d1c12c150 2021-05-25T12:00:36.943 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.904+0000 7f3d067fc700 1 -- 172.21.15.102:0/537647259 >> v1:172.21.15.102:6790/0 conn(0x7f3d1c098930 legacy=0x7f3d1c092540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:36.943 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.904+0000 7f3d067fc700 1 -- 172.21.15.102:0/537647259 >> v1:172.21.15.181:6789/0 conn(0x7f3d1c12bca0 legacy=0x7f3d1c0972e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:36.943 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.904+0000 7f3d067fc700 10 monclient: _finish_hunting 0 2021-05-25T12:00:36.943 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.904+0000 7f3d067fc700 1 monclient: found mon.a 2021-05-25T12:00:36.943 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.904+0000 7f3d067fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:00:36.944 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.904+0000 7f3d067fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:00:36.944 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.904+0000 7f3d067fc700 1 -- 172.21.15.102:0/537647259 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d1c094e20 con 0x7f3d1c12c150 2021-05-25T12:00:36.944 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.904+0000 7f3d067fc700 10 monclient: _finish_auth 0 2021-05-25T12:00:36.944 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.904+0000 7f3d067fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:00:36.945 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.904+0000 7f3d1b59e700 5 monclient: authenticate success, global_id 4455 2021-05-25T12:00:36.945 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.904+0000 7f3d1b59e700 10 monclient: _renew_subs 2021-05-25T12:00:36.945 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.904+0000 7f3d1b59e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:00:36.945 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.904+0000 7f3d1b59e700 1 -- 172.21.15.102:0/537647259 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d1c128820 con 0x7f3d1c12c150 2021-05-25T12:00:36.946 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.904+0000 7f3d1b59e700 10 client.4455.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:00:36.946 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.904+0000 7f3d1b59e700 10 monclient: _renew_subs 2021-05-25T12:00:36.946 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.904+0000 7f3d1b59e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:00:36.946 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.904+0000 7f3d1b59e700 1 -- 172.21.15.102:0/537647259 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d1c0979f0 con 0x7f3d1c12c150 2021-05-25T12:00:36.947 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.904+0000 7f3d1b59e700 1 librados: init done 2021-05-25T12:00:36.947 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.904+0000 7f3d067fc700 1 -- 172.21.15.102:0/537647259 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3d0c001520 con 0x7f3d1c12c150 2021-05-25T12:00:36.947 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.904+0000 7f3d067fc700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:00:36.947 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.904+0000 7f3d067fc700 1 -- 172.21.15.102:0/537647259 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f3d0c001830 con 0x7f3d1c12c150 2021-05-25T12:00:36.947 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.904+0000 7f3d067fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:36.948 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.904+0000 7f3d18d99700 4 set_mon_vals no callback set 2021-05-25T12:00:36.948 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.904+0000 7f3d067fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:00:36.948 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.904+0000 7f3d067fc700 10 monclient: dump: 2021-05-25T12:00:36.948 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:36.948 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:36.949 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:36.949 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:36.949 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:36.949 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:36.949 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:36.950 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:36.950 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:36.950 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:36.950 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.905+0000 7f3d067fc700 1 -- 172.21.15.102:0/537647259 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f3d0c001990 con 0x7f3d1c12c150 2021-05-25T12:00:36.950 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.905+0000 7f3d067fc700 1 -- 172.21.15.102:0/537647259 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(15..15 src has 1..15) v4 ==== 3549+0+0 (unknown 3948819498 0 0) 0x7f3d0c03a910 con 0x7f3d1c12c150 2021-05-25T12:00:36.951 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.905+0000 7f3d067fc700 10 client.4455.objecter ms_dispatch 0x7f3d1c001550 osd_map(15..15 src has 1..15) v4 2021-05-25T12:00:36.951 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.905+0000 7f3d067fc700 3 client.4455.objecter handle_osd_map got epochs [15,15] > 0 2021-05-25T12:00:36.951 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.905+0000 7f3d067fc700 3 client.4455.objecter handle_osd_map decoding full epoch 15 2021-05-25T12:00:36.952 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.905+0000 7f3d1b59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:00:36.952 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.905+0000 7f3d1b59e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:00:36.952 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.905+0000 7f3d1b59e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:00:36.952 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.905+0000 7f3d067fc700 20 client.4455.objecter dump_active .. 0 homeless 2021-05-25T12:00:36.952 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.905+0000 7f3d1b59e700 1 -- 172.21.15.102:0/537647259 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cfc004db0 con 0x7f3d1c12c150 2021-05-25T12:00:36.953 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.905+0000 7f3d1a59c700 10 monclient: get_auth_request con 0x7f3cf8033030 auth_method 0 2021-05-25T12:00:36.953 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.905+0000 7f3d067fc700 10 client.4455.objecter ms_handle_connect 0x7f3cf8033030 2021-05-25T12:00:36.953 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.909+0000 7f3d067fc700 1 -- 172.21.15.102:0/537647259 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7f3d0c0020a0 con 0x7f3d1c12c150 2021-05-25T12:00:36.953 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.909+0000 7f3d067fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:00:36.953 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:36.909+0000 7f3d067fc700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:00:37.117 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.115+0000 7f3d1b59e700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2021-05-25T12:00:37.118 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.115+0000 7f3d1b59e700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2021-05-25T12:00:37.118 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.115+0000 7f3d1b59e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:00:37.118 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.115+0000 7f3d1b59e700 1 -- 172.21.15.102:0/537647259 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f3cfc0055e0 con 0x7f3d1c12c150 2021-05-25T12:00:37.119 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.116+0000 7f3d067fc700 1 -- 172.21.15.102:0/537647259 <== mon.0 v1:172.21.15.102:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1342 (unknown 254746657 0 3243003726) 0x7f3d0c010830 con 0x7f3d1c12c150 2021-05-25T12:00:37.119 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.116+0000 7f3d067fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2021-05-25T12:00:37.119 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.116+0000 7f3d067fc700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:00:37.119 INFO:teuthology.orchestra.run.smithi102.stdout:{"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138303579357183","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"20ee91b6-8935-4b98-b868-85c03789ccc4","modified":"2021-05-25T11:59:11.011522Z","created":"2021-05-25T11:59:11.011522Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6790","nonce":0}]},"addr":"172.21.15.102:6790/0","public_addr":"172.21.15.102:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2021-05-25T12:00:37.121 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.117+0000 7f3d1b59e700 10 librados: watch_flush enter 2021-05-25T12:00:37.121 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.117+0000 7f3d1b59e700 10 librados: watch_flush exit 2021-05-25T12:00:37.121 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.117+0000 7f3d1b59e700 20 client.4455.objecter shutdown clearing up homeless session... 2021-05-25T12:00:37.122 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.117+0000 7f3d1b59e700 10 client.4455.objecter successfully canceled tick 2021-05-25T12:00:37.122 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.117+0000 7f3d1b59e700 1 -- 172.21.15.102:0/537647259 >> v1:172.21.15.181:6812/29582 conn(0x7f3cf8033030 legacy=0x7f3cf80354f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:37.122 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.117+0000 7f3d1b59e700 10 monclient: shutdown 2021-05-25T12:00:37.122 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.117+0000 7f3d1b59e700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:00:37.122 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.117+0000 7f3d1b59e700 1 -- 172.21.15.102:0/537647259 >> v1:172.21.15.102:6789/0 conn(0x7f3d1c12c150 legacy=0x7f3d1c142f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:37.123 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.117+0000 7f3d1b59e700 1 -- 172.21.15.102:0/537647259 shutdown_connections 2021-05-25T12:00:37.123 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.117+0000 7f3d1b59e700 1 -- 172.21.15.102:0/537647259 >> 172.21.15.102:0/537647259 conn(0x7f3d1c09e9d0 msgr2=0x7f3d1c09c800 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:00:37.123 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.117+0000 7f3d1b59e700 1 -- 172.21.15.102:0/537647259 shutdown_connections 2021-05-25T12:00:37.123 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.118+0000 7f3d1b59e700 1 -- 172.21.15.102:0/537647259 wait complete. 2021-05-25T12:00:37.123 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.118+0000 7f3d1b59e700 1 librados: shutdown 2021-05-25T12:00:37.130 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2021-05-25T12:00:37.130 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2021-05-25T12:00:37.240 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.238+0000 7fa04f59e700 10 monclient: get_monmap_and_config 2021-05-25T12:00:37.240 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.238+0000 7fa04f59e700 10 monclient: build_initial_monmap 2021-05-25T12:00:37.241 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.238+0000 7fa04f59e700 10 monclient: monmap: 2021-05-25T12:00:37.241 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:00:37.241 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:37.241 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:00:37.239848+0000 2021-05-25T12:00:37.242 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:00:37.239848+0000 2021-05-25T12:00:37.242 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:00:37.242 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:37.243 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:00:37.243 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:00:37.243 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:00:37.244 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:37.273 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.239+0000 7fa04f59e700 1 Processor -- start 2021-05-25T12:00:37.274 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.239+0000 7fa04f59e700 1 -- start start 2021-05-25T12:00:37.274 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.239+0000 7fa04f59e700 10 monclient: init 2021-05-25T12:00:37.275 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.240+0000 7fa04f59e700 10 monclient: _reopen_session rank -1 2021-05-25T12:00:37.275 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.240+0000 7fa04f59e700 10 monclient: _add_conns ranks=[1,0,2] 2021-05-25T12:00:37.276 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.240+0000 7fa04f59e700 10 monclient(hunting): picked mon.noname-c con 0x7fa05009b4e0 addr v1:172.21.15.181:6789/0 2021-05-25T12:00:37.276 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.240+0000 7fa04f59e700 10 monclient(hunting): picked mon.noname-a con 0x7fa050068b40 addr v1:172.21.15.102:6789/0 2021-05-25T12:00:37.276 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.240+0000 7fa04f59e700 10 monclient(hunting): picked mon.noname-b con 0x7fa050068ff0 addr v1:172.21.15.102:6790/0 2021-05-25T12:00:37.277 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.240+0000 7fa04f59e700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa050096d20 con 0x7fa050068b40 2021-05-25T12:00:37.277 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.240+0000 7fa04f59e700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa050096fc0 con 0x7fa05009b4e0 2021-05-25T12:00:37.278 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.240+0000 7fa04f59e700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa050097260 con 0x7fa050068ff0 2021-05-25T12:00:37.278 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.240+0000 7fa04f59e700 10 monclient(hunting): _renew_subs 2021-05-25T12:00:37.279 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.240+0000 7fa04f59e700 10 monclient(hunting): authenticate will time out at 837.837188s 2021-05-25T12:00:37.279 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.240+0000 7fa04ed9d700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fa050068ff0 0x7fa050134000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:54184/0 (socket says 172.21.15.102:54184) 2021-05-25T12:00:37.279 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.240+0000 7fa04dd9b700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fa050068b40 0x7fa0501308d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:59302/0 (socket says 172.21.15.102:59302) 2021-05-25T12:00:37.280 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.240+0000 7fa04ed9d700 1 -- 172.21.15.102:0/837313298 learned_addr learned my addr 172.21.15.102:0/837313298 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:00:37.280 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.240+0000 7fa04d59a700 1 -- 172.21.15.102:0/837313298 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 118417106 0 0) 0x7fa050097260 con 0x7fa050068ff0 2021-05-25T12:00:37.281 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.240+0000 7fa04d59a700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:37.281 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.240+0000 7fa04d59a700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:37.281 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.240+0000 7fa04d59a700 10 monclient(hunting): my global_id is 4442 2021-05-25T12:00:37.282 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.240+0000 7fa04d59a700 1 -- 172.21.15.102:0/837313298 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa034003270 con 0x7fa050068ff0 2021-05-25T12:00:37.282 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.240+0000 7fa04d59a700 1 -- 172.21.15.102:0/837313298 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2572655980 0 0) 0x7fa050096d20 con 0x7fa050068b40 2021-05-25T12:00:37.283 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.240+0000 7fa04d59a700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:37.283 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.240+0000 7fa04d59a700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:37.284 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.240+0000 7fa04d59a700 10 monclient(hunting): my global_id is 4458 2021-05-25T12:00:37.284 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.240+0000 7fa04d59a700 1 -- 172.21.15.102:0/837313298 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa050097260 con 0x7fa050068b40 2021-05-25T12:00:37.284 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.240+0000 7fa04d59a700 1 -- 172.21.15.102:0/837313298 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3625052415 0 0) 0x7fa050096fc0 con 0x7fa05009b4e0 2021-05-25T12:00:37.285 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.240+0000 7fa04d59a700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:37.285 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.240+0000 7fa04d59a700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:37.286 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.240+0000 7fa04d59a700 10 monclient(hunting): my global_id is 74098 2021-05-25T12:00:37.286 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.240+0000 7fa04d59a700 1 -- 172.21.15.102:0/837313298 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa050096d20 con 0x7fa05009b4e0 2021-05-25T12:00:37.287 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.240+0000 7fa04d59a700 1 -- 172.21.15.102:0/837313298 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4081653396 0 0) 0x7fa034003270 con 0x7fa050068ff0 2021-05-25T12:00:37.287 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.240+0000 7fa04d59a700 1 -- 172.21.15.102:0/837313298 >> v1:172.21.15.181:6789/0 conn(0x7fa05009b4e0 legacy=0x7fa050068430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:37.288 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.240+0000 7fa04d59a700 1 -- 172.21.15.102:0/837313298 >> v1:172.21.15.102:6789/0 conn(0x7fa050068b40 legacy=0x7fa0501308d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:37.288 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.240+0000 7fa04d59a700 10 monclient: _finish_hunting 0 2021-05-25T12:00:37.288 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.240+0000 7fa04d59a700 1 monclient: found mon.noname-b 2021-05-25T12:00:37.289 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.240+0000 7fa04d59a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:00:37.289 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.240+0000 7fa04d59a700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T12:00:37.290 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.240+0000 7fa04d59a700 1 -- 172.21.15.102:0/837313298 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa050090990 con 0x7fa050068ff0 2021-05-25T12:00:37.290 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.240+0000 7fa04d59a700 10 monclient: _finish_auth 0 2021-05-25T12:00:37.291 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.240+0000 7fa04d59a700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:00:37.291 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.240+0000 7fa04d59a700 1 -- 172.21.15.102:0/837313298 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fa040001990 con 0x7fa050068ff0 2021-05-25T12:00:37.292 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.240+0000 7fa04f59e700 5 monclient: authenticate success, global_id 4442 2021-05-25T12:00:37.292 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.240+0000 7fa04d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:37.292 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.240+0000 7fa04d59a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:00:37.293 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.240+0000 7fa04d59a700 10 monclient: dump: 2021-05-25T12:00:37.293 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:37.294 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:37.294 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:37.295 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:37.295 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:37.295 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:37.296 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:37.296 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:37.297 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:37.297 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:37.298 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.240+0000 7fa04f59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:00:37.298 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.240+0000 7fa04d59a700 1 -- 172.21.15.102:0/837313298 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa040001c00 con 0x7fa050068ff0 2021-05-25T12:00:37.299 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.240+0000 7fa04d59a700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:00:37.299 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.240+0000 7fa04d59a700 1 -- 172.21.15.102:0/837313298 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fa040001f90 con 0x7fa050068ff0 2021-05-25T12:00:37.299 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.240+0000 7fa04f59e700 10 monclient: get_monmap_and_config success 2021-05-25T12:00:37.300 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.240+0000 7fa04d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:37.300 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.240+0000 7fa04f59e700 4 set_mon_vals no callback set 2021-05-25T12:00:37.301 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.240+0000 7fa04d59a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:00:37.301 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.240+0000 7fa04d59a700 10 monclient: dump: 2021-05-25T12:00:37.301 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:37.302 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:37.302 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:37.303 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:37.303 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:37.304 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:37.304 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:37.304 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:37.305 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:37.305 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:37.305 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.241+0000 7fa04f59e700 10 monclient: shutdown 2021-05-25T12:00:37.305 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.241+0000 7fa04f59e700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:00:37.306 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.241+0000 7fa04f59e700 1 -- 172.21.15.102:0/837313298 >> v1:172.21.15.102:6790/0 conn(0x7fa050068ff0 legacy=0x7fa050134000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:37.306 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.241+0000 7fa04f59e700 1 -- 172.21.15.102:0/837313298 shutdown_connections 2021-05-25T12:00:37.306 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.241+0000 7fa04f59e700 1 -- 172.21.15.102:0/837313298 >> 172.21.15.102:0/837313298 conn(0x7fa050097f90 msgr2=0x7fa05009a9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:00:37.306 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.241+0000 7fa04f59e700 1 -- 172.21.15.102:0/837313298 shutdown_connections 2021-05-25T12:00:37.307 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.241+0000 7fa04f59e700 1 -- 172.21.15.102:0/837313298 wait complete. 2021-05-25T12:00:37.307 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.241+0000 7fa04f59e700 10 monclient: build_initial_monmap 2021-05-25T12:00:37.307 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.241+0000 7fa04f59e700 10 monclient: monmap: 2021-05-25T12:00:37.307 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:00:37.308 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:00:37.308 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:00:37.308 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:00:37.308 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:00:37.308 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:00:37.309 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:00:37.309 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:00:37.309 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:00:37.309 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:37.309 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.242+0000 7fa04f59e700 1 librados: starting msgr at 2021-05-25T12:00:37.310 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.242+0000 7fa04f59e700 1 librados: starting objecter 2021-05-25T12:00:37.310 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.242+0000 7fa04f59e700 1 Processor -- start 2021-05-25T12:00:37.310 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.242+0000 7fa04f59e700 1 -- start start 2021-05-25T12:00:37.310 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.242+0000 7fa04f59e700 1 librados: setting wanted keys 2021-05-25T12:00:37.311 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.242+0000 7fa04f59e700 1 librados: calling monclient init 2021-05-25T12:00:37.311 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.242+0000 7fa04f59e700 10 monclient: init 2021-05-25T12:00:37.311 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.242+0000 7fa04f59e700 10 monclient: _reopen_session rank -1 2021-05-25T12:00:37.311 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.242+0000 7fa04f59e700 10 monclient: _add_conns ranks=[2,0,1] 2021-05-25T12:00:37.311 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.242+0000 7fa04f59e700 10 monclient(hunting): picked mon.noname-c con 0x7fa050068b40 addr v1:172.21.15.102:6790/0 2021-05-25T12:00:37.312 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.242+0000 7fa04f59e700 10 monclient(hunting): picked mon.noname-a con 0x7fa050068ff0 addr v1:172.21.15.102:6789/0 2021-05-25T12:00:37.312 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.242+0000 7fa04f59e700 10 monclient(hunting): picked mon.noname-b con 0x7fa05009b4e0 addr v1:172.21.15.181:6789/0 2021-05-25T12:00:37.312 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.242+0000 7fa04f59e700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa05013b770 con 0x7fa050068ff0 2021-05-25T12:00:37.312 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.242+0000 7fa04f59e700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa050094720 con 0x7fa05009b4e0 2021-05-25T12:00:37.313 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.242+0000 7fa04f59e700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa05011bd10 con 0x7fa050068b40 2021-05-25T12:00:37.313 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.242+0000 7fa04f59e700 10 monclient(hunting): _renew_subs 2021-05-25T12:00:37.313 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.242+0000 7fa04f59e700 10 monclient(hunting): authenticate will time out at 837.839733s 2021-05-25T12:00:37.313 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.242+0000 7fa04dd9b700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fa050068ff0 0x7fa0501345d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:59310/0 (socket says 172.21.15.102:59310) 2021-05-25T12:00:37.313 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.242+0000 7fa04dd9b700 1 -- 172.21.15.102:0/3870335105 learned_addr learned my addr 172.21.15.102:0/3870335105 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:00:37.314 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.242+0000 7fa04e59c700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fa050068b40 0x7fa05013b060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:54186/0 (socket says 172.21.15.102:54186) 2021-05-25T12:00:37.314 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.242+0000 7fa03e7fc700 10 client.?.objecter ms_handle_connect 0x7fa050068ff0 2021-05-25T12:00:37.314 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.242+0000 7fa03e7fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:00:37.314 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.242+0000 7fa03e7fc700 10 client.?.objecter ms_handle_connect 0x7fa050068b40 2021-05-25T12:00:37.315 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.242+0000 7fa03e7fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:00:37.315 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.242+0000 7fa03e7fc700 10 client.?.objecter ms_handle_connect 0x7fa05009b4e0 2021-05-25T12:00:37.315 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.242+0000 7fa03e7fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:00:37.315 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.242+0000 7fa03e7fc700 1 -- 172.21.15.102:0/3870335105 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3381777512 0 0) 0x7fa05013b770 con 0x7fa050068ff0 2021-05-25T12:00:37.315 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.243+0000 7fa03e7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:37.316 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.243+0000 7fa03e7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:37.316 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.243+0000 7fa03e7fc700 10 monclient(hunting): my global_id is 4461 2021-05-25T12:00:37.316 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.243+0000 7fa03e7fc700 1 -- 172.21.15.102:0/3870335105 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa02c004c10 con 0x7fa050068ff0 2021-05-25T12:00:37.316 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.243+0000 7fa03e7fc700 1 -- 172.21.15.102:0/3870335105 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1872292434 0 0) 0x7fa05011bd10 con 0x7fa050068b40 2021-05-25T12:00:37.317 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.243+0000 7fa03e7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:37.317 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.243+0000 7fa03e7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:37.317 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.243+0000 7fa03e7fc700 10 monclient(hunting): my global_id is 4445 2021-05-25T12:00:37.317 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.243+0000 7fa03e7fc700 1 -- 172.21.15.102:0/3870335105 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa05013b770 con 0x7fa050068b40 2021-05-25T12:00:37.317 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.243+0000 7fa03e7fc700 1 -- 172.21.15.102:0/3870335105 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1811907058 0 0) 0x7fa050094720 con 0x7fa05009b4e0 2021-05-25T12:00:37.318 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.243+0000 7fa03e7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:37.318 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.243+0000 7fa03e7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:37.318 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.243+0000 7fa03e7fc700 10 monclient(hunting): my global_id is 74101 2021-05-25T12:00:37.318 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.243+0000 7fa03e7fc700 1 -- 172.21.15.102:0/3870335105 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa05011bd10 con 0x7fa05009b4e0 2021-05-25T12:00:37.318 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.243+0000 7fa03e7fc700 1 -- 172.21.15.102:0/3870335105 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 210769755 0 0) 0x7fa05013b770 con 0x7fa050068b40 2021-05-25T12:00:37.319 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.243+0000 7fa03e7fc700 1 -- 172.21.15.102:0/3870335105 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa050094720 con 0x7fa050068b40 2021-05-25T12:00:37.319 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.243+0000 7fa03e7fc700 1 -- 172.21.15.102:0/3870335105 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fa0380018c0 con 0x7fa050068b40 2021-05-25T12:00:37.319 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.243+0000 7fa03e7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:37.319 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.243+0000 7fa03e7fc700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:00:37.320 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.243+0000 7fa03e7fc700 10 monclient(hunting): dump: 2021-05-25T12:00:37.320 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:37.320 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:37.320 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:37.320 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:37.321 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:37.321 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:37.321 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:37.321 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:37.322 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:37.322 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:37.322 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.243+0000 7fa03e7fc700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:00:37.322 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.243+0000 7fa03e7fc700 1 -- 172.21.15.102:0/3870335105 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3178268399 0 0) 0x7fa02c004c10 con 0x7fa050068ff0 2021-05-25T12:00:37.323 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.243+0000 7fa03e7fc700 1 -- 172.21.15.102:0/3870335105 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa05013b770 con 0x7fa050068ff0 2021-05-25T12:00:37.323 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.243+0000 7fa03e7fc700 1 -- 172.21.15.102:0/3870335105 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2724321661 0 0) 0x7fa05011bd10 con 0x7fa05009b4e0 2021-05-25T12:00:37.323 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.243+0000 7fa03e7fc700 1 -- 172.21.15.102:0/3870335105 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa02c004c10 con 0x7fa05009b4e0 2021-05-25T12:00:37.323 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.243+0000 7fa03e7fc700 1 -- 172.21.15.102:0/3870335105 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fa0440018a0 con 0x7fa050068ff0 2021-05-25T12:00:37.323 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.243+0000 7fa03e7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:37.324 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.243+0000 7fa03e7fc700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2021-05-25T12:00:37.324 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.243+0000 7fa03e7fc700 10 monclient(hunting): dump: 2021-05-25T12:00:37.324 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:37.324 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:37.325 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:37.325 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:37.325 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:37.325 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:37.325 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:37.326 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:37.326 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:37.326 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:37.326 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.243+0000 7fa03e7fc700 1 -- 172.21.15.102:0/3870335105 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fa040001950 con 0x7fa05009b4e0 2021-05-25T12:00:37.327 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.243+0000 7fa03e7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:37.327 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.243+0000 7fa03e7fc700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2021-05-25T12:00:37.327 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.243+0000 7fa03e7fc700 10 monclient(hunting): dump: 2021-05-25T12:00:37.327 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:37.327 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:37.328 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:37.328 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:37.328 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:37.328 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:37.328 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:37.329 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:37.329 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:37.329 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:37.329 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.243+0000 7fa03e7fc700 1 -- 172.21.15.102:0/3870335105 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2941620833 0 0) 0x7fa050094720 con 0x7fa050068b40 2021-05-25T12:00:37.330 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.243+0000 7fa03e7fc700 1 -- 172.21.15.102:0/3870335105 >> v1:172.21.15.181:6789/0 conn(0x7fa05009b4e0 legacy=0x7fa050134ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:37.330 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.243+0000 7fa03e7fc700 1 -- 172.21.15.102:0/3870335105 >> v1:172.21.15.102:6789/0 conn(0x7fa050068ff0 legacy=0x7fa0501345d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:37.330 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.243+0000 7fa03e7fc700 10 monclient: _finish_hunting 0 2021-05-25T12:00:37.330 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.243+0000 7fa03e7fc700 1 monclient: found mon.c 2021-05-25T12:00:37.330 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.243+0000 7fa03e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:00:37.331 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.243+0000 7fa03e7fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:00:37.331 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.243+0000 7fa03e7fc700 1 -- 172.21.15.102:0/3870335105 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa05011bfb0 con 0x7fa050068b40 2021-05-25T12:00:37.331 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.243+0000 7fa03e7fc700 10 monclient: _finish_auth 0 2021-05-25T12:00:37.331 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.243+0000 7fa03e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:00:37.332 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.243+0000 7fa04f59e700 5 monclient: authenticate success, global_id 4445 2021-05-25T12:00:37.332 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.243+0000 7fa04f59e700 10 monclient: _renew_subs 2021-05-25T12:00:37.332 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.243+0000 7fa04f59e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:00:37.332 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.243+0000 7fa04f59e700 1 -- 172.21.15.102:0/3870335105 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa05013ac80 con 0x7fa050068b40 2021-05-25T12:00:37.333 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.243+0000 7fa04f59e700 10 client.4445.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:00:37.333 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.243+0000 7fa04f59e700 10 monclient: _renew_subs 2021-05-25T12:00:37.333 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.243+0000 7fa04f59e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:00:37.333 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.243+0000 7fa04f59e700 1 -- 172.21.15.102:0/3870335105 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa050143fe0 con 0x7fa050068b40 2021-05-25T12:00:37.333 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.243+0000 7fa04f59e700 1 librados: init done 2021-05-25T12:00:37.334 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.243+0000 7fa03e7fc700 1 -- 172.21.15.102:0/3870335105 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa038001ae0 con 0x7fa050068b40 2021-05-25T12:00:37.334 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.243+0000 7fa03e7fc700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:00:37.334 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.243+0000 7fa03e7fc700 1 -- 172.21.15.102:0/3870335105 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fa038001df0 con 0x7fa050068b40 2021-05-25T12:00:37.334 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.243+0000 7fa03e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:37.335 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.243+0000 7fa03e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:00:37.335 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.243+0000 7fa04cd99700 4 set_mon_vals no callback set 2021-05-25T12:00:37.335 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.243+0000 7fa03e7fc700 10 monclient: dump: 2021-05-25T12:00:37.335 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:37.335 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:37.336 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:37.336 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:37.336 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:37.336 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:37.337 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:37.337 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:37.337 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:37.337 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:37.337 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.244+0000 7fa04f59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:00:37.338 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.244+0000 7fa04f59e700 1 -- 172.21.15.102:0/3870335105 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fa0300065a0 con 0x7fa030002840 2021-05-25T12:00:37.338 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.244+0000 7fa03e7fc700 1 -- 172.21.15.102:0/3870335105 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7fa038001fd0 con 0x7fa050068b40 2021-05-25T12:00:37.338 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.244+0000 7fa03e7fc700 10 client.4445.objecter ms_handle_connect 0x7fa030002840 2021-05-25T12:00:37.338 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.244+0000 7fa03e7fc700 10 client.4445.objecter resend_mon_ops 2021-05-25T12:00:37.338 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.244+0000 7fa03e7fc700 1 -- 172.21.15.102:0/3870335105 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(15..15 src has 1..15) v4 ==== 3549+0+0 (unknown 3948819498 0 0) 0x7fa03803cd50 con 0x7fa050068b40 2021-05-25T12:00:37.339 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.244+0000 7fa03e7fc700 10 client.4445.objecter ms_dispatch 0x7fa050001550 osd_map(15..15 src has 1..15) v4 2021-05-25T12:00:37.339 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.244+0000 7fa03e7fc700 3 client.4445.objecter handle_osd_map got epochs [15,15] > 0 2021-05-25T12:00:37.339 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.244+0000 7fa03e7fc700 3 client.4445.objecter handle_osd_map decoding full epoch 15 2021-05-25T12:00:37.339 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.244+0000 7fa03e7fc700 20 client.4445.objecter dump_active .. 0 homeless 2021-05-25T12:00:37.340 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.244+0000 7fa03e7fc700 1 -- 172.21.15.102:0/3870335105 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2103982063 0 0) 0x7fa0300065a0 con 0x7fa030002840 2021-05-25T12:00:37.340 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.244+0000 7fa03e7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:37.340 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.244+0000 7fa03e7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:37.340 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.244+0000 7fa03e7fc700 10 monclient(hunting): my global_id is 4464 2021-05-25T12:00:37.340 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.244+0000 7fa03e7fc700 1 -- 172.21.15.102:0/3870335105 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa050094720 con 0x7fa030002840 2021-05-25T12:00:37.341 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.244+0000 7fa04ed9d700 10 monclient: get_auth_request con 0x7fa02c033600 auth_method 0 2021-05-25T12:00:37.341 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.244+0000 7fa03e7fc700 1 -- 172.21.15.102:0/3870335105 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1284954388 0 0) 0x7fa050094720 con 0x7fa030002840 2021-05-25T12:00:37.341 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.244+0000 7fa03e7fc700 1 -- 172.21.15.102:0/3870335105 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa0300065a0 con 0x7fa030002840 2021-05-25T12:00:37.341 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.244+0000 7fa03e7fc700 1 -- 172.21.15.102:0/3870335105 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fa044004530 con 0x7fa030002840 2021-05-25T12:00:37.342 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.244+0000 7fa03e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:37.342 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.244+0000 7fa03e7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:00:37.342 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.244+0000 7fa03e7fc700 10 monclient: dump: 2021-05-25T12:00:37.342 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:37.343 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:37.343 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:37.343 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:37.343 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:37.343 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:37.344 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:37.344 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:37.344 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:37.344 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:37.345 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.244+0000 7fa03e7fc700 10 client.4445.objecter ms_handle_connect 0x7fa02c033600 2021-05-25T12:00:37.345 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.244+0000 7fa03e7fc700 1 -- 172.21.15.102:0/3870335105 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1241893926 0 0) 0x7fa0300065a0 con 0x7fa030002840 2021-05-25T12:00:37.345 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.244+0000 7fa03e7fc700 1 -- 172.21.15.102:0/3870335105 --> v1:172.21.15.102:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa030009e40 con 0x7fa030002840 2021-05-25T12:00:37.345 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.245+0000 7fa03e7fc700 1 -- 172.21.15.102:0/3870335105 <== mon.0 v1:172.21.15.102:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7fa044006840 con 0x7fa030002840 2021-05-25T12:00:37.345 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.245+0000 7fa03e7fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:00:37.346 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.245+0000 7fa03e7fc700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:00:37.346 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.245+0000 7fa03e7fc700 1 -- 172.21.15.102:0/3870335105 >> v1:172.21.15.102:6789/0 conn(0x7fa030002840 legacy=0x7fa030004ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:37.346 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.245+0000 7fa03e7fc700 1 -- 172.21.15.102:0/3870335105 >> v1:172.21.15.102:6789/0 conn(0x7fa030002840 legacy=0x7fa030004ce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:00:37.346 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.251+0000 7fa04f59e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:00:37.347 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.251+0000 7fa04f59e700 1 -- 172.21.15.102:0/3870335105 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fa030009ad0 con 0x7fa030008d90 2021-05-25T12:00:37.347 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.251+0000 7fa03e7fc700 10 client.4445.objecter ms_handle_connect 0x7fa030008d90 2021-05-25T12:00:37.347 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.251+0000 7fa03e7fc700 10 client.4445.objecter resend_mon_ops 2021-05-25T12:00:37.347 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.251+0000 7fa03e7fc700 1 -- 172.21.15.102:0/3870335105 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1057414162 0 0) 0x7fa030009ad0 con 0x7fa030008d90 2021-05-25T12:00:37.347 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.251+0000 7fa03e7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:37.348 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.251+0000 7fa03e7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:37.348 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.251+0000 7fa03e7fc700 10 monclient(hunting): my global_id is 4467 2021-05-25T12:00:37.348 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.251+0000 7fa03e7fc700 1 -- 172.21.15.102:0/3870335105 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0300065a0 con 0x7fa030008d90 2021-05-25T12:00:37.348 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.252+0000 7fa03e7fc700 1 -- 172.21.15.102:0/3870335105 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3200745183 0 0) 0x7fa0300065a0 con 0x7fa030008d90 2021-05-25T12:00:37.349 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.252+0000 7fa03e7fc700 1 -- 172.21.15.102:0/3870335105 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa030009ad0 con 0x7fa030008d90 2021-05-25T12:00:37.349 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.252+0000 7fa03e7fc700 1 -- 172.21.15.102:0/3870335105 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fa044004cb0 con 0x7fa030008d90 2021-05-25T12:00:37.349 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.252+0000 7fa03e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:37.349 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.252+0000 7fa03e7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:00:37.349 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.252+0000 7fa03e7fc700 10 monclient: dump: 2021-05-25T12:00:37.350 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:37.350 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:37.350 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:37.350 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:37.351 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:37.351 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:37.351 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:37.351 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:37.351 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:37.352 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:37.352 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.252+0000 7fa03e7fc700 1 -- 172.21.15.102:0/3870335105 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2963694091 0 0) 0x7fa030009ad0 con 0x7fa030008d90 2021-05-25T12:00:37.352 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.252+0000 7fa03e7fc700 1 -- 172.21.15.102:0/3870335105 --> v1:172.21.15.102:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa030008740 con 0x7fa030008d90 2021-05-25T12:00:37.352 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.252+0000 7fa03e7fc700 1 -- 172.21.15.102:0/3870335105 <== mon.0 v1:172.21.15.102:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3942 (unknown 0 0 716253740) 0x7fa044006a50 con 0x7fa030008d90 2021-05-25T12:00:37.353 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.252+0000 7fa03e7fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:00:37.353 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.252+0000 7fa03e7fc700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:00:37.353 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.252+0000 7fa03e7fc700 1 -- 172.21.15.102:0/3870335105 >> v1:172.21.15.102:6789/0 conn(0x7fa030008d90 legacy=0x7fa030009220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:37.353 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.252+0000 7fa03e7fc700 1 -- 172.21.15.102:0/3870335105 >> v1:172.21.15.102:6789/0 conn(0x7fa030008d90 legacy=0x7fa030009220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:00:37.354 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:00:37.354 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:00:37.354 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:00:37.355 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "leader", 2021-05-25T12:00:37.355 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 14, 2021-05-25T12:00:37.355 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:00:37.355 INFO:teuthology.orchestra.run.smithi102.stdout: 0, 2021-05-25T12:00:37.355 INFO:teuthology.orchestra.run.smithi102.stdout: 1, 2021-05-25T12:00:37.356 INFO:teuthology.orchestra.run.smithi102.stdout: 2 2021-05-25T12:00:37.356 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:00:37.356 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 0, 2021-05-25T12:00:37.356 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:00:37.357 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:00:37.357 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:00:37.357 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:00:37.357 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:00:37.357 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:00:37.358 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:00:37.358 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:00:37.358 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:00:37.358 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:00:37.359 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:00:37.359 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:00:37.359 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:00:37.359 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:00:37.359 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:00:37.360 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:00:37.360 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:00:37.360 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:00:37.360 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:00:37.360 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:00:37.361 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:00:37.361 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:00:37.361 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:00:37.361 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:00:37.361 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:00:37.362 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:37.362 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:00:37.362 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:00:37.362 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:00:37.362 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:00:37.363 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:00:37.363 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:00:37.363 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:00:37.363 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:00:37.364 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:00:37.364 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:00:37.364 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:00:37.364 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:00:37.364 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:00:37.365 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:00:37.365 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:00:37.365 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:00:37.365 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:00:37.365 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:00:37.366 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:00:37.366 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:00:37.366 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:00:37.366 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:00:37.366 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:00:37.367 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:00:37.367 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:00:37.367 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:00:37.367 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:37.368 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:00:37.368 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:37.368 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:00:37.368 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:00:37.368 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:00:37.369 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:00:37.369 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:37.369 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:00:37.369 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:00:37.369 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:00:37.370 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:37.370 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:00:37.370 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:37.370 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:00:37.370 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:00:37.371 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:00:37.371 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:00:37.371 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:00:37.371 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:37.372 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:37.372 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:00:37.372 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:00:37.372 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:00:37.372 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:00:37.373 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:37.373 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:00:37.373 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:00:37.373 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:00:37.373 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:37.374 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:00:37.374 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:37.374 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:00:37.374 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:00:37.375 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:00:37.375 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:00:37.375 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:00:37.375 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:37.375 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:37.376 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:00:37.376 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:00:37.376 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:00:37.376 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:00:37.376 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:37.377 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:00:37.377 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:00:37.377 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:00:37.377 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:37.377 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:00:37.378 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:37.378 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:00:37.378 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:00:37.378 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:00:37.379 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:00:37.379 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:00:37.379 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:37.379 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:00:37.379 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:37.380 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:00:37.380 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:00:37.380 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:37.380 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:00:37.381 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:00:37.381 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:00:37.381 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:37.381 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:00:37.382 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": [ 2021-05-25T12:00:37.382 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:37.382 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:00:37.382 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:00:37.383 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:00:37.383 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:37.383 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:00:37.383 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:00:37.383 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:37.384 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:00:37.384 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:00:37.384 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 4 2021-05-25T12:00:37.384 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:37.384 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:00:37.385 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:37.385 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:00:37.385 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:00:37.387 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.252+0000 7fa04f59e700 10 librados: watch_flush enter 2021-05-25T12:00:37.388 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.252+0000 7fa04f59e700 10 librados: watch_flush exit 2021-05-25T12:00:37.388 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.252+0000 7fa04f59e700 20 client.4445.objecter shutdown clearing up homeless session... 2021-05-25T12:00:37.388 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.252+0000 7fa04f59e700 10 client.4445.objecter successfully canceled tick 2021-05-25T12:00:37.388 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.252+0000 7fa04f59e700 1 -- 172.21.15.102:0/3870335105 >> v1:172.21.15.181:6812/29582 conn(0x7fa02c033600 legacy=0x7fa02c035ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:37.389 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.252+0000 7fa04f59e700 10 monclient: shutdown 2021-05-25T12:00:37.389 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.252+0000 7fa04f59e700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:00:37.389 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.252+0000 7fa04f59e700 1 -- 172.21.15.102:0/3870335105 >> v1:172.21.15.102:6790/0 conn(0x7fa050068b40 legacy=0x7fa05013b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:37.389 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.252+0000 7fa04ed9d700 1 -- 172.21.15.102:0/3870335105 reap_dead start 2021-05-25T12:00:37.389 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.252+0000 7fa04ed9d700 1 -- 172.21.15.102:0/3870335105 reap_dead start 2021-05-25T12:00:37.390 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.252+0000 7fa04f59e700 1 -- 172.21.15.102:0/3870335105 shutdown_connections 2021-05-25T12:00:37.390 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.252+0000 7fa04f59e700 1 -- 172.21.15.102:0/3870335105 >> 172.21.15.102:0/3870335105 conn(0x7fa050097f90 msgr2=0x7fa0500923e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:00:37.390 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.252+0000 7fa04f59e700 1 -- 172.21.15.102:0/3870335105 shutdown_connections 2021-05-25T12:00:37.390 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.252+0000 7fa04f59e700 1 -- 172.21.15.102:0/3870335105 wait complete. 2021-05-25T12:00:37.390 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.252+0000 7fa04f59e700 1 librados: shutdown 2021-05-25T12:00:37.391 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2021-05-25T12:00:37.501 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.500+0000 7f1be5c57700 10 monclient: get_monmap_and_config 2021-05-25T12:00:37.502 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.500+0000 7f1be5c57700 10 monclient: build_initial_monmap 2021-05-25T12:00:37.502 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.500+0000 7f1be5c57700 10 monclient: monmap: 2021-05-25T12:00:37.502 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:00:37.502 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:37.503 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:00:37.500985+0000 2021-05-25T12:00:37.503 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:00:37.500985+0000 2021-05-25T12:00:37.503 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:00:37.503 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:37.503 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:00:37.504 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:00:37.504 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:00:37.504 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:37.516 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.500+0000 7f1be5c57700 1 Processor -- start 2021-05-25T12:00:37.517 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.500+0000 7f1be5c57700 1 -- start start 2021-05-25T12:00:37.517 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.500+0000 7f1be5c57700 10 monclient: init 2021-05-25T12:00:37.517 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.501+0000 7f1be5c57700 10 monclient: _reopen_session rank -1 2021-05-25T12:00:37.517 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.501+0000 7f1be5c57700 10 monclient: _add_conns ranks=[0,1,2] 2021-05-25T12:00:37.518 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.501+0000 7f1be5c57700 10 monclient(hunting): picked mon.noname-a con 0x7f1be0090990 addr v1:172.21.15.102:6789/0 2021-05-25T12:00:37.518 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.501+0000 7f1be5c57700 10 monclient(hunting): picked mon.noname-c con 0x7f1be0090d70 addr v1:172.21.15.181:6789/0 2021-05-25T12:00:37.518 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.501+0000 7f1be5c57700 10 monclient(hunting): picked mon.noname-b con 0x7f1be012e330 addr v1:172.21.15.102:6790/0 2021-05-25T12:00:37.518 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.501+0000 7f1be5c57700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1be0096fa0 con 0x7f1be0090990 2021-05-25T12:00:37.519 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.501+0000 7f1be5c57700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1be00926a0 con 0x7f1be0090d70 2021-05-25T12:00:37.519 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.501+0000 7f1be5c57700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1be0092940 con 0x7f1be012e330 2021-05-25T12:00:37.519 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.501+0000 7f1be5c57700 10 monclient(hunting): _renew_subs 2021-05-25T12:00:37.519 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.501+0000 7f1be5c57700 10 monclient(hunting): authenticate will time out at 838.098310s 2021-05-25T12:00:37.520 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.501+0000 7f1bdffff700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f1be012e330 0x7f1be0130720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:54202/0 (socket says 172.21.15.102:54202) 2021-05-25T12:00:37.520 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.501+0000 7f1bdffff700 1 -- 172.21.15.102:0/4287584298 learned_addr learned my addr 172.21.15.102:0/4287584298 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:00:37.520 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.501+0000 7f1bdf7fe700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f1be0090990 0x7f1be0093240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:59320/0 (socket says 172.21.15.102:59320) 2021-05-25T12:00:37.520 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.501+0000 7f1bde7fc700 1 -- 172.21.15.102:0/4287584298 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3385801455 0 0) 0x7f1be0096fa0 con 0x7f1be0090990 2021-05-25T12:00:37.521 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.501+0000 7f1bde7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:37.521 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.501+0000 7f1bde7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:37.521 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.501+0000 7f1bde7fc700 10 monclient(hunting): my global_id is 4470 2021-05-25T12:00:37.521 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.501+0000 7f1bde7fc700 1 -- 172.21.15.102:0/4287584298 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bc4004380 con 0x7f1be0090990 2021-05-25T12:00:37.521 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.501+0000 7f1bde7fc700 1 -- 172.21.15.102:0/4287584298 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 243710757 0 0) 0x7f1be0092940 con 0x7f1be012e330 2021-05-25T12:00:37.522 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.501+0000 7f1bde7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:37.522 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.501+0000 7f1bde7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:37.522 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.501+0000 7f1bde7fc700 10 monclient(hunting): my global_id is 4448 2021-05-25T12:00:37.522 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.501+0000 7f1bde7fc700 1 -- 172.21.15.102:0/4287584298 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1be0096fa0 con 0x7f1be012e330 2021-05-25T12:00:37.523 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.501+0000 7f1bde7fc700 1 -- 172.21.15.102:0/4287584298 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1318382903 0 0) 0x7f1be00926a0 con 0x7f1be0090d70 2021-05-25T12:00:37.523 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.501+0000 7f1bde7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:37.523 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.501+0000 7f1bde7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:37.523 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.501+0000 7f1bde7fc700 10 monclient(hunting): my global_id is 74104 2021-05-25T12:00:37.523 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.501+0000 7f1bde7fc700 1 -- 172.21.15.102:0/4287584298 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1be0092940 con 0x7f1be0090d70 2021-05-25T12:00:37.524 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.501+0000 7f1bde7fc700 1 -- 172.21.15.102:0/4287584298 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3648524261 0 0) 0x7f1bc4004380 con 0x7f1be0090990 2021-05-25T12:00:37.524 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.501+0000 7f1bde7fc700 1 -- 172.21.15.102:0/4287584298 >> v1:172.21.15.102:6790/0 conn(0x7f1be012e330 legacy=0x7f1be0130720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:37.524 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.501+0000 7f1bde7fc700 1 -- 172.21.15.102:0/4287584298 >> v1:172.21.15.181:6789/0 conn(0x7f1be0090d70 legacy=0x7f1be0091200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:37.524 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.501+0000 7f1bde7fc700 10 monclient: _finish_hunting 0 2021-05-25T12:00:37.524 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.501+0000 7f1bde7fc700 1 monclient: found mon.noname-a 2021-05-25T12:00:37.525 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.501+0000 7f1bde7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:00:37.525 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.501+0000 7f1bde7fc700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T12:00:37.525 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.501+0000 7f1bde7fc700 1 -- 172.21.15.102:0/4287584298 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1be0092be0 con 0x7f1be0090990 2021-05-25T12:00:37.525 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.501+0000 7f1bde7fc700 10 monclient: _finish_auth 0 2021-05-25T12:00:37.526 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.501+0000 7f1bde7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:00:37.526 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.501+0000 7f1bde7fc700 1 -- 172.21.15.102:0/4287584298 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f1bd0001610 con 0x7f1be0090990 2021-05-25T12:00:37.526 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.501+0000 7f1bde7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:37.526 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.501+0000 7f1bde7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:00:37.526 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.501+0000 7f1bde7fc700 10 monclient: dump: 2021-05-25T12:00:37.527 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:37.527 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:37.527 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:37.527 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:37.528 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:37.528 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:37.528 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:37.528 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:37.528 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:37.529 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:37.529 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.502+0000 7f1be5c57700 5 monclient: authenticate success, global_id 4470 2021-05-25T12:00:37.529 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.502+0000 7f1be5c57700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:00:37.529 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.502+0000 7f1bde7fc700 1 -- 172.21.15.102:0/4287584298 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1bd0001880 con 0x7f1be0090990 2021-05-25T12:00:37.529 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.502+0000 7f1bde7fc700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:00:37.530 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.502+0000 7f1bde7fc700 1 -- 172.21.15.102:0/4287584298 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f1bd0001c10 con 0x7f1be0090990 2021-05-25T12:00:37.530 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.502+0000 7f1be5c57700 10 monclient: get_monmap_and_config success 2021-05-25T12:00:37.530 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.502+0000 7f1be5c57700 4 set_mon_vals no callback set 2021-05-25T12:00:37.530 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.502+0000 7f1bde7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:37.531 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.502+0000 7f1bde7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:00:37.531 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.502+0000 7f1bde7fc700 10 monclient: dump: 2021-05-25T12:00:37.531 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:37.531 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:37.531 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:37.532 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:37.532 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:37.532 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:37.532 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:37.533 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:37.533 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:37.533 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:37.533 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.502+0000 7f1be5c57700 10 monclient: shutdown 2021-05-25T12:00:37.533 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.502+0000 7f1be5c57700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:00:37.534 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.502+0000 7f1be5c57700 1 -- 172.21.15.102:0/4287584298 >> v1:172.21.15.102:6789/0 conn(0x7f1be0090990 legacy=0x7f1be0093240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:37.534 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.502+0000 7f1be5c57700 1 -- 172.21.15.102:0/4287584298 shutdown_connections 2021-05-25T12:00:37.534 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.502+0000 7f1be5c57700 1 -- 172.21.15.102:0/4287584298 >> 172.21.15.102:0/4287584298 conn(0x7f1be009b370 msgr2=0x7f1be0097430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:00:37.534 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.502+0000 7f1be5c57700 1 -- 172.21.15.102:0/4287584298 shutdown_connections 2021-05-25T12:00:37.535 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.502+0000 7f1be5c57700 1 -- 172.21.15.102:0/4287584298 wait complete. 2021-05-25T12:00:37.535 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.503+0000 7f1be5c57700 10 monclient: build_initial_monmap 2021-05-25T12:00:37.535 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.503+0000 7f1be5c57700 10 monclient: monmap: 2021-05-25T12:00:37.535 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:00:37.535 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:00:37.536 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:00:37.536 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:00:37.536 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:00:37.536 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:00:37.536 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:00:37.537 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:00:37.537 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:00:37.537 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:37.537 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.503+0000 7f1be5c57700 1 librados: starting msgr at 2021-05-25T12:00:37.538 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.503+0000 7f1be5c57700 1 librados: starting objecter 2021-05-25T12:00:37.538 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.503+0000 7f1be5c57700 1 Processor -- start 2021-05-25T12:00:37.538 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.503+0000 7f1be5c57700 1 -- start start 2021-05-25T12:00:37.538 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.503+0000 7f1be5c57700 1 librados: setting wanted keys 2021-05-25T12:00:37.538 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.503+0000 7f1be5c57700 1 librados: calling monclient init 2021-05-25T12:00:37.539 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.503+0000 7f1be5c57700 10 monclient: init 2021-05-25T12:00:37.539 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.504+0000 7f1be5c57700 10 monclient: _reopen_session rank -1 2021-05-25T12:00:37.539 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.504+0000 7f1be5c57700 10 monclient: _add_conns ranks=[1,2,0] 2021-05-25T12:00:37.539 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.504+0000 7f1be5c57700 10 monclient(hunting): picked mon.noname-b con 0x7f1be0090990 addr v1:172.21.15.181:6789/0 2021-05-25T12:00:37.540 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.504+0000 7f1be5c57700 10 monclient(hunting): picked mon.noname-c con 0x7f1be0090d70 addr v1:172.21.15.102:6790/0 2021-05-25T12:00:37.540 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.504+0000 7f1be5c57700 10 monclient(hunting): picked mon.noname-a con 0x7f1be012e330 addr v1:172.21.15.102:6789/0 2021-05-25T12:00:37.540 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.504+0000 7f1be5c57700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1be0091fe0 con 0x7f1be012e330 2021-05-25T12:00:37.540 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.504+0000 7f1be5c57700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1be00687e0 con 0x7f1be0090990 2021-05-25T12:00:37.540 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.504+0000 7f1be5c57700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1be01142a0 con 0x7f1be0090d70 2021-05-25T12:00:37.541 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.504+0000 7f1be5c57700 10 monclient(hunting): _renew_subs 2021-05-25T12:00:37.541 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.504+0000 7f1be5c57700 10 monclient(hunting): authenticate will time out at 838.101137s 2021-05-25T12:00:37.541 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.504+0000 7f1bdffff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f1be012e330 0x7f1be01413c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:59330/0 (socket says 172.21.15.102:59330) 2021-05-25T12:00:37.541 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.504+0000 7f1bdffff700 1 -- 172.21.15.102:0/3568772934 learned_addr learned my addr 172.21.15.102:0/3568772934 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:00:37.542 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.504+0000 7f1bcb7fe700 10 client.?.objecter ms_handle_connect 0x7f1be012e330 2021-05-25T12:00:37.542 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.504+0000 7f1bcb7fe700 10 client.?.objecter resend_mon_ops 2021-05-25T12:00:37.542 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.504+0000 7f1bcb7fe700 10 client.?.objecter ms_handle_connect 0x7f1be0090d70 2021-05-25T12:00:37.542 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.504+0000 7f1bcb7fe700 10 client.?.objecter resend_mon_ops 2021-05-25T12:00:37.542 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.504+0000 7f1bcb7fe700 10 client.?.objecter ms_handle_connect 0x7f1be0090990 2021-05-25T12:00:37.543 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.504+0000 7f1bcb7fe700 10 client.?.objecter resend_mon_ops 2021-05-25T12:00:37.543 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.504+0000 7f1bcb7fe700 1 -- 172.21.15.102:0/3568772934 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3137275465 0 0) 0x7f1be0091fe0 con 0x7f1be012e330 2021-05-25T12:00:37.543 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.504+0000 7f1bcb7fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:37.543 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.504+0000 7f1bcb7fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:37.543 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.504+0000 7f1bcb7fe700 10 monclient(hunting): my global_id is 4473 2021-05-25T12:00:37.544 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.504+0000 7f1bcb7fe700 1 -- 172.21.15.102:0/3568772934 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bc0004c10 con 0x7f1be012e330 2021-05-25T12:00:37.544 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.504+0000 7f1bcb7fe700 1 -- 172.21.15.102:0/3568772934 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2422962714 0 0) 0x7f1be01142a0 con 0x7f1be0090d70 2021-05-25T12:00:37.544 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.504+0000 7f1bcb7fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:37.544 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.504+0000 7f1bcb7fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:37.545 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.504+0000 7f1bcb7fe700 10 monclient(hunting): my global_id is 4451 2021-05-25T12:00:37.545 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.504+0000 7f1bcb7fe700 1 -- 172.21.15.102:0/3568772934 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1be0091fe0 con 0x7f1be0090d70 2021-05-25T12:00:37.545 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.504+0000 7f1bcb7fe700 1 -- 172.21.15.102:0/3568772934 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4082871142 0 0) 0x7f1be00687e0 con 0x7f1be0090990 2021-05-25T12:00:37.545 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.504+0000 7f1bcb7fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:37.545 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.504+0000 7f1bcb7fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:37.546 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.504+0000 7f1bcb7fe700 10 monclient(hunting): my global_id is 74107 2021-05-25T12:00:37.546 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.504+0000 7f1bcb7fe700 1 -- 172.21.15.102:0/3568772934 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1be01142a0 con 0x7f1be0090990 2021-05-25T12:00:37.546 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.504+0000 7f1bcb7fe700 1 -- 172.21.15.102:0/3568772934 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3235426201 0 0) 0x7f1bc0004c10 con 0x7f1be012e330 2021-05-25T12:00:37.546 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.504+0000 7f1bcb7fe700 1 -- 172.21.15.102:0/3568772934 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1be00687e0 con 0x7f1be012e330 2021-05-25T12:00:37.546 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.504+0000 7f1bcb7fe700 1 -- 172.21.15.102:0/3568772934 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f1bd4001d50 con 0x7f1be012e330 2021-05-25T12:00:37.547 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.504+0000 7f1bcb7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:37.547 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.504+0000 7f1bcb7fe700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:00:37.547 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.504+0000 7f1bcb7fe700 10 monclient(hunting): dump: 2021-05-25T12:00:37.547 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:37.548 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:37.548 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:37.548 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:37.548 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:37.548 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:37.549 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:37.549 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:37.549 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:37.549 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:37.549 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.504+0000 7f1bcb7fe700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:00:37.550 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.504+0000 7f1bcb7fe700 1 -- 172.21.15.102:0/3568772934 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3763401968 0 0) 0x7f1be01142a0 con 0x7f1be0090990 2021-05-25T12:00:37.550 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.504+0000 7f1bcb7fe700 1 -- 172.21.15.102:0/3568772934 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1bc0004c10 con 0x7f1be0090990 2021-05-25T12:00:37.550 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.504+0000 7f1bcb7fe700 1 -- 172.21.15.102:0/3568772934 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f1bd00015d0 con 0x7f1be0090990 2021-05-25T12:00:37.550 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.504+0000 7f1bcb7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:37.551 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.504+0000 7f1bcb7fe700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2021-05-25T12:00:37.551 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.504+0000 7f1bcb7fe700 10 monclient(hunting): dump: 2021-05-25T12:00:37.551 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:37.551 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:37.552 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:37.552 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:37.552 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:37.552 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:37.552 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:37.553 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:37.553 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:37.553 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:37.553 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.504+0000 7f1bcb7fe700 1 -- 172.21.15.102:0/3568772934 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2581748732 0 0) 0x7f1be0091fe0 con 0x7f1be0090d70 2021-05-25T12:00:37.553 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.504+0000 7f1bcb7fe700 1 -- 172.21.15.102:0/3568772934 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1be01142a0 con 0x7f1be0090d70 2021-05-25T12:00:37.554 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.504+0000 7f1bcb7fe700 1 -- 172.21.15.102:0/3568772934 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f1bcc001840 con 0x7f1be0090d70 2021-05-25T12:00:37.554 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.504+0000 7f1bcb7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:37.554 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.504+0000 7f1bcb7fe700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2021-05-25T12:00:37.554 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.504+0000 7f1bcb7fe700 10 monclient(hunting): dump: 2021-05-25T12:00:37.555 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:37.555 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:37.555 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:37.555 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:37.555 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:37.556 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:37.556 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:37.556 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:37.556 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:37.557 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:37.557 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.504+0000 7f1bcb7fe700 1 -- 172.21.15.102:0/3568772934 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3767729062 0 0) 0x7f1be00687e0 con 0x7f1be012e330 2021-05-25T12:00:37.557 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.504+0000 7f1bcb7fe700 1 -- 172.21.15.102:0/3568772934 >> v1:172.21.15.102:6790/0 conn(0x7f1be0090d70 legacy=0x7f1be013dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:37.557 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.504+0000 7f1bcb7fe700 1 -- 172.21.15.102:0/3568772934 >> v1:172.21.15.181:6789/0 conn(0x7f1be0090990 legacy=0x7f1be01292d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:37.557 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.504+0000 7f1bcb7fe700 10 monclient: _finish_hunting 0 2021-05-25T12:00:37.558 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.504+0000 7f1bcb7fe700 1 monclient: found mon.a 2021-05-25T12:00:37.558 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.504+0000 7f1bcb7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:00:37.558 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.504+0000 7f1bcb7fe700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:00:37.558 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.504+0000 7f1bcb7fe700 1 -- 172.21.15.102:0/3568772934 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1be0115280 con 0x7f1be012e330 2021-05-25T12:00:37.559 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.504+0000 7f1bcb7fe700 10 monclient: _finish_auth 0 2021-05-25T12:00:37.559 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.504+0000 7f1bcb7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:00:37.559 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.504+0000 7f1be5c57700 5 monclient: authenticate success, global_id 4473 2021-05-25T12:00:37.559 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.504+0000 7f1be5c57700 10 monclient: _renew_subs 2021-05-25T12:00:37.559 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.504+0000 7f1be5c57700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:00:37.560 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.504+0000 7f1be5c57700 1 -- 172.21.15.102:0/3568772934 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1be01147b0 con 0x7f1be012e330 2021-05-25T12:00:37.560 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.504+0000 7f1be5c57700 10 client.4473.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:00:37.560 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.504+0000 7f1be5c57700 10 monclient: _renew_subs 2021-05-25T12:00:37.560 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.504+0000 7f1be5c57700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:00:37.560 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.504+0000 7f1be5c57700 1 -- 172.21.15.102:0/3568772934 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1be0067c40 con 0x7f1be012e330 2021-05-25T12:00:37.561 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.504+0000 7f1be5c57700 1 librados: init done 2021-05-25T12:00:37.561 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.505+0000 7f1bcb7fe700 1 -- 172.21.15.102:0/3568772934 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1bd40020a0 con 0x7f1be012e330 2021-05-25T12:00:37.561 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.505+0000 7f1bcb7fe700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:00:37.561 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.505+0000 7f1bddffb700 4 set_mon_vals no callback set 2021-05-25T12:00:37.562 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.505+0000 7f1bcb7fe700 1 -- 172.21.15.102:0/3568772934 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f1bd4004d00 con 0x7f1be012e330 2021-05-25T12:00:37.562 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.505+0000 7f1bcb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:37.562 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.505+0000 7f1bcb7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:00:37.562 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.505+0000 7f1bcb7fe700 10 monclient: dump: 2021-05-25T12:00:37.562 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:37.563 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:37.563 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:37.563 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:37.563 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:37.564 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:37.564 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:37.564 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:37.564 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:37.564 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:37.565 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.505+0000 7f1be5c57700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:00:37.565 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.505+0000 7f1be5c57700 1 -- 172.21.15.102:0/3568772934 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1bb80065a0 con 0x7f1bb8002840 2021-05-25T12:00:37.565 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.505+0000 7f1bcb7fe700 1 -- 172.21.15.102:0/3568772934 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f1bd400e020 con 0x7f1be012e330 2021-05-25T12:00:37.565 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.505+0000 7f1bcb7fe700 10 client.4473.objecter ms_handle_connect 0x7f1bb8002840 2021-05-25T12:00:37.565 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.505+0000 7f1bcb7fe700 10 client.4473.objecter resend_mon_ops 2021-05-25T12:00:37.566 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.505+0000 7f1bcb7fe700 1 -- 172.21.15.102:0/3568772934 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(15..15 src has 1..15) v4 ==== 3549+0+0 (unknown 3948819498 0 0) 0x7f1bd403d2c0 con 0x7f1be012e330 2021-05-25T12:00:37.566 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.505+0000 7f1bcb7fe700 10 client.4473.objecter ms_dispatch 0x7f1be0001550 osd_map(15..15 src has 1..15) v4 2021-05-25T12:00:37.566 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.505+0000 7f1bcb7fe700 3 client.4473.objecter handle_osd_map got epochs [15,15] > 0 2021-05-25T12:00:37.566 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.505+0000 7f1bcb7fe700 3 client.4473.objecter handle_osd_map decoding full epoch 15 2021-05-25T12:00:37.567 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.506+0000 7f1bcb7fe700 20 client.4473.objecter dump_active .. 0 homeless 2021-05-25T12:00:37.567 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.506+0000 7f1bcb7fe700 1 -- 172.21.15.102:0/3568772934 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 339275507 0 0) 0x7f1bb80065a0 con 0x7f1bb8002840 2021-05-25T12:00:37.567 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.506+0000 7f1bcb7fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:37.567 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.506+0000 7f1bcb7fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:37.567 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.506+0000 7f1bcb7fe700 10 monclient(hunting): my global_id is 4454 2021-05-25T12:00:37.568 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.506+0000 7f1bcb7fe700 1 -- 172.21.15.102:0/3568772934 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1be00687e0 con 0x7f1bb8002840 2021-05-25T12:00:37.568 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.506+0000 7f1bdeffd700 10 monclient: get_auth_request con 0x7f1bc0033500 auth_method 0 2021-05-25T12:00:37.568 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.506+0000 7f1bcb7fe700 1 -- 172.21.15.102:0/3568772934 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1802009611 0 0) 0x7f1be00687e0 con 0x7f1bb8002840 2021-05-25T12:00:37.568 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.506+0000 7f1bcb7fe700 1 -- 172.21.15.102:0/3568772934 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1bb80065a0 con 0x7f1bb8002840 2021-05-25T12:00:37.569 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.506+0000 7f1bcb7fe700 10 client.4473.objecter ms_handle_connect 0x7f1bc0033500 2021-05-25T12:00:37.569 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.506+0000 7f1bcb7fe700 1 -- 172.21.15.102:0/3568772934 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f1bd0004b70 con 0x7f1bb8002840 2021-05-25T12:00:37.569 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.506+0000 7f1bcb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:37.569 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.506+0000 7f1bcb7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:00:37.569 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.506+0000 7f1bcb7fe700 10 monclient: dump: 2021-05-25T12:00:37.570 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:37.570 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:37.570 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:37.570 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:37.571 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:37.571 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:37.571 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:37.571 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:37.571 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:37.572 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:37.572 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.506+0000 7f1bcb7fe700 1 -- 172.21.15.102:0/3568772934 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3034497819 0 0) 0x7f1bb80065a0 con 0x7f1bb8002840 2021-05-25T12:00:37.572 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.506+0000 7f1bcb7fe700 1 -- 172.21.15.102:0/3568772934 --> v1:172.21.15.102:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1bb8009e40 con 0x7f1bb8002840 2021-05-25T12:00:37.572 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.506+0000 7f1bcb7fe700 1 -- 172.21.15.102:0/3568772934 <== mon.2 v1:172.21.15.102:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7f1bd0006f80 con 0x7f1bb8002840 2021-05-25T12:00:37.573 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.506+0000 7f1bcb7fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:00:37.573 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.506+0000 7f1bcb7fe700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:00:37.573 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.506+0000 7f1bcb7fe700 1 -- 172.21.15.102:0/3568772934 >> v1:172.21.15.102:6790/0 conn(0x7f1bb8002840 legacy=0x7f1bb8004ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:37.573 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.506+0000 7f1bcb7fe700 1 -- 172.21.15.102:0/3568772934 >> v1:172.21.15.102:6790/0 conn(0x7f1bb8002840 legacy=0x7f1bb8004ce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:00:37.573 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.513+0000 7f1be5c57700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:00:37.574 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.513+0000 7f1be5c57700 1 -- 172.21.15.102:0/3568772934 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1bb8007510 con 0x7f1bb80068b0 2021-05-25T12:00:37.574 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.513+0000 7f1bcb7fe700 10 client.4473.objecter ms_handle_connect 0x7f1bb80068b0 2021-05-25T12:00:37.574 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.513+0000 7f1bcb7fe700 10 client.4473.objecter resend_mon_ops 2021-05-25T12:00:37.574 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.513+0000 7f1bcb7fe700 1 -- 172.21.15.102:0/3568772934 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1850704866 0 0) 0x7f1bb8007510 con 0x7f1bb80068b0 2021-05-25T12:00:37.574 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.513+0000 7f1bcb7fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:37.575 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.513+0000 7f1bcb7fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:37.575 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.513+0000 7f1bcb7fe700 10 monclient(hunting): my global_id is 4457 2021-05-25T12:00:37.575 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.513+0000 7f1bcb7fe700 1 -- 172.21.15.102:0/3568772934 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bb80065a0 con 0x7f1bb80068b0 2021-05-25T12:00:37.575 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.513+0000 7f1bcb7fe700 1 -- 172.21.15.102:0/3568772934 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3521530148 0 0) 0x7f1bb80065a0 con 0x7f1bb80068b0 2021-05-25T12:00:37.576 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.513+0000 7f1bcb7fe700 1 -- 172.21.15.102:0/3568772934 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1bb8007510 con 0x7f1bb80068b0 2021-05-25T12:00:37.576 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.513+0000 7f1bcb7fe700 1 -- 172.21.15.102:0/3568772934 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f1bd00051a0 con 0x7f1bb80068b0 2021-05-25T12:00:37.576 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.513+0000 7f1bcb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:37.576 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.513+0000 7f1bcb7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:00:37.576 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.513+0000 7f1bcb7fe700 10 monclient: dump: 2021-05-25T12:00:37.577 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:37.577 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:37.577 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:37.577 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:37.577 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:37.578 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:37.578 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:37.578 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:37.578 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:37.579 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:37.579 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.513+0000 7f1bcb7fe700 1 -- 172.21.15.102:0/3568772934 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2292941394 0 0) 0x7f1bb8007510 con 0x7f1bb80068b0 2021-05-25T12:00:37.579 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.513+0000 7f1bcb7fe700 1 -- 172.21.15.102:0/3568772934 --> v1:172.21.15.102:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f1bb8008770 con 0x7f1bb80068b0 2021-05-25T12:00:37.579 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.514+0000 7f1bcb7fe700 1 -- 172.21.15.102:0/3568772934 <== mon.2 v1:172.21.15.102:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4110 (unknown 0 0 4100274033) 0x7f1bd00064b0 con 0x7f1bb80068b0 2021-05-25T12:00:37.579 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.514+0000 7f1bcb7fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:00:37.580 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.514+0000 7f1bcb7fe700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:00:37.580 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.514+0000 7f1bcb7fe700 1 -- 172.21.15.102:0/3568772934 >> v1:172.21.15.102:6790/0 conn(0x7f1bb80068b0 legacy=0x7f1bb8006d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:37.580 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.514+0000 7f1bcb7fe700 1 -- 172.21.15.102:0/3568772934 >> v1:172.21.15.102:6790/0 conn(0x7f1bb80068b0 legacy=0x7f1bb8006d40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:00:37.581 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:00:37.581 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:00:37.581 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:00:37.581 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "peon", 2021-05-25T12:00:37.582 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 14, 2021-05-25T12:00:37.582 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:00:37.582 INFO:teuthology.orchestra.run.smithi102.stdout: 0, 2021-05-25T12:00:37.582 INFO:teuthology.orchestra.run.smithi102.stdout: 1, 2021-05-25T12:00:37.583 INFO:teuthology.orchestra.run.smithi102.stdout: 2 2021-05-25T12:00:37.583 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:00:37.583 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 0, 2021-05-25T12:00:37.583 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:00:37.583 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:00:37.584 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:00:37.584 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:00:37.584 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:00:37.584 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:00:37.584 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:00:37.585 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:00:37.585 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:00:37.585 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:00:37.585 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:00:37.585 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:00:37.586 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:00:37.586 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:00:37.586 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:00:37.586 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:00:37.587 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:00:37.587 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:00:37.587 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:00:37.587 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:00:37.587 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:00:37.588 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:00:37.588 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:00:37.588 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:00:37.588 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:00:37.588 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:37.589 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:00:37.589 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:00:37.589 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:00:37.589 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:00:37.589 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:00:37.590 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:00:37.590 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:00:37.590 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:00:37.590 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:00:37.591 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:00:37.591 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:00:37.591 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:00:37.591 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:00:37.591 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:00:37.592 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:00:37.592 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:00:37.592 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:00:37.592 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:00:37.592 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:00:37.593 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:00:37.593 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:00:37.593 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:00:37.593 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:00:37.594 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:00:37.594 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:00:37.594 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:00:37.594 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:37.594 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:00:37.595 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:37.595 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:00:37.595 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:00:37.595 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:00:37.595 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:00:37.596 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:37.596 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:00:37.596 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:00:37.596 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:00:37.596 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:37.597 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:00:37.597 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:37.597 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:00:37.597 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:00:37.598 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:00:37.598 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:00:37.598 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:00:37.598 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:37.598 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:37.599 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:00:37.599 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:00:37.599 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:00:37.599 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:00:37.599 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:37.600 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:00:37.600 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:00:37.600 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:00:37.600 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:37.601 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:00:37.601 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:37.601 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:00:37.601 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:00:37.601 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:00:37.602 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:00:37.602 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:00:37.602 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:37.602 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:37.602 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:00:37.603 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:00:37.603 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:00:37.603 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:00:37.603 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:37.603 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:00:37.604 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:00:37.604 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:00:37.604 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:37.604 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:00:37.605 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:37.605 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:00:37.605 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:00:37.605 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:00:37.605 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:00:37.606 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:00:37.606 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:37.606 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:00:37.606 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:37.606 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:00:37.607 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:00:37.607 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:37.607 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:00:37.607 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:00:37.607 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:00:37.608 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:37.608 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:00:37.608 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": [ 2021-05-25T12:00:37.608 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:37.609 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:00:37.609 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:00:37.609 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 5 2021-05-25T12:00:37.609 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:37.609 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:00:37.610 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:00:37.610 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:37.610 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:00:37.610 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:00:37.611 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:00:37.611 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:37.611 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:00:37.611 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": [ 2021-05-25T12:00:37.611 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:37.612 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:00:37.612 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:00:37.612 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:00:37.612 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:37.615 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:00:37.615 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:37.615 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:00:37.616 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:00:37.618 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.514+0000 7f1be5c57700 10 librados: watch_flush enter 2021-05-25T12:00:37.618 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.514+0000 7f1be5c57700 10 librados: watch_flush exit 2021-05-25T12:00:37.618 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.514+0000 7f1be5c57700 20 client.4473.objecter shutdown clearing up homeless session... 2021-05-25T12:00:37.619 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.514+0000 7f1be5c57700 10 client.4473.objecter successfully canceled tick 2021-05-25T12:00:37.619 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.514+0000 7f1be5c57700 1 -- 172.21.15.102:0/3568772934 >> v1:172.21.15.181:6812/29582 conn(0x7f1bc0033500 legacy=0x7f1bc00359c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:37.619 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.514+0000 7f1be5c57700 10 monclient: shutdown 2021-05-25T12:00:37.619 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.514+0000 7f1be5c57700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:00:37.619 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.514+0000 7f1be5c57700 1 -- 172.21.15.102:0/3568772934 >> v1:172.21.15.102:6789/0 conn(0x7f1be012e330 legacy=0x7f1be01413c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:37.620 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.514+0000 7f1bdffff700 1 -- 172.21.15.102:0/3568772934 reap_dead start 2021-05-25T12:00:37.620 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.514+0000 7f1bdffff700 1 -- 172.21.15.102:0/3568772934 reap_dead start 2021-05-25T12:00:37.620 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.514+0000 7f1be5c57700 1 -- 172.21.15.102:0/3568772934 shutdown_connections 2021-05-25T12:00:37.620 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.514+0000 7f1be5c57700 1 -- 172.21.15.102:0/3568772934 >> 172.21.15.102:0/3568772934 conn(0x7f1be009b370 msgr2=0x7f1be0093a60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:00:37.621 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.514+0000 7f1be5c57700 1 -- 172.21.15.102:0/3568772934 shutdown_connections 2021-05-25T12:00:37.621 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.514+0000 7f1be5c57700 1 -- 172.21.15.102:0/3568772934 wait complete. 2021-05-25T12:00:37.621 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.514+0000 7f1be5c57700 1 librados: shutdown 2021-05-25T12:00:37.621 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2021-05-25T12:00:37.731 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.730+0000 7fcfddb5b700 10 monclient: get_monmap_and_config 2021-05-25T12:00:37.731 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.730+0000 7fcfddb5b700 10 monclient: build_initial_monmap 2021-05-25T12:00:37.732 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.730+0000 7fcfddb5b700 10 monclient: monmap: 2021-05-25T12:00:37.732 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:00:37.732 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:37.732 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:00:37.730992+0000 2021-05-25T12:00:37.732 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:00:37.730992+0000 2021-05-25T12:00:37.733 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:00:37.733 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:37.733 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:00:37.733 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:00:37.734 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:00:37.734 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:37.748 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.730+0000 7fcfddb5b700 1 Processor -- start 2021-05-25T12:00:37.748 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.730+0000 7fcfddb5b700 1 -- start start 2021-05-25T12:00:37.748 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.730+0000 7fcfddb5b700 10 monclient: init 2021-05-25T12:00:37.749 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.731+0000 7fcfddb5b700 10 monclient: _reopen_session rank -1 2021-05-25T12:00:37.749 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.731+0000 7fcfddb5b700 10 monclient: _add_conns ranks=[2,0,1] 2021-05-25T12:00:37.749 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.731+0000 7fcfddb5b700 10 monclient(hunting): picked mon.noname-b con 0x7fcfd809a5e0 addr v1:172.21.15.102:6790/0 2021-05-25T12:00:37.749 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.731+0000 7fcfddb5b700 10 monclient(hunting): picked mon.noname-a con 0x7fcfd8063dc0 addr v1:172.21.15.102:6789/0 2021-05-25T12:00:37.750 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.731+0000 7fcfddb5b700 10 monclient(hunting): picked mon.noname-c con 0x7fcfd81325c0 addr v1:172.21.15.181:6789/0 2021-05-25T12:00:37.750 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.731+0000 7fcfddb5b700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfd8099fe0 con 0x7fcfd8063dc0 2021-05-25T12:00:37.750 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.731+0000 7fcfddb5b700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfd809a1e0 con 0x7fcfd81325c0 2021-05-25T12:00:37.750 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.731+0000 7fcfddb5b700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfd809b400 con 0x7fcfd809a5e0 2021-05-25T12:00:37.750 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.731+0000 7fcfddb5b700 10 monclient(hunting): _renew_subs 2021-05-25T12:00:37.751 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.731+0000 7fcfddb5b700 10 monclient(hunting): authenticate will time out at 838.328358s 2021-05-25T12:00:37.751 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.731+0000 7fcfd6ffd700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fcfd8063dc0 0x7fcfd8064240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:59340/0 (socket says 172.21.15.102:59340) 2021-05-25T12:00:37.751 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.731+0000 7fcfd6ffd700 1 -- 172.21.15.102:0/3032095580 learned_addr learned my addr 172.21.15.102:0/3032095580 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:00:37.751 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.731+0000 7fcfd67fc700 1 -- 172.21.15.102:0/3032095580 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1788860484 0 0) 0x7fcfd809b400 con 0x7fcfd809a5e0 2021-05-25T12:00:37.752 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.731+0000 7fcfd67fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:37.752 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.731+0000 7fcfd67fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:37.752 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.731+0000 7fcfd67fc700 10 monclient(hunting): my global_id is 4460 2021-05-25T12:00:37.752 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.731+0000 7fcfd67fc700 1 -- 172.21.15.102:0/3032095580 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcfbc003270 con 0x7fcfd809a5e0 2021-05-25T12:00:37.752 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.731+0000 7fcfd67fc700 1 -- 172.21.15.102:0/3032095580 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4120583082 0 0) 0x7fcfd8099fe0 con 0x7fcfd8063dc0 2021-05-25T12:00:37.753 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.731+0000 7fcfd67fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:37.753 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.731+0000 7fcfd67fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:37.753 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.731+0000 7fcfd67fc700 10 monclient(hunting): my global_id is 4476 2021-05-25T12:00:37.753 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.731+0000 7fcfd67fc700 1 -- 172.21.15.102:0/3032095580 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcfd809b400 con 0x7fcfd8063dc0 2021-05-25T12:00:37.754 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.731+0000 7fcfd67fc700 1 -- 172.21.15.102:0/3032095580 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2082353743 0 0) 0x7fcfd809a1e0 con 0x7fcfd81325c0 2021-05-25T12:00:37.754 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.731+0000 7fcfd67fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:37.754 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.731+0000 7fcfd67fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:37.754 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.731+0000 7fcfd67fc700 10 monclient(hunting): my global_id is 74110 2021-05-25T12:00:37.754 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.731+0000 7fcfd67fc700 1 -- 172.21.15.102:0/3032095580 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcfd8099fe0 con 0x7fcfd81325c0 2021-05-25T12:00:37.755 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.731+0000 7fcfd67fc700 1 -- 172.21.15.102:0/3032095580 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 188100639 0 0) 0x7fcfbc003270 con 0x7fcfd809a5e0 2021-05-25T12:00:37.755 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.731+0000 7fcfd67fc700 1 -- 172.21.15.102:0/3032095580 >> v1:172.21.15.181:6789/0 conn(0x7fcfd81325c0 legacy=0x7fcfd81349b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:37.755 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.731+0000 7fcfd67fc700 1 -- 172.21.15.102:0/3032095580 >> v1:172.21.15.102:6789/0 conn(0x7fcfd8063dc0 legacy=0x7fcfd8064240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:37.755 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.731+0000 7fcfd67fc700 10 monclient: _finish_hunting 0 2021-05-25T12:00:37.756 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.731+0000 7fcfd67fc700 1 monclient: found mon.noname-b 2021-05-25T12:00:37.756 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.731+0000 7fcfd67fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:00:37.756 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.731+0000 7fcfd67fc700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T12:00:37.756 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.731+0000 7fcfd67fc700 1 -- 172.21.15.102:0/3032095580 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfd80963f0 con 0x7fcfd809a5e0 2021-05-25T12:00:37.756 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.731+0000 7fcfd67fc700 10 monclient: _finish_auth 0 2021-05-25T12:00:37.757 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.731+0000 7fcfd67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:00:37.757 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.731+0000 7fcfd67fc700 1 -- 172.21.15.102:0/3032095580 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fcfc4001650 con 0x7fcfd809a5e0 2021-05-25T12:00:37.757 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.731+0000 7fcfddb5b700 5 monclient: authenticate success, global_id 4460 2021-05-25T12:00:37.757 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.731+0000 7fcfddb5b700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:00:37.758 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.731+0000 7fcfd67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:37.758 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.731+0000 7fcfd67fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:00:37.758 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.732+0000 7fcfd67fc700 10 monclient: dump: 2021-05-25T12:00:37.758 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:37.758 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:37.759 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:37.759 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:37.759 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:37.759 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:37.759 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:37.760 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:37.760 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:37.760 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:37.760 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.732+0000 7fcfddb5b700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:00:37.761 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.732+0000 7fcfd67fc700 1 -- 172.21.15.102:0/3032095580 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcfc40017f0 con 0x7fcfd809a5e0 2021-05-25T12:00:37.761 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.732+0000 7fcfd67fc700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:00:37.761 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.732+0000 7fcfddb5b700 10 monclient: get_monmap_and_config success 2021-05-25T12:00:37.761 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.732+0000 7fcfd67fc700 1 -- 172.21.15.102:0/3032095580 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fcfc4001b80 con 0x7fcfd809a5e0 2021-05-25T12:00:37.761 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.732+0000 7fcfd67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:37.762 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.732+0000 7fcfddb5b700 4 set_mon_vals no callback set 2021-05-25T12:00:37.762 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.732+0000 7fcfd67fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:00:37.762 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.732+0000 7fcfd67fc700 10 monclient: dump: 2021-05-25T12:00:37.762 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:37.762 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:37.763 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:37.763 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:37.763 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:37.763 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:37.764 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:37.764 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:37.764 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:37.764 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:37.764 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.732+0000 7fcfddb5b700 10 monclient: shutdown 2021-05-25T12:00:37.765 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.732+0000 7fcfddb5b700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:00:37.765 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.732+0000 7fcfddb5b700 1 -- 172.21.15.102:0/3032095580 >> v1:172.21.15.102:6790/0 conn(0x7fcfd809a5e0 legacy=0x7fcfd80636b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:37.765 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.732+0000 7fcfddb5b700 1 -- 172.21.15.102:0/3032095580 shutdown_connections 2021-05-25T12:00:37.765 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.732+0000 7fcfddb5b700 1 -- 172.21.15.102:0/3032095580 >> 172.21.15.102:0/3032095580 conn(0x7fcfd809bc30 msgr2=0x7fcfd809c040 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:00:37.765 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.732+0000 7fcfddb5b700 1 -- 172.21.15.102:0/3032095580 shutdown_connections 2021-05-25T12:00:37.766 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.732+0000 7fcfddb5b700 1 -- 172.21.15.102:0/3032095580 wait complete. 2021-05-25T12:00:37.766 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.733+0000 7fcfddb5b700 10 monclient: build_initial_monmap 2021-05-25T12:00:37.766 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.733+0000 7fcfddb5b700 10 monclient: monmap: 2021-05-25T12:00:37.766 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:00:37.767 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:00:37.767 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:00:37.767 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:00:37.767 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:00:37.767 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:00:37.768 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:00:37.768 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:00:37.768 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:00:37.768 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:37.768 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.733+0000 7fcfddb5b700 1 librados: starting msgr at 2021-05-25T12:00:37.769 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.733+0000 7fcfddb5b700 1 librados: starting objecter 2021-05-25T12:00:37.769 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.733+0000 7fcfddb5b700 1 Processor -- start 2021-05-25T12:00:37.769 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.733+0000 7fcfddb5b700 1 -- start start 2021-05-25T12:00:37.769 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.733+0000 7fcfddb5b700 1 librados: setting wanted keys 2021-05-25T12:00:37.769 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.733+0000 7fcfddb5b700 1 librados: calling monclient init 2021-05-25T12:00:37.770 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.733+0000 7fcfddb5b700 10 monclient: init 2021-05-25T12:00:37.770 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.733+0000 7fcfddb5b700 10 monclient: _reopen_session rank -1 2021-05-25T12:00:37.770 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.733+0000 7fcfddb5b700 10 monclient: _add_conns ranks=[1,0,2] 2021-05-25T12:00:37.770 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.734+0000 7fcfddb5b700 10 monclient(hunting): picked mon.noname-b con 0x7fcfd8063dc0 addr v1:172.21.15.181:6789/0 2021-05-25T12:00:37.771 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.734+0000 7fcfddb5b700 10 monclient(hunting): picked mon.noname-a con 0x7fcfd809a5e0 addr v1:172.21.15.102:6789/0 2021-05-25T12:00:37.771 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.734+0000 7fcfddb5b700 10 monclient(hunting): picked mon.noname-c con 0x7fcfd81325c0 addr v1:172.21.15.102:6790/0 2021-05-25T12:00:37.771 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.734+0000 7fcfddb5b700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfd812c2c0 con 0x7fcfd809a5e0 2021-05-25T12:00:37.771 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.734+0000 7fcfddb5b700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfd8116320 con 0x7fcfd8063dc0 2021-05-25T12:00:37.771 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.734+0000 7fcfddb5b700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfd81165c0 con 0x7fcfd81325c0 2021-05-25T12:00:37.772 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.734+0000 7fcfddb5b700 10 monclient(hunting): _renew_subs 2021-05-25T12:00:37.772 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.734+0000 7fcfddb5b700 10 monclient(hunting): authenticate will time out at 838.331089s 2021-05-25T12:00:37.772 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.734+0000 7fcfd7fff700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fcfd81325c0 0x7fcfd81433c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:54226/0 (socket says 172.21.15.102:54226) 2021-05-25T12:00:37.772 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.734+0000 7fcfd6ffd700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fcfd809a5e0 0x7fcfd813b4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:59346/0 (socket says 172.21.15.102:59346) 2021-05-25T12:00:37.773 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.734+0000 7fcfd7fff700 1 -- 172.21.15.102:0/2926588347 learned_addr learned my addr 172.21.15.102:0/2926588347 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:00:37.773 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.734+0000 7fcfc37fe700 10 client.?.objecter ms_handle_connect 0x7fcfd809a5e0 2021-05-25T12:00:37.773 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.734+0000 7fcfc37fe700 10 client.?.objecter resend_mon_ops 2021-05-25T12:00:37.773 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.734+0000 7fcfc37fe700 10 client.?.objecter ms_handle_connect 0x7fcfd81325c0 2021-05-25T12:00:37.773 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.734+0000 7fcfc37fe700 10 client.?.objecter resend_mon_ops 2021-05-25T12:00:37.774 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.734+0000 7fcfc37fe700 10 client.?.objecter ms_handle_connect 0x7fcfd8063dc0 2021-05-25T12:00:37.774 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.734+0000 7fcfc37fe700 10 client.?.objecter resend_mon_ops 2021-05-25T12:00:37.774 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.734+0000 7fcfc37fe700 1 -- 172.21.15.102:0/2926588347 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3812186565 0 0) 0x7fcfd812c2c0 con 0x7fcfd809a5e0 2021-05-25T12:00:37.774 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.734+0000 7fcfc37fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:37.774 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.734+0000 7fcfc37fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:37.775 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.734+0000 7fcfc37fe700 10 monclient(hunting): my global_id is 4479 2021-05-25T12:00:37.775 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.734+0000 7fcfc37fe700 1 -- 172.21.15.102:0/2926588347 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcfb8004c10 con 0x7fcfd809a5e0 2021-05-25T12:00:37.775 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.734+0000 7fcfc37fe700 1 -- 172.21.15.102:0/2926588347 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 211885535 0 0) 0x7fcfd81165c0 con 0x7fcfd81325c0 2021-05-25T12:00:37.775 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.734+0000 7fcfc37fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:37.775 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.734+0000 7fcfc37fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:37.776 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.734+0000 7fcfc37fe700 10 monclient(hunting): my global_id is 4463 2021-05-25T12:00:37.776 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.734+0000 7fcfc37fe700 1 -- 172.21.15.102:0/2926588347 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcfd812c2c0 con 0x7fcfd81325c0 2021-05-25T12:00:37.776 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.734+0000 7fcfc37fe700 1 -- 172.21.15.102:0/2926588347 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2089365396 0 0) 0x7fcfd8116320 con 0x7fcfd8063dc0 2021-05-25T12:00:37.776 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.734+0000 7fcfc37fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:37.777 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.734+0000 7fcfc37fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:37.777 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.734+0000 7fcfc37fe700 10 monclient(hunting): my global_id is 74113 2021-05-25T12:00:37.777 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.734+0000 7fcfc37fe700 1 -- 172.21.15.102:0/2926588347 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcfd81165c0 con 0x7fcfd8063dc0 2021-05-25T12:00:37.777 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.734+0000 7fcfc37fe700 1 -- 172.21.15.102:0/2926588347 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3893399527 0 0) 0x7fcfb8004c10 con 0x7fcfd809a5e0 2021-05-25T12:00:37.777 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.734+0000 7fcfc37fe700 1 -- 172.21.15.102:0/2926588347 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcfd8116320 con 0x7fcfd809a5e0 2021-05-25T12:00:37.778 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.734+0000 7fcfc37fe700 1 -- 172.21.15.102:0/2926588347 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fcfc80019d0 con 0x7fcfd809a5e0 2021-05-25T12:00:37.778 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.734+0000 7fcfc37fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:37.778 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.734+0000 7fcfc37fe700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:00:37.778 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.734+0000 7fcfc37fe700 10 monclient(hunting): dump: 2021-05-25T12:00:37.778 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:37.779 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:37.779 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:37.779 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:37.779 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:37.780 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:37.780 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:37.780 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:37.780 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:37.780 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:37.781 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.734+0000 7fcfc37fe700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:00:37.781 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.734+0000 7fcfc37fe700 1 -- 172.21.15.102:0/2926588347 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 128597530 0 0) 0x7fcfd812c2c0 con 0x7fcfd81325c0 2021-05-25T12:00:37.781 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.734+0000 7fcfc37fe700 1 -- 172.21.15.102:0/2926588347 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcfb8004c10 con 0x7fcfd81325c0 2021-05-25T12:00:37.781 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.734+0000 7fcfc37fe700 1 -- 172.21.15.102:0/2926588347 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2857609697 0 0) 0x7fcfd81165c0 con 0x7fcfd8063dc0 2021-05-25T12:00:37.781 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.734+0000 7fcfc37fe700 1 -- 172.21.15.102:0/2926588347 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcfd812c2c0 con 0x7fcfd8063dc0 2021-05-25T12:00:37.782 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.734+0000 7fcfc37fe700 1 -- 172.21.15.102:0/2926588347 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fcfcc001c80 con 0x7fcfd81325c0 2021-05-25T12:00:37.782 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.734+0000 7fcfc37fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:37.782 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.734+0000 7fcfc37fe700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2021-05-25T12:00:37.782 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.734+0000 7fcfc37fe700 10 monclient(hunting): dump: 2021-05-25T12:00:37.783 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:37.783 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:37.783 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:37.783 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:37.783 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:37.784 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:37.784 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:37.784 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:37.784 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:37.785 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:37.785 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.734+0000 7fcfc37fe700 1 -- 172.21.15.102:0/2926588347 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fcfc4001b00 con 0x7fcfd8063dc0 2021-05-25T12:00:37.785 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.734+0000 7fcfc37fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:37.785 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.734+0000 7fcfc37fe700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2021-05-25T12:00:37.785 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.734+0000 7fcfc37fe700 10 monclient(hunting): dump: 2021-05-25T12:00:37.786 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:37.786 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:37.786 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:37.786 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:37.786 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:37.787 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:37.787 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:37.787 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:37.787 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:37.788 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:37.788 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.734+0000 7fcfc37fe700 1 -- 172.21.15.102:0/2926588347 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 84788465 0 0) 0x7fcfd8116320 con 0x7fcfd809a5e0 2021-05-25T12:00:37.788 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.734+0000 7fcfc37fe700 1 -- 172.21.15.102:0/2926588347 >> v1:172.21.15.102:6790/0 conn(0x7fcfd81325c0 legacy=0x7fcfd81433c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:37.788 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.734+0000 7fcfc37fe700 1 -- 172.21.15.102:0/2926588347 >> v1:172.21.15.181:6789/0 conn(0x7fcfd8063dc0 legacy=0x7fcfd812bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:37.788 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.734+0000 7fcfc37fe700 10 monclient: _finish_hunting 0 2021-05-25T12:00:37.789 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.734+0000 7fcfc37fe700 1 monclient: found mon.a 2021-05-25T12:00:37.789 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.734+0000 7fcfc37fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:00:37.789 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.734+0000 7fcfc37fe700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:00:37.789 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.734+0000 7fcfc37fe700 1 -- 172.21.15.102:0/2926588347 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfd813bbc0 con 0x7fcfd809a5e0 2021-05-25T12:00:37.789 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.734+0000 7fcfc37fe700 10 monclient: _finish_auth 0 2021-05-25T12:00:37.790 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.734+0000 7fcfc37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:00:37.790 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.734+0000 7fcfddb5b700 5 monclient: authenticate success, global_id 4479 2021-05-25T12:00:37.790 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.734+0000 7fcfddb5b700 10 monclient: _renew_subs 2021-05-25T12:00:37.790 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.734+0000 7fcfddb5b700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:00:37.791 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.734+0000 7fcfddb5b700 1 -- 172.21.15.102:0/2926588347 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfd8094ac0 con 0x7fcfd809a5e0 2021-05-25T12:00:37.791 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.734+0000 7fcfddb5b700 10 client.4479.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:00:37.791 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.734+0000 7fcfddb5b700 10 monclient: _renew_subs 2021-05-25T12:00:37.791 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.734+0000 7fcfddb5b700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:00:37.791 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.734+0000 7fcfddb5b700 1 -- 172.21.15.102:0/2926588347 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfd8093420 con 0x7fcfd809a5e0 2021-05-25T12:00:37.792 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.734+0000 7fcfddb5b700 1 librados: init done 2021-05-25T12:00:37.792 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.735+0000 7fcfc37fe700 1 -- 172.21.15.102:0/2926588347 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcfc8004530 con 0x7fcfd809a5e0 2021-05-25T12:00:37.792 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.735+0000 7fcfc37fe700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:00:37.792 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.735+0000 7fcfc37fe700 1 -- 172.21.15.102:0/2926588347 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fcfc8004840 con 0x7fcfd809a5e0 2021-05-25T12:00:37.792 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.735+0000 7fcfc37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:37.793 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.735+0000 7fcfd5ffb700 4 set_mon_vals no callback set 2021-05-25T12:00:37.793 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.735+0000 7fcfc37fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:00:37.793 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.735+0000 7fcfc37fe700 10 monclient: dump: 2021-05-25T12:00:37.793 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:37.793 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:37.794 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:37.794 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:37.794 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:37.794 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:37.795 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:37.795 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:37.795 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:37.795 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:37.795 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.735+0000 7fcfc37fe700 1 -- 172.21.15.102:0/2926588347 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7fcfc800dbe0 con 0x7fcfd809a5e0 2021-05-25T12:00:37.796 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.735+0000 7fcfddb5b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:00:37.796 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.735+0000 7fcfddb5b700 1 -- 172.21.15.102:0/2926588347 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fcfb00065a0 con 0x7fcfb0002840 2021-05-25T12:00:37.796 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.735+0000 7fcfc37fe700 1 -- 172.21.15.102:0/2926588347 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(15..15 src has 1..15) v4 ==== 3549+0+0 (unknown 3948819498 0 0) 0x7fcfc803ccb0 con 0x7fcfd809a5e0 2021-05-25T12:00:37.796 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.735+0000 7fcfc37fe700 10 client.4479.objecter ms_dispatch 0x7fcfd8001550 osd_map(15..15 src has 1..15) v4 2021-05-25T12:00:37.796 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.735+0000 7fcfc37fe700 3 client.4479.objecter handle_osd_map got epochs [15,15] > 0 2021-05-25T12:00:37.797 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.735+0000 7fcfc37fe700 3 client.4479.objecter handle_osd_map decoding full epoch 15 2021-05-25T12:00:37.797 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.735+0000 7fcfc37fe700 20 client.4479.objecter dump_active .. 0 homeless 2021-05-25T12:00:37.797 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.735+0000 7fcfc37fe700 10 client.4479.objecter ms_handle_connect 0x7fcfb0002840 2021-05-25T12:00:37.797 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.735+0000 7fcfc37fe700 10 client.4479.objecter resend_mon_ops 2021-05-25T12:00:37.798 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.735+0000 7fcfd7fff700 10 monclient: get_auth_request con 0x7fcfb8033520 auth_method 0 2021-05-25T12:00:37.798 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.735+0000 7fcfc37fe700 1 -- 172.21.15.102:0/2926588347 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3431524407 0 0) 0x7fcfb00065a0 con 0x7fcfb0002840 2021-05-25T12:00:37.798 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.735+0000 7fcfc37fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:37.798 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.735+0000 7fcfc37fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:37.798 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.735+0000 7fcfc37fe700 10 monclient(hunting): my global_id is 74116 2021-05-25T12:00:37.799 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.735+0000 7fcfc37fe700 1 -- 172.21.15.102:0/2926588347 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcfd8116320 con 0x7fcfb0002840 2021-05-25T12:00:37.799 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.736+0000 7fcfc37fe700 10 client.4479.objecter ms_handle_connect 0x7fcfb8033520 2021-05-25T12:00:37.799 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.736+0000 7fcfc37fe700 1 -- 172.21.15.102:0/2926588347 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2391519680 0 0) 0x7fcfd8116320 con 0x7fcfb0002840 2021-05-25T12:00:37.799 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.736+0000 7fcfc37fe700 1 -- 172.21.15.102:0/2926588347 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcfb00065a0 con 0x7fcfb0002840 2021-05-25T12:00:37.799 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.736+0000 7fcfc37fe700 1 -- 172.21.15.102:0/2926588347 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fcfc4002620 con 0x7fcfb0002840 2021-05-25T12:00:37.800 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.736+0000 7fcfc37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:37.800 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.736+0000 7fcfc37fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T12:00:37.800 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.736+0000 7fcfc37fe700 10 monclient: dump: 2021-05-25T12:00:37.800 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:37.801 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:37.801 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:37.801 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:37.801 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:37.801 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:37.802 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:37.802 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:37.802 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:37.802 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:37.802 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.736+0000 7fcfc37fe700 1 -- 172.21.15.102:0/2926588347 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1637616963 0 0) 0x7fcfb00065a0 con 0x7fcfb0002840 2021-05-25T12:00:37.803 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.736+0000 7fcfc37fe700 1 -- 172.21.15.102:0/2926588347 --> v1:172.21.15.181:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcfb0009e40 con 0x7fcfb0002840 2021-05-25T12:00:37.803 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.736+0000 7fcfc37fe700 1 -- 172.21.15.102:0/2926588347 <== mon.1 v1:172.21.15.181:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7fcfc4002810 con 0x7fcfb0002840 2021-05-25T12:00:37.803 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.736+0000 7fcfc37fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:00:37.803 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.736+0000 7fcfc37fe700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:00:37.804 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.736+0000 7fcfc37fe700 1 -- 172.21.15.102:0/2926588347 >> v1:172.21.15.181:6789/0 conn(0x7fcfb0002840 legacy=0x7fcfb0004ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:37.804 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.736+0000 7fcfc37fe700 1 -- 172.21.15.102:0/2926588347 >> v1:172.21.15.181:6789/0 conn(0x7fcfb0002840 legacy=0x7fcfb0004ce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:00:37.804 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.743+0000 7fcfddb5b700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:00:37.804 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.743+0000 7fcfddb5b700 1 -- 172.21.15.102:0/2926588347 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fcfb0009ab0 con 0x7fcfb0008d70 2021-05-25T12:00:37.804 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.743+0000 7fcfc37fe700 10 client.4479.objecter ms_handle_connect 0x7fcfb0008d70 2021-05-25T12:00:37.805 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.743+0000 7fcfc37fe700 10 client.4479.objecter resend_mon_ops 2021-05-25T12:00:37.805 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.743+0000 7fcfc37fe700 1 -- 172.21.15.102:0/2926588347 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3867500930 0 0) 0x7fcfb0009ab0 con 0x7fcfb0008d70 2021-05-25T12:00:37.805 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.743+0000 7fcfc37fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:37.805 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.743+0000 7fcfc37fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:37.805 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.743+0000 7fcfc37fe700 10 monclient(hunting): my global_id is 74119 2021-05-25T12:00:37.806 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.743+0000 7fcfc37fe700 1 -- 172.21.15.102:0/2926588347 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcfb00065a0 con 0x7fcfb0008d70 2021-05-25T12:00:37.806 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.743+0000 7fcfc37fe700 1 -- 172.21.15.102:0/2926588347 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2424071615 0 0) 0x7fcfb00065a0 con 0x7fcfb0008d70 2021-05-25T12:00:37.806 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.743+0000 7fcfc37fe700 1 -- 172.21.15.102:0/2926588347 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcfb0009ab0 con 0x7fcfb0008d70 2021-05-25T12:00:37.806 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.743+0000 7fcfc37fe700 1 -- 172.21.15.102:0/2926588347 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fcfc4002e30 con 0x7fcfb0008d70 2021-05-25T12:00:37.806 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.743+0000 7fcfc37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:37.807 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.743+0000 7fcfc37fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T12:00:37.807 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.743+0000 7fcfc37fe700 10 monclient: dump: 2021-05-25T12:00:37.807 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:37.807 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:37.808 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:37.808 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:37.808 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:37.808 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:37.808 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:37.809 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:37.809 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:37.809 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:37.809 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.743+0000 7fcfc37fe700 1 -- 172.21.15.102:0/2926588347 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3651979245 0 0) 0x7fcfb0009ab0 con 0x7fcfb0008d70 2021-05-25T12:00:37.809 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.743+0000 7fcfc37fe700 1 -- 172.21.15.102:0/2926588347 --> v1:172.21.15.181:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fcfb0008740 con 0x7fcfb0008d70 2021-05-25T12:00:37.810 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.744+0000 7fcfc37fe700 1 -- 172.21.15.102:0/2926588347 <== mon.1 v1:172.21.15.181:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3770 (unknown 0 0 364835484) 0x7fcfc4003010 con 0x7fcfb0008d70 2021-05-25T12:00:37.810 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.744+0000 7fcfc37fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:00:37.810 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.744+0000 7fcfc37fe700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:00:37.810 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.744+0000 7fcfc37fe700 1 -- 172.21.15.102:0/2926588347 >> v1:172.21.15.181:6789/0 conn(0x7fcfb0008d70 legacy=0x7fcfb0009200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:37.810 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.744+0000 7fcfc37fe700 1 -- 172.21.15.102:0/2926588347 >> v1:172.21.15.181:6789/0 conn(0x7fcfb0008d70 legacy=0x7fcfb0009200 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:00:37.811 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:00:37.811 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:00:37.811 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:00:37.812 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "peon", 2021-05-25T12:00:37.812 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 14, 2021-05-25T12:00:37.812 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:00:37.812 INFO:teuthology.orchestra.run.smithi102.stdout: 0, 2021-05-25T12:00:37.812 INFO:teuthology.orchestra.run.smithi102.stdout: 1, 2021-05-25T12:00:37.813 INFO:teuthology.orchestra.run.smithi102.stdout: 2 2021-05-25T12:00:37.813 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:00:37.813 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 0, 2021-05-25T12:00:37.813 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:00:37.814 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:00:37.814 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:00:37.814 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:00:37.814 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:00:37.814 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:00:37.815 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:00:37.815 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:00:37.815 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:00:37.815 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:00:37.815 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:00:37.816 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:00:37.816 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:00:37.816 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:00:37.816 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:00:37.817 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:00:37.817 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:00:37.817 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:00:37.817 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:00:37.817 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:00:37.818 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:00:37.818 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:00:37.818 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:00:37.818 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:00:37.818 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:00:37.819 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:37.819 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:00:37.819 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:00:37.819 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:00:37.820 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:00:37.820 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:00:37.820 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:00:37.820 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:00:37.820 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:00:37.821 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:00:37.821 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:00:37.821 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:00:37.821 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:00:37.822 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:00:37.822 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:00:37.822 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:00:37.822 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:00:37.822 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:00:37.823 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:00:37.823 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:00:37.823 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:00:37.823 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:00:37.823 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:00:37.824 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:00:37.824 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:00:37.824 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:00:37.824 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:00:37.824 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:37.825 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:00:37.825 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:37.825 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:00:37.825 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:00:37.825 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:00:37.826 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:00:37.826 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:37.826 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:00:37.826 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:00:37.826 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:00:37.827 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:37.827 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:00:37.827 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:37.827 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:00:37.828 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:00:37.828 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:00:37.828 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:00:37.828 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:00:37.828 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:37.829 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:37.829 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:00:37.829 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:00:37.829 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:00:37.829 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:00:37.830 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:37.830 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:00:37.830 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:00:37.830 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:00:37.830 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:37.831 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:00:37.831 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:37.831 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:00:37.831 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:00:37.831 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:00:37.832 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:00:37.832 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:00:37.832 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:37.832 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:37.832 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:00:37.833 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:00:37.833 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:00:37.833 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:00:37.833 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:37.833 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:00:37.834 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:00:37.834 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:00:37.834 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:37.834 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:00:37.834 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:37.835 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:00:37.835 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:00:37.835 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:00:37.835 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:00:37.835 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:00:37.836 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:37.836 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:00:37.836 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:37.836 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:00:37.837 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:00:37.837 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:37.837 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:00:37.837 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:00:37.837 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:00:37.838 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:37.838 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:00:37.838 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:00:37.838 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:37.838 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:00:37.839 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:00:37.839 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:00:37.839 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:37.839 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:00:37.839 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:37.840 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:00:37.840 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:00:37.842 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.744+0000 7fcfddb5b700 10 librados: watch_flush enter 2021-05-25T12:00:37.842 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.744+0000 7fcfddb5b700 10 librados: watch_flush exit 2021-05-25T12:00:37.842 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.744+0000 7fcfddb5b700 20 client.4479.objecter shutdown clearing up homeless session... 2021-05-25T12:00:37.843 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.744+0000 7fcfddb5b700 10 client.4479.objecter successfully canceled tick 2021-05-25T12:00:37.843 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.744+0000 7fcfddb5b700 1 -- 172.21.15.102:0/2926588347 >> v1:172.21.15.181:6812/29582 conn(0x7fcfb8033520 legacy=0x7fcfb80359e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:37.843 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.744+0000 7fcfddb5b700 10 monclient: shutdown 2021-05-25T12:00:37.843 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.744+0000 7fcfddb5b700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:00:37.844 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.744+0000 7fcfddb5b700 1 -- 172.21.15.102:0/2926588347 >> v1:172.21.15.102:6789/0 conn(0x7fcfd809a5e0 legacy=0x7fcfd813b4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:37.844 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.744+0000 7fcfd7fff700 1 -- 172.21.15.102:0/2926588347 reap_dead start 2021-05-25T12:00:37.844 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.744+0000 7fcfd7fff700 1 -- 172.21.15.102:0/2926588347 reap_dead start 2021-05-25T12:00:37.844 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.744+0000 7fcfddb5b700 1 -- 172.21.15.102:0/2926588347 shutdown_connections 2021-05-25T12:00:37.844 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.744+0000 7fcfddb5b700 1 -- 172.21.15.102:0/2926588347 >> 172.21.15.102:0/2926588347 conn(0x7fcfd809bc30 msgr2=0x7fcfd809cb60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:00:37.845 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.744+0000 7fcfddb5b700 1 -- 172.21.15.102:0/2926588347 shutdown_connections 2021-05-25T12:00:37.845 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.744+0000 7fcfddb5b700 1 -- 172.21.15.102:0/2926588347 wait complete. 2021-05-25T12:00:37.845 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.744+0000 7fcfddb5b700 1 librados: shutdown 2021-05-25T12:00:37.846 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2021-05-25T12:00:37.846 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2021-05-25T12:00:37.955 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.953+0000 7f727b9ad700 10 monclient: get_monmap_and_config 2021-05-25T12:00:37.955 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.953+0000 7f727b9ad700 10 monclient: build_initial_monmap 2021-05-25T12:00:37.955 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.953+0000 7f727b9ad700 10 monclient: monmap: 2021-05-25T12:00:37.956 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:00:37.956 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:37.956 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:00:37.954807+0000 2021-05-25T12:00:37.956 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:00:37.954807+0000 2021-05-25T12:00:37.956 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:00:37.957 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:37.957 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:00:37.957 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:00:37.957 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:00:37.957 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:37.969 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.954+0000 7f727b9ad700 1 Processor -- start 2021-05-25T12:00:37.969 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.954+0000 7f727b9ad700 1 -- start start 2021-05-25T12:00:37.969 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.954+0000 7f727b9ad700 10 monclient: init 2021-05-25T12:00:37.969 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.955+0000 7f727b9ad700 10 monclient: _reopen_session rank -1 2021-05-25T12:00:37.970 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.955+0000 7f727b9ad700 10 monclient: _add_conns ranks=[1,2,0] 2021-05-25T12:00:37.970 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.955+0000 7f727b9ad700 10 monclient(hunting): picked mon.noname-c con 0x7f7274069190 addr v1:172.21.15.181:6789/0 2021-05-25T12:00:37.970 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.955+0000 7f727b9ad700 10 monclient(hunting): picked mon.noname-b con 0x7f7274069570 addr v1:172.21.15.102:6790/0 2021-05-25T12:00:37.970 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.955+0000 7f727b9ad700 10 monclient(hunting): picked mon.noname-a con 0x7f7274131e80 addr v1:172.21.15.102:6789/0 2021-05-25T12:00:37.971 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.955+0000 7f727b9ad700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72740917b0 con 0x7f7274131e80 2021-05-25T12:00:37.971 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.955+0000 7f727b9ad700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7274096bf0 con 0x7f7274069190 2021-05-25T12:00:37.971 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.955+0000 7f727b9ad700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72740910e0 con 0x7f7274069570 2021-05-25T12:00:37.971 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.955+0000 7f727b9ad700 10 monclient(hunting): _renew_subs 2021-05-25T12:00:37.971 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.955+0000 7f727b9ad700 10 monclient(hunting): authenticate will time out at 838.552215s 2021-05-25T12:00:37.972 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.955+0000 7f7279f4a700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f7274131e80 0x7f7274134270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:59360/0 (socket says 172.21.15.102:59360) 2021-05-25T12:00:37.972 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.955+0000 7f7278f48700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f7274069570 0x7f72740699f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:54236/0 (socket says 172.21.15.102:54236) 2021-05-25T12:00:37.972 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.955+0000 7f7279f4a700 1 -- 172.21.15.102:0/356894361 learned_addr learned my addr 172.21.15.102:0/356894361 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:00:37.972 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.955+0000 7f726bfff700 1 -- 172.21.15.102:0/356894361 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1306921645 0 0) 0x7f72740910e0 con 0x7f7274069570 2021-05-25T12:00:37.973 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.955+0000 7f726bfff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:37.973 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.955+0000 7f726bfff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:37.973 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.955+0000 7f726bfff700 10 monclient(hunting): my global_id is 4466 2021-05-25T12:00:37.973 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.955+0000 7f726bfff700 1 -- 172.21.15.102:0/356894361 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7260003270 con 0x7f7274069570 2021-05-25T12:00:37.973 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.955+0000 7f726bfff700 1 -- 172.21.15.102:0/356894361 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1031590535 0 0) 0x7f7270001490 con 0x7f7274131e80 2021-05-25T12:00:37.974 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.955+0000 7f726bfff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:37.974 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.955+0000 7f726bfff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:37.974 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.955+0000 7f726bfff700 10 monclient(hunting): my global_id is 4482 2021-05-25T12:00:37.974 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.955+0000 7f726bfff700 1 -- 172.21.15.102:0/356894361 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72740910e0 con 0x7f7274131e80 2021-05-25T12:00:37.975 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.955+0000 7f726bfff700 1 -- 172.21.15.102:0/356894361 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1975651055 0 0) 0x7f7274096bf0 con 0x7f7274069190 2021-05-25T12:00:37.975 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.955+0000 7f726bfff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:37.975 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.955+0000 7f726bfff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:37.975 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.955+0000 7f726bfff700 10 monclient(hunting): my global_id is 74122 2021-05-25T12:00:37.975 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.955+0000 7f726bfff700 1 -- 172.21.15.102:0/356894361 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7270001490 con 0x7f7274069190 2021-05-25T12:00:37.976 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.955+0000 7f726bfff700 1 -- 172.21.15.102:0/356894361 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2053700140 0 0) 0x7f7260003270 con 0x7f7274069570 2021-05-25T12:00:37.976 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.955+0000 7f726bfff700 1 -- 172.21.15.102:0/356894361 >> v1:172.21.15.181:6789/0 conn(0x7f7274069190 legacy=0x7f72740964e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:37.976 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.955+0000 7f726bfff700 1 -- 172.21.15.102:0/356894361 >> v1:172.21.15.102:6789/0 conn(0x7f7274131e80 legacy=0x7f7274134270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:37.976 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.955+0000 7f726bfff700 10 monclient: _finish_hunting 0 2021-05-25T12:00:37.977 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.955+0000 7f726bfff700 1 monclient: found mon.noname-b 2021-05-25T12:00:37.977 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.955+0000 7f726bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:00:37.977 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.955+0000 7f726bfff700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T12:00:37.977 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.955+0000 7f726bfff700 1 -- 172.21.15.102:0/356894361 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72740914a0 con 0x7f7274069570 2021-05-25T12:00:37.977 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.955+0000 7f726bfff700 10 monclient: _finish_auth 0 2021-05-25T12:00:37.978 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.955+0000 7f726bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:00:37.978 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.955+0000 7f726bfff700 1 -- 172.21.15.102:0/356894361 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f7264001580 con 0x7f7274069570 2021-05-25T12:00:37.978 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.955+0000 7f727b9ad700 5 monclient: authenticate success, global_id 4466 2021-05-25T12:00:37.978 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.955+0000 7f727b9ad700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:00:37.979 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.955+0000 7f726bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:37.979 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.955+0000 7f726bfff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:00:37.979 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.955+0000 7f726bfff700 10 monclient: dump: 2021-05-25T12:00:37.979 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:37.979 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:37.980 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:37.980 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:37.980 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:37.980 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:37.980 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:37.981 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:37.981 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:37.981 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:37.981 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.955+0000 7f727b9ad700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:00:37.981 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.955+0000 7f726bfff700 1 -- 172.21.15.102:0/356894361 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7264001810 con 0x7f7274069570 2021-05-25T12:00:37.982 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.955+0000 7f726bfff700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:00:37.982 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.955+0000 7f726bfff700 1 -- 172.21.15.102:0/356894361 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f7264001ba0 con 0x7f7274069570 2021-05-25T12:00:37.982 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.955+0000 7f727b9ad700 10 monclient: get_monmap_and_config success 2021-05-25T12:00:37.982 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.955+0000 7f727b9ad700 4 set_mon_vals no callback set 2021-05-25T12:00:37.983 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.955+0000 7f726bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:37.983 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.955+0000 7f726bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:00:37.983 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.955+0000 7f726bfff700 10 monclient: dump: 2021-05-25T12:00:37.983 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:37.984 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:37.984 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:37.984 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:37.984 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:37.984 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:37.985 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:37.985 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:37.985 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:37.985 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:37.985 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.956+0000 7f727b9ad700 10 monclient: shutdown 2021-05-25T12:00:37.986 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.956+0000 7f727b9ad700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:00:37.986 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.956+0000 7f727b9ad700 1 -- 172.21.15.102:0/356894361 >> v1:172.21.15.102:6790/0 conn(0x7f7274069570 legacy=0x7f72740699f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:37.986 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.956+0000 7f727b9ad700 1 -- 172.21.15.102:0/356894361 shutdown_connections 2021-05-25T12:00:37.986 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.956+0000 7f727b9ad700 1 -- 172.21.15.102:0/356894361 >> 172.21.15.102:0/356894361 conn(0x7f727409af90 msgr2=0x7f727409b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:00:37.987 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.956+0000 7f727b9ad700 1 -- 172.21.15.102:0/356894361 shutdown_connections 2021-05-25T12:00:37.987 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.956+0000 7f727b9ad700 1 -- 172.21.15.102:0/356894361 wait complete. 2021-05-25T12:00:37.987 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.956+0000 7f727b9ad700 10 monclient: build_initial_monmap 2021-05-25T12:00:37.987 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.956+0000 7f727b9ad700 10 monclient: monmap: 2021-05-25T12:00:37.988 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:00:37.988 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:00:37.988 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:00:37.988 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:00:37.988 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:00:37.989 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:00:37.989 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:00:37.989 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:00:37.989 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:00:37.989 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:37.990 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.957+0000 7f727b9ad700 1 librados: starting msgr at 2021-05-25T12:00:37.990 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.957+0000 7f727b9ad700 1 librados: starting objecter 2021-05-25T12:00:37.990 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.957+0000 7f727b9ad700 1 Processor -- start 2021-05-25T12:00:37.990 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.957+0000 7f727b9ad700 1 -- start start 2021-05-25T12:00:37.991 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.957+0000 7f727b9ad700 1 librados: setting wanted keys 2021-05-25T12:00:37.991 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.957+0000 7f727b9ad700 1 librados: calling monclient init 2021-05-25T12:00:37.991 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.957+0000 7f727b9ad700 10 monclient: init 2021-05-25T12:00:37.992 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.957+0000 7f727b9ad700 10 monclient: _reopen_session rank -1 2021-05-25T12:00:37.992 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.957+0000 7f727b9ad700 10 monclient: _add_conns ranks=[2,1,0] 2021-05-25T12:00:37.993 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.957+0000 7f727b9ad700 10 monclient(hunting): picked mon.noname-c con 0x7f7274069190 addr v1:172.21.15.102:6790/0 2021-05-25T12:00:37.993 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.957+0000 7f727b9ad700 10 monclient(hunting): picked mon.noname-b con 0x7f7274069570 addr v1:172.21.15.181:6789/0 2021-05-25T12:00:37.993 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.957+0000 7f727b9ad700 10 monclient(hunting): picked mon.noname-a con 0x7f7274131e80 addr v1:172.21.15.102:6789/0 2021-05-25T12:00:37.993 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.957+0000 7f727b9ad700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7274130690 con 0x7f7274131e80 2021-05-25T12:00:37.994 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.957+0000 7f727b9ad700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7274112090 con 0x7f7274069570 2021-05-25T12:00:37.994 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.957+0000 7f727b9ad700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7274111610 con 0x7f7274069190 2021-05-25T12:00:37.994 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.957+0000 7f727b9ad700 10 monclient(hunting): _renew_subs 2021-05-25T12:00:37.995 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.957+0000 7f727b9ad700 10 monclient(hunting): authenticate will time out at 838.554914s 2021-05-25T12:00:37.995 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.957+0000 7f7279749700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f7274069190 0x7f727413b160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:54240/0 (socket says 172.21.15.102:54240) 2021-05-25T12:00:37.995 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.957+0000 7f7279f4a700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f7274131e80 0x7f72741433d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:59366/0 (socket says 172.21.15.102:59366) 2021-05-25T12:00:37.995 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.957+0000 7f7279749700 1 -- 172.21.15.102:0/261531002 learned_addr learned my addr 172.21.15.102:0/261531002 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:00:37.996 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.958+0000 7f72697fa700 10 client.?.objecter ms_handle_connect 0x7f7274131e80 2021-05-25T12:00:37.996 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.958+0000 7f72697fa700 10 client.?.objecter resend_mon_ops 2021-05-25T12:00:37.996 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.958+0000 7f72697fa700 10 client.?.objecter ms_handle_connect 0x7f7274069190 2021-05-25T12:00:37.996 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.958+0000 7f72697fa700 10 client.?.objecter resend_mon_ops 2021-05-25T12:00:37.997 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.958+0000 7f72697fa700 10 client.?.objecter ms_handle_connect 0x7f7274069570 2021-05-25T12:00:37.997 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.958+0000 7f72697fa700 10 client.?.objecter resend_mon_ops 2021-05-25T12:00:37.997 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.958+0000 7f72697fa700 1 -- 172.21.15.102:0/261531002 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2878454116 0 0) 0x7f7274111610 con 0x7f7274069190 2021-05-25T12:00:37.997 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.958+0000 7f72697fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:37.998 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.958+0000 7f72697fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:37.998 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.958+0000 7f72697fa700 10 monclient(hunting): my global_id is 4469 2021-05-25T12:00:37.998 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.958+0000 7f72697fa700 1 -- 172.21.15.102:0/261531002 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7254004c10 con 0x7f7274069190 2021-05-25T12:00:37.998 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.958+0000 7f72697fa700 1 -- 172.21.15.102:0/261531002 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 813483807 0 0) 0x7f7274130690 con 0x7f7274131e80 2021-05-25T12:00:37.998 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.958+0000 7f72697fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:37.999 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.958+0000 7f72697fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:37.999 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.958+0000 7f72697fa700 10 monclient(hunting): my global_id is 4485 2021-05-25T12:00:37.999 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.958+0000 7f72697fa700 1 -- 172.21.15.102:0/261531002 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7274111610 con 0x7f7274131e80 2021-05-25T12:00:37.999 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.958+0000 7f72697fa700 1 -- 172.21.15.102:0/261531002 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1994482694 0 0) 0x7f7274112090 con 0x7f7274069570 2021-05-25T12:00:38.000 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.958+0000 7f72697fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:38.000 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.958+0000 7f72697fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:38.000 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.958+0000 7f72697fa700 10 monclient(hunting): my global_id is 74125 2021-05-25T12:00:38.000 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.958+0000 7f72697fa700 1 -- 172.21.15.102:0/261531002 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7274130690 con 0x7f7274069570 2021-05-25T12:00:38.001 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.958+0000 7f72697fa700 1 -- 172.21.15.102:0/261531002 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4245516332 0 0) 0x7f7254004c10 con 0x7f7274069190 2021-05-25T12:00:38.001 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.958+0000 7f72697fa700 1 -- 172.21.15.102:0/261531002 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7274112090 con 0x7f7274069190 2021-05-25T12:00:38.001 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.958+0000 7f72697fa700 1 -- 172.21.15.102:0/261531002 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f725c001980 con 0x7f7274069190 2021-05-25T12:00:38.001 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.958+0000 7f72697fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:38.002 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.958+0000 7f72697fa700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:00:38.002 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.958+0000 7f72697fa700 10 monclient(hunting): dump: 2021-05-25T12:00:38.002 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:38.002 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:38.003 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:38.003 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:38.003 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:38.003 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:38.004 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:38.004 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:38.004 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:38.004 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:38.005 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.958+0000 7f72697fa700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:00:38.005 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.958+0000 7f72697fa700 1 -- 172.21.15.102:0/261531002 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 854458003 0 0) 0x7f7274111610 con 0x7f7274131e80 2021-05-25T12:00:38.005 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.958+0000 7f72697fa700 1 -- 172.21.15.102:0/261531002 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7254004c10 con 0x7f7274131e80 2021-05-25T12:00:38.006 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.958+0000 7f72697fa700 1 -- 172.21.15.102:0/261531002 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f7270001d40 con 0x7f7274131e80 2021-05-25T12:00:38.006 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.958+0000 7f72697fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:38.006 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.958+0000 7f72697fa700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2021-05-25T12:00:38.006 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.958+0000 7f72697fa700 10 monclient(hunting): dump: 2021-05-25T12:00:38.006 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:38.007 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:38.007 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:38.007 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:38.007 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:38.008 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:38.008 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:38.008 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:38.008 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:38.009 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:38.009 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.958+0000 7f72697fa700 1 -- 172.21.15.102:0/261531002 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 257846780 0 0) 0x7f7274130690 con 0x7f7274069570 2021-05-25T12:00:38.009 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.958+0000 7f72697fa700 1 -- 172.21.15.102:0/261531002 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7274111610 con 0x7f7274069570 2021-05-25T12:00:38.009 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.958+0000 7f72697fa700 1 -- 172.21.15.102:0/261531002 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f7264001540 con 0x7f7274069570 2021-05-25T12:00:38.010 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.958+0000 7f72697fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:38.010 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.958+0000 7f72697fa700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2021-05-25T12:00:38.010 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.958+0000 7f72697fa700 10 monclient(hunting): dump: 2021-05-25T12:00:38.011 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:38.011 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:38.011 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:38.011 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:38.012 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:38.012 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:38.012 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:38.012 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:38.013 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:38.013 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:38.013 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.958+0000 7f72697fa700 1 -- 172.21.15.102:0/261531002 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3604049665 0 0) 0x7f7274112090 con 0x7f7274069190 2021-05-25T12:00:38.014 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.958+0000 7f72697fa700 1 -- 172.21.15.102:0/261531002 >> v1:172.21.15.181:6789/0 conn(0x7f7274069570 legacy=0x7f727413fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:38.014 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.958+0000 7f72697fa700 1 -- 172.21.15.102:0/261531002 >> v1:172.21.15.102:6789/0 conn(0x7f7274131e80 legacy=0x7f72741433d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:38.014 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.958+0000 7f72697fa700 10 monclient: _finish_hunting 0 2021-05-25T12:00:38.014 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.958+0000 7f72697fa700 1 monclient: found mon.c 2021-05-25T12:00:38.015 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.958+0000 7f72697fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:00:38.015 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.958+0000 7f72697fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:00:38.015 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.958+0000 7f72697fa700 1 -- 172.21.15.102:0/261531002 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f727413b870 con 0x7f7274069190 2021-05-25T12:00:38.015 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.958+0000 7f72697fa700 10 monclient: _finish_auth 0 2021-05-25T12:00:38.016 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.958+0000 7f72697fa700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:00:38.016 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.958+0000 7f727b9ad700 5 monclient: authenticate success, global_id 4469 2021-05-25T12:00:38.016 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.958+0000 7f727b9ad700 10 monclient: _renew_subs 2021-05-25T12:00:38.016 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.958+0000 7f727b9ad700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:00:38.017 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.958+0000 7f727b9ad700 1 -- 172.21.15.102:0/261531002 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7274096e80 con 0x7f7274069190 2021-05-25T12:00:38.017 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.958+0000 7f727b9ad700 10 client.4469.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:00:38.017 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.958+0000 7f727b9ad700 10 monclient: _renew_subs 2021-05-25T12:00:38.017 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.958+0000 7f727b9ad700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:00:38.018 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.958+0000 7f727b9ad700 1 -- 172.21.15.102:0/261531002 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7274131090 con 0x7f7274069190 2021-05-25T12:00:38.018 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.958+0000 7f727b9ad700 1 librados: init done 2021-05-25T12:00:38.018 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.958+0000 7f72697fa700 1 -- 172.21.15.102:0/261531002 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f725c002d00 con 0x7f7274069190 2021-05-25T12:00:38.019 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.958+0000 7f72697fa700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:00:38.019 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.958+0000 7f72697fa700 1 -- 172.21.15.102:0/261531002 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f725c002100 con 0x7f7274069190 2021-05-25T12:00:38.019 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.958+0000 7f72697fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:38.019 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.958+0000 7f726b7fe700 4 set_mon_vals no callback set 2021-05-25T12:00:38.020 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.958+0000 7f72697fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:00:38.020 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.958+0000 7f72697fa700 10 monclient: dump: 2021-05-25T12:00:38.020 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:38.020 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:38.020 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:38.021 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:38.021 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:38.021 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:38.021 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:38.022 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:38.022 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:38.022 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:38.022 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.959+0000 7f72697fa700 1 -- 172.21.15.102:0/261531002 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f725c002260 con 0x7f7274069190 2021-05-25T12:00:38.023 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.959+0000 7f727b9ad700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:00:38.023 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.959+0000 7f727b9ad700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:00:38.023 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.959+0000 7f727b9ad700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:00:38.023 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.959+0000 7f727b9ad700 1 -- 172.21.15.102:0/261531002 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f724c004db0 con 0x7f7274069190 2021-05-25T12:00:38.024 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.959+0000 7f72697fa700 1 -- 172.21.15.102:0/261531002 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(15..15 src has 1..15) v4 ==== 3549+0+0 (unknown 3948819498 0 0) 0x7f725c03cde0 con 0x7f7274069190 2021-05-25T12:00:38.024 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.959+0000 7f72697fa700 10 client.4469.objecter ms_dispatch 0x7f7274001550 osd_map(15..15 src has 1..15) v4 2021-05-25T12:00:38.024 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.959+0000 7f72697fa700 3 client.4469.objecter handle_osd_map got epochs [15,15] > 0 2021-05-25T12:00:38.024 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.959+0000 7f72697fa700 3 client.4469.objecter handle_osd_map decoding full epoch 15 2021-05-25T12:00:38.025 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.959+0000 7f72697fa700 20 client.4469.objecter dump_active .. 0 homeless 2021-05-25T12:00:38.025 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.959+0000 7f7278f48700 10 monclient: get_auth_request con 0x7f72540335a0 auth_method 0 2021-05-25T12:00:38.025 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.960+0000 7f72697fa700 10 client.4469.objecter ms_handle_connect 0x7f72540335a0 2021-05-25T12:00:38.025 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.963+0000 7f72697fa700 1 -- 172.21.15.102:0/261531002 <== mon.2 v1:172.21.15.102:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7f725c012cd0 con 0x7f7274069190 2021-05-25T12:00:38.025 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.963+0000 7f72697fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:00:38.026 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:37.963+0000 7f72697fa700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:00:38.172 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:38.171+0000 7f727b9ad700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2021-05-25T12:00:38.173 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:38.171+0000 7f727b9ad700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2021-05-25T12:00:38.173 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:38.171+0000 7f727b9ad700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:00:38.173 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:38.171+0000 7f727b9ad700 1 -- 172.21.15.102:0/261531002 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f724c005870 con 0x7f7274069190 2021-05-25T12:00:38.174 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:38.172+0000 7f72697fa700 1 -- 172.21.15.102:0/261531002 <== mon.2 v1:172.21.15.102:6790/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7f725c0128c0 con 0x7f7274069190 2021-05-25T12:00:38.174 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:38.172+0000 7f72697fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2021-05-25T12:00:38.174 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:38.172+0000 7f72697fa700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:00:38.176 INFO:tasks.ceph.mon.a.smithi102.stderr:2021-05-25T12:00:38.172+0000 7fcad34e7700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:00:38.176 INFO:tasks.ceph.mon.a.smithi102.stderr:2021-05-25T12:00:38.172+0000 7fcad34e7700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=12,config=2,health=12,kv=6,logm=66,mdsmap=2} crc {auth=49601416,config=3140514009,health=2319137676,kv=1184736763,logm=1736605113,mdsmap=3585761405}) 2021-05-25T12:00:38.176 INFO:tasks.ceph.mon.a.smithi102.stderr:2021-05-25T12:00:38.173+0000 7fcad34e7700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=13,config=2,health=11,kv=6,logm=63,mdsmap=3,mgr=2} crc {auth=2655164091,config=3140514009,health=3490199527,kv=1184736763,logm=228350975,mdsmap=1531151339,mgr=983774903}) 2021-05-25T12:00:38.177 INFO:tasks.ceph.mon.a.smithi102.stderr:2021-05-25T12:00:38.173+0000 7fcad34e7700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:00:38.177 INFO:tasks.ceph.mon.a.smithi102.stderr:2021-05-25T12:00:38.173+0000 7fcad34e7700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=12,config=2,health=12,kv=6,logm=66,mdsmap=2} crc {auth=49601416,config=3140514009,health=2319137676,kv=1184736763,logm=1736605113,mdsmap=3585761405}) 2021-05-25T12:00:38.177 INFO:tasks.ceph.mon.a.smithi102.stderr:2021-05-25T12:00:38.173+0000 7fcad34e7700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=14,config=1,health=12,kv=5,logm=65,mdsmap=2,mgr=1} crc {auth=2344358682,config=1228700967,health=2319137676,kv=3385806472,logm=4243293233,mdsmap=3140514009,mgr=1065898531}) 2021-05-25T12:00:38.178 INFO:tasks.ceph.mon.a.smithi102.stderr:2021-05-25T12:00:38.174+0000 7fcad34e7700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:00:38.178 INFO:tasks.ceph.mon.a.smithi102.stderr:2021-05-25T12:00:38.174+0000 7fcad34e7700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mdsmap=1,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=4,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=32} crc {mdsmap=1228700967,mgr=2178563563,mgr_command_descs=3824783157,mgr_metadata=1267308364,mgrstat=1167288616,mon_config_key=4118394475,monmap=3809614708,osd_metadata=725478305,osd_pg_creating=534015559,osdmap=3329101012}) 2021-05-25T12:00:38.178 INFO:tasks.ceph.mon.a.smithi102.stderr:2021-05-25T12:00:38.174+0000 7fcad34e7700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mdsmap=1,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=4,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=32} crc {mdsmap=1228700967,mgr=2996579194,mgr_command_descs=3824783157,mgr_metadata=1267308364,mgrstat=1167288616,mon_config_key=4118394475,monmap=3809614708,osd_metadata=3667066319,osd_pg_creating=534015559,osdmap=3197078820}) 2021-05-25T12:00:38.179 INFO:tasks.ceph.mon.a.smithi102.stderr:2021-05-25T12:00:38.174+0000 7fcad34e7700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:00:38.179 INFO:tasks.ceph.mon.a.smithi102.stderr:2021-05-25T12:00:38.174+0000 7fcad34e7700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mdsmap=1,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=4,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=32} crc {mdsmap=1228700967,mgr=2178563563,mgr_command_descs=3824783157,mgr_metadata=1267308364,mgrstat=1167288616,mon_config_key=4118394475,monmap=3809614708,osd_metadata=725478305,osd_pg_creating=534015559,osdmap=3329101012}) 2021-05-25T12:00:38.179 INFO:tasks.ceph.mon.a.smithi102.stderr:2021-05-25T12:00:38.174+0000 7fcad34e7700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=4,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=32} crc {mgr=2996579194,mgr_command_descs=3824783157,mgr_metadata=1267308364,mgrstat=3405982556,mon_config_key=4118394475,monmap=3809614708,osd_metadata=725478305,osd_pg_creating=534015559,osdmap=4144276082}) 2021-05-25T12:00:38.181 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:38.174+0000 7f727b9ad700 10 librados: watch_flush enter 2021-05-25T12:00:38.181 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:38.174+0000 7f727b9ad700 10 librados: watch_flush exit 2021-05-25T12:00:38.181 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:38.174+0000 7f727b9ad700 20 client.4469.objecter shutdown clearing up homeless session... 2021-05-25T12:00:38.182 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:38.174+0000 7f727b9ad700 10 client.4469.objecter successfully canceled tick 2021-05-25T12:00:38.182 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:38.174+0000 7f727b9ad700 1 -- 172.21.15.102:0/261531002 >> v1:172.21.15.181:6812/29582 conn(0x7f72540335a0 legacy=0x7f7254035a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:38.182 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:38.174+0000 7f727b9ad700 10 monclient: shutdown 2021-05-25T12:00:38.182 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:38.174+0000 7f727b9ad700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:00:38.182 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:38.174+0000 7f727b9ad700 1 -- 172.21.15.102:0/261531002 >> v1:172.21.15.102:6790/0 conn(0x7f7274069190 legacy=0x7f727413b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:38.183 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:38.174+0000 7f727b9ad700 1 -- 172.21.15.102:0/261531002 shutdown_connections 2021-05-25T12:00:38.183 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:38.174+0000 7f727b9ad700 1 -- 172.21.15.102:0/261531002 >> 172.21.15.102:0/261531002 conn(0x7f727409af90 msgr2=0x7f7274094850 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:00:38.183 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:38.174+0000 7f727b9ad700 1 -- 172.21.15.102:0/261531002 shutdown_connections 2021-05-25T12:00:38.183 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:38.174+0000 7f727b9ad700 1 -- 172.21.15.102:0/261531002 wait complete. 2021-05-25T12:00:38.184 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:38.174+0000 7f727b9ad700 1 librados: shutdown 2021-05-25T12:00:38.187 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2021-05-25T12:00:38.691 INFO:tasks.workunit.client.0.smithi181.stderr:Note: switching to 'a16ab1caacd8b41a7fe74bc40a10ec74ee8f5802'. 2021-05-25T12:00:38.691 INFO:tasks.workunit.client.0.smithi181.stderr: 2021-05-25T12:00:38.692 INFO:tasks.workunit.client.0.smithi181.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2021-05-25T12:00:38.692 INFO:tasks.workunit.client.0.smithi181.stderr:changes and commit them, and you can discard any commits you make in this 2021-05-25T12:00:38.692 INFO:tasks.workunit.client.0.smithi181.stderr:state without impacting any branches by switching back to a branch. 2021-05-25T12:00:38.693 INFO:tasks.workunit.client.0.smithi181.stderr: 2021-05-25T12:00:38.693 INFO:tasks.workunit.client.0.smithi181.stderr:If you want to create a new branch to retain commits you create, you may 2021-05-25T12:00:38.693 INFO:tasks.workunit.client.0.smithi181.stderr:do so (now or later) by using -c with the switch command. Example: 2021-05-25T12:00:38.693 INFO:tasks.workunit.client.0.smithi181.stderr: 2021-05-25T12:00:38.694 INFO:tasks.workunit.client.0.smithi181.stderr: git switch -c 2021-05-25T12:00:38.694 INFO:tasks.workunit.client.0.smithi181.stderr: 2021-05-25T12:00:38.694 INFO:tasks.workunit.client.0.smithi181.stderr:Or undo this operation with: 2021-05-25T12:00:38.695 INFO:tasks.workunit.client.0.smithi181.stderr: 2021-05-25T12:00:38.695 INFO:tasks.workunit.client.0.smithi181.stderr: git switch - 2021-05-25T12:00:38.695 INFO:tasks.workunit.client.0.smithi181.stderr: 2021-05-25T12:00:38.695 INFO:tasks.workunit.client.0.smithi181.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2021-05-25T12:00:38.696 INFO:tasks.workunit.client.0.smithi181.stderr: 2021-05-25T12:00:38.696 INFO:tasks.workunit.client.0.smithi181.stderr:HEAD is now at a16ab1caacd Merge branch 'wip-cmake-libasan.6' of https://github.com/tchaikov/ceph into wip-kefu-testing-2021-05-25-1039 2021-05-25T12:00:38.697 DEBUG:teuthology.orchestra.run.smithi181:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2021-05-25T12:00:38.768 INFO:tasks.workunit.client.0.smithi181.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2021-05-25T12:00:38.770 INFO:tasks.workunit.client.0.smithi181.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2021-05-25T12:00:38.770 INFO:tasks.workunit.client.0.smithi181.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2021-05-25T12:00:39.188 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2021-05-25T12:00:39.189 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2021-05-25T12:00:39.318 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.305+0000 7fdbbf7d0700 10 monclient: get_monmap_and_config 2021-05-25T12:00:39.318 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.305+0000 7fdbbf7d0700 10 monclient: build_initial_monmap 2021-05-25T12:00:39.318 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.305+0000 7fdbbf7d0700 10 monclient: monmap: 2021-05-25T12:00:39.319 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:00:39.319 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:39.319 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:00:39.306196+0000 2021-05-25T12:00:39.319 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:00:39.306196+0000 2021-05-25T12:00:39.320 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:00:39.320 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:39.320 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:00:39.320 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:00:39.320 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:00:39.321 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:39.321 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.305+0000 7fdbbf7d0700 1 Processor -- start 2021-05-25T12:00:39.321 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.305+0000 7fdbbf7d0700 1 -- start start 2021-05-25T12:00:39.322 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.305+0000 7fdbbf7d0700 10 monclient: init 2021-05-25T12:00:39.322 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.306+0000 7fdbbf7d0700 10 monclient: _reopen_session rank -1 2021-05-25T12:00:39.322 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.306+0000 7fdbbf7d0700 10 monclient: _add_conns ranks=[1,0,2] 2021-05-25T12:00:39.323 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.306+0000 7fdbbf7d0700 10 monclient(hunting): picked mon.noname-c con 0x7fdbb8090990 addr v1:172.21.15.181:6789/0 2021-05-25T12:00:39.323 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.306+0000 7fdbbf7d0700 10 monclient(hunting): picked mon.noname-a con 0x7fdbb8090d70 addr v1:172.21.15.102:6789/0 2021-05-25T12:00:39.323 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.306+0000 7fdbbf7d0700 10 monclient(hunting): picked mon.noname-b con 0x7fdbb812e330 addr v1:172.21.15.102:6790/0 2021-05-25T12:00:39.323 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.306+0000 7fdbbf7d0700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbb8096fa0 con 0x7fdbb8090d70 2021-05-25T12:00:39.324 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.306+0000 7fdbbf7d0700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbb80926a0 con 0x7fdbb8090990 2021-05-25T12:00:39.324 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.306+0000 7fdbbf7d0700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbb8092940 con 0x7fdbb812e330 2021-05-25T12:00:39.324 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.306+0000 7fdbbf7d0700 10 monclient(hunting): _renew_subs 2021-05-25T12:00:39.324 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.306+0000 7fdbbf7d0700 10 monclient(hunting): authenticate will time out at 839.903476s 2021-05-25T12:00:39.324 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.306+0000 7fdbbdd6d700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fdbb812e330 0x7fdbb8130720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:54248/0 (socket says 172.21.15.102:54248) 2021-05-25T12:00:39.325 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.306+0000 7fdbbcd6b700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fdbb8090d70 0x7fdbb8091200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:59370/0 (socket says 172.21.15.102:59370) 2021-05-25T12:00:39.325 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.306+0000 7fdbbdd6d700 1 -- 172.21.15.102:0/2529763540 learned_addr learned my addr 172.21.15.102:0/2529763540 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:00:39.325 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.306+0000 7fdba7fff700 1 -- 172.21.15.102:0/2529763540 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 51444802 0 0) 0x7fdbb8096fa0 con 0x7fdbb8090d70 2021-05-25T12:00:39.326 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.306+0000 7fdba7fff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:39.326 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.306+0000 7fdba7fff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:39.326 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.306+0000 7fdba7fff700 10 monclient(hunting): my global_id is 4488 2021-05-25T12:00:39.326 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.306+0000 7fdba7fff700 1 -- 172.21.15.102:0/2529763540 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdba0004380 con 0x7fdbb8090d70 2021-05-25T12:00:39.327 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.306+0000 7fdba7fff700 1 -- 172.21.15.102:0/2529763540 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 868377708 0 0) 0x7fdbb8092940 con 0x7fdbb812e330 2021-05-25T12:00:39.327 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.306+0000 7fdba7fff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:39.328 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.306+0000 7fdba7fff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:39.328 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.306+0000 7fdba7fff700 10 monclient(hunting): my global_id is 4472 2021-05-25T12:00:39.328 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.306+0000 7fdba7fff700 1 -- 172.21.15.102:0/2529763540 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbb8096fa0 con 0x7fdbb812e330 2021-05-25T12:00:39.328 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.307+0000 7fdba7fff700 1 -- 172.21.15.102:0/2529763540 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 799716033 0 0) 0x7fdbb80926a0 con 0x7fdbb8090990 2021-05-25T12:00:39.329 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.307+0000 7fdba7fff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:39.329 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.307+0000 7fdba7fff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:39.329 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.307+0000 7fdba7fff700 10 monclient(hunting): my global_id is 74128 2021-05-25T12:00:39.329 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.307+0000 7fdba7fff700 1 -- 172.21.15.102:0/2529763540 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbb8092940 con 0x7fdbb8090990 2021-05-25T12:00:39.329 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.307+0000 7fdba7fff700 1 -- 172.21.15.102:0/2529763540 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1980018218 0 0) 0x7fdba0004380 con 0x7fdbb8090d70 2021-05-25T12:00:39.330 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.307+0000 7fdba7fff700 1 -- 172.21.15.102:0/2529763540 >> v1:172.21.15.102:6790/0 conn(0x7fdbb812e330 legacy=0x7fdbb8130720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:39.330 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.307+0000 7fdba7fff700 1 -- 172.21.15.102:0/2529763540 >> v1:172.21.15.181:6789/0 conn(0x7fdbb8090990 legacy=0x7fdbb8093240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:39.330 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.307+0000 7fdba7fff700 10 monclient: _finish_hunting 0 2021-05-25T12:00:39.330 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.307+0000 7fdba7fff700 1 monclient: found mon.noname-a 2021-05-25T12:00:39.332 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.307+0000 7fdba7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:00:39.332 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.307+0000 7fdba7fff700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T12:00:39.333 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.307+0000 7fdba7fff700 1 -- 172.21.15.102:0/2529763540 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbb8092be0 con 0x7fdbb8090d70 2021-05-25T12:00:39.333 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.307+0000 7fdba7fff700 10 monclient: _finish_auth 0 2021-05-25T12:00:39.333 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.307+0000 7fdba7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:00:39.333 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.307+0000 7fdbbf7d0700 5 monclient: authenticate success, global_id 4488 2021-05-25T12:00:39.333 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.307+0000 7fdbbf7d0700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:00:39.334 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.307+0000 7fdba7fff700 1 -- 172.21.15.102:0/2529763540 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fdbac001610 con 0x7fdbb8090d70 2021-05-25T12:00:39.334 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.307+0000 7fdba7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:39.334 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.307+0000 7fdba7fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:00:39.334 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.307+0000 7fdba7fff700 10 monclient: dump: 2021-05-25T12:00:39.335 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:39.335 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:39.335 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:39.335 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:39.335 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:39.336 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:39.336 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:39.336 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:39.336 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:39.336 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:39.337 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.307+0000 7fdbbf7d0700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:00:39.337 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.307+0000 7fdba7fff700 1 -- 172.21.15.102:0/2529763540 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdbac001880 con 0x7fdbb8090d70 2021-05-25T12:00:39.337 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.307+0000 7fdba7fff700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:00:39.338 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.307+0000 7fdbbf7d0700 10 monclient: get_monmap_and_config success 2021-05-25T12:00:39.338 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.307+0000 7fdba7fff700 1 -- 172.21.15.102:0/2529763540 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fdbac001c10 con 0x7fdbb8090d70 2021-05-25T12:00:39.338 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.307+0000 7fdba7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:39.339 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.307+0000 7fdbbf7d0700 4 set_mon_vals no callback set 2021-05-25T12:00:39.339 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.307+0000 7fdba7fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:00:39.339 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.307+0000 7fdba7fff700 10 monclient: dump: 2021-05-25T12:00:39.339 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:39.340 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:39.340 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:39.340 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:39.340 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:39.340 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:39.341 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:39.341 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:39.341 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:39.341 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:39.341 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.307+0000 7fdbbf7d0700 10 monclient: shutdown 2021-05-25T12:00:39.342 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.307+0000 7fdbbf7d0700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:00:39.342 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.307+0000 7fdbbf7d0700 1 -- 172.21.15.102:0/2529763540 >> v1:172.21.15.102:6789/0 conn(0x7fdbb8090d70 legacy=0x7fdbb8091200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:39.342 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.307+0000 7fdbbf7d0700 1 -- 172.21.15.102:0/2529763540 shutdown_connections 2021-05-25T12:00:39.342 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.307+0000 7fdbbf7d0700 1 -- 172.21.15.102:0/2529763540 >> 172.21.15.102:0/2529763540 conn(0x7fdbb809b370 msgr2=0x7fdbb8097430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:00:39.343 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.307+0000 7fdbbf7d0700 1 -- 172.21.15.102:0/2529763540 shutdown_connections 2021-05-25T12:00:39.343 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.307+0000 7fdbbf7d0700 1 -- 172.21.15.102:0/2529763540 wait complete. 2021-05-25T12:00:39.343 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.308+0000 7fdbbf7d0700 10 monclient: build_initial_monmap 2021-05-25T12:00:39.343 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.308+0000 7fdbbf7d0700 10 monclient: monmap: 2021-05-25T12:00:39.343 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:00:39.344 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:00:39.344 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:00:39.344 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:00:39.344 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:00:39.345 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:00:39.345 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:00:39.345 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:00:39.345 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:00:39.345 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:39.346 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.308+0000 7fdbbf7d0700 1 librados: starting msgr at 2021-05-25T12:00:39.346 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.308+0000 7fdbbf7d0700 1 librados: starting objecter 2021-05-25T12:00:39.346 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.308+0000 7fdbbf7d0700 1 Processor -- start 2021-05-25T12:00:39.346 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.308+0000 7fdbbf7d0700 1 -- start start 2021-05-25T12:00:39.346 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.308+0000 7fdbbf7d0700 1 librados: setting wanted keys 2021-05-25T12:00:39.347 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.308+0000 7fdbbf7d0700 1 librados: calling monclient init 2021-05-25T12:00:39.347 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.308+0000 7fdbbf7d0700 10 monclient: init 2021-05-25T12:00:39.347 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.309+0000 7fdbbf7d0700 10 monclient: _reopen_session rank -1 2021-05-25T12:00:39.347 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.309+0000 7fdbbf7d0700 10 monclient: _add_conns ranks=[1,0,2] 2021-05-25T12:00:39.348 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.309+0000 7fdbbf7d0700 10 monclient(hunting): picked mon.noname-b con 0x7fdbb8090990 addr v1:172.21.15.181:6789/0 2021-05-25T12:00:39.348 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.309+0000 7fdbbf7d0700 10 monclient(hunting): picked mon.noname-a con 0x7fdbb8090d70 addr v1:172.21.15.102:6789/0 2021-05-25T12:00:39.348 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.309+0000 7fdbbf7d0700 10 monclient(hunting): picked mon.noname-c con 0x7fdbb812e330 addr v1:172.21.15.102:6790/0 2021-05-25T12:00:39.348 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.309+0000 7fdbbf7d0700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbb8091fe0 con 0x7fdbb8090d70 2021-05-25T12:00:39.348 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.309+0000 7fdbbf7d0700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbb8114a00 con 0x7fdbb8090990 2021-05-25T12:00:39.349 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.309+0000 7fdbbf7d0700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbb8114ca0 con 0x7fdbb812e330 2021-05-25T12:00:39.349 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.309+0000 7fdbbf7d0700 10 monclient(hunting): _renew_subs 2021-05-25T12:00:39.349 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.309+0000 7fdbbf7d0700 10 monclient(hunting): authenticate will time out at 839.906223s 2021-05-25T12:00:39.349 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.309+0000 7fdbbcd6b700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fdbb8090d70 0x7fdbb8138960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:59378/0 (socket says 172.21.15.102:59378) 2021-05-25T12:00:39.349 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.309+0000 7fdbbcd6b700 1 -- 172.21.15.102:0/922796431 learned_addr learned my addr 172.21.15.102:0/922796431 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:00:39.350 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.309+0000 7fdbbdd6d700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fdbb812e330 0x7fdbb8140cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:54254/0 (socket says 172.21.15.102:54254) 2021-05-25T12:00:39.350 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.309+0000 7fdba57fa700 10 client.?.objecter ms_handle_connect 0x7fdbb8090d70 2021-05-25T12:00:39.350 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.309+0000 7fdba57fa700 10 client.?.objecter resend_mon_ops 2021-05-25T12:00:39.350 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.309+0000 7fdba57fa700 10 client.?.objecter ms_handle_connect 0x7fdbb812e330 2021-05-25T12:00:39.351 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.309+0000 7fdba57fa700 10 client.?.objecter resend_mon_ops 2021-05-25T12:00:39.351 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.309+0000 7fdba57fa700 10 client.?.objecter ms_handle_connect 0x7fdbb8090990 2021-05-25T12:00:39.351 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.309+0000 7fdba57fa700 10 client.?.objecter resend_mon_ops 2021-05-25T12:00:39.351 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.309+0000 7fdba57fa700 1 -- 172.21.15.102:0/922796431 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1896310739 0 0) 0x7fdbb8091fe0 con 0x7fdbb8090d70 2021-05-25T12:00:39.351 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.309+0000 7fdba57fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:39.352 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.309+0000 7fdba57fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:39.352 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.309+0000 7fdba57fa700 10 monclient(hunting): my global_id is 4491 2021-05-25T12:00:39.352 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.309+0000 7fdba57fa700 1 -- 172.21.15.102:0/922796431 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb98004c10 con 0x7fdbb8090d70 2021-05-25T12:00:39.352 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.309+0000 7fdba57fa700 1 -- 172.21.15.102:0/922796431 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 100636922 0 0) 0x7fdbb8114ca0 con 0x7fdbb812e330 2021-05-25T12:00:39.353 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.309+0000 7fdba57fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:39.353 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.309+0000 7fdba57fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:39.353 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.309+0000 7fdba57fa700 10 monclient(hunting): my global_id is 4475 2021-05-25T12:00:39.353 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.309+0000 7fdba57fa700 1 -- 172.21.15.102:0/922796431 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbb8091fe0 con 0x7fdbb812e330 2021-05-25T12:00:39.353 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.309+0000 7fdba57fa700 1 -- 172.21.15.102:0/922796431 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 174413816 0 0) 0x7fdbb8114a00 con 0x7fdbb8090990 2021-05-25T12:00:39.354 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.309+0000 7fdba57fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:39.354 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.309+0000 7fdba57fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:39.354 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.309+0000 7fdba57fa700 10 monclient(hunting): my global_id is 74131 2021-05-25T12:00:39.354 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.309+0000 7fdba57fa700 1 -- 172.21.15.102:0/922796431 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbb8114ca0 con 0x7fdbb8090990 2021-05-25T12:00:39.355 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.309+0000 7fdba57fa700 1 -- 172.21.15.102:0/922796431 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3071846926 0 0) 0x7fdb98004c10 con 0x7fdbb8090d70 2021-05-25T12:00:39.355 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.309+0000 7fdba57fa700 1 -- 172.21.15.102:0/922796431 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdbb8114a00 con 0x7fdbb8090d70 2021-05-25T12:00:39.355 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.309+0000 7fdba57fa700 1 -- 172.21.15.102:0/922796431 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fdbac001610 con 0x7fdbb8090d70 2021-05-25T12:00:39.355 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.309+0000 7fdba57fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:39.355 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.309+0000 7fdba57fa700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:00:39.356 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.309+0000 7fdba57fa700 10 monclient(hunting): dump: 2021-05-25T12:00:39.356 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:39.356 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:39.356 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:39.357 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:39.357 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:39.357 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:39.357 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:39.357 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:39.357 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:39.358 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:39.358 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.309+0000 7fdba57fa700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:00:39.358 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.309+0000 7fdba57fa700 1 -- 172.21.15.102:0/922796431 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3194921892 0 0) 0x7fdbb8091fe0 con 0x7fdbb812e330 2021-05-25T12:00:39.358 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.309+0000 7fdba57fa700 1 -- 172.21.15.102:0/922796431 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdb98004c10 con 0x7fdbb812e330 2021-05-25T12:00:39.359 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.309+0000 7fdba57fa700 1 -- 172.21.15.102:0/922796431 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fdbb4001bb0 con 0x7fdbb812e330 2021-05-25T12:00:39.359 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.309+0000 7fdba57fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:39.359 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.309+0000 7fdba57fa700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2021-05-25T12:00:39.359 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.309+0000 7fdba57fa700 10 monclient(hunting): dump: 2021-05-25T12:00:39.359 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:39.360 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:39.360 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:39.360 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:39.360 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:39.360 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:39.361 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:39.361 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:39.361 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:39.361 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:39.361 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.309+0000 7fdba57fa700 1 -- 172.21.15.102:0/922796431 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2098371367 0 0) 0x7fdbb8114ca0 con 0x7fdbb8090990 2021-05-25T12:00:39.362 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.309+0000 7fdba57fa700 1 -- 172.21.15.102:0/922796431 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdbb8091fe0 con 0x7fdbb8090990 2021-05-25T12:00:39.362 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.309+0000 7fdba57fa700 1 -- 172.21.15.102:0/922796431 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fdba8001840 con 0x7fdbb8090990 2021-05-25T12:00:39.362 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.309+0000 7fdba57fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:39.362 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.309+0000 7fdba57fa700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2021-05-25T12:00:39.362 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.309+0000 7fdba57fa700 10 monclient(hunting): dump: 2021-05-25T12:00:39.363 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:39.363 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:39.363 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:39.363 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:39.364 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:39.364 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:39.364 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:39.364 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:39.364 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:39.365 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:39.365 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.309+0000 7fdba57fa700 1 -- 172.21.15.102:0/922796431 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1082628216 0 0) 0x7fdbb8114a00 con 0x7fdbb8090d70 2021-05-25T12:00:39.365 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.309+0000 7fdba57fa700 1 -- 172.21.15.102:0/922796431 >> v1:172.21.15.102:6790/0 conn(0x7fdbb812e330 legacy=0x7fdbb8140cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:39.365 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.310+0000 7fdba57fa700 1 -- 172.21.15.102:0/922796431 >> v1:172.21.15.181:6789/0 conn(0x7fdbb8090990 legacy=0x7fdbb8128270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:39.365 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.310+0000 7fdba57fa700 10 monclient: _finish_hunting 0 2021-05-25T12:00:39.366 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.310+0000 7fdba57fa700 1 monclient: found mon.a 2021-05-25T12:00:39.366 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.310+0000 7fdba57fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:00:39.366 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.310+0000 7fdba57fa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:00:39.366 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.310+0000 7fdba57fa700 1 -- 172.21.15.102:0/922796431 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbb8128980 con 0x7fdbb8090d70 2021-05-25T12:00:39.367 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.310+0000 7fdba57fa700 10 monclient: _finish_auth 0 2021-05-25T12:00:39.367 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.310+0000 7fdba57fa700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:00:39.367 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.310+0000 7fdbbf7d0700 5 monclient: authenticate success, global_id 4491 2021-05-25T12:00:39.367 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.310+0000 7fdbbf7d0700 10 monclient: _renew_subs 2021-05-25T12:00:39.367 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.310+0000 7fdbbf7d0700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:00:39.368 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.310+0000 7fdbbf7d0700 1 -- 172.21.15.102:0/922796431 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbb8112b10 con 0x7fdbb8090d70 2021-05-25T12:00:39.368 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.310+0000 7fdbbf7d0700 10 client.4491.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:00:39.368 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.310+0000 7fdbbf7d0700 10 monclient: _renew_subs 2021-05-25T12:00:39.368 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.310+0000 7fdbbf7d0700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:00:39.368 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.310+0000 7fdbbf7d0700 1 -- 172.21.15.102:0/922796431 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbb8139230 con 0x7fdbb8090d70 2021-05-25T12:00:39.369 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.310+0000 7fdbbf7d0700 1 librados: init done 2021-05-25T12:00:39.369 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.310+0000 7fdba57fa700 1 -- 172.21.15.102:0/922796431 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdbac004940 con 0x7fdbb8090d70 2021-05-25T12:00:39.369 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.310+0000 7fdba57fa700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:00:39.369 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.310+0000 7fdba57fa700 1 -- 172.21.15.102:0/922796431 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fdbac004b30 con 0x7fdbb8090d70 2021-05-25T12:00:39.369 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.310+0000 7fdba57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:39.370 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.310+0000 7fdba57fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:00:39.370 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.310+0000 7fdba77fe700 4 set_mon_vals no callback set 2021-05-25T12:00:39.370 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.310+0000 7fdba57fa700 10 monclient: dump: 2021-05-25T12:00:39.370 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:39.371 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:39.371 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:39.371 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:39.371 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:39.371 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:39.372 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:39.372 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:39.372 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:39.372 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:39.372 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.310+0000 7fdba57fa700 1 -- 172.21.15.102:0/922796431 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7fdbac004d10 con 0x7fdbb8090d70 2021-05-25T12:00:39.373 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.310+0000 7fdbbf7d0700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:00:39.373 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.310+0000 7fdbbf7d0700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:00:39.373 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.310+0000 7fdbbf7d0700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:00:39.373 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.310+0000 7fdbbf7d0700 1 -- 172.21.15.102:0/922796431 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb90004db0 con 0x7fdbb8090d70 2021-05-25T12:00:39.374 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.310+0000 7fdba57fa700 1 -- 172.21.15.102:0/922796431 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(15..15 src has 1..15) v4 ==== 3549+0+0 (unknown 3948819498 0 0) 0x7fdbac03cf60 con 0x7fdbb8090d70 2021-05-25T12:00:39.374 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.310+0000 7fdba57fa700 10 client.4491.objecter ms_dispatch 0x7fdbb8001550 osd_map(15..15 src has 1..15) v4 2021-05-25T12:00:39.374 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.310+0000 7fdba57fa700 3 client.4491.objecter handle_osd_map got epochs [15,15] > 0 2021-05-25T12:00:39.374 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.310+0000 7fdba57fa700 3 client.4491.objecter handle_osd_map decoding full epoch 15 2021-05-25T12:00:39.374 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.311+0000 7fdba57fa700 20 client.4491.objecter dump_active .. 0 homeless 2021-05-25T12:00:39.375 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.311+0000 7fdbbd56c700 10 monclient: get_auth_request con 0x7fdb980335c0 auth_method 0 2021-05-25T12:00:39.375 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.311+0000 7fdba57fa700 10 client.4491.objecter ms_handle_connect 0x7fdb980335c0 2021-05-25T12:00:39.375 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.315+0000 7fdba57fa700 1 -- 172.21.15.102:0/922796431 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7fdbac006380 con 0x7fdbb8090d70 2021-05-25T12:00:39.375 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.315+0000 7fdba57fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:00:39.375 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.315+0000 7fdba57fa700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:00:39.430 INFO:tasks.workunit.client.0.smithi181.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2021-05-25T12:00:39.479 INFO:tasks.workunit.client.0.smithi181.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2021-05-25T12:00:39.515 INFO:tasks.workunit.client.0.smithi181.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2021-05-25T12:00:39.516 INFO:tasks.workunit.client.0.smithi181.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2021-05-25T12:00:39.517 INFO:tasks.workunit.client.0.smithi181.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2021-05-25T12:00:39.524 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.522+0000 7fdbbf7d0700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2021-05-25T12:00:39.524 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.522+0000 7fdbbf7d0700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2021-05-25T12:00:39.524 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.522+0000 7fdbbf7d0700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:00:39.524 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.522+0000 7fdbbf7d0700 1 -- 172.21.15.102:0/922796431 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fdb900055e0 con 0x7fdbb8090d70 2021-05-25T12:00:39.525 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.522+0000 7fdba57fa700 1 -- 172.21.15.102:0/922796431 <== mon.0 v1:172.21.15.102:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1342 (unknown 254746657 0 1977479342) 0x7fdbac012b60 con 0x7fdbb8090d70 2021-05-25T12:00:39.525 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.522+0000 7fdba57fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2021-05-25T12:00:39.525 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.522+0000 7fdba57fa700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:00:39.525 INFO:teuthology.orchestra.run.smithi102.stdout:{"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138303579357183","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"20ee91b6-8935-4b98-b868-85c03789ccc4","modified":"2021-05-25T11:59:11.011522Z","created":"2021-05-25T11:59:11.011522Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6790","nonce":0}]},"addr":"172.21.15.102:6790/0","public_addr":"172.21.15.102:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2021-05-25T12:00:39.527 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.524+0000 7fdbbf7d0700 10 librados: watch_flush enter 2021-05-25T12:00:39.527 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.524+0000 7fdbbf7d0700 10 librados: watch_flush exit 2021-05-25T12:00:39.527 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.524+0000 7fdbbf7d0700 20 client.4491.objecter shutdown clearing up homeless session... 2021-05-25T12:00:39.528 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.524+0000 7fdbbf7d0700 10 client.4491.objecter successfully canceled tick 2021-05-25T12:00:39.528 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.524+0000 7fdbbf7d0700 1 -- 172.21.15.102:0/922796431 >> v1:172.21.15.181:6812/29582 conn(0x7fdb980335c0 legacy=0x7fdb98035a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:39.528 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.524+0000 7fdbbf7d0700 10 monclient: shutdown 2021-05-25T12:00:39.528 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.524+0000 7fdbbf7d0700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:00:39.529 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.524+0000 7fdbbf7d0700 1 -- 172.21.15.102:0/922796431 >> v1:172.21.15.102:6789/0 conn(0x7fdbb8090d70 legacy=0x7fdbb8138960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:39.529 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.524+0000 7fdbbf7d0700 1 -- 172.21.15.102:0/922796431 shutdown_connections 2021-05-25T12:00:39.529 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.524+0000 7fdbbf7d0700 1 -- 172.21.15.102:0/922796431 >> 172.21.15.102:0/922796431 conn(0x7fdbb809b370 msgr2=0x7fdbb80973c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:00:39.529 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.524+0000 7fdbbf7d0700 1 -- 172.21.15.102:0/922796431 shutdown_connections 2021-05-25T12:00:39.529 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.524+0000 7fdbbf7d0700 1 -- 172.21.15.102:0/922796431 wait complete. 2021-05-25T12:00:39.530 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.524+0000 7fdbbf7d0700 1 librados: shutdown 2021-05-25T12:00:39.536 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2021-05-25T12:00:39.537 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2021-05-25T12:00:39.537 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2021-05-25T12:00:39.551 INFO:tasks.workunit.client.0.smithi181.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2021-05-25T12:00:39.554 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2021-05-25T12:00:39.554 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2021-05-25T12:00:39.608 INFO:tasks.workunit:Running workunits matching rados/test.sh on client.0... 2021-05-25T12:00:39.609 INFO:tasks.workunit:Running workunit rados/test.sh... 2021-05-25T12:00:39.610 DEBUG:teuthology.orchestra.run.smithi181:workunit test rados/test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a16ab1caacd8b41a7fe74bc40a10ec74ee8f5802 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh 2021-05-25T12:00:39.647 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.645+0000 7f8984f47700 10 monclient: get_monmap_and_config 2021-05-25T12:00:39.648 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.645+0000 7f8984f47700 10 monclient: build_initial_monmap 2021-05-25T12:00:39.648 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.645+0000 7f8984f47700 10 monclient: monmap: 2021-05-25T12:00:39.648 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:00:39.648 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:39.648 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:00:39.646868+0000 2021-05-25T12:00:39.649 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:00:39.646868+0000 2021-05-25T12:00:39.649 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:00:39.649 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:39.649 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:00:39.650 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:00:39.650 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:00:39.650 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:39.650 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.646+0000 7f8984f47700 1 Processor -- start 2021-05-25T12:00:39.650 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.646+0000 7f8984f47700 1 -- start start 2021-05-25T12:00:39.651 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.646+0000 7f8984f47700 10 monclient: init 2021-05-25T12:00:39.664 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.646+0000 7f8984f47700 10 monclient: _reopen_session rank -1 2021-05-25T12:00:39.665 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.647+0000 7f8984f47700 10 monclient: _add_conns ranks=[2,1,0] 2021-05-25T12:00:39.665 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.647+0000 7f8984f47700 10 monclient(hunting): picked mon.noname-b con 0x7f8980093fc0 addr v1:172.21.15.102:6790/0 2021-05-25T12:00:39.665 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.647+0000 7f8984f47700 10 monclient(hunting): picked mon.noname-c con 0x7f89800692d0 addr v1:172.21.15.181:6789/0 2021-05-25T12:00:39.665 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.647+0000 7f8984f47700 10 monclient(hunting): picked mon.noname-a con 0x7f898012f130 addr v1:172.21.15.102:6789/0 2021-05-25T12:00:39.666 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.647+0000 7f8984f47700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8980097160 con 0x7f898012f130 2021-05-25T12:00:39.666 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.647+0000 7f8984f47700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8980092fe0 con 0x7f89800692d0 2021-05-25T12:00:39.666 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.647+0000 7f8984f47700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8980093280 con 0x7f8980093fc0 2021-05-25T12:00:39.666 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.647+0000 7f8984f47700 10 monclient(hunting): _renew_subs 2021-05-25T12:00:39.666 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.647+0000 7f8984f47700 10 monclient(hunting): authenticate will time out at 840.244081s 2021-05-25T12:00:39.667 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.647+0000 7f897ed9d700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f898012f130 0x7f8980131520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:59382/0 (socket says 172.21.15.102:59382) 2021-05-25T12:00:39.667 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.647+0000 7f897e59c700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f8980093fc0 0x7f8980096340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:54260/0 (socket says 172.21.15.102:54260) 2021-05-25T12:00:39.667 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.647+0000 7f897ed9d700 1 -- 172.21.15.102:0/3380238143 learned_addr learned my addr 172.21.15.102:0/3380238143 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:00:39.667 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.647+0000 7f897d59a700 1 -- 172.21.15.102:0/3380238143 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3690080794 0 0) 0x7f8980097160 con 0x7f898012f130 2021-05-25T12:00:39.667 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.647+0000 7f897d59a700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:39.668 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.647+0000 7f897d59a700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:39.668 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.647+0000 7f897d59a700 10 monclient(hunting): my global_id is 4494 2021-05-25T12:00:39.668 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.647+0000 7f897d59a700 1 -- 172.21.15.102:0/3380238143 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8964003270 con 0x7f898012f130 2021-05-25T12:00:39.668 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.647+0000 7f897d59a700 1 -- 172.21.15.102:0/3380238143 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3648226611 0 0) 0x7f8980093280 con 0x7f8980093fc0 2021-05-25T12:00:39.668 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.647+0000 7f897d59a700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:39.669 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.647+0000 7f897d59a700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:39.669 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.647+0000 7f897d59a700 10 monclient(hunting): my global_id is 4478 2021-05-25T12:00:39.669 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.647+0000 7f897d59a700 1 -- 172.21.15.102:0/3380238143 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8980097160 con 0x7f8980093fc0 2021-05-25T12:00:39.669 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.647+0000 7f897d59a700 1 -- 172.21.15.102:0/3380238143 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2301552949 0 0) 0x7f8980092fe0 con 0x7f89800692d0 2021-05-25T12:00:39.669 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.647+0000 7f897d59a700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:39.670 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.647+0000 7f897d59a700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:39.670 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.647+0000 7f897d59a700 10 monclient(hunting): my global_id is 74134 2021-05-25T12:00:39.670 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.647+0000 7f897d59a700 1 -- 172.21.15.102:0/3380238143 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8980093280 con 0x7f89800692d0 2021-05-25T12:00:39.670 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.647+0000 7f897d59a700 1 -- 172.21.15.102:0/3380238143 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1658801797 0 0) 0x7f8980097160 con 0x7f8980093fc0 2021-05-25T12:00:39.671 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.647+0000 7f897d59a700 1 -- 172.21.15.102:0/3380238143 >> v1:172.21.15.181:6789/0 conn(0x7f89800692d0 legacy=0x7f8980096a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:39.671 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.647+0000 7f897d59a700 1 -- 172.21.15.102:0/3380238143 >> v1:172.21.15.102:6789/0 conn(0x7f898012f130 legacy=0x7f8980131520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:39.671 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.647+0000 7f897d59a700 10 monclient: _finish_hunting 0 2021-05-25T12:00:39.671 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.647+0000 7f897d59a700 1 monclient: found mon.noname-b 2021-05-25T12:00:39.671 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.647+0000 7f897d59a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:00:39.672 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.647+0000 7f897d59a700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T12:00:39.672 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.647+0000 7f897d59a700 1 -- 172.21.15.102:0/3380238143 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8980093520 con 0x7f8980093fc0 2021-05-25T12:00:39.672 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.647+0000 7f897d59a700 10 monclient: _finish_auth 0 2021-05-25T12:00:39.672 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.647+0000 7f897d59a700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:00:39.672 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.647+0000 7f8984f47700 5 monclient: authenticate success, global_id 4478 2021-05-25T12:00:39.673 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.647+0000 7f8984f47700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:00:39.673 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.647+0000 7f897d59a700 1 -- 172.21.15.102:0/3380238143 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f8974001520 con 0x7f8980093fc0 2021-05-25T12:00:39.673 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.647+0000 7f897d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:39.673 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.647+0000 7f897d59a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:00:39.673 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.647+0000 7f897d59a700 10 monclient: dump: 2021-05-25T12:00:39.674 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:39.674 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:39.674 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:39.674 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:39.674 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:39.675 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:39.675 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:39.675 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:39.675 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:39.675 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:39.676 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.647+0000 7f8984f47700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:00:39.676 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.647+0000 7f897d59a700 1 -- 172.21.15.102:0/3380238143 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f89740017c0 con 0x7f8980093fc0 2021-05-25T12:00:39.676 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.647+0000 7f897d59a700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:00:39.676 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.647+0000 7f8984f47700 10 monclient: get_monmap_and_config success 2021-05-25T12:00:39.676 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.647+0000 7f897d59a700 1 -- 172.21.15.102:0/3380238143 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f8974001b50 con 0x7f8980093fc0 2021-05-25T12:00:39.677 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.647+0000 7f8984f47700 4 set_mon_vals no callback set 2021-05-25T12:00:39.677 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.647+0000 7f897d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:39.677 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.647+0000 7f897d59a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:00:39.677 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.647+0000 7f897d59a700 10 monclient: dump: 2021-05-25T12:00:39.678 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:39.678 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:39.678 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:39.678 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:39.678 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:39.679 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:39.679 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:39.679 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:39.679 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:39.680 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:39.680 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.648+0000 7f8984f47700 10 monclient: shutdown 2021-05-25T12:00:39.680 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.648+0000 7f8984f47700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:00:39.680 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.648+0000 7f8984f47700 1 -- 172.21.15.102:0/3380238143 >> v1:172.21.15.102:6790/0 conn(0x7f8980093fc0 legacy=0x7f8980096340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:39.680 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.648+0000 7f8984f47700 1 -- 172.21.15.102:0/3380238143 shutdown_connections 2021-05-25T12:00:39.681 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.648+0000 7f8984f47700 1 -- 172.21.15.102:0/3380238143 >> 172.21.15.102:0/3380238143 conn(0x7f898009b370 msgr2=0x7f8980097430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:00:39.681 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.648+0000 7f8984f47700 1 -- 172.21.15.102:0/3380238143 shutdown_connections 2021-05-25T12:00:39.681 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.648+0000 7f8984f47700 1 -- 172.21.15.102:0/3380238143 wait complete. 2021-05-25T12:00:39.681 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.648+0000 7f8984f47700 10 monclient: build_initial_monmap 2021-05-25T12:00:39.681 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.648+0000 7f8984f47700 10 monclient: monmap: 2021-05-25T12:00:39.682 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:00:39.682 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:00:39.682 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:00:39.682 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:00:39.683 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:00:39.683 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:00:39.683 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:00:39.683 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:00:39.683 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:00:39.684 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:39.684 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.649+0000 7f8984f47700 1 librados: starting msgr at 2021-05-25T12:00:39.684 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.649+0000 7f8984f47700 1 librados: starting objecter 2021-05-25T12:00:39.684 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.649+0000 7f8984f47700 1 Processor -- start 2021-05-25T12:00:39.685 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.649+0000 7f8984f47700 1 -- start start 2021-05-25T12:00:39.685 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.649+0000 7f8984f47700 1 librados: setting wanted keys 2021-05-25T12:00:39.685 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.649+0000 7f8984f47700 1 librados: calling monclient init 2021-05-25T12:00:39.685 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.649+0000 7f8984f47700 10 monclient: init 2021-05-25T12:00:39.685 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.649+0000 7f8984f47700 10 monclient: _reopen_session rank -1 2021-05-25T12:00:39.686 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.649+0000 7f8984f47700 10 monclient: _add_conns ranks=[2,1,0] 2021-05-25T12:00:39.686 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.649+0000 7f8984f47700 10 monclient(hunting): picked mon.noname-c con 0x7f89800692d0 addr v1:172.21.15.102:6790/0 2021-05-25T12:00:39.686 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.649+0000 7f8984f47700 10 monclient(hunting): picked mon.noname-b con 0x7f8980093fc0 addr v1:172.21.15.181:6789/0 2021-05-25T12:00:39.686 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.649+0000 7f8984f47700 10 monclient(hunting): picked mon.noname-a con 0x7f898012f130 addr v1:172.21.15.102:6789/0 2021-05-25T12:00:39.687 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.649+0000 7f8984f47700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f898012b3a0 con 0x7f898012f130 2021-05-25T12:00:39.687 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.649+0000 7f8984f47700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89801127b0 con 0x7f8980093fc0 2021-05-25T12:00:39.687 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.649+0000 7f8984f47700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8980112a50 con 0x7f89800692d0 2021-05-25T12:00:39.687 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.649+0000 7f8984f47700 10 monclient(hunting): _renew_subs 2021-05-25T12:00:39.687 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.649+0000 7f8984f47700 10 monclient(hunting): authenticate will time out at 840.246695s 2021-05-25T12:00:39.688 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.649+0000 7f897ed9d700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f898012f130 0x7f89801414f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:59390/0 (socket says 172.21.15.102:59390) 2021-05-25T12:00:39.688 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.649+0000 7f897e59c700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f89800692d0 0x7f8980127570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:54264/0 (socket says 172.21.15.102:54264) 2021-05-25T12:00:39.688 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.649+0000 7f897ed9d700 1 -- 172.21.15.102:0/3246953083 learned_addr learned my addr 172.21.15.102:0/3246953083 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:00:39.688 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.649+0000 7f896e7fc700 10 client.?.objecter ms_handle_connect 0x7f898012f130 2021-05-25T12:00:39.688 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.649+0000 7f896e7fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:00:39.689 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.649+0000 7f896e7fc700 10 client.?.objecter ms_handle_connect 0x7f89800692d0 2021-05-25T12:00:39.689 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.649+0000 7f896e7fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:00:39.689 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.649+0000 7f896e7fc700 10 client.?.objecter ms_handle_connect 0x7f8980093fc0 2021-05-25T12:00:39.689 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.649+0000 7f896e7fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:00:39.689 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.649+0000 7f896e7fc700 1 -- 172.21.15.102:0/3246953083 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1227736437 0 0) 0x7f898012b3a0 con 0x7f898012f130 2021-05-25T12:00:39.690 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.649+0000 7f896e7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:39.690 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.650+0000 7f896e7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:39.690 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.650+0000 7f896e7fc700 10 monclient(hunting): my global_id is 4497 2021-05-25T12:00:39.690 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.650+0000 7f896e7fc700 1 -- 172.21.15.102:0/3246953083 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f895c004c10 con 0x7f898012f130 2021-05-25T12:00:39.691 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.650+0000 7f896e7fc700 1 -- 172.21.15.102:0/3246953083 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1440749909 0 0) 0x7f8980112a50 con 0x7f89800692d0 2021-05-25T12:00:39.691 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.650+0000 7f896e7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:39.691 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.650+0000 7f896e7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:39.691 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.650+0000 7f896e7fc700 10 monclient(hunting): my global_id is 4481 2021-05-25T12:00:39.691 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.650+0000 7f896e7fc700 1 -- 172.21.15.102:0/3246953083 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f898012b3a0 con 0x7f89800692d0 2021-05-25T12:00:39.692 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.650+0000 7f896e7fc700 1 -- 172.21.15.102:0/3246953083 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1490765080 0 0) 0x7f89801127b0 con 0x7f8980093fc0 2021-05-25T12:00:39.692 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.650+0000 7f896e7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:39.692 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.650+0000 7f896e7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:39.692 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.650+0000 7f896e7fc700 10 monclient(hunting): my global_id is 74137 2021-05-25T12:00:39.693 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.650+0000 7f896e7fc700 1 -- 172.21.15.102:0/3246953083 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8980112a50 con 0x7f8980093fc0 2021-05-25T12:00:39.693 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.650+0000 7f896e7fc700 1 -- 172.21.15.102:0/3246953083 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4239696494 0 0) 0x7f898012b3a0 con 0x7f89800692d0 2021-05-25T12:00:39.693 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.650+0000 7f896e7fc700 1 -- 172.21.15.102:0/3246953083 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f89801127b0 con 0x7f89800692d0 2021-05-25T12:00:39.693 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.650+0000 7f896e7fc700 1 -- 172.21.15.102:0/3246953083 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f8974001b90 con 0x7f89800692d0 2021-05-25T12:00:39.693 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.650+0000 7f896e7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:39.694 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.650+0000 7f896e7fc700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:00:39.694 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.650+0000 7f896e7fc700 10 monclient(hunting): dump: 2021-05-25T12:00:39.694 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:39.694 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:39.695 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:39.695 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:39.695 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:39.695 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:39.695 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:39.696 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:39.696 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:39.696 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:39.696 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.650+0000 7f896e7fc700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:00:39.696 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.650+0000 7f896e7fc700 1 -- 172.21.15.102:0/3246953083 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2141291421 0 0) 0x7f895c004c10 con 0x7f898012f130 2021-05-25T12:00:39.697 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.650+0000 7f896e7fc700 1 -- 172.21.15.102:0/3246953083 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f898012b3a0 con 0x7f898012f130 2021-05-25T12:00:39.697 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.650+0000 7f896e7fc700 1 -- 172.21.15.102:0/3246953083 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3261097124 0 0) 0x7f8980112a50 con 0x7f8980093fc0 2021-05-25T12:00:39.697 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.650+0000 7f896e7fc700 1 -- 172.21.15.102:0/3246953083 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f895c004c10 con 0x7f8980093fc0 2021-05-25T12:00:39.697 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.650+0000 7f896e7fc700 1 -- 172.21.15.102:0/3246953083 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f8970001540 con 0x7f898012f130 2021-05-25T12:00:39.697 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.650+0000 7f896e7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:39.698 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.650+0000 7f896e7fc700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2021-05-25T12:00:39.698 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.650+0000 7f896e7fc700 10 monclient(hunting): dump: 2021-05-25T12:00:39.698 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:39.698 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:39.699 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:39.699 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:39.699 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:39.699 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:39.699 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:39.700 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:39.700 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:39.700 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:39.700 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.650+0000 7f896e7fc700 1 -- 172.21.15.102:0/3246953083 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f8968001840 con 0x7f8980093fc0 2021-05-25T12:00:39.700 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.650+0000 7f896e7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:39.701 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.650+0000 7f896e7fc700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2021-05-25T12:00:39.701 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.650+0000 7f896e7fc700 10 monclient(hunting): dump: 2021-05-25T12:00:39.701 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:39.701 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:39.701 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:39.702 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:39.702 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:39.702 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:39.702 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:39.703 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:39.703 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:39.703 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:39.703 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.650+0000 7f896e7fc700 1 -- 172.21.15.102:0/3246953083 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3009693863 0 0) 0x7f89801127b0 con 0x7f89800692d0 2021-05-25T12:00:39.703 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.650+0000 7f896e7fc700 1 -- 172.21.15.102:0/3246953083 >> v1:172.21.15.181:6789/0 conn(0x7f8980093fc0 legacy=0x7f898013ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:39.704 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.650+0000 7f896e7fc700 1 -- 172.21.15.102:0/3246953083 >> v1:172.21.15.102:6789/0 conn(0x7f898012f130 legacy=0x7f89801414f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:39.704 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.650+0000 7f896e7fc700 10 monclient: _finish_hunting 0 2021-05-25T12:00:39.704 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.650+0000 7f896e7fc700 1 monclient: found mon.c 2021-05-25T12:00:39.704 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.650+0000 7f896e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:00:39.704 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.650+0000 7f896e7fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:00:39.705 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.650+0000 7f896e7fc700 1 -- 172.21.15.102:0/3246953083 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8980112cf0 con 0x7f89800692d0 2021-05-25T12:00:39.705 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.650+0000 7f896e7fc700 10 monclient: _finish_auth 0 2021-05-25T12:00:39.705 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.650+0000 7f896e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:00:39.705 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.650+0000 7f8984f47700 5 monclient: authenticate success, global_id 4481 2021-05-25T12:00:39.706 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.650+0000 7f8984f47700 10 monclient: _renew_subs 2021-05-25T12:00:39.706 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.650+0000 7f8984f47700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:00:39.706 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.650+0000 7f8984f47700 1 -- 172.21.15.102:0/3246953083 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8980127c80 con 0x7f89800692d0 2021-05-25T12:00:39.706 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.650+0000 7f896e7fc700 1 -- 172.21.15.102:0/3246953083 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8974004970 con 0x7f89800692d0 2021-05-25T12:00:39.706 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.650+0000 7f896e7fc700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:00:39.707 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.650+0000 7f8984f47700 10 client.4481.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:00:39.707 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.650+0000 7f8984f47700 10 monclient: _renew_subs 2021-05-25T12:00:39.707 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.650+0000 7f896e7fc700 1 -- 172.21.15.102:0/3246953083 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f8974004d00 con 0x7f89800692d0 2021-05-25T12:00:39.707 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.650+0000 7f8984f47700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:00:39.707 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.650+0000 7f8984f47700 1 -- 172.21.15.102:0/3246953083 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8980067a40 con 0x7f89800692d0 2021-05-25T12:00:39.708 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.650+0000 7f8984f47700 1 librados: init done 2021-05-25T12:00:39.708 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.650+0000 7f896e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:39.708 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.650+0000 7f897cd99700 4 set_mon_vals no callback set 2021-05-25T12:00:39.708 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.650+0000 7f896e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:00:39.708 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.650+0000 7f896e7fc700 10 monclient: dump: 2021-05-25T12:00:39.709 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:39.709 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:39.709 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:39.709 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:39.710 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:39.710 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:39.710 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:39.710 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:39.710 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:39.711 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:39.711 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.651+0000 7f8984f47700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:00:39.711 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.651+0000 7f8984f47700 1 -- 172.21.15.102:0/3246953083 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f8980059730 con 0x7f8980139590 2021-05-25T12:00:39.711 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.651+0000 7f896e7fc700 1 -- 172.21.15.102:0/3246953083 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f897400e0f0 con 0x7f89800692d0 2021-05-25T12:00:39.712 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.651+0000 7f896e7fc700 10 client.4481.objecter ms_handle_connect 0x7f8980139590 2021-05-25T12:00:39.712 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.651+0000 7f896e7fc700 10 client.4481.objecter resend_mon_ops 2021-05-25T12:00:39.712 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.651+0000 7f896e7fc700 1 -- 172.21.15.102:0/3246953083 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(15..15 src has 1..15) v4 ==== 3549+0+0 (unknown 3948819498 0 0) 0x7f897403cfc0 con 0x7f89800692d0 2021-05-25T12:00:39.712 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.651+0000 7f896e7fc700 10 client.4481.objecter ms_dispatch 0x7f8980001550 osd_map(15..15 src has 1..15) v4 2021-05-25T12:00:39.712 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.651+0000 7f896e7fc700 3 client.4481.objecter handle_osd_map got epochs [15,15] > 0 2021-05-25T12:00:39.713 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.651+0000 7f896e7fc700 3 client.4481.objecter handle_osd_map decoding full epoch 15 2021-05-25T12:00:39.713 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.651+0000 7f896e7fc700 20 client.4481.objecter dump_active .. 0 homeless 2021-05-25T12:00:39.713 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.651+0000 7f896e7fc700 1 -- 172.21.15.102:0/3246953083 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1890089192 0 0) 0x7f8980059730 con 0x7f8980139590 2021-05-25T12:00:39.713 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.651+0000 7f896e7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:39.713 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.651+0000 7f896e7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:39.714 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.651+0000 7f896e7fc700 10 monclient(hunting): my global_id is 4500 2021-05-25T12:00:39.714 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.651+0000 7f896e7fc700 1 -- 172.21.15.102:0/3246953083 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89801127b0 con 0x7f8980139590 2021-05-25T12:00:39.714 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.651+0000 7f896e7fc700 1 -- 172.21.15.102:0/3246953083 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3514928391 0 0) 0x7f89801127b0 con 0x7f8980139590 2021-05-25T12:00:39.714 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.651+0000 7f896e7fc700 1 -- 172.21.15.102:0/3246953083 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8980059730 con 0x7f8980139590 2021-05-25T12:00:39.715 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.651+0000 7f896e7fc700 1 -- 172.21.15.102:0/3246953083 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f8968001d70 con 0x7f8980139590 2021-05-25T12:00:39.715 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.651+0000 7f896e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:39.715 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.651+0000 7f896e7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:00:39.715 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.651+0000 7f896e7fc700 10 monclient: dump: 2021-05-25T12:00:39.715 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:39.716 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:39.716 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:39.716 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:39.716 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:39.716 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:39.717 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:39.717 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:39.717 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:39.717 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:39.717 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.651+0000 7f897ed9d700 10 monclient: get_auth_request con 0x7f895c0335c0 auth_method 0 2021-05-25T12:00:39.718 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.651+0000 7f896e7fc700 1 -- 172.21.15.102:0/3246953083 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2833666530 0 0) 0x7f8980059730 con 0x7f8980139590 2021-05-25T12:00:39.718 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.651+0000 7f896e7fc700 1 -- 172.21.15.102:0/3246953083 --> v1:172.21.15.102:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f89800008d0 con 0x7f8980139590 2021-05-25T12:00:39.718 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.651+0000 7f896e7fc700 10 client.4481.objecter ms_handle_connect 0x7f895c0335c0 2021-05-25T12:00:39.718 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.652+0000 7f896e7fc700 1 -- 172.21.15.102:0/3246953083 <== mon.0 v1:172.21.15.102:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7f89680040e0 con 0x7f8980139590 2021-05-25T12:00:39.718 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.652+0000 7f896e7fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:00:39.719 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.652+0000 7f896e7fc700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:00:39.719 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.652+0000 7f896e7fc700 1 -- 172.21.15.102:0/3246953083 >> v1:172.21.15.102:6789/0 conn(0x7f8980139590 legacy=0x7f8980058d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:39.719 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.652+0000 7f896e7fc700 1 -- 172.21.15.102:0/3246953083 >> v1:172.21.15.102:6789/0 conn(0x7f8980139590 legacy=0x7f8980058d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:00:39.719 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.658+0000 7f8984f47700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:00:39.720 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.658+0000 7f8984f47700 1 -- 172.21.15.102:0/3246953083 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f8980092350 con 0x7f898012c5c0 2021-05-25T12:00:39.720 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.658+0000 7f896e7fc700 10 client.4481.objecter ms_handle_connect 0x7f898012c5c0 2021-05-25T12:00:39.720 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.658+0000 7f896e7fc700 10 client.4481.objecter resend_mon_ops 2021-05-25T12:00:39.720 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.658+0000 7f896e7fc700 1 -- 172.21.15.102:0/3246953083 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2876076769 0 0) 0x7f8980092350 con 0x7f898012c5c0 2021-05-25T12:00:39.720 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.658+0000 7f896e7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:39.721 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.658+0000 7f896e7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:39.721 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.658+0000 7f896e7fc700 10 monclient(hunting): my global_id is 4503 2021-05-25T12:00:39.721 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.658+0000 7f896e7fc700 1 -- 172.21.15.102:0/3246953083 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8980059730 con 0x7f898012c5c0 2021-05-25T12:00:39.721 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.659+0000 7f896e7fc700 1 -- 172.21.15.102:0/3246953083 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1029987124 0 0) 0x7f8980059730 con 0x7f898012c5c0 2021-05-25T12:00:39.722 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.659+0000 7f896e7fc700 1 -- 172.21.15.102:0/3246953083 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8980092350 con 0x7f898012c5c0 2021-05-25T12:00:39.722 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.659+0000 7f896e7fc700 1 -- 172.21.15.102:0/3246953083 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f89680024f0 con 0x7f898012c5c0 2021-05-25T12:00:39.722 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.659+0000 7f896e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:39.722 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.659+0000 7f896e7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:00:39.722 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.659+0000 7f896e7fc700 10 monclient: dump: 2021-05-25T12:00:39.723 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:39.723 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:39.723 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:39.723 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:39.723 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:39.724 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:39.724 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:39.724 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:39.724 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:39.724 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:39.725 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.659+0000 7f896e7fc700 1 -- 172.21.15.102:0/3246953083 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3276773654 0 0) 0x7f8980092350 con 0x7f898012c5c0 2021-05-25T12:00:39.725 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.659+0000 7f896e7fc700 1 -- 172.21.15.102:0/3246953083 --> v1:172.21.15.102:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f8980059420 con 0x7f898012c5c0 2021-05-25T12:00:39.725 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.659+0000 7f896e7fc700 1 -- 172.21.15.102:0/3246953083 <== mon.0 v1:172.21.15.102:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3942 (unknown 0 0 3222130028) 0x7f8968006310 con 0x7f898012c5c0 2021-05-25T12:00:39.725 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.659+0000 7f896e7fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:00:39.725 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.659+0000 7f896e7fc700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:00:39.726 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.659+0000 7f896e7fc700 1 -- 172.21.15.102:0/3246953083 >> v1:172.21.15.102:6789/0 conn(0x7f898012c5c0 legacy=0x7f8980146c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:39.726 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.659+0000 7f896e7fc700 1 -- 172.21.15.102:0/3246953083 >> v1:172.21.15.102:6789/0 conn(0x7f898012c5c0 legacy=0x7f8980146c30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:00:39.726 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:00:39.727 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:00:39.727 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:00:39.728 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "leader", 2021-05-25T12:00:39.728 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 14, 2021-05-25T12:00:39.728 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:00:39.728 INFO:teuthology.orchestra.run.smithi102.stdout: 0, 2021-05-25T12:00:39.728 INFO:teuthology.orchestra.run.smithi102.stdout: 1, 2021-05-25T12:00:39.729 INFO:teuthology.orchestra.run.smithi102.stdout: 2 2021-05-25T12:00:39.729 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:00:39.729 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 2, 2021-05-25T12:00:39.729 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:00:39.729 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:00:39.730 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:00:39.730 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:00:39.730 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:00:39.730 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:00:39.731 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:00:39.731 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:00:39.731 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:00:39.731 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:00:39.731 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:00:39.732 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:00:39.732 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:00:39.732 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:00:39.732 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:00:39.732 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:00:39.733 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:00:39.733 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:00:39.733 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:00:39.733 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:00:39.733 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:00:39.734 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:00:39.734 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:00:39.734 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:00:39.734 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:00:39.734 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:39.735 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:00:39.735 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:00:39.735 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:00:39.735 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:00:39.735 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:00:39.736 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:00:39.736 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:00:39.736 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:00:39.736 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:00:39.737 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:00:39.737 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:00:39.737 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:00:39.737 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:00:39.737 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:00:39.738 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:00:39.738 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:00:39.738 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:00:39.738 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:00:39.738 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:00:39.739 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:00:39.739 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:00:39.739 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:00:39.739 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:00:39.740 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:00:39.740 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:00:39.740 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:00:39.740 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:39.740 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:00:39.741 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:39.741 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:00:39.741 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:00:39.741 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:00:39.741 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:00:39.742 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:39.742 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:00:39.742 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:00:39.742 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:00:39.742 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:39.743 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:00:39.743 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:39.743 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:00:39.743 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:00:39.743 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:00:39.744 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:00:39.744 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:00:39.744 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:39.744 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:39.744 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:00:39.745 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:00:39.745 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:00:39.745 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:00:39.745 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:39.746 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:00:39.746 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:00:39.746 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:00:39.746 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:39.746 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:00:39.747 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:39.747 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:00:39.747 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:00:39.747 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:00:39.747 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:00:39.748 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:00:39.748 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:39.748 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:39.748 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:00:39.748 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:00:39.749 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:00:39.749 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:00:39.749 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:39.749 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:00:39.749 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:00:39.750 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:00:39.750 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:39.750 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:00:39.750 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:39.751 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:00:39.751 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:00:39.751 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:00:39.751 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:00:39.751 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:00:39.752 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:39.752 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:00:39.752 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:39.752 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:00:39.753 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:00:39.753 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:39.753 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:00:39.753 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:00:39.753 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:00:39.754 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:39.754 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:00:39.754 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": [ 2021-05-25T12:00:39.754 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:39.755 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:00:39.755 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:00:39.755 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:00:39.755 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:39.755 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:00:39.756 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:00:39.756 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:39.756 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:00:39.756 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:00:39.756 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 4 2021-05-25T12:00:39.757 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:39.757 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:00:39.757 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:39.757 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:00:39.757 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:00:39.760 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.659+0000 7f8984f47700 10 librados: watch_flush enter 2021-05-25T12:00:39.760 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.659+0000 7f8984f47700 10 librados: watch_flush exit 2021-05-25T12:00:39.760 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.659+0000 7f8984f47700 20 client.4481.objecter shutdown clearing up homeless session... 2021-05-25T12:00:39.761 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.659+0000 7f8984f47700 10 client.4481.objecter successfully canceled tick 2021-05-25T12:00:39.761 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.659+0000 7f8984f47700 1 -- 172.21.15.102:0/3246953083 >> v1:172.21.15.181:6812/29582 conn(0x7f895c0335c0 legacy=0x7f895c035a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:39.761 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.659+0000 7f8984f47700 10 monclient: shutdown 2021-05-25T12:00:39.761 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.659+0000 7f8984f47700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:00:39.761 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.659+0000 7f8984f47700 1 -- 172.21.15.102:0/3246953083 >> v1:172.21.15.102:6790/0 conn(0x7f89800692d0 legacy=0x7f8980127570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:39.762 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.659+0000 7f897ed9d700 1 -- 172.21.15.102:0/3246953083 reap_dead start 2021-05-25T12:00:39.762 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.659+0000 7f897ed9d700 1 -- 172.21.15.102:0/3246953083 reap_dead start 2021-05-25T12:00:39.762 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.659+0000 7f8984f47700 1 -- 172.21.15.102:0/3246953083 shutdown_connections 2021-05-25T12:00:39.762 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.659+0000 7f8984f47700 1 -- 172.21.15.102:0/3246953083 >> 172.21.15.102:0/3246953083 conn(0x7f898009b370 msgr2=0x7f8980095b60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:00:39.763 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.659+0000 7f8984f47700 1 -- 172.21.15.102:0/3246953083 shutdown_connections 2021-05-25T12:00:39.763 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.659+0000 7f8984f47700 1 -- 172.21.15.102:0/3246953083 wait complete. 2021-05-25T12:00:39.763 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.659+0000 7f8984f47700 1 librados: shutdown 2021-05-25T12:00:39.764 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2021-05-25T12:00:39.765 INFO:tasks.workunit.client.0.smithi181.stderr:+ parallel=1 2021-05-25T12:00:39.765 INFO:tasks.workunit.client.0.smithi181.stderr:+ '[' '' = --serial ']' 2021-05-25T12:00:39.766 INFO:tasks.workunit.client.0.smithi181.stderr:+ color= 2021-05-25T12:00:39.766 INFO:tasks.workunit.client.0.smithi181.stderr:+ '[' -t 1 ']' 2021-05-25T12:00:39.766 INFO:tasks.workunit.client.0.smithi181.stderr:+ trap cleanup EXIT ERR HUP INT QUIT 2021-05-25T12:00:39.766 INFO:tasks.workunit.client.0.smithi181.stderr:+ declare -A pids 2021-05-25T12:00:39.767 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2021-05-25T12:00:39.767 INFO:tasks.workunit.client.0.smithi181.stderr:+ '[' 1 -eq 1 ']' 2021-05-25T12:00:39.767 INFO:tasks.workunit.client.0.smithi181.stderr:++ printf %25s api_aio 2021-05-25T12:00:39.767 INFO:tasks.workunit.client.0.smithi181.stderr:+ r=' api_aio' 2021-05-25T12:00:39.767 INFO:tasks.workunit.client.0.smithi181.stderr:++ echo api_aio 2021-05-25T12:00:39.768 INFO:tasks.workunit.client.0.smithi181.stderr:++ awk '{print $1}' 2021-05-25T12:00:39.768 INFO:tasks.workunit.client.0.smithi181.stderr:+ ff=api_aio 2021-05-25T12:00:39.768 INFO:tasks.workunit.client.0.smithi181.stderr:+ pid=32837 2021-05-25T12:00:39.768 INFO:tasks.workunit.client.0.smithi181.stderr:+ echo 'test api_aio on pid 32837' 2021-05-25T12:00:39.769 INFO:tasks.workunit.client.0.smithi181.stdout:test api_aio on pid 32837 2021-05-25T12:00:39.771 INFO:tasks.workunit.client.0.smithi181.stderr:+ pids[$f]=32837 2021-05-25T12:00:39.771 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2021-05-25T12:00:39.771 INFO:tasks.workunit.client.0.smithi181.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio 2>&1 | tee ceph_test_rados_api_aio.log | sed "s/^/ api_aio: /"' 2021-05-25T12:00:39.772 INFO:tasks.workunit.client.0.smithi181.stderr:+ '[' 1 -eq 1 ']' 2021-05-25T12:00:39.772 INFO:tasks.workunit.client.0.smithi181.stderr:++ printf %25s api_aio_pp 2021-05-25T12:00:39.772 INFO:tasks.workunit.client.0.smithi181.stderr:+ r=' api_aio_pp' 2021-05-25T12:00:39.772 INFO:tasks.workunit.client.0.smithi181.stderr:++ echo api_aio_pp 2021-05-25T12:00:39.772 INFO:tasks.workunit.client.0.smithi181.stderr:++ awk '{print $1}' 2021-05-25T12:00:39.773 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph_test_rados_api_aio 2021-05-25T12:00:39.773 INFO:tasks.workunit.client.0.smithi181.stderr:+ tee ceph_test_rados_api_aio.log 2021-05-25T12:00:39.773 INFO:tasks.workunit.client.0.smithi181.stderr:+ sed 's/^/ api_aio: /' 2021-05-25T12:00:39.773 INFO:tasks.workunit.client.0.smithi181.stderr:+ ff=api_aio_pp 2021-05-25T12:00:39.773 INFO:tasks.workunit.client.0.smithi181.stderr:+ pid=32845 2021-05-25T12:00:39.774 INFO:tasks.workunit.client.0.smithi181.stdout:test api_aio_pp on pid 32845 2021-05-25T12:00:39.776 INFO:tasks.workunit.client.0.smithi181.stderr:+ echo 'test api_aio_pp on pid 32845' 2021-05-25T12:00:39.776 INFO:tasks.workunit.client.0.smithi181.stderr:+ pids[$f]=32845 2021-05-25T12:00:39.776 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2021-05-25T12:00:39.776 INFO:tasks.workunit.client.0.smithi181.stderr:+ '[' 1 -eq 1 ']' 2021-05-25T12:00:39.777 INFO:tasks.workunit.client.0.smithi181.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio_pp 2>&1 | tee ceph_test_rados_api_aio_pp.log | sed "s/^/ api_aio_pp: /"' 2021-05-25T12:00:39.777 INFO:tasks.workunit.client.0.smithi181.stderr:++ printf %25s api_io 2021-05-25T12:00:39.777 INFO:tasks.workunit.client.0.smithi181.stderr:+ r=' api_io' 2021-05-25T12:00:39.777 INFO:tasks.workunit.client.0.smithi181.stderr:++ echo api_io 2021-05-25T12:00:39.777 INFO:tasks.workunit.client.0.smithi181.stderr:++ awk '{print $1}' 2021-05-25T12:00:39.778 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph_test_rados_api_aio_pp 2021-05-25T12:00:39.778 INFO:tasks.workunit.client.0.smithi181.stderr:+ tee ceph_test_rados_api_aio_pp.log 2021-05-25T12:00:39.778 INFO:tasks.workunit.client.0.smithi181.stderr:+ sed 's/^/ api_aio_pp: /' 2021-05-25T12:00:39.778 INFO:tasks.workunit.client.0.smithi181.stderr:+ ff=api_io 2021-05-25T12:00:39.779 INFO:tasks.workunit.client.0.smithi181.stderr:+ pid=32853 2021-05-25T12:00:39.779 INFO:tasks.workunit.client.0.smithi181.stderr:+ echo 'test api_io on pid 32853' 2021-05-25T12:00:39.779 INFO:tasks.workunit.client.0.smithi181.stdout:test api_io on pid 32853 2021-05-25T12:00:39.781 INFO:tasks.workunit.client.0.smithi181.stderr:+ pids[$f]=32853 2021-05-25T12:00:39.782 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2021-05-25T12:00:39.782 INFO:tasks.workunit.client.0.smithi181.stderr:+ '[' 1 -eq 1 ']' 2021-05-25T12:00:39.782 INFO:tasks.workunit.client.0.smithi181.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io 2>&1 | tee ceph_test_rados_api_io.log | sed "s/^/ api_io: /"' 2021-05-25T12:00:39.782 INFO:tasks.workunit.client.0.smithi181.stderr:++ printf %25s api_io_pp 2021-05-25T12:00:39.783 INFO:tasks.workunit.client.0.smithi181.stderr:+ r=' api_io_pp' 2021-05-25T12:00:39.783 INFO:tasks.workunit.client.0.smithi181.stderr:++ echo api_io_pp 2021-05-25T12:00:39.783 INFO:tasks.workunit.client.0.smithi181.stderr:++ awk '{print $1}' 2021-05-25T12:00:39.783 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph_test_rados_api_io 2021-05-25T12:00:39.783 INFO:tasks.workunit.client.0.smithi181.stderr:+ tee ceph_test_rados_api_io.log 2021-05-25T12:00:39.784 INFO:tasks.workunit.client.0.smithi181.stderr:+ sed 's/^/ api_io: /' 2021-05-25T12:00:39.784 INFO:tasks.workunit.client.0.smithi181.stderr:+ ff=api_io_pp 2021-05-25T12:00:39.784 INFO:tasks.workunit.client.0.smithi181.stderr:+ pid=32861 2021-05-25T12:00:39.784 INFO:tasks.workunit.client.0.smithi181.stderr:+ echo 'test api_io_pp on pid 32861' 2021-05-25T12:00:39.785 INFO:tasks.workunit.client.0.smithi181.stdout:test api_io_pp on pid 32861 2021-05-25T12:00:39.786 INFO:tasks.workunit.client.0.smithi181.stderr:+ pids[$f]=32861 2021-05-25T12:00:39.787 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2021-05-25T12:00:39.787 INFO:tasks.workunit.client.0.smithi181.stderr:+ '[' 1 -eq 1 ']' 2021-05-25T12:00:39.787 INFO:tasks.workunit.client.0.smithi181.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io_pp 2>&1 | tee ceph_test_rados_api_io_pp.log | sed "s/^/ api_io_pp: /"' 2021-05-25T12:00:39.787 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph_test_rados_api_io_pp 2021-05-25T12:00:39.788 INFO:tasks.workunit.client.0.smithi181.stderr:+ tee ceph_test_rados_api_io_pp.log 2021-05-25T12:00:39.788 INFO:tasks.workunit.client.0.smithi181.stderr:+ sed 's/^/ api_io_pp: /' 2021-05-25T12:00:39.788 INFO:tasks.workunit.client.0.smithi181.stderr:++ printf %25s api_asio 2021-05-25T12:00:39.788 INFO:tasks.workunit.client.0.smithi181.stderr:+ r=' api_asio' 2021-05-25T12:00:39.788 INFO:tasks.workunit.client.0.smithi181.stderr:++ echo api_asio 2021-05-25T12:00:39.789 INFO:tasks.workunit.client.0.smithi181.stderr:++ awk '{print $1}' 2021-05-25T12:00:39.789 INFO:tasks.workunit.client.0.smithi181.stderr:+ ff=api_asio 2021-05-25T12:00:39.789 INFO:tasks.workunit.client.0.smithi181.stderr:+ pid=32869 2021-05-25T12:00:39.789 INFO:tasks.workunit.client.0.smithi181.stderr:+ echo 'test api_asio on pid 32869' 2021-05-25T12:00:39.790 INFO:tasks.workunit.client.0.smithi181.stdout:test api_asio on pid 32869 2021-05-25T12:00:39.791 INFO:tasks.workunit.client.0.smithi181.stderr:+ pids[$f]=32869 2021-05-25T12:00:39.792 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2021-05-25T12:00:39.792 INFO:tasks.workunit.client.0.smithi181.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_asio 2>&1 | tee ceph_test_rados_api_asio.log | sed "s/^/ api_asio: /"' 2021-05-25T12:00:39.792 INFO:tasks.workunit.client.0.smithi181.stderr:+ '[' 1 -eq 1 ']' 2021-05-25T12:00:39.792 INFO:tasks.workunit.client.0.smithi181.stderr:++ printf %25s api_list 2021-05-25T12:00:39.792 INFO:tasks.workunit.client.0.smithi181.stderr:+ r=' api_list' 2021-05-25T12:00:39.793 INFO:tasks.workunit.client.0.smithi181.stderr:++ echo api_list 2021-05-25T12:00:39.793 INFO:tasks.workunit.client.0.smithi181.stderr:++ awk '{print $1}' 2021-05-25T12:00:39.793 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph_test_rados_api_asio 2021-05-25T12:00:39.793 INFO:tasks.workunit.client.0.smithi181.stderr:+ tee ceph_test_rados_api_asio.log 2021-05-25T12:00:39.794 INFO:tasks.workunit.client.0.smithi181.stderr:+ sed 's/^/ api_asio: /' 2021-05-25T12:00:39.794 INFO:tasks.workunit.client.0.smithi181.stderr:+ ff=api_list 2021-05-25T12:00:39.794 INFO:tasks.workunit.client.0.smithi181.stderr:+ pid=32877 2021-05-25T12:00:39.794 INFO:tasks.workunit.client.0.smithi181.stderr:+ echo 'test api_list on pid 32877' 2021-05-25T12:00:39.795 INFO:tasks.workunit.client.0.smithi181.stdout:test api_list on pid 32877 2021-05-25T12:00:39.796 INFO:tasks.workunit.client.0.smithi181.stderr:+ pids[$f]=32877 2021-05-25T12:00:39.796 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2021-05-25T12:00:39.797 INFO:tasks.workunit.client.0.smithi181.stderr:+ '[' 1 -eq 1 ']' 2021-05-25T12:00:39.797 INFO:tasks.workunit.client.0.smithi181.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_list 2>&1 | tee ceph_test_rados_api_list.log | sed "s/^/ api_list: /"' 2021-05-25T12:00:39.797 INFO:tasks.workunit.client.0.smithi181.stderr:++ printf %25s api_lock 2021-05-25T12:00:39.797 INFO:tasks.workunit.client.0.smithi181.stderr:+ r=' api_lock' 2021-05-25T12:00:39.797 INFO:tasks.workunit.client.0.smithi181.stderr:++ echo api_lock 2021-05-25T12:00:39.798 INFO:tasks.workunit.client.0.smithi181.stderr:++ awk '{print $1}' 2021-05-25T12:00:39.798 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph_test_rados_api_list 2021-05-25T12:00:39.798 INFO:tasks.workunit.client.0.smithi181.stderr:+ tee ceph_test_rados_api_list.log 2021-05-25T12:00:39.798 INFO:tasks.workunit.client.0.smithi181.stderr:+ sed 's/^/ api_list: /' 2021-05-25T12:00:39.799 INFO:tasks.workunit.client.0.smithi181.stdout:test api_lock on pid 32885 2021-05-25T12:00:39.800 INFO:tasks.workunit.client.0.smithi181.stderr:+ ff=api_lock 2021-05-25T12:00:39.800 INFO:tasks.workunit.client.0.smithi181.stderr:+ pid=32885 2021-05-25T12:00:39.801 INFO:tasks.workunit.client.0.smithi181.stderr:+ echo 'test api_lock on pid 32885' 2021-05-25T12:00:39.801 INFO:tasks.workunit.client.0.smithi181.stderr:+ pids[$f]=32885 2021-05-25T12:00:39.801 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2021-05-25T12:00:39.801 INFO:tasks.workunit.client.0.smithi181.stderr:+ '[' 1 -eq 1 ']' 2021-05-25T12:00:39.801 INFO:tasks.workunit.client.0.smithi181.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock 2>&1 | tee ceph_test_rados_api_lock.log | sed "s/^/ api_lock: /"' 2021-05-25T12:00:39.802 INFO:tasks.workunit.client.0.smithi181.stderr:++ printf %25s api_lock_pp 2021-05-25T12:00:39.802 INFO:tasks.workunit.client.0.smithi181.stderr:+ r=' api_lock_pp' 2021-05-25T12:00:39.802 INFO:tasks.workunit.client.0.smithi181.stderr:++ echo api_lock_pp 2021-05-25T12:00:39.802 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph_test_rados_api_lock 2021-05-25T12:00:39.803 INFO:tasks.workunit.client.0.smithi181.stderr:+ tee ceph_test_rados_api_lock.log 2021-05-25T12:00:39.803 INFO:tasks.workunit.client.0.smithi181.stderr:+ sed 's/^/ api_lock: /' 2021-05-25T12:00:39.803 INFO:tasks.workunit.client.0.smithi181.stderr:++ awk '{print $1}' 2021-05-25T12:00:39.803 INFO:tasks.workunit.client.0.smithi181.stderr:+ ff=api_lock_pp 2021-05-25T12:00:39.804 INFO:tasks.workunit.client.0.smithi181.stdout:test api_lock_pp on pid 32921 2021-05-25T12:00:39.805 INFO:tasks.workunit.client.0.smithi181.stderr:+ pid=32921 2021-05-25T12:00:39.805 INFO:tasks.workunit.client.0.smithi181.stderr:+ echo 'test api_lock_pp on pid 32921' 2021-05-25T12:00:39.806 INFO:tasks.workunit.client.0.smithi181.stderr:+ pids[$f]=32921 2021-05-25T12:00:39.806 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2021-05-25T12:00:39.806 INFO:tasks.workunit.client.0.smithi181.stderr:+ '[' 1 -eq 1 ']' 2021-05-25T12:00:39.806 INFO:tasks.workunit.client.0.smithi181.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock_pp 2>&1 | tee ceph_test_rados_api_lock_pp.log | sed "s/^/ api_lock_pp: /"' 2021-05-25T12:00:39.807 INFO:tasks.workunit.client.0.smithi181.stderr:+ tee ceph_test_rados_api_lock_pp.log 2021-05-25T12:00:39.807 INFO:tasks.workunit.client.0.smithi181.stderr:+ sed 's/^/ api_lock_pp: /' 2021-05-25T12:00:39.807 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph_test_rados_api_lock_pp 2021-05-25T12:00:39.807 INFO:tasks.workunit.client.0.smithi181.stderr:++ printf %25s api_misc 2021-05-25T12:00:39.807 INFO:tasks.workunit.client.0.smithi181.stderr:+ r=' api_misc' 2021-05-25T12:00:39.808 INFO:tasks.workunit.client.0.smithi181.stderr:++ echo api_misc 2021-05-25T12:00:39.808 INFO:tasks.workunit.client.0.smithi181.stderr:++ awk '{print $1}' 2021-05-25T12:00:39.808 INFO:tasks.workunit.client.0.smithi181.stderr:+ ff=api_misc 2021-05-25T12:00:39.808 INFO:tasks.workunit.client.0.smithi181.stderr:+ pid=32971 2021-05-25T12:00:39.808 INFO:tasks.workunit.client.0.smithi181.stderr:+ echo 'test api_misc on pid 32971' 2021-05-25T12:00:39.809 INFO:tasks.workunit.client.0.smithi181.stdout:test api_misc on pid 32971 2021-05-25T12:00:39.810 INFO:tasks.workunit.client.0.smithi181.stderr:+ pids[$f]=32971 2021-05-25T12:00:39.811 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2021-05-25T12:00:39.811 INFO:tasks.workunit.client.0.smithi181.stderr:+ '[' 1 -eq 1 ']' 2021-05-25T12:00:39.811 INFO:tasks.workunit.client.0.smithi181.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc 2>&1 | tee ceph_test_rados_api_misc.log | sed "s/^/ api_misc: /"' 2021-05-25T12:00:39.811 INFO:tasks.workunit.client.0.smithi181.stderr:++ printf %25s api_misc_pp 2021-05-25T12:00:39.811 INFO:tasks.workunit.client.0.smithi181.stderr:+ r=' api_misc_pp' 2021-05-25T12:00:39.812 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph_test_rados_api_misc 2021-05-25T12:00:39.812 INFO:tasks.workunit.client.0.smithi181.stderr:+ tee ceph_test_rados_api_misc.log 2021-05-25T12:00:39.812 INFO:tasks.workunit.client.0.smithi181.stderr:+ sed 's/^/ api_misc: /' 2021-05-25T12:00:39.812 INFO:tasks.workunit.client.0.smithi181.stderr:++ echo api_misc_pp 2021-05-25T12:00:39.813 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: [==========] Running 12 tests from 1 test suite. 2021-05-25T12:00:39.814 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: [----------] Global test environment set-up. 2021-05-25T12:00:39.814 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: [----------] 12 tests from AsioRados 2021-05-25T12:00:39.814 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.721+0000 7ff88fe07540 10 monclient: get_monmap_and_config 2021-05-25T12:00:39.814 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.721+0000 7ff88fe07540 10 monclient: build_initial_monmap 2021-05-25T12:00:39.815 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.721+0000 7ff88fe07540 10 monclient: monmap: 2021-05-25T12:00:39.815 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: epoch 0 2021-05-25T12:00:39.815 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:00:39.815 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: last_changed 0.000000 2021-05-25T12:00:39.815 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: created 0.000000 2021-05-25T12:00:39.816 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: min_mon_release 0 (unknown) 2021-05-25T12:00:39.816 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: election_strategy: 1 2021-05-25T12:00:39.816 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:00:39.816 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:00:39.817 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:00:39.817 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.817 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.721+0000 7ff88fe07540 1 Processor -- start 2021-05-25T12:00:39.817 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.721+0000 7ff88fe07540 1 -- start start 2021-05-25T12:00:39.817 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.721+0000 7ff88fe07540 10 monclient: init 2021-05-25T12:00:39.818 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.722+0000 7ff88fe07540 10 monclient: _reopen_session rank -1 2021-05-25T12:00:39.818 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.722+0000 7ff88fe07540 10 monclient: _add_conns ranks=[2,0,1] 2021-05-25T12:00:39.818 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.722+0000 7ff88fe07540 10 monclient(hunting): picked mon.noname-c con 0x55fe52544870 addr v1:172.21.15.102:6790/0 2021-05-25T12:00:39.818 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.722+0000 7ff88fe07540 10 monclient(hunting): picked mon.noname-a con 0x55fe52566e20 addr v1:172.21.15.102:6789/0 2021-05-25T12:00:39.819 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.722+0000 7ff88fe07540 10 monclient(hunting): picked mon.noname-b con 0x55fe525fa920 addr v1:172.21.15.181:6789/0 2021-05-25T12:00:39.819 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.722+0000 7ff88fe07540 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x55fe525673e0 con 0x55fe52566e20 2021-05-25T12:00:39.819 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.722+0000 7ff88fe07540 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x55fe5253f860 con 0x55fe525fa920 2021-05-25T12:00:39.819 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.722+0000 7ff88fe07540 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x55fe525fc7f0 con 0x55fe52544870 2021-05-25T12:00:39.819 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.722+0000 7ff88fe07540 10 monclient(hunting): _renew_subs 2021-05-25T12:00:39.820 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.722+0000 7ff88fe07540 10 monclient(hunting): authenticate will time out at 839.206475s 2021-05-25T12:00:39.820 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.722+0000 7ff887a96700 1 --1- >> v1:172.21.15.181:6789/0 conn(0x55fe525fa920 0x55fe52606ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:34922/0 (socket says 172.21.15.181:34922) 2021-05-25T12:00:39.820 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.722+0000 7ff887a96700 1 -- 172.21.15.181:0/3620044669 learned_addr learned my addr 172.21.15.181:0/3620044669 (peer_addr_for_me v1:172.21.15.181:0/0) 2021-05-25T12:00:39.820 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.722+0000 7ff886293700 1 -- 172.21.15.181:0/3620044669 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1139637784 0 0) 0x55fe5253f860 con 0x55fe525fa920 2021-05-25T12:00:39.820 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.722+0000 7ff886293700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:39.821 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.722+0000 7ff886293700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:39.821 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.722+0000 7ff886293700 10 monclient(hunting): my global_id is 74167 2021-05-25T12:00:39.821 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.722+0000 7ff886293700 1 -- 172.21.15.181:0/3620044669 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff874002050 con 0x55fe525fa920 2021-05-25T12:00:39.821 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.722+0000 7ff886293700 1 -- 172.21.15.181:0/3620044669 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3210262149 0 0) 0x55fe525fc7f0 con 0x55fe52544870 2021-05-25T12:00:39.821 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.723+0000 7ff886293700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:39.822 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.723+0000 7ff886293700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:39.822 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.723+0000 7ff886293700 10 monclient(hunting): my global_id is 4508 2021-05-25T12:00:39.822 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.723+0000 7ff886293700 1 -- 172.21.15.181:0/3620044669 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x55fe5253f860 con 0x55fe52544870 2021-05-25T12:00:39.822 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.723+0000 7ff886293700 1 -- 172.21.15.181:0/3620044669 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3003422392 0 0) 0x55fe5253f860 con 0x55fe52544870 2021-05-25T12:00:39.823 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.723+0000 7ff886293700 1 -- 172.21.15.181:0/3620044669 >> v1:172.21.15.181:6789/0 conn(0x55fe525fa920 legacy=0x55fe52606ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:39.823 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.723+0000 7ff886293700 1 -- 172.21.15.181:0/3620044669 >> v1:172.21.15.102:6789/0 conn(0x55fe52566e20 legacy=0x55fe525fed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:39.823 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.723+0000 7ff886293700 10 monclient: _finish_hunting 0 2021-05-25T12:00:39.823 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.723+0000 7ff886293700 1 monclient: found mon.noname-c 2021-05-25T12:00:39.823 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.723+0000 7ff886293700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:00:39.824 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.723+0000 7ff886293700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.102:6790/0 2021-05-25T12:00:39.824 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.723+0000 7ff886293700 1 -- 172.21.15.181:0/3620044669 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fe525fd580 con 0x55fe52544870 2021-05-25T12:00:39.824 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.723+0000 7ff886293700 10 monclient: _finish_auth 0 2021-05-25T12:00:39.824 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.723+0000 7ff886293700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:00:39.824 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.723+0000 7ff886293700 1 -- 172.21.15.181:0/3620044669 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7ff870001550 con 0x55fe52544870 2021-05-25T12:00:39.825 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.723+0000 7ff886293700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:39.825 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.723+0000 7ff886293700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:00:39.825 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.723+0000 7ff886293700 10 monclient: dump: 2021-05-25T12:00:39.825 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: epoch 1 2021-05-25T12:00:39.825 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:39.826 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:39.826 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:39.826 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: min_mon_release 17 (quincy) 2021-05-25T12:00:39.826 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: election_strategy: 3 2021-05-25T12:00:39.827 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:39.827 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:39.827 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:39.827 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.827 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.723+0000 7ff88fe07540 5 monclient: authenticate success, global_id 4508 2021-05-25T12:00:39.828 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.723+0000 7ff88fe07540 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:00:39.828 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.723+0000 7ff886293700 1 -- 172.21.15.181:0/3620044669 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff870001770 con 0x55fe52544870 2021-05-25T12:00:39.828 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.723+0000 7ff886293700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:00:39.829 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.723+0000 7ff88fe07540 10 monclient: get_monmap_and_config success 2021-05-25T12:00:39.829 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.723+0000 7ff88fe07540 4 set_mon_vals no callback set 2021-05-25T12:00:39.829 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.723+0000 7ff886293700 1 -- 172.21.15.181:0/3620044669 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7ff870001b40 con 0x55fe52544870 2021-05-25T12:00:39.829 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.723+0000 7ff886293700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:39.829 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.723+0000 7ff886293700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:00:39.830 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.723+0000 7ff886293700 10 monclient: dump: 2021-05-25T12:00:39.830 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: epoch 1 2021-05-25T12:00:39.830 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:39.830 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:39.831 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:39.831 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: min_mon_release 17 (quincy) 2021-05-25T12:00:39.831 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: election_strategy: 3 2021-05-25T12:00:39.831 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:39.831 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:39.832 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:39.832 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.832 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.724+0000 7ff88fe07540 10 monclient: shutdown 2021-05-25T12:00:39.832 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.724+0000 7ff88fe07540 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:00:39.832 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.724+0000 7ff88fe07540 1 -- 172.21.15.181:0/3620044669 >> v1:172.21.15.102:6790/0 conn(0x55fe52544870 legacy=0x55fe525fe640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:39.833 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.724+0000 7ff88fe07540 1 -- 172.21.15.181:0/3620044669 shutdown_connections 2021-05-25T12:00:39.833 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.724+0000 7ff88fe07540 1 -- 172.21.15.181:0/3620044669 >> 172.21.15.181:0/3620044669 conn(0x55fe52564a30 msgr2=0x55fe525fbd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:00:39.833 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.724+0000 7ff88fe07540 1 -- 172.21.15.181:0/3620044669 shutdown_connections 2021-05-25T12:00:39.833 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.724+0000 7ff88fe07540 1 -- 172.21.15.181:0/3620044669 wait complete. 2021-05-25T12:00:39.833 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.724+0000 7ff88fe07540 10 monclient: build_initial_monmap 2021-05-25T12:00:39.834 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.724+0000 7ff88fe07540 10 monclient: monmap: 2021-05-25T12:00:39.834 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: epoch 0 2021-05-25T12:00:39.834 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:00:39.834 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: last_changed 0.000000 2021-05-25T12:00:39.835 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: created 0.000000 2021-05-25T12:00:39.835 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: min_mon_release 0 (unknown) 2021-05-25T12:00:39.835 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: election_strategy: 1 2021-05-25T12:00:39.835 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:00:39.835 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:00:39.836 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:00:39.836 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.836 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.724+0000 7ff88fe07540 1 librados: starting msgr at 2021-05-25T12:00:39.836 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.724+0000 7ff88fe07540 1 librados: starting objecter 2021-05-25T12:00:39.836 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.724+0000 7ff88fe07540 1 Processor -- start 2021-05-25T12:00:39.837 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.725+0000 7ff88fe07540 1 -- start start 2021-05-25T12:00:39.837 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.725+0000 7ff88fe07540 1 librados: setting wanted keys 2021-05-25T12:00:39.837 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.725+0000 7ff88fe07540 1 librados: calling monclient init 2021-05-25T12:00:39.837 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.725+0000 7ff88fe07540 10 monclient: init 2021-05-25T12:00:39.837 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.725+0000 7ff88fe07540 10 monclient: _reopen_session rank -1 2021-05-25T12:00:39.838 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.725+0000 7ff88fe07540 10 monclient: _add_conns ranks=[0,2,1] 2021-05-25T12:00:39.838 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.725+0000 7ff88fe07540 10 monclient(hunting): picked mon.noname-a con 0x55fe52544870 addr v1:172.21.15.102:6789/0 2021-05-25T12:00:39.838 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.725+0000 7ff88fe07540 10 monclient(hunting): picked mon.noname-c con 0x55fe52566e20 addr v1:172.21.15.102:6790/0 2021-05-25T12:00:39.838 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.725+0000 7ff88fe07540 10 monclient(hunting): picked mon.noname-b con 0x55fe525fa920 addr v1:172.21.15.181:6789/0 2021-05-25T12:00:39.838 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.725+0000 7ff88fe07540 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x55fe52540ff0 con 0x55fe52544870 2021-05-25T12:00:39.839 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.725+0000 7ff88fe07540 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x55fe5255eaa0 con 0x55fe525fa920 2021-05-25T12:00:39.839 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.725+0000 7ff88fe07540 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x55fe5255ed40 con 0x55fe52566e20 2021-05-25T12:00:39.839 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.725+0000 7ff88fe07540 10 monclient(hunting): _renew_subs 2021-05-25T12:00:39.839 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.725+0000 7ff88fe07540 10 monclient(hunting): authenticate will time out at 839.209682s 2021-05-25T12:00:39.840 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.725+0000 7ff87ea94700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x55fe52566e20 0x55fe525f2fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.181:53954/0 (socket says 172.21.15.181:53954) 2021-05-25T12:00:39.840 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.725+0000 7ff87ea94700 1 -- 172.21.15.181:0/4180131642 learned_addr learned my addr 172.21.15.181:0/4180131642 (peer_addr_for_me v1:172.21.15.181:0/0) 2021-05-25T12:00:39.840 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.726+0000 7ff884a90700 10 client.?.objecter ms_handle_connect 0x55fe52566e20 2021-05-25T12:00:39.840 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.726+0000 7ff884a90700 10 client.?.objecter resend_mon_ops 2021-05-25T12:00:39.840 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.726+0000 7ff884a90700 10 client.?.objecter ms_handle_connect 0x55fe52544870 2021-05-25T12:00:39.841 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.726+0000 7ff884a90700 10 client.?.objecter resend_mon_ops 2021-05-25T12:00:39.841 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.726+0000 7ff884a90700 10 client.?.objecter ms_handle_connect 0x55fe525fa920 2021-05-25T12:00:39.841 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.726+0000 7ff884a90700 10 client.?.objecter resend_mon_ops 2021-05-25T12:00:39.841 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.726+0000 7ff884a90700 1 -- 172.21.15.181:0/4180131642 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1828047852 0 0) 0x55fe5255ed40 con 0x55fe52566e20 2021-05-25T12:00:39.841 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.726+0000 7ff884a90700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:39.842 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.726+0000 7ff884a90700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:39.842 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.726+0000 7ff884a90700 10 monclient(hunting): my global_id is 4511 2021-05-25T12:00:39.842 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.726+0000 7ff884a90700 1 -- 172.21.15.181:0/4180131642 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff874001e10 con 0x55fe52566e20 2021-05-25T12:00:39.842 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.726+0000 7ff884a90700 1 -- 172.21.15.181:0/4180131642 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2058279365 0 0) 0x55fe5255eaa0 con 0x55fe525fa920 2021-05-25T12:00:39.842 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.726+0000 7ff884a90700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:39.843 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.726+0000 7ff884a90700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:39.843 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.726+0000 7ff884a90700 10 monclient(hunting): my global_id is 74170 2021-05-25T12:00:39.843 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.726+0000 7ff884a90700 1 -- 172.21.15.181:0/4180131642 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x55fe5255ed40 con 0x55fe525fa920 2021-05-25T12:00:39.843 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.726+0000 7ff884a90700 1 -- 172.21.15.181:0/4180131642 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1575801615 0 0) 0x7ff874001e10 con 0x55fe52566e20 2021-05-25T12:00:39.844 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.726+0000 7ff884a90700 1 -- 172.21.15.181:0/4180131642 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x55fe5255eaa0 con 0x55fe52566e20 2021-05-25T12:00:39.844 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.726+0000 7ff884a90700 1 -- 172.21.15.181:0/4180131642 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2242085628 0 0) 0x55fe5255ed40 con 0x55fe525fa920 2021-05-25T12:00:39.844 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.726+0000 7ff884a90700 1 -- 172.21.15.181:0/4180131642 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff874001e10 con 0x55fe525fa920 2021-05-25T12:00:39.844 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.726+0000 7ff884a90700 1 -- 172.21.15.181:0/4180131642 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7ff878002d20 con 0x55fe52566e20 2021-05-25T12:00:39.844 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.726+0000 7ff884a90700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:39.845 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.726+0000 7ff884a90700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:00:39.845 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.726+0000 7ff884a90700 10 monclient(hunting): dump: 2021-05-25T12:00:39.845 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: epoch 1 2021-05-25T12:00:39.845 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:39.845 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:39.846 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:39.846 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: min_mon_release 17 (quincy) 2021-05-25T12:00:39.846 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: election_strategy: 3 2021-05-25T12:00:39.846 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:39.846 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:39.847 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:39.847 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.847 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.726+0000 7ff884a90700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:00:39.847 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.726+0000 7ff884a90700 1 -- 172.21.15.181:0/4180131642 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7ff880001370 con 0x55fe525fa920 2021-05-25T12:00:39.848 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.726+0000 7ff884a90700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:39.848 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.726+0000 7ff884a90700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2021-05-25T12:00:39.848 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.726+0000 7ff884a90700 10 monclient(hunting): dump: 2021-05-25T12:00:39.848 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: epoch 1 2021-05-25T12:00:39.848 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:39.849 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:39.849 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:39.849 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: min_mon_release 17 (quincy) 2021-05-25T12:00:39.849 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: election_strategy: 3 2021-05-25T12:00:39.849 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:39.850 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:39.850 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:39.850 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.850 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.726+0000 7ff884a90700 1 -- 172.21.15.181:0/4180131642 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3329707765 0 0) 0x55fe5255eaa0 con 0x55fe52566e20 2021-05-25T12:00:39.851 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.726+0000 7ff884a90700 1 -- 172.21.15.181:0/4180131642 >> v1:172.21.15.181:6789/0 conn(0x55fe525fa920 legacy=0x55fe5260dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:39.851 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.726+0000 7ff884a90700 1 -- 172.21.15.181:0/4180131642 >> v1:172.21.15.102:6789/0 conn(0x55fe52544870 legacy=0x55fe525f2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:39.851 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.726+0000 7ff884a90700 10 monclient: _finish_hunting 0 2021-05-25T12:00:39.851 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.726+0000 7ff884a90700 1 monclient: found mon.c 2021-05-25T12:00:39.851 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.727+0000 7ff884a90700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:00:39.852 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.727+0000 7ff884a90700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:00:39.852 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.727+0000 7ff884a90700 1 -- 172.21.15.181:0/4180131642 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fe525f9e80 con 0x55fe52566e20 2021-05-25T12:00:39.852 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.727+0000 7ff884a90700 10 monclient: _finish_auth 0 2021-05-25T12:00:39.852 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.727+0000 7ff884a90700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:00:39.852 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.727+0000 7ff88fe07540 5 monclient: authenticate success, global_id 4511 2021-05-25T12:00:39.853 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.727+0000 7ff88fe07540 10 monclient: _renew_subs 2021-05-25T12:00:39.853 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.727+0000 7ff88fe07540 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:00:39.853 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.727+0000 7ff88fe07540 1 -- 172.21.15.181:0/4180131642 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x55fe525f93b0 con 0x55fe52566e20 2021-05-25T12:00:39.853 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.727+0000 7ff88fe07540 10 client.4511.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:00:39.854 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.727+0000 7ff88fe07540 10 monclient: _renew_subs 2021-05-25T12:00:39.854 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.727+0000 7ff88fe07540 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:00:39.854 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.727+0000 7ff88fe07540 1 -- 172.21.15.181:0/4180131642 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x55fe5255f6f0 con 0x55fe52566e20 2021-05-25T12:00:39.854 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.727+0000 7ff88fe07540 1 librados: init done 2021-05-25T12:00:39.854 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.727+0000 7ff88fe07540 10 librados: wait_for_osdmap waiting 2021-05-25T12:00:39.855 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.727+0000 7ff884a90700 1 -- 172.21.15.181:0/4180131642 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff878001780 con 0x55fe52566e20 2021-05-25T12:00:39.855 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.727+0000 7ff884a90700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:00:39.855 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.727+0000 7ff884a90700 1 -- 172.21.15.181:0/4180131642 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7ff878001a90 con 0x55fe52566e20 2021-05-25T12:00:39.855 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.727+0000 7ff884a90700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:39.855 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.727+0000 7ff884a90700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:00:39.856 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.727+0000 7ff886293700 4 set_mon_vals no callback set 2021-05-25T12:00:39.856 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.727+0000 7ff884a90700 10 monclient: dump: 2021-05-25T12:00:39.856 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: epoch 1 2021-05-25T12:00:39.856 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:39.857 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:39.857 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:39.857 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: min_mon_release 17 (quincy) 2021-05-25T12:00:39.857 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: election_strategy: 3 2021-05-25T12:00:39.857 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:39.858 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:39.858 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:39.858 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.858 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.727+0000 7ff884a90700 1 -- 172.21.15.181:0/4180131642 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7ff878001c70 con 0x55fe52566e20 2021-05-25T12:00:39.859 INFO:tasks.workunit.client.0.smithi181.stderr:++ awk '{print $1}' 2021-05-25T12:00:39.859 INFO:tasks.workunit.client.0.smithi181.stderr:+ ff=api_misc_pp 2021-05-25T12:00:39.860 INFO:tasks.workunit.client.0.smithi181.stderr:+ pid=32996 2021-05-25T12:00:39.860 INFO:tasks.workunit.client.0.smithi181.stderr:+ echo 'test api_misc_pp on pid 32996' 2021-05-25T12:00:39.860 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.728+0000 7ff884a90700 1 -- 172.21.15.181:0/4180131642 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(15..15 src has 1..15) v4 ==== 3549+0+0 (unknown 3948819498 0 0) 0test api_misc_pp on pid 32996 2021-05-25T12:00:39.862 INFO:tasks.workunit.client.0.smithi181.stderr:+ pids[$f]=32996 2021-05-25T12:00:39.862 INFO:tasks.workunit.client.0.smithi181.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc_pp 2>&1 | tee ceph_test_rados_api_misc_pp.log | sed "s/^/ api_misc_pp: /"' 2021-05-25T12:00:39.862 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2021-05-25T12:00:39.863 INFO:tasks.workunit.client.0.smithi181.stderr:+ '[' 1 -eq 1 ']' 2021-05-25T12:00:39.863 INFO:tasks.workunit.client.0.smithi181.stderr:++ printf %25s api_tier_pp 2021-05-25T12:00:39.863 INFO:tasks.workunit.client.0.smithi181.stderr:+ r=' api_tier_pp' 2021-05-25T12:00:39.863 INFO:tasks.workunit.client.0.smithi181.stderr:++ echo api_tier_pp 2021-05-25T12:00:39.863 INFO:tasks.workunit.client.0.smithi181.stderr:++ awk '{print $1}' 2021-05-25T12:00:39.864 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph_test_rados_api_misc_pp 2021-05-25T12:00:39.864 INFO:tasks.workunit.client.0.smithi181.stderr:+ tee ceph_test_rados_api_misc_pp.log 2021-05-25T12:00:39.864 INFO:tasks.workunit.client.0.smithi181.stderr:+ ff=api_tier_pp 2021-05-25T12:00:39.864 INFO:tasks.workunit.client.0.smithi181.stderr:+ pid=33004 2021-05-25T12:00:39.865 INFO:tasks.workunit.client.0.smithi181.stdout:test api_tier_pp on pid 33004 2021-05-25T12:00:39.867 INFO:tasks.workunit.client.0.smithi181.stderr:+ echo 'test api_tier_pp on pid 33004' 2021-05-25T12:00:39.867 INFO:tasks.workunit.client.0.smithi181.stderr:+ pids[$f]=33004 2021-05-25T12:00:39.867 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2021-05-25T12:00:39.867 INFO:tasks.workunit.client.0.smithi181.stderr:+ '[' 1 -eq 1 ']' 2021-05-25T12:00:39.867 INFO:tasks.workunit.client.0.smithi181.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_tier_pp 2>&1 | tee ceph_test_rados_api_tier_pp.log | sed "s/^/ api_tier_pp: /"' 2021-05-25T12:00:39.868 INFO:tasks.workunit.client.0.smithi181.stderr:++ printf %25s api_pool 2021-05-25T12:00:39.868 INFO:tasks.workunit.client.0.smithi181.stderr:+ r=' api_pool' 2021-05-25T12:00:39.868 INFO:tasks.workunit.client.0.smithi181.stderr:+ sed 's/^/ api_misc_pp: /' 2021-05-25T12:00:39.868 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph_test_rados_api_tier_pp 2021-05-25T12:00:39.869 INFO:tasks.workunit.client.0.smithi181.stderr:+ sed 's/^/ api_tier_pp: /' 2021-05-25T12:00:39.869 INFO:tasks.workunit.client.0.smithi181.stderr:++ echo api_pool 2021-05-25T12:00:39.869 INFO:tasks.workunit.client.0.smithi181.stderr:++ awk '{print $1}' 2021-05-25T12:00:39.869 INFO:tasks.workunit.client.0.smithi181.stderr:+ ff=api_pool 2021-05-25T12:00:39.869 INFO:tasks.workunit.client.0.smithi181.stderr:+ pid=33041 2021-05-25T12:00:39.870 INFO:tasks.workunit.client.0.smithi181.stderr:+ tee ceph_test_rados_api_tier_pp.log 2021-05-25T12:00:39.870 INFO:tasks.workunit.client.0.smithi181.stderr:+ echo 'test api_pool on pid 33041' 2021-05-25T12:00:39.870 INFO:tasks.workunit.client.0.smithi181.stdout:test api_pool on pid 33041 2021-05-25T12:00:39.871 INFO:tasks.workunit.client.0.smithi181.stderr:+ pids[$f]=33041 2021-05-25T12:00:39.872 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2021-05-25T12:00:39.872 INFO:tasks.workunit.client.0.smithi181.stderr:+ '[' 1 -eq 1 ']' 2021-05-25T12:00:39.872 INFO:tasks.workunit.client.0.smithi181.stderr:++ printf %25s api_snapshots 2021-05-25T12:00:39.872 INFO:tasks.workunit.client.0.smithi181.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_pool 2>&1 | tee ceph_test_rados_api_pool.log | sed "s/^/ api_pool: /"' 2021-05-25T12:00:39.873 INFO:tasks.workunit.client.0.smithi181.stderr:+ r=' api_snapshots' 2021-05-25T12:00:39.873 INFO:tasks.workunit.client.0.smithi181.stderr:++ echo api_snapshots 2021-05-25T12:00:39.873 INFO:tasks.workunit.client.0.smithi181.stderr:++ awk '{print $1}' 2021-05-25T12:00:39.873 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph_test_rados_api_pool 2021-05-25T12:00:39.874 INFO:tasks.workunit.client.0.smithi181.stdout:test api_snapshots on pid 33061 2021-05-25T12:00:39.875 INFO:tasks.workunit.client.0.smithi181.stderr:+ ff=api_snapshots 2021-05-25T12:00:39.875 INFO:tasks.workunit.client.0.smithi181.stderr:+ pid=33061 2021-05-25T12:00:39.875 INFO:tasks.workunit.client.0.smithi181.stderr:+ echo 'test api_snapshots on pid 33061' 2021-05-25T12:00:39.876 INFO:tasks.workunit.client.0.smithi181.stderr:+ pids[$f]=33061 2021-05-25T12:00:39.876 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2021-05-25T12:00:39.876 INFO:tasks.workunit.client.0.smithi181.stderr:+ '[' 1 -eq 1 ']' 2021-05-25T12:00:39.876 INFO:tasks.workunit.client.0.smithi181.stderr:++ printf %25s api_snapshots_pp 2021-05-25T12:00:39.877 INFO:tasks.workunit.client.0.smithi181.stderr:+ r=' api_snapshots_pp' 2021-05-25T12:00:39.877 INFO:tasks.workunit.client.0.smithi181.stderr:++ awk '{print $1}' 2021-05-25T12:00:39.877 INFO:tasks.workunit.client.0.smithi181.stderr:+ tee ceph_test_rados_api_pool.log 2021-05-25T12:00:39.877 INFO:tasks.workunit.client.0.smithi181.stderr:+ sed 's/^/ api_pool: /' 2021-05-25T12:00:39.877 INFO:tasks.workunit.client.0.smithi181.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots 2>&1 | tee ceph_test_rados_api_snapshots.log | sed "s/^/ api_snapshots: /"' 2021-05-25T12:00:39.878 INFO:tasks.workunit.client.0.smithi181.stderr:++ echo api_snapshots_pp 2021-05-25T12:00:39.878 INFO:tasks.workunit.client.0.smithi181.stdout:test api_snapshots_pp on pid 33067 2021-05-25T12:00:39.880 INFO:tasks.workunit.client.0.smithi181.stderr:+ ff=api_snapshots_pp 2021-05-25T12:00:39.880 INFO:tasks.workunit.client.0.smithi181.stderr:+ pid=33067 2021-05-25T12:00:39.880 INFO:tasks.workunit.client.0.smithi181.stderr:+ echo 'test api_snapshots_pp on pid 33067' 2021-05-25T12:00:39.881 INFO:tasks.workunit.client.0.smithi181.stderr:+ pids[$f]=33067 2021-05-25T12:00:39.881 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2021-05-25T12:00:39.881 INFO:tasks.workunit.client.0.smithi181.stderr:+ '[' 1 -eq 1 ']' 2021-05-25T12:00:39.881 INFO:tasks.workunit.client.0.smithi181.stderr:++ printf %25s api_stat 2021-05-25T12:00:39.881 INFO:tasks.workunit.client.0.smithi181.stderr:+ sed 's/^/ api_snapshots: /' 2021-05-25T12:00:39.882 INFO:tasks.workunit.client.0.smithi181.stderr:+ r=' api_stat' 2021-05-25T12:00:39.882 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph_test_rados_api_snapshots 2021-05-25T12:00:39.882 INFO:tasks.workunit.client.0.smithi181.stderr:++ echo api_stat 2021-05-25T12:00:39.882 INFO:tasks.workunit.client.0.smithi181.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots_pp 2>&1 | tee ceph_test_rados_api_snapshots_pp.log | sed "s/^/ api_snapshots_pp: /"' 2021-05-25T12:00:39.883 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph_test_rados_api_snapshots_pp 2021-05-25T12:00:39.883 INFO:tasks.workunit.client.0.smithi181.stderr:+ sed 's/^/ api_snapshots_pp: /' 2021-05-25T12:00:39.883 INFO:tasks.workunit.client.0.smithi181.stderr:+ tee ceph_test_rados_api_snapshots_pp.log 2021-05-25T12:00:39.883 INFO:tasks.workunit.client.0.smithi181.stderr:++ awk '{print $1}' 2021-05-25T12:00:39.883 INFO:tasks.workunit.client.0.smithi181.stderr:+ ff=api_stat 2021-05-25T12:00:39.884 INFO:tasks.workunit.client.0.smithi181.stderr:+ pid=33090 2021-05-25T12:00:39.884 INFO:tasks.workunit.client.0.smithi181.stderr:+ echo 'test api_stat on pid 33090' 2021-05-25T12:00:39.884 INFO:tasks.workunit.client.0.smithi181.stdout:test api_stat on pid 33090 2021-05-25T12:00:39.886 INFO:tasks.workunit.client.0.smithi181.stderr:+ pids[$f]=33090 2021-05-25T12:00:39.886 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2021-05-25T12:00:39.886 INFO:tasks.workunit.client.0.smithi181.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat 2>&1 | tee ceph_test_rados_api_stat.log | sed "s/^/ api_stat: /"' 2021-05-25T12:00:39.886 INFO:tasks.workunit.client.0.smithi181.stderr:+ '[' 1 -eq 1 ']' 2021-05-25T12:00:39.887 INFO:tasks.workunit.client.0.smithi181.stderr:++ printf %25s api_stat_pp 2021-05-25T12:00:39.887 INFO:tasks.workunit.client.0.smithi181.stderr:+ r=' api_stat_pp' 2021-05-25T12:00:39.887 INFO:tasks.workunit.client.0.smithi181.stderr:++ awk '{print $1}' 2021-05-25T12:00:39.887 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph_test_rados_api_stat 2021-05-25T12:00:39.887 INFO:tasks.workunit.client.0.smithi181.stderr:+ tee ceph_test_rados_api_stat.log 2021-05-25T12:00:39.888 INFO:tasks.workunit.client.0.smithi181.stderr:+ tee ceph_test_rados_api_snapshots.log 2021-05-25T12:00:39.888 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: [==========] Running 11 tests from 4 test suites. 2021-05-25T12:00:39.888 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: [----------] Global test environment set-up. 2021-05-25T12:00:39.889 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion 2021-05-25T12:00:39.889 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: [ RUN ] LibRadosMiscVersion.Version 2021-05-25T12:00:39.889 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: [ OK ] LibRadosMiscVersion.Version (0 ms) 2021-05-25T12:00:39.889 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2021-05-25T12:00:39.889 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.890 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: [----------] 1 test from LibRadosMiscConnectFailure 2021-05-25T12:00:39.890 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectFailure 2021-05-25T12:00:39.890 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 0 2021-05-25T12:00:39.890 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.753+0000 7f4d7a69b540 10 monclient: get_monmap_and_config 2021-05-25T12:00:39.890 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.753+0000 7f4d7a69b540 10 monclient: build_initial_monmap 2021-05-25T12:00:39.891 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.753+0000 7f4d7a69b540 10 monclient: monmap: 2021-05-25T12:00:39.891 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: epoch 0 2021-05-25T12:00:39.891 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:39.891 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: last_changed 2021-05-25T12:00:39.755383+0000 2021-05-25T12:00:39.892 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: created 2021-05-25T12:00:39.755383+0000 2021-05-25T12:00:39.892 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: min_mon_release 0 (unknown) 2021-05-25T12:00:39.892 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: election_strategy: 3 2021-05-25T12:00:39.892 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:00:39.892 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:00:39.893 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:00:39.893 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.893 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.757+0000 7f4d7a69b540 1 Processor -- start 2021-05-25T12:00:39.893 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.757+0000 7f4d7a69b540 1 -- start start 2021-05-25T12:00:39.893 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.757+0000 7f4d7a69b540 10 monclient: init 2021-05-25T12:00:39.894 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.758+0000 7f4d7a69b540 10 monclient: _reopen_session rank -1 2021-05-25T12:00:39.894 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.758+0000 7f4d7a69b540 10 monclient: _add_conns ranks=[1,2,0] 2021-05-25T12:00:39.894 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.758+0000 7f4d7a69b540 10 monclient(hunting): picked mon.noname-c con 0x55ddff636f60 addr v1:172.21.15.181:6789/0 2021-05-25T12:00:39.894 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.758+0000 7f4d7a69b540 10 monclient(hunting): picked mon.noname-b con 0x55ddff63d300 addr v1:172.21.15.102:6790/0 2021-05-25T12:00:39.895 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.758+0000 7f4d7a69b540 10 monclient(hunting): picked mon.noname-a con 0x55ddff63d7b0 addr v1:172.21.15.102:6789/0 2021-05-25T12:00:39.895 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.758+0000 7f4d7a69b540 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x55ddff562a10 con 0x55ddff63d7b0 2021-05-25T12:00:39.895 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.758+0000 7f4d7a69b540 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x55ddff546410 con 0x55ddff636f60 2021-05-25T12:00:39.895 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.758+0000 7f4d7a69b540 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x55ddff637e10 con 0x55ddff63d300 2021-05-25T12:00:39.895 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.758+0000 7f4d7a69b540 10 monclient(hunting): _renew_subs 2021-05-25T12:00:39.896 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.758+0000 7f4d7a69b540 10 monclient(hunting): authenticate will time out at 539.242682s 2021-05-25T12:00:39.896 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.759+0000 7f4d6b7fe700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x55ddff63d300 0x55ddff701110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.181:54004/0 (socket says 172.21.15.181:54004) 2021-05-25T12:00:39.896 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.759+0000 7f4d6bfff700 1 --1- >> v1:172.21.15.181:6789/0 conn(0x55ddff636f60 0x55ddff6f8e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:34978/0 (socket says 172.21.15.181:34978) 2021-05-25T12:00:39.897 INFO:tasks.workunit.client.0.smithi181.stderr:+ sed 's/^/ api_stat: /' 2021-05-25T12:00:39.898 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.759+0000 7f4d6b7fe700 1 -- 172.21.15.181:0/149592805 learned_addr learned my addr 172.21.15.181:0/149592805 (peer_addr_for_me v1:172.21.15.181:0/0) 2021-05-25T12:00:39.898 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.759+0000 7f4d7a69b540 0 monclient(hunting): authenticate timed out after 1e-09 2021-05-25T12:00:39.899 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.759+0000 7f4d7a69b540 10 monclient(hunting): shutdown 2021-05-25T12:00:39.899 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.759+0000 7f4d7a69b540 20 monclient(hunting): shutdown discarding 1 pending message(s) 2021-05-25T12:00:39.899 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.759+0000 7f4d7a69b540 1 -- 172.21.15.181:0/149592805 >> v1:172.21.15.102:6790/0 conn(0x55ddff63d300 legacy=0x55ddff701110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:39.899 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.759+0000 7f4d7a69b540 1 -- 172.21.15.181:0/149592805 >> v1:172.21.15.181:6789/0 conn(0x55ddff636f60 legacy=0x55ddff6f8e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:39.900 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.759+0000 7f4d7a69b540 1 -- 172.21.15.181:0/149592805 >> v1:172.21.15.102:6789/0 conn(0x55ddff63d7b0 legacy=0x55ddff704840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-05-25T12:00:39.900 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.759+0000 7f4d7a69b540 10 monclient: init 2021-05-25T12:00:39.900 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.760+0000 7f4d7a69b540 10 monclient: _reopen_session rank -1 2021-05-25T12:00:39.900 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.760+0000 7f4d7a69b540 10 monclient: _add_conns ranks=[1,2,0] 2021-05-25T12:00:39.901 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.760+0000 7f4d7a69b540 10 monclient(hunting): picked mon.noname-c con 0x55ddff6fb9a0 addr v1:172.21.15.181:6789/0 2021-05-25T12:00:39.901 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.760+0000 7f4d7a69b540 10 monclient(hunting): picked mon.noname-b con 0x55ddff66abc0 addr v1:172.21.15.102:6790/0 2021-05-25T12:00:39.901 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.760+0000 7f4d7a69b540 10 monclient(hunting): picked mon.noname-a con 0x55ddff63d7b0 addr v1:172.21.15.102:6789/0 2021-05-25T12:00:39.901 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.760+0000 7f4d7a69b540 1 -- 172.21.15.181:0/149592805 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x55ddff562a10 con 0x55ddff63d7b0 2021-05-25T12:00:39.901 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.760+0000 7f4d7a69b540 1 -- 172.21.15.181:0/149592805 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x55ddff637e10 con 0x55ddff6fb9a0 2021-05-25T12:00:39.902 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.760+0000 7f4d7a69b540 1 -- 172.21.15.181:0/149592805 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x55ddff637630 con 0x55ddff66abc0 2021-05-25T12:00:39.902 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.760+0000 7f4d7a69b540 10 monclient(hunting): _renew_subs 2021-05-25T12:00:39.902 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.760+0000 7f4d7a69b540 10 monclient(hunting): authenticate will time out at 539.244433s 2021-05-25T12:00:39.902 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.761+0000 7f4d7a69b540 0 monclient(hunting): authenticate timed out after 1e-09 2021-05-25T12:00:39.902 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.761+0000 7f4d7a69b540 10 monclient(hunting): shutdown 2021-05-25T12:00:39.903 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.761+0000 7f4d7a69b540 20 monclient(hunting): shutdown discarding 1 pending message(s) 2021-05-25T12:00:39.903 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.761+0000 7f4d7a69b540 1 -- 172.21.15.181:0/149592805 >> v1:172.21.15.102:6790/0 conn(0x55ddff66abc0 legacy=0x55ddff707680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:39.903 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.761+0000 7f4d7a69b540 1 -- 172.21.15.181:0/149592805 >> v1:172.21.15.181:6789/0 conn(0x55ddff6fb9a0 legacy=0x55ddff706f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:39.903 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.761+0000 7f4d7a69b540 1 -- 172.21.15.181:0/149592805 >> v1:172.21.15.102:6789/0 conn(0x55ddff63d7b0 legacy=0x55ddff704840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-05-25T12:00:39.904 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.761+0000 7f4d7a69b540 10 monclient: init 2021-05-25T12:00:39.904 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.761+0000 7f4d7a69b540 10 monclient: _reopen_session rank -1 2021-05-25T12:00:39.904 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.761+0000 7f4d7a69b540 10 monclient: _add_conns ranks=[0,1,2] 2021-05-25T12:00:39.904 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.761+0000 7f4d7a69b540 10 monclient(hunting): picked mon.noname-a con 0x55ddff63d7b0 addr v1:172.21.15.102:6789/0 2021-05-25T12:00:39.904 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.761+0000 7f4d7a69b540 10 monclient(hunting): picked mon.noname-c con 0x55ddff6fb9a0 addr v1:172.21.15.181:6789/0 2021-05-25T12:00:39.905 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.761+0000 7f4d7a69b540 10 monclient(hunting): picked mon.noname-b con 0x55ddff66abc0 addr v1:172.21.15.102:6790/0 2021-05-25T12:00:39.905 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.761+0000 7f4d7a69b540 1 -- 172.21.15.181:0/149592805 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x55ddff562a10 con 0x55ddff63d7b0 2021-05-25T12:00:39.905 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.761+0000 7f4d7a69b540 1 -- 172.21.15.181:0/149592805 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x55ddff637e10 con 0x55ddff6fb9a0 2021-05-25T12:00:39.905 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.761+0000 7f4d7a69b540 1 -- 172.21.15.181:0/149592805 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x55ddff637630 con 0x55ddff66abc0 2021-05-25T12:00:39.905 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.761+0000 7f4d7a69b540 10 monclient(hunting): _renew_subs 2021-05-25T12:00:39.906 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.761+0000 7f4d7a69b540 10 monclient(hunting): authenticate will time out at 539.245712s 2021-05-25T12:00:39.906 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.761+0000 7f4d7a69b540 0 monclient(hunting): authenticate timed out after 1e-09 2021-05-25T12:00:39.906 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.761+0000 7f4d7a69b540 10 monclient(hunting): shutdown 2021-05-25T12:00:39.906 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.761+0000 7f4d7a69b540 20 monclient(hunting): shutdown discarding 1 pending message(s) 2021-05-25T12:00:39.907 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.761+0000 7f4d7a69b540 1 -- 172.21.15.181:0/149592805 >> v1:172.21.15.102:6790/0 conn(0x55ddff66abc0 legacy=0x55ddff707580 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-05-25T12:00:39.907 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.761+0000 7f4d7a69b540 1 -- 172.21.15.181:0/149592805 >> v1:172.21.15.181:6789/0 conn(0x55ddff6fb9a0 legacy=0x55ddff706e70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-05-25T12:00:39.907 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.762+0000 7f4d7a69b540 1 -- 172.21.15.181:0/149592805 >> v1:172.21.15.102:6789/0 conn(0x55ddff63d7b0 legacy=0x55ddff6f81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:39.907 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.762+0000 7f4d7a69b540 10 monclient: init 2021-05-25T12:00:39.907 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.762+0000 7f4d7a69b540 10 monclient: _reopen_session rank -1 2021-05-25T12:00:39.908 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.762+0000 7f4d7a69b540 10 monclient: _add_conns ranks=[2,0,1] 2021-05-25T12:00:39.908 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.762+0000 7f4d7a69b540 10 monclient(hunting): picked mon.noname-b con 0x55ddff66abc0 addr v1:172.21.15.102:6790/0 2021-05-25T12:00:39.908 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.762+0000 7f4d7a69b540 10 monclient(hunting): picked mon.noname-a con 0x55ddff63d7b0 addr v1:172.21.15.102:6789/0 2021-05-25T12:00:39.908 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.762+0000 7f4d7a69b540 10 monclient(hunting): picked mon.noname-c con 0x55ddff6fb9a0 addr v1:172.21.15.181:6789/0 2021-05-25T12:00:39.909 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.762+0000 7f4d7a69b540 1 -- 172.21.15.181:0/149592805 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x55ddff637630 con 0x55ddff63d7b0 2021-05-25T12:00:39.909 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.762+0000 7f4d7a69b540 1 -- 172.21.15.181:0/149592805 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x55ddff725a40 con 0x55ddff6fb9a0 2021-05-25T12:00:39.909 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.762+0000 7f4d7a69b540 1 -- 172.21.15.181:0/149592805 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x55ddff6f8d10 con 0x55ddff66abc0 2021-05-25T12:00:39.909 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.762+0000 7f4d7a69b540 10 monclient(hunting): _renew_subs 2021-05-25T12:00:39.909 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.762+0000 7f4d7a69b540 10 monclient(hunting): authenticate will time out at 539.246792s 2021-05-25T12:00:39.910 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.762+0000 7f4d7a69b540 0 monclient(hunting): authenticate timed out after 1e-09 2021-05-25T12:00:39.910 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.762+0000 7f4d7a69b540 10 monclient(hunting): shutdown 2021-05-25T12:00:39.910 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.762+0000 7f4d7a69b540 20 monclient(hunting): shutdown discarding 1 pending message(s) 2021-05-25T12:00:39.910 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.762+0000 7f4d7a69b540 1 -- 172.21.15.181:0/149592805 >> v1:172.21.15.102:6790/0 conn(0x55ddff66abc0 legacy=0x55ddff707580 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-05-25T12:00:39.911 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.762+0000 7f4d7a69b540 1 -- 172.21.15.181:0/149592805 >> v1:172.21.15.181:6789/0 conn(0x55ddff6fb9a0 legacy=0x55ddff706e70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-05-25T12:00:39.911 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.762+0000 7f4d7a69b540 1 -- 172.21.15.181:0/149592805 >> v1:172.21.15.102:6789/0 conn(0x55ddff63d7b0 legacy=0x55ddff6f80a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-05-25T12:00:39.911 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.763+0000 7f4d7a69b540 10 monclient: init 2021-05-25T12:00:39.911 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.763+0000 7f4d7a69b540 10 monclient: _reopen_session rank -1 2021-05-25T12:00:39.911 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.763+0000 7f4d7a69b540 10 monclient: _add_conns ranks=[2,0,1] 2021-05-25T12:00:39.912 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.763+0000 7f4d7a69b540 10 monclient(hunting): picked mon.noname-b con 0x55ddff66abc0 addr v1:172.21.15.102:6790/0 2021-05-25T12:00:39.912 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.763+0000 7f4d7a69b540 10 monclient(hunting): picked mon.noname-a con 0x55ddff63d7b0 addr v1:172.21.15.102:6789/0 2021-05-25T12:00:39.912 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.763+0000 7f4d7a69b540 10 monclient(hunting): picked mon.noname-c con 0x55ddff6fb9a0 addr v1:172.21.15.181:6789/0 2021-05-25T12:00:39.912 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.763+0000 7f4d7a69b540 1 -- 172.21.15.181:0/149592805 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x55ddff637630 con 0x55ddff63d7b0 2021-05-25T12:00:39.912 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.763+0000 7f4d7a69b540 1 -- 172.21.15.181:0/149592805 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x55ddff725a40 con 0x55ddff6fb9a0 2021-05-25T12:00:39.913 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.763+0000 7f4d7a69b540 1 -- 172.21.15.181:0/149592805 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x55ddff6f8d10 con 0x55ddff66abc0 2021-05-25T12:00:39.913 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.763+0000 7f4d7a69b540 10 monclient(hunting): _renew_subs 2021-05-25T12:00:39.913 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.763+0000 7f4d7a69b540 10 monclient(hunting): authenticate will time out at 539.247454s 2021-05-25T12:00:39.913 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.763+0000 7f4d7a69b540 0 monclient(hunting): authenticate timed out after 1e-09 2021-05-25T12:00:39.914 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.763+0000 7f4d7a69b540 10 monclient(hunting): shutdown 2021-05-25T12:00:39.914 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.763+0000 7f4d7a69b540 20 monclient(hunting): shutdown discarding 1 pending message(s) 2021-05-25T12:00:39.914 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.763+0000 7f4d7a69b540 1 -- 172.21.15.181:0/149592805 >> v1:172.21.15.102:6790/0 conn(0x55ddff66abc0 legacy=0x55ddff721e90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-05-25T12:00:39.914 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.763+0000 7f4d7a69b540 1 -- 172.21.15.181:0/149592805 >> v1:172.21.15.181:6789/0 conn(0x55ddff6fb9a0 legacy=0x55ddff66b000 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-05-25T12:00:39.914 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.763+0000 7f4d7a69b540 1 -- 172.21.15.181:0/149592805 >> v1:172.21.15.102:6789/0 conn(0x55ddff63d7b0 legacy=0x55ddff6f80a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-05-25T12:00:39.915 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.763+0000 7f4d7a69b540 10 monclient: init 2021-05-25T12:00:39.915 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.764+0000 7f4d7a69b540 10 monclient: _reopen_session rank -1 2021-05-25T12:00:39.915 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.764+0000 7f4d7a69b540 10 monclient: _add_conns ranks=[2,1,0] 2021-05-25T12:00:39.915 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.764+0000 7f4d7a69b540 10 monclient(hunting): picked mon.noname-b con 0x55ddff66abc0 addr v1:172.21.15.102:6790/0 2021-05-25T12:00:39.916 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.764+0000 7f4d7a69b540 10 monclient(hunting): picked mon.noname-c con 0x55ddff6fb9a0 addr v1:172.21.15.181:6789/0 2021-05-25T12:00:39.916 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.764+0000 7f4d7a69b540 10 monclient(hunting): picked mon.noname-a con 0x55ddff63d7b0 addr v1:172.21.15.102:6789/0 2021-05-25T12:00:39.916 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.764+0000 7f4d7a69b540 1 -- 172.21.15.181:0/149592805 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x55ddff637630 con 0x55ddff63d7b0 2021-05-25T12:00:39.916 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.764+0000 7f4d7a69b540 1 -- 172.21.15.181:0/149592805 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x55ddff725a40 con 0x55ddff6fb9a0 2021-05-25T12:00:39.916 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.764+0000 7f4d7a69b540 1 -- 172.21.15.181:0/149592805 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x55ddff6f8d10 con 0x55ddff66abc0 2021-05-25T12:00:39.917 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.764+0000 7f4d7a69b540 10 monclient(hunting): _renew_subs 2021-05-25T12:00:39.917 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.764+0000 7f4d7a69b540 10 monclient(hunting): authenticate will time out at 539.248271s 2021-05-25T12:00:39.917 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.764+0000 7f4d7a69b540 0 monclient(hunting): authenticate timed out after 1e-09 2021-05-25T12:00:39.917 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.764+0000 7f4d7a69b540 10 monclient(hunting): shutdown 2021-05-25T12:00:39.917 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.764+0000 7f4d7a69b540 20 monclient(hunting): shutdown discarding 1 pending message(s) 2021-05-25T12:00:39.918 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.764+0000 7f4d7a69b540 1 -- 172.21.15.181:0/149592805 >> v1:172.21.15.102:6790/0 conn(0x55ddff66abc0 legacy=0x55ddff721e90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-05-25T12:00:39.918 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.764+0000 7f4d7a69b540 1 -- 172.21.15.181:0/149592805 >> v1:172.21.15.181:6789/0 conn(0x55ddff6fb9a0 legacy=0x55ddff66b000 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-05-25T12:00:39.918 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.764+0000 7f4d7a69b540 1 -- 172.21.15.181:0/149592805 >> v1:172.21.15.102:6789/0 conn(0x55ddff63d7b0 legacy=0x55ddff6f80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-05-25T12:00:39.918 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.764+0000 7f4d7a69b540 10 monclient: init 2021-05-25T12:00:39.918 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.765+0000 7f4d7a69b540 10 monclient: _reopen_session rank -1 2021-05-25T12:00:39.919 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.765+0000 7f4d7a69b540 10 monclient: _add_conns ranks=[1,2,0] 2021-05-25T12:00:39.919 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.765+0000 7f4d7a69b540 10 monclient(hunting): picked mon.noname-c con 0x55ddff6fb9a0 addr v1:172.21.15.181:6789/0 2021-05-25T12:00:39.919 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.765+0000 7f4d7a69b540 10 monclient(hunting): picked mon.noname-b con 0x55ddff66abc0 addr v1:172.21.15.102:6790/0 2021-05-25T12:00:39.919 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.765+0000 7f4d7a69b540 10 monclient(hunting): picked mon.noname-a con 0x55ddff63d7b0 addr v1:172.21.15.102:6789/0 2021-05-25T12:00:39.920 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.765+0000 7f4d7a69b540 1 -- 172.21.15.181:0/149592805 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x55ddff637630 con 0x55ddff63d7b0 2021-05-25T12:00:39.920 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.765+0000 7f4d7a69b540 1 -- 172.21.15.181:0/149592805 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x55ddff6f8d10 con 0x55ddff6fb9a0 2021-05-25T12:00:39.920 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.765+0000 7f4d7a69b540 1 -- 172.21.15.181:0/149592805 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x55ddff633d60 con 0x55ddff66abc0 2021-05-25T12:00:39.920 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.765+0000 7f4d7a69b540 10 monclient(hunting): _renew_subs 2021-05-25T12:00:39.920 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.765+0000 7f4d7a69b540 10 monclient(hunting): authenticate will time out at 539.249211s 2021-05-25T12:00:39.921 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.765+0000 7f4d7a69b540 0 monclient(hunting): authenticate timed out after 1e-09 2021-05-25T12:00:39.921 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.765+0000 7f4d7a69b540 10 monclient(hunting): shutdown 2021-05-25T12:00:39.921 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.765+0000 7f4d7a69b540 20 monclient(hunting): shutdown discarding 1 pending message(s) 2021-05-25T12:00:39.921 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.765+0000 7f4d7a69b540 1 -- 172.21.15.181:0/149592805 >> v1:172.21.15.102:6790/0 conn(0x55ddff66abc0 legacy=0x55ddff721e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-05-25T12:00:39.921 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.765+0000 7f4d7a69b540 1 -- 172.21.15.181:0/149592805 >> v1:172.21.15.181:6789/0 conn(0x55ddff6fb9a0 legacy=0x55ddff66b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:39.922 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.765+0000 7f4d7a69b540 1 -- 172.21.15.181:0/149592805 >> v1:172.21.15.102:6789/0 conn(0x55ddff63d7b0 legacy=0x55ddff707110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:39.922 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.765+0000 7f4d7a69b540 10 monclient: init 2021-05-25T12:00:39.922 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.766+0000 7f4d7a69b540 10 monclient: _reopen_session rank -1 2021-05-25T12:00:39.922 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.766+0000 7f4d7a69b540 10 monclient: _add_conns ranks=[0,2,1] 2021-05-25T12:00:39.923 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.766+0000 7f4d7a69b540 10 monclient(hunting): picked mon.noname-a con 0x55ddff63d7b0 addr v1:172.21.15.102:6789/0 2021-05-25T12:00:39.923 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.766+0000 7f4d7a69b540 10 monclient(hunting): picked mon.noname-b con 0x55ddff66abc0 addr v1:172.21.15.102:6790/0 2021-05-25T12:00:39.923 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.766+0000 7f4d7a69b540 10 monclient(hunting): picked mon.noname-c con 0x55ddff6fb9a0 addr v1:172.21.15.181:6789/0 2021-05-25T12:00:39.923 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.766+0000 7f4d7a69b540 1 -- 172.21.15.181:0/149592805 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x55ddff633d60 con 0x55ddff63d7b0 2021-05-25T12:00:39.923 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.766+0000 7f4d7a69b540 1 -- 172.21.15.181:0/149592805 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x55ddff707bb0 con 0x55ddff6fb9a0 2021-05-25T12:00:39.924 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.766+0000 7f4d7a69b540 1 -- 172.21.15.181:0/149592805 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x55ddff707eb0 con 0x55ddff66abc0 2021-05-25T12:00:39.924 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.766+0000 7f4d7a69b540 10 monclient(hunting): _renew_subs 2021-05-25T12:00:39.924 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.766+0000 7f4d7a69b540 10 monclient(hunting): authenticate will time out at 539.250219s 2021-05-25T12:00:39.924 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.766+0000 7f4d7a69b540 0 monclient(hunting): authenticate timed out after 1e-09 2021-05-25T12:00:39.924 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.766+0000 7f4d7a69b540 10 monclient(hunting): shutdown 2021-05-25T12:00:39.925 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.766+0000 7f4d7a69b540 20 monclient(hunting): shutdown discarding 1 pending message(s) 2021-05-25T12:00:39.925 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.766+0000 7f4d7a69b540 1 -- 172.21.15.181:0/149592805 >> v1:172.21.15.102:6790/0 conn(0x55ddff66abc0 legacy=0x55ddff721e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-05-25T12:00:39.925 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.766+0000 7f4d7a69b540 1 -- 172.21.15.181:0/149592805 >> v1:172.21.15.181:6789/0 conn(0x55ddff6fb9a0 legacy=0x55ddff66b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-05-25T12:00:39.925 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.766+0000 7f4d7a69b540 1 -- 172.21.15.181:0/149592805 >> v1:172.21.15.102:6789/0 conn(0x55ddff63d7b0 legacy=0x55ddff707110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-05-25T12:00:39.926 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.766+0000 7f4d7a69b540 10 monclient: init 2021-05-25T12:00:39.926 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.766+0000 7f4d7a69b540 10 monclient: _reopen_session rank -1 2021-05-25T12:00:39.926 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.766+0000 7f4d7a69b540 10 monclient: _add_conns ranks=[0,1,2] 2021-05-25T12:00:39.926 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.766+0000 7f4d7a69b540 10 monclient(hunting): picked mon.noname-a con 0x55ddff63d7b0 addr v1:172.21.15.102:6789/0 2021-05-25T12:00:39.926 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.766+0000 7f4d7a69b540 10 monclient(hunting): picked mon.noname-c con 0x55ddff6fb9a0 addr v1:172.21.15.181:6789/0 2021-05-25T12:00:39.927 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.766+0000 7f4d7a69b540 10 monclient(hunting): picked mon.noname-b con 0x55ddff66abc0 addr v1:172.21.15.102:6790/0 2021-05-25T12:00:39.927 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.766+0000 7f4d7a69b540 1 -- 172.21.15.181:0/149592805 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x55ddff633d60 con 0x55ddff63d7b0 2021-05-25T12:00:39.927 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.766+0000 7f4d7a69b540 1 -- 172.21.15.181:0/149592805 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x55ddff707bb0 con 0x55ddff6fb9a0 2021-05-25T12:00:39.927 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.766+0000 7f4d7a69b540 1 -- 172.21.15.181:0/149592805 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x55ddff707eb0 con 0x55ddff66abc0 2021-05-25T12:00:39.927 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.766+0000 7f4d7a69b540 10 monclient(hunting): _renew_subs 2021-05-25T12:00:39.928 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.766+0000 7f4d7a69b540 10 monclient(hunting): authenticate will time out at 539.250879s 2021-05-25T12:00:39.928 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.766+0000 7f4d7a69b540 0 monclient(hunting): authenticate timed out after 1e-09 2021-05-25T12:00:39.928 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.766+0000 7f4d7a69b540 10 monclient(hunting): shutdown 2021-05-25T12:00:39.929 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.766+0000 7f4d7a69b540 20 monclient(hunting): shutdown discarding 1 pending message(s) 2021-05-25T12:00:39.929 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.766+0000 7f4d7a69b540 1 -- 172.21.15.181:0/149592805 >> v1:172.21.15.102:6790/0 conn(0x55ddff66abc0 legacy=0x55ddff721e90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-05-25T12:00:39.929 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.767+0000 7f4d7a69b540 1 -- 172.21.15.181:0/149592805 >> v1:172.21.15.181:6789/0 conn(0x55ddff6fb9a0 legacy=0x55ddff66b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:39.929 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.767+0000 7f4d6affd700 1 -- 172.21.15.181:0/149592805 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 980669450 0 0) 0x55ddff707bb0 con 0x55ddff6fb9a0 2021-05-25T12:00:39.929 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.767+0000 7f4d7a69b540 1 -- 172.21.15.181:0/149592805 >> v1:172.21.15.102:6789/0 conn(0x55ddff63d7b0 legacy=0x55ddff707110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:39.930 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.767+0000 7f4d6affd700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2021-05-25T12:00:39.930 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.767+0000 7f4d7a69b540 10 monclient: init 2021-05-25T12:00:39.930 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.767+0000 7f4d7a69b540 10 monclient: _reopen_session rank -1 2021-05-25T12:00:39.930 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.767+0000 7f4d7a69b540 10 monclient: _add_conns ranks=[0,1,2] 2021-05-25T12:00:39.930 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.767+0000 7f4d7a69b540 10 monclient(hunting): picked mon.noname-a con 0x55ddff63d7b0 addr v1:172.21.15.102:6789/0 2021-05-25T12:00:39.931 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.768+0000 7f4d7a69b540 10 monclient(hunting): picked mon.noname-c con 0x55ddff6fb9a0 addr v1:172.21.15.181:6789/0 2021-05-25T12:00:39.931 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.768+0000 7f4d7a69b540 10 monclient(hunting): picked mon.noname-b con 0x55ddff66abc0 addr v1:172.21.15.102:6790/0 2021-05-25T12:00:39.931 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.768+0000 7f4d7a69b540 1 -- 172.21.15.181:0/149592805 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x55ddff707eb0 con 0x55ddff63d7b0 2021-05-25T12:00:39.931 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.768+0000 7f4d7a69b540 1 -- 172.21.15.181:0/149592805 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x55ddff725bd0 con 0x55ddff6fb9a0 2021-05-25T12:00:39.932 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.768+0000 7f4d7a69b540 1 -- 172.21.15.181:0/149592805 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x55ddff6f39f0 con 0x55ddff66abc0 2021-05-25T12:00:39.932 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.768+0000 7f4d7a69b540 10 monclient(hunting): _renew_subs 2021-05-25T12:00:39.932 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.768+0000 7f4d7a69b540 10 monclient(hunting): authenticate will time out at 539.252168s 2021-05-25T12:00:39.932 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.768+0000 7f4d7a69b540 0 monclient(hunting): authenticate timed out after 1e-09 2021-05-25T12:00:39.933 INFO:tasks.workunit.client.0.smithi181.stderr:++ echo api_stat_pp 2021-05-25T12:00:39.934 INFO:tasks.workunit.client.0.smithi181.stderr:+ ff=api_stat_pp 2021-05-25T12:00:39.934 INFO:tasks.workunit.client.0.smithi181.stderr:+ pid=33141 2021-05-25T12:00:39.934 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.768+0000 7f4d7a69b540 10 monclient(huntest api_stat_pp on pid 33141 2021-05-25T12:00:39.938 INFO:tasks.workunit.client.0.smithi181.stderr:+ echo 'test api_stat_pp on pid 33141' 2021-05-25T12:00:39.938 INFO:tasks.workunit.client.0.smithi181.stderr:+ pids[$f]=33141 2021-05-25T12:00:39.938 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2021-05-25T12:00:39.938 INFO:tasks.workunit.client.0.smithi181.stderr:+ '[' 1 -eq 1 ']' 2021-05-25T12:00:39.939 INFO:tasks.workunit.client.0.smithi181.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat_pp 2>&1 | tee ceph_test_rados_api_stat_pp.log | sed "s/^/ api_stat_pp: /"' 2021-05-25T12:00:39.939 INFO:tasks.workunit.client.0.smithi181.stderr:++ printf %25s api_watch_notify 2021-05-25T12:00:39.939 INFO:tasks.workunit.client.0.smithi181.stderr:+ r=' api_watch_notify' 2021-05-25T12:00:39.939 INFO:tasks.workunit.client.0.smithi181.stderr:++ echo api_watch_notify 2021-05-25T12:00:39.939 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph_test_rados_api_stat_pp 2021-05-25T12:00:39.940 INFO:tasks.workunit.client.0.smithi181.stderr:+ sed 's/^/ api_stat_pp: /' 2021-05-25T12:00:39.940 INFO:tasks.workunit.client.0.smithi181.stderr:++ awk '{print $1}' 2021-05-25T12:00:39.940 INFO:tasks.workunit.client.0.smithi181.stderr:+ tee ceph_test_rados_api_stat_pp.log 2021-05-25T12:00:39.940 INFO:tasks.workunit.client.0.smithi181.stderr:+ ff=api_watch_notify 2021-05-25T12:00:39.940 INFO:tasks.workunit.client.0.smithi181.stderr:+ pid=33175 2021-05-25T12:00:39.941 INFO:tasks.workunit.client.0.smithi181.stdout:test api_watch_notify on pid 33175 2021-05-25T12:00:39.944 INFO:tasks.workunit.client.0.smithi181.stderr:+ echo 'test api_watch_notify on pid 33175' 2021-05-25T12:00:39.944 INFO:tasks.workunit.client.0.smithi181.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify 2>&1 | tee ceph_test_rados_api_watch_notify.log | sed "s/^/ api_watch_notify: /"' 2021-05-25T12:00:39.944 INFO:tasks.workunit.client.0.smithi181.stderr:+ pids[$f]=33175 2021-05-25T12:00:39.945 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2021-05-25T12:00:39.945 INFO:tasks.workunit.client.0.smithi181.stderr:+ '[' 1 -eq 1 ']' 2021-05-25T12:00:39.945 INFO:tasks.workunit.client.0.smithi181.stderr:++ printf %25s api_watch_notify_pp 2021-05-25T12:00:39.945 INFO:tasks.workunit.client.0.smithi181.stderr:+ r=' api_watch_notify_pp' 2021-05-25T12:00:39.946 INFO:tasks.workunit.client.0.smithi181.stderr:++ awk '{print $1}' 2021-05-25T12:00:39.946 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph_test_rados_api_watch_notify 2021-05-25T12:00:39.946 INFO:tasks.workunit.client.0.smithi181.stderr:+ sed 's/^/ api_watch_notify: /' 2021-05-25T12:00:39.946 INFO:tasks.workunit.client.0.smithi181.stderr:+ tee ceph_test_rados_api_watch_notify.log 2021-05-25T12:00:39.946 INFO:tasks.workunit.client.0.smithi181.stderr:++ echo api_watch_notify_pp 2021-05-25T12:00:39.947 INFO:tasks.workunit.client.0.smithi181.stdout:test api_watch_notify_pp on pid 33194 2021-05-25T12:00:39.949 INFO:tasks.workunit.client.0.smithi181.stderr:+ ff=api_watch_notify_pp 2021-05-25T12:00:39.950 INFO:tasks.workunit.client.0.smithi181.stderr:+ pid=33194 2021-05-25T12:00:39.950 INFO:tasks.workunit.client.0.smithi181.stderr:+ echo 'test api_watch_notify_pp on pid 33194' 2021-05-25T12:00:39.950 INFO:tasks.workunit.client.0.smithi181.stderr:+ pids[$f]=33194 2021-05-25T12:00:39.950 INFO:tasks.workunit.client.0.smithi181.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify_pp 2>&1 | tee ceph_test_rados_api_watch_notify_pp.log | sed "s/^/ api_watch_notify_pp: /"' 2021-05-25T12:00:39.951 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2021-05-25T12:00:39.951 INFO:tasks.workunit.client.0.smithi181.stderr:+ '[' 1 -eq 1 ']' 2021-05-25T12:00:39.951 INFO:tasks.workunit.client.0.smithi181.stderr:+ tee ceph_test_rados_api_watch_notify_pp.log 2021-05-25T12:00:39.951 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph_test_rados_api_watch_notify_pp 2021-05-25T12:00:39.952 INFO:tasks.workunit.client.0.smithi181.stderr:++ printf %25s api_cmd 2021-05-25T12:00:39.952 INFO:tasks.workunit.client.0.smithi181.stderr:+ sed 's/^/ api_watch_notify_pp: /' 2021-05-25T12:00:39.952 INFO:tasks.workunit.client.0.smithi181.stderr:+ r=' api_cmd' 2021-05-25T12:00:39.952 INFO:tasks.workunit.client.0.smithi181.stderr:++ awk '{print $1}' 2021-05-25T12:00:39.952 INFO:tasks.workunit.client.0.smithi181.stderr:++ echo api_cmd 2021-05-25T12:00:39.953 INFO:tasks.workunit.client.0.smithi181.stderr:+ ff=api_cmd 2021-05-25T12:00:39.953 INFO:tasks.workunit.client.0.smithi181.stdout:test api_cmd on pid 33233 2021-05-25T12:00:39.956 INFO:tasks.workunit.client.0.smithi181.stderr:+ pid=33233 2021-05-25T12:00:39.956 INFO:tasks.workunit.client.0.smithi181.stderr:+ echo 'test api_cmd on pid 33233' 2021-05-25T12:00:39.956 INFO:tasks.workunit.client.0.smithi181.stderr:+ pids[$f]=33233 2021-05-25T12:00:39.957 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2021-05-25T12:00:39.957 INFO:tasks.workunit.client.0.smithi181.stderr:+ '[' 1 -eq 1 ']' 2021-05-25T12:00:39.957 INFO:tasks.workunit.client.0.smithi181.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd 2>&1 | tee ceph_test_rados_api_cmd.log | sed "s/^/ api_cmd: /"' 2021-05-25T12:00:39.957 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph_test_rados_api_cmd 2021-05-25T12:00:39.957 INFO:tasks.workunit.client.0.smithi181.stderr:+ sed 's/^/ api_cmd: /' 2021-05-25T12:00:39.958 INFO:tasks.workunit.client.0.smithi181.stderr:++ printf %25s api_cmd_pp 2021-05-25T12:00:39.958 INFO:tasks.workunit.client.0.smithi181.stderr:+ r=' api_cmd_pp' 2021-05-25T12:00:39.958 INFO:tasks.workunit.client.0.smithi181.stderr:++ echo api_cmd_pp 2021-05-25T12:00:39.958 INFO:tasks.workunit.client.0.smithi181.stderr:++ awk '{print $1}' 2021-05-25T12:00:39.959 INFO:tasks.workunit.client.0.smithi181.stderr:+ tee ceph_test_rados_api_cmd.log 2021-05-25T12:00:39.959 INFO:tasks.workunit.client.0.smithi181.stderr:+ ff=api_cmd_pp 2021-05-25T12:00:39.959 INFO:tasks.workunit.client.0.smithi181.stdout:test api_cmd_pp on pid 33255 2021-05-25T12:00:39.961 INFO:tasks.workunit.client.0.smithi181.stderr:+ pid=33255 2021-05-25T12:00:39.962 INFO:tasks.workunit.client.0.smithi181.stderr:+ echo 'test api_cmd_pp on pid 33255' 2021-05-25T12:00:39.962 INFO:tasks.workunit.client.0.smithi181.stderr:+ pids[$f]=33255 2021-05-25T12:00:39.962 INFO:tasks.workunit.client.0.smithi181.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd_pp 2>&1 | tee ceph_test_rados_api_cmd_pp.log | sed "s/^/ api_cmd_pp: /"' 2021-05-25T12:00:39.962 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2021-05-25T12:00:39.963 INFO:tasks.workunit.client.0.smithi181.stderr:+ '[' 1 -eq 1 ']' 2021-05-25T12:00:39.963 INFO:tasks.workunit.client.0.smithi181.stderr:++ printf %25s api_service 2021-05-25T12:00:39.963 INFO:tasks.workunit.client.0.smithi181.stderr:+ r=' api_service' 2021-05-25T12:00:39.963 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph_test_rados_api_cmd_pp 2021-05-25T12:00:39.963 INFO:tasks.workunit.client.0.smithi181.stderr:+ tee ceph_test_rados_api_cmd_pp.log 2021-05-25T12:00:39.964 INFO:tasks.workunit.client.0.smithi181.stderr:++ awk '{print $1}' 2021-05-25T12:00:39.964 INFO:tasks.workunit.client.0.smithi181.stderr:++ echo api_service 2021-05-25T12:00:39.964 INFO:tasks.workunit.client.0.smithi181.stdout:test api_service on pid 33286 2021-05-25T12:00:39.967 INFO:tasks.workunit.client.0.smithi181.stderr:+ ff=api_service 2021-05-25T12:00:39.967 INFO:tasks.workunit.client.0.smithi181.stderr:+ pid=33286 2021-05-25T12:00:39.967 INFO:tasks.workunit.client.0.smithi181.stderr:+ echo 'test api_service on pid 33286' 2021-05-25T12:00:39.968 INFO:tasks.workunit.client.0.smithi181.stderr:+ pids[$f]=33286 2021-05-25T12:00:39.968 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2021-05-25T12:00:39.968 INFO:tasks.workunit.client.0.smithi181.stderr:+ '[' 1 -eq 1 ']' 2021-05-25T12:00:39.968 INFO:tasks.workunit.client.0.smithi181.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service 2>&1 | tee ceph_test_rados_api_service.log | sed "s/^/ api_service: /"' 2021-05-25T12:00:39.968 INFO:tasks.workunit.client.0.smithi181.stderr:++ printf %25s api_service_pp 2021-05-25T12:00:39.969 INFO:tasks.workunit.client.0.smithi181.stderr:+ r=' api_service_pp' 2021-05-25T12:00:39.969 INFO:tasks.workunit.client.0.smithi181.stderr:++ awk '{print $1}' 2021-05-25T12:00:39.969 INFO:tasks.workunit.client.0.smithi181.stderr:+ tee ceph_test_rados_api_service.log 2021-05-25T12:00:39.969 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph_test_rados_api_service 2021-05-25T12:00:39.970 INFO:tasks.workunit.client.0.smithi181.stderr:+ sed 's/^/ api_service: /' 2021-05-25T12:00:39.970 INFO:tasks.workunit.client.0.smithi181.stderr:++ echo api_service_pp 2021-05-25T12:00:39.970 INFO:tasks.workunit.client.0.smithi181.stdout:test api_service_pp on pid 33305 2021-05-25T12:00:39.973 INFO:tasks.workunit.client.0.smithi181.stderr:+ ff=api_service_pp 2021-05-25T12:00:39.973 INFO:tasks.workunit.client.0.smithi181.stderr:+ pid=33305 2021-05-25T12:00:39.974 INFO:tasks.workunit.client.0.smithi181.stderr:+ echo 'test api_service_pp on pid 33305' 2021-05-25T12:00:39.974 INFO:tasks.workunit.client.0.smithi181.stderr:+ pids[$f]=33305 2021-05-25T12:00:39.974 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2021-05-25T12:00:39.974 INFO:tasks.workunit.client.0.smithi181.stderr:+ '[' 1 -eq 1 ']' 2021-05-25T12:00:39.975 INFO:tasks.workunit.client.0.smithi181.stderr:++ printf %25s api_c_write_operations 2021-05-25T12:00:39.975 INFO:tasks.workunit.client.0.smithi181.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service_pp 2>&1 | tee ceph_test_rados_api_service_pp.log | sed "s/^/ api_service_pp: /"' 2021-05-25T12:00:39.975 INFO:tasks.workunit.client.0.smithi181.stderr:+ r=' api_c_write_operations' 2021-05-25T12:00:39.975 INFO:tasks.workunit.client.0.smithi181.stderr:++ echo api_c_write_operations 2021-05-25T12:00:39.976 INFO:tasks.workunit.client.0.smithi181.stderr:++ awk '{print $1}' 2021-05-25T12:00:39.976 INFO:tasks.workunit.client.0.smithi181.stderr:+ sed 's/^/ api_cmd_pp: /' 2021-05-25T12:00:39.976 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph_test_rados_api_service_pp 2021-05-25T12:00:39.976 INFO:tasks.workunit.client.0.smithi181.stderr:+ sed 's/^/ api_service_pp: /' 2021-05-25T12:00:39.976 INFO:tasks.workunit.client.0.smithi181.stderr:+ tee ceph_test_rados_api_service_pp.log 2021-05-25T12:00:39.977 INFO:tasks.workunit.client.0.smithi181.stderr:+ ff=api_c_write_operations 2021-05-25T12:00:39.977 INFO:tasks.workunit.client.0.smithi181.stdout:test api_c_write_operations on pid 33323 2021-05-25T12:00:39.980 INFO:tasks.workunit.client.0.smithi181.stderr:+ pid=33323 2021-05-25T12:00:39.980 INFO:tasks.workunit.client.0.smithi181.stderr:+ echo 'test api_c_write_operations on pid 33323' 2021-05-25T12:00:39.980 INFO:tasks.workunit.client.0.smithi181.stderr:+ pids[$f]=33323 2021-05-25T12:00:39.980 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2021-05-25T12:00:39.981 INFO:tasks.workunit.client.0.smithi181.stderr:+ '[' 1 -eq 1 ']' 2021-05-25T12:00:39.981 INFO:tasks.workunit.client.0.smithi181.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_write_operations 2>&1 | tee ceph_test_rados_api_c_write_operations.log | sed "s/^/ api_c_write_operations: /"' 2021-05-25T12:00:39.981 INFO:tasks.workunit.client.0.smithi181.stderr:++ printf %25s api_c_read_operations 2021-05-25T12:00:39.981 INFO:tasks.workunit.client.0.smithi181.stderr:+ r=' api_c_read_operations' 2021-05-25T12:00:39.981 INFO:tasks.workunit.client.0.smithi181.stderr:++ echo api_c_read_operations 2021-05-25T12:00:39.982 INFO:tasks.workunit.client.0.smithi181.stderr:++ awk '{print $1}' 2021-05-25T12:00:39.982 INFO:tasks.workunit.client.0.smithi181.stderr:+ tee ceph_test_rados_api_c_write_operations.log 2021-05-25T12:00:39.982 INFO:tasks.workunit.client.0.smithi181.stderr:+ sed 's/^/ api_c_write_operations: /' 2021-05-25T12:00:39.982 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph_test_rados_api_c_write_operations 2021-05-25T12:00:39.983 INFO:tasks.workunit.client.0.smithi181.stderr:+ ff=api_c_read_operations 2021-05-25T12:00:39.983 INFO:tasks.workunit.client.0.smithi181.stdout:test api_c_read_operations on pid 33339 2021-05-25T12:00:39.985 INFO:tasks.workunit.client.0.smithi181.stderr:+ pid=33339 2021-05-25T12:00:39.986 INFO:tasks.workunit.client.0.smithi181.stderr:+ echo 'test api_c_read_operations on pid 33339' 2021-05-25T12:00:39.986 INFO:tasks.workunit.client.0.smithi181.stderr:+ pids[$f]=33339 2021-05-25T12:00:39.986 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2021-05-25T12:00:39.986 INFO:tasks.workunit.client.0.smithi181.stderr:+ '[' 1 -eq 1 ']' 2021-05-25T12:00:39.987 INFO:tasks.workunit.client.0.smithi181.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_read_operations 2>&1 | tee ceph_test_rados_api_c_read_operations.log | sed "s/^/ api_c_read_operations: /"' 2021-05-25T12:00:39.987 INFO:tasks.workunit.client.0.smithi181.stderr:+ tee ceph_test_rados_api_c_read_operations.log 2021-05-25T12:00:39.987 INFO:tasks.workunit.client.0.smithi181.stderr:+ sed 's/^/ api_c_read_operations: /' 2021-05-25T12:00:39.987 INFO:tasks.workunit.client.0.smithi181.stderr:++ printf %25s api_cls_remote_reads 2021-05-25T12:00:39.988 INFO:tasks.workunit.client.0.smithi181.stderr:+ r=' api_cls_remote_reads' 2021-05-25T12:00:39.988 INFO:tasks.workunit.client.0.smithi181.stderr:++ awk '{print $1}' 2021-05-25T12:00:39.988 INFO:tasks.workunit.client.0.smithi181.stderr:++ echo api_cls_remote_reads 2021-05-25T12:00:39.988 INFO:tasks.workunit.client.0.smithi181.stderr:+ ff=api_cls_remote_reads 2021-05-25T12:00:39.989 INFO:tasks.workunit.client.0.smithi181.stdout:test api_cls_remote_reads on pid 33355 2021-05-25T12:00:39.989 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.887+0000 7f722759e700 10 monclient: get_monmap_and_config 2021-05-25T12:00:39.990 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.887+0000 7f722759e700 10 monclient: build_initial_monmap 2021-05-25T12:00:39.990 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.887+0000 7f722759e700 10 monclient: monmap: 2021-05-25T12:00:39.990 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:00:39.991 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:39.991 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:00:39.887958+0000 2021-05-25T12:00:39.991 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:00:39.887958+0000 2021-05-25T12:00:39.991 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:00:39.991 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:39.992 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:00:39.992 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:00:39.992 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:00:39.992 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:39.992 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.887+0000 7f722759e700 1 Processor -- start 2021-05-25T12:00:39.993 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.887+0000 7f722759e700 1 -- start start 2021-05-25T12:00:39.993 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.887+0000 7f722759e700 10 monclient: init 2021-05-25T12:00:39.993 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.888+0000 7f722759e700 10 monclient: _reopen_session rank -1 2021-05-25T12:00:39.993 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.888+0000 7f722759e700 10 monclient: _add_conns ranks=[1,2,0] 2021-05-25T12:00:39.994 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.888+0000 7f722759e700 10 monclient(hunting): picked mon.noname-c con 0x7f7228098930 addr v1:172.21.15.181:6789/0 2021-05-25T12:00:39.994 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.888+0000 7f722759e700 10 monclient(hunting): picked mon.noname-b con 0x7f722812bca0 addr v1:172.21.15.102:6790/0 2021-05-25T12:00:39.994 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.888+0000 7f722759e700 10 monclient(hunting): picked mon.noname-a con 0x7f722812c150 addr v1:172.21.15.102:6789/0 2021-05-25T12:00:39.994 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.888+0000 7f7225d9b700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f722812bca0 0x7f72281308c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:54278/0 (socket says 172.21.15.102:54278) 2021-05-25T12:00:39.994 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.888+0000 7f7225d9b700 1 -- 172.21.15.102:0/3000242063 learned_addr learned my addr 172.21.15.102:0/3000242063 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:00:39.995 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.888+0000 7f722759e700 1 -- 172.21.15.102:0/3000242063 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7228099b20 con 0x7f722812c150 2021-05-25T12:00:39.995 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.888+0000 7f722759e700 1 -- 172.21.15.102:0/3000242063 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f722809a3a0 con 0x7f7228098930 2021-05-25T12:00:39.995 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.888+0000 7f722759e700 1 -- 172.21.15.102:0/3000242063 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f722809ac60 con 0x7f722812bca0 2021-05-25T12:00:39.995 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.888+0000 7f722759e700 10 monclient(hunting): _renew_subs 2021-05-25T12:00:39.996 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.888+0000 7f722759e700 10 monclient(hunting): authenticate will time out at 840.485633s 2021-05-25T12:00:39.996 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.888+0000 7f722559a700 1 -- 172.21.15.102:0/3000242063 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 329172913 0 0) 0x7f722809ac60 con 0x7f722812bca0 2021-05-25T12:00:39.996 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.888+0000 7f722559a700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:39.996 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.888+0000 7f722559a700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:39.996 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.888+0000 7f722559a700 10 monclient(hunting): my global_id is 4628 2021-05-25T12:00:39.997 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.888+0000 7f722559a700 1 -- 172.21.15.102:0/3000242063 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f720c003270 con 0x7f722812bca0 2021-05-25T12:00:39.997 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.889+0000 7f722559a700 1 -- 172.21.15.102:0/3000242063 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1606319066 0 0) 0x7f722809a3a0 con 0x7f7228098930 2021-05-25T12:00:39.997 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.889+0000 7f722559a700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:39.997 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.889+0000 7f722559a700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:39.997 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.889+0000 7f722559a700 10 monclient(hunting): my global_id is 74302 2021-05-25T12:00:39.998 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.889+0000 7f722559a700 1 -- 172.21.15.102:0/3000242063 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f722809ac60 con 0x7f7228098930 2021-05-25T12:00:39.998 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.889+0000 7f722559a700 1 -- 172.21.15.102:0/3000242063 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 937155698 0 0) 0x7f720c003270 con 0x7f722812bca0 2021-05-25T12:00:39.998 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.889+0000 7f722559a700 1 -- 172.21.15.102:0/3000242063 >> v1:172.21.15.181:6789/0 conn(0x7f7228098930 legacy=0x7f722812b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:39.998 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.889+0000 7f722559a700 1 -- 172.21.15.102:0/3000242063 >> v1:172.21.15.102:6789/0 conn(0x7f722812c150 legacy=0x7f7228133ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:39.999 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.889+0000 7f722559a700 10 monclient: _finish_hunting 0 2021-05-25T12:00:39.999 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.889+0000 7f722559a700 1 monclient: found mon.noname-b 2021-05-25T12:00:39.999 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.889+0000 7f722559a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:00:39.999 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.889+0000 7f722559a700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T12:00:39.999 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.889+0000 7f722559a700 1 -- 172.21.15.102:0/3000242063 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f722809af00 con 0x7f722812bca0 2021-05-25T12:00:40.000 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.889+0000 7f722559a700 10 monclient: _finish_auth 0 2021-05-25T12:00:40.000 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.889+0000 7f722559a700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:00:40.000 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.889+0000 7f722559a700 1 -- 172.21.15.102:0/3000242063 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f7210001610 con 0x7f722812bca0 2021-05-25T12:00:40.000 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.889+0000 7f722759e700 5 monclient: authenticate success, global_id 4628 2021-05-25T12:00:40.000 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.889+0000 7f722759e700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:00:40.001 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.889+0000 7f722559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:40.001 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.889+0000 7f722559a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:00:40.001 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.889+0000 7f722559a700 10 monclient: dump: 2021-05-25T12:00:40.001 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:40.002 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:40.002 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:40.002 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:40.002 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:40.002 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:40.003 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:40.003 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:40.003 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:40.003 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:40.003 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.889+0000 7f722759e700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:00:40.004 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.889+0000 7f722559a700 1 -- 172.21.15.102:0/3000242063 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7210001830 con 0x7f722812bca0 2021-05-25T12:00:40.004 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.889+0000 7f722559a700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:00:40.004 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.889+0000 7f722759e700 10 monclient: get_monmap_and_config success 2021-05-25T12:00:40.004 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.889+0000 7f722559a700 1 -- 172.21.15.102:0/3000242063 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f7210001b40 con 0x7f722812bca0 2021-05-25T12:00:40.005 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.889+0000 7f722759e700 4 set_mon_vals no callback set 2021-05-25T12:00:40.005 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.889+0000 7f722559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:40.005 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.889+0000 7f722559a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:00:40.005 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.889+0000 7f722559a700 10 monclient: dump: 2021-05-25T12:00:40.005 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:40.006 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:40.006 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:40.006 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:40.006 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:40.006 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:40.007 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:40.007 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:40.007 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:40.007 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:40.008 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.889+0000 7f722759e700 10 monclient: shutdown 2021-05-25T12:00:40.008 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.889+0000 7f722759e700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:00:40.008 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.889+0000 7f722759e700 1 -- 172.21.15.102:0/3000242063 >> v1:172.21.15.102:6790/0 conn(0x7f722812bca0 legacy=0x7f72281308c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:40.008 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.889+0000 7f722759e700 1 -- 172.21.15.102:0/3000242063 shutdown_connections 2021-05-25T12:00:40.008 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.889+0000 7f722759e700 1 -- 172.21.15.102:0/3000242063 >> 172.21.15.102:0/3000242063 conn(0x7f722809e9d0 msgr2=0x7f722809ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:00:40.009 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.889+0000 7f722759e700 1 -- 172.21.15.102:0/3000242063 shutdown_connections 2021-05-25T12:00:40.009 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.889+0000 7f722759e700 1 -- 172.21.15.102:0/3000242063 wait complete. 2021-05-25T12:00:40.009 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.890+0000 7f722759e700 10 monclient: build_initial_monmap 2021-05-25T12:00:40.009 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.890+0000 7f722759e700 10 monclient: monmap: 2021-05-25T12:00:40.010 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:00:40.010 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:00:40.010 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:00:40.010 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:00:40.010 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:00:40.011 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:00:40.011 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:00:40.011 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:00:40.011 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:00:40.011 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:40.012 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.890+0000 7f722759e700 1 librados: starting msgr at 2021-05-25T12:00:40.012 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.890+0000 7f722759e700 1 librados: starting objecter 2021-05-25T12:00:40.012 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.890+0000 7f722759e700 1 Processor -- start 2021-05-25T12:00:40.012 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.890+0000 7f722759e700 1 -- start start 2021-05-25T12:00:40.013 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.890+0000 7f722759e700 1 librados: setting wanted keys 2021-05-25T12:00:40.013 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.890+0000 7f722759e700 1 librados: calling monclient init 2021-05-25T12:00:40.013 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.890+0000 7f722759e700 10 monclient: init 2021-05-25T12:00:40.013 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.891+0000 7f722759e700 10 monclient: _reopen_session rank -1 2021-05-25T12:00:40.013 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.891+0000 7f722759e700 10 monclient: _add_conns ranks=[0,2,1] 2021-05-25T12:00:40.014 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.891+0000 7f722759e700 10 monclient(hunting): picked mon.noname-a con 0x7f7228098930 addr v1:172.21.15.102:6789/0 2021-05-25T12:00:40.014 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.891+0000 7f722759e700 10 monclient(hunting): picked mon.noname-c con 0x7f722812bca0 addr v1:172.21.15.102:6790/0 2021-05-25T12:00:40.014 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.891+0000 7f722759e700 10 monclient(hunting): picked mon.noname-b con 0x7f722812c150 addr v1:172.21.15.181:6789/0 2021-05-25T12:00:40.014 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.891+0000 7f722759e700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72280921c0 con 0x7f7228098930 2021-05-25T12:00:40.015 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.891+0000 7f722759e700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7228092fa0 con 0x7f722812c150 2021-05-25T12:00:40.015 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.891+0000 7f722759e700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7228097be0 con 0x7f722812bca0 2021-05-25T12:00:40.015 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.891+0000 7f722759e700 10 monclient(hunting): _renew_subs 2021-05-25T12:00:40.015 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.891+0000 7f722759e700 10 monclient(hunting): authenticate will time out at 840.488204s 2021-05-25T12:00:40.015 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.891+0000 7f7225d9b700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f722812bca0 0x7f72280974d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:54284/0 (socket says 172.21.15.102:54284) 2021-05-25T12:00:40.016 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.891+0000 7f7225d9b700 1 -- 172.21.15.102:0/879158789 learned_addr learned my addr 172.21.15.102:0/879158789 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:00:40.016 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.891+0000 7f722659c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f7228098930 0x7f7228091ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:59404/0 (socket says 172.21.15.102:59404) 2021-05-25T12:00:40.016 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.891+0000 7f72167fc700 10 client.?.objecter ms_handle_connect 0x7f7228098930 2021-05-25T12:00:40.016 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.891+0000 7f72167fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:00:40.017 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.891+0000 7f72167fc700 10 client.?.objecter ms_handle_connect 0x7f722812bca0 2021-05-25T12:00:40.017 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.891+0000 7f72167fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:00:40.017 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.891+0000 7f72167fc700 1 -- 172.21.15.102:0/879158789 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 631540476 0 0) 0x7f7228097be0 con 0x7f722812bca0 2021-05-25T12:00:40.017 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.891+0000 7f72167fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:40.017 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.891+0000 7f72167fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:40.018 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.891+0000 7f72167fc700 10 monclient(hunting): my global_id is 4631 2021-05-25T12:00:40.018 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.891+0000 7f72167fc700 1 -- 172.21.15.102:0/879158789 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7204004c10 con 0x7f722812bca0 2021-05-25T12:00:40.018 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.891+0000 7f72167fc700 10 client.?.objecter ms_handle_connect 0x7f722812c150 2021-05-25T12:00:40.018 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.891+0000 7f72167fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:00:40.018 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.891+0000 7f72167fc700 1 -- 172.21.15.102:0/879158789 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3002428090 0 0) 0x7f7204004c10 con 0x7f722812bca0 2021-05-25T12:00:40.019 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.891+0000 7f72167fc700 1 -- 172.21.15.102:0/879158789 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7228097be0 con 0x7f722812bca0 2021-05-25T12:00:40.019 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.891+0000 7f72167fc700 1 -- 172.21.15.102:0/879158789 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f72100015d0 con 0x7f722812bca0 2021-05-25T12:00:40.019 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.891+0000 7f72167fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:40.019 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.891+0000 7f72167fc700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:00:40.020 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.891+0000 7f72167fc700 10 monclient(hunting): dump: 2021-05-25T12:00:40.020 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:40.020 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:40.020 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:40.020 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:40.021 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:40.021 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:40.021 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:40.021 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:40.021 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:40.022 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:40.022 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.891+0000 7f72167fc700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:00:40.022 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.891+0000 7f72167fc700 1 -- 172.21.15.102:0/879158789 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3293729812 0 0) 0x7f7228092fa0 con 0x7f722812c150 2021-05-25T12:00:40.022 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.891+0000 7f72167fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:40.023 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.891+0000 7f72167fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:40.023 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.891+0000 7f72167fc700 10 monclient(hunting): my global_id is 74305 2021-05-25T12:00:40.023 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.891+0000 7f72167fc700 1 -- 172.21.15.102:0/879158789 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7204004c10 con 0x7f722812c150 2021-05-25T12:00:40.023 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.892+0000 7f72167fc700 1 -- 172.21.15.102:0/879158789 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 304410892 0 0) 0x7f7228097be0 con 0x7f722812bca0 2021-05-25T12:00:40.023 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.892+0000 7f72167fc700 1 -- 172.21.15.102:0/879158789 >> v1:172.21.15.181:6789/0 conn(0x7f722812c150 legacy=0x7f72281430d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:40.024 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.892+0000 7f72167fc700 1 -- 172.21.15.102:0/879158789 >> v1:172.21.15.102:6789/0 conn(0x7f7228098930 legacy=0x7f7228091ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:40.024 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.892+0000 7f72167fc700 10 monclient: _finish_hunting 0 2021-05-25T12:00:40.024 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.892+0000 7f72167fc700 1 monclient: found mon.c 2021-05-25T12:00:40.024 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.892+0000 7f72167fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:00:40.024 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.892+0000 7f72167fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:00:40.025 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.892+0000 7f72167fc700 1 -- 172.21.15.102:0/879158789 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7228093980 con 0x7f722812bca0 2021-05-25T12:00:40.025 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.892+0000 7f72167fc700 10 monclient: _finish_auth 0 2021-05-25T12:00:40.025 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.892+0000 7f72167fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:00:40.025 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.892+0000 7f722759e700 5 monclient: authenticate success, global_id 4631 2021-05-25T12:00:40.025 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.892+0000 7f722759e700 10 monclient: _renew_subs 2021-05-25T12:00:40.026 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.892+0000 7f722759e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:00:40.026 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.892+0000 7f722759e700 1 -- 172.21.15.102:0/879158789 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72280945b0 con 0x7f722812bca0 2021-05-25T12:00:40.026 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.892+0000 7f722759e700 10 client.4631.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:00:40.026 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.892+0000 7f722759e700 10 monclient: _renew_subs 2021-05-25T12:00:40.027 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.892+0000 7f722759e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:00:40.027 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.892+0000 7f722759e700 1 -- 172.21.15.102:0/879158789 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7228093390 con 0x7f722812bca0 2021-05-25T12:00:40.027 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.892+0000 7f722759e700 1 librados: init done 2021-05-25T12:00:40.027 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.892+0000 7f72167fc700 1 -- 172.21.15.102:0/879158789 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7210004930 con 0x7f722812bca0 2021-05-25T12:00:40.027 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.892+0000 7f72167fc700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:00:40.028 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.892+0000 7f72167fc700 1 -- 172.21.15.102:0/879158789 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f7210004cc0 con 0x7f722812bca0 2021-05-25T12:00:40.028 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.892+0000 7f72167fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:40.028 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.892+0000 7f7224d99700 4 set_mon_vals no callback set 2021-05-25T12:00:40.028 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.892+0000 7f72167fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:00:40.029 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.892+0000 7f72167fc700 10 monclient: dump: 2021-05-25T12:00:40.029 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:40.029 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:40.029 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:40.029 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:40.030 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:40.030 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:40.030 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:40.030 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:40.030 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:40.031 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:40.031 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.892+0000 7f722759e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:00:40.031 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.892+0000 7f722759e700 1 -- 172.21.15.102:0/879158789 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f72080065a0 con 0x7f7208002840 2021-05-25T12:00:40.031 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.893+0000 7f72167fc700 10 client.4631.objecter ms_handle_connect 0x7f7208002840 2021-05-25T12:00:40.032 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.893+0000 7f72167fc700 10 client.4631.objecter resend_mon_ops 2021-05-25T12:00:40.032 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.893+0000 7f72167fc700 1 -- 172.21.15.102:0/879158789 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f7210004ea0 con 0x7f722812bca0 2021-05-25T12:00:40.032 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.893+0000 7f72167fc700 1 -- 172.21.15.102:0/879158789 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(15..15 src has 1..15) v4 ==== 3549+0+0 (unknown 3948819498 0 0) 0x7f721003d180 con 0x7f722812bca0 2021-05-25T12:00:40.032 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.893+0000 7f72167fc700 10 client.4631.objecter ms_dispatch 0x7f7228001550 osd_map(15..15 src has 1..15) v4 2021-05-25T12:00:40.032 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.893+0000 7f72167fc700 3 client.4631.objecter handle_osd_map got epochs [15,15] > 0 2021-05-25T12:00:40.033 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.893+0000 7f72167fc700 3 client.4631.objecter handle_osd_map decoding full epoch 15 2021-05-25T12:00:40.033 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.893+0000 7f72167fc700 20 client.4631.objecter dump_active .. 0 homeless 2021-05-25T12:00:40.033 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.893+0000 7f72167fc700 1 -- 172.21.15.102:0/879158789 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4014960586 0 0) 0x7f72080065a0 con 0x7f7208002840 2021-05-25T12:00:40.033 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.893+0000 7f72167fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:40.034 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.893+0000 7f72167fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:40.034 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.893+0000 7f72167fc700 10 monclient(hunting): my global_id is 4637 2021-05-25T12:00:40.034 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.893+0000 7f72167fc700 1 -- 172.21.15.102:0/879158789 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7228097be0 con 0x7f7208002840 2021-05-25T12:00:40.034 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.893+0000 7f72167fc700 1 -- 172.21.15.102:0/879158789 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2981596874 0 0) 0x7f7228097be0 con 0x7f7208002840 2021-05-25T12:00:40.034 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.893+0000 7f72167fc700 1 -- 172.21.15.102:0/879158789 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f72080065a0 con 0x7f7208002840 2021-05-25T12:00:40.035 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.893+0000 7f72167fc700 1 -- 172.21.15.102:0/879158789 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f721c001660 con 0x7f7208002840 2021-05-25T12:00:40.035 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.893+0000 7f72167fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:40.035 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.893+0000 7f72167fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:00:40.035 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.893+0000 7f72167fc700 10 monclient: dump: 2021-05-25T12:00:40.035 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:40.036 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:40.036 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:40.036 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:40.036 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:40.036 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:40.037 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:40.037 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:40.037 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:40.037 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:40.038 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.893+0000 7f7226d9d700 10 monclient: get_auth_request con 0x7f7204033170 auth_method 0 2021-05-25T12:00:40.038 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.893+0000 7f72167fc700 1 -- 172.21.15.102:0/879158789 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 812751288 0 0) 0x7f72080065a0 con 0x7f7208002840 2021-05-25T12:00:40.038 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.893+0000 7f72167fc700 1 -- 172.21.15.102:0/879158789 --> v1:172.21.15.102:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7208009e40 con 0x7f7208002840 2021-05-25T12:00:40.038 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.893+0000 7f72167fc700 10 client.4631.objecter ms_handle_connect 0x7f7204033170 2021-05-25T12:00:40.038 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.894+0000 7f72167fc700 1 -- 172.21.15.102:0/879158789 <== mon.2 v1:172.21.15.102:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7f721c002b50 con 0x7f7208002840 2021-05-25T12:00:40.039 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.894+0000 7f72167fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:00:40.039 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.894+0000 7f72167fc700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:00:40.039 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.894+0000 7f72167fc700 1 -- 172.21.15.102:0/879158789 >> v1:172.21.15.102:6790/0 conn(0x7f7208002840 legacy=0x7f7208004ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:40.039 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.894+0000 7f72167fc700 1 -- 172.21.15.102:0/879158789 >> v1:172.21.15.102:6790/0 conn(0x7f7208002840 legacy=0x7f7208004ce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:00:40.040 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.901+0000 7f722759e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:00:40.040 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.901+0000 7f722759e700 1 -- 172.21.15.102:0/879158789 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f7208007510 con 0x7f72080068b0 2021-05-25T12:00:40.040 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.902+0000 7f72167fc700 10 client.4631.objecter ms_handle_connect 0x7f72080068b0 2021-05-25T12:00:40.040 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.902+0000 7f72167fc700 10 client.4631.objecter resend_mon_ops 2021-05-25T12:00:40.040 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.902+0000 7f72167fc700 1 -- 172.21.15.102:0/879158789 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1899255288 0 0) 0x7f7208007510 con 0x7f72080068b0 2021-05-25T12:00:40.041 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.902+0000 7f72167fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:40.041 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.902+0000 7f72167fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:40.041 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.902+0000 7f72167fc700 10 monclient(hunting): my global_id is 4655 2021-05-25T12:00:40.041 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.902+0000 7f72167fc700 1 -- 172.21.15.102:0/879158789 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72080065a0 con 0x7f72080068b0 2021-05-25T12:00:40.041 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.902+0000 7f72167fc700 1 -- 172.21.15.102:0/879158789 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1344239429 0 0) 0x7f72080065a0 con 0x7f72080068b0 2021-05-25T12:00:40.042 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.902+0000 7f72167fc700 1 -- 172.21.15.102:0/879158789 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7208007510 con 0x7f72080068b0 2021-05-25T12:00:40.042 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.902+0000 7f72167fc700 1 -- 172.21.15.102:0/879158789 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f721c001c10 con 0x7f72080068b0 2021-05-25T12:00:40.042 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.902+0000 7f72167fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:40.042 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.902+0000 7f72167fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:00:40.042 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.902+0000 7f72167fc700 10 monclient: dump: 2021-05-25T12:00:40.043 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:40.043 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:40.043 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:40.043 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:40.044 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:40.044 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:40.044 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:40.044 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:40.044 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:40.045 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:40.045 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.902+0000 7f72167fc700 1 -- 172.21.15.102:0/879158789 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1002598466 0 0) 0x7f7208007510 con 0x7f72080068b0 2021-05-25T12:00:40.045 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.902+0000 7f72167fc700 1 -- 172.21.15.102:0/879158789 --> v1:172.21.15.102:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f7208008770 con 0x7f72080068b0 2021-05-25T12:00:40.045 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.903+0000 7f72167fc700 1 -- 172.21.15.102:0/879158789 <== mon.2 v1:172.21.15.102:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4111 (unknown 0 0 1388598372) 0x7f721c005510 con 0x7f72080068b0 2021-05-25T12:00:40.045 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.903+0000 7f72167fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:00:40.046 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.903+0000 7f72167fc700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:00:40.046 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.903+0000 7f72167fc700 1 -- 172.21.15.102:0/879158789 >> v1:172.21.15.102:6790/0 conn(0x7f72080068b0 legacy=0x7f7208006d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:40.046 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.903+0000 7f72167fc700 1 -- 172.21.15.102:0/879158789 >> v1:172.21.15.102:6790/0 conn(0x7f72080068b0 legacy=0x7f7208006d40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:00:40.047 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:00:40.047 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:00:40.047 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:00:40.047 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "peon", 2021-05-25T12:00:40.048 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 14, 2021-05-25T12:00:40.048 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:00:40.048 INFO:teuthology.orchestra.run.smithi102.stdout: 0, 2021-05-25T12:00:40.048 INFO:teuthology.orchestra.run.smithi102.stdout: 1, 2021-05-25T12:00:40.049 INFO:teuthology.orchestra.run.smithi102.stdout: 2 2021-05-25T12:00:40.049 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:00:40.049 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 3, 2021-05-25T12:00:40.049 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:00:40.049 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:00:40.050 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:00:40.050 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:00:40.050 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:00:40.050 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:00:40.050 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:00:40.051 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:00:40.051 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:00:40.051 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:00:40.051 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:00:40.052 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:00:40.052 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:00:40.052 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:00:40.052 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:00:40.052 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:00:40.053 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:00:40.053 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:00:40.053 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:00:40.053 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:00:40.053 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:00:40.054 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:00:40.054 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:00:40.054 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:00:40.054 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:00:40.055 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:40.055 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:00:40.055 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:00:40.055 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:00:40.055 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:00:40.056 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:00:40.056 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:00:40.056 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:00:40.056 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:00:40.056 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:00:40.057 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:00:40.057 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:00:40.057 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:00:40.057 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:00:40.058 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:00:40.058 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:00:40.058 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:00:40.058 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:00:40.058 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:00:40.059 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:00:40.059 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:00:40.059 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:00:40.059 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:00:40.059 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:00:40.060 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:00:40.060 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:00:40.060 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:00:40.060 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:40.061 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:00:40.061 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:40.061 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:00:40.061 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:00:40.061 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:00:40.062 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:00:40.062 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:40.062 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:00:40.062 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:00:40.062 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:00:40.063 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:40.063 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:00:40.063 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:40.063 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:00:40.063 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:00:40.064 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:00:40.064 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:00:40.064 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:00:40.064 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:40.064 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:40.065 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:00:40.065 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:00:40.065 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:00:40.065 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:00:40.065 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:40.066 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:00:40.066 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:00:40.066 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:00:40.066 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:40.067 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:00:40.067 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:40.067 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:00:40.067 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:00:40.067 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:00:40.068 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:00:40.068 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:00:40.068 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:40.068 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:40.068 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:00:40.069 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:00:40.069 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:00:40.069 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:00:40.069 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:40.069 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:00:40.070 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:00:40.070 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:00:40.070 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:40.070 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:00:40.071 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:40.071 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:00:40.071 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:00:40.071 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:00:40.071 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:00:40.072 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:00:40.072 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:40.072 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:00:40.072 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:40.072 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:00:40.073 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:00:40.073 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:40.073 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:00:40.073 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:00:40.074 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:00:40.074 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:40.074 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:00:40.074 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": [ 2021-05-25T12:00:40.074 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:40.075 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:00:40.075 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:00:40.075 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 5 2021-05-25T12:00:40.075 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:40.076 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:00:40.076 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:00:40.076 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:40.076 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:00:40.076 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:00:40.077 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 20 2021-05-25T12:00:40.077 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:40.077 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:00:40.077 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": [ 2021-05-25T12:00:40.077 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:40.078 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:00:40.078 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:00:40.078 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:00:40.078 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:40.078 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:00:40.079 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:40.079 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:00:40.079 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:00:40.080 INFO:tasks.workunit.client.0.smithi181.stderr:+ pid=33355 2021-05-25T12:00:40.080 INFO:tasks.workunit.client.0.smithi181.stderr:+ echo 'test api_cls_remote_reads on pid 33355' 2021-05-25T12:00:40.081 INFO:tasks.workunit.client.0.smithi181.stderr:+ pids[$f]=33355 2021-05-25T12:00:40.081 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2021-05-25T12:00:40.081 INFO:tasks.workunit.client.0.smithi181.stderr:+ '[' 1 -eq 1 ']' 2021-05-25T12:00:40.081 INFO:tasks.workunit.client.0.smithi181.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cls_remote_reads 2>&1 | tee ceph_test_rados_api_cls_remote_reads.log | sed "s/^/ api_cls_remote_reads: /"' 2021-05-25T12:00:40.081 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph_test_rados_api_cls_remote_reads 2021-05-25T12:00:40.082 INFO:tasks.workunit.client.0.smithi181.stderr:++ printf %25s list_parallel 2021-05-25T12:00:40.082 INFO:tasks.workunit.client.0.smithi181.stderr:+ r=' list_parallel' 2021-05-25T12:00:40.082 INFO:tasks.workunit.client.0.smithi181.stderr:+ sed 's/^/ api_cls_remote_reads: /' 2021-05-25T12:00:40.082 INFO:tasks.workunit.client.0.smithi181.stderr:++ awk '{print $1}' 2021-05-25T12:00:40.083 INFO:tasks.workunit.client.0.smithi181.stderr:++ echo list_parallel 2021-05-25T12:00:40.083 INFO:tasks.workunit.client.0.smithi181.stdout:test list_parallel on pid 33374 2021-05-25T12:00:40.085 INFO:tasks.workunit.client.0.smithi181.stderr:+ ff=list_parallel 2021-05-25T12:00:40.086 INFO:tasks.workunit.client.0.smithi181.stderr:+ pid=33374 2021-05-25T12:00:40.086 INFO:tasks.workunit.client.0.smithi181.stderr:+ echo 'test list_parallel on pid 33374' 2021-05-25T12:00:40.086 INFO:tasks.workunit.client.0.smithi181.stderr:+ pids[$f]=33374 2021-05-25T12:00:40.086 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2021-05-25T12:00:40.087 INFO:tasks.workunit.client.0.smithi181.stderr:+ '[' 1 -eq 1 ']' 2021-05-25T12:00:40.087 INFO:tasks.workunit.client.0.smithi181.stderr:+ bash -o pipefail -exc 'ceph_test_rados_list_parallel 2>&1 | tee ceph_test_rados_list_parallel.log | sed "s/^/ list_parallel: /"' 2021-05-25T12:00:40.087 INFO:tasks.workunit.client.0.smithi181.stderr:++ printf %25s open_pools_parallel 2021-05-25T12:00:40.087 INFO:tasks.workunit.client.0.smithi181.stderr:+ sed 's/^/ list_parallel: /' 2021-05-25T12:00:40.087 INFO:tasks.workunit.client.0.smithi181.stderr:+ tee ceph_test_rados_list_parallel.log 2021-05-25T12:00:40.088 INFO:tasks.workunit.client.0.smithi181.stderr:+ r=' open_pools_parallel' 2021-05-25T12:00:40.088 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph_test_rados_list_parallel 2021-05-25T12:00:40.088 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph_test_rados_api_c_read_operations 2021-05-25T12:00:40.088 INFO:tasks.workunit.client.0.smithi181.stderr:+ tee ceph_test_rados_api_cls_remote_reads.log 2021-05-25T12:00:40.089 INFO:tasks.workunit.client.0.smithi181.stderr:++ awk '{print $1}' 2021-05-25T12:00:40.089 INFO:tasks.workunit.client.0.smithi181.stderr:++ echo open_pools_parallel 2021-05-25T12:00:40.089 INFO:tasks.workunit.client.0.smithi181.stdout:test open_pools_parallel on pid 33415 2021-05-25T12:00:40.090 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.903+0000 7f722759e700 10 librados: watch_flush enter 2021-05-25T12:00:40.091 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.903+0000 7f722759e700 10 librados: watch_flush exit 2021-05-25T12:00:40.091 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.903+0000 7f722759e700 20 client.4631.objecter shutdown clearing up homeless session... 2021-05-25T12:00:40.091 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.903+0000 7f722759e700 10 client.4631.objecter successfully canceled tick 2021-05-25T12:00:40.091 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.903+0000 7f722759e700 1 -- 172.21.15.102:0/879158789 >> v1:172.21.15.181:6812/29582 conn(0x7f7204033170 legacy=0x7f7204035630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:40.091 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.903+0000 7f722759e700 10 monclient: shutdown 2021-05-25T12:00:40.092 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.903+0000 7f722759e700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:00:40.092 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.903+0000 7f722759e700 1 -- 172.21.15.102:0/879158789 >> v1:172.21.15.102:6790/0 conn(0x7f722812bca0 legacy=0x7f72280974d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:40.092 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.903+0000 7f7226d9d700 1 -- 172.21.15.102:0/879158789 reap_dead start 2021-05-25T12:00:40.092 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.903+0000 7f7226d9d700 1 -- 172.21.15.102:0/879158789 reap_dead start 2021-05-25T12:00:40.092 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.903+0000 7f722759e700 1 -- 172.21.15.102:0/879158789 shutdown_connections 2021-05-25T12:00:40.093 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.903+0000 7f722759e700 1 -- 172.21.15.102:0/879158789 >> 172.21.15.102:0/879158789 conn(0x7f722809e9d0 msgr2=0x7f7228128770 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:00:40.093 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.903+0000 7f722759e700 1 -- 172.21.15.102:0/879158789 shutdown_connections 2021-05-25T12:00:40.093 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.903+0000 7f722759e700 1 -- 172.21.15.102:0/879158789 wait complete. 2021-05-25T12:00:40.093 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:39.903+0000 7f722759e700 1 librados: shutdown 2021-05-25T12:00:40.094 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2021-05-25T12:00:40.095 INFO:tasks.workunit.client.0.smithi181.stderr:+ ff=open_pools_parallel 2021-05-25T12:00:40.096 INFO:tasks.workunit.client.0.smithi181.stderr:+ pid=33415 2021-05-25T12:00:40.096 INFO:tasks.workunit.client.0.smithi181.stderr:+ echo 'test open_pools_parallel on pid 33415' 2021-05-25T12:00:40.096 INFO:tasks.workunit.client.0.smithi181.stderr:+ pids[$f]=33415 2021-05-25T12:00:40.096 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2021-05-25T12:00:40.097 INFO:tasks.workunit.client.0.smithi181.stderr:+ '[' 1 -eq 1 ']' 2021-05-25T12:00:40.097 INFO:tasks.workunit.client.0.smithi181.stderr:+ bash -o pipefail -exc 'ceph_test_rados_open_pools_parallel 2>&1 | tee ceph_test_rados_open_pools_parallel.log | sed "s/^/ open_pools_parallel: /"' 2021-05-25T12:00:40.097 INFO:tasks.workunit.client.0.smithi181.stderr:++ printf %25s delete_pools_parallel 2021-05-25T12:00:40.097 INFO:tasks.workunit.client.0.smithi181.stderr:+ sed 's/^/ open_pools_parallel: /' 2021-05-25T12:00:40.098 INFO:tasks.workunit.client.0.smithi181.stderr:+ r=' delete_pools_parallel' 2021-05-25T12:00:40.098 INFO:tasks.workunit.client.0.smithi181.stderr:+ tee ceph_test_rados_open_pools_parallel.log 2021-05-25T12:00:40.098 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph_test_rados_open_pools_parallel 2021-05-25T12:00:40.098 INFO:tasks.workunit.client.0.smithi181.stderr:++ awk '{print $1}' 2021-05-25T12:00:40.098 INFO:tasks.workunit.client.0.smithi181.stderr:++ echo delete_pools_parallel 2021-05-25T12:00:40.099 INFO:tasks.workunit.client.0.smithi181.stdout:test delete_pools_parallel on pid 33440 2021-05-25T12:00:40.100 INFO:tasks.workunit.client.0.smithi181.stderr:+ ff=delete_pools_parallel 2021-05-25T12:00:40.100 INFO:tasks.workunit.client.0.smithi181.stderr:+ pid=33440 2021-05-25T12:00:40.100 INFO:tasks.workunit.client.0.smithi181.stderr:+ echo 'test delete_pools_parallel on pid 33440' 2021-05-25T12:00:40.100 INFO:tasks.workunit.client.0.smithi181.stderr:+ pids[$f]=33440 2021-05-25T12:00:40.101 INFO:tasks.workunit.client.0.smithi181.stderr:+ ret=0 2021-05-25T12:00:40.101 INFO:tasks.workunit.client.0.smithi181.stderr:+ '[' 1 -eq 1 ']' 2021-05-25T12:00:40.101 INFO:tasks.workunit.client.0.smithi181.stderr:+ for t in "${!pids[@]}" 2021-05-25T12:00:40.101 INFO:tasks.workunit.client.0.smithi181.stderr:+ pid=33061 2021-05-25T12:00:40.101 INFO:tasks.workunit.client.0.smithi181.stderr:+ wait 33061 2021-05-25T12:00:40.102 INFO:tasks.workunit.client.0.smithi181.stderr:+ bash -o pipefail -exc 'ceph_test_rados_delete_pools_parallel 2>&1 | tee ceph_test_rados_delete_pools_parallel.log | sed "s/^/ delete_pools_parallel: /"' 2021-05-25T12:00:40.102 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph_test_rados_delete_pools_parallel 2021-05-25T12:00:40.102 INFO:tasks.workunit.client.0.smithi181.stderr:+ sed 's/^/ delete_pools_parallel: /' 2021-05-25T12:00:40.103 INFO:tasks.workunit.client.0.smithi181.stderr:+ tee ceph_test_rados_delete_pools_parallel.log 2021-05-25T12:00:40.129 INFO:tasks.workunit.client.0.smithi181.stdout:x7ff87803be50 con 0x55fe52566e20 2021-05-25T12:00:40.129 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.728+0000 7ff884a90700 10 client.4511.objecter ms_dispatch 0x55fe524d4560 osd_map(15..15 src has 1..15) v4 2021-05-25T12:00:40.129 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.728+0000 7ff884a90700 3 client.4511.objecter handle_osd_map got epochs [15,15] > 0 2021-05-25T12:00:40.129 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.728+0000 7ff884a90700 3 client.4511.objecter handle_osd_map decoding full epoch 15 2021-05-25T12:00:40.130 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.728+0000 7ff884a90700 20 client.4511.objecter dump_active .. 0 homeless 2021-05-25T12:00:40.130 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.728+0000 7ff887295700 10 monclient: get_auth_request con 0x7ff874031be0 auth_method 0 2021-05-25T12:00:40.130 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.728+0000 7ff88fe07540 10 librados: wait_for_osdmap done waiting 2021-05-25T12:00:40.130 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.728+0000 7ff88fe07540 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:00:40.130 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.728+0000 7ff88fe07540 1 -- 172.21.15.181:0/4180131642 --> v1:172.21.15.102:6790/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff880001930 con 0x55fe52566e20 2021-05-25T12:00:40.131 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.728+0000 7ff884a90700 10 client.4511.objecter ms_handle_connect 0x7ff874031be0 2021-05-25T12:00:40.131 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.728+0000 7ff884a90700 1 -- 172.21.15.181:0/4180131642 <== mon.2 v1:172.21.15.102:6790/0 9 ==== mon_get_version_reply(handle=1 version=15) v2 ==== 24+0+0 (unknown 2942983318 0 0) 0x7ff8780024a0 con 0x55fe52566e20 2021-05-25T12:00:40.131 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.728+0000 7ff884a90700 10 monclient: handle_get_version_reply finishing 1 version 15 2021-05-25T12:00:40.131 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.728+0000 7ff885a92700 10 client.4511.objecter _get_latest_version latest 15, have it 2021-05-25T12:00:40.132 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.728+0000 7ff88fe07540 10 client.4511.objecter create_pool name=ceph_test_rados_api_asio 2021-05-25T12:00:40.132 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.728+0000 7ff88fe07540 10 client.4511.objecter pool_op_submit 1 2021-05-25T12:00:40.132 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.728+0000 7ff88fe07540 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:00:40.132 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:39.728+0000 7ff88fe07540 1 -- 172.21.15.181:0/4180131642 --> v1:172.21.15.102:6790/0 -- pool_op(create pool 0 tid 1 name ceph_test_rados_api_asio v0) v4 -- 0x55fe524bb630 con 0x55fe52566e20 2021-05-25T12:00:40.132 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:40.126+0000 7ff884a90700 1 -- 172.21.15.181:0/4180131642 <== mon.2 v1:172.21.15.102:6790/0 10 ==== pool_op_reply(tid 1 (0) Success v16) v1 ==== 43+0+0 (unknown 457885425 0 0) 0x7ff878011b10 con 0x55fe52566e20 2021-05-25T12:00:40.133 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:40.126+0000 7ff884a90700 10 client.4511.objecter ms_dispatch 0x55fe524d4560 pool_op_reply(tid 1 (0) Success v16) v1 2021-05-25T12:00:40.133 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:40.126+0000 7ff884a90700 10 client.4511.objecter handle_pool_op_reply pool_op_reply(tid 1 (0) Success v16) v1 2021-05-25T12:00:40.133 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:40.126+0000 7ff884a90700 10 client.4511.objecter have request 1 at 0x55fe5255e8e0 Op: create 2021-05-25T12:00:40.133 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:40.126+0000 7ff884a90700 20 client.4511.objecter waiting for client to reach epoch 16 before calling back 2021-05-25T12:00:40.133 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:40.126+0000 7ff884a90700 10 client.4511.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:00:40.134 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:40.126+0000 7ff884a90700 10 monclient: _renew_subs 2021-05-25T12:00:40.134 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:40.126+0000 7ff884a90700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:00:40.134 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:40.126+0000 7ff884a90700 1 -- 172.21.15.181:0/4180131642 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=16}) v3 -- 0x55fe5255eaa0 con 0x55fe52566e20 2021-05-25T12:00:40.134 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:40.126+0000 7ff884a90700 10 client.4511.objecter done 2021-05-25T12:00:40.236 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.234+0000 7efc4d982700 10 monclient: get_monmap_and_config 2021-05-25T12:00:40.236 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.234+0000 7efc4d982700 10 monclient: build_initial_monmap 2021-05-25T12:00:40.236 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.234+0000 7efc4d982700 10 monclient: monmap: 2021-05-25T12:00:40.237 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:00:40.237 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:40.237 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:00:40.235429+0000 2021-05-25T12:00:40.237 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:00:40.235429+0000 2021-05-25T12:00:40.237 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:00:40.238 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:40.238 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:00:40.238 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:00:40.238 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:00:40.239 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:40.239 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.234+0000 7efc4d982700 1 Processor -- start 2021-05-25T12:00:40.239 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.235+0000 7efc4d982700 1 -- start start 2021-05-25T12:00:40.239 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.235+0000 7efc4d982700 10 monclient: init 2021-05-25T12:00:40.253 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.235+0000 7efc4d982700 10 monclient: _reopen_session rank -1 2021-05-25T12:00:40.254 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.235+0000 7efc4d982700 10 monclient: _add_conns ranks=[2,0,1] 2021-05-25T12:00:40.254 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.235+0000 7efc4d982700 10 monclient(hunting): picked mon.noname-b con 0x7efc48068d50 addr v1:172.21.15.102:6790/0 2021-05-25T12:00:40.254 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.235+0000 7efc4d982700 10 monclient(hunting): picked mon.noname-a con 0x7efc48069130 addr v1:172.21.15.102:6789/0 2021-05-25T12:00:40.254 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.235+0000 7efc4d982700 10 monclient(hunting): picked mon.noname-c con 0x7efc480695e0 addr v1:172.21.15.181:6789/0 2021-05-25T12:00:40.255 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.235+0000 7efc4d982700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc48094380 con 0x7efc48069130 2021-05-25T12:00:40.255 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.235+0000 7efc4d982700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc48094620 con 0x7efc480695e0 2021-05-25T12:00:40.255 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.235+0000 7efc4d982700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc480948c0 con 0x7efc48068d50 2021-05-25T12:00:40.255 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.235+0000 7efc4d982700 10 monclient(hunting): _renew_subs 2021-05-25T12:00:40.255 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.235+0000 7efc4d982700 10 monclient(hunting): authenticate will time out at 840.832562s 2021-05-25T12:00:40.256 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.235+0000 7efc46ffd700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7efc48068d50 0x7efc48129c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:54294/0 (socket says 172.21.15.102:54294) 2021-05-25T12:00:40.256 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.235+0000 7efc46ffd700 1 -- 172.21.15.102:0/1215139199 learned_addr learned my addr 172.21.15.102:0/1215139199 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:00:40.256 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.235+0000 7efc45ffb700 1 -- 172.21.15.102:0/1215139199 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1954307544 0 0) 0x7efc480948c0 con 0x7efc48068d50 2021-05-25T12:00:40.256 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.235+0000 7efc45ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:40.257 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.235+0000 7efc45ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:40.257 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.235+0000 7efc45ffb700 10 monclient(hunting): my global_id is 4700 2021-05-25T12:00:40.257 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.235+0000 7efc45ffb700 1 -- 172.21.15.102:0/1215139199 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc34003270 con 0x7efc48068d50 2021-05-25T12:00:40.257 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.236+0000 7efc45ffb700 1 -- 172.21.15.102:0/1215139199 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 417345805 0 0) 0x7efc48094620 con 0x7efc480695e0 2021-05-25T12:00:40.257 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.236+0000 7efc45ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:40.258 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.236+0000 7efc45ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:40.258 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.236+0000 7efc45ffb700 10 monclient(hunting): my global_id is 74374 2021-05-25T12:00:40.258 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.236+0000 7efc45ffb700 1 -- 172.21.15.102:0/1215139199 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc480948c0 con 0x7efc480695e0 2021-05-25T12:00:40.258 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.236+0000 7efc45ffb700 1 -- 172.21.15.102:0/1215139199 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3301281805 0 0) 0x7efc34003270 con 0x7efc48068d50 2021-05-25T12:00:40.258 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.236+0000 7efc45ffb700 1 -- 172.21.15.102:0/1215139199 >> v1:172.21.15.181:6789/0 conn(0x7efc480695e0 legacy=0x7efc48132160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:40.259 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.236+0000 7efc45ffb700 1 -- 172.21.15.102:0/1215139199 >> v1:172.21.15.102:6789/0 conn(0x7efc48069130 legacy=0x7efc4812ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:40.259 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.236+0000 7efc45ffb700 10 monclient: _finish_hunting 0 2021-05-25T12:00:40.259 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.236+0000 7efc45ffb700 1 monclient: found mon.noname-b 2021-05-25T12:00:40.259 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.236+0000 7efc45ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:00:40.260 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.236+0000 7efc45ffb700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T12:00:40.260 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.236+0000 7efc45ffb700 1 -- 172.21.15.102:0/1215139199 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc48094c80 con 0x7efc48068d50 2021-05-25T12:00:40.260 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.236+0000 7efc45ffb700 10 monclient: _finish_auth 0 2021-05-25T12:00:40.260 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.236+0000 7efc45ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:00:40.260 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.236+0000 7efc45ffb700 1 -- 172.21.15.102:0/1215139199 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7efc3c001610 con 0x7efc48068d50 2021-05-25T12:00:40.261 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.236+0000 7efc45ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:40.261 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.236+0000 7efc45ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:00:40.261 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.236+0000 7efc45ffb700 10 monclient: dump: 2021-05-25T12:00:40.261 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:40.262 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:40.262 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:40.262 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:40.262 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:40.262 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:40.263 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:40.263 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:40.263 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:40.263 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:40.263 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.236+0000 7efc4d982700 5 monclient: authenticate success, global_id 4700 2021-05-25T12:00:40.264 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.236+0000 7efc4d982700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:00:40.264 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.236+0000 7efc45ffb700 1 -- 172.21.15.102:0/1215139199 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efc3c001830 con 0x7efc48068d50 2021-05-25T12:00:40.264 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.236+0000 7efc45ffb700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:00:40.264 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.236+0000 7efc4d982700 10 monclient: get_monmap_and_config success 2021-05-25T12:00:40.265 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.236+0000 7efc45ffb700 1 -- 172.21.15.102:0/1215139199 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7efc3c001bc0 con 0x7efc48068d50 2021-05-25T12:00:40.265 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.236+0000 7efc4d982700 4 set_mon_vals no callback set 2021-05-25T12:00:40.265 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.236+0000 7efc45ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:40.265 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.236+0000 7efc45ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:00:40.265 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.236+0000 7efc45ffb700 10 monclient: dump: 2021-05-25T12:00:40.266 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:40.266 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:40.266 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:40.266 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:40.266 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:40.267 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:40.267 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:40.267 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:40.267 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:40.267 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:40.268 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.236+0000 7efc4d982700 10 monclient: shutdown 2021-05-25T12:00:40.268 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.236+0000 7efc4d982700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:00:40.268 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.236+0000 7efc4d982700 1 -- 172.21.15.102:0/1215139199 >> v1:172.21.15.102:6790/0 conn(0x7efc48068d50 legacy=0x7efc48129c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:40.268 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.236+0000 7efc4d982700 1 -- 172.21.15.102:0/1215139199 shutdown_connections 2021-05-25T12:00:40.269 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.236+0000 7efc4d982700 1 -- 172.21.15.102:0/1215139199 >> 172.21.15.102:0/1215139199 conn(0x7efc4809b370 msgr2=0x7efc48097430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:00:40.269 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.236+0000 7efc4d982700 1 -- 172.21.15.102:0/1215139199 shutdown_connections 2021-05-25T12:00:40.269 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.236+0000 7efc4d982700 1 -- 172.21.15.102:0/1215139199 wait complete. 2021-05-25T12:00:40.269 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.237+0000 7efc4d982700 10 monclient: build_initial_monmap 2021-05-25T12:00:40.269 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.237+0000 7efc4d982700 10 monclient: monmap: 2021-05-25T12:00:40.270 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:00:40.270 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:00:40.270 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:00:40.270 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:00:40.270 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:00:40.271 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:00:40.271 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:00:40.271 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:00:40.271 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:00:40.272 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:40.272 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.237+0000 7efc4d982700 1 librados: starting msgr at 2021-05-25T12:00:40.272 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.237+0000 7efc4d982700 1 librados: starting objecter 2021-05-25T12:00:40.272 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.237+0000 7efc4d982700 1 Processor -- start 2021-05-25T12:00:40.272 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.237+0000 7efc4d982700 1 -- start start 2021-05-25T12:00:40.273 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.237+0000 7efc4d982700 1 librados: setting wanted keys 2021-05-25T12:00:40.273 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.237+0000 7efc4d982700 1 librados: calling monclient init 2021-05-25T12:00:40.273 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.237+0000 7efc4d982700 10 monclient: init 2021-05-25T12:00:40.273 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.238+0000 7efc4d982700 10 monclient: _reopen_session rank -1 2021-05-25T12:00:40.273 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.238+0000 7efc4d982700 10 monclient: _add_conns ranks=[0,1,2] 2021-05-25T12:00:40.274 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.238+0000 7efc4d982700 10 monclient(hunting): picked mon.noname-a con 0x7efc48068d50 addr v1:172.21.15.102:6789/0 2021-05-25T12:00:40.274 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.238+0000 7efc4d982700 10 monclient(hunting): picked mon.noname-b con 0x7efc48069130 addr v1:172.21.15.181:6789/0 2021-05-25T12:00:40.274 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.238+0000 7efc4d982700 10 monclient(hunting): picked mon.noname-c con 0x7efc480695e0 addr v1:172.21.15.102:6790/0 2021-05-25T12:00:40.274 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.238+0000 7efc4d982700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc48121b90 con 0x7efc48068d50 2021-05-25T12:00:40.275 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.238+0000 7efc4d982700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc48091240 con 0x7efc48069130 2021-05-25T12:00:40.275 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.238+0000 7efc4d982700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc48121dc0 con 0x7efc480695e0 2021-05-25T12:00:40.275 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.238+0000 7efc4d982700 10 monclient(hunting): _renew_subs 2021-05-25T12:00:40.275 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.238+0000 7efc4d982700 10 monclient(hunting): authenticate will time out at 840.835215s 2021-05-25T12:00:40.276 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.238+0000 7efc46ffd700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7efc48068d50 0x7efc48138e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:59420/0 (socket says 172.21.15.102:59420) 2021-05-25T12:00:40.276 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.238+0000 7efc46ffd700 1 -- 172.21.15.102:0/3517856888 learned_addr learned my addr 172.21.15.102:0/3517856888 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:00:40.276 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.238+0000 7efc2effd700 10 client.?.objecter ms_handle_connect 0x7efc480695e0 2021-05-25T12:00:40.276 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.238+0000 7efc2effd700 10 client.?.objecter resend_mon_ops 2021-05-25T12:00:40.276 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.238+0000 7efc2effd700 10 client.?.objecter ms_handle_connect 0x7efc48068d50 2021-05-25T12:00:40.277 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.238+0000 7efc2effd700 10 client.?.objecter resend_mon_ops 2021-05-25T12:00:40.277 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.238+0000 7efc2effd700 10 client.?.objecter ms_handle_connect 0x7efc48069130 2021-05-25T12:00:40.277 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.238+0000 7efc2effd700 10 client.?.objecter resend_mon_ops 2021-05-25T12:00:40.277 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.238+0000 7efc2effd700 1 -- 172.21.15.102:0/3517856888 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1421509837 0 0) 0x7efc48121dc0 con 0x7efc480695e0 2021-05-25T12:00:40.278 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.238+0000 7efc2effd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:40.278 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.238+0000 7efc2effd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:40.278 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.238+0000 7efc2effd700 10 monclient(hunting): my global_id is 4703 2021-05-25T12:00:40.278 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.238+0000 7efc2effd700 1 -- 172.21.15.102:0/3517856888 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc28004c10 con 0x7efc480695e0 2021-05-25T12:00:40.278 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.238+0000 7efc2effd700 1 -- 172.21.15.102:0/3517856888 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 373141021 0 0) 0x7efc48091240 con 0x7efc48069130 2021-05-25T12:00:40.279 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.238+0000 7efc2effd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:40.279 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.238+0000 7efc2effd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:40.279 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.238+0000 7efc2effd700 10 monclient(hunting): my global_id is 74377 2021-05-25T12:00:40.279 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.238+0000 7efc2effd700 1 -- 172.21.15.102:0/3517856888 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc48121dc0 con 0x7efc48069130 2021-05-25T12:00:40.279 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.238+0000 7efc2effd700 1 -- 172.21.15.102:0/3517856888 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3206682935 0 0) 0x7efc28004c10 con 0x7efc480695e0 2021-05-25T12:00:40.280 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.238+0000 7efc2effd700 1 -- 172.21.15.102:0/3517856888 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efc48091240 con 0x7efc480695e0 2021-05-25T12:00:40.280 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.238+0000 7efc2effd700 1 -- 172.21.15.102:0/3517856888 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7efc38001d10 con 0x7efc480695e0 2021-05-25T12:00:40.280 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.238+0000 7efc2effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:40.280 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.238+0000 7efc2effd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:00:40.280 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.238+0000 7efc2effd700 10 monclient(hunting): dump: 2021-05-25T12:00:40.281 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:40.281 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:40.281 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:40.281 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:40.282 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:40.282 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:40.282 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:40.282 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:40.282 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:40.283 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:40.283 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.238+0000 7efc2effd700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:00:40.283 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.238+0000 7efc2effd700 1 -- 172.21.15.102:0/3517856888 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1319355413 0 0) 0x7efc48121dc0 con 0x7efc48069130 2021-05-25T12:00:40.283 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.238+0000 7efc2effd700 1 -- 172.21.15.102:0/3517856888 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efc28004c10 con 0x7efc48069130 2021-05-25T12:00:40.283 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.238+0000 7efc2effd700 1 -- 172.21.15.102:0/3517856888 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7efc300017f0 con 0x7efc48069130 2021-05-25T12:00:40.284 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.238+0000 7efc2effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:40.284 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.238+0000 7efc2effd700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2021-05-25T12:00:40.284 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.238+0000 7efc2effd700 10 monclient(hunting): dump: 2021-05-25T12:00:40.284 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:40.284 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:40.285 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:40.285 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:40.285 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:40.285 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:40.286 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:40.286 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:40.286 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:40.286 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:40.286 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.238+0000 7efc2effd700 1 -- 172.21.15.102:0/3517856888 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1748724659 0 0) 0x7efc48091240 con 0x7efc480695e0 2021-05-25T12:00:40.287 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.238+0000 7efc2effd700 1 -- 172.21.15.102:0/3517856888 >> v1:172.21.15.181:6789/0 conn(0x7efc48069130 legacy=0x7efc48125530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:40.287 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.238+0000 7efc2effd700 1 -- 172.21.15.102:0/3517856888 >> v1:172.21.15.102:6789/0 conn(0x7efc48068d50 legacy=0x7efc48138e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:40.287 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.238+0000 7efc2effd700 10 monclient: _finish_hunting 0 2021-05-25T12:00:40.287 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.238+0000 7efc2effd700 1 monclient: found mon.c 2021-05-25T12:00:40.287 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.238+0000 7efc2effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:00:40.288 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.238+0000 7efc2effd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:00:40.288 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.238+0000 7efc2effd700 1 -- 172.21.15.102:0/3517856888 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc48122060 con 0x7efc480695e0 2021-05-25T12:00:40.288 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.238+0000 7efc2effd700 10 monclient: _finish_auth 0 2021-05-25T12:00:40.288 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.238+0000 7efc2effd700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:00:40.288 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.238+0000 7efc4d982700 5 monclient: authenticate success, global_id 4703 2021-05-25T12:00:40.289 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.238+0000 7efc4d982700 10 monclient: _renew_subs 2021-05-25T12:00:40.289 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.238+0000 7efc4d982700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:00:40.289 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.238+0000 7efc4d982700 1 -- 172.21.15.102:0/3517856888 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc480685b0 con 0x7efc480695e0 2021-05-25T12:00:40.289 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.238+0000 7efc4d982700 10 client.4703.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:00:40.290 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.238+0000 7efc4d982700 10 monclient: _renew_subs 2021-05-25T12:00:40.290 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.238+0000 7efc4d982700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:00:40.290 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.238+0000 7efc4d982700 1 -- 172.21.15.102:0/3517856888 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efc48125c40 con 0x7efc480695e0 2021-05-25T12:00:40.290 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.238+0000 7efc4d982700 1 librados: init done 2021-05-25T12:00:40.290 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.239+0000 7efc2effd700 1 -- 172.21.15.102:0/3517856888 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efc380022f0 con 0x7efc480695e0 2021-05-25T12:00:40.291 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.239+0000 7efc2effd700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:00:40.291 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.239+0000 7efc2effd700 1 -- 172.21.15.102:0/3517856888 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7efc38002680 con 0x7efc480695e0 2021-05-25T12:00:40.291 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.239+0000 7efc2effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:40.291 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.239+0000 7efc2effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:00:40.291 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.239+0000 7efc2effd700 10 monclient: dump: 2021-05-25T12:00:40.292 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:40.292 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:40.292 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:40.292 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:40.293 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:40.293 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:40.293 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:40.293 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:40.293 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:40.294 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:40.294 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.239+0000 7efc457fa700 4 set_mon_vals no callback set 2021-05-25T12:00:40.294 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.239+0000 7efc2effd700 1 -- 172.21.15.102:0/3517856888 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7efc38002860 con 0x7efc480695e0 2021-05-25T12:00:40.294 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.239+0000 7efc2effd700 1 -- 172.21.15.102:0/3517856888 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(16..16 src has 1..16) v4 ==== 9119+0+0 (unknown 962814075 0 0) 0x7efc3803c150 con 0x7efc480695e0 2021-05-25T12:00:40.294 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.239+0000 7efc4d982700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:00:40.295 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.239+0000 7efc2effd700 10 client.4703.objecter ms_dispatch 0x7efc48001550 osd_map(16..16 src has 1..16) v4 2021-05-25T12:00:40.295 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.239+0000 7efc2effd700 3 client.4703.objecter handle_osd_map got epochs [16,16] > 0 2021-05-25T12:00:40.295 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.239+0000 7efc2effd700 3 client.4703.objecter handle_osd_map decoding full epoch 16 2021-05-25T12:00:40.295 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.239+0000 7efc4d982700 1 -- 172.21.15.102:0/3517856888 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7efc200065a0 con 0x7efc20002840 2021-05-25T12:00:40.296 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.239+0000 7efc2effd700 20 client.4703.objecter dump_active .. 0 homeless 2021-05-25T12:00:40.296 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.239+0000 7efc46ffd700 10 monclient: get_auth_request con 0x7efc280332f0 auth_method 0 2021-05-25T12:00:40.296 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.240+0000 7efc2effd700 10 client.4703.objecter ms_handle_connect 0x7efc20002840 2021-05-25T12:00:40.296 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.240+0000 7efc2effd700 10 client.4703.objecter resend_mon_ops 2021-05-25T12:00:40.296 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.240+0000 7efc2effd700 1 -- 172.21.15.102:0/3517856888 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1277641150 0 0) 0x7efc200065a0 con 0x7efc20002840 2021-05-25T12:00:40.297 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.240+0000 7efc2effd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:40.297 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.240+0000 7efc2effd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:40.297 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.240+0000 7efc2effd700 10 monclient(hunting): my global_id is 74380 2021-05-25T12:00:40.297 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.240+0000 7efc2effd700 1 -- 172.21.15.102:0/3517856888 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc48091240 con 0x7efc20002840 2021-05-25T12:00:40.297 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.240+0000 7efc2effd700 10 client.4703.objecter ms_handle_connect 0x7efc280332f0 2021-05-25T12:00:40.298 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.240+0000 7efc2effd700 1 -- 172.21.15.102:0/3517856888 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2137944405 0 0) 0x7efc48091240 con 0x7efc20002840 2021-05-25T12:00:40.298 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.240+0000 7efc2effd700 1 -- 172.21.15.102:0/3517856888 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efc200065a0 con 0x7efc20002840 2021-05-25T12:00:40.298 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.240+0000 7efc2effd700 1 -- 172.21.15.102:0/3517856888 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7efc30001e60 con 0x7efc20002840 2021-05-25T12:00:40.298 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.240+0000 7efc2effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:40.299 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.240+0000 7efc2effd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T12:00:40.299 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.240+0000 7efc2effd700 10 monclient: dump: 2021-05-25T12:00:40.299 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:40.299 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:40.299 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:40.300 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:40.300 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:40.300 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:40.300 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:40.300 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:40.301 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:40.301 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:40.301 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.240+0000 7efc2effd700 1 -- 172.21.15.102:0/3517856888 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3593210789 0 0) 0x7efc200065a0 con 0x7efc20002840 2021-05-25T12:00:40.301 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.240+0000 7efc2effd700 1 -- 172.21.15.102:0/3517856888 --> v1:172.21.15.181:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7efc20009e40 con 0x7efc20002840 2021-05-25T12:00:40.301 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.240+0000 7efc2effd700 1 -- 172.21.15.102:0/3517856888 <== mon.1 v1:172.21.15.181:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7efc30002180 con 0x7efc20002840 2021-05-25T12:00:40.302 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.240+0000 7efc2effd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:00:40.302 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.240+0000 7efc2effd700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:00:40.302 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.240+0000 7efc2effd700 1 -- 172.21.15.102:0/3517856888 >> v1:172.21.15.181:6789/0 conn(0x7efc20002840 legacy=0x7efc20004ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:40.302 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.240+0000 7efc2effd700 1 -- 172.21.15.102:0/3517856888 >> v1:172.21.15.181:6789/0 conn(0x7efc20002840 legacy=0x7efc20004ce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:00:40.303 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.247+0000 7efc4d982700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:00:40.303 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.247+0000 7efc4d982700 1 -- 172.21.15.102:0/3517856888 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7efc20007510 con 0x7efc200068b0 2021-05-25T12:00:40.303 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.248+0000 7efc2effd700 10 client.4703.objecter ms_handle_connect 0x7efc200068b0 2021-05-25T12:00:40.303 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.248+0000 7efc2effd700 10 client.4703.objecter resend_mon_ops 2021-05-25T12:00:40.303 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.248+0000 7efc2effd700 1 -- 172.21.15.102:0/3517856888 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1113468514 0 0) 0x7efc20007510 con 0x7efc200068b0 2021-05-25T12:00:40.304 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.248+0000 7efc2effd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:40.304 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.248+0000 7efc2effd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:40.304 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.248+0000 7efc2effd700 10 monclient(hunting): my global_id is 74383 2021-05-25T12:00:40.304 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.248+0000 7efc2effd700 1 -- 172.21.15.102:0/3517856888 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc200065a0 con 0x7efc200068b0 2021-05-25T12:00:40.304 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.248+0000 7efc2effd700 1 -- 172.21.15.102:0/3517856888 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1168393069 0 0) 0x7efc200065a0 con 0x7efc200068b0 2021-05-25T12:00:40.305 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.248+0000 7efc2effd700 1 -- 172.21.15.102:0/3517856888 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efc20007510 con 0x7efc200068b0 2021-05-25T12:00:40.305 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.248+0000 7efc2effd700 1 -- 172.21.15.102:0/3517856888 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7efc300027e0 con 0x7efc200068b0 2021-05-25T12:00:40.305 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.248+0000 7efc2effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:40.305 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.248+0000 7efc2effd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T12:00:40.306 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.248+0000 7efc2effd700 10 monclient: dump: 2021-05-25T12:00:40.306 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:40.306 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:40.306 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:40.306 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:40.307 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:40.307 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:40.307 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:40.307 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:40.308 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:40.308 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:40.308 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.248+0000 7efc2effd700 1 -- 172.21.15.102:0/3517856888 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3962518212 0 0) 0x7efc20007510 con 0x7efc200068b0 2021-05-25T12:00:40.308 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.248+0000 7efc2effd700 1 -- 172.21.15.102:0/3517856888 --> v1:172.21.15.181:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7efc20008770 con 0x7efc200068b0 2021-05-25T12:00:40.308 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.248+0000 7efc2effd700 1 -- 172.21.15.102:0/3517856888 <== mon.1 v1:172.21.15.181:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3771 (unknown 0 0 2347099091) 0x7efc30002ac0 con 0x7efc200068b0 2021-05-25T12:00:40.309 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.248+0000 7efc2effd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:00:40.309 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.248+0000 7efc2effd700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:00:40.309 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.248+0000 7efc2effd700 1 -- 172.21.15.102:0/3517856888 >> v1:172.21.15.181:6789/0 conn(0x7efc200068b0 legacy=0x7efc20006d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:40.309 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.248+0000 7efc2effd700 1 -- 172.21.15.102:0/3517856888 >> v1:172.21.15.181:6789/0 conn(0x7efc200068b0 legacy=0x7efc20006d40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:00:40.310 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:00:40.310 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:00:40.310 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:00:40.311 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "peon", 2021-05-25T12:00:40.311 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 14, 2021-05-25T12:00:40.311 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:00:40.311 INFO:teuthology.orchestra.run.smithi102.stdout: 0, 2021-05-25T12:00:40.311 INFO:teuthology.orchestra.run.smithi102.stdout: 1, 2021-05-25T12:00:40.312 INFO:teuthology.orchestra.run.smithi102.stdout: 2 2021-05-25T12:00:40.312 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:00:40.312 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 3, 2021-05-25T12:00:40.312 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:00:40.312 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:00:40.313 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:00:40.313 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:00:40.313 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:00:40.313 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:00:40.314 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:00:40.314 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:00:40.314 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:00:40.314 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:00:40.314 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:00:40.315 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:00:40.315 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:00:40.315 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:00:40.315 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:00:40.315 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:00:40.316 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:00:40.316 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:00:40.316 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:00:40.316 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:00:40.317 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:00:40.317 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:00:40.317 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:00:40.317 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:00:40.317 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:00:40.318 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:40.318 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:00:40.318 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:00:40.318 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:00:40.318 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:00:40.319 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:00:40.319 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:00:40.319 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:00:40.319 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:00:40.320 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:00:40.320 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:00:40.320 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:00:40.320 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:00:40.320 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:00:40.321 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:00:40.321 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:00:40.321 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:00:40.321 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:00:40.321 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:00:40.322 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:00:40.322 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:00:40.322 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:00:40.322 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:00:40.322 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:00:40.323 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:00:40.323 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:00:40.323 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:00:40.323 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:40.324 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:00:40.324 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:40.324 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:00:40.324 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:00:40.324 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:00:40.325 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:00:40.325 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:40.325 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:00:40.325 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:00:40.325 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:00:40.326 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:40.326 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:00:40.326 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:40.326 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:00:40.326 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:00:40.327 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:00:40.327 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:00:40.327 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:00:40.327 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:40.328 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:40.328 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:00:40.328 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:00:40.328 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:00:40.328 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:00:40.329 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:40.329 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:00:40.329 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:00:40.329 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:00:40.329 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:40.330 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:00:40.330 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:40.330 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:00:40.330 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:00:40.330 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:00:40.331 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:00:40.331 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:00:40.331 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:40.331 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:40.331 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:00:40.332 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:00:40.332 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:00:40.332 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:00:40.332 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:40.332 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:00:40.333 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:00:40.333 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:00:40.333 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:40.333 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:00:40.333 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:40.334 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:00:40.334 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:00:40.334 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:00:40.334 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:00:40.335 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:00:40.335 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:40.335 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:00:40.335 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:40.335 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:00:40.336 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:00:40.336 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:40.336 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:00:40.336 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:00:40.336 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:00:40.337 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:40.337 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:00:40.337 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:00:40.337 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:40.338 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:00:40.338 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:00:40.338 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 12 2021-05-25T12:00:40.338 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:40.338 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:00:40.339 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:40.339 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:00:40.339 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:00:40.341 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.249+0000 7efc4d982700 10 librados: watch_flush enter 2021-05-25T12:00:40.341 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.249+0000 7efc4d982700 10 librados: watch_flush exit 2021-05-25T12:00:40.341 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.249+0000 7efc4d982700 20 client.4703.objecter shutdown clearing up homeless session... 2021-05-25T12:00:40.342 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.249+0000 7efc4d982700 10 client.4703.objecter successfully canceled tick 2021-05-25T12:00:40.342 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.249+0000 7efc4d982700 1 -- 172.21.15.102:0/3517856888 >> v1:172.21.15.181:6812/29582 conn(0x7efc280332f0 legacy=0x7efc280357b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:40.342 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.249+0000 7efc4d982700 10 monclient: shutdown 2021-05-25T12:00:40.342 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.249+0000 7efc4d982700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:00:40.343 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.249+0000 7efc4d982700 1 -- 172.21.15.102:0/3517856888 >> v1:172.21.15.102:6790/0 conn(0x7efc480695e0 legacy=0x7efc481411a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:40.343 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.249+0000 7efc477fe700 1 -- 172.21.15.102:0/3517856888 reap_dead start 2021-05-25T12:00:40.343 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.249+0000 7efc477fe700 1 -- 172.21.15.102:0/3517856888 reap_dead start 2021-05-25T12:00:40.343 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.249+0000 7efc4d982700 1 -- 172.21.15.102:0/3517856888 shutdown_connections 2021-05-25T12:00:40.343 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.249+0000 7efc4d982700 1 -- 172.21.15.102:0/3517856888 >> 172.21.15.102:0/3517856888 conn(0x7efc4809b370 msgr2=0x7efc480988b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:00:40.344 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.249+0000 7efc4d982700 1 -- 172.21.15.102:0/3517856888 shutdown_connections 2021-05-25T12:00:40.344 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.249+0000 7efc4d982700 1 -- 172.21.15.102:0/3517856888 wait complete. 2021-05-25T12:00:40.344 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.249+0000 7efc4d982700 1 librados: shutdown 2021-05-25T12:00:40.345 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2021-05-25T12:00:40.345 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2021-05-25T12:00:40.345 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2021-05-25T12:00:40.345 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2021-05-25T12:00:40.346 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2021-05-25T12:00:40.346 INFO:teuthology.orchestra.run:waiting for 300 2021-05-25T12:00:40.355 INFO:tasks.ceph.mon.a:Stopped 2021-05-25T12:00:40.356 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2021-05-25T12:00:40.356 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2021-05-25T12:00:40.503 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.502+0000 7fe06c4ed700 10 monclient: get_monmap_and_config 2021-05-25T12:00:40.504 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.502+0000 7fe06c4ed700 10 monclient: build_initial_monmap 2021-05-25T12:00:40.504 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.502+0000 7fe06c4ed700 10 monclient: monmap: 2021-05-25T12:00:40.504 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:00:40.504 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:40.504 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:00:40.503204+0000 2021-05-25T12:00:40.505 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:00:40.503204+0000 2021-05-25T12:00:40.505 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:00:40.505 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:40.505 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:00:40.506 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:00:40.506 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:00:40.506 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:40.517 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.502+0000 7fe06c4ed700 1 Processor -- start 2021-05-25T12:00:40.517 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.502+0000 7fe06c4ed700 1 -- start start 2021-05-25T12:00:40.518 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.502+0000 7fe06c4ed700 10 monclient: init 2021-05-25T12:00:40.518 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.503+0000 7fe06c4ed700 10 monclient: _reopen_session rank -1 2021-05-25T12:00:40.518 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.503+0000 7fe06c4ed700 10 monclient: _add_conns ranks=[1,2,0] 2021-05-25T12:00:40.518 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.503+0000 7fe06c4ed700 10 monclient(hunting): picked mon.noname-c con 0x7fe064069190 addr v1:172.21.15.181:6789/0 2021-05-25T12:00:40.518 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.503+0000 7fe06c4ed700 10 monclient(hunting): picked mon.noname-b con 0x7fe064069570 addr v1:172.21.15.102:6790/0 2021-05-25T12:00:40.519 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.503+0000 7fe06c4ed700 10 monclient(hunting): picked mon.noname-a con 0x7fe064131e80 addr v1:172.21.15.102:6789/0 2021-05-25T12:00:40.519 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.503+0000 7fe06c4ed700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0640917b0 con 0x7fe064131e80 2021-05-25T12:00:40.519 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.503+0000 7fe06c4ed700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe064096bf0 con 0x7fe064069190 2021-05-25T12:00:40.519 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.503+0000 7fe06c4ed700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0640910e0 con 0x7fe064069570 2021-05-25T12:00:40.520 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.503+0000 7fe06c4ed700 10 monclient(hunting): _renew_subs 2021-05-25T12:00:40.520 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.503+0000 7fe06c4ed700 10 monclient(hunting): authenticate will time out at 841.100577s 2021-05-25T12:00:40.520 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.503+0000 7fe06aa8a700 1 -- >> v1:172.21.15.102:6789/0 conn(0x7fe064131e80 legacy=0x7fe064134270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:00:40.520 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.503+0000 7fe069a88700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fe064069570 0x7fe0640699f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:54320/0 (socket says 172.21.15.102:54320) 2021-05-25T12:00:40.520 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.503+0000 7fe069a88700 1 -- 172.21.15.102:0/3211776284 learned_addr learned my addr 172.21.15.102:0/3211776284 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:00:40.521 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.503+0000 7fe069287700 1 -- 172.21.15.102:0/3211776284 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1100771601 0 0) 0x7fe0640910e0 con 0x7fe064069570 2021-05-25T12:00:40.521 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.503+0000 7fe069287700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:40.521 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.503+0000 7fe069287700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:40.521 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.503+0000 7fe069287700 10 monclient(hunting): my global_id is 4706 2021-05-25T12:00:40.521 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.503+0000 7fe069287700 1 -- 172.21.15.102:0/3211776284 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe04c003270 con 0x7fe064069570 2021-05-25T12:00:40.522 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.504+0000 7fe069287700 1 -- 172.21.15.102:0/3211776284 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 812095994 0 0) 0x7fe064096bf0 con 0x7fe064069190 2021-05-25T12:00:40.522 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.504+0000 7fe069287700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:40.522 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.504+0000 7fe069287700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:40.522 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.504+0000 7fe069287700 10 monclient(hunting): my global_id is 74386 2021-05-25T12:00:40.523 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.504+0000 7fe069287700 1 -- 172.21.15.102:0/3211776284 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0640910e0 con 0x7fe064069190 2021-05-25T12:00:40.523 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.504+0000 7fe069287700 1 -- 172.21.15.102:0/3211776284 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2303663061 0 0) 0x7fe04c003270 con 0x7fe064069570 2021-05-25T12:00:40.523 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.504+0000 7fe069287700 1 -- 172.21.15.102:0/3211776284 >> v1:172.21.15.181:6789/0 conn(0x7fe064069190 legacy=0x7fe0640964e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:40.523 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.504+0000 7fe069287700 1 -- 172.21.15.102:0/3211776284 >> v1:172.21.15.102:6789/0 conn(0x7fe064131e80 legacy=0x7fe064134270 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-05-25T12:00:40.523 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.504+0000 7fe069287700 10 monclient: _finish_hunting 0 2021-05-25T12:00:40.524 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.504+0000 7fe069287700 1 monclient: found mon.noname-b 2021-05-25T12:00:40.524 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.504+0000 7fe069287700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:00:40.524 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.504+0000 7fe069287700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T12:00:40.524 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.504+0000 7fe069287700 1 -- 172.21.15.102:0/3211776284 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0640914a0 con 0x7fe064069570 2021-05-25T12:00:40.524 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.504+0000 7fe069287700 10 monclient: _finish_auth 0 2021-05-25T12:00:40.525 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.504+0000 7fe069287700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:00:40.525 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.504+0000 7fe069287700 1 -- 172.21.15.102:0/3211776284 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fe054001650 con 0x7fe064069570 2021-05-25T12:00:40.525 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.504+0000 7fe06c4ed700 5 monclient: authenticate success, global_id 4706 2021-05-25T12:00:40.525 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.504+0000 7fe069287700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:40.526 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.504+0000 7fe069287700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:00:40.526 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.504+0000 7fe069287700 10 monclient: dump: 2021-05-25T12:00:40.526 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:40.526 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:40.526 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:40.527 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:40.527 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:40.527 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:40.527 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:40.527 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:40.528 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:40.528 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:40.528 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.504+0000 7fe06c4ed700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:00:40.528 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.504+0000 7fe069287700 1 -- 172.21.15.102:0/3211776284 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe054001870 con 0x7fe064069570 2021-05-25T12:00:40.528 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.504+0000 7fe069287700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:00:40.529 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.504+0000 7fe06c4ed700 10 monclient: get_monmap_and_config success 2021-05-25T12:00:40.529 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.504+0000 7fe069287700 1 -- 172.21.15.102:0/3211776284 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fe054001c00 con 0x7fe064069570 2021-05-25T12:00:40.529 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.504+0000 7fe069287700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:40.529 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.504+0000 7fe06c4ed700 4 set_mon_vals no callback set 2021-05-25T12:00:40.529 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.504+0000 7fe069287700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:00:40.530 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.504+0000 7fe069287700 10 monclient: dump: 2021-05-25T12:00:40.530 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:40.530 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:40.530 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:40.531 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:40.531 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:40.531 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:40.531 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:40.531 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:40.532 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:40.532 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:40.532 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.504+0000 7fe06c4ed700 10 monclient: shutdown 2021-05-25T12:00:40.532 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.504+0000 7fe06c4ed700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:00:40.532 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.504+0000 7fe06c4ed700 1 -- 172.21.15.102:0/3211776284 >> v1:172.21.15.102:6790/0 conn(0x7fe064069570 legacy=0x7fe0640699f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:40.533 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.504+0000 7fe06c4ed700 1 -- 172.21.15.102:0/3211776284 shutdown_connections 2021-05-25T12:00:40.533 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.504+0000 7fe06c4ed700 1 -- 172.21.15.102:0/3211776284 >> 172.21.15.102:0/3211776284 conn(0x7fe06409af90 msgr2=0x7fe06409b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:00:40.533 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.504+0000 7fe06c4ed700 1 -- 172.21.15.102:0/3211776284 shutdown_connections 2021-05-25T12:00:40.533 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.504+0000 7fe06c4ed700 1 -- 172.21.15.102:0/3211776284 wait complete. 2021-05-25T12:00:40.533 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.505+0000 7fe06c4ed700 10 monclient: build_initial_monmap 2021-05-25T12:00:40.534 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.505+0000 7fe06c4ed700 10 monclient: monmap: 2021-05-25T12:00:40.534 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:00:40.534 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:00:40.534 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:00:40.534 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:00:40.535 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:00:40.535 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:00:40.535 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:00:40.535 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:00:40.536 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:00:40.536 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:40.536 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.505+0000 7fe06c4ed700 1 librados: starting msgr at 2021-05-25T12:00:40.536 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.505+0000 7fe06c4ed700 1 librados: starting objecter 2021-05-25T12:00:40.536 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.505+0000 7fe06c4ed700 1 Processor -- start 2021-05-25T12:00:40.537 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.505+0000 7fe06c4ed700 1 -- start start 2021-05-25T12:00:40.537 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.505+0000 7fe06c4ed700 1 librados: setting wanted keys 2021-05-25T12:00:40.537 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.505+0000 7fe06c4ed700 1 librados: calling monclient init 2021-05-25T12:00:40.537 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.505+0000 7fe06c4ed700 10 monclient: init 2021-05-25T12:00:40.537 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.505+0000 7fe06c4ed700 10 monclient: _reopen_session rank -1 2021-05-25T12:00:40.538 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.505+0000 7fe06c4ed700 10 monclient: _add_conns ranks=[0,2,1] 2021-05-25T12:00:40.538 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.505+0000 7fe06c4ed700 10 monclient(hunting): picked mon.noname-a con 0x7fe064069190 addr v1:172.21.15.102:6789/0 2021-05-25T12:00:40.538 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.505+0000 7fe06c4ed700 10 monclient(hunting): picked mon.noname-c con 0x7fe064069570 addr v1:172.21.15.102:6790/0 2021-05-25T12:00:40.538 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.505+0000 7fe06c4ed700 10 monclient(hunting): picked mon.noname-b con 0x7fe064131e80 addr v1:172.21.15.181:6789/0 2021-05-25T12:00:40.539 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.505+0000 7fe06c4ed700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe06412f240 con 0x7fe064069190 2021-05-25T12:00:40.539 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.505+0000 7fe06c4ed700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe06412f4e0 con 0x7fe064131e80 2021-05-25T12:00:40.539 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.505+0000 7fe06c4ed700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe06412f780 con 0x7fe064069570 2021-05-25T12:00:40.539 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.505+0000 7fe06c4ed700 10 monclient(hunting): _renew_subs 2021-05-25T12:00:40.539 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.506+0000 7fe06c4ed700 10 monclient(hunting): authenticate will time out at 841.103004s 2021-05-25T12:00:40.540 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.506+0000 7fe06a289700 1 -- >> v1:172.21.15.102:6789/0 conn(0x7fe064069190 legacy=0x7fe06412cbc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:00:40.540 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.506+0000 7fe069a88700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fe064069570 0x7fe06412d2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:54328/0 (socket says 172.21.15.102:54328) 2021-05-25T12:00:40.540 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.506+0000 7fe069a88700 1 -- 172.21.15.102:0/1290528767 learned_addr learned my addr 172.21.15.102:0/1290528767 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:00:40.540 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.506+0000 7fe0527fc700 10 client.?.objecter ms_handle_connect 0x7fe064069570 2021-05-25T12:00:40.540 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.506+0000 7fe0527fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:00:40.541 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.506+0000 7fe0527fc700 10 client.?.objecter ms_handle_connect 0x7fe064131e80 2021-05-25T12:00:40.541 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.506+0000 7fe0527fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:00:40.541 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.506+0000 7fe0527fc700 1 -- 172.21.15.102:0/1290528767 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2311192964 0 0) 0x7fe06412f780 con 0x7fe064069570 2021-05-25T12:00:40.541 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.506+0000 7fe0527fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:40.541 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.506+0000 7fe0527fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:40.542 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.506+0000 7fe0527fc700 10 monclient(hunting): my global_id is 4709 2021-05-25T12:00:40.542 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.506+0000 7fe0527fc700 1 -- 172.21.15.102:0/1290528767 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe044004c10 con 0x7fe064069570 2021-05-25T12:00:40.542 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.506+0000 7fe0527fc700 1 -- 172.21.15.102:0/1290528767 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1672565740 0 0) 0x7fe06412f4e0 con 0x7fe064131e80 2021-05-25T12:00:40.542 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.506+0000 7fe0527fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:40.542 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.506+0000 7fe0527fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:40.543 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.506+0000 7fe0527fc700 10 monclient(hunting): my global_id is 74389 2021-05-25T12:00:40.543 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.506+0000 7fe0527fc700 1 -- 172.21.15.102:0/1290528767 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe06412f780 con 0x7fe064131e80 2021-05-25T12:00:40.543 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.506+0000 7fe0527fc700 1 -- 172.21.15.102:0/1290528767 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2273801443 0 0) 0x7fe044004c10 con 0x7fe064069570 2021-05-25T12:00:40.543 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.506+0000 7fe0527fc700 1 -- 172.21.15.102:0/1290528767 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe06412f4e0 con 0x7fe064069570 2021-05-25T12:00:40.544 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.506+0000 7fe0527fc700 1 -- 172.21.15.102:0/1290528767 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fe054001550 con 0x7fe064069570 2021-05-25T12:00:40.544 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.506+0000 7fe0527fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:40.544 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.506+0000 7fe0527fc700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:00:40.544 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.506+0000 7fe0527fc700 10 monclient(hunting): dump: 2021-05-25T12:00:40.544 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:40.545 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:40.545 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:40.545 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:40.545 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:40.545 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:40.546 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:40.546 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:40.546 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:40.546 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:40.546 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.506+0000 7fe0527fc700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:00:40.547 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.506+0000 7fe0527fc700 1 -- 172.21.15.102:0/1290528767 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3794711834 0 0) 0x7fe06412f780 con 0x7fe064131e80 2021-05-25T12:00:40.547 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.506+0000 7fe0527fc700 1 -- 172.21.15.102:0/1290528767 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe044004c10 con 0x7fe064131e80 2021-05-25T12:00:40.547 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.506+0000 7fe0527fc700 1 -- 172.21.15.102:0/1290528767 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fe060004060 con 0x7fe064131e80 2021-05-25T12:00:40.547 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.506+0000 7fe0527fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:40.548 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.506+0000 7fe0527fc700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2021-05-25T12:00:40.548 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.506+0000 7fe0527fc700 10 monclient(hunting): dump: 2021-05-25T12:00:40.548 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:40.548 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:40.548 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:40.549 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:40.549 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:40.549 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:40.549 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:40.549 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:40.550 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:40.550 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:40.550 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.506+0000 7fe0527fc700 1 -- 172.21.15.102:0/1290528767 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1721356052 0 0) 0x7fe06412f4e0 con 0x7fe064069570 2021-05-25T12:00:40.550 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.506+0000 7fe0527fc700 1 -- 172.21.15.102:0/1290528767 >> v1:172.21.15.181:6789/0 conn(0x7fe064131e80 legacy=0x7fe064143ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:40.550 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.506+0000 7fe0527fc700 1 -- 172.21.15.102:0/1290528767 >> v1:172.21.15.102:6789/0 conn(0x7fe064069190 legacy=0x7fe06412cbc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-05-25T12:00:40.551 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.506+0000 7fe0527fc700 10 monclient: _finish_hunting 0 2021-05-25T12:00:40.551 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.506+0000 7fe0527fc700 1 monclient: found mon.c 2021-05-25T12:00:40.551 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.506+0000 7fe0527fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:00:40.551 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.506+0000 7fe0527fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:00:40.552 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.506+0000 7fe0527fc700 1 -- 172.21.15.102:0/1290528767 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe06412fb40 con 0x7fe064069570 2021-05-25T12:00:40.552 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.506+0000 7fe0527fc700 10 monclient: _finish_auth 0 2021-05-25T12:00:40.552 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.506+0000 7fe0527fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:00:40.552 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.506+0000 7fe06c4ed700 5 monclient: authenticate success, global_id 4709 2021-05-25T12:00:40.552 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.506+0000 7fe06c4ed700 10 monclient: _renew_subs 2021-05-25T12:00:40.553 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.506+0000 7fe06c4ed700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:00:40.553 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.506+0000 7fe06c4ed700 1 -- 172.21.15.102:0/1290528767 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe064094fb0 con 0x7fe064069570 2021-05-25T12:00:40.553 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.506+0000 7fe06c4ed700 10 client.4709.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:00:40.553 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.506+0000 7fe06c4ed700 10 monclient: _renew_subs 2021-05-25T12:00:40.553 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.506+0000 7fe06c4ed700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:00:40.554 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.506+0000 7fe06c4ed700 1 -- 172.21.15.102:0/1290528767 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe06412b5f0 con 0x7fe064069570 2021-05-25T12:00:40.554 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.506+0000 7fe06c4ed700 1 librados: init done 2021-05-25T12:00:40.554 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.506+0000 7fe0527fc700 1 -- 172.21.15.102:0/1290528767 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe0540048f0 con 0x7fe064069570 2021-05-25T12:00:40.554 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.506+0000 7fe0527fc700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:00:40.554 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.506+0000 7fe0527fc700 1 -- 172.21.15.102:0/1290528767 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fe054004c80 con 0x7fe064069570 2021-05-25T12:00:40.555 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.506+0000 7fe0527fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:40.555 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.506+0000 7fe0527fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:00:40.555 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.506+0000 7fe068a86700 4 set_mon_vals no callback set 2021-05-25T12:00:40.555 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.506+0000 7fe0527fc700 10 monclient: dump: 2021-05-25T12:00:40.556 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:40.556 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:40.556 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:40.556 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:40.556 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:40.557 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:40.557 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:40.557 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:40.557 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:40.557 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:40.558 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.507+0000 7fe06c4ed700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:00:40.558 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.507+0000 7fe0527fc700 1 -- 172.21.15.102:0/1290528767 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7fe054004e60 con 0x7fe064069570 2021-05-25T12:00:40.558 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.507+0000 7fe06c4ed700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:00:40.558 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.507+0000 7fe06c4ed700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:00:40.559 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.507+0000 7fe06c4ed700 1 -- 172.21.15.102:0/1290528767 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe048004db0 con 0x7fe064069570 2021-05-25T12:00:40.559 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.507+0000 7fe0527fc700 1 -- 172.21.15.102:0/1290528767 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(16..16 src has 1..16) v4 ==== 9119+0+0 (unknown 962814075 0 0) 0x7fe05403e690 con 0x7fe064069570 2021-05-25T12:00:40.559 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.507+0000 7fe0527fc700 10 client.4709.objecter ms_dispatch 0x7fe064001550 osd_map(16..16 src has 1..16) v4 2021-05-25T12:00:40.559 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.507+0000 7fe0527fc700 3 client.4709.objecter handle_osd_map got epochs [16,16] > 0 2021-05-25T12:00:40.559 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.507+0000 7fe0527fc700 3 client.4709.objecter handle_osd_map decoding full epoch 16 2021-05-25T12:00:40.560 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.507+0000 7fe0527fc700 20 client.4709.objecter dump_active .. 0 homeless 2021-05-25T12:00:40.560 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.507+0000 7fe06a289700 10 monclient: get_auth_request con 0x7fe0440332a0 auth_method 0 2021-05-25T12:00:40.560 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.508+0000 7fe0527fc700 10 client.4709.objecter ms_handle_connect 0x7fe0440332a0 2021-05-25T12:00:40.560 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.511+0000 7fe0527fc700 1 -- 172.21.15.102:0/1290528767 <== mon.2 v1:172.21.15.102:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7fe0540130a0 con 0x7fe064069570 2021-05-25T12:00:40.560 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.511+0000 7fe0527fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:00:40.561 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.511+0000 7fe0527fc700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:00:40.720 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.718+0000 7fe06c4ed700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2021-05-25T12:00:40.721 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.718+0000 7fe06c4ed700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2021-05-25T12:00:40.721 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.718+0000 7fe06c4ed700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:00:40.722 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.718+0000 7fe06c4ed700 1 -- 172.21.15.102:0/1290528767 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fe0480055e0 con 0x7fe064069570 2021-05-25T12:00:40.722 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.719+0000 7fe0527fc700 1 -- 172.21.15.102:0/1290528767 <== mon.2 v1:172.21.15.102:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1342 (unknown 254746657 0 798306142) 0x7fe054012c90 con 0x7fe064069570 2021-05-25T12:00:40.723 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.719+0000 7fe0527fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2021-05-25T12:00:40.723 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.719+0000 7fe0527fc700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:00:40.724 INFO:teuthology.orchestra.run.smithi102.stdout:{"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138303579357183","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"20ee91b6-8935-4b98-b868-85c03789ccc4","modified":"2021-05-25T11:59:11.011522Z","created":"2021-05-25T11:59:11.011522Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6790","nonce":0}]},"addr":"172.21.15.102:6790/0","public_addr":"172.21.15.102:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2021-05-25T12:00:40.727 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.720+0000 7fe06c4ed700 10 librados: watch_flush enter 2021-05-25T12:00:40.727 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.720+0000 7fe06c4ed700 10 librados: watch_flush exit 2021-05-25T12:00:40.728 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.720+0000 7fe06c4ed700 20 client.4709.objecter shutdown clearing up homeless session... 2021-05-25T12:00:40.728 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.720+0000 7fe06c4ed700 10 client.4709.objecter successfully canceled tick 2021-05-25T12:00:40.729 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.720+0000 7fe06c4ed700 1 -- 172.21.15.102:0/1290528767 >> v1:172.21.15.181:6812/29582 conn(0x7fe0440332a0 legacy=0x7fe044035760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:40.729 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.720+0000 7fe06c4ed700 10 monclient: shutdown 2021-05-25T12:00:40.730 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.720+0000 7fe06c4ed700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:00:40.730 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.720+0000 7fe06c4ed700 1 -- 172.21.15.102:0/1290528767 >> v1:172.21.15.102:6790/0 conn(0x7fe064069570 legacy=0x7fe06412d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:40.731 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.720+0000 7fe06c4ed700 1 -- 172.21.15.102:0/1290528767 shutdown_connections 2021-05-25T12:00:40.731 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.720+0000 7fe06c4ed700 1 -- 172.21.15.102:0/1290528767 >> 172.21.15.102:0/1290528767 conn(0x7fe06409af90 msgr2=0x7fe06412b800 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:00:40.731 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.720+0000 7fe06c4ed700 1 -- 172.21.15.102:0/1290528767 shutdown_connections 2021-05-25T12:00:40.732 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.720+0000 7fe06c4ed700 1 -- 172.21.15.102:0/1290528767 wait complete. 2021-05-25T12:00:40.732 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:40.720+0000 7fe06c4ed700 1 librados: shutdown 2021-05-25T12:00:43.735 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2021-05-25T12:00:43.852 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.851+0000 7f422613c700 10 monclient: get_monmap_and_config 2021-05-25T12:00:43.852 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.851+0000 7f422613c700 10 monclient: build_initial_monmap 2021-05-25T12:00:43.853 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.851+0000 7f422613c700 10 monclient: monmap: 2021-05-25T12:00:43.853 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:00:43.853 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:43.853 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:00:43.852053+0000 2021-05-25T12:00:43.854 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:00:43.852053+0000 2021-05-25T12:00:43.854 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:00:43.854 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:43.854 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:00:43.854 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:00:43.855 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:00:43.855 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:43.864 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.851+0000 7f422613c700 1 Processor -- start 2021-05-25T12:00:43.865 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.851+0000 7f422613c700 1 -- start start 2021-05-25T12:00:43.865 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.851+0000 7f422613c700 10 monclient: init 2021-05-25T12:00:43.865 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.852+0000 7f422613c700 10 monclient: _reopen_session rank -1 2021-05-25T12:00:43.865 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.852+0000 7f422613c700 10 monclient: _add_conns ranks=[0,2,1] 2021-05-25T12:00:43.865 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.852+0000 7f422613c700 10 monclient(hunting): picked mon.noname-a con 0x7f4220069190 addr v1:172.21.15.102:6789/0 2021-05-25T12:00:43.866 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.852+0000 7f422613c700 10 monclient(hunting): picked mon.noname-b con 0x7f4220069c80 addr v1:172.21.15.102:6790/0 2021-05-25T12:00:43.866 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.852+0000 7f422613c700 10 monclient(hunting): picked mon.noname-c con 0x7f4220096ab0 addr v1:172.21.15.181:6789/0 2021-05-25T12:00:43.866 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.852+0000 7f422613c700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42200959b0 con 0x7f4220069190 2021-05-25T12:00:43.866 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.852+0000 7f422613c700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42200914e0 con 0x7f4220096ab0 2021-05-25T12:00:43.867 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.852+0000 7f422613c700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4220095c10 con 0x7f4220069c80 2021-05-25T12:00:43.867 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.852+0000 7f422613c700 10 monclient(hunting): _renew_subs 2021-05-25T12:00:43.867 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.852+0000 7f422613c700 10 monclient(hunting): authenticate will time out at 844.449488s 2021-05-25T12:00:43.867 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.852+0000 7f421f7fe700 1 -- >> v1:172.21.15.102:6789/0 conn(0x7f4220069190 legacy=0x7f4220069570 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:00:43.867 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.852+0000 7f421effd700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f4220069c80 0x7f4220130e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:54336/0 (socket says 172.21.15.102:54336) 2021-05-25T12:00:43.868 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.852+0000 7f421effd700 1 -- 172.21.15.102:0/1723347030 learned_addr learned my addr 172.21.15.102:0/1723347030 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:00:43.868 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.852+0000 7f421e7fc700 1 -- 172.21.15.102:0/1723347030 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1247182675 0 0) 0x7f4220095c10 con 0x7f4220069c80 2021-05-25T12:00:43.868 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.852+0000 7f421e7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:43.868 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.852+0000 7f421e7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:43.869 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.852+0000 7f421e7fc700 10 monclient(hunting): my global_id is 4712 2021-05-25T12:00:43.869 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.852+0000 7f421e7fc700 1 -- 172.21.15.102:0/1723347030 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4204003270 con 0x7f4220069c80 2021-05-25T12:00:43.869 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.853+0000 7f421e7fc700 1 -- 172.21.15.102:0/1723347030 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2449050446 0 0) 0x7f4204003270 con 0x7f4220069c80 2021-05-25T12:00:43.869 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.853+0000 7f421e7fc700 1 -- 172.21.15.102:0/1723347030 >> v1:172.21.15.181:6789/0 conn(0x7f4220096ab0 legacy=0x7f4220134560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:43.869 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.853+0000 7f421e7fc700 1 -- 172.21.15.102:0/1723347030 >> v1:172.21.15.102:6789/0 conn(0x7f4220069190 legacy=0x7f4220069570 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-05-25T12:00:43.870 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.853+0000 7f421e7fc700 10 monclient: _finish_hunting 0 2021-05-25T12:00:43.870 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.853+0000 7f421e7fc700 1 monclient: found mon.noname-b 2021-05-25T12:00:43.870 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.853+0000 7f421e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:00:43.870 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.853+0000 7f421e7fc700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T12:00:43.871 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.853+0000 7f421e7fc700 1 -- 172.21.15.102:0/1723347030 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4220095eb0 con 0x7f4220069c80 2021-05-25T12:00:43.871 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.853+0000 7f421e7fc700 10 monclient: _finish_auth 0 2021-05-25T12:00:43.871 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.853+0000 7f421e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:00:43.871 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.853+0000 7f422613c700 5 monclient: authenticate success, global_id 4712 2021-05-25T12:00:43.871 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.853+0000 7f421e7fc700 1 -- 172.21.15.102:0/1723347030 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f4210001610 con 0x7f4220069c80 2021-05-25T12:00:43.872 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.853+0000 7f422613c700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:00:43.872 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.853+0000 7f421e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:43.872 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.853+0000 7f421e7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:00:43.872 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.853+0000 7f421e7fc700 10 monclient: dump: 2021-05-25T12:00:43.872 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:43.873 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:43.873 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:43.873 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:43.873 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:43.874 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:43.874 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:43.874 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:43.874 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:43.874 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:43.875 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.853+0000 7f422613c700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:00:43.875 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.853+0000 7f421e7fc700 1 -- 172.21.15.102:0/1723347030 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4210001930 con 0x7f4220069c80 2021-05-25T12:00:43.875 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.853+0000 7f421e7fc700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:00:43.875 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.853+0000 7f421e7fc700 1 -- 172.21.15.102:0/1723347030 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f4210001cc0 con 0x7f4220069c80 2021-05-25T12:00:43.875 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.853+0000 7f422613c700 10 monclient: get_monmap_and_config success 2021-05-25T12:00:43.876 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.853+0000 7f422613c700 4 set_mon_vals no callback set 2021-05-25T12:00:43.876 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.853+0000 7f421e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:43.876 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.853+0000 7f421e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:00:43.876 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.853+0000 7f421e7fc700 10 monclient: dump: 2021-05-25T12:00:43.876 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:43.877 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:43.877 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:43.878 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:43.878 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:43.878 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:43.878 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:43.878 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:43.879 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:43.879 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:43.879 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.853+0000 7f422613c700 10 monclient: shutdown 2021-05-25T12:00:43.879 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.853+0000 7f422613c700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:00:43.880 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.853+0000 7f422613c700 1 -- 172.21.15.102:0/1723347030 >> v1:172.21.15.102:6790/0 conn(0x7f4220069c80 legacy=0x7f4220130e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:43.880 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.853+0000 7f422613c700 1 -- 172.21.15.102:0/1723347030 shutdown_connections 2021-05-25T12:00:43.880 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.853+0000 7f422613c700 1 -- 172.21.15.102:0/1723347030 >> 172.21.15.102:0/1723347030 conn(0x7f422009af90 msgr2=0x7f422009b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:00:43.880 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.853+0000 7f422613c700 1 -- 172.21.15.102:0/1723347030 shutdown_connections 2021-05-25T12:00:43.880 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.853+0000 7f422613c700 1 -- 172.21.15.102:0/1723347030 wait complete. 2021-05-25T12:00:43.881 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.854+0000 7f422613c700 10 monclient: build_initial_monmap 2021-05-25T12:00:43.881 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.854+0000 7f422613c700 10 monclient: monmap: 2021-05-25T12:00:43.881 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:00:43.881 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:00:43.881 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:00:43.882 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:00:43.882 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:00:43.882 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:00:43.882 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:00:43.882 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:00:43.883 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:00:43.883 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:43.883 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.854+0000 7f422613c700 1 librados: starting msgr at 2021-05-25T12:00:43.883 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.854+0000 7f422613c700 1 librados: starting objecter 2021-05-25T12:00:43.884 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.854+0000 7f422613c700 1 Processor -- start 2021-05-25T12:00:43.884 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.854+0000 7f422613c700 1 -- start start 2021-05-25T12:00:43.884 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.854+0000 7f422613c700 1 librados: setting wanted keys 2021-05-25T12:00:43.884 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.854+0000 7f422613c700 1 librados: calling monclient init 2021-05-25T12:00:43.884 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.854+0000 7f422613c700 10 monclient: init 2021-05-25T12:00:43.885 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.855+0000 7f422613c700 10 monclient: _reopen_session rank -1 2021-05-25T12:00:43.885 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.855+0000 7f422613c700 10 monclient: _add_conns ranks=[0,2,1] 2021-05-25T12:00:43.885 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.855+0000 7f422613c700 10 monclient(hunting): picked mon.noname-a con 0x7f4220069190 addr v1:172.21.15.102:6789/0 2021-05-25T12:00:43.885 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.855+0000 7f422613c700 10 monclient(hunting): picked mon.noname-c con 0x7f4220069c80 addr v1:172.21.15.102:6790/0 2021-05-25T12:00:43.885 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.855+0000 7f422613c700 10 monclient(hunting): picked mon.noname-b con 0x7f4220096ab0 addr v1:172.21.15.181:6789/0 2021-05-25T12:00:43.886 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.855+0000 7f422613c700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f422011ab60 con 0x7f4220069190 2021-05-25T12:00:43.886 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.855+0000 7f422613c700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f422011ae00 con 0x7f4220096ab0 2021-05-25T12:00:43.886 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.855+0000 7f422613c700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f422011b0a0 con 0x7f4220069c80 2021-05-25T12:00:43.886 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.855+0000 7f422613c700 10 monclient(hunting): _renew_subs 2021-05-25T12:00:43.887 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.855+0000 7f422613c700 10 monclient(hunting): authenticate will time out at 844.452221s 2021-05-25T12:00:43.887 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.855+0000 7f421f7fe700 1 -- >> v1:172.21.15.102:6789/0 conn(0x7f4220069190 legacy=0x7f422012d660 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:00:43.887 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.855+0000 7f421effd700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f4220069c80 0x7f422013ffd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:54342/0 (socket says 172.21.15.102:54342) 2021-05-25T12:00:43.887 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.855+0000 7f421effd700 1 -- 172.21.15.102:0/1267418286 learned_addr learned my addr 172.21.15.102:0/1267418286 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:00:43.887 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.855+0000 7f420b7fe700 10 client.?.objecter ms_handle_connect 0x7f4220069c80 2021-05-25T12:00:43.888 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.855+0000 7f420b7fe700 10 client.?.objecter resend_mon_ops 2021-05-25T12:00:43.888 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.855+0000 7f420b7fe700 1 -- 172.21.15.102:0/1267418286 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3692851442 0 0) 0x7f422011b0a0 con 0x7f4220069c80 2021-05-25T12:00:43.888 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.855+0000 7f420b7fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:43.888 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.855+0000 7f420b7fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:43.888 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.855+0000 7f420b7fe700 10 monclient(hunting): my global_id is 4715 2021-05-25T12:00:43.889 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.855+0000 7f420b7fe700 1 -- 172.21.15.102:0/1267418286 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4200004c10 con 0x7f4220069c80 2021-05-25T12:00:43.889 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.855+0000 7f420b7fe700 10 client.?.objecter ms_handle_connect 0x7f4220096ab0 2021-05-25T12:00:43.889 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.855+0000 7f420b7fe700 10 client.?.objecter resend_mon_ops 2021-05-25T12:00:43.889 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.855+0000 7f420b7fe700 1 -- 172.21.15.102:0/1267418286 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1756043962 0 0) 0x7f422011ae00 con 0x7f4220096ab0 2021-05-25T12:00:43.889 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.855+0000 7f420b7fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:43.890 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.855+0000 7f420b7fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:43.890 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.855+0000 7f420b7fe700 10 monclient(hunting): my global_id is 74395 2021-05-25T12:00:43.890 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.855+0000 7f420b7fe700 1 -- 172.21.15.102:0/1267418286 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f422011b0a0 con 0x7f4220096ab0 2021-05-25T12:00:43.890 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.855+0000 7f420b7fe700 1 -- 172.21.15.102:0/1267418286 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2830574088 0 0) 0x7f4200004c10 con 0x7f4220069c80 2021-05-25T12:00:43.891 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.855+0000 7f420b7fe700 1 -- 172.21.15.102:0/1267418286 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f422011ae00 con 0x7f4220069c80 2021-05-25T12:00:43.891 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.855+0000 7f420b7fe700 1 -- 172.21.15.102:0/1267418286 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f4210001c40 con 0x7f4220069c80 2021-05-25T12:00:43.891 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.855+0000 7f420b7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:43.891 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.855+0000 7f420b7fe700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:00:43.891 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.855+0000 7f420b7fe700 10 monclient(hunting): dump: 2021-05-25T12:00:43.892 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:43.892 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:43.892 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:43.892 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:43.892 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:43.893 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:43.893 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:43.893 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:43.893 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:43.893 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:43.894 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.855+0000 7f420b7fe700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:00:43.894 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.855+0000 7f420b7fe700 1 -- 172.21.15.102:0/1267418286 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1068343363 0 0) 0x7f422011ae00 con 0x7f4220069c80 2021-05-25T12:00:43.894 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.855+0000 7f420b7fe700 1 -- 172.21.15.102:0/1267418286 >> v1:172.21.15.181:6789/0 conn(0x7f4220096ab0 legacy=0x7f4220143700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:43.894 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.855+0000 7f420b7fe700 1 -- 172.21.15.102:0/1267418286 >> v1:172.21.15.102:6789/0 conn(0x7f4220069190 legacy=0x7f422012d660 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-05-25T12:00:43.895 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.855+0000 7f420b7fe700 10 monclient: _finish_hunting 0 2021-05-25T12:00:43.895 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.855+0000 7f420b7fe700 1 monclient: found mon.c 2021-05-25T12:00:43.895 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.855+0000 7f420b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:00:43.895 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.855+0000 7f420b7fe700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:00:43.895 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.855+0000 7f420b7fe700 1 -- 172.21.15.102:0/1267418286 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4220118f30 con 0x7f4220069c80 2021-05-25T12:00:43.896 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.855+0000 7f420b7fe700 10 monclient: _finish_auth 0 2021-05-25T12:00:43.896 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.855+0000 7f420b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:00:43.896 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.855+0000 7f422613c700 5 monclient: authenticate success, global_id 4715 2021-05-25T12:00:43.896 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.855+0000 7f422613c700 10 monclient: _renew_subs 2021-05-25T12:00:43.897 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.855+0000 7f422613c700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:00:43.897 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.855+0000 7f422613c700 1 -- 172.21.15.102:0/1267418286 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f422012dd70 con 0x7f4220069c80 2021-05-25T12:00:43.897 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.855+0000 7f422613c700 10 client.4715.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:00:43.897 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.855+0000 7f422613c700 10 monclient: _renew_subs 2021-05-25T12:00:43.897 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.855+0000 7f422613c700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:00:43.898 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.855+0000 7f422613c700 1 -- 172.21.15.102:0/1267418286 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4220118420 con 0x7f4220069c80 2021-05-25T12:00:43.898 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.855+0000 7f422613c700 1 librados: init done 2021-05-25T12:00:43.898 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.856+0000 7f420b7fe700 1 -- 172.21.15.102:0/1267418286 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4210004a50 con 0x7f4220069c80 2021-05-25T12:00:43.898 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.856+0000 7f420b7fe700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:00:43.898 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.856+0000 7f420b7fe700 1 -- 172.21.15.102:0/1267418286 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f4210004de0 con 0x7f4220069c80 2021-05-25T12:00:43.899 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.856+0000 7f420b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:43.899 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.856+0000 7f421dffb700 4 set_mon_vals no callback set 2021-05-25T12:00:43.899 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.856+0000 7f420b7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:00:43.899 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.856+0000 7f420b7fe700 10 monclient: dump: 2021-05-25T12:00:43.899 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:43.900 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:43.900 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:43.900 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:43.900 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:43.901 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:43.901 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:43.901 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:43.901 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:43.901 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:43.902 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.856+0000 7f420b7fe700 1 -- 172.21.15.102:0/1267418286 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f4210004fc0 con 0x7f4220069c80 2021-05-25T12:00:43.902 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.856+0000 7f422613c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:00:43.902 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.856+0000 7f422613c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:00:43.902 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.856+0000 7f422613c700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:00:43.902 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.856+0000 7f422613c700 1 -- 172.21.15.102:0/1267418286 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41f8004db0 con 0x7f4220069c80 2021-05-25T12:00:43.903 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.856+0000 7f420b7fe700 1 -- 172.21.15.102:0/1267418286 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(16..16 src has 1..16) v4 ==== 9119+0+0 (unknown 962814075 0 0) 0x7f421003e6f0 con 0x7f4220069c80 2021-05-25T12:00:43.903 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.856+0000 7f420b7fe700 10 client.4715.objecter ms_dispatch 0x7f4220001550 osd_map(16..16 src has 1..16) v4 2021-05-25T12:00:43.903 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.856+0000 7f420b7fe700 3 client.4715.objecter handle_osd_map got epochs [16,16] > 0 2021-05-25T12:00:43.903 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.856+0000 7f420b7fe700 3 client.4715.objecter handle_osd_map decoding full epoch 16 2021-05-25T12:00:43.903 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.857+0000 7f420b7fe700 20 client.4715.objecter dump_active .. 0 homeless 2021-05-25T12:00:43.904 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.857+0000 7f421f7fe700 10 monclient: get_auth_request con 0x7f4200033150 auth_method 0 2021-05-25T12:00:43.904 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.857+0000 7f420b7fe700 10 client.4715.objecter ms_handle_connect 0x7f4200033150 2021-05-25T12:00:43.904 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.861+0000 7f420b7fe700 1 -- 172.21.15.102:0/1267418286 <== mon.2 v1:172.21.15.102:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7f4210012ed0 con 0x7f4220069c80 2021-05-25T12:00:43.904 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.861+0000 7f420b7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:00:43.905 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:43.861+0000 7f420b7fe700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:00:44.071 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:44.069+0000 7f422613c700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2021-05-25T12:00:44.072 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:44.070+0000 7f422613c700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2021-05-25T12:00:44.072 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:44.070+0000 7f422613c700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:00:44.072 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:44.070+0000 7f422613c700 1 -- 172.21.15.102:0/1267418286 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f41f80055e0 con 0x7f4220069c80 2021-05-25T12:00:44.072 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:44.070+0000 7f420b7fe700 1 -- 172.21.15.102:0/1267418286 <== mon.2 v1:172.21.15.102:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1342 (unknown 254746657 0 1129981551) 0x7f4210011be0 con 0x7f4220069c80 2021-05-25T12:00:44.073 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:44.070+0000 7f420b7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2021-05-25T12:00:44.073 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:44.070+0000 7f420b7fe700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:00:44.073 INFO:teuthology.orchestra.run.smithi102.stdout:{"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540138303579357183","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"20ee91b6-8935-4b98-b868-85c03789ccc4","modified":"2021-05-25T11:59:11.011522Z","created":"2021-05-25T11:59:11.011522Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6790","nonce":0}]},"addr":"172.21.15.102:6790/0","public_addr":"172.21.15.102:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2021-05-25T12:00:44.075 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:44.071+0000 7f422613c700 10 librados: watch_flush enter 2021-05-25T12:00:44.075 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:44.071+0000 7f422613c700 10 librados: watch_flush exit 2021-05-25T12:00:44.075 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:44.071+0000 7f422613c700 20 client.4715.objecter shutdown clearing up homeless session... 2021-05-25T12:00:44.076 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:44.071+0000 7f422613c700 10 client.4715.objecter successfully canceled tick 2021-05-25T12:00:44.076 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:44.071+0000 7f422613c700 1 -- 172.21.15.102:0/1267418286 >> v1:172.21.15.181:6812/29582 conn(0x7f4200033150 legacy=0x7f4200035610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:44.076 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:44.071+0000 7f422613c700 10 monclient: shutdown 2021-05-25T12:00:44.076 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:44.071+0000 7f422613c700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:00:44.076 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:44.071+0000 7f422613c700 1 -- 172.21.15.102:0/1267418286 >> v1:172.21.15.102:6790/0 conn(0x7f4220069c80 legacy=0x7f422013ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:44.077 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:44.071+0000 7f422613c700 1 -- 172.21.15.102:0/1267418286 shutdown_connections 2021-05-25T12:00:44.077 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:44.071+0000 7f422613c700 1 -- 172.21.15.102:0/1267418286 >> 172.21.15.102:0/1267418286 conn(0x7f422009af90 msgr2=0x7f422012eac0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:00:44.077 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:44.071+0000 7f422613c700 1 -- 172.21.15.102:0/1267418286 shutdown_connections 2021-05-25T12:00:44.077 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:44.071+0000 7f422613c700 1 -- 172.21.15.102:0/1267418286 wait complete. 2021-05-25T12:00:44.077 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:44.072+0000 7f422613c700 1 librados: shutdown 2021-05-25T12:00:46.728 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:40.127+0000 7ff884a90700 1 -- 172.21.15.181:0/4180131642 <== mon.2 v1:172.21.15.102:6790/0 11 ==== osd_map(16..16 src has 1..16) v4 ==== 5850+0+0 (unknown 823670192 0 0) 0x7ff878010110 con 0x55fe52566e20 2021-05-25T12:00:46.729 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:40.127+0000 7ff884a90700 10 client.4511.objecter ms_dispatch 0x55fe524d4560 osd_map(16..16 src has 1..16) v4 2021-05-25T12:00:46.730 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:40.127+0000 7ff884a90700 3 client.4511.objecter handle_osd_map got epochs [16,16] > 15 2021-05-25T12:00:46.730 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:40.127+0000 7ff884a90700 3 client.4511.objecter handle_osd_map decoding incremental epoch 16 2021-05-25T12:00:46.730 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:40.127+0000 7ff884a90700 20 client.4511.objecter dump_active .. 0 homeless 2021-05-25T12:00:46.732 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:40.127+0000 7ff88fe07540 10 librados: set snap read head -> 1 2021-05-25T12:00:46.732 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:40.127+0000 7ff88fe07540 10 librados: writefull oid=exist nspace= 2021-05-25T12:00:46.732 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:40.127+0000 7ff88fe07540 10 client.4511.objecter _op_submit op 0x55fe526102d0 2021-05-25T12:00:46.732 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:40.127+0000 7ff88fe07540 20 client.4511.objecter _calc_target epoch 16 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2021-05-25T12:00:46.733 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:40.127+0000 7ff88fe07540 20 client.4511.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2021-05-25T12:00:46.733 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:40.127+0000 7ff88fe07540 10 client.4511.objecter _calc_target raw pgid 9.d3a59201 -> actual 9.1 acting [0,2] primary 0 2021-05-25T12:00:46.733 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:40.127+0000 7ff88fe07540 20 client.4511.objecter _get_session s=0x55fe524c84a0 osd=0 3 2021-05-25T12:00:46.733 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:40.127+0000 7ff88fe07540 10 client.4511.objecter _op_submit oid exist '@9' '@9' [writefull 0~5 in=5b] tid 2 osd.0 2021-05-25T12:00:46.734 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:40.127+0000 7ff88fe07540 20 client.4511.objecter get_session s=0x55fe524c84a0 osd=0 3 2021-05-25T12:00:46.734 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:40.127+0000 7ff88fe07540 15 client.4511.objecter _session_op_assign 0 2 2021-05-25T12:00:46.734 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:40.127+0000 7ff88fe07540 15 client.4511.objecter _send_op 2 to 9.1 on osd.0 2021-05-25T12:00:46.734 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:40.127+0000 7ff88fe07540 1 -- 172.21.15.181:0/4180131642 --> v1:172.21.15.102:6807/29973 -- osd_op(unknown.0.0:2 9.1 9:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected e16) v8 -- 0x55fe526147e0 con 0x55fe52610cd0 2021-05-25T12:00:46.734 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:40.127+0000 7ff88fe07540 20 client.4511.objecter put_session s=0x55fe524c84a0 osd=0 4 2021-05-25T12:00:46.735 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:40.127+0000 7ff88fe07540 5 client.4511.objecter 1 in flight 2021-05-25T12:00:46.735 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:40.127+0000 7ff887a96700 10 monclient: get_auth_request con 0x55fe52610cd0 auth_method 0 2021-05-25T12:00:46.735 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:40.128+0000 7ff884a90700 10 client.4511.objecter ms_handle_connect 0x55fe52610cd0 2021-05-25T12:00:46.735 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:42.726+0000 7ff87f7fe700 10 monclient: tick 2021-05-25T12:00:46.736 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:42.726+0000 7ff87f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:00:46.736 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:42.726+0000 7ff87f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:00:46.736 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:43.726+0000 7ff87f7fe700 10 monclient: tick 2021-05-25T12:00:46.736 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:43.726+0000 7ff87f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:00:46.736 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:44.726+0000 7ff87f7fe700 10 monclient: tick 2021-05-25T12:00:46.737 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:44.726+0000 7ff87f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:00:46.737 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:44.728+0000 7ff885291700 10 client.4511.objecter tick 2021-05-25T12:00:46.737 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:45.727+0000 7ff87f7fe700 10 monclient: tick 2021-05-25T12:00:46.737 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:45.727+0000 7ff87f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:00:46.738 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:46.727+0000 7ff87f7fe700 10 monclient: tick 2021-05-25T12:00:47.085 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2021-05-25T12:00:47.429 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:47.426+0000 7fc367b94700 10 monclient: get_monmap_and_config 2021-05-25T12:00:47.429 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:47.426+0000 7fc367b94700 10 monclient: build_initial_monmap 2021-05-25T12:00:47.429 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:47.426+0000 7fc367b94700 10 monclient: monmap: 2021-05-25T12:00:47.429 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:00:47.430 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:47.430 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:00:47.428127+0000 2021-05-25T12:00:47.431 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:00:47.428127+0000 2021-05-25T12:00:47.431 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:00:47.431 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:47.432 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:00:47.432 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:00:47.433 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:00:47.433 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:47.433 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:47.426+0000 7fc367b94700 1 Processor -- start 2021-05-25T12:00:47.434 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:47.426+0000 7fc367b94700 1 -- start start 2021-05-25T12:00:47.434 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:47.426+0000 7fc367b94700 10 monclient: init 2021-05-25T12:00:47.437 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:47.427+0000 7fc367b94700 10 monclient: _reopen_session rank -1 2021-05-25T12:00:47.438 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:47.427+0000 7fc367b94700 10 monclient: _add_conns ranks=[0,2,1] 2021-05-25T12:00:47.438 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:47.427+0000 7fc367b94700 10 monclient(hunting): picked mon.noname-a con 0x7fc360090990 addr v1:172.21.15.102:6789/0 2021-05-25T12:00:47.439 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:47.427+0000 7fc367b94700 10 monclient(hunting): picked mon.noname-b con 0x7fc360090d70 addr v1:172.21.15.102:6790/0 2021-05-25T12:00:47.439 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:47.427+0000 7fc367b94700 10 monclient(hunting): picked mon.noname-c con 0x7fc36012e330 addr v1:172.21.15.181:6789/0 2021-05-25T12:00:47.440 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:47.427+0000 7fc367b94700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc360096fa0 con 0x7fc360090990 2021-05-25T12:00:47.440 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:47.427+0000 7fc367b94700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3600926a0 con 0x7fc36012e330 2021-05-25T12:00:47.440 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:47.427+0000 7fc367b94700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc360092940 con 0x7fc360090d70 2021-05-25T12:00:47.441 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:47.427+0000 7fc367b94700 10 monclient(hunting): _renew_subs 2021-05-25T12:00:47.441 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:47.427+0000 7fc367b94700 10 monclient(hunting): authenticate will time out at 848.025445s 2021-05-25T12:00:47.442 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:47.427+0000 7fc365930700 1 -- >> v1:172.21.15.102:6789/0 conn(0x7fc360090990 legacy=0x7fc360093240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:00:47.442 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:47.427+0000 7fc36512f700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fc360090d70 0x7fc360091200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:54350/0 (socket says 172.21.15.102:54350) 2021-05-25T12:00:47.443 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:47.427+0000 7fc36512f700 1 -- 172.21.15.102:0/1704341051 learned_addr learned my addr 172.21.15.102:0/1704341051 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:00:47.630 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:47.628+0000 7fc365930700 1 -- 172.21.15.102:0/1704341051 >> v1:172.21.15.102:6789/0 conn(0x7fc360090990 legacy=0x7fc360093240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:00:48.031 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:48.029+0000 7fc365930700 1 -- 172.21.15.102:0/1704341051 >> v1:172.21.15.102:6789/0 conn(0x7fc360090990 legacy=0x7fc360093240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:00:48.832 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:48.830+0000 7fc365930700 1 -- 172.21.15.102:0/1704341051 >> v1:172.21.15.102:6789/0 conn(0x7fc360090990 legacy=0x7fc360093240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:00:50.431 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:50.428+0000 7fc34f7fe700 10 monclient(hunting): tick 2021-05-25T12:00:50.431 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:50.428+0000 7fc34f7fe700 1 monclient(hunting): continuing hunt 2021-05-25T12:00:50.431 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:50.428+0000 7fc34f7fe700 10 monclient(hunting): _reopen_session rank -1 2021-05-25T12:00:50.432 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:50.428+0000 7fc34f7fe700 1 -- 172.21.15.102:0/1704341051 >> v1:172.21.15.102:6790/0 conn(0x7fc360090d70 legacy=0x7fc360091200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:50.432 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:50.428+0000 7fc34f7fe700 1 -- 172.21.15.102:0/1704341051 >> v1:172.21.15.181:6789/0 conn(0x7fc36012e330 legacy=0x7fc360130720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:50.432 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:50.428+0000 7fc34f7fe700 1 -- 172.21.15.102:0/1704341051 >> v1:172.21.15.102:6789/0 conn(0x7fc360090990 legacy=0x7fc360093240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-05-25T12:00:50.432 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:50.428+0000 7fc34f7fe700 10 monclient: _add_conns ranks=[2,1,0] 2021-05-25T12:00:50.432 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:50.428+0000 7fc34f7fe700 10 monclient(hunting): picked mon.noname-b con 0x7fc3400044f0 addr v1:172.21.15.102:6790/0 2021-05-25T12:00:50.433 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:50.428+0000 7fc34f7fe700 10 monclient(hunting): picked mon.noname-c con 0x7fc36012e330 addr v1:172.21.15.181:6789/0 2021-05-25T12:00:50.433 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:50.428+0000 7fc34f7fe700 10 monclient(hunting): picked mon.noname-a con 0x7fc360090990 addr v1:172.21.15.102:6789/0 2021-05-25T12:00:50.433 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:50.428+0000 7fc34f7fe700 1 -- 172.21.15.102:0/1704341051 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc360092be0 con 0x7fc360090990 2021-05-25T12:00:50.433 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:50.428+0000 7fc34f7fe700 1 -- 172.21.15.102:0/1704341051 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc360096fa0 con 0x7fc36012e330 2021-05-25T12:00:50.433 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:50.428+0000 7fc34f7fe700 1 -- 172.21.15.102:0/1704341051 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3400111a0 con 0x7fc3400044f0 2021-05-25T12:00:50.434 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:50.428+0000 7fc34f7fe700 10 monclient(hunting): _renew_subs 2021-05-25T12:00:50.434 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:50.428+0000 7fc366131700 1 -- 172.21.15.102:0/1704341051 >> v1:172.21.15.102:6789/0 conn(0x7fc360090990 legacy=0x7fc34000d9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:00:50.630 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:50.629+0000 7fc366131700 1 -- 172.21.15.102:0/1704341051 >> v1:172.21.15.102:6789/0 conn(0x7fc360090990 legacy=0x7fc34000d9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:00:51.031 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:51.030+0000 7fc366131700 1 -- 172.21.15.102:0/1704341051 >> v1:172.21.15.102:6789/0 conn(0x7fc360090990 legacy=0x7fc34000d9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:00:51.832 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:51.831+0000 7fc366131700 1 -- 172.21.15.102:0/1704341051 >> v1:172.21.15.102:6789/0 conn(0x7fc360090990 legacy=0x7fc34000d9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:00:53.431 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:53.428+0000 7fc34f7fe700 10 monclient(hunting): tick 2021-05-25T12:00:53.431 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:53.428+0000 7fc34f7fe700 1 monclient(hunting): continuing hunt 2021-05-25T12:00:53.431 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:53.428+0000 7fc34f7fe700 10 monclient(hunting): _reopen_session rank -1 2021-05-25T12:00:53.431 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:53.428+0000 7fc34f7fe700 1 -- 172.21.15.102:0/1704341051 >> v1:172.21.15.102:6790/0 conn(0x7fc3400044f0 legacy=0x7fc340006940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:53.432 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:53.428+0000 7fc34f7fe700 1 -- 172.21.15.102:0/1704341051 >> v1:172.21.15.181:6789/0 conn(0x7fc36012e330 legacy=0x7fc34000a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:53.432 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:53.428+0000 7fc34f7fe700 1 -- 172.21.15.102:0/1704341051 >> v1:172.21.15.102:6789/0 conn(0x7fc360090990 legacy=0x7fc34000d9d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-05-25T12:00:53.432 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:53.428+0000 7fc34f7fe700 10 monclient: _add_conns ranks=[2,1,0] 2021-05-25T12:00:53.432 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:53.428+0000 7fc34f7fe700 10 monclient(hunting): picked mon.noname-b con 0x7fc340010220 addr v1:172.21.15.102:6790/0 2021-05-25T12:00:53.433 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:53.429+0000 7fc34f7fe700 10 monclient(hunting): picked mon.noname-c con 0x7fc36012e330 addr v1:172.21.15.181:6789/0 2021-05-25T12:00:53.433 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:53.429+0000 7fc34f7fe700 10 monclient(hunting): picked mon.noname-a con 0x7fc360090990 addr v1:172.21.15.102:6789/0 2021-05-25T12:00:53.433 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:53.429+0000 7fc34f7fe700 1 -- 172.21.15.102:0/1704341051 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc340012410 con 0x7fc360090990 2021-05-25T12:00:53.433 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:53.429+0000 7fc34f7fe700 1 -- 172.21.15.102:0/1704341051 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc360092be0 con 0x7fc36012e330 2021-05-25T12:00:53.433 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:53.429+0000 7fc34f7fe700 1 -- 172.21.15.102:0/1704341051 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc340010ef0 con 0x7fc340010220 2021-05-25T12:00:53.434 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:53.429+0000 7fc34f7fe700 10 monclient(hunting): _renew_subs 2021-05-25T12:00:53.434 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:53.429+0000 7fc366131700 1 -- 172.21.15.102:0/1704341051 >> v1:172.21.15.102:6789/0 conn(0x7fc360090990 legacy=0x7fc34000d9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:00:53.631 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:53.630+0000 7fc366131700 1 -- 172.21.15.102:0/1704341051 >> v1:172.21.15.102:6789/0 conn(0x7fc360090990 legacy=0x7fc34000d9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:00:54.032 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:54.031+0000 7fc366131700 1 -- 172.21.15.102:0/1704341051 >> v1:172.21.15.102:6789/0 conn(0x7fc360090990 legacy=0x7fc34000d9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:00:54.833 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:54.832+0000 7fc366131700 1 -- 172.21.15.102:0/1704341051 >> v1:172.21.15.102:6789/0 conn(0x7fc360090990 legacy=0x7fc34000d9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:00:55.142 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.141+0000 7fc36492e700 1 -- 172.21.15.102:0/1704341051 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3973900368 0 0) 0x7fc360092be0 con 0x7fc36012e330 2021-05-25T12:00:55.142 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.141+0000 7fc36492e700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:55.143 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.141+0000 7fc36492e700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:55.144 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.141+0000 7fc36492e700 10 monclient(hunting): my global_id is 74509 2021-05-25T12:00:55.145 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.141+0000 7fc36492e700 1 -- 172.21.15.102:0/1704341051 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc348003270 con 0x7fc36012e330 2021-05-25T12:00:55.150 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.141+0000 7fc36492e700 1 -- 172.21.15.102:0/1704341051 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3767199859 0 0) 0x7fc348003270 con 0x7fc36012e330 2021-05-25T12:00:55.150 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.141+0000 7fc36492e700 1 -- 172.21.15.102:0/1704341051 >> v1:172.21.15.102:6790/0 conn(0x7fc340010220 legacy=0x7fc340010660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:55.150 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.141+0000 7fc36492e700 1 -- 172.21.15.102:0/1704341051 >> v1:172.21.15.102:6789/0 conn(0x7fc360090990 legacy=0x7fc34000d9d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-05-25T12:00:55.150 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.141+0000 7fc36492e700 10 monclient: _finish_hunting 0 2021-05-25T12:00:55.151 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.141+0000 7fc36492e700 1 monclient: found mon.noname-c 2021-05-25T12:00:55.151 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.141+0000 7fc36492e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:00:55.151 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.141+0000 7fc36492e700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.181:6789/0 2021-05-25T12:00:55.151 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.141+0000 7fc36492e700 1 -- 172.21.15.102:0/1704341051 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc340005940 con 0x7fc36012e330 2021-05-25T12:00:55.152 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.141+0000 7fc36492e700 10 monclient: _finish_auth 0 2021-05-25T12:00:55.152 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.141+0000 7fc36492e700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:00:55.152 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.141+0000 7fc36492e700 1 -- 172.21.15.102:0/1704341051 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fc354001550 con 0x7fc36012e330 2021-05-25T12:00:55.152 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.141+0000 7fc36492e700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:55.153 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.141+0000 7fc36492e700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:00:55.153 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.141+0000 7fc36492e700 10 monclient: dump: 2021-05-25T12:00:55.153 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:55.153 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:55.153 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:55.154 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:55.154 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:55.154 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:55.154 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:55.155 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:55.155 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:55.155 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:55.155 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.141+0000 7fc367b94700 5 monclient: authenticate success, global_id 74509 2021-05-25T12:00:55.156 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.141+0000 7fc367b94700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:00:55.156 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.141+0000 7fc36492e700 1 -- 172.21.15.102:0/1704341051 <== mon.1 v1:172.21.15.181:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc3540017a0 con 0x7fc36012e330 2021-05-25T12:00:55.156 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.141+0000 7fc36492e700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:00:55.156 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.141+0000 7fc36492e700 1 -- 172.21.15.102:0/1704341051 <== mon.1 v1:172.21.15.181:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fc354001b70 con 0x7fc36012e330 2021-05-25T12:00:55.156 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.141+0000 7fc367b94700 10 monclient: get_monmap_and_config success 2021-05-25T12:00:55.157 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.141+0000 7fc367b94700 4 set_mon_vals no callback set 2021-05-25T12:00:55.157 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.141+0000 7fc36492e700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:55.157 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.141+0000 7fc36492e700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T12:00:55.157 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.141+0000 7fc36492e700 10 monclient: dump: 2021-05-25T12:00:55.157 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:55.158 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:55.158 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:55.158 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:55.158 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:55.158 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:55.159 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:55.159 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:55.159 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:55.159 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:55.160 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.142+0000 7fc367b94700 10 monclient: shutdown 2021-05-25T12:00:55.160 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.142+0000 7fc367b94700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:00:55.160 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.142+0000 7fc367b94700 1 -- 172.21.15.102:0/1704341051 >> v1:172.21.15.181:6789/0 conn(0x7fc36012e330 legacy=0x7fc34000a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:55.160 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.142+0000 7fc367b94700 1 -- 172.21.15.102:0/1704341051 shutdown_connections 2021-05-25T12:00:55.160 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.142+0000 7fc367b94700 1 -- 172.21.15.102:0/1704341051 >> 172.21.15.102:0/1704341051 conn(0x7fc36009b370 msgr2=0x7fc360097430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:00:55.161 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.142+0000 7fc367b94700 1 -- 172.21.15.102:0/1704341051 shutdown_connections 2021-05-25T12:00:55.161 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.142+0000 7fc367b94700 1 -- 172.21.15.102:0/1704341051 wait complete. 2021-05-25T12:00:55.161 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.143+0000 7fc367b94700 10 monclient: build_initial_monmap 2021-05-25T12:00:55.161 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.143+0000 7fc367b94700 10 monclient: monmap: 2021-05-25T12:00:55.161 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:00:55.162 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:00:55.162 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:00:55.162 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:00:55.162 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:00:55.162 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:00:55.163 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:00:55.163 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:00:55.163 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:00:55.163 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:55.163 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.143+0000 7fc367b94700 1 librados: starting msgr at 2021-05-25T12:00:55.164 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.143+0000 7fc367b94700 1 librados: starting objecter 2021-05-25T12:00:55.164 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.143+0000 7fc367b94700 1 Processor -- start 2021-05-25T12:00:55.164 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.143+0000 7fc367b94700 1 -- start start 2021-05-25T12:00:55.164 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.143+0000 7fc367b94700 1 librados: setting wanted keys 2021-05-25T12:00:55.164 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.143+0000 7fc367b94700 1 librados: calling monclient init 2021-05-25T12:00:55.165 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.143+0000 7fc367b94700 10 monclient: init 2021-05-25T12:00:55.165 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.144+0000 7fc367b94700 10 monclient: _reopen_session rank -1 2021-05-25T12:00:55.165 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.144+0000 7fc367b94700 10 monclient: _add_conns ranks=[0,1,2] 2021-05-25T12:00:55.165 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.144+0000 7fc367b94700 10 monclient(hunting): picked mon.noname-a con 0x7fc340010220 addr v1:172.21.15.102:6789/0 2021-05-25T12:00:55.165 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.144+0000 7fc367b94700 10 monclient(hunting): picked mon.noname-b con 0x7fc360090990 addr v1:172.21.15.181:6789/0 2021-05-25T12:00:55.166 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.144+0000 7fc367b94700 10 monclient(hunting): picked mon.noname-c con 0x7fc36012e330 addr v1:172.21.15.102:6790/0 2021-05-25T12:00:55.166 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.144+0000 7fc367b94700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3600681c0 con 0x7fc340010220 2021-05-25T12:00:55.166 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.144+0000 7fc367b94700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc360068460 con 0x7fc360090990 2021-05-25T12:00:55.166 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.144+0000 7fc367b94700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc360068700 con 0x7fc36012e330 2021-05-25T12:00:55.167 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.144+0000 7fc367b94700 10 monclient(hunting): _renew_subs 2021-05-25T12:00:55.167 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.144+0000 7fc367b94700 10 monclient(hunting): authenticate will time out at 855.741501s 2021-05-25T12:00:55.167 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.144+0000 7fc365930700 1 -- >> v1:172.21.15.102:6789/0 conn(0x7fc340010220 legacy=0x7fc360064c80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:00:55.167 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.144+0000 7fc366131700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fc36012e330 0x7fc360065aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:54388/0 (socket says 172.21.15.102:54388) 2021-05-25T12:00:55.167 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.144+0000 7fc366131700 1 -- 172.21.15.102:0/987554529 learned_addr learned my addr 172.21.15.102:0/987554529 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:00:55.168 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.144+0000 7fc34dffb700 10 client.?.objecter ms_handle_connect 0x7fc36012e330 2021-05-25T12:00:55.168 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.144+0000 7fc34dffb700 10 client.?.objecter resend_mon_ops 2021-05-25T12:00:55.168 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.144+0000 7fc34dffb700 10 client.?.objecter ms_handle_connect 0x7fc360090990 2021-05-25T12:00:55.168 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.144+0000 7fc34dffb700 10 client.?.objecter resend_mon_ops 2021-05-25T12:00:55.336 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.334+0000 7fc34dffb700 1 -- 172.21.15.102:0/987554529 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4269923248 0 0) 0x7fc360068700 con 0x7fc36012e330 2021-05-25T12:00:55.336 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.335+0000 7fc34dffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:55.337 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.335+0000 7fc34dffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:55.337 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.335+0000 7fc34dffb700 10 monclient(hunting): my global_id is 4829 2021-05-25T12:00:55.337 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.335+0000 7fc34dffb700 1 -- 172.21.15.102:0/987554529 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3400108f0 con 0x7fc36012e330 2021-05-25T12:00:55.341 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.337+0000 7fc34dffb700 1 -- 172.21.15.102:0/987554529 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1713385412 0 0) 0x7fc360068460 con 0x7fc360090990 2021-05-25T12:00:55.342 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.337+0000 7fc34dffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:55.342 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.337+0000 7fc34dffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:55.342 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.337+0000 7fc34dffb700 10 monclient(hunting): my global_id is 74515 2021-05-25T12:00:55.342 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.337+0000 7fc34dffb700 1 -- 172.21.15.102:0/987554529 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc360068700 con 0x7fc360090990 2021-05-25T12:00:55.342 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.337+0000 7fc34dffb700 1 -- 172.21.15.102:0/987554529 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1808865862 0 0) 0x7fc3400108f0 con 0x7fc36012e330 2021-05-25T12:00:55.343 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.337+0000 7fc34dffb700 1 -- 172.21.15.102:0/987554529 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc360068460 con 0x7fc36012e330 2021-05-25T12:00:55.343 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.337+0000 7fc34dffb700 1 -- 172.21.15.102:0/987554529 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fc35c002ca0 con 0x7fc36012e330 2021-05-25T12:00:55.343 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.337+0000 7fc34dffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:55.343 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.337+0000 7fc34dffb700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:00:55.344 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.337+0000 7fc34dffb700 10 monclient(hunting): dump: 2021-05-25T12:00:55.344 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:55.344 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:55.344 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:55.344 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:55.345 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:55.345 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:55.345 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:55.345 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:55.346 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:55.346 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:55.346 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.337+0000 7fc34dffb700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:00:55.346 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.337+0000 7fc34dffb700 1 -- 172.21.15.102:0/987554529 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1464136578 0 0) 0x7fc360068700 con 0x7fc360090990 2021-05-25T12:00:55.347 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.337+0000 7fc34dffb700 1 -- 172.21.15.102:0/987554529 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc3400108f0 con 0x7fc360090990 2021-05-25T12:00:55.347 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.337+0000 7fc34dffb700 1 -- 172.21.15.102:0/987554529 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fc354001ab0 con 0x7fc360090990 2021-05-25T12:00:55.347 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.337+0000 7fc34dffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:55.347 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.337+0000 7fc34dffb700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2021-05-25T12:00:55.347 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.337+0000 7fc34dffb700 10 monclient(hunting): dump: 2021-05-25T12:00:55.348 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:55.348 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:55.348 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:55.348 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:55.348 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:55.349 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:55.349 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:55.349 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:55.349 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:55.350 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:55.350 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.338+0000 7fc34dffb700 1 -- 172.21.15.102:0/987554529 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3825839703 0 0) 0x7fc360068460 con 0x7fc36012e330 2021-05-25T12:00:55.350 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.338+0000 7fc34dffb700 1 -- 172.21.15.102:0/987554529 >> v1:172.21.15.181:6789/0 conn(0x7fc360090990 legacy=0x7fc360065390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:55.350 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.338+0000 7fc34dffb700 1 -- 172.21.15.102:0/987554529 >> v1:172.21.15.102:6789/0 conn(0x7fc340010220 legacy=0x7fc360064c80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-05-25T12:00:55.350 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.338+0000 7fc34dffb700 10 monclient: _finish_hunting 0 2021-05-25T12:00:55.351 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.338+0000 7fc34dffb700 1 monclient: found mon.c 2021-05-25T12:00:55.351 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.338+0000 7fc34dffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:00:55.351 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.338+0000 7fc34dffb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:00:55.351 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.338+0000 7fc34dffb700 1 -- 172.21.15.102:0/987554529 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc360067310 con 0x7fc36012e330 2021-05-25T12:00:55.352 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.338+0000 7fc34dffb700 10 monclient: _finish_auth 0 2021-05-25T12:00:55.352 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.338+0000 7fc34dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:00:55.352 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.338+0000 7fc367b94700 5 monclient: authenticate success, global_id 4829 2021-05-25T12:00:55.352 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.338+0000 7fc367b94700 10 monclient: _renew_subs 2021-05-25T12:00:55.352 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.338+0000 7fc367b94700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:00:55.353 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.338+0000 7fc367b94700 1 -- 172.21.15.102:0/987554529 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc360066830 con 0x7fc36012e330 2021-05-25T12:00:55.353 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.338+0000 7fc367b94700 10 client.4829.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:00:55.353 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.338+0000 7fc367b94700 10 monclient: _renew_subs 2021-05-25T12:00:55.353 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.338+0000 7fc367b94700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:00:55.354 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.338+0000 7fc367b94700 1 -- 172.21.15.102:0/987554529 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc360115100 con 0x7fc36012e330 2021-05-25T12:00:55.354 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.338+0000 7fc367b94700 1 librados: init done 2021-05-25T12:00:55.354 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.338+0000 7fc34dffb700 1 -- 172.21.15.102:0/987554529 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc35c003070 con 0x7fc36012e330 2021-05-25T12:00:55.354 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.338+0000 7fc34dffb700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:00:55.354 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.338+0000 7fc34dffb700 1 -- 172.21.15.102:0/987554529 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fc35c001ab0 con 0x7fc36012e330 2021-05-25T12:00:55.355 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.338+0000 7fc34dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:55.355 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.338+0000 7fc34dffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:00:55.355 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.338+0000 7fc34ffff700 4 set_mon_vals no callback set 2021-05-25T12:00:55.355 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.338+0000 7fc34dffb700 10 monclient: dump: 2021-05-25T12:00:55.355 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:55.356 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:55.356 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:55.356 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:55.356 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:55.356 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:55.357 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:55.357 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:55.357 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:55.357 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:55.357 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.339+0000 7fc367b94700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:00:55.358 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.339+0000 7fc34dffb700 1 -- 172.21.15.102:0/987554529 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7fc35c001c90 con 0x7fc36012e330 2021-05-25T12:00:55.358 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.339+0000 7fc367b94700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:00:55.358 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.339+0000 7fc367b94700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:00:55.358 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.339+0000 7fc367b94700 1 -- 172.21.15.102:0/987554529 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc33c004db0 con 0x7fc36012e330 2021-05-25T12:00:55.359 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.339+0000 7fc34dffb700 1 -- 172.21.15.102:0/987554529 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(16..16 src has 1..16) v4 ==== 9119+0+0 (unknown 962814075 0 0) 0x7fc35c03e770 con 0x7fc36012e330 2021-05-25T12:00:55.359 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.339+0000 7fc34dffb700 10 client.4829.objecter ms_dispatch 0x7fc360001550 osd_map(16..16 src has 1..16) v4 2021-05-25T12:00:55.359 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.339+0000 7fc34dffb700 3 client.4829.objecter handle_osd_map got epochs [16,16] > 0 2021-05-25T12:00:55.359 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.339+0000 7fc34dffb700 3 client.4829.objecter handle_osd_map decoding full epoch 16 2021-05-25T12:00:55.359 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.339+0000 7fc34dffb700 20 client.4829.objecter dump_active .. 0 homeless 2021-05-25T12:00:55.360 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.339+0000 7fc365930700 10 monclient: get_auth_request con 0x7fc3400343e0 auth_method 0 2021-05-25T12:00:55.360 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.340+0000 7fc34dffb700 10 client.4829.objecter ms_handle_connect 0x7fc3400343e0 2021-05-25T12:00:55.361 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:46.727+0000 7ff87f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:00:55.361 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:47.727+0000 7ff87f7fe700 10 monclient: tick 2021-05-25T12:00:55.361 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:47.727+0000 7ff87f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:00:55.361 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:48.727+0000 7ff87f7fe700 10 monclient: tick 2021-05-25T12:00:55.361 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:48.727+0000 7ff87f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:00:55.362 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:49.727+0000 7ff87f7fe700 10 monclient: tick 2021-05-25T12:00:55.362 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:49.727+0000 7ff87f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:00:55.362 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:49.729+0000 7ff885291700 10 client.4511.objecter tick 2021-05-25T12:00:55.362 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:50.727+0000 7ff87f7fe700 10 monclient: tick 2021-05-25T12:00:55.363 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:50.727+0000 7ff87f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:00:55.363 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:51.728+0000 7ff87f7fe700 10 monclient: tick 2021-05-25T12:00:55.363 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:51.728+0000 7ff87f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:00:55.363 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:52.728+0000 7ff87f7fe700 10 monclient: tick 2021-05-25T12:00:55.364 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:52.728+0000 7ff87f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:00:55.364 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:52.728+0000 7ff87f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:00:55.364 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:53.728+0000 7ff87f7fe700 10 monclient: tick 2021-05-25T12:00:55.364 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:53.728+0000 7ff87f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:00:55.364 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:54.728+0000 7ff87f7fe700 10 monclient: tick 2021-05-25T12:00:55.365 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:54.728+0000 7ff87f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:00:55.365 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:54.730+0000 7ff885291700 10 client.4511.objecter tick 2021-05-25T12:00:55.365 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:54.730+0000 7ff885291700 2 client.4511.objecter tid 2 on osd.0 is laggy 2021-05-25T12:00:55.365 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:54.730+0000 7ff885291700 10 client.4511.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:00:55.365 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:54.730+0000 7ff885291700 10 monclient: _renew_subs 2021-05-25T12:00:55.366 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:54.730+0000 7ff885291700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:00:55.366 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:54.730+0000 7ff885291700 1 -- 172.21.15.181:0/4180131642 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=17}) v3 -- 0x7ff858002f90 con 0x55fe52566e20 2021-05-25T12:00:55.366 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:54.730+0000 7ff885291700 1 -- 172.21.15.181:0/4180131642 --> v1:172.21.15.102:6807/29973 -- ping magic: 0 v1 -- 0x7ff858003330 con 0x55fe52610cd0 2021-05-25T12:00:55.367 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.345+0000 7ff884a90700 1 -- 172.21.15.181:0/4180131642 <== mon.2 v1:172.21.15.102:6790/0 12 ==== osd_map(17..17 src has 1..17) v4 ==== 5313+0+0 (unknown 2286356532 0 0) 0x7ff87803a670 con 0x55fe52566e20 2021-05-25T12:00:55.367 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.345+0000 7ff884a90700 10 client.4511.objecter ms_dispatch 0x55fe524d4560 osd_map(17..17 src has 1..17) v4 2021-05-25T12:00:55.367 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.345+0000 7ff884a90700 3 client.4511.objecter handle_osd_map got epochs [17,17] > 16 2021-05-25T12:00:55.367 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.345+0000 7ff884a90700 3 client.4511.objecter handle_osd_map decoding incremental epoch 17 2021-05-25T12:00:55.368 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.345+0000 7ff884a90700 10 client.4511.objecter checking op 2 2021-05-25T12:00:55.368 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.345+0000 7ff884a90700 20 client.4511.objecter _calc_target epoch 17 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2021-05-25T12:00:55.368 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.343+0000 7fc34dffb700 1 -- 172.21.15.102:0/987554529 <== mon.2 v1:172.21.15.102:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7fc35c013060 con 0x7fc36012e330 2021-05-25T12:00:55.368 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.343+0000 7fc34dffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:00:55.369 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.343+0000 7fc34dffb700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:00:55.472 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.345+0000 7ff884a90700 20 client.4511.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2021-05-25T12:00:55.472 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.345+0000 7ff884a90700 20 client.4511.objecter dump_active .. 0 homeless 2021-05-25T12:00:55.472 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.345+0000 7ff884a90700 20 client.4511.objecter 2 9.d3a59201 osd.0 exist [writefull 0~5 in=5b] 2021-05-25T12:00:55.472 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.438+0000 7ff884a90700 1 -- 172.21.15.181:0/4180131642 <== osd.0 v1:172.21.15.102:6807/29973 1 ==== osd_map(17..17 src has 1..17) v4 ==== 5313+0+0 (unknown 2286356532 0 0) 0x7ff880004290 con 0x55fe52610cd0 2021-05-25T12:00:55.473 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.438+0000 7ff884a90700 10 client.4511.objecter ms_dispatch 0x55fe524d4560 osd_map(17..17 src has 1..17) v4 2021-05-25T12:00:55.473 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.438+0000 7ff884a90700 3 client.4511.objecter handle_osd_map ignoring epochs [17,17] <= 17 2021-05-25T12:00:55.473 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.438+0000 7ff884a90700 20 client.4511.objecter dump_active .. 0 homeless 2021-05-25T12:00:55.473 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.438+0000 7ff884a90700 20 client.4511.objecter 2 9.d3a59201 osd.0 exist [writefull 0~5 in=5b] 2021-05-25T12:00:55.474 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.470+0000 7ff887a96700 1 -- 172.21.15.181:0/4180131642 <== osd.0 v1:172.21.15.102:6807/29973 2 ==== osd_op_reply(2 exist [writefull 0~5] v17'1 uv1 ondisk = 0) v8 ==== 149+0+0 (unknown 828200432 0 0) 0x7ff880004550 con 0x55fe52610cd0 2021-05-25T12:00:55.474 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.470+0000 7ff887a96700 10 client.4511.objecter ms_dispatch 0x55fe524d4560 osd_op_reply(2 exist [writefull 0~5] v17'1 uv1 ondisk = 0) v8 2021-05-25T12:00:55.474 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.470+0000 7ff887a96700 10 client.4511.objecter in handle_osd_op_reply 2021-05-25T12:00:55.474 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: [ RUN ] AsioRados.AsyncReadCallback 2021-05-25T12:00:55.474 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.470+0000 7ff887a96700 7 client.4511.objecter handle_osd_op_reply 2 ondisk uv 1 in 9.1 attempt 0 2021-05-25T12:00:55.475 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.470+0000 7ff887a96700 10 client.4511.objecter op 0 rval 0 len 0 2021-05-25T12:00:55.475 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.470+0000 7ff887a96700 15 client.4511.objecter handle_osd_op_reply completed tid 2 2021-05-25T12:00:55.475 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.470+0000 7ff887a96700 15 client.4511.objecter _finish_op 2 2021-05-25T12:00:55.475 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.470+0000 7ff887a96700 20 client.4511.objecter put_session s=0x55fe524c84a0 osd=0 4 2021-05-25T12:00:55.476 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.470+0000 7ff887a96700 15 client.4511.objecter _session_op_remove 0 2 2021-05-25T12:00:55.476 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.470+0000 7ff887a96700 5 client.4511.objecter 0 in flight 2021-05-25T12:00:55.476 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.470+0000 7ff88fe07540 10 librados: Objecter returned from writefull r=0 2021-05-25T12:00:55.476 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.470+0000 7ff88fe07540 10 client.4511.objecter _op_submit op 0x55fe5260f2c0 2021-05-25T12:00:55.477 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.470+0000 7ff88fe07540 20 client.4511.objecter _calc_target epoch 17 base exist @9 precalc_pgid 0 pgid 0.0 is_read 2021-05-25T12:00:55.477 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.470+0000 7ff88fe07540 20 client.4511.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2021-05-25T12:00:55.477 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.470+0000 7ff88fe07540 10 client.4511.objecter _calc_target raw pgid 9.d3a59201 -> actual 9.1 acting [0,2] primary 0 2021-05-25T12:00:55.477 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.470+0000 7ff88fe07540 20 client.4511.objecter _get_session s=0x55fe524c84a0 osd=0 3 2021-05-25T12:00:55.478 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.470+0000 7ff88fe07540 10 client.4511.objecter _op_submit oid exist '@9' '@9' [read 0~256] tid 3 osd.0 2021-05-25T12:00:55.478 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.470+0000 7ff88fe07540 20 client.4511.objecter get_session s=0x55fe524c84a0 osd=0 3 2021-05-25T12:00:55.478 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.470+0000 7ff88fe07540 15 client.4511.objecter _session_op_assign 0 3 2021-05-25T12:00:55.478 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.470+0000 7ff88fe07540 15 client.4511.objecter _send_op 3 to 9.1 on osd.0 2021-05-25T12:00:55.479 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.470+0000 7ff88fe07540 1 -- 172.21.15.181:0/4180131642 --> v1:172.21.15.102:6807/29973 -- osd_op(unknown.0.0:3 9.1 9:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected e17) v8 -- 0x55fe5260f7b0 con 0x55fe52610cd0 2021-05-25T12:00:55.480 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.470+0000 7ff88fe07540 20 client.4511.objecter put_session s=0x55fe524c84a0 osd=0 4 2021-05-25T12:00:55.480 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.470+0000 7ff88fe07540 5 client.4511.objecter 1 in flight 2021-05-25T12:00:55.480 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.470+0000 7ff88fe07540 10 client.4511.objecter _op_submit op 0x55fe5260fc10 2021-05-25T12:00:55.480 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.470+0000 7ff88fe07540 20 client.4511.objecter _calc_target epoch 17 base noexist @9 precalc_pgid 0 pgid 0.0 is_read 2021-05-25T12:00:55.481 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.470+0000 7ff88fe07540 20 client.4511.objecter _calc_target target noexist @9 -> pgid 9.a7bbaa1 2021-05-25T12:00:55.481 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.470+0000 7ff88fe07540 10 client.4511.objecter _calc_target raw pgid 9.a7bbaa1 -> actual 9.1 acting [0,2] primary 0 2021-05-25T12:00:55.481 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.470+0000 7ff88fe07540 20 client.4511.objecter _get_session s=0x55fe524c84a0 osd=0 4 2021-05-25T12:00:55.481 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.470+0000 7ff88fe07540 10 client.4511.objecter _op_submit oid noexist '@9' '@9' [read 0~256] tid 4 osd.0 2021-05-25T12:00:55.481 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.470+0000 7ff88fe07540 20 client.4511.objecter get_session s=0x55fe524c84a0 osd=0 4 2021-05-25T12:00:55.482 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.470+0000 7ff88fe07540 15 client.4511.objecter _session_op_assign 0 4 2021-05-25T12:00:55.482 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.470+0000 7ff88fe07540 15 client.4511.objecter _send_op 4 to 9.1 on osd.0 2021-05-25T12:00:55.482 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.470+0000 7ff88fe07540 1 -- 172.21.15.181:0/4180131642 --> v1:172.21.15.102:6807/29973 -- osd_op(unknown.0.0:4 9.1 9:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected e17) v8 -- 0x55fe52610100 con 0x55fe52610cd0 2021-05-25T12:00:55.482 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.470+0000 7ff88fe07540 20 client.4511.objecter put_session s=0x55fe524c84a0 osd=0 5 2021-05-25T12:00:55.483 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.470+0000 7ff88fe07540 5 client.4511.objecter 2 in flight 2021-05-25T12:00:55.483 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.473+0000 7ff887a96700 1 -- 172.21.15.181:0/4180131642 <== osd.0 v1:172.21.15.102:6807/29973 3 ==== osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (unknown 2177760655 0 3741568377) 0x7ff880004550 con 0x55fe52610cd0 2021-05-25T12:00:55.483 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.473+0000 7ff887a96700 10 client.4511.objecter ms_dispatch 0x55fe524d4560 osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2021-05-25T12:00:55.483 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.473+0000 7ff887a96700 10 client.4511.objecter in handle_osd_op_reply 2021-05-25T12:00:55.483 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.473+0000 7ff887a96700 7 client.4511.objecter handle_osd_op_reply 3 ondisk uv 1 in 9.1 attempt 0 2021-05-25T12:00:55.484 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.473+0000 7ff887a96700 10 client.4511.objecter op 0 rval 0 len 5 2021-05-25T12:00:55.484 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.473+0000 7ff887a96700 15 client.4511.objecter handle_osd_op_reply completed tid 3 2021-05-25T12:00:55.484 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.473+0000 7ff887a96700 15 client.4511.objecter _finish_op 3 2021-05-25T12:00:55.484 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.473+0000 7ff887a96700 20 client.4511.objecter put_session s=0x55fe524c84a0 osd=0 5 2021-05-25T12:00:55.484 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.473+0000 7ff887a96700 15 client.4511.objecter _session_op_remove 0 3 2021-05-25T12:00:55.485 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.473+0000 7ff887a96700 5 client.4511.objecter 1 in flight 2021-05-25T12:00:55.485 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.473+0000 7ff887a96700 1 -- 172.21.15.181:0/4180131642 <== osd.0 v1:172.21.15.102:6807/29973 4 ==== osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (unknown 370542934 0 0) 0x7ff880004550 con 0x55fe52610cd0[ OK ] AsioRados.AsyncReadCallback 2021-05-25T12:00:55.485 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.473+0000 7ff887a96700 10 client.4511.objecter ms_dispatch 0x55fe524d4560 osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 (3 ms) 2021-05-25T12:00:55.485 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: [ RUN ] AsioRados.AsyncReadFuture 2021-05-25T12:00:55.485 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.486 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.473+0000 7ff887a96700 10 client.4511.objecter in handle_osd_op_reply 2021-05-25T12:00:55.486 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.473+0000 7ff887a96700 7 client.4511.objecter handle_osd_op_reply 4 ondisk uv 0 in 9.1 attempt 0 2021-05-25T12:00:55.486 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.473+0000 7ff887a96700 10 client.4511.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2021-05-25T12:00:55.486 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.473+0000 7ff887a96700 10 client.4511.objecter op 0 rval 0 len 0 2021-05-25T12:00:55.486 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.473+0000 7ff887a96700 15 client.4511.objecter handle_osd_op_reply completed tid 4 2021-05-25T12:00:55.487 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.473+0000 7ff887a96700 15 client.4511.objecter _finish_op 4 2021-05-25T12:00:55.487 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.473+0000 7ff887a96700 20 client.4511.objecter put_session s=0x55fe524c84a0 osd=0 4 2021-05-25T12:00:55.487 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.473+0000 7ff887a96700 15 client.4511.objecter _session_op_remove 0 4 2021-05-25T12:00:55.487 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.473+0000 7ff887a96700 5 client.4511.objecter 0 in flight 2021-05-25T12:00:55.487 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.473+0000 7ff88fe07540 10 client.4511.objecter _op_submit op 0x55fe5260f2c0 2021-05-25T12:00:55.488 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.473+0000 7ff88fe07540 20 client.4511.objecter _calc_target epoch 17 base exist @9 precalc_pgid 0 pgid 0.0 is_read 2021-05-25T12:00:55.488 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.473+0000 7ff88fe07540 20 client.4511.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2021-05-25T12:00:55.488 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.473+0000 7ff88fe07540 10 client.4511.objecter _calc_target raw pgid 9.d3a59201 -> actual 9.1 acting [0,2] primary 0 2021-05-25T12:00:55.488 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.473+0000 7ff88fe07540 20 client.4511.objecter _get_session s=0x55fe524c84a0 osd=0 3 2021-05-25T12:00:55.489 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.473+0000 7ff88fe07540 10 client.4511.objecter _op_submit oid exist '@9' '@9' [read 0~256] tid 5 osd.0 2021-05-25T12:00:55.489 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.473+0000 7ff88fe07540 20 client.4511.objecter get_session s=0x55fe524c84a0 osd=0 3 2021-05-25T12:00:55.489 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.473+0000 7ff88fe07540 15 client.4511.objecter _session_op_assign 0 5 2021-05-25T12:00:55.489 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.473+0000 7ff88fe07540 15 client.4511.objecter _send_op 5 to 9.1 on osd.0 2021-05-25T12:00:55.489 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.473+0000 7ff88fe07540 1 -- 172.21.15.181:0/4180131642 --> v1:172.21.15.102:6807/29973 -- osd_op(unknown.0.0:5 9.1 9:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected e17) v8 -- 0x55fe5260fbf0 con 0x55fe52610cd0 2021-05-25T12:00:55.490 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.473+0000 7ff88fe07540 20 client.4511.objecter put_session s=0x55fe524c84a0 osd=0 4 2021-05-25T12:00:55.490 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.473+0000 7ff88fe07540 5 client.4511.objecter 1 in flight 2021-05-25T12:00:55.490 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.473+0000 7ff88fe07540 10 client.4511.objecter _op_submit op 0x55fe52614b90 2021-05-25T12:00:55.490 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.473+0000 7ff88fe07540 20 client.4511.objecter _calc_target epoch 17 base noexist @9 precalc_pgid 0 pgid 0.0 is_read 2021-05-25T12:00:55.490 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.473+0000 7ff88fe07540 20 client.4511.objecter _calc_target target noexist @9 -> pgid 9.a7bbaa1 2021-05-25T12:00:55.491 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.473+0000 7ff88fe07540 10 client.4511.objecter _calc_target raw pgid 9.a7bbaa1 -> actual 9.1 acting [0,2] primary 0 2021-05-25T12:00:55.491 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.473+0000 7ff88fe07540 20 client.4511.objecter _get_session s=0x55fe524c84a0 osd=0 4 2021-05-25T12:00:55.491 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.473+0000 7ff88fe07540 10 client.4511.objecter _op_submit oid noexist '@9' '@9' [read 0~256] tid 6 osd.0 2021-05-25T12:00:55.491 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.473+0000 7ff88fe07540 20 client.4511.objecter get_session s=0x55fe524c84a0 osd=0 4 2021-05-25T12:00:55.491 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.473+0000 7ff88fe07540 15 client.4511.objecter _session_op_assign 0 6 2021-05-25T12:00:55.492 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.473+0000 7ff88fe07540 15 client.4511.objecter _send_op 6 to 9.1 on osd.0 2021-05-25T12:00:55.492 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.473+0000 7ff88fe07540 1 -- 172.21.15.181:0/4180131642 --> v1:172.21.15.102:6807/29973 -- osd_op(unknown.0.0:6 9.1 9:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected e17) v8 -- 0x55fe52615080 con 0x55fe52610cd0 2021-05-25T12:00:55.492 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.473+0000 7ff88fe07540 20 client.4511.objecter put_session s=0x55fe524c84a0 osd=0 5 2021-05-25T12:00:55.492 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.473+0000 7ff88fe07540 5 client.4511.objecter 2 in flight 2021-05-25T12:00:55.492 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.474+0000 7ff887a96700 1 -- 172.21.15.181:0/4180131642 <== osd.0 v1:172.21.15.102:6807/29973 5 ==== osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (unknown 2177760655 0 3741568377) 0x7ff880004550 con 0x55fe52610cd0 2021-05-25T12:00:55.493 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.474+0000 7ff887a96700 10 client.4511.objecter ms_dispatch 0x55fe524d4560 osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2021-05-25T12:00:55.493 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.474+0000 7ff887a96700 10 client.4511.objecter in handle_osd_op_reply 2021-05-25T12:00:55.493 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.474+0000 7ff887a96700 7 client.4511.objecter handle_osd_op_reply 5 ondisk uv 1 in 9.1 attempt 0 2021-05-25T12:00:55.493 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.474+0000 7ff887a96700 10 client.4511.objecter op 0 rval 0 len 5 2021-05-25T12:00:55.493 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.474+0000 7ff887a96700 15 client.4511.objecter handle_osd_op_reply completed tid 5 2021-05-25T12:00:55.494 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.474+0000 7ff887a96700 15 client.4511.objecter _finish_op 5 2021-05-25T12:00:55.494 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.474+0000 7ff887a96700 20 client.4511.objecter put_session s=0x55fe524c84a0 osd=0 5 2021-05-25T12:00:55.494 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.474+0000 7ff887a96700 15 client.4511.objecter _session_op_remove 0 5 2021-05-25T12:00:55.494 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.474+0000 7ff887a96700 5 client.4511.objecter 1 in flight 2021-05-25T12:00:55.495 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.474+0000 7ff887a96700 1 -- 172.21.15.181:0/4180131642 <== osd.0 v1:172.21.15.102:6807/29973 6 ==== osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (unknown 370542934 0 0) 0x7ff880004550 con 0x55fe52610cd0 2021-05-25T12:00:55.495 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.474+0000 7ff887a96700 10 client.4511.objecter ms_dispatch 0x55fe524d4560 osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2021-05-25T12:00:55.495 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.474+0000 7ff887a96700 10 client.4511.objecter in handle_osd_op_reply 2021-05-25T12:00:55.495 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.474+0000 7ff887a96700 7 client.4511.objecter handle_osd_op_reply 6 ondisk uv 0 in 9.1 attempt 0 2021-05-25T12:00:55.495 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.474+0000 7ff887a96700 10 client.4511.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2021-05-25T12:00:55.496 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.474+0000 7ff887a96700 10 client.4511.objecter op 0 rval 0 len 0 2021-05-25T12:00:55.496 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.474+0000 7ff887a96700 15 client.4511.objecter handle_osd_op_reply completed tid 6 2021-05-25T12:00:55.496 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.474+0000 7ff887a96700 15 client.4511.objecter _finish_op 6 2021-05-25T12:00:55.496 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.474+0000 7ff887a96700 20 client.4511.objecter put_session s=0x55fe524c84a0 osd=0 4 2021-05-25T12:00:55.496 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.474+0000 7ff887a96700 15 client.4511.objecter _session_op_remove 0 6 2021-05-25T12:00:55.497 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.474+0000 7ff887a96700 5 client.4511.objecter 0 in flight 2021-05-25T12:00:55.497 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: [ OK ] AsioRados.AsyncReadFuture (1 ms) 2021-05-25T12:00:55.497 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: [ RUN ] AsioRados.AsyncReadYield 2021-05-25T12:00:55.497 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.474+0000 7ff88fe07540 10 client.4511.objecter _op_submit op 0x55fe5260f2c0 2021-05-25T12:00:55.498 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.474+0000 7ff88fe07540 20 client.4511.objecter _calc_target epoch 17 base exist @9 precalc_pgid 0 pgid 0.0 is_read 2021-05-25T12:00:55.498 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.475+0000 7ff88fe07540 20 client.4511.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2021-05-25T12:00:55.498 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.475+0000 7ff88fe07540 10 client.4511.objecter _calc_target raw pgid 9.d3a59201 -> actual 9.1 acting [0,2] primary 0 2021-05-25T12:00:55.498 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.475+0000 7ff88fe07540 20 client.4511.objecter _get_session s=0x55fe524c84a0 osd=0 3 2021-05-25T12:00:55.498 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.475+0000 7ff88fe07540 10 client.4511.objecter _op_submit oid exist '@9' '@9' [read 0~256] tid 7 osd.0 2021-05-25T12:00:55.499 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.475+0000 7ff88fe07540 20 client.4511.objecter get_session s=0x55fe524c84a0 osd=0 3 2021-05-25T12:00:55.499 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.475+0000 7ff88fe07540 15 client.4511.objecter _session_op_assign 0 7 2021-05-25T12:00:55.499 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.475+0000 7ff88fe07540 15 client.4511.objecter _send_op 7 to 9.1 on osd.0 2021-05-25T12:00:55.499 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.475+0000 7ff88fe07540 1 -- 172.21.15.181:0/4180131642 --> v1:172.21.15.102:6807/29973 -- osd_op(unknown.0.0:7 9.1 9:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected e17) v8 -- 0x55fe52614c70 con 0x55fe52610cd0 2021-05-25T12:00:55.499 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.475+0000 7ff88fe07540 20 client.4511.objecter put_session s=0x55fe524c84a0 osd=0 4 2021-05-25T12:00:55.500 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.475+0000 7ff88fe07540 5 client.4511.objecter 1 in flight 2021-05-25T12:00:55.500 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.475+0000 7ff88fe07540 10 client.4511.objecter _op_submit op 0x55fe52635c10 2021-05-25T12:00:55.500 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.475+0000 7ff88fe07540 20 client.4511.objecter _calc_target epoch 17 base noexist @9 precalc_pgid 0 pgid 0.0 is_read 2021-05-25T12:00:55.500 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.475+0000 7ff88fe07540 20 client.4511.objecter _calc_target target noexist @9 -> pgid 9.a7bbaa1 2021-05-25T12:00:55.501 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.475+0000 7ff88fe07540 10 client.4511.objecter _calc_target raw pgid 9.a7bbaa1 -> actual 9.1 acting [0,2] primary 0 2021-05-25T12:00:55.501 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.475+0000 7ff88fe07540 20 client.4511.objecter _get_session s=0x55fe524c84a0 osd=0 4 2021-05-25T12:00:55.501 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.475+0000 7ff88fe07540 10 client.4511.objecter _op_submit oid noexist '@9' '@9' [read 0~256] tid 8 osd.0 2021-05-25T12:00:55.501 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.475+0000 7ff88fe07540 20 client.4511.objecter get_session s=0x55fe524c84a0 osd=0 4 2021-05-25T12:00:55.501 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.475+0000 7ff88fe07540 15 client.4511.objecter _session_op_assign 0 8 2021-05-25T12:00:55.502 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.475+0000 7ff88fe07540 15 client.4511.objecter _send_op 8 to 9.1 on osd.0 2021-05-25T12:00:55.502 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.475+0000 7ff88fe07540 1 -- 172.21.15.181:0/4180131642 --> v1:172.21.15.102:6807/29973 -- osd_op(unknown.0.0:8 9.1 9:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected e17) v8 -- 0x55fe526361c0 con 0x55fe52610cd0 2021-05-25T12:00:55.502 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.475+0000 7ff88fe07540 20 client.4511.objecter put_session s=0x55fe524c84a0 osd=0 5 2021-05-25T12:00:55.502 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.475+0000 7ff88fe07540 5 client.4511.objecter 2 in flight 2021-05-25T12:00:55.502 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.475+0000 7ff887a96700 1 -- 172.21.15.181:0/4180131642 <== osd.0 v1:172.21.15.102:6807/29973 7 ==== osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (unknown 2177760655 0 3741568377) 0x7ff880004550 con 0x55fe52610cd0 2021-05-25T12:00:55.503 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.475+0000 7ff887a96700 10 client.4511.objecter ms_dispatch 0x55fe524d4560 osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2021-05-25T12:00:55.503 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.475+0000 7ff887a96700 10 client.4511.objecter in handle_osd_op_reply 2021-05-25T12:00:55.503 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.475+0000 7ff887a96700 7 client.4511.objecter handle_osd_op_reply 7 ondisk uv 1 in 9.1 attempt 0 2021-05-25T12:00:55.503 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.475+0000 7ff887a96700 10 client.4511.objecter op 0 rval 0 len 5 2021-05-25T12:00:55.504 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.475+0000 7ff887a96700 15 client.4511.objecter handle_osd_op_reply completed tid 7 2021-05-25T12:00:55.504 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.475+0000 7ff887a96700 15 client.4511.objecter _finish_op 7 2021-05-25T12:00:55.504 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.475+0000 7ff887a96700 20 client.4511.objecter put_session s=0x55fe524c84a0 osd=0 5 2021-05-25T12:00:55.504 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.475+0000 7ff887a96700 15 client.4511.objecter _session_op_remove 0 7 2021-05-25T12:00:55.504 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.475+0000 7ff887a96700 5 client.4511.objecter 1 in flight 2021-05-25T12:00:55.505 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.475+0000 7ff887a96700 1 -- 172.21.15.181:0/4180131642 <== osd.0 v1:172.21.15.102:6807/29973 8 ==== osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (unknown 370542934 0 0) 0x7ff880004550 con 0x55fe52610cd0 2021-05-25T12:00:55.505 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.475+0000 7ff887a96700 10 client.4511.objecter ms_dispatch 0x55fe524d4560 osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2021-05-25T12:00:55.505 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.475+0000 7ff887a96700 10 client.4511.objecter in handle_osd_op_reply 2021-05-25T12:00:55.505 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.475+0000 7ff887a96700 7 client.4511.objecter handle_osd_op_reply 8 ondisk uv 0 in 9.1 attempt 0 2021-05-25T12:00:55.505 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: [ OK ] AsioRados.AsyncReadYield2021-05-25T12:00:55.475+0000 7ff887a96700 10 client.4511.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2021-05-25T12:00:55.506 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: (1 ms) 2021-05-25T12:00:55.506 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.475+0000 7ff887a96700 10 client.4511.objecter op 0 rval 0 len 0[ RUN ] AsioRados.AsyncWriteCallback 2021-05-25T12:00:55.506 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.506 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.475+0000 7ff887a96700 15 client.4511.objecter handle_osd_op_reply completed tid 8 2021-05-25T12:00:55.506 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.475+0000 7ff887a96700 15 client.4511.objecter _finish_op 8 2021-05-25T12:00:55.507 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.475+0000 7ff887a96700 20 client.4511.objecter put_session s=0x55fe524c84a0 osd=0 4 2021-05-25T12:00:55.507 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.475+0000 7ff887a96700 15 client.4511.objecter _session_op_remove 0 8 2021-05-25T12:00:55.507 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.475+0000 7ff887a96700 5 client.4511.objecter 0 in flight 2021-05-25T12:00:55.507 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.475+0000 7ff88fe07540 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2021-05-25T12:00:55.508 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.475+0000 7ff88fe07540 20 librados: queue_aio_write 0x55fe524d0970 completion 0x55fe5260f2c0 write_seq 1 2021-05-25T12:00:55.508 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.475+0000 7ff88fe07540 10 client.4511.objecter _op_submit op 0x55fe52635bf0 2021-05-25T12:00:55.508 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.475+0000 7ff88fe07540 20 client.4511.objecter _calc_target epoch 17 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2021-05-25T12:00:55.508 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.475+0000 7ff88fe07540 20 client.4511.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2021-05-25T12:00:55.508 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.475+0000 7ff88fe07540 10 client.4511.objecter _calc_target raw pgid 9.d3a59201 -> actual 9.1 acting [0,2] primary 0 2021-05-25T12:00:55.509 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.475+0000 7ff88fe07540 20 client.4511.objecter _get_session s=0x55fe524c84a0 osd=0 3 2021-05-25T12:00:55.509 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.475+0000 7ff88fe07540 10 client.4511.objecter _op_submit oid exist '@9' '@9' [write 0~5 in=5b] tid 9 osd.0 2021-05-25T12:00:55.509 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.475+0000 7ff88fe07540 20 client.4511.objecter get_session s=0x55fe524c84a0 osd=0 3 2021-05-25T12:00:55.509 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.475+0000 7ff88fe07540 15 client.4511.objecter _session_op_assign 0 9 2021-05-25T12:00:55.510 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.475+0000 7ff88fe07540 15 client.4511.objecter _send_op 9 to 9.1 on osd.0 2021-05-25T12:00:55.511 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.475+0000 7ff88fe07540 1 -- 172.21.15.181:0/4180131642 --> v1:172.21.15.102:6807/29973 -- osd_op(unknown.0.0:9 9.1 9:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected e17) v8 -- 0x55fe5260f3d0 con 0x55fe52610cd0 2021-05-25T12:00:55.512 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.475+0000 7ff88fe07540 20 client.4511.objecter put_session s=0x55fe524c84a0 osd=0 4 2021-05-25T12:00:55.512 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.475+0000 7ff88fe07540 5 client.4511.objecter 1 in flight 2021-05-25T12:00:55.512 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.475+0000 7ff88fe07540 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2021-05-25T12:00:55.512 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.479+0000 7ff887a96700 1 -- 172.21.15.181:0/4180131642 <== osd.0 v1:172.21.15.102:6807/29973 9 ==== osd_op_reply(9 exist [write 0~5] v17'2 uv2 ondisk = 0) v8 ==== 149+0+0 (unknown 837496600 0 0) 0x7ff880004550 con 0x55fe52610cd0 2021-05-25T12:00:55.513 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.479+0000 7ff887a96700 10 client.4511.objecter ms_dispatch 0x55fe524d4560 osd_op_reply(9 exist [write 0~5] v17'2 uv2 ondisk = 0) v8 2021-05-25T12:00:55.513 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.479+0000 7ff887a96700 10 client.4511.objecter in handle_osd_op_reply 2021-05-25T12:00:55.513 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: [ OK ] AsioRados.AsyncWriteCallback2021-05-25T12:00:55.479+0000 7ff887a96700 7 client.4511.objecter handle_osd_op_reply 9 ondisk uv 2 in 9.1 attempt 0 2021-05-25T12:00:55.513 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: (4 ms) 2021-05-25T12:00:55.513 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteFuture 2021-05-25T12:00:55.514 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.479+0000 7ff887a96700 10 client.4511.objecter op 0 rval 0 len 0 2021-05-25T12:00:55.514 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.479+0000 7ff887a96700 15 client.4511.objecter handle_osd_op_reply completed tid 9 2021-05-25T12:00:55.514 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.479+0000 7ff887a96700 15 client.4511.objecter _finish_op 9 2021-05-25T12:00:55.514 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.479+0000 7ff887a96700 20 client.4511.objecter put_session s=0x55fe524c84a0 osd=0 4 2021-05-25T12:00:55.515 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.479+0000 7ff887a96700 15 client.4511.objecter _session_op_remove 0 9 2021-05-25T12:00:55.515 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.479+0000 7ff887a96700 5 client.4511.objecter 0 in flight 2021-05-25T12:00:55.515 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.479+0000 7ff887a96700 20 librados: complete_aio_write 0x55fe5260f2c0 2021-05-25T12:00:55.515 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.479+0000 7ff88fe07540 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2021-05-25T12:00:55.515 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.479+0000 7ff88fe07540 20 librados: queue_aio_write 0x55fe524d0970 completion 0x55fe52616440 write_seq 2 2021-05-25T12:00:55.516 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.479+0000 7ff88fe07540 10 client.4511.objecter _op_submit op 0x55fe52635bf0 2021-05-25T12:00:55.516 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.479+0000 7ff88fe07540 20 client.4511.objecter _calc_target epoch 17 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2021-05-25T12:00:55.516 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.479+0000 7ff88fe07540 20 client.4511.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2021-05-25T12:00:55.516 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.479+0000 7ff88fe07540 10 client.4511.objecter _calc_target raw pgid 9.d3a59201 -> actual 9.1 acting [0,2] primary 0 2021-05-25T12:00:55.516 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.479+0000 7ff88fe07540 20 client.4511.objecter _get_session s=0x55fe524c84a0 osd=0 3 2021-05-25T12:00:55.517 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.479+0000 7ff88fe07540 10 client.4511.objecter _op_submit oid exist '@9' '@9' [write 0~5 in=5b] tid 10 osd.0 2021-05-25T12:00:55.517 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.479+0000 7ff88fe07540 20 client.4511.objecter get_session s=0x55fe524c84a0 osd=0 3 2021-05-25T12:00:55.517 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.479+0000 7ff88fe07540 15 client.4511.objecter _session_op_assign 0 10 2021-05-25T12:00:55.517 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.479+0000 7ff88fe07540 15 client.4511.objecter _send_op 10 to 9.1 on osd.0 2021-05-25T12:00:55.517 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.479+0000 7ff88fe07540 1 -- 172.21.15.181:0/4180131642 --> v1:172.21.15.102:6807/29973 -- osd_op(unknown.0.0:10 9.1 9:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected e17) v8 -- 0x55fe5260f3d0 con 0x55fe52610cd0 2021-05-25T12:00:55.518 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.480+0000 7ff88fe07540 20 client.4511.objecter put_session s=0x55fe524c84a0 osd=0 4 2021-05-25T12:00:55.518 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.480+0000 7ff88fe07540 5 client.4511.objecter 1 in flight 2021-05-25T12:00:55.518 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.480+0000 7ff88fe07540 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2021-05-25T12:00:55.518 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.481+0000 7ff887a96700 1 -- 172.21.15.181:0/4180131642 <== osd.0 v1:172.21.15.102:6807/29973 10 ==== osd_op_reply(10 exist [write 0~5] v17'3 uv3 ondisk = 0) v8 ==== 149+0+0 (unknown 1164678348 0 0) 0x7ff880004550 con 0x55fe52610cd0 2021-05-25T12:00:55.519 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.481+0000 7ff887a96700 10 client.4511.objecter ms_dispatch 0x55fe524d4560 osd_op_reply(10 exist [write 0~5] v17'3 uv3 ondisk = 0) v8 2021-05-25T12:00:55.519 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.481+0000 7ff887a96700 10 client.4511.objecter in handle_osd_op_reply 2021-05-25T12:00:55.519 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.481+0000 7ff887a96700 7 client.4511.objecter handle_osd_op_reply 10 ondisk uv 3 in 9.1 attempt 0 2021-05-25T12:00:55.519 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.481+0000 7ff887a96700 10 client.4511.objecter op 0 rval 0 len 0 2021-05-25T12:00:55.519 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.481+0000 7ff887a96700 15 client.4511.objecter handle_osd_op_reply completed tid 10[ OK ] AsioRados.AsyncWriteFuture 2021-05-25T12:00:55.520 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: (1 ms) 2021-05-25T12:00:55.520 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.481+0000 7ff887a96700 15 client.4511.objecter _finish_op 10 2021-05-25T12:00:55.520 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteYield 2021-05-25T12:00:55.520 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.481+0000 7ff887a96700 20 client.4511.objecter put_session s=0x55fe524c84a0 osd=0 4 2021-05-25T12:00:55.520 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.481+0000 7ff887a96700 15 client.4511.objecter _session_op_remove 0 10 2021-05-25T12:00:55.521 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.481+0000 7ff887a96700 5 client.4511.objecter 0 in flight 2021-05-25T12:00:55.521 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.481+0000 7ff887a96700 20 librados: complete_aio_write 0x55fe52616440 2021-05-25T12:00:55.521 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.481+0000 7ff88fe07540 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2021-05-25T12:00:55.521 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.481+0000 7ff88fe07540 20 librados: queue_aio_write 0x55fe524d0970 completion 0x55fe52616550 write_seq 3 2021-05-25T12:00:55.521 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.481+0000 7ff88fe07540 10 client.4511.objecter _op_submit op 0x55fe52628b60 2021-05-25T12:00:55.522 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.481+0000 7ff88fe07540 20 client.4511.objecter _calc_target epoch 17 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2021-05-25T12:00:55.522 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.481+0000 7ff88fe07540 20 client.4511.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2021-05-25T12:00:55.522 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.481+0000 7ff88fe07540 10 client.4511.objecter _calc_target raw pgid 9.d3a59201 -> actual 9.1 acting [0,2] primary 0 2021-05-25T12:00:55.522 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.481+0000 7ff88fe07540 20 client.4511.objecter _get_session s=0x55fe524c84a0 osd=0 3 2021-05-25T12:00:55.522 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.481+0000 7ff88fe07540 10 client.4511.objecter _op_submit oid exist '@9' '@9' [write 0~5 in=5b] tid 11 osd.0 2021-05-25T12:00:55.523 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.481+0000 7ff88fe07540 20 client.4511.objecter get_session s=0x55fe524c84a0 osd=0 3 2021-05-25T12:00:55.523 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.481+0000 7ff88fe07540 15 client.4511.objecter _session_op_assign 0 11 2021-05-25T12:00:55.523 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.481+0000 7ff88fe07540 15 client.4511.objecter _send_op 11 to 9.1 on osd.0 2021-05-25T12:00:55.523 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.481+0000 7ff88fe07540 1 -- 172.21.15.181:0/4180131642 --> v1:172.21.15.102:6807/29973 -- osd_op(unknown.0.0:11 9.1 9:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected e17) v8 -- 0x55fe52629050 con 0x55fe52610cd0 2021-05-25T12:00:55.523 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.481+0000 7ff88fe07540 20 client.4511.objecter put_session s=0x55fe524c84a0 osd=0 4 2021-05-25T12:00:55.524 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.481+0000 7ff88fe07540 5 client.4511.objecter 1 in flight 2021-05-25T12:00:55.524 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.481+0000 7ff88fe07540 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2021-05-25T12:00:55.524 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff887a96700 1 -- 172.21.15.181:0/4180131642 <== osd.0 v1:172.21.15.102:6807/29973 11 ==== osd_op_reply(11 exist [write 0~5] v17'4 uv4 ondisk = 0) v8 ==== 149+0+0 (unknown 251969809 0 0) 0x7ff880004550 con 0x55fe52610cd0 2021-05-25T12:00:55.524 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff887a96700 10 client.4511.objecter ms_dispatch 0x55fe524d4560 osd_op_reply(11 exist [write 0~5] v17'4 uv4 ondisk = 0) v8 2021-05-25T12:00:55.524 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff887a96700 10 client.4511.objecter in handle_osd_op_reply 2021-05-25T12:00:55.525 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff887a96700 7 client.4511.objecter handle_osd_op_reply 11 ondisk uv 4 in 9.1 attempt 0[ OK ] AsioRados.AsyncWriteYield 2021-05-25T12:00:55.525 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff887a96700 10 client.4511.objecter op 0 rval 0 len 0 (1 ms) 2021-05-25T12:00:55.525 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationCallback 2021-05-25T12:00:55.525 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.525 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff887a96700 15 client.4511.objecter handle_osd_op_reply completed tid 11 2021-05-25T12:00:55.526 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff887a96700 15 client.4511.objecter _finish_op 11 2021-05-25T12:00:55.526 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff887a96700 20 client.4511.objecter put_session s=0x55fe524c84a0 osd=0 4 2021-05-25T12:00:55.526 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff887a96700 15 client.4511.objecter _session_op_remove 0 11 2021-05-25T12:00:55.526 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff887a96700 5 client.4511.objecter 0 in flight 2021-05-25T12:00:55.526 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff887a96700 20 librados: complete_aio_write 0x55fe52616550 2021-05-25T12:00:55.527 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff88fe07540 10 client.4511.objecter _op_submit op 0x55fe52615680 2021-05-25T12:00:55.527 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff88fe07540 20 client.4511.objecter _calc_target epoch 17 base exist @9 precalc_pgid 0 pgid 0.0 is_read 2021-05-25T12:00:55.527 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff88fe07540 20 client.4511.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2021-05-25T12:00:55.527 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff88fe07540 10 client.4511.objecter _calc_target raw pgid 9.d3a59201 -> actual 9.1 acting [0,2] primary 0 2021-05-25T12:00:55.527 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff88fe07540 20 client.4511.objecter _get_session s=0x55fe524c84a0 osd=0 3 2021-05-25T12:00:55.528 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff88fe07540 10 client.4511.objecter _op_submit oid exist '@9' '@9' [read 0~0] tid 12 osd.0 2021-05-25T12:00:55.528 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff88fe07540 20 client.4511.objecter get_session s=0x55fe524c84a0 osd=0 3 2021-05-25T12:00:55.528 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff88fe07540 15 client.4511.objecter _session_op_assign 0 12 2021-05-25T12:00:55.528 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff88fe07540 15 client.4511.objecter _send_op 12 to 9.1 on osd.0 2021-05-25T12:00:55.528 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff88fe07540 1 -- 172.21.15.181:0/4180131642 --> v1:172.21.15.102:6807/29973 -- osd_op(unknown.0.0:12 9.1 9:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected e17) v8 -- 0x55fe52615b70 con 0x55fe52610cd0 2021-05-25T12:00:55.529 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff88fe07540 20 client.4511.objecter put_session s=0x55fe524c84a0 osd=0 4 2021-05-25T12:00:55.529 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff88fe07540 5 client.4511.objecter 1 in flight 2021-05-25T12:00:55.529 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff88fe07540 10 client.4511.objecter _op_submit op 0x55fe52629480 2021-05-25T12:00:55.529 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff88fe07540 20 client.4511.objecter _calc_target epoch 17 base noexist @9 precalc_pgid 0 pgid 0.0 is_read 2021-05-25T12:00:55.529 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff88fe07540 20 client.4511.objecter _calc_target target noexist @9 -> pgid 9.a7bbaa1 2021-05-25T12:00:55.530 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff88fe07540 10 client.4511.objecter _calc_target raw pgid 9.a7bbaa1 -> actual 9.1 acting [0,2] primary 0 2021-05-25T12:00:55.530 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff88fe07540 20 client.4511.objecter _get_session s=0x55fe524c84a0 osd=0 4 2021-05-25T12:00:55.530 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff88fe07540 10 client.4511.objecter _op_submit oid noexist '@9' '@9' [read 0~0] tid 13 osd.0 2021-05-25T12:00:55.530 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff88fe07540 20 client.4511.objecter get_session s=0x55fe524c84a0 osd=0 4 2021-05-25T12:00:55.531 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff88fe07540 15 client.4511.objecter _session_op_assign 0 13 2021-05-25T12:00:55.531 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff88fe07540 15 client.4511.objecter _send_op 13 to 9.1 on osd.0 2021-05-25T12:00:55.531 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff88fe07540 1 -- 172.21.15.181:0/4180131642 --> v1:172.21.15.102:6807/29973 -- osd_op(unknown.0.0:13 9.1 9:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected e17) v8 -- 0x55fe52615b70 con 0x55fe52610cd0 2021-05-25T12:00:55.531 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff88fe07540 20 client.4511.objecter put_session s=0x55fe524c84a0 osd=0 5 2021-05-25T12:00:55.531 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff88fe07540 5 client.4511.objecter 2 in flight 2021-05-25T12:00:55.532 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff887a96700 1 -- 172.21.15.181:0/4180131642 <== osd.0 v1:172.21.15.102:6807/29973 12 ==== osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (unknown 4087291774 0 3741568377) 0x7ff880004550 con 0x55fe52610cd0 2021-05-25T12:00:55.532 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff887a96700 10 client.4511.objecter ms_dispatch 0x55fe524d4560 osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2021-05-25T12:00:55.532 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff887a96700 10 client.4511.objecter in handle_osd_op_reply 2021-05-25T12:00:55.532 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff887a96700 7 client.4511.objecter handle_osd_op_reply 12 ondisk uv 4 in 9.1 attempt 0 2021-05-25T12:00:55.532 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff887a96700 10 client.4511.objecter op 0 rval 0 len 5 2021-05-25T12:00:55.533 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff887a96700 15 client.4511.objecter handle_osd_op_reply completed tid 12 2021-05-25T12:00:55.533 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff887a96700 15 client.4511.objecter _finish_op 12 2021-05-25T12:00:55.533 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff887a96700 20 client.4511.objecter put_session s=0x55fe524c84a0 osd=0 5 2021-05-25T12:00:55.533 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff887a96700 15 client.4511.objecter _session_op_remove 0 12 2021-05-25T12:00:55.533 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff887a96700 5 client.4511.objecter 1 in flight 2021-05-25T12:00:55.534 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff887a96700 1 -- 172.21.15.181:0/4180131642 <== osd.0 v1:172.21.15.102:6807/29973 13 ==== osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (unknown 2073966446 0 0) 0x7ff880004550 con 0x55fe52610cd0 2021-05-25T12:00:55.534 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff887a96700 10 client.4511.objecter ms_dispatch 0x55fe524d4560 osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2021-05-25T12:00:55.534 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff887a96700 10 client.4511.objecter in handle_osd_op_reply 2021-05-25T12:00:55.534 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff887a96700 7 client.4511.objecter handle_osd_op_reply 13 ondisk uv 0 in 9.1 attempt 0 2021-05-25T12:00:55.534 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationCallback2021-05-25T12:00:55.482+0000 7ff887a96700 10 client.4511.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2021-05-25T12:00:55.535 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: (1 ms) 2021-05-25T12:00:55.535 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationFuture 2021-05-25T12:00:55.535 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff887a96700 10 client.4511.objecter op 0 rval 0 len 0 2021-05-25T12:00:55.535 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff887a96700 15 client.4511.objecter handle_osd_op_reply completed tid 13 2021-05-25T12:00:55.535 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff887a96700 15 client.4511.objecter _finish_op 13 2021-05-25T12:00:55.536 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff887a96700 20 client.4511.objecter put_session s=0x55fe524c84a0 osd=0 4 2021-05-25T12:00:55.536 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff887a96700 15 client.4511.objecter _session_op_remove 0 13 2021-05-25T12:00:55.536 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff887a96700 5 client.4511.objecter 0 in flight 2021-05-25T12:00:55.536 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff88fe07540 10 client.4511.objecter _op_submit op 0x55fe52615680 2021-05-25T12:00:55.536 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff88fe07540 20 client.4511.objecter _calc_target epoch 17 base exist @9 precalc_pgid 0 pgid 0.0 is_read 2021-05-25T12:00:55.537 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff88fe07540 20 client.4511.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2021-05-25T12:00:55.537 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff88fe07540 10 client.4511.objecter _calc_target raw pgid 9.d3a59201 -> actual 9.1 acting [0,2] primary 0 2021-05-25T12:00:55.537 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff88fe07540 20 client.4511.objecter _get_session s=0x55fe524c84a0 osd=0 3 2021-05-25T12:00:55.537 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff88fe07540 10 client.4511.objecter _op_submit oid exist '@9' '@9' [read 0~0] tid 14 osd.0 2021-05-25T12:00:55.537 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff88fe07540 20 client.4511.objecter get_session s=0x55fe524c84a0 osd=0 3 2021-05-25T12:00:55.538 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff88fe07540 15 client.4511.objecter _session_op_assign 0 14 2021-05-25T12:00:55.538 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff88fe07540 15 client.4511.objecter _send_op 14 to 9.1 on osd.0 2021-05-25T12:00:55.538 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff88fe07540 1 -- 172.21.15.181:0/4180131642 --> v1:172.21.15.102:6807/29973 -- osd_op(unknown.0.0:14 9.1 9:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected e17) v8 -- 0x55fe52615b70 con 0x55fe52610cd0 2021-05-25T12:00:55.538 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff88fe07540 20 client.4511.objecter put_session s=0x55fe524c84a0 osd=0 4 2021-05-25T12:00:55.538 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff88fe07540 5 client.4511.objecter 1 in flight 2021-05-25T12:00:55.539 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff88fe07540 10 client.4511.objecter _op_submit op 0x55fe52629430 2021-05-25T12:00:55.539 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff88fe07540 20 client.4511.objecter _calc_target epoch 17 base noexist @9 precalc_pgid 0 pgid 0.0 is_read 2021-05-25T12:00:55.539 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff88fe07540 20 client.4511.objecter _calc_target target noexist @9 -> pgid 9.a7bbaa1 2021-05-25T12:00:55.539 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff88fe07540 10 client.4511.objecter _calc_target raw pgid 9.a7bbaa1 -> actual 9.1 acting [0,2] primary 0 2021-05-25T12:00:55.540 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff88fe07540 20 client.4511.objecter _get_session s=0x55fe524c84a0 osd=0 4 2021-05-25T12:00:55.540 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff88fe07540 10 client.4511.objecter _op_submit oid noexist '@9' '@9' [read 0~0] tid 15 osd.0 2021-05-25T12:00:55.540 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff88fe07540 20 client.4511.objecter get_session s=0x55fe524c84a0 osd=0 4 2021-05-25T12:00:55.540 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff88fe07540 15 client.4511.objecter _session_op_assign 0 15 2021-05-25T12:00:55.540 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff88fe07540 15 client.4511.objecter _send_op 15 to 9.1 on osd.0 2021-05-25T12:00:55.541 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff88fe07540 1 -- 172.21.15.181:0/4180131642 --> v1:172.21.15.102:6807/29973 -- osd_op(unknown.0.0:15 9.1 9:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected e17) v8 -- 0x55fe52629920 con 0x55fe52610cd0 2021-05-25T12:00:55.541 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff88fe07540 20 client.4511.objecter put_session s=0x55fe524c84a0 osd=0 5 2021-05-25T12:00:55.541 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff88fe07540 5 client.4511.objecter 2 in flight 2021-05-25T12:00:55.541 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff887a96700 1 -- 172.21.15.181:0/4180131642 <== osd.0 v1:172.21.15.102:6807/29973 14 ==== osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (unknown 4087291774 0 3741568377) 0x7ff880004550 con 0x55fe52610cd0 2021-05-25T12:00:55.541 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff887a96700 10 client.4511.objecter ms_dispatch 0x55fe524d4560 osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2021-05-25T12:00:55.542 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff887a96700 10 client.4511.objecter in handle_osd_op_reply 2021-05-25T12:00:55.542 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff887a96700 7 client.4511.objecter handle_osd_op_reply 14 ondisk uv 4 in 9.1 attempt 0 2021-05-25T12:00:55.542 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff887a96700 10 client.4511.objecter op 0 rval 0 len 5 2021-05-25T12:00:55.542 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff887a96700 15 client.4511.objecter handle_osd_op_reply completed tid 14 2021-05-25T12:00:55.542 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff887a96700 15 client.4511.objecter _finish_op 14 2021-05-25T12:00:55.543 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff887a96700 20 client.4511.objecter put_session s=0x55fe524c84a0 osd=0 5 2021-05-25T12:00:55.543 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff887a96700 15 client.4511.objecter _session_op_remove 0 14 2021-05-25T12:00:55.543 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff887a96700 5 client.4511.objecter 1 in flight 2021-05-25T12:00:55.543 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.482+0000 7ff887a96700 1 -- 172.21.15.181:0/4180131642 <== osd.0 v1:172.21.15.102:6807/29973 15 ==== osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (unknown 2073966446 0 0) 0x7ff880004550 con 0x55fe52610cd0 2021-05-25T12:00:55.543 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.483+0000 7ff887a96700 10 client.4511.objecter ms_dispatch 0x55fe524d4560 osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2021-05-25T12:00:55.544 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.483+0000 7ff887a96700 10 client.4511.objecter in handle_osd_op_reply 2021-05-25T12:00:55.544 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.483+0000 7ff887a96700 7 client.4511.objecter handle_osd_op_reply 15 ondisk uv 0 in 9.1 attempt 0 2021-05-25T12:00:55.544 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.483+0000 7ff887a96700 10 client.4511.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2021-05-25T12:00:55.544 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.483+0000 7ff887a96700 10 client.4511.objecter op 0 rval 0 len 0 2021-05-25T12:00:55.545 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.483+0000 7ff887a96700 15 client.4511.objecter handle_osd_op_reply completed tid 15 2021-05-25T12:00:55.545 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.483+0000 7ff887a96700 15 client.4511.objecter _finish_op 15 2021-05-25T12:00:55.545 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.483+0000 7ff887a96700 20 client.4511.objecter put_session s=0x55fe524c84a0 osd=0 4 2021-05-25T12:00:55.545 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationFuture (0 ms) 2021-05-25T12:00:55.545 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationYield 2021-05-25T12:00:55.546 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.483+0000 7ff887a96700 15 client.4511.objecter _session_op_remove 0 15 2021-05-25T12:00:55.546 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.483+0000 7ff887a96700 5 client.4511.objecter 0 in flight 2021-05-25T12:00:55.546 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.483+0000 7ff88fe07540 10 client.4511.objecter _op_submit op 0x55fe526284f0 2021-05-25T12:00:55.546 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.483+0000 7ff88fe07540 20 client.4511.objecter _calc_target epoch 17 base exist @9 precalc_pgid 0 pgid 0.0 is_read 2021-05-25T12:00:55.546 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.483+0000 7ff88fe07540 20 client.4511.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2021-05-25T12:00:55.547 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.483+0000 7ff88fe07540 10 client.4511.objecter _calc_target raw pgid 9.d3a59201 -> actual 9.1 acting [0,2] primary 0 2021-05-25T12:00:55.547 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.483+0000 7ff88fe07540 20 client.4511.objecter _get_session s=0x55fe524c84a0 osd=0 3 2021-05-25T12:00:55.547 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.483+0000 7ff88fe07540 10 client.4511.objecter _op_submit oid exist '@9' '@9' [read 0~0] tid 16 osd.0 2021-05-25T12:00:55.547 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.483+0000 7ff88fe07540 20 client.4511.objecter get_session s=0x55fe524c84a0 osd=0 3 2021-05-25T12:00:55.547 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.483+0000 7ff88fe07540 15 client.4511.objecter _session_op_assign 0 16 2021-05-25T12:00:55.548 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.483+0000 7ff88fe07540 15 client.4511.objecter _send_op 16 to 9.1 on osd.0 2021-05-25T12:00:55.548 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.483+0000 7ff88fe07540 1 -- 172.21.15.181:0/4180131642 --> v1:172.21.15.102:6807/29973 -- osd_op(unknown.0.0:16 9.1 9:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected e17) v8 -- 0x55fe526289e0 con 0x55fe52610cd0 2021-05-25T12:00:55.548 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.483+0000 7ff88fe07540 20 client.4511.objecter put_session s=0x55fe524c84a0 osd=0 4 2021-05-25T12:00:55.548 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.483+0000 7ff88fe07540 5 client.4511.objecter 1 in flight 2021-05-25T12:00:55.548 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.483+0000 7ff88fe07540 10 client.4511.objecter _op_submit op 0x55fe52628f10 2021-05-25T12:00:55.549 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.483+0000 7ff88fe07540 20 client.4511.objecter _calc_target epoch 17 base noexist @9 precalc_pgid 0 pgid 0.0 is_read 2021-05-25T12:00:55.549 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.483+0000 7ff88fe07540 20 client.4511.objecter _calc_target target noexist @9 -> pgid 9.a7bbaa1 2021-05-25T12:00:55.549 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.483+0000 7ff88fe07540 10 client.4511.objecter _calc_target raw pgid 9.a7bbaa1 -> actual 9.1 acting [0,2] primary 0 2021-05-25T12:00:55.549 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.483+0000 7ff88fe07540 20 client.4511.objecter _get_session s=0x55fe524c84a0 osd=0 4 2021-05-25T12:00:55.550 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.483+0000 7ff88fe07540 10 client.4511.objecter _op_submit oid noexist '@9' '@9' [read 0~0] tid 17 osd.0 2021-05-25T12:00:55.550 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.483+0000 7ff88fe07540 20 client.4511.objecter get_session s=0x55fe524c84a0 osd=0 4 2021-05-25T12:00:55.550 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.483+0000 7ff88fe07540 15 client.4511.objecter _session_op_assign 0 17 2021-05-25T12:00:55.550 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.483+0000 7ff88fe07540 15 client.4511.objecter _send_op 17 to 9.1 on osd.0 2021-05-25T12:00:55.550 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.483+0000 7ff88fe07540 1 -- 172.21.15.181:0/4180131642 --> v1:172.21.15.102:6807/29973 -- osd_op(unknown.0.0:17 9.1 9:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected e17) v8 -- 0x55fe52629450 con 0x55fe52610cd0 2021-05-25T12:00:55.551 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.483+0000 7ff88fe07540 20 client.4511.objecter put_session s=0x55fe524c84a0 osd=0 5 2021-05-25T12:00:55.551 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.483+0000 7ff88fe07540 5 client.4511.objecter 2 in flight 2021-05-25T12:00:55.551 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.483+0000 7ff887a96700 1 -- 172.21.15.181:0/4180131642 <== osd.0 v1:172.21.15.102:6807/29973 16 ==== osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (unknown 4087291774 0 3741568377) 0x7ff880004550 con 0x55fe52610cd0 2021-05-25T12:00:55.551 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.483+0000 7ff887a96700 10 client.4511.objecter ms_dispatch 0x55fe524d4560 osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2021-05-25T12:00:55.551 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.483+0000 7ff887a96700 10 client.4511.objecter in handle_osd_op_reply 2021-05-25T12:00:55.552 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.483+0000 7ff887a96700 7 client.4511.objecter handle_osd_op_reply 16 ondisk uv 4 in 9.1 attempt 0 2021-05-25T12:00:55.552 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.483+0000 7ff887a96700 10 client.4511.objecter op 0 rval 0 len 5 2021-05-25T12:00:55.552 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.483+0000 7ff887a96700 15 client.4511.objecter handle_osd_op_reply completed tid 16 2021-05-25T12:00:55.552 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.483+0000 7ff887a96700 15 client.4511.objecter _finish_op 16 2021-05-25T12:00:55.552 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.483+0000 7ff887a96700 20 client.4511.objecter put_session s=0x55fe524c84a0 osd=0 5 2021-05-25T12:00:55.553 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.483+0000 7ff887a96700 15 client.4511.objecter _session_op_remove 0 16 2021-05-25T12:00:55.553 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.483+0000 7ff887a96700 5 client.4511.objecter 1 in flight 2021-05-25T12:00:55.553 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.483+0000 7ff887a96700 1 -- 172.21.15.181:0/4180131642 <== osd.0 v1:172.21.15.102:6807/29973 17 ==== osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (unknown 2073966446 0 0) 0x7ff880004550 con 0x55fe52610cd0 2021-05-25T12:00:55.553 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.483+0000 7ff887a96700 10 client.4511.objecter ms_dispatch 0x55fe524d4560 osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2021-05-25T12:00:55.553 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.483+0000 7ff887a96700 10 client.4511.objecter in handle_osd_op_reply 2021-05-25T12:00:55.554 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.483+0000 7ff887a96700 7 client.4511.objecter handle_osd_op_reply 17 ondisk uv 0 in 9.1 attempt 0 2021-05-25T12:00:55.554 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.483+0000 7ff887a96700 10 client.4511.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2021-05-25T12:00:55.554 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationYield2021-05-25T12:00:55.483+0000 7ff887a96700 10 client.4511.objecter op 0 rval 0 len 0 2021-05-25T12:00:55.554 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: (1 ms) 2021-05-25T12:00:55.555 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationCallback 2021-05-25T12:00:55.555 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.483+0000 7ff887a96700 15 client.4511.objecter handle_osd_op_reply completed tid 17 2021-05-25T12:00:55.555 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.483+0000 7ff887a96700 15 client.4511.objecter _finish_op 17 2021-05-25T12:00:55.555 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.483+0000 7ff887a96700 20 client.4511.objecter put_session s=0x55fe524c84a0 osd=0 4 2021-05-25T12:00:55.555 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.483+0000 7ff887a96700 15 client.4511.objecter _session_op_remove 0 17 2021-05-25T12:00:55.556 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.483+0000 7ff887a96700 5 client.4511.objecter 0 in flight 2021-05-25T12:00:55.556 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.483+0000 7ff88fe07540 20 librados: queue_aio_write 0x55fe524d0970 completion 0x55fe52615d40 write_seq 4 2021-05-25T12:00:55.556 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.483+0000 7ff88fe07540 10 client.4511.objecter _op_submit op 0x55fe52628f10 2021-05-25T12:00:55.556 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.483+0000 7ff88fe07540 20 client.4511.objecter _calc_target epoch 17 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2021-05-25T12:00:55.556 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.483+0000 7ff88fe07540 20 client.4511.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2021-05-25T12:00:55.557 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.483+0000 7ff88fe07540 10 client.4511.objecter _calc_target raw pgid 9.d3a59201 -> actual 9.1 acting [0,2] primary 0 2021-05-25T12:00:55.557 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.483+0000 7ff88fe07540 20 client.4511.objecter _get_session s=0x55fe524c84a0 osd=0 3 2021-05-25T12:00:55.557 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.483+0000 7ff88fe07540 10 client.4511.objecter _op_submit oid exist '@9' '@9' [writefull 0~5 in=5b] tid 18 osd.0 2021-05-25T12:00:55.557 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.483+0000 7ff88fe07540 20 client.4511.objecter get_session s=0x55fe524c84a0 osd=0 3 2021-05-25T12:00:55.558 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.483+0000 7ff88fe07540 15 client.4511.objecter _session_op_assign 0 18 2021-05-25T12:00:55.558 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.483+0000 7ff88fe07540 15 client.4511.objecter _send_op 18 to 9.1 on osd.0 2021-05-25T12:00:55.558 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.483+0000 7ff88fe07540 1 -- 172.21.15.181:0/4180131642 --> v1:172.21.15.102:6807/29973 -- osd_op(unknown.0.0:18 9.1 9:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected e17) v8 -- 0x55fe52615680 con 0x55fe52610cd0 2021-05-25T12:00:55.558 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.483+0000 7ff88fe07540 20 client.4511.objecter put_session s=0x55fe524c84a0 osd=0 4 2021-05-25T12:00:55.559 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.483+0000 7ff88fe07540 5 client.4511.objecter 1 in flight 2021-05-25T12:00:55.559 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.484+0000 7ff887a96700 1 -- 172.21.15.181:0/4180131642 <== osd.0 v1:172.21.15.102:6807/29973 18 ==== osd_op_reply(18 exist [writefull 0~5] v17'5 uv5 ondisk = 0) v8 ==== 149+0+0 (unknown 3870542929 0 0) 0x7ff880004550 con 0x55fe52610cd0 2021-05-25T12:00:55.559 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.484+0000 7ff887a96700 10 client.4511.objecter ms_dispatch 0x55fe524d4560 osd_op_reply(18 exist [writefull 0~5] v17'5 uv5 ondisk = 0) v8 2021-05-25T12:00:55.559 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationCallback2021-05-25T12:00:55.484+0000 7ff887a96700 10 client.4511.objecter in handle_osd_op_reply 2021-05-25T12:00:55.560 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.484+0000 7ff887a96700 7 client.4511.objecter handle_osd_op_reply 18 ondisk uv 5 in 9.1 attempt 0 2021-05-25T12:00:55.560 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: (1 ms) 2021-05-25T12:00:55.560 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationFuture 2021-05-25T12:00:55.560 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.484+0000 7ff887a96700 10 client.4511.objecter op 0 rval 0 len 0 2021-05-25T12:00:55.560 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.484+0000 7ff887a96700 15 client.4511.objecter handle_osd_op_reply completed tid 18 2021-05-25T12:00:55.561 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.484+0000 7ff887a96700 15 client.4511.objecter _finish_op 18 2021-05-25T12:00:55.561 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.484+0000 7ff887a96700 20 client.4511.objecter put_session s=0x55fe524c84a0 osd=0 4 2021-05-25T12:00:55.561 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.484+0000 7ff887a96700 15 client.4511.objecter _session_op_remove 0 18 2021-05-25T12:00:55.561 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.484+0000 7ff887a96700 5 client.4511.objecter 0 in flight 2021-05-25T12:00:55.561 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.484+0000 7ff887a96700 20 librados: complete_aio_write 0x55fe52615d40 2021-05-25T12:00:55.562 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.484+0000 7ff88fe07540 20 librados: queue_aio_write 0x55fe524d0970 completion 0x55fe52628de0 write_seq 5 2021-05-25T12:00:55.562 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.484+0000 7ff88fe07540 10 client.4511.objecter _op_submit op 0x55fe52628ef0 2021-05-25T12:00:55.562 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.484+0000 7ff88fe07540 20 client.4511.objecter _calc_target epoch 17 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2021-05-25T12:00:55.562 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.484+0000 7ff88fe07540 20 client.4511.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2021-05-25T12:00:55.562 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.484+0000 7ff88fe07540 10 client.4511.objecter _calc_target raw pgid 9.d3a59201 -> actual 9.1 acting [0,2] primary 0 2021-05-25T12:00:55.563 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.484+0000 7ff88fe07540 20 client.4511.objecter _get_session s=0x55fe524c84a0 osd=0 3 2021-05-25T12:00:55.563 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.484+0000 7ff88fe07540 10 client.4511.objecter _op_submit oid exist '@9' '@9' [writefull 0~5 in=5b] tid 19 osd.0 2021-05-25T12:00:55.563 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.484+0000 7ff88fe07540 20 client.4511.objecter get_session s=0x55fe524c84a0 osd=0 3 2021-05-25T12:00:55.563 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.484+0000 7ff88fe07540 15 client.4511.objecter _session_op_assign 0 19 2021-05-25T12:00:55.563 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.484+0000 7ff88fe07540 15 client.4511.objecter _send_op 19 to 9.1 on osd.0 2021-05-25T12:00:55.564 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.484+0000 7ff88fe07540 1 -- 172.21.15.181:0/4180131642 --> v1:172.21.15.102:6807/29973 -- osd_op(unknown.0.0:19 9.1 9:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected e17) v8 -- 0x55fe52615680 con 0x55fe52610cd0 2021-05-25T12:00:55.564 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.484+0000 7ff88fe07540 20 client.4511.objecter put_session s=0x55fe524c84a0 osd=0 4 2021-05-25T12:00:55.564 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.484+0000 7ff88fe07540 5 client.4511.objecter 1 in flight 2021-05-25T12:00:55.564 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.485+0000 7ff887a96700 1 -- 172.21.15.181:0/4180131642 <== osd.0 v1:172.21.15.102:6807/29973 19 ==== osd_op_reply(19 exist [writefull 0~5] v17'6 uv6 ondisk = 0) v8 ==== 149+0+0 (unknown 2066874413 0 0) 0x7ff880004550 con 0x55fe52610cd0 2021-05-25T12:00:55.564 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.485+0000 7ff887a96700 10 client.4511.objecter ms_dispatch 0x55fe524d4560 osd_op_reply(19 exist [writefull 0~5] v17'6 uv6 ondisk = 0) v8 2021-05-25T12:00:55.565 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.485+0000 7ff887a96700 10 client.4511.objecter in handle_osd_op_reply 2021-05-25T12:00:55.565 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.485+0000 7ff887a96700 7 client.4511.objecter handle_osd_op_reply 19 ondisk uv 6 in 9.1 attempt 0 2021-05-25T12:00:55.565 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.485+0000 7ff887a96700 10 client.4511.objecter op 0 rval 0 len 0 2021-05-25T12:00:55.565 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.485+0000 7ff887a96700 15 client.4511.objecter handle_osd_op_reply completed tid 19 2021-05-25T12:00:55.565 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.485+0000 7ff887a96700 15 client.4511.objecter _finish_op 19 2021-05-25T12:00:55.566 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.485+0000 7ff887a96700 20 client.4511.objecter put_session s=0x55fe524c84a0 osd=0 4 2021-05-25T12:00:55.566 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationFuture2021-05-25T12:00:55.485+0000 7ff887a96700 15 client.4511.objecter _session_op_remove 0 19 2021-05-25T12:00:55.566 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: (1 ms) 2021-05-25T12:00:55.566 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: [ RUN ] 2021-05-25T12:00:55.485+0000 7ff887a96700 5 client.4511.objecter 0 in flightAsioRados.AsyncWriteOperationYield 2021-05-25T12:00:55.567 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.567 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.485+0000 7ff887a96700 20 librados: complete_aio_write 0x55fe52628de0 2021-05-25T12:00:55.567 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.485+0000 7ff88fe07540 20 librados: queue_aio_write 0x55fe524d0970 completion 0x55fe52615ab0 write_seq 6 2021-05-25T12:00:55.567 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.485+0000 7ff88fe07540 10 client.4511.objecter _op_submit op 0x55fe526284f0 2021-05-25T12:00:55.567 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.485+0000 7ff88fe07540 20 client.4511.objecter _calc_target epoch 17 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2021-05-25T12:00:55.568 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.485+0000 7ff88fe07540 20 client.4511.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2021-05-25T12:00:55.568 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.485+0000 7ff88fe07540 10 client.4511.objecter _calc_target raw pgid 9.d3a59201 -> actual 9.1 acting [0,2] primary 0 2021-05-25T12:00:55.568 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.485+0000 7ff88fe07540 20 client.4511.objecter _get_session s=0x55fe524c84a0 osd=0 3 2021-05-25T12:00:55.568 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.485+0000 7ff88fe07540 10 client.4511.objecter _op_submit oid exist '@9' '@9' [writefull 0~5 in=5b] tid 20 osd.0 2021-05-25T12:00:55.568 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.485+0000 7ff88fe07540 20 client.4511.objecter get_session s=0x55fe524c84a0 osd=0 3 2021-05-25T12:00:55.569 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.485+0000 7ff88fe07540 15 client.4511.objecter _session_op_assign 0 20 2021-05-25T12:00:55.569 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.485+0000 7ff88fe07540 15 client.4511.objecter _send_op 20 to 9.1 on osd.0 2021-05-25T12:00:55.569 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.485+0000 7ff88fe07540 1 -- 172.21.15.181:0/4180131642 --> v1:172.21.15.102:6807/29973 -- osd_op(unknown.0.0:20 9.1 9:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected e17) v8 -- 0x55fe52615680 con 0x55fe52610cd0 2021-05-25T12:00:55.569 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.485+0000 7ff88fe07540 20 client.4511.objecter put_session s=0x55fe524c84a0 osd=0 4 2021-05-25T12:00:55.569 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.485+0000 7ff88fe07540 5 client.4511.objecter 1 in flight 2021-05-25T12:00:55.570 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.487+0000 7ff887a96700 1 -- 172.21.15.181:0/4180131642 <== osd.0 v1:172.21.15.102:6807/29973 20 ==== osd_op_reply(20 exist [writefull 0~5] v17'7 uv7 ondisk = 0) v8 ==== 149+0+0 (unknown 263371769 0 0) 0x7ff880004550 con 0x55fe52610cd0 2021-05-25T12:00:55.570 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.487+0000 7ff887a96700 10 client.4511.objecter ms_dispatch 0x55fe524d4560 osd_op_reply(20 exist [writefull 0~5] v17'7 uv7 ondisk = 0) v8 2021-05-25T12:00:55.570 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.487+0000 7ff887a96700 10 client.4511.objecter in handle_osd_op_reply[ OK ] AsioRados.AsyncWriteOperationYield 2021-05-25T12:00:55.570 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: (2 ms) 2021-05-25T12:00:55.570 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.487+0000 7ff887a96700 7 client.4511.objecter handle_osd_op_reply 20 ondisk uv 7 in 9.1 attempt 0 2021-05-25T12:00:55.571 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.487+0000 7ff887a96700 10 client.4511.objecter op 0 rval 0 len 0 2021-05-25T12:00:55.571 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.487+0000 7ff887a96700 15 client.4511.objecter handle_osd_op_reply completed tid 20 2021-05-25T12:00:55.571 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.487+0000 7ff887a96700 15 client.4511.objecter _finish_op 20 2021-05-25T12:00:55.571 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.487+0000 7ff887a96700 20 client.4511.objecter put_session s=0x55fe524c84a0 osd=0 4 2021-05-25T12:00:55.571 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.487+0000 7ff887a96700 15 client.4511.objecter _session_op_remove 0 20 2021-05-25T12:00:55.572 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.487+0000 7ff887a96700 5 client.4511.objecter 0 in flight 2021-05-25T12:00:55.572 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.487+0000 7ff887a96700 20 librados: complete_aio_write 0x55fe52615ab0 2021-05-25T12:00:55.572 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.487+0000 7ff88fe07540 10 librados: watch_flush enter 2021-05-25T12:00:55.572 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.487+0000 7ff88fe07540 10 librados: watch_flush exit 2021-05-25T12:00:55.572 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.487+0000 7ff88fe07540 10 client.4511.objecter close_session for osd.0 2021-05-25T12:00:55.573 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.487+0000 7ff88fe07540 1 -- 172.21.15.181:0/4180131642 >> v1:172.21.15.102:6807/29973 conn(0x55fe52610cd0 legacy=0x55fe526130c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:55.573 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.487+0000 7ff88fe07540 20 client.4511.objecter put_session s=0x55fe524c84a0 osd=0 1 2021-05-25T12:00:55.573 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.487+0000 7ff88fe07540 20 client.4511.objecter shutdown clearing up homeless session... 2021-05-25T12:00:55.573 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.487+0000 7ff88fe07540 10 client.4511.objecter successfully canceled tick 2021-05-25T12:00:55.573 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.487+0000 7ff88fe07540 1 -- 172.21.15.181:0/4180131642 >> v1:172.21.15.181:6812/29582 conn(0x7ff874031be0 legacy=0x7ff8740340a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:55.574 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.487+0000 7ff88fe07540 10 monclient: shutdown 2021-05-25T12:00:55.574 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.487+0000 7ff88fe07540 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:00:55.574 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.487+0000 7ff88fe07540 1 -- 172.21.15.181:0/4180131642 >> v1:172.21.15.102:6790/0 conn(0x55fe52566e20 legacy=0x55fe525f2fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:55.574 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.487+0000 7ff887a96700 1 -- 172.21.15.181:0/4180131642 reap_dead start 2021-05-25T12:00:55.574 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.487+0000 7ff88fe07540 1 -- 172.21.15.181:0/4180131642 shutdown_connections 2021-05-25T12:00:55.575 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.487+0000 7ff88fe07540 1 -- 172.21.15.181:0/4180131642 >> 172.21.15.181:0/4180131642 conn(0x55fe52564a30 msgr2=0x55fe525fe640 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:00:55.575 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.487+0000 7ff88fe07540 1 -- 172.21.15.181:0/4180131642 shutdown_connections 2021-05-25T12:00:55.575 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.487+0000 7ff88fe07540 1 -- 172.21.15.181:0/4180131642 wait complete. 2021-05-25T12:00:55.575 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.487+0000 7ff88fe07540 1 librados: shutdown 2021-05-25T12:00:55.576 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: [----------] 12 tests from AsioRados (17 ms total) 2021-05-25T12:00:55.576 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: 2021-05-25T12:00:55.576 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: [----------] Global test environment tear-down 2021-05-25T12:00:55.576 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: [==========] 12 tests from 1 test suite ran. (15765 ms total) 2021-05-25T12:00:55.576 INFO:tasks.workunit.client.0.smithi181.stdout: api_asio: [ PASSED ] 12 tests. 2021-05-25T12:00:55.713 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.711+0000 7fc367b94700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2021-05-25T12:00:55.714 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.711+0000 7fc367b94700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2021-05-25T12:00:55.714 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.711+0000 7fc367b94700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:00:55.714 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.711+0000 7fc367b94700 1 -- 172.21.15.102:0/987554529 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fc33c005060 con 0x7fc36012e330 2021-05-25T12:00:55.714 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.712+0000 7fc34dffb700 1 -- 172.21.15.102:0/987554529 <== mon.2 v1:172.21.15.102:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1336 (unknown 254746657 0 2310284968) 0x7fc35c012c50 con 0x7fc36012e330 2021-05-25T12:00:55.714 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.712+0000 7fc34dffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2021-05-25T12:00:55.715 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.712+0000 7fc34dffb700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:00:55.715 INFO:teuthology.orchestra.run.smithi102.stdout:{"election_epoch":18,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138303579357183","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"20ee91b6-8935-4b98-b868-85c03789ccc4","modified":"2021-05-25T11:59:11.011522Z","created":"2021-05-25T11:59:11.011522Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6790","nonce":0}]},"addr":"172.21.15.102:6790/0","public_addr":"172.21.15.102:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2021-05-25T12:00:55.717 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.714+0000 7fc367b94700 10 librados: watch_flush enter 2021-05-25T12:00:55.717 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.714+0000 7fc367b94700 10 librados: watch_flush exit 2021-05-25T12:00:55.717 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.714+0000 7fc367b94700 20 client.4829.objecter shutdown clearing up homeless session... 2021-05-25T12:00:55.717 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.714+0000 7fc367b94700 10 client.4829.objecter successfully canceled tick 2021-05-25T12:00:55.717 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.714+0000 7fc367b94700 1 -- 172.21.15.102:0/987554529 >> v1:172.21.15.181:6812/29582 conn(0x7fc3400343e0 legacy=0x7fc3400368a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:55.718 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.714+0000 7fc367b94700 10 monclient: shutdown 2021-05-25T12:00:55.718 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.714+0000 7fc367b94700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:00:55.718 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.714+0000 7fc367b94700 1 -- 172.21.15.102:0/987554529 >> v1:172.21.15.102:6790/0 conn(0x7fc36012e330 legacy=0x7fc360065aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:55.718 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.714+0000 7fc367b94700 1 -- 172.21.15.102:0/987554529 shutdown_connections 2021-05-25T12:00:55.719 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.714+0000 7fc367b94700 1 -- 172.21.15.102:0/987554529 >> 172.21.15.102:0/987554529 conn(0x7fc36009b370 msgr2=0x7fc3601292f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:00:55.719 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.714+0000 7fc367b94700 1 -- 172.21.15.102:0/987554529 shutdown_connections 2021-05-25T12:00:55.719 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.714+0000 7fc367b94700 1 -- 172.21.15.102:0/987554529 wait complete. 2021-05-25T12:00:55.719 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.714+0000 7fc367b94700 1 librados: shutdown 2021-05-25T12:00:55.728 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2021-05-25T12:00:55.729 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2021-05-25T12:00:55.853 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.852+0000 7f03f332e700 10 monclient: get_monmap_and_config 2021-05-25T12:00:55.853 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.852+0000 7f03f332e700 10 monclient: build_initial_monmap 2021-05-25T12:00:55.854 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.852+0000 7f03f332e700 10 monclient: monmap: 2021-05-25T12:00:55.854 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:00:55.854 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:55.854 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:00:55.852996+0000 2021-05-25T12:00:55.855 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:00:55.852996+0000 2021-05-25T12:00:55.855 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:00:55.855 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:55.855 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:00:55.855 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:00:55.856 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:00:55.856 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:55.870 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.852+0000 7f03f332e700 1 Processor -- start 2021-05-25T12:00:55.870 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.852+0000 7f03f332e700 1 -- start start 2021-05-25T12:00:55.870 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.852+0000 7f03f332e700 10 monclient: init 2021-05-25T12:00:55.871 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.853+0000 7f03f332e700 10 monclient: _reopen_session rank -1 2021-05-25T12:00:55.871 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.853+0000 7f03f332e700 10 monclient: _add_conns ranks=[1,0,2] 2021-05-25T12:00:55.871 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.853+0000 7f03f332e700 10 monclient(hunting): picked mon.noname-c con 0x7f03ec098af0 addr v1:172.21.15.181:6789/0 2021-05-25T12:00:55.871 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.853+0000 7f03f332e700 10 monclient(hunting): picked mon.noname-a con 0x7f03ec097f40 addr v1:172.21.15.102:6789/0 2021-05-25T12:00:55.871 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.853+0000 7f03f332e700 10 monclient(hunting): picked mon.noname-b con 0x7f03ec0983f0 addr v1:172.21.15.102:6790/0 2021-05-25T12:00:55.872 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.853+0000 7f03f332e700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03ec12c3b0 con 0x7f03ec097f40 2021-05-25T12:00:55.872 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.853+0000 7f03f332e700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03ec098ed0 con 0x7f03ec098af0 2021-05-25T12:00:55.872 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.853+0000 7f03f332e700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03ec09b0e0 con 0x7f03ec0983f0 2021-05-25T12:00:55.872 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.853+0000 7f03f332e700 10 monclient(hunting): _renew_subs 2021-05-25T12:00:55.873 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.853+0000 7f03f332e700 10 monclient(hunting): authenticate will time out at 856.450369s 2021-05-25T12:00:55.873 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.853+0000 7f03f1b2b700 1 -- >> v1:172.21.15.102:6789/0 conn(0x7f03ec097f40 legacy=0x7f03ec130a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:00:55.873 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.853+0000 7f03f2b2d700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f03ec0983f0 0x7f03ec1341a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:54408/0 (socket says 172.21.15.102:54408) 2021-05-25T12:00:55.873 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.853+0000 7f03f2b2d700 1 -- 172.21.15.102:0/892192759 learned_addr learned my addr 172.21.15.102:0/892192759 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:00:55.874 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.853+0000 7f03f132a700 1 -- 172.21.15.102:0/892192759 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2299922889 0 0) 0x7f03ec09b0e0 con 0x7f03ec0983f0 2021-05-25T12:00:55.874 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.853+0000 7f03f132a700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:55.874 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.853+0000 7f03f132a700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:55.874 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.853+0000 7f03f132a700 10 monclient(hunting): my global_id is 4868 2021-05-25T12:00:55.875 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.853+0000 7f03f132a700 1 -- 172.21.15.102:0/892192759 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03d4003270 con 0x7f03ec0983f0 2021-05-25T12:00:55.875 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.853+0000 7f03f132a700 1 -- 172.21.15.102:0/892192759 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3336680365 0 0) 0x7f03ec098ed0 con 0x7f03ec098af0 2021-05-25T12:00:55.875 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.853+0000 7f03f132a700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:55.875 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.853+0000 7f03f132a700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:55.875 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.853+0000 7f03f132a700 10 monclient(hunting): my global_id is 74554 2021-05-25T12:00:55.876 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.853+0000 7f03f132a700 1 -- 172.21.15.102:0/892192759 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03ec09b0e0 con 0x7f03ec098af0 2021-05-25T12:00:55.876 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.853+0000 7f03f132a700 1 -- 172.21.15.102:0/892192759 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2733646091 0 0) 0x7f03d4003270 con 0x7f03ec0983f0 2021-05-25T12:00:55.876 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.853+0000 7f03f132a700 1 -- 172.21.15.102:0/892192759 >> v1:172.21.15.181:6789/0 conn(0x7f03ec098af0 legacy=0x7f03ec12bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:55.876 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.853+0000 7f03f132a700 1 -- 172.21.15.102:0/892192759 >> v1:172.21.15.102:6789/0 conn(0x7f03ec097f40 legacy=0x7f03ec130a70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-05-25T12:00:55.877 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.853+0000 7f03f132a700 10 monclient: _finish_hunting 0 2021-05-25T12:00:55.877 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.853+0000 7f03f132a700 1 monclient: found mon.noname-b 2021-05-25T12:00:55.877 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.853+0000 7f03f132a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:00:55.877 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.853+0000 7f03f132a700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T12:00:55.877 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.853+0000 7f03f132a700 1 -- 172.21.15.102:0/892192759 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03ec090b10 con 0x7f03ec0983f0 2021-05-25T12:00:55.878 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.854+0000 7f03f132a700 10 monclient: _finish_auth 0 2021-05-25T12:00:55.878 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.854+0000 7f03f132a700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:00:55.878 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.854+0000 7f03f132a700 1 -- 172.21.15.102:0/892192759 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f03e8001990 con 0x7f03ec0983f0 2021-05-25T12:00:55.878 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.854+0000 7f03f132a700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:55.878 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.854+0000 7f03f132a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:00:55.879 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.854+0000 7f03f132a700 10 monclient: dump: 2021-05-25T12:00:55.879 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:55.879 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:55.880 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:55.880 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:55.880 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:55.880 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:55.881 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:55.881 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:55.881 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:55.881 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:55.882 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.854+0000 7f03f332e700 5 monclient: authenticate success, global_id 4868 2021-05-25T12:00:55.882 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.854+0000 7f03f332e700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:00:55.882 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.854+0000 7f03f132a700 1 -- 172.21.15.102:0/892192759 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f03e8001c70 con 0x7f03ec0983f0 2021-05-25T12:00:55.882 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.854+0000 7f03f132a700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:00:55.882 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.854+0000 7f03f132a700 1 -- 172.21.15.102:0/892192759 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f03e8002000 con 0x7f03ec0983f0 2021-05-25T12:00:55.883 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.854+0000 7f03f132a700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:55.883 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.854+0000 7f03f132a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:00:55.883 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.854+0000 7f03f132a700 10 monclient: dump: 2021-05-25T12:00:55.883 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:55.884 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:55.884 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:55.884 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:55.884 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:55.884 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:55.885 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:55.885 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:55.885 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:55.885 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:55.885 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.854+0000 7f03f332e700 10 monclient: get_monmap_and_config success 2021-05-25T12:00:55.886 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.854+0000 7f03f332e700 4 set_mon_vals no callback set 2021-05-25T12:00:55.886 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.854+0000 7f03f332e700 10 monclient: shutdown 2021-05-25T12:00:55.886 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.854+0000 7f03f332e700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:00:55.886 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.854+0000 7f03f332e700 1 -- 172.21.15.102:0/892192759 >> v1:172.21.15.102:6790/0 conn(0x7f03ec0983f0 legacy=0x7f03ec1341a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:55.887 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.855+0000 7f03f332e700 1 -- 172.21.15.102:0/892192759 shutdown_connections 2021-05-25T12:00:55.887 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.855+0000 7f03f332e700 1 -- 172.21.15.102:0/892192759 >> 172.21.15.102:0/892192759 conn(0x7f03ec09ea40 msgr2=0x7f03ec09ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:00:55.887 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.855+0000 7f03f332e700 1 -- 172.21.15.102:0/892192759 shutdown_connections 2021-05-25T12:00:55.887 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.855+0000 7f03f332e700 1 -- 172.21.15.102:0/892192759 wait complete. 2021-05-25T12:00:55.887 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.855+0000 7f03f332e700 10 monclient: build_initial_monmap 2021-05-25T12:00:55.888 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.855+0000 7f03f332e700 10 monclient: monmap: 2021-05-25T12:00:55.888 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:00:55.888 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:00:55.889 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:00:55.889 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:00:55.889 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:00:55.889 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:00:55.889 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:00:55.890 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:00:55.890 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:00:55.890 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:55.890 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.855+0000 7f03f332e700 1 librados: starting msgr at 2021-05-25T12:00:55.891 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.855+0000 7f03f332e700 1 librados: starting objecter 2021-05-25T12:00:55.891 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.855+0000 7f03f332e700 1 Processor -- start 2021-05-25T12:00:55.891 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.856+0000 7f03f332e700 1 -- start start 2021-05-25T12:00:55.891 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.856+0000 7f03f332e700 1 librados: setting wanted keys 2021-05-25T12:00:55.891 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.856+0000 7f03f332e700 1 librados: calling monclient init 2021-05-25T12:00:55.892 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.856+0000 7f03f332e700 10 monclient: init 2021-05-25T12:00:55.892 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.856+0000 7f03f332e700 10 monclient: _reopen_session rank -1 2021-05-25T12:00:55.892 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.856+0000 7f03f332e700 10 monclient: _add_conns ranks=[1,0,2] 2021-05-25T12:00:55.892 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.856+0000 7f03f332e700 10 monclient(hunting): picked mon.noname-b con 0x7f03ec097f40 addr v1:172.21.15.181:6789/0 2021-05-25T12:00:55.892 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.856+0000 7f03f332e700 10 monclient(hunting): picked mon.noname-a con 0x7f03ec0983f0 addr v1:172.21.15.102:6789/0 2021-05-25T12:00:55.893 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.856+0000 7f03f332e700 10 monclient(hunting): picked mon.noname-c con 0x7f03ec098af0 addr v1:172.21.15.102:6790/0 2021-05-25T12:00:55.893 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.856+0000 7f03f332e700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03ec09a430 con 0x7f03ec0983f0 2021-05-25T12:00:55.893 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.856+0000 7f03f332e700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03ec094e20 con 0x7f03ec097f40 2021-05-25T12:00:55.893 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.856+0000 7f03f332e700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03ec1155b0 con 0x7f03ec098af0 2021-05-25T12:00:55.894 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.856+0000 7f03f332e700 10 monclient(hunting): _renew_subs 2021-05-25T12:00:55.894 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.856+0000 7f03f332e700 10 monclient(hunting): authenticate will time out at 856.453493s 2021-05-25T12:00:55.894 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.856+0000 7f03f1b2b700 1 -- >> v1:172.21.15.102:6789/0 conn(0x7f03ec0983f0 legacy=0x7f03ec0946f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:00:55.894 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.856+0000 7f03f2b2d700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f03ec098af0 0x7f03ec143190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:54416/0 (socket says 172.21.15.102:54416) 2021-05-25T12:00:55.894 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.856+0000 7f03f2b2d700 1 -- 172.21.15.102:0/3698397661 learned_addr learned my addr 172.21.15.102:0/3698397661 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:00:55.895 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.856+0000 7f03da7fc700 10 client.?.objecter ms_handle_connect 0x7f03ec098af0 2021-05-25T12:00:55.895 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.856+0000 7f03da7fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:00:55.895 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.856+0000 7f03da7fc700 10 client.?.objecter ms_handle_connect 0x7f03ec097f40 2021-05-25T12:00:55.895 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.856+0000 7f03da7fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:00:55.896 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.856+0000 7f03da7fc700 1 -- 172.21.15.102:0/3698397661 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3498742515 0 0) 0x7f03ec1155b0 con 0x7f03ec098af0 2021-05-25T12:00:55.896 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.856+0000 7f03da7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:55.896 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.856+0000 7f03da7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:55.896 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.856+0000 7f03da7fc700 10 monclient(hunting): my global_id is 4871 2021-05-25T12:00:55.896 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.856+0000 7f03da7fc700 1 -- 172.21.15.102:0/3698397661 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03cc004c10 con 0x7f03ec098af0 2021-05-25T12:00:55.897 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.856+0000 7f03da7fc700 1 -- 172.21.15.102:0/3698397661 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1907907558 0 0) 0x7f03ec094e20 con 0x7f03ec097f40 2021-05-25T12:00:55.897 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.856+0000 7f03da7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:55.897 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.856+0000 7f03da7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:55.898 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.856+0000 7f03da7fc700 10 monclient(hunting): my global_id is 74557 2021-05-25T12:00:55.898 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.856+0000 7f03da7fc700 1 -- 172.21.15.102:0/3698397661 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03ec1155b0 con 0x7f03ec097f40 2021-05-25T12:00:55.898 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.857+0000 7f03da7fc700 1 -- 172.21.15.102:0/3698397661 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4292204500 0 0) 0x7f03cc004c10 con 0x7f03ec098af0 2021-05-25T12:00:55.898 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.857+0000 7f03da7fc700 1 -- 172.21.15.102:0/3698397661 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f03ec094e20 con 0x7f03ec098af0 2021-05-25T12:00:55.899 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.857+0000 7f03da7fc700 1 -- 172.21.15.102:0/3698397661 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f03e8001950 con 0x7f03ec098af0 2021-05-25T12:00:55.899 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.857+0000 7f03da7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:55.899 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.857+0000 7f03da7fc700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:00:55.899 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.857+0000 7f03da7fc700 10 monclient(hunting): dump: 2021-05-25T12:00:55.899 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:55.900 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:55.900 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:55.900 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:55.900 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:55.901 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:55.901 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:55.901 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:55.901 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:55.901 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:55.902 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.857+0000 7f03da7fc700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:00:55.902 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.857+0000 7f03da7fc700 1 -- 172.21.15.102:0/3698397661 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1738480331 0 0) 0x7f03ec1155b0 con 0x7f03ec097f40 2021-05-25T12:00:55.902 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.857+0000 7f03da7fc700 1 -- 172.21.15.102:0/3698397661 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f03cc004c10 con 0x7f03ec097f40 2021-05-25T12:00:55.902 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.857+0000 7f03da7fc700 1 -- 172.21.15.102:0/3698397661 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f03dc001990 con 0x7f03ec097f40 2021-05-25T12:00:55.903 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.857+0000 7f03da7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:55.903 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.857+0000 7f03da7fc700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2021-05-25T12:00:55.903 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.857+0000 7f03da7fc700 10 monclient(hunting): dump: 2021-05-25T12:00:55.903 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:55.904 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:55.904 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:55.904 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:55.904 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:55.904 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:55.905 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:55.905 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:55.905 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:55.905 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:55.906 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.857+0000 7f03da7fc700 1 -- 172.21.15.102:0/3698397661 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 391533781 0 0) 0x7f03cc004c10 con 0x7f03ec097f40 2021-05-25T12:00:55.906 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.857+0000 7f03da7fc700 1 -- 172.21.15.102:0/3698397661 >> v1:172.21.15.102:6790/0 conn(0x7f03ec098af0 legacy=0x7f03ec143190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:55.906 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.857+0000 7f03da7fc700 1 -- 172.21.15.102:0/3698397661 >> v1:172.21.15.102:6789/0 conn(0x7f03ec0983f0 legacy=0x7f03ec0946f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-05-25T12:00:55.906 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.857+0000 7f03da7fc700 10 monclient: _finish_hunting 0 2021-05-25T12:00:55.906 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.857+0000 7f03da7fc700 1 monclient: found mon.b 2021-05-25T12:00:55.907 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.857+0000 7f03da7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:00:55.907 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.857+0000 7f03da7fc700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T12:00:55.907 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.857+0000 7f03da7fc700 1 -- 172.21.15.102:0/3698397661 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03ec115850 con 0x7f03ec097f40 2021-05-25T12:00:55.907 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.857+0000 7f03da7fc700 10 monclient: _finish_auth 0 2021-05-25T12:00:55.908 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.857+0000 7f03da7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:00:55.908 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.857+0000 7f03f332e700 5 monclient: authenticate success, global_id 74557 2021-05-25T12:00:55.908 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.857+0000 7f03f332e700 10 monclient: _renew_subs 2021-05-25T12:00:55.908 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.857+0000 7f03f332e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T12:00:55.908 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.857+0000 7f03f332e700 1 -- 172.21.15.102:0/3698397661 --> v1:172.21.15.181:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03ec0965a0 con 0x7f03ec097f40 2021-05-25T12:00:55.909 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.857+0000 7f03f332e700 10 client.74557.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:00:55.909 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.857+0000 7f03f332e700 10 monclient: _renew_subs 2021-05-25T12:00:55.909 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.857+0000 7f03f332e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T12:00:55.909 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.857+0000 7f03f332e700 1 -- 172.21.15.102:0/3698397661 --> v1:172.21.15.181:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f03ec115bd0 con 0x7f03ec097f40 2021-05-25T12:00:55.909 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.857+0000 7f03f332e700 1 librados: init done 2021-05-25T12:00:55.910 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.857+0000 7f03da7fc700 1 -- 172.21.15.102:0/3698397661 <== mon.1 v1:172.21.15.181:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f03dc001f20 con 0x7f03ec097f40 2021-05-25T12:00:55.910 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.857+0000 7f03da7fc700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:00:55.910 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.857+0000 7f03da7fc700 1 -- 172.21.15.102:0/3698397661 <== mon.1 v1:172.21.15.181:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f03dc002230 con 0x7f03ec097f40 2021-05-25T12:00:55.910 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.857+0000 7f03da7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:55.911 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.857+0000 7f03f0b29700 4 set_mon_vals no callback set 2021-05-25T12:00:55.911 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.857+0000 7f03da7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T12:00:55.911 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.857+0000 7f03da7fc700 10 monclient: dump: 2021-05-25T12:00:55.911 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:55.911 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:55.912 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:55.912 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:55.912 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:55.912 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:55.912 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:55.913 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:55.913 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:55.913 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:55.913 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.858+0000 7f03da7fc700 1 -- 172.21.15.102:0/3698397661 <== mon.1 v1:172.21.15.181:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f03dc002410 con 0x7f03ec097f40 2021-05-25T12:00:55.914 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.858+0000 7f03f332e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:00:55.914 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.858+0000 7f03f332e700 1 -- 172.21.15.102:0/3698397661 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f03d00065a0 con 0x7f03d0002840 2021-05-25T12:00:55.914 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.858+0000 7f03da7fc700 1 -- 172.21.15.102:0/3698397661 <== mon.1 v1:172.21.15.181:6789/0 8 ==== osd_map(17..17 src has 1..17) v4 ==== 11815+0+0 (unknown 4088861535 0 0) 0x7f03dc03c730 con 0x7f03ec097f40 2021-05-25T12:00:55.914 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.858+0000 7f03da7fc700 10 client.74557.objecter ms_dispatch 0x7f03ec001550 osd_map(17..17 src has 1..17) v4 2021-05-25T12:00:55.914 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.858+0000 7f03da7fc700 3 client.74557.objecter handle_osd_map got epochs [17,17] > 0 2021-05-25T12:00:55.915 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.858+0000 7f03da7fc700 3 client.74557.objecter handle_osd_map decoding full epoch 17 2021-05-25T12:00:55.915 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.858+0000 7f03da7fc700 20 client.74557.objecter dump_active .. 0 homeless 2021-05-25T12:00:55.915 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.858+0000 7f03da7fc700 10 client.74557.objecter ms_handle_connect 0x7f03d0002840 2021-05-25T12:00:55.915 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.858+0000 7f03da7fc700 10 client.74557.objecter resend_mon_ops 2021-05-25T12:00:55.916 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.858+0000 7f03f2b2d700 10 monclient: get_auth_request con 0x7f03cc033280 auth_method 0 2021-05-25T12:00:55.916 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.858+0000 7f03da7fc700 1 -- 172.21.15.102:0/3698397661 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3639182430 0 0) 0x7f03d00065a0 con 0x7f03d0002840 2021-05-25T12:00:55.916 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.858+0000 7f03da7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:55.916 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.858+0000 7f03da7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:55.916 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.858+0000 7f03da7fc700 10 monclient(hunting): my global_id is 4874 2021-05-25T12:00:55.917 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.858+0000 7f03da7fc700 1 -- 172.21.15.102:0/3698397661 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03cc004c10 con 0x7f03d0002840 2021-05-25T12:00:55.917 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.858+0000 7f03da7fc700 10 client.74557.objecter ms_handle_connect 0x7f03cc033280 2021-05-25T12:00:55.917 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.858+0000 7f03da7fc700 1 -- 172.21.15.102:0/3698397661 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2726712919 0 0) 0x7f03cc004c10 con 0x7f03d0002840 2021-05-25T12:00:55.917 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.858+0000 7f03da7fc700 1 -- 172.21.15.102:0/3698397661 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f03d00065a0 con 0x7f03d0002840 2021-05-25T12:00:55.917 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.858+0000 7f03da7fc700 1 -- 172.21.15.102:0/3698397661 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f03e0003d30 con 0x7f03d0002840 2021-05-25T12:00:55.918 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.858+0000 7f03da7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:55.918 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.858+0000 7f03da7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:00:55.918 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.858+0000 7f03da7fc700 10 monclient: dump: 2021-05-25T12:00:55.918 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:55.919 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:55.919 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:55.919 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:55.919 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:55.919 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:55.920 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:55.920 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:55.920 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:55.920 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:55.920 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.858+0000 7f03da7fc700 1 -- 172.21.15.102:0/3698397661 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4186448442 0 0) 0x7f03d00065a0 con 0x7f03d0002840 2021-05-25T12:00:55.921 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.858+0000 7f03da7fc700 1 -- 172.21.15.102:0/3698397661 --> v1:172.21.15.102:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f03d0009e40 con 0x7f03d0002840 2021-05-25T12:00:55.921 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.859+0000 7f03da7fc700 1 -- 172.21.15.102:0/3698397661 <== mon.2 v1:172.21.15.102:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7f03e00051f0 con 0x7f03d0002840 2021-05-25T12:00:55.921 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.859+0000 7f03da7fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:00:55.921 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.859+0000 7f03da7fc700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:00:55.921 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.859+0000 7f03da7fc700 1 -- 172.21.15.102:0/3698397661 >> v1:172.21.15.102:6790/0 conn(0x7f03d0002840 legacy=0x7f03d0004ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:55.922 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.859+0000 7f03da7fc700 1 -- 172.21.15.102:0/3698397661 >> v1:172.21.15.102:6790/0 conn(0x7f03d0002840 legacy=0x7f03d0004ce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:00:55.922 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.866+0000 7f03f332e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:00:55.922 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.866+0000 7f03f332e700 1 -- 172.21.15.102:0/3698397661 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f03d0009b00 con 0x7f03d0008dc0 2021-05-25T12:00:55.922 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.866+0000 7f03da7fc700 10 client.74557.objecter ms_handle_connect 0x7f03d0008dc0 2021-05-25T12:00:55.922 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.866+0000 7f03da7fc700 10 client.74557.objecter resend_mon_ops 2021-05-25T12:00:55.923 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.866+0000 7f03da7fc700 1 -- 172.21.15.102:0/3698397661 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 606052948 0 0) 0x7f03d0009b00 con 0x7f03d0008dc0 2021-05-25T12:00:55.923 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.866+0000 7f03da7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:55.923 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.866+0000 7f03da7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:55.923 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.866+0000 7f03da7fc700 10 monclient(hunting): my global_id is 4877 2021-05-25T12:00:55.924 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.866+0000 7f03da7fc700 1 -- 172.21.15.102:0/3698397661 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03d00065a0 con 0x7f03d0008dc0 2021-05-25T12:00:55.924 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.866+0000 7f03da7fc700 1 -- 172.21.15.102:0/3698397661 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2326767472 0 0) 0x7f03d00065a0 con 0x7f03d0008dc0 2021-05-25T12:00:55.924 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.866+0000 7f03da7fc700 1 -- 172.21.15.102:0/3698397661 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f03d0009b00 con 0x7f03d0008dc0 2021-05-25T12:00:55.924 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.866+0000 7f03da7fc700 1 -- 172.21.15.102:0/3698397661 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f03e00043e0 con 0x7f03d0008dc0 2021-05-25T12:00:55.924 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.866+0000 7f03da7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:55.925 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.866+0000 7f03da7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:00:55.925 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.866+0000 7f03da7fc700 10 monclient: dump: 2021-05-25T12:00:55.925 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:55.925 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:55.926 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:55.926 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:55.926 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:55.926 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:55.926 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:55.927 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:55.927 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:55.927 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:55.927 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.866+0000 7f03da7fc700 1 -- 172.21.15.102:0/3698397661 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 885405658 0 0) 0x7f03d0009b00 con 0x7f03d0008dc0 2021-05-25T12:00:55.928 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.866+0000 7f03da7fc700 1 -- 172.21.15.102:0/3698397661 --> v1:172.21.15.102:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f03d0008740 con 0x7f03d0008dc0 2021-05-25T12:00:55.928 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.866+0000 7f03da7fc700 1 -- 172.21.15.102:0/3698397661 <== mon.2 v1:172.21.15.102:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4102 (unknown 0 0 179326346) 0x7f03e00047e0 con 0x7f03d0008dc0 2021-05-25T12:00:55.928 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.866+0000 7f03da7fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:00:55.928 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.866+0000 7f03da7fc700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:00:55.928 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.866+0000 7f03da7fc700 1 -- 172.21.15.102:0/3698397661 >> v1:172.21.15.102:6790/0 conn(0x7f03d0008dc0 legacy=0x7f03d0009250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:55.929 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.866+0000 7f03da7fc700 1 -- 172.21.15.102:0/3698397661 >> v1:172.21.15.102:6790/0 conn(0x7f03d0008dc0 legacy=0x7f03d0009250 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:00:55.929 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:00:55.929 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:00:55.930 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:00:55.930 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "leader", 2021-05-25T12:00:55.930 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 18, 2021-05-25T12:00:55.930 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:00:55.931 INFO:teuthology.orchestra.run.smithi102.stdout: 1, 2021-05-25T12:00:55.931 INFO:teuthology.orchestra.run.smithi102.stdout: 2 2021-05-25T12:00:55.931 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:00:55.931 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 0, 2021-05-25T12:00:55.932 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:00:55.932 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:00:55.932 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:00:55.932 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:00:55.932 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:00:55.933 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:00:55.933 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:00:55.933 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:00:55.933 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:00:55.934 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:00:55.934 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:00:55.934 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:00:55.934 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:00:55.935 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:00:55.935 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:00:55.935 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:00:55.935 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:00:55.936 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:00:55.936 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:00:55.936 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:00:55.936 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:00:55.936 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:00:55.937 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:00:55.937 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:00:55.937 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:00:55.937 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:55.937 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:00:55.938 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:00:55.938 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:00:55.938 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:00:55.938 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:00:55.939 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:00:55.939 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:00:55.939 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:00:55.939 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:00:55.939 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:00:55.940 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:00:55.940 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:00:55.940 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:00:55.940 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:00:55.940 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:00:55.941 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:00:55.941 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:00:55.941 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:00:55.941 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:00:55.942 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:00:55.942 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:00:55.942 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:00:55.942 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:00:55.943 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:00:55.943 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:00:55.943 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:00:55.943 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:55.944 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:00:55.944 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:55.944 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:00:55.944 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:00:55.944 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:00:55.945 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:00:55.945 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:55.945 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:00:55.945 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:00:55.945 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:00:55.946 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:55.946 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:00:55.946 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:55.946 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:00:55.946 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:00:55.947 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:00:55.947 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:00:55.947 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:00:55.947 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:55.947 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:55.948 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:00:55.948 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:00:55.948 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:00:55.948 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:00:55.949 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:55.949 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:00:55.949 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:00:55.949 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:00:55.949 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:55.950 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:00:55.950 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:55.950 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:00:55.950 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:00:55.951 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:00:55.951 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:00:55.951 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:00:55.951 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:55.951 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:55.952 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:00:55.952 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:00:55.952 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:00:55.953 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:00:55.953 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:55.953 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:00:55.953 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:00:55.953 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:00:55.954 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:55.954 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:00:55.954 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:55.954 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:00:55.955 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:00:55.955 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:00:55.955 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:00:55.955 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:00:55.956 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:55.956 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:00:55.956 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:55.956 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:00:55.957 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:00:55.957 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:55.957 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:00:55.957 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:00:55.958 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:00:55.958 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:55.958 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:00:55.959 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": [ 2021-05-25T12:00:55.959 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:55.959 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:00:55.959 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:00:55.960 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 6 2021-05-25T12:00:55.960 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:55.960 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:00:55.961 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:00:55.961 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:55.961 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:00:55.961 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:00:55.961 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 22 2021-05-25T12:00:55.962 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:55.962 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:00:55.962 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": [ 2021-05-25T12:00:55.962 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:55.963 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:00:55.963 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:00:55.963 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:00:55.963 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:55.963 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:00:55.964 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:55.964 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:00:55.964 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:00:55.966 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.867+0000 7f03f332e700 10 librados: watch_flush enter 2021-05-25T12:00:55.967 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.867+0000 7f03f332e700 10 librados: watch_flush exit 2021-05-25T12:00:55.967 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.867+0000 7f03f332e700 20 client.74557.objecter shutdown clearing up homeless session... 2021-05-25T12:00:55.967 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.867+0000 7f03f332e700 10 client.74557.objecter successfully canceled tick 2021-05-25T12:00:55.967 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.867+0000 7f03f332e700 1 -- 172.21.15.102:0/3698397661 >> v1:172.21.15.181:6812/29582 conn(0x7f03cc033280 legacy=0x7f03cc035740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:55.968 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.867+0000 7f03f332e700 10 monclient: shutdown 2021-05-25T12:00:55.968 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.867+0000 7f03f332e700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:00:55.968 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.867+0000 7f03f332e700 1 -- 172.21.15.102:0/3698397661 >> v1:172.21.15.181:6789/0 conn(0x7f03ec097f40 legacy=0x7f03ec093fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:55.968 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.867+0000 7f03f2b2d700 1 -- 172.21.15.102:0/3698397661 reap_dead start 2021-05-25T12:00:55.969 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.867+0000 7f03f2b2d700 1 -- 172.21.15.102:0/3698397661 reap_dead start 2021-05-25T12:00:55.969 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.867+0000 7f03f332e700 1 -- 172.21.15.102:0/3698397661 shutdown_connections 2021-05-25T12:00:55.969 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.867+0000 7f03f332e700 1 -- 172.21.15.102:0/3698397661 >> 172.21.15.102:0/3698397661 conn(0x7f03ec09ea40 msgr2=0x7f03ec09c870 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:00:55.970 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.867+0000 7f03f332e700 1 -- 172.21.15.102:0/3698397661 shutdown_connections 2021-05-25T12:00:55.970 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.867+0000 7f03f332e700 1 -- 172.21.15.102:0/3698397661 wait complete. 2021-05-25T12:00:55.970 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:55.867+0000 7f03f332e700 1 librados: shutdown 2021-05-25T12:00:55.971 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2021-05-25T12:00:56.100 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.098+0000 7ff3da968700 10 monclient: get_monmap_and_config 2021-05-25T12:00:56.100 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.098+0000 7ff3da968700 10 monclient: build_initial_monmap 2021-05-25T12:00:56.100 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.098+0000 7ff3da968700 10 monclient: monmap: 2021-05-25T12:00:56.100 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:00:56.101 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:56.101 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:00:56.099431+0000 2021-05-25T12:00:56.101 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:00:56.099431+0000 2021-05-25T12:00:56.101 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:00:56.102 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:56.103 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:00:56.103 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:00:56.104 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:00:56.105 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:56.134 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.099+0000 7ff3da968700 1 Processor -- start 2021-05-25T12:00:56.135 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.099+0000 7ff3da968700 1 -- start start 2021-05-25T12:00:56.135 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.099+0000 7ff3da968700 10 monclient: init 2021-05-25T12:00:56.135 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.099+0000 7ff3da968700 10 monclient: _reopen_session rank -1 2021-05-25T12:00:56.136 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.099+0000 7ff3da968700 10 monclient: _add_conns ranks=[0,2,1] 2021-05-25T12:00:56.136 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.099+0000 7ff3da968700 10 monclient(hunting): picked mon.noname-a con 0x7ff3d4063840 addr v1:172.21.15.102:6789/0 2021-05-25T12:00:56.136 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.099+0000 7ff3da968700 10 monclient(hunting): picked mon.noname-b con 0x7ff3d4094da0 addr v1:172.21.15.102:6790/0 2021-05-25T12:00:56.136 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.099+0000 7ff3da968700 10 monclient(hunting): picked mon.noname-c con 0x7ff3d41323c0 addr v1:172.21.15.181:6789/0 2021-05-25T12:00:56.137 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.099+0000 7ff3da968700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3d4063fb0 con 0x7ff3d4063840 2021-05-25T12:00:56.137 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.100+0000 7ff3da968700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3d40982f0 con 0x7ff3d41323c0 2021-05-25T12:00:56.137 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.100+0000 7ff3da968700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3d4064510 con 0x7ff3d4094da0 2021-05-25T12:00:56.137 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.100+0000 7ff3da968700 10 monclient(hunting): _renew_subs 2021-05-25T12:00:56.138 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.100+0000 7ff3da968700 10 monclient(hunting): authenticate will time out at 856.697029s 2021-05-25T12:00:56.138 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.100+0000 7ff3d3fff700 1 -- >> v1:172.21.15.102:6789/0 conn(0x7ff3d4063840 legacy=0x7ff3d40974d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:00:56.138 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.100+0000 7ff3d37fe700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7ff3d4094da0 0x7ff3d4097be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:54426/0 (socket says 172.21.15.102:54426) 2021-05-25T12:00:56.138 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.100+0000 7ff3d37fe700 1 -- 172.21.15.102:0/1671582004 learned_addr learned my addr 172.21.15.102:0/1671582004 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:00:56.138 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.100+0000 7ff3d2ffd700 1 -- 172.21.15.102:0/1671582004 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4113058309 0 0) 0x7ff3d4064510 con 0x7ff3d4094da0 2021-05-25T12:00:56.139 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.100+0000 7ff3d2ffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:56.139 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.100+0000 7ff3d2ffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:56.139 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.100+0000 7ff3d2ffd700 10 monclient(hunting): my global_id is 4922 2021-05-25T12:00:56.139 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.100+0000 7ff3d2ffd700 1 -- 172.21.15.102:0/1671582004 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3b8003270 con 0x7ff3d4094da0 2021-05-25T12:00:56.140 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.100+0000 7ff3d2ffd700 1 -- 172.21.15.102:0/1671582004 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 838389944 0 0) 0x7ff3d40982f0 con 0x7ff3d41323c0 2021-05-25T12:00:56.140 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.100+0000 7ff3d2ffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:56.140 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.100+0000 7ff3d2ffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:56.140 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.100+0000 7ff3d2ffd700 10 monclient(hunting): my global_id is 74608 2021-05-25T12:00:56.140 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.100+0000 7ff3d2ffd700 1 -- 172.21.15.102:0/1671582004 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3d4064510 con 0x7ff3d41323c0 2021-05-25T12:00:56.141 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.100+0000 7ff3d2ffd700 1 -- 172.21.15.102:0/1671582004 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1534055946 0 0) 0x7ff3d4064510 con 0x7ff3d41323c0 2021-05-25T12:00:56.141 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.100+0000 7ff3d2ffd700 1 -- 172.21.15.102:0/1671582004 >> v1:172.21.15.102:6790/0 conn(0x7ff3d4094da0 legacy=0x7ff3d4097be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:56.141 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.100+0000 7ff3d2ffd700 1 -- 172.21.15.102:0/1671582004 >> v1:172.21.15.102:6789/0 conn(0x7ff3d4063840 legacy=0x7ff3d40974d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-05-25T12:00:56.141 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.100+0000 7ff3d2ffd700 10 monclient: _finish_hunting 0 2021-05-25T12:00:56.141 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.100+0000 7ff3d2ffd700 1 monclient: found mon.noname-c 2021-05-25T12:00:56.142 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.100+0000 7ff3d2ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:00:56.142 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.100+0000 7ff3d2ffd700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.181:6789/0 2021-05-25T12:00:56.142 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.100+0000 7ff3d2ffd700 1 -- 172.21.15.102:0/1671582004 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3d40647b0 con 0x7ff3d41323c0 2021-05-25T12:00:56.142 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.100+0000 7ff3d2ffd700 10 monclient: _finish_auth 0 2021-05-25T12:00:56.143 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.100+0000 7ff3d2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:00:56.143 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.100+0000 7ff3d2ffd700 1 -- 172.21.15.102:0/1671582004 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7ff3c8001970 con 0x7ff3d41323c0 2021-05-25T12:00:56.143 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.100+0000 7ff3d2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:56.143 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.100+0000 7ff3d2ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:00:56.143 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.100+0000 7ff3d2ffd700 10 monclient: dump: 2021-05-25T12:00:56.144 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:56.144 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:56.144 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:56.144 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:56.144 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:56.145 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:56.145 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:56.145 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:56.145 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:56.146 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:56.146 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.100+0000 7ff3da968700 5 monclient: authenticate success, global_id 74608 2021-05-25T12:00:56.146 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.100+0000 7ff3da968700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:00:56.146 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.100+0000 7ff3d2ffd700 1 -- 172.21.15.102:0/1671582004 <== mon.1 v1:172.21.15.181:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff3c8001c20 con 0x7ff3d41323c0 2021-05-25T12:00:56.146 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.100+0000 7ff3d2ffd700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:00:56.147 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.100+0000 7ff3da968700 10 monclient: get_monmap_and_config success 2021-05-25T12:00:56.147 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.100+0000 7ff3d2ffd700 1 -- 172.21.15.102:0/1671582004 <== mon.1 v1:172.21.15.181:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7ff3c8001fb0 con 0x7ff3d41323c0 2021-05-25T12:00:56.147 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.100+0000 7ff3d2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:56.147 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.100+0000 7ff3da968700 4 set_mon_vals no callback set 2021-05-25T12:00:56.148 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.100+0000 7ff3d2ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T12:00:56.148 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.100+0000 7ff3d2ffd700 10 monclient: dump: 2021-05-25T12:00:56.148 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:56.148 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:56.148 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:56.149 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:56.149 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:56.149 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:56.149 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:56.150 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:56.150 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:56.150 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:56.150 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.101+0000 7ff3da968700 10 monclient: shutdown 2021-05-25T12:00:56.150 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.101+0000 7ff3da968700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:00:56.151 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.101+0000 7ff3da968700 1 -- 172.21.15.102:0/1671582004 >> v1:172.21.15.181:6789/0 conn(0x7ff3d41323c0 legacy=0x7ff3d41347b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:56.151 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.101+0000 7ff3da968700 1 -- 172.21.15.102:0/1671582004 shutdown_connections 2021-05-25T12:00:56.151 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.101+0000 7ff3da968700 1 -- 172.21.15.102:0/1671582004 >> 172.21.15.102:0/1671582004 conn(0x7ff3d409cb30 msgr2=0x7ff3d409cf40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:00:56.151 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.101+0000 7ff3da968700 1 -- 172.21.15.102:0/1671582004 shutdown_connections 2021-05-25T12:00:56.151 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.101+0000 7ff3da968700 1 -- 172.21.15.102:0/1671582004 wait complete. 2021-05-25T12:00:56.152 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.101+0000 7ff3da968700 10 monclient: build_initial_monmap 2021-05-25T12:00:56.152 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.101+0000 7ff3da968700 10 monclient: monmap: 2021-05-25T12:00:56.152 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:00:56.152 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:00:56.153 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:00:56.153 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:00:56.153 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:00:56.153 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:00:56.153 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:00:56.154 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:00:56.154 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:00:56.154 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:56.154 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.102+0000 7ff3da968700 1 librados: starting msgr at 2021-05-25T12:00:56.155 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.102+0000 7ff3da968700 1 librados: starting objecter 2021-05-25T12:00:56.155 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.102+0000 7ff3da968700 1 Processor -- start 2021-05-25T12:00:56.155 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.102+0000 7ff3da968700 1 -- start start 2021-05-25T12:00:56.155 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.102+0000 7ff3da968700 1 librados: setting wanted keys 2021-05-25T12:00:56.155 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.102+0000 7ff3da968700 1 librados: calling monclient init 2021-05-25T12:00:56.156 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.102+0000 7ff3da968700 10 monclient: init 2021-05-25T12:00:56.156 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.102+0000 7ff3da968700 10 monclient: _reopen_session rank -1 2021-05-25T12:00:56.156 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.102+0000 7ff3da968700 10 monclient: _add_conns ranks=[0,1,2] 2021-05-25T12:00:56.156 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.102+0000 7ff3da968700 10 monclient(hunting): picked mon.noname-a con 0x7ff3d4063840 addr v1:172.21.15.102:6789/0 2021-05-25T12:00:56.157 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.102+0000 7ff3da968700 10 monclient(hunting): picked mon.noname-b con 0x7ff3d4094da0 addr v1:172.21.15.181:6789/0 2021-05-25T12:00:56.157 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.102+0000 7ff3da968700 10 monclient(hunting): picked mon.noname-c con 0x7ff3d41323c0 addr v1:172.21.15.102:6790/0 2021-05-25T12:00:56.157 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.102+0000 7ff3da968700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3d412f020 con 0x7ff3d4063840 2021-05-25T12:00:56.157 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.102+0000 7ff3da968700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3d4131a00 con 0x7ff3d4094da0 2021-05-25T12:00:56.157 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.102+0000 7ff3da968700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3d4131ca0 con 0x7ff3d41323c0 2021-05-25T12:00:56.158 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.102+0000 7ff3da968700 10 monclient(hunting): _renew_subs 2021-05-25T12:00:56.158 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.102+0000 7ff3d3fff700 1 -- >> v1:172.21.15.102:6789/0 conn(0x7ff3d4063840 legacy=0x7ff3d412e200 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:00:56.158 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.102+0000 7ff3da968700 10 monclient(hunting): authenticate will time out at 856.699952s 2021-05-25T12:00:56.158 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.103+0000 7ff3d8f05700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7ff3d41323c0 0x7ff3d41433a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:54434/0 (socket says 172.21.15.102:54434) 2021-05-25T12:00:56.159 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.103+0000 7ff3d8f05700 1 -- 172.21.15.102:0/947090450 learned_addr learned my addr 172.21.15.102:0/947090450 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:00:56.159 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.103+0000 7ff3bffff700 10 client.?.objecter ms_handle_connect 0x7ff3d41323c0 2021-05-25T12:00:56.159 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.103+0000 7ff3bffff700 10 client.?.objecter resend_mon_ops 2021-05-25T12:00:56.159 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.103+0000 7ff3bffff700 10 client.?.objecter ms_handle_connect 0x7ff3d4094da0 2021-05-25T12:00:56.159 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.103+0000 7ff3bffff700 10 client.?.objecter resend_mon_ops 2021-05-25T12:00:56.160 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.103+0000 7ff3bffff700 1 -- 172.21.15.102:0/947090450 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 994680322 0 0) 0x7ff3d4131ca0 con 0x7ff3d41323c0 2021-05-25T12:00:56.160 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.103+0000 7ff3bffff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:56.160 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.103+0000 7ff3bffff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:56.160 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.103+0000 7ff3bffff700 10 monclient(hunting): my global_id is 4925 2021-05-25T12:00:56.161 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.103+0000 7ff3bffff700 1 -- 172.21.15.102:0/947090450 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3b4004c10 con 0x7ff3d41323c0 2021-05-25T12:00:56.161 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.103+0000 7ff3bffff700 1 -- 172.21.15.102:0/947090450 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3697814544 0 0) 0x7ff3d4131a00 con 0x7ff3d4094da0 2021-05-25T12:00:56.161 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.103+0000 7ff3bffff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:56.161 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.103+0000 7ff3bffff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:56.161 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.103+0000 7ff3bffff700 10 monclient(hunting): my global_id is 74611 2021-05-25T12:00:56.162 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.103+0000 7ff3bffff700 1 -- 172.21.15.102:0/947090450 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3d4131ca0 con 0x7ff3d4094da0 2021-05-25T12:00:56.162 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.103+0000 7ff3bffff700 1 -- 172.21.15.102:0/947090450 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3820241914 0 0) 0x7ff3b4004c10 con 0x7ff3d41323c0 2021-05-25T12:00:56.162 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.103+0000 7ff3bffff700 1 -- 172.21.15.102:0/947090450 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff3d4131a00 con 0x7ff3d41323c0 2021-05-25T12:00:56.162 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.103+0000 7ff3bffff700 1 -- 172.21.15.102:0/947090450 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7ff3c80019f0 con 0x7ff3d41323c0 2021-05-25T12:00:56.163 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.103+0000 7ff3bffff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:56.163 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.103+0000 7ff3bffff700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:00:56.163 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.103+0000 7ff3bffff700 10 monclient(hunting): dump: 2021-05-25T12:00:56.163 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:56.163 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:56.164 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:56.164 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:56.164 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:56.164 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:56.165 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:56.165 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:56.165 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:56.165 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:56.165 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.103+0000 7ff3bffff700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:00:56.166 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.103+0000 7ff3bffff700 1 -- 172.21.15.102:0/947090450 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2806362959 0 0) 0x7ff3d4131ca0 con 0x7ff3d4094da0 2021-05-25T12:00:56.166 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.103+0000 7ff3bffff700 1 -- 172.21.15.102:0/947090450 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff3b4004c10 con 0x7ff3d4094da0 2021-05-25T12:00:56.166 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.103+0000 7ff3bffff700 1 -- 172.21.15.102:0/947090450 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7ff3c40019f0 con 0x7ff3d4094da0 2021-05-25T12:00:56.166 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.103+0000 7ff3bffff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:56.167 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.103+0000 7ff3bffff700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2021-05-25T12:00:56.167 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.103+0000 7ff3bffff700 10 monclient(hunting): dump: 2021-05-25T12:00:56.167 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:56.167 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:56.167 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:56.168 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:56.168 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:56.168 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:56.168 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:56.169 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:56.169 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:56.169 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:56.169 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.103+0000 7ff3bffff700 1 -- 172.21.15.102:0/947090450 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3454734699 0 0) 0x7ff3d4131a00 con 0x7ff3d41323c0 2021-05-25T12:00:56.169 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.103+0000 7ff3bffff700 1 -- 172.21.15.102:0/947090450 >> v1:172.21.15.181:6789/0 conn(0x7ff3d4094da0 legacy=0x7ff3d412e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:56.170 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.103+0000 7ff3bffff700 1 -- 172.21.15.102:0/947090450 >> v1:172.21.15.102:6789/0 conn(0x7ff3d4063840 legacy=0x7ff3d412e200 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-05-25T12:00:56.170 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.103+0000 7ff3bffff700 10 monclient: _finish_hunting 0 2021-05-25T12:00:56.170 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.103+0000 7ff3bffff700 1 monclient: found mon.c 2021-05-25T12:00:56.170 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.103+0000 7ff3bffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:00:56.170 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.103+0000 7ff3bffff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:00:56.171 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.103+0000 7ff3bffff700 1 -- 172.21.15.102:0/947090450 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3d4131f40 con 0x7ff3d41323c0 2021-05-25T12:00:56.171 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.103+0000 7ff3bffff700 10 monclient: _finish_auth 0 2021-05-25T12:00:56.171 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.103+0000 7ff3bffff700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:00:56.171 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.103+0000 7ff3da968700 5 monclient: authenticate success, global_id 4925 2021-05-25T12:00:56.171 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.103+0000 7ff3da968700 10 monclient: _renew_subs 2021-05-25T12:00:56.172 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.103+0000 7ff3da968700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:00:56.172 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.103+0000 7ff3da968700 1 -- 172.21.15.102:0/947090450 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3d412fe40 con 0x7ff3d41323c0 2021-05-25T12:00:56.172 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.103+0000 7ff3da968700 10 client.4925.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:00:56.172 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.103+0000 7ff3da968700 10 monclient: _renew_subs 2021-05-25T12:00:56.173 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.103+0000 7ff3da968700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:00:56.173 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.103+0000 7ff3da968700 1 -- 172.21.15.102:0/947090450 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3d41308b0 con 0x7ff3d41323c0 2021-05-25T12:00:56.173 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.103+0000 7ff3da968700 1 librados: init done 2021-05-25T12:00:56.173 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.103+0000 7ff3bffff700 1 -- 172.21.15.102:0/947090450 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff3c8002640 con 0x7ff3d41323c0 2021-05-25T12:00:56.174 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.103+0000 7ff3bffff700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:00:56.174 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.103+0000 7ff3bffff700 1 -- 172.21.15.102:0/947090450 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7ff3c80029d0 con 0x7ff3d41323c0 2021-05-25T12:00:56.174 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.103+0000 7ff3bffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:56.174 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.103+0000 7ff3d27fc700 4 set_mon_vals no callback set 2021-05-25T12:00:56.174 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.103+0000 7ff3bffff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:00:56.175 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.103+0000 7ff3bffff700 10 monclient: dump: 2021-05-25T12:00:56.175 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:56.175 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:56.175 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:56.176 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:56.176 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:56.176 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:56.176 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:56.176 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:56.177 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:56.177 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:56.177 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.104+0000 7ff3da968700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:00:56.177 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.104+0000 7ff3da968700 1 -- 172.21.15.102:0/947090450 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff3ac0065a0 con 0x7ff3ac002840 2021-05-25T12:00:56.178 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.104+0000 7ff3bffff700 1 -- 172.21.15.102:0/947090450 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7ff3c8002bb0 con 0x7ff3d41323c0 2021-05-25T12:00:56.178 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.104+0000 7ff3bffff700 1 -- 172.21.15.102:0/947090450 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(17..17 src has 1..17) v4 ==== 11815+0+0 (unknown 4088861535 0 0) 0x7ff3c803f3c0 con 0x7ff3d41323c0 2021-05-25T12:00:56.178 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.104+0000 7ff3bffff700 10 client.4925.objecter ms_dispatch 0x7ff3d4001550 osd_map(17..17 src has 1..17) v4 2021-05-25T12:00:56.178 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.104+0000 7ff3bffff700 3 client.4925.objecter handle_osd_map got epochs [17,17] > 0 2021-05-25T12:00:56.178 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.104+0000 7ff3bffff700 3 client.4925.objecter handle_osd_map decoding full epoch 17 2021-05-25T12:00:56.179 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.104+0000 7ff3d37fe700 10 monclient: get_auth_request con 0x7ff3b40332f0 auth_method 0 2021-05-25T12:00:56.179 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.104+0000 7ff3bffff700 20 client.4925.objecter dump_active .. 0 homeless 2021-05-25T12:00:56.179 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.104+0000 7ff3bffff700 10 client.4925.objecter ms_handle_connect 0x7ff3ac002840 2021-05-25T12:00:56.179 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.104+0000 7ff3bffff700 10 client.4925.objecter resend_mon_ops 2021-05-25T12:00:56.179 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.104+0000 7ff3bffff700 1 -- 172.21.15.102:0/947090450 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1106794707 0 0) 0x7ff3ac0065a0 con 0x7ff3ac002840 2021-05-25T12:00:56.180 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.104+0000 7ff3bffff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:56.180 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.104+0000 7ff3bffff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:56.180 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.104+0000 7ff3bffff700 10 monclient(hunting): my global_id is 74614 2021-05-25T12:00:56.180 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.104+0000 7ff3bffff700 1 -- 172.21.15.102:0/947090450 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3d4131a00 con 0x7ff3ac002840 2021-05-25T12:00:56.181 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.105+0000 7ff3bffff700 10 client.4925.objecter ms_handle_connect 0x7ff3b40332f0 2021-05-25T12:00:56.181 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.105+0000 7ff3bffff700 1 -- 172.21.15.102:0/947090450 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3531722202 0 0) 0x7ff3d4131a00 con 0x7ff3ac002840 2021-05-25T12:00:56.181 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.105+0000 7ff3bffff700 1 -- 172.21.15.102:0/947090450 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff3ac0065a0 con 0x7ff3ac002840 2021-05-25T12:00:56.181 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.105+0000 7ff3bffff700 1 -- 172.21.15.102:0/947090450 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7ff3c0003d00 con 0x7ff3ac002840 2021-05-25T12:00:56.181 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.105+0000 7ff3bffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:56.182 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.105+0000 7ff3bffff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T12:00:56.182 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.105+0000 7ff3bffff700 10 monclient: dump: 2021-05-25T12:00:56.182 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:56.182 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:56.183 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:56.183 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:56.183 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:56.183 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:56.183 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:56.184 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:56.184 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:56.184 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:56.184 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.105+0000 7ff3bffff700 1 -- 172.21.15.102:0/947090450 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1073816120 0 0) 0x7ff3ac0065a0 con 0x7ff3ac002840 2021-05-25T12:00:56.185 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.105+0000 7ff3bffff700 1 -- 172.21.15.102:0/947090450 --> v1:172.21.15.181:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff3ac009e40 con 0x7ff3ac002840 2021-05-25T12:00:56.185 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.105+0000 7ff3bffff700 1 -- 172.21.15.102:0/947090450 <== mon.1 v1:172.21.15.181:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7ff3c00051c0 con 0x7ff3ac002840 2021-05-25T12:00:56.185 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.105+0000 7ff3bffff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:00:56.185 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.105+0000 7ff3bffff700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:00:56.185 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.105+0000 7ff3bffff700 1 -- 172.21.15.102:0/947090450 >> v1:172.21.15.181:6789/0 conn(0x7ff3ac002840 legacy=0x7ff3ac004ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:56.186 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.105+0000 7ff3bffff700 1 -- 172.21.15.102:0/947090450 >> v1:172.21.15.181:6789/0 conn(0x7ff3ac002840 legacy=0x7ff3ac004ce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:00:56.186 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.114+0000 7ff3da968700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:00:56.186 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.114+0000 7ff3da968700 1 -- 172.21.15.102:0/947090450 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff3ac009b00 con 0x7ff3ac008dc0 2021-05-25T12:00:56.186 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.114+0000 7ff3bffff700 10 client.4925.objecter ms_handle_connect 0x7ff3ac008dc0 2021-05-25T12:00:56.187 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.114+0000 7ff3bffff700 10 client.4925.objecter resend_mon_ops 2021-05-25T12:00:56.187 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.114+0000 7ff3bffff700 1 -- 172.21.15.102:0/947090450 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2448359037 0 0) 0x7ff3ac009b00 con 0x7ff3ac008dc0 2021-05-25T12:00:56.187 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.114+0000 7ff3bffff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:00:56.187 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.114+0000 7ff3bffff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:00:56.187 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.114+0000 7ff3bffff700 10 monclient(hunting): my global_id is 74617 2021-05-25T12:00:56.188 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.114+0000 7ff3bffff700 1 -- 172.21.15.102:0/947090450 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3ac0065a0 con 0x7ff3ac008dc0 2021-05-25T12:00:56.188 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.115+0000 7ff3bffff700 1 -- 172.21.15.102:0/947090450 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 27951622 0 0) 0x7ff3ac0065a0 con 0x7ff3ac008dc0 2021-05-25T12:00:56.188 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.115+0000 7ff3bffff700 1 -- 172.21.15.102:0/947090450 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff3ac009b00 con 0x7ff3ac008dc0 2021-05-25T12:00:56.188 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.115+0000 7ff3bffff700 1 -- 172.21.15.102:0/947090450 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7ff3c00041c0 con 0x7ff3ac008dc0 2021-05-25T12:00:56.189 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.115+0000 7ff3bffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:00:56.189 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.115+0000 7ff3bffff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T12:00:56.189 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.115+0000 7ff3bffff700 10 monclient: dump: 2021-05-25T12:00:56.189 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:00:56.189 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:00:56.190 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:56.190 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:00:56.190 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:00:56.190 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:00:56.190 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:00:56.191 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:00:56.191 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:00:56.191 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:00:56.191 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.115+0000 7ff3bffff700 1 -- 172.21.15.102:0/947090450 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2775124554 0 0) 0x7ff3ac009b00 con 0x7ff3ac008dc0 2021-05-25T12:00:56.192 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.115+0000 7ff3bffff700 1 -- 172.21.15.102:0/947090450 --> v1:172.21.15.181:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ff3ac008740 con 0x7ff3ac008dc0 2021-05-25T12:00:56.192 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.115+0000 7ff3bffff700 1 -- 172.21.15.102:0/947090450 <== mon.1 v1:172.21.15.181:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3760 (unknown 0 0 227777409) 0x7ff3c0006ec0 con 0x7ff3ac008dc0 2021-05-25T12:00:56.192 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.115+0000 7ff3bffff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:00:56.192 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.115+0000 7ff3bffff700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:00:56.192 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.115+0000 7ff3bffff700 1 -- 172.21.15.102:0/947090450 >> v1:172.21.15.181:6789/0 conn(0x7ff3ac008dc0 legacy=0x7ff3ac009250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:56.193 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.115+0000 7ff3bffff700 1 -- 172.21.15.102:0/947090450 >> v1:172.21.15.181:6789/0 conn(0x7ff3ac008dc0 legacy=0x7ff3ac009250 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:00:56.193 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:00:56.193 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:00:56.194 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:00:56.194 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "peon", 2021-05-25T12:00:56.194 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 18, 2021-05-25T12:00:56.194 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:00:56.195 INFO:teuthology.orchestra.run.smithi102.stdout: 1, 2021-05-25T12:00:56.195 INFO:teuthology.orchestra.run.smithi102.stdout: 2 2021-05-25T12:00:56.195 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:00:56.195 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 0, 2021-05-25T12:00:56.195 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:00:56.196 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:00:56.196 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:00:56.196 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:00:56.196 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:00:56.197 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:00:56.197 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:00:56.197 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:00:56.197 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:00:56.197 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:00:56.198 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:00:56.198 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:00:56.198 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:00:56.198 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:00:56.199 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:00:56.199 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:00:56.199 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:00:56.199 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:00:56.199 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:00:56.200 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:00:56.200 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:00:56.200 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:00:56.200 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:00:56.200 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:00:56.201 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:00:56.201 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:56.201 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:00:56.201 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:00:56.202 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:00:56.202 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:00:56.202 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:00:56.202 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:00:56.202 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:00:56.203 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:00:56.203 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:00:56.203 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:00:56.203 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:00:56.203 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:00:56.204 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:00:56.204 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:00:56.204 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:00:56.204 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:00:56.205 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:00:56.205 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:00:56.205 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:00:56.205 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:00:56.205 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:00:56.206 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:00:56.206 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:00:56.206 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:00:56.206 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:00:56.207 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:00:56.207 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:56.207 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:00:56.207 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:56.208 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:00:56.208 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:00:56.208 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:00:56.208 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:00:56.208 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:56.209 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:00:56.209 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:00:56.209 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:00:56.209 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:56.210 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:00:56.210 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:56.210 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:00:56.210 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:00:56.211 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:00:56.211 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:00:56.211 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:00:56.211 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:56.211 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:56.212 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:00:56.212 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:00:56.212 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:00:56.212 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:00:56.212 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:56.213 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:00:56.213 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:00:56.213 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:00:56.213 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:56.214 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:00:56.214 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:56.214 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:00:56.214 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:00:56.215 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:00:56.215 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:00:56.215 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:00:56.215 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:56.215 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:56.216 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:00:56.216 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:00:56.216 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:00:56.216 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:00:56.216 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:56.217 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:00:56.217 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:00:56.217 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:00:56.217 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:56.218 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:00:56.218 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:56.218 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:00:56.218 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:00:56.218 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:00:56.219 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:00:56.219 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:00:56.219 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:56.219 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:00:56.219 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:56.220 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:00:56.220 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:00:56.220 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:56.220 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:00:56.221 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:00:56.221 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:00:56.221 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:56.221 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:00:56.221 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:00:56.222 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:00:56.222 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:00:56.222 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:00:56.222 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 30 2021-05-25T12:00:56.222 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:00:56.223 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:00:56.223 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:00:56.223 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:00:56.223 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:00:56.225 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.116+0000 7ff3da968700 10 librados: watch_flush enter 2021-05-25T12:00:56.225 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.116+0000 7ff3da968700 10 librados: watch_flush exit 2021-05-25T12:00:56.226 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.116+0000 7ff3da968700 20 client.4925.objecter shutdown clearing up homeless session... 2021-05-25T12:00:56.226 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.116+0000 7ff3da968700 10 client.4925.objecter successfully canceled tick 2021-05-25T12:00:56.226 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.116+0000 7ff3da968700 1 -- 172.21.15.102:0/947090450 >> v1:172.21.15.181:6812/29582 conn(0x7ff3b40332f0 legacy=0x7ff3b40357b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:56.226 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.116+0000 7ff3da968700 10 monclient: shutdown 2021-05-25T12:00:56.227 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.116+0000 7ff3da968700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:00:56.227 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.116+0000 7ff3da968700 1 -- 172.21.15.102:0/947090450 >> v1:172.21.15.102:6790/0 conn(0x7ff3d41323c0 legacy=0x7ff3d41433a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:00:56.227 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.116+0000 7ff3d8f05700 1 -- 172.21.15.102:0/947090450 reap_dead start 2021-05-25T12:00:56.227 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.116+0000 7ff3d8f05700 1 -- 172.21.15.102:0/947090450 reap_dead start 2021-05-25T12:00:56.228 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.116+0000 7ff3da968700 1 -- 172.21.15.102:0/947090450 shutdown_connections 2021-05-25T12:00:56.228 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.116+0000 7ff3da968700 1 -- 172.21.15.102:0/947090450 >> 172.21.15.102:0/947090450 conn(0x7ff3d409cb30 msgr2=0x7ff3d41349e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:00:56.228 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.116+0000 7ff3da968700 1 -- 172.21.15.102:0/947090450 shutdown_connections 2021-05-25T12:00:56.228 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.116+0000 7ff3da968700 1 -- 172.21.15.102:0/947090450 wait complete. 2021-05-25T12:00:56.228 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:00:56.116+0000 7ff3da968700 1 librados: shutdown 2021-05-25T12:00:56.229 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2021-05-25T12:00:56.230 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_1_[33413]: starting. 2021-05-25T12:00:56.230 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_1_[33413]: creating pool ceph_test_rados_list_parallel.smithi181-33377 2021-05-25T12:00:56.230 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_1_[33413]: created object 0... 2021-05-25T12:00:56.230 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_1_[33413]: created object 25... 2021-05-25T12:00:56.231 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_1_[33413]: created object 49... 2021-05-25T12:00:56.231 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_1_[33413]: finishing. 2021-05-25T12:00:56.231 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_1_[33413]: shutting down. 2021-05-25T12:00:56.231 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_2_[33414]: starting. 2021-05-25T12:00:56.231 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_2_[33414]: listing objects. 2021-05-25T12:00:56.232 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_2_[33414]: listed object 0... 2021-05-25T12:00:56.232 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_2_[33414]: listed object 25... 2021-05-25T12:00:56.232 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_2_[33414]: saw 50 objects 2021-05-25T12:00:56.232 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_2_[33414]: shutting down. 2021-05-25T12:00:56.232 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: ******************************* 2021-05-25T12:00:56.233 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_3_[34042]: starting. 2021-05-25T12:00:56.233 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_3_[34042]: creating pool ceph_test_rados_list_parallel.smithi181-33377 2021-05-25T12:00:56.233 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_3_[34042]: created object 0... 2021-05-25T12:00:56.233 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_3_[34042]: created object 25... 2021-05-25T12:00:56.233 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_3_[34042]: created object 49... 2021-05-25T12:00:56.234 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_3_[34042]: finishing. 2021-05-25T12:00:56.234 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_3_[34042]: shutting down. 2021-05-25T12:00:56.234 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: ******************************* 2021-05-25T12:00:56.234 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_4_[34043]: starting. 2021-05-25T12:00:56.235 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_4_[34043]: listing objects. 2021-05-25T12:00:56.235 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_4_[34043]: listed object 0... 2021-05-25T12:00:56.235 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_4_[34043]: listed object 25... 2021-05-25T12:00:56.235 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_4_[34043]: saw 47 objects 2021-05-25T12:00:56.235 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_4_[34043]: shutting down. 2021-05-25T12:00:56.236 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: ******************************* 2021-05-25T12:00:56.236 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_5_[34044]: starting. 2021-05-25T12:00:56.236 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_5_[34044]: removed 25 objects... 2021-05-25T12:00:56.236 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_5_[34044]: removed half of the objects 2021-05-25T12:00:56.236 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_5_[34044]: removed 50 objects... 2021-05-25T12:00:56.237 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_5_[34044]: removed 50 objects 2021-05-25T12:00:56.237 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_5_[34044]: shutting down. 2021-05-25T12:00:56.237 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: ******************************* 2021-05-25T12:00:56.237 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: ******************************* 2021-05-25T12:00:56.238 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_6_[34093]: starting. 2021-05-25T12:00:56.238 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_6_[34093]: creating pool ceph_test_rados_list_parallel.smithi181-33377 2021-05-25T12:00:56.238 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_6_[34093]: created object 0... 2021-05-25T12:00:56.238 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_6_[34093]: created object 25... 2021-05-25T12:00:56.238 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_6_[34093]: created object 49... 2021-05-25T12:00:56.239 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_6_[34093]: finishing. 2021-05-25T12:00:56.239 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_6_[34093]: shutting down. 2021-05-25T12:00:56.239 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: ******************************* 2021-05-25T12:00:56.239 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: ******************************* 2021-05-25T12:00:56.239 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_7_[34094]: starting. 2021-05-25T12:00:56.240 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_7_[34094]: listing objects. 2021-05-25T12:00:56.240 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_7_[34094]: listed object 0... 2021-05-25T12:00:56.240 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_7_[34094]: listed object 25... 2021-05-25T12:00:56.240 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_7_[34094]: listed object 50... 2021-05-25T12:00:56.241 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_7_[34094]: saw 53 objects 2021-05-25T12:00:56.241 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_7_[34094]: shutting down. 2021-05-25T12:00:56.241 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: ******************************* 2021-05-25T12:00:56.241 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: ******************************* 2021-05-25T12:00:56.241 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_8_[34095]: starting. 2021-05-25T12:00:56.242 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_8_[34095]: added 25 objects... 2021-05-25T12:00:56.242 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_8_[34095]: added half of the objects 2021-05-25T12:00:56.242 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_8_[34095]: added 50 objects... 2021-05-25T12:00:56.242 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_8_[34095]: added 50 objects 2021-05-25T12:00:56.243 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_8_[34095]: shutting down. 2021-05-25T12:00:56.243 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: ******************************* 2021-05-25T12:00:56.243 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: ******************************* 2021-05-25T12:00:56.652 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: ******************************* 2021-05-25T12:00:56.652 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_9_[34144]: starting. 2021-05-25T12:00:56.653 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_9_[34144]: creating pool ceph_test_rados_list_parallel.smithi181-33377 2021-05-25T12:00:56.653 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_9_[34144]: created object 0... 2021-05-25T12:00:56.653 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_9_[34144]: created object 25... 2021-05-25T12:00:56.653 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_9_[34144]: created object 49... 2021-05-25T12:00:56.654 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_9_[34144]: finishing. 2021-05-25T12:00:56.654 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_9_[34144]: shutting down. 2021-05-25T12:00:56.654 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: ******************************* 2021-05-25T12:00:56.654 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: ******************************* 2021-05-25T12:00:56.655 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: ******************************* 2021-05-25T12:00:56.655 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_10_[34145]: starting. 2021-05-25T12:00:56.655 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_10_[34145]: listing objects. 2021-05-25T12:00:56.655 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_10_[34145]: listed object 0... 2021-05-25T12:00:56.655 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_10_[34145]: listed object 25... 2021-05-25T12:00:56.656 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_10_[34145]: listed object 50... 2021-05-25T12:00:56.656 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_10_[34145]: listed object 75... 2021-05-25T12:00:56.656 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_10_[34145]: saw 99 objects 2021-05-25T12:00:56.656 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_10_[34145]: shutting down. 2021-05-25T12:00:56.656 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: ******************************* 2021-05-25T12:00:56.657 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: ******************************* 2021-05-25T12:00:56.657 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: ******************************* 2021-05-25T12:00:56.657 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_12_[34147]: starting. 2021-05-25T12:00:56.657 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_12_[34147]: added 25 objects... 2021-05-25T12:00:56.658 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_12_[34147]: added half of the objects 2021-05-25T12:00:56.658 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_12_[34147]: added 50 objects... 2021-05-25T12:00:56.658 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_12_[34147]: added 50 objects 2021-05-25T12:00:56.658 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_12_[34147]: shutting down. 2021-05-25T12:00:56.658 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: ******************************* 2021-05-25T12:00:56.659 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: ******************************* 2021-05-25T12:00:56.659 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: ******************************* 2021-05-25T12:00:56.659 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_13_[34148]: starting. 2021-05-25T12:00:56.659 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_13_[34148]: removed 25 objects... 2021-05-25T12:00:56.659 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_13_[34148]: removed half of the objects 2021-05-25T12:00:56.660 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_13_[34148]: removed 50 objects... 2021-05-25T12:00:56.660 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_13_[34148]: removed 50 objects 2021-05-25T12:00:56.660 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_13_[34148]: shutting down. 2021-05-25T12:00:56.660 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: ******************************* 2021-05-25T12:00:56.661 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: ******************************* 2021-05-25T12:00:56.661 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: ******************************* 2021-05-25T12:00:56.661 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_11_[34146]: starting. 2021-05-25T12:00:56.661 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_11_[34146]: added 25 objects... 2021-05-25T12:00:56.661 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_11_[34146]: added half of the objects 2021-05-25T12:00:56.662 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_11_[34146]: added 50 objects... 2021-05-25T12:00:56.662 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_11_[34146]: added 50 objects 2021-05-25T12:00:56.662 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_11_[34146]: shutting down. 2021-05-25T12:00:56.662 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: ******************************* 2021-05-25T12:00:56.662 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: ******************************* 2021-05-25T12:00:56.663 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: ******************************* 2021-05-25T12:00:56.663 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: ******************************* 2021-05-25T12:00:56.663 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_14_[34229]: starting. 2021-05-25T12:00:56.663 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_14_[34229]: creating pool ceph_test_rados_list_parallel.smithi181-33377 2021-05-25T12:00:56.664 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_14_[34229]: created object 0... 2021-05-25T12:00:56.664 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_14_[34229]: created object 25... 2021-05-25T12:00:56.664 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_14_[34229]: created object 49... 2021-05-25T12:00:56.664 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_14_[34229]: finishing. 2021-05-25T12:00:56.664 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_14_[34229]: shutting down. 2021-05-25T12:00:56.665 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: ******************************* 2021-05-25T12:00:56.665 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: ******************************* 2021-05-25T12:00:56.665 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: ******************************* 2021-05-25T12:00:56.665 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: ******************************* 2021-05-25T12:00:56.665 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_15_[34230]: starting. 2021-05-25T12:00:56.666 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_15_[34230]: listing objects. 2021-05-25T12:00:56.666 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_15_[34230]: listed object 0... 2021-05-25T12:00:56.813 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_15_[34230]: listed object 2 api_list: [==========] Running 11 tests from 3 test suites. 2021-05-25T12:00:56.813 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: [----------] Global test environment set-up. 2021-05-25T12:00:56.813 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: [----------] 7 tests from LibRadosList 2021-05-25T12:00:56.814 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: [ RUN ] LibRadosList.ListObjects 2021-05-25T12:00:56.814 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: [ OK ] LibRadosList.ListObjects (166 ms) 2021-05-25T12:00:56.814 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: [ RUN ] LibRadosList.ListObjectsZeroInName 2021-05-25T12:00:56.814 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: [ OK ] LibRadosList.ListObjectsZeroInName (22 ms) 2021-05-25T12:00:56.815 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: [ RUN ] LibRadosList.ListObjectsNS 2021-05-25T12:00:56.815 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: myset foo1,foo2,foo3 2021-05-25T12:00:56.815 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: foo1 2021-05-25T12:00:56.815 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: foo2 2021-05-25T12:00:56.816 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: foo3 2021-05-25T12:00:56.816 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: myset foo1,foo4,foo5 2021-05-25T12:00:56.816 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: foo4 2021-05-25T12:00:56.816 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: foo5 2021-05-25T12:00:56.816 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: foo1 2021-05-25T12:00:56.817 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: myset foo6,foo7 2021-05-25T12:00:56.817 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: foo7 2021-05-25T12:00:56.817 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: foo6 2021-05-25T12:00:56.817 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2021-05-25T12:00:56.817 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: ns1:foo4 2021-05-25T12:00:56.818 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: ns1:foo5 2021-05-25T12:00:56.818 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: ns2:foo7 2021-05-25T12:00:56.818 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: ns2:foo6 2021-05-25T12:00:56.818 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: ns1:foo1 2021-05-25T12:00:56.819 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: :foo1 2021-05-25T12:00:56.819 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: :foo2 2021-05-25T12:00:56.819 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: :foo3 2021-05-25T12:00:56.819 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: [ OK ] LibRadosList.ListObjectsNS (86 ms) 2021-05-25T12:00:56.819 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: [ RUN ] LibRadosList.ListObjectsStart 2021-05-25T12:00:56.820 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: 1 0 2021-05-25T12:00:56.820 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: 10 0 2021-05-25T12:00:56.820 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: 13 0 2021-05-25T12:00:56.820 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: 7 0 2021-05-25T12:00:56.820 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: 14 0 2021-05-25T12:00:56.821 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: 0 0 2021-05-25T12:00:56.821 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: 15 0 2021-05-25T12:00:56.821 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: 11 0 2021-05-25T12:00:56.821 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: 5 0 2021-05-25T12:00:56.821 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: 8 0 2021-05-25T12:00:56.822 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: 6 0 2021-05-25T12:00:56.822 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: 3 0 2021-05-25T12:00:56.822 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: 4 0 2021-05-25T12:00:56.822 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: 12 0 2021-05-25T12:00:56.823 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: 9 0 2021-05-25T12:00:56.823 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: 2 0 2021-05-25T12:00:56.823 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2021-05-25T12:00:56.823 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: [ OK ] LibRadosList.ListObjectsStart (93 ms) 2021-05-25T12:00:56.823 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: [ RUN ] LibRadosList.ListObjectsCursor 2021-05-25T12:00:56.824 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: x cursor=MIN 2021-05-25T12:00:56.824 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: > oid=1 cursor=8:02547ec2:::1:head 2021-05-25T12:00:56.824 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: > oid=10 cursor=8:52ea6a34:::10:head 2021-05-25T12:00:56.824 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: > oid=13 cursor=8:566253c9:::13:head 2021-05-25T12:00:56.824 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: > oid=7 cursor=8:5c6b0b28:::7:head 2021-05-25T12:00:56.825 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: > oid=14 cursor=8:62a1935d:::14:head 2021-05-25T12:00:56.825 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: > oid=0 cursor=8:6cac518f:::0:head 2021-05-25T12:00:56.825 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: > oid=15 cursor=8:863748b0:::15:head 2021-05-25T12:00:56.825 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: > oid=11 cursor=8:89d3ae78:::11:head 2021-05-25T12:00:56.825 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: > oid=5 cursor=8:b29083e3:::5:head 2021-05-25T12:00:56.826 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: > oid=8 cursor=8:bd63b0f1:::8:head 2021-05-25T12:00:56.826 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: > oid=6 cursor=8:c4fdafeb:::6:head 2021-05-25T12:00:56.826 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: > oid=3 cursor=8:cfc208b3:::3:head 2021-05-25T12:00:56.826 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: > oid=4 cursor=8:d83876eb:::4:head 2021-05-25T12:00:56.826 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: > oid=12 cursor=8:de5d7c5f:::12:head 2021-05-25T12:00:56.827 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: > oid=9 cursor=8:e960b815:::9:head 2021-05-25T12:00:56.827 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: > oid=2 cursor=8:f905c69b:::2:head 2021-05-25T12:00:56.827 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: FIRST> seek to MIN oid=1 2021-05-25T12:00:56.827 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: : oid=1 cursor=8:02547ec2:::1:head 2021-05-25T12:00:56.828 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: : seek to 8:02547ec2:::1:head 2021-05-25T12:00:56.828 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: > 8:02547ec2:::1:head -> 1 2021-05-25T12:00:56.828 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: : oid=10 cursor=8:52ea6a34:::10:head 2021-05-25T12:00:56.828 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: : seek to 8:52ea6a34:::10:head 2021-05-25T12:00:56.828 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: > 8:52ea6a34:::10:head -> 10 2021-05-25T12:00:56.829 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: : oid=13 cursor=8:566253c9:::13:head 2021-05-25T12:00:56.829 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: : seek to 8:566253c9:::13:head 2021-05-25T12:00:56.829 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: > 8:566253c9:::13:head -> 13 2021-05-25T12:00:56.829 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: : oid=7 cursor=8:5c6b0b28:::7:head 2021-05-25T12:00:56.829 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: : seek to 8:5c6b0b28:::7:head 2021-05-25T12:00:56.830 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: > 8:5c6b0b28:::7:head -> 7 2021-05-25T12:00:56.830 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: : oid=14 cursor=8:62a1935d:::14:head 2021-05-25T12:00:56.881 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: : seek to 8:62a1935d:::14:head 2021-05-25T12:00:56.882 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: > 8:62a1935d:::14:head -> 14 2021-05-25T12:00:56.882 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: : oid=0 cursor=8:6cac518f:::0:head 2021-05-25T12:00:56.882 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: : seek to 8:6cac518f:::0:head 2021-05-25T12:00:56.883 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: > 8:6cac518f:::0:head -> 0 2021-05-25T12:00:56.883 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: : oid=15 cursor=8:863748b0:::15:head 2021-05-25T12:00:56.883 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: : seek to 8:863748b0:::15:head 2021-05-25T12:00:56.883 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: > 8:863748b0:::15:head -> 15 2021-05-25T12:00:56.883 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: : oid=11 cursor=8:89d3ae78:::11:head 2021-05-25T12:00:56.884 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: : seek to 8:89d3ae78:::11:head 2021-05-25T12:00:56.884 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: > 8:89d3ae78:::11:head -> 11 2021-05-25T12:00:56.884 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: : oid=5 cursor=8:b29083e3:::5:head 2021-05-25T12:00:56.884 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: : seek to 8:b29083e3:::5:head 2021-05-25T12:00:56.885 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: > 8:b29083e3:::5:head -> 5 2021-05-25T12:00:56.885 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: : oid=8 cursor=8:bd63b0f1:::8:head 2021-05-25T12:00:56.885 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: : seek to 8:bd63b0f1:::8:head 2021-05-25T12:00:56.885 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: > 8:bd63b0f1:::8:head -> 8 2021-05-25T12:00:56.885 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: : oid=6 cursor=8:c4fdafeb:::6:head 2021-05-25T12:00:56.886 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: : seek to 8:c4fdafeb:::6:head 2021-05-25T12:00:56.886 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: > 8:c4fdafeb:::6:head -> 6 2021-05-25T12:00:56.886 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: : oid=3 cursor=8:cfc208b3:::3:head 2021-05-25T12:00:56.886 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: : seek to 8:cfc208b3:::3:head 2021-05-25T12:00:56.887 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: > 8:cfc208b3:::3:head -> 3 2021-05-25T12:00:56.887 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: : oid=4 cursor=8:d83876eb:::4:head 2021-05-25T12:00:56.887 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: : seek to 8:d83876eb:::4:head 2021-05-25T12:00:56.887 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: > 8:d83876eb:::4:head -> 4 2021-05-25T12:00:56.887 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: : oid=12 cursor=8:de5d7c5f:::12:head 2021-05-25T12:00:56.888 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: : seek to 8:de5d7c5f:::12:head 2021-05-25T12:00:56.888 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: > 8:de5d7c5f:::12:head -> 12 2021-05-25T12:00:56.888 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: : oid=9 cursor=8:e960b815:::9:head 2021-05-25T12:00:56.888 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: : seek to 8:e960b815:::9:head 2021-05-25T12:00:56.889 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: > 8:e960b815:::9:head -> 9 2021-05-25T12:00:56.889 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: : oid=2 cursor=8:f905c69b:::2:head 2021-05-25T12:00:56.889 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: : seek to 8:f905c69b:::2:head 2021-05-25T12:00:56.889 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: > 8:f905c69b:::2:head -> 2 2021-05-25T12:00:56.889 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: : seek to 8:566253c9:::13:head 2021-05-25T12:00:56.890 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: : cursor()=8:566253c9:::13:head expected=8:566253c9:::13:head 2021-05-25T12:00:56.890 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: > 8:566253c9:::13:head -> 13 2021-05-25T12:00:56.890 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: : entry=13 expected=13 2021-05-25T12:00:56.890 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: : seek to 8:02547ec2:::1:head 2021-05-25T12:00:56.891 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: : cursor()=8:02547ec2:::1:head expected=8:02547ec2:::1:head 2021-05-25T12:00:56.891 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: > 8:02547ec2:::1:head -> 1 2021-05-25T12:00:56.891 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: : entry=1 expected=1 2021-05-25T12:00:56.891 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: : seek to 8:f905c69b:::2:head 2021-05-25T12:00:56.891 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: : cursor()=8:f905c69b:::2:head expected=8:f905c69b:::2:head 2021-05-25T12:00:56.892 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: > 8:f905c69b:::2:head -> 2 2021-05-25T12:00:56.892 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: : entry=2 expected=2 2021-05-25T12:00:56.892 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: : seek to 8:5c6b0b28:::7:head 2021-05-25T12:00:56.892 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: : cursor()=8:5c6b0b28:::7:head expected=8:5c6b0b28:::7:head 2021-05-25T12:00:56.893 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: > 8:5c6b0b28:::7:head -> 7 2021-05-25T12:00:56.893 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: : entry=7 expected=7 2021-05-25T12:00:56.893 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: : seek to 8:b29083e3:::5:head 2021-05-25T12:00:56.893 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: : cursor()=8:b29083e3:::5:head expected=8:b29083e3:::5:head 2021-05-25T12:00:56.894 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: > 8:b29083e3:::5:head -> 5 2021-05-25T12:00:56.894 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: : entry=5 expected=5 2021-05-25T12:00:56.894 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: : seek to 8:89d3ae78:::11:head 2021-05-25T12:00:56.894 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: : cursor()=8:89d3ae78:::11:head expected=8:89d3ae78:::11:head 2021-05-25T12:00:56.894 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: > 8:89d3ae78:::11:head -> 11 2021-05-25T12:00:56.895 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: : entry=11 expected=11 2021-05-25T12:00:56.895 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: : seek to 8:863748b0:::15:head 2021-05-25T12:00:56.895 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: : cursor()=8:863748b0:::15:head expected=8:863748b0:::15:head 2021-05-25T12:00:56.895 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: > 8:863748b0:::15:head -> 15 2021-05-25T12:00:56.896 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: : entry=15 expected=15 2021-05-25T12:00:56.896 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: : seek to 8:6cac518f:::0:head 2021-05-25T12:00:56.896 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: : cursor()=8:6cac518f:::0:head expected=8:6cac518f:::0:head 2021-05-25T12:00:56.896 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: > 8:6cac518f:::0:head -> 0 2021-05-25T12:00:56.896 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: : entry=0 expected=0 2021-05-25T12:00:56.897 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: : seek to 8:62a1935d:::14:head 2021-05-25T12:00:57.383 INFO:tasks.workunit.client.0.smithi181.stdout: api api_cmd_pp: Running main() from gmock_main.cc 2021-05-25T12:00:57.384 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd_pp: [==========] Running 3 tests from 1 test suite. 2021-05-25T12:00:57.384 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd_pp: [----------] Global test environment set-up. 2021-05-25T12:00:57.384 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd 2021-05-25T12:00:57.384 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.MonDescribePP 2021-05-25T12:00:57.385 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd_pp: [ OK ] LibRadosCmd.MonDescribePP (49 ms) 2021-05-25T12:00:57.385 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.OSDCmdPP 2021-05-25T12:00:57.385 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd_pp: [ OK ] LibRadosCmd.OSDCmdPP (34 ms) 2021-05-25T12:00:57.385 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.PGCmdPP 2021-05-25T12:00:57.386 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd_pp: [ OK ] LibRadosCmd.PGCmdPP (17473 ms) 2021-05-25T12:00:57.386 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd (17556 ms total) 2021-05-25T12:00:57.386 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd_pp: 2021-05-25T12:00:57.386 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd_pp: [----------] Global test environment tear-down 2021-05-25T12:00:57.386 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd_pp: [==========] 3 tests from 1 test suite ran. (17556 ms total) 2021-05-25T12:00:57.387 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd_pp: [ PASSED ] 3 tests. 2021-05-25T12:00:57.387 INFO:tasks.workunit.client.0.smithi181.stdout:5... 2021-05-25T12:00:57.387 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_15_[34230]: listed object 50... 2021-05-25T12:00:57.388 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_15_[34230]: listed object 75... 2021-05-25T12:00:57.388 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_15_[34230]: listed object 100... 2021-05-25T12:00:57.388 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_15_[34230]: listed object 125... 2021-05-25T12:00:57.388 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_15_[34230]: saw 150 objects 2021-05-25T12:00:57.389 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_15_[34230]: shutting down. 2021-05-25T12:00:57.389 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: ******************************* 2021-05-25T12:00:57.389 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: ******************************* 2021-05-25T12:00:57.389 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: ******************************* 2021-05-25T12:00:57.389 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: ******************************* 2021-05-25T12:00:57.390 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_16_[34231]: starting. 2021-05-25T12:00:57.390 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_16_[34231]: added 25 objects... 2021-05-25T12:00:57.390 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_16_[34231]: added half of the objects 2021-05-25T12:00:57.390 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_16_[34231]: added 50 objects... 2021-05-25T12:00:57.391 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_16_[34231]: added 50 objects 2021-05-25T12:00:57.391 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: process_16_[34231]: shutting down. 2021-05-25T12:00:57.391 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: ******************************* 2021-05-25T12:00:57.391 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: ******************************* 2021-05-25T12:00:57.391 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: ******************************* 2021-05-25T12:00:57.392 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: ******************************* 2021-05-25T12:00:57.392 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: ******************************* 2021-05-25T12:00:57.392 INFO:tasks.workunit.client.0.smithi181.stdout: list_parallel: ******* SUCCESS ********** 2021-05-25T12:00:57.393 INFO:tasks.workunit.client.0.smithi181.stdout: api_cls_remote_reads: [==========] Running 1 test from 1 test suite. 2021-05-25T12:00:57.393 INFO:tasks.workunit.client.0.smithi181.stdout: api_cls_remote_reads: [----------] Global test environment set-up. 2021-05-25T12:00:57.393 INFO:tasks.workunit.client.0.smithi181.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads 2021-05-25T12:00:57.394 INFO:tasks.workunit.client.0.smithi181.stdout: api_cls_remote_reads: [ RUN ] ClsTestRemoteReads.TestGather 2021-05-25T12:00:57.394 INFO:tasks.workunit.client.0.smithi181.stdout: api_cls_remote_reads: [ OK ] ClsTestRemoteReads.TestGather (17515 ms) 2021-05-25T12:00:57.394 INFO:tasks.workunit.client.0.smithi181.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads (17515 ms total) 2021-05-25T12:00:57.394 INFO:tasks.workunit.client.0.smithi181.stdout: api_cls_remote_reads: 2021-05-25T12:00:57.395 INFO:tasks.workunit.client.0.smithi181.stdout: api_cls_remote_reads: [----------] Global test environment tear-down 2021-05-25T12:00:57.395 INFO:tasks.workunit.client.0.smithi181.stdout: api_cls_remote_reads: [==========] 1 test from 1 test suite ran. (17515 ms total) 2021-05-25T12:00:57.395 INFO:tasks.workunit.client.0.smithi181.stdout: api_cls_remote_reads: [ PASSED ] 1 test. 2021-05-25T12:00:58.431 INFO:tasks.workunit.client.0.smithi181.stdout: api_c_read_operations: Running main() from gmock_main.cc 2021-05-25T12:00:58.431 INFO:tasks.workunit.client.0.smithi181.stdout: api_c_read_operations: [==========] Running 16 tests from 1 test suite. 2021-05-25T12:00:58.431 INFO:tasks.workunit.client.0.smithi181.stdout: api_c_read_operations: [----------] Global test environment set-up. 2021-05-25T12:00:58.432 INFO:tasks.workunit.client.0.smithi181.stdout: api_c_read_operations: [----------] 16 tests from CReadOpsTest 2021-05-25T12:00:58.432 INFO:tasks.workunit.client.0.smithi181.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.NewDelete 2021-05-25T12:00:58.432 INFO:tasks.workunit.client.0.smithi181.stdout: api_c_read_operations: [ OK ] CReadOpsTest.NewDelete (0 ms) 2021-05-25T12:00:58.432 INFO:tasks.workunit.client.0.smithi181.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.SetOpFlags 2021-05-25T12:00:58.432 INFO:tasks.workunit.client.0.smithi181.stdout: api_c_read_operations: [ OK ] CReadOpsTest.SetOpFlags (93 ms) 2021-05-25T12:00:58.433 INFO:tasks.workunit.client.0.smithi181.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertExists 2021-05-25T12:00:58.433 INFO:tasks.workunit.client.0.smithi181.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertExists (12 ms) 2021-05-25T12:00:58.433 INFO:tasks.workunit.client.0.smithi181.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertVersion 2021-05-25T12:00:58.433 INFO:tasks.workunit.client.0.smithi181.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertVersion (9 ms) 2021-05-25T12:00:58.433 INFO:tasks.workunit.client.0.smithi181.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpXattr 2021-05-25T12:00:58.434 INFO:tasks.workunit.client.0.smithi181.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpXattr (28 ms) 2021-05-25T12:00:58.434 INFO:tasks.workunit.client.0.smithi181.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Read 2021-05-25T12:00:58.434 INFO:tasks.workunit.client.0.smithi181.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Read (7 ms) 2021-05-25T12:00:58.434 INFO:tasks.workunit.client.0.smithi181.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Checksum 2021-05-25T12:00:58.435 INFO:tasks.workunit.client.0.smithi181.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Checksum (7 ms) 2021-05-25T12:00:58.435 INFO:tasks.workunit.client.0.smithi181.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.RWOrderedRead 2021-05-25T12:00:58.435 INFO:tasks.workunit.client.0.smithi181.stdout: api_c_read_operations: [ OK ] CReadOpsTest.RWOrderedRead (6 ms) 2021-05-25T12:00:58.435 INFO:tasks.workunit.client.0.smithi181.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ShortRead 2021-05-25T12:00:58.435 INFO:tasks.workunit.client.0.smithi181.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ShortRead (9 ms) 2021-05-25T12:00:58.436 INFO:tasks.workunit.client.0.smithi181.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Exec 2021-05-25T12:00:58.436 INFO:tasks.workunit.client.0.smithi181.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Exec (7 ms) 2021-05-25T12:00:58.436 INFO:tasks.workunit.client.0.smithi181.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ExecUserBuf 2021-05-25T12:00:58.436 INFO:tasks.workunit.client.0.smithi181.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ExecUserBuf (7 ms) 2021-05-25T12:00:58.437 INFO:tasks.workunit.client.0.smithi181.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat 2021-05-25T12:00:58.437 INFO:tasks.workunit.client.0.smithi181.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat (6 ms) 2021-05-25T12:00:58.437 INFO:tasks.workunit.client.0.smithi181.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Omap 2021-05-25T12:00:58.437 INFO:tasks.workunit.client.0.smithi181.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Omap (15 ms) 2021-05-25T12:00:58.437 INFO:tasks.workunit.client.0.smithi181.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.OmapNuls 2021-05-25T12:00:58.438 INFO:tasks.workunit.client.0.smithi181.stdout: api_c_read_operations: [ OK ] CReadOpsTest.OmapNuls (14 ms) 2021-05-25T12:00:58.438 INFO:tasks.workunit.client.0.smithi181.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.GetXattrs 2021-05-25T12:00:58.438 INFO:tasks.workunit.client.0.smithi181.stdout: api_c_read_operations: [ OK ] CReadOpsTest.GetXattrs (16 ms) 2021-05-25T12:00:58.438 INFO:tasks.workunit.client.0.smithi181.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpExt 2021-05-25T12:00:58.439 INFO:tasks.workunit.client.0.smithi181.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpExt (8 ms) 2021-05-25T12:00:58.439 INFO:tasks.workunit.client.0.smithi181.stdout: api_c_read_operations: [----------] 16 tests from CReadOpsTest (245 ms total) 2021-05-25T12:00:58.439 INFO:tasks.workunit.client.0.smithi181.stdout: api_c_read_operations: 2021-05-25T12:00:58.439 INFO:tasks.workunit.client.0.smithi181.stdout: api_c_read_operations: [----------] Global test environment tear-down 2021-05-25T12:00:58.439 INFO:tasks.workunit.client.0.smithi181.stdout: api_c_read_operations: [==========] 16 tests from 1 test suite ran. (18562 ms total) 2021-05-25T12:00:58.440 INFO:tasks.workunit.client.0.smithi181.stdout: api_c_read_operations: [ PASSED ] 16 tests. 2021-05-25T12:00:58.484 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: Running main() from gmock_main.cc 2021-05-25T12:00:58.484 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: [==========] Running 4 tests from 1 test suite. 2021-05-25T12:00:58.484 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: [----------] Global test environment set-up. 2021-05-25T12:00:58.485 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: [----------] 4 tests from LibRadosCmd 2021-05-25T12:00:58.485 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: [ RUN ] LibRadosCmd.MonDescribe 2021-05-25T12:00:58.485 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: [ OK ] LibRadosCmd.MonDescribe (41 ms) 2021-05-25T12:00:58.485 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: [ RUN ] LibRadosCmd.OSDCmd 2021-05-25T12:00:58.486 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: [ OK ] LibRadosCmd.OSDCmd (53 ms) 2021-05-25T12:00:58.486 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: [ RUN ] LibRadosCmd.PGCmd 2021-05-25T12:00:58.486 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: [ OK ] LibRadosCmd.PGCmd (18515 ms) 2021-05-25T12:00:58.486 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: [ RUN ] LibRadosCmd.WatchLog 2021-05-25T12:00:58.486 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:00:57.377968+0000 mon.c [INF] from='client.74371 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi181-33235-1","app": "rados","yes_i_really_mean_it": true}]': finished 2021-05-25T12:00:58.487 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:00:57.378076+0000 mon.c [INF] from='client.74335 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi181-33341-1","app": "rados","yes_i_really_mean_it": true}]': finished 2021-05-25T12:00:58.487 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:00:57.378185+0000 mon.c [INF] from='client.74530 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi181-33015-4","app": "rados","yes_i_really_mean_it": true}]': finished 2021-05-25T12:00:58.487 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:00:57.378316+0000 mon.c [INF] from='client.? v1:172.21.15.181:0/1206631197' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi181-32863-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2021-05-25T12:00:58.487 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:00:57.378395+0000 mon.c [INF] from='client.? v1:172.21.15.181:0/3650250362' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi181-33095-6", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2021-05-25T12:00:58.488 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:00:57.378542+0000 mon.c [INF] from='client.74656 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi181-32858-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2021-05-25T12:00:58.488 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:00:57.381146+0000 mon.c [INF] from='client.? v1:172.21.15.181:0/1206631197' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi181-32863-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi181-32863-22"}]: dispatch 2021-05-25T12:00:58.488 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:00:57.381617+0000 mon.c [INF] from='client.? v1:172.21.15.181:0/3650250362' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi181-33095-6", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi181-33095-6"}]: dispatch 2021-05-25T12:00:58.488 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:00:57.384959+0000 mon.b [INF] from='client.? v1:172.21.15.181:0/570907553' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi181-32842-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-05-25T12:00:58.489 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:00:57.385110+0000 mon.b [INF] from='client.? v1:172.21.15.181:0/2768366830' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi181-32858-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi181-32858-15"}]: dispatch 2021-05-25T12:00:58.489 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:00:57.395491+0000 mon.c [INF] from='client.74650 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi181-32842-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-05-25T12:00:59.484 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:00:57.395774+0000 mon.c [INF] from='client.74656 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi181-32858-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi181-32858-15"}]: dispatch 2021-05-25T12:00:59.485 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:00:57.461912+0000 mon.b [INF] from='client.? v1:172.21.15.181:0/2288382365' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi181-33162-7"}]: dispatch 2021-05-25T12:00:59.485 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:00:57.492774+0000 mon.c [INF] from='client.74698 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi181-33162-7"}]: dispatch 2021-05-25T12:00:59.485 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:00:57.493554+0000 mon.b [INF] from='client.? v1:172.21.15.181:0/2288382365' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi181-33162-7"}]: dispatch 2021-05-25T12:00:59.485 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:00:57.581324+0000 mon.c [INF] from='client.74698 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi181-33162-7"}]: dispatch 2021-05-25T12:00:59.486 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:00:57.581889+0000 mon.b [INF] from='client.? v1:172.21.15.181:0/2288382365' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi181-33162-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2021-05-25T12:00:59.486 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:00:57.582366+0000 mon.c [INF] from='client.74698 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi181-33162-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2021-05-25T12:00:59.486 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:00:58.359583+0000 mon.c [INF] from='client.? v1:172.21.15.181:0/758857343' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi181-33204-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi181-33204-1"}]': finished 2021-05-25T12:00:59.486 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:00:58.400918+0000 mon.c [INF] from='client.74650 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi181-32842-2","app": "rados","yes_i_really_mean_it": true}]': finished 2021-05-25T12:00:59.487 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:00:58.425775+0000 mon.c [INF] from='client.74698 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi181-33162-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2021-05-25T12:00:59.487 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:00:58.426988+0000 mon.b [INF] from='client.? v1:172.21.15.181:0/2288382365' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi181-33162-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi181-33162-7"}]: dispatch 2021-05-25T12:00:59.487 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:00:58.427252+0000 mon.b [INF] from='client.? v1:172.21.15.181:0/1323872308' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi181-33335-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-05-25T12:00:59.487 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:00:58.427677+0000 mon.b [INF] from='client.? v1:172.21.15.181:0/2689409629' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi181-33015-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-05-25T12:00:59.488 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:00:58.428151+0000 mon.b [INF] from='client.? v1:172.21.15.181:0/2399728201' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi181-32850-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-05-25T12:00:59.488 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:00:58.428734+0000 mon.c [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2021-05-25T12:00:59.488 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:00:58.429056+0000 mon.b [INF] from='client.? v1:172.21.15.181:0/2821227853' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi181-33057-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-05-25T12:00:59.488 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:00:58.431464+0000 mon.c [INF] from='client.74698 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi181-33162-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi181-33162-7"}]: dispatch 2021-05-25T12:00:59.488 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:00:58.432240+0000 mon.c [INF] from='client.74707 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi181-33335-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-05-25T12:00:59.489 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:00:58.432561+0000 mon.c [INF] from='client.74530 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi181-33015-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-05-25T12:00:59.489 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:00:58.432773+0000 mon.c [INF] from='client.74704 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi181-32850-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-05-25T12:00:59.489 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:00:58.433009+0000 mon.c [INF] from='client.74701 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi181-33057-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-05-25T12:00:59.489 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:00:58.480219+0000 mon.c [INF] from='client.? v1:172.21.15.181:0/322065534' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi181-32930-10"}]: dispatch 2021-05-25T12:00:59.490 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:00:58.481718+0000 mon.b [INF] from='client.? v1:172.21.15.181:0/3023005921' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2021-05-25T12:00:59.490 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:00:58.481861+0000 mon.c [INF] from='client.? v1:172.21.15.181:0/322065534' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi181-32930-10"}]: dispatch 2021-05-25T12:00:59.490 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:00:58.481908+0000 client.admin [INF] onexx 2021-05-25T12:00:59.490 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:00:58.482039+0000 mon.c [INF] from='client.74728 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2021-05-25T12:00:59.491 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:00:58.482552+0000 mon.c [INF] from='client.? v1:172.21.15.181:0/322065534' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi181-32930-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2021-05-25T12:00:59.491 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:00:59.362184+0000 mon.c [INF] from='client.? v1:172.21.15.181:0/1206631197' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi181-32863-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi181-32863-22"}]': finished 2021-05-25T12:00:59.491 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:00:59.362255+0000 mon.c [INF] from='client.? v1:172.21.15.181:0/3650250362' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi181-33095-6", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi181-33095-6"}]': finished 2021-05-25T12:00:59.491 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:00:59.362305+0000 mon.c [INF] from='client.74656 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi181-32858-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi181-32858-15"}]': finished 2021-05-25T12:01:00.397 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:00:59.382022+ delete_pools_parallel: process_1_[33524]: starting. 2021-05-25T12:01:00.398 INFO:tasks.workunit.client.0.smithi181.stdout: delete_pools_parallel: process_1_[33524]: creating pool ceph_test_rados_delete_pools_parallel.smithi181-33460 2021-05-25T12:01:00.398 INFO:tasks.workunit.client.0.smithi181.stdout: delete_pools_parallel: process_1_[33524]: created object 0... 2021-05-25T12:01:00.398 INFO:tasks.workunit.client.0.smithi181.stdout: delete_pools_parallel: process_1_[33524]: created object 25... 2021-05-25T12:01:00.399 INFO:tasks.workunit.client.0.smithi181.stdout: delete_pools_parallel: process_1_[33524]: created object 49... 2021-05-25T12:01:00.399 INFO:tasks.workunit.client.0.smithi181.stdout: delete_pools_parallel: process_1_[33524]: finishing. 2021-05-25T12:01:00.399 INFO:tasks.workunit.client.0.smithi181.stdout: delete_pools_parallel: process_1_[33524]: shutting down. 2021-05-25T12:01:00.399 INFO:tasks.workunit.client.0.smithi181.stdout: delete_pools_parallel: process_2_[33525]: starting. 2021-05-25T12:01:00.399 INFO:tasks.workunit.client.0.smithi181.stdout: delete_pools_parallel: process_2_[33525]: deleting pool ceph_test_rados_delete_pools_parallel.smithi181-33460 2021-05-25T12:01:00.400 INFO:tasks.workunit.client.0.smithi181.stdout: delete_pools_parallel: process_2_[33525]: shutting down. 2021-05-25T12:01:00.400 INFO:tasks.workunit.client.0.smithi181.stdout: delete_pools_parallel: ******************************* 2021-05-25T12:01:00.400 INFO:tasks.workunit.client.0.smithi181.stdout: delete_pools_parallel: process_3_[34360]: starting. 2021-05-25T12:01:00.400 INFO:tasks.workunit.client.0.smithi181.stdout: delete_pools_parallel: process_3_[34360]: creating pool ceph_test_rados_delete_pools_parallel.smithi181-33460 2021-05-25T12:01:00.401 INFO:tasks.workunit.client.0.smithi181.stdout: delete_pools_parallel: process_3_[34360]: created object 0... 2021-05-25T12:01:00.401 INFO:tasks.workunit.client.0.smithi181.stdout: delete_pools_parallel: process_3_[34360]: created object 25... 2021-05-25T12:01:00.401 INFO:tasks.workunit.client.0.smithi181.stdout: delete_pools_parallel: process_3_[34360]: created object 49... 2021-05-25T12:01:00.401 INFO:tasks.workunit.client.0.smithi181.stdout: delete_pools_parallel: process_3_[34360]: finishing. 2021-05-25T12:01:00.401 INFO:tasks.workunit.client.0.smithi181.stdout: delete_pools_parallel: process_3_[34360]: shutting down. 2021-05-25T12:01:00.402 INFO:tasks.workunit.client.0.smithi181.stdout: delete_pools_parallel: ******************************* 2021-05-25T12:01:00.402 INFO:tasks.workunit.client.0.smithi181.stdout: delete_pools_parallel: process_5_[34362]: starting. 2021-05-25T12:01:00.402 INFO:tasks.workunit.client.0.smithi181.stdout: delete_pools_parallel: process_5_[34362]: listing objects. 2021-05-25T12:01:00.402 INFO:tasks.workunit.client.0.smithi181.stdout: delete_pools_parallel: process_5_[34362]: listed object 0... 2021-05-25T12:01:00.402 INFO:tasks.workunit.client.0.smithi181.stdout: delete_pools_parallel: process_5_[34362]: listed object 25... 2021-05-25T12:01:00.403 INFO:tasks.workunit.client.0.smithi181.stdout: delete_pools_parallel: process_5_[34362]: saw 50 objects 2021-05-25T12:01:00.403 INFO:tasks.workunit.client.0.smithi181.stdout: delete_pools_parallel: process_5_[34362]: shutting down. 2021-05-25T12:01:00.403 INFO:tasks.workunit.client.0.smithi181.stdout: delete_pools_parallel: ******************************* 2021-05-25T12:01:00.403 INFO:tasks.workunit.client.0.smithi181.stdout: delete_pools_parallel: process_4_[34361]: starting. 2021-05-25T12:01:00.404 INFO:tasks.workunit.client.0.smithi181.stdout: delete_pools_parallel: process_4_[34361]: deleting pool ceph_test_rados_delete_pools_parallel.smithi181-33460 2021-05-25T12:01:00.404 INFO:tasks.workunit.client.0.smithi181.stdout: delete_pools_parallel: process_4_[34361]: shutting down. 2021-05-25T12:01:00.404 INFO:tasks.workunit.client.0.smithi181.stdout: delete_pools_parallel: ******************************* 2021-05-25T12:01:00.404 INFO:tasks.workunit.client.0.smithi181.stdout: delete_pools_parallel: ******************************* 2021-05-25T12:01:00.404 INFO:tasks.workunit.client.0.smithi181.stdout: delete_pools_parallel: ******* SUCCESS ********** 2021-05-25T12:01:00.513 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: Running main() from gmock_main.cc 2021-05-25T12:01:00.514 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [==========] Running 37 tests from 2 test suites. 2021-05-25T12:01:00.514 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [----------] Global test environment set-up. 2021-05-25T12:01:00.514 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP 2021-05-25T12:01:00.514 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: seed 32863 2021-05-25T12:01:00.515 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TooBigPP 2021-05-25T12:01:00.515 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [ OK ] LibRadosIoPP.TooBigPP (0 ms) 2021-05-25T12:01:00.515 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SimpleWritePP 2021-05-25T12:01:00.515 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [ OK ] LibRadosIoPP.SimpleWritePP (17 ms) 2021-05-25T12:01:00.515 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadOpPP 2021-05-25T12:01:00.516 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadOpPP (15 ms) 2021-05-25T12:01:00.516 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadOpPP 2021-05-25T12:01:00.516 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadOpPP (2 ms) 2021-05-25T12:01:00.516 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP 2021-05-25T12:01:00.517 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP (2 ms) 2021-05-25T12:01:00.517 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP2 2021-05-25T12:01:00.517 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP2 (3 ms) 2021-05-25T12:01:00.517 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [ RUN ] LibRadosIoPP.Checksum 2021-05-25T12:01:00.518 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [ OK ] LibRadosIoPP.Checksum (4 ms) 2021-05-25T12:01:00.518 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadIntoBufferlist 2021-05-25T12:01:00.518 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadIntoBufferlist (5 ms) 2021-05-25T12:01:00.518 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [ RUN ] LibRadosIoPP.OverlappingWriteRoundTripPP 2021-05-25T12:01:00.518 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [ OK ] LibRadosIoPP.OverlappingWriteRoundTripPP (6 ms) 2021-05-25T12:01:00.519 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP 2021-05-25T12:01:00.519 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP (10 ms) 2021-05-25T12:01:00.519 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP2 2021-05-25T12:01:00.519 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP2 (6 ms) 2021-05-25T12:01:00.520 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [ RUN ] LibRadosIoPP.AppendRoundTripPP 2021-05-25T12:01:00.520 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [ OK ] LibRadosIoPP.AppendRoundTripPP (9 ms) 2021-05-25T12:01:00.520 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TruncTestPP 2021-05-25T12:01:00.520 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [ OK ] LibRadosIoPP.TruncTestPP (8 ms) 2021-05-25T12:01:00.520 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RemoveTestPP 2021-05-25T12:01:00.521 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [ OK ] LibRadosIoPP.RemoveTestPP (8 ms) 2021-05-25T12:01:00.521 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrsRoundTripPP 2021-05-25T12:01:00.521 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrsRoundTripPP (9 ms) 2021-05-25T12:01:00.521 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RmXattrPP 2021-05-25T12:01:00.521 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [ OK ] LibRadosIoPP.RmXattrPP (31 ms) 2021-05-25T12:01:00.522 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrListPP 2021-05-25T12:01:00.522 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrListPP (12 ms) 2021-05-25T12:01:00.522 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtPP 2021-05-25T12:01:00.522 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtPP (11 ms) 2021-05-25T12:01:00.523 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtDNEPP 2021-05-25T12:01:00.523 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtDNEPP (8 ms) 2021-05-25T12:01:00.523 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtMismatchPP 2021-05-25T12:01:00.523 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtMismatchPP (5 ms) 2021-05-25T12:01:00.523 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP (171 ms total) 2021-05-25T12:01:00.524 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: 2021-05-25T12:01:00.524 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP 2021-05-25T12:01:00.524 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SimpleWritePP 2021-05-25T12:01:00.524 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SimpleWritePP (1094 ms) 2021-05-25T12:01:00.524 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.ReadOpPP 2021-05-25T12:01:00.525 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [ OK ] LibRadosIoECPP.ReadOpPP (15 ms) 2021-05-25T12:01:00.525 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SparseReadOpPP 2021-05-25T12:01:00.525 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SparseReadOpPP (5 ms) 2021-05-25T12:01:00.525 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP 2021-05-25T12:01:00.526 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP (4 ms) 2021-05-25T12:01:00.526 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP2 2021-05-25T12:01:00.526 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP2 (6 ms) 2021-05-25T12:01:01.392 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [ RUN open_pools_parallel: process_1_[33486]: starting. 2021-05-25T12:01:01.393 INFO:tasks.workunit.client.0.smithi181.stdout: open_pools_parallel: process_1_[33486]: creating pool ceph_test_rados_open_pools_parallel.smithi181-33428 2021-05-25T12:01:01.393 INFO:tasks.workunit.client.0.smithi181.stdout: open_pools_parallel: process_1_[33486]: created object 0... 2021-05-25T12:01:01.393 INFO:tasks.workunit.client.0.smithi181.stdout: open_pools_parallel: process_1_[33486]: created object 25... 2021-05-25T12:01:01.393 INFO:tasks.workunit.client.0.smithi181.stdout: open_pools_parallel: process_1_[33486]: created object 49... 2021-05-25T12:01:01.394 INFO:tasks.workunit.client.0.smithi181.stdout: open_pools_parallel: process_1_[33486]: finishing. 2021-05-25T12:01:01.394 INFO:tasks.workunit.client.0.smithi181.stdout: open_pools_parallel: process_1_[33486]: shutting down. 2021-05-25T12:01:01.394 INFO:tasks.workunit.client.0.smithi181.stdout: open_pools_parallel: process_2_[33488]: starting. 2021-05-25T12:01:01.394 INFO:tasks.workunit.client.0.smithi181.stdout: open_pools_parallel: process_2_[33488]: rados_pool_create. 2021-05-25T12:01:01.394 INFO:tasks.workunit.client.0.smithi181.stdout: open_pools_parallel: process_2_[33488]: rados_ioctx_create. 2021-05-25T12:01:01.395 INFO:tasks.workunit.client.0.smithi181.stdout: open_pools_parallel: process_2_[33488]: shutting down. 2021-05-25T12:01:01.395 INFO:tasks.workunit.client.0.smithi181.stdout: open_pools_parallel: ******************************* 2021-05-25T12:01:01.395 INFO:tasks.workunit.client.0.smithi181.stdout: open_pools_parallel: process_3_[34528]: starting. 2021-05-25T12:01:01.395 INFO:tasks.workunit.client.0.smithi181.stdout: open_pools_parallel: process_3_[34528]: creating pool ceph_test_rados_open_pools_parallel.smithi181-33428 2021-05-25T12:01:01.396 INFO:tasks.workunit.client.0.smithi181.stdout: open_pools_parallel: process_3_[34528]: created object 0... 2021-05-25T12:01:01.396 INFO:tasks.workunit.client.0.smithi181.stdout: open_pools_parallel: process_3_[34528]: created object 25... 2021-05-25T12:01:01.396 INFO:tasks.workunit.client.0.smithi181.stdout: open_pools_parallel: process_3_[34528]: created object 49... 2021-05-25T12:01:01.396 INFO:tasks.workunit.client.0.smithi181.stdout: open_pools_parallel: process_3_[34528]: finishing. 2021-05-25T12:01:01.396 INFO:tasks.workunit.client.0.smithi181.stdout: open_pools_parallel: process_3_[34528]: shutting down. 2021-05-25T12:01:01.397 INFO:tasks.workunit.client.0.smithi181.stdout: open_pools_parallel: ******************************* 2021-05-25T12:01:01.397 INFO:tasks.workunit.client.0.smithi181.stdout: open_pools_parallel: process_4_[34529]: starting. 2021-05-25T12:01:01.397 INFO:tasks.workunit.client.0.smithi181.stdout: open_pools_parallel: process_4_[34529]: rados_pool_create. 2021-05-25T12:01:01.397 INFO:tasks.workunit.client.0.smithi181.stdout: open_pools_parallel: process_4_[34529]: rados_ioctx_create. 2021-05-25T12:01:01.398 INFO:tasks.workunit.client.0.smithi181.stdout: open_pools_parallel: process_4_[34529]: shutting down. 2021-05-25T12:01:01.398 INFO:tasks.workunit.client.0.smithi181.stdout: open_pools_parallel: ******************************* 2021-05-25T12:01:01.398 INFO:tasks.workunit.client.0.smithi181.stdout: open_pools_parallel: ******************************* 2021-05-25T12:01:01.398 INFO:tasks.workunit.client.0.smithi181.stdout: open_pools_parallel: ******* SUCCESS ********** 2021-05-25T12:01:02.538 INFO:tasks.workunit.client.0.smithi181.stdout:0000 mon.c [INF] from='client.74707 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi181-33335-2","app": "rados","yes_i_really_mean_it": true}]': finished 2021-05-25T12:01:02.539 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:00:59.382110+0000 mon.c [INF] from='client.74530 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi181-33015-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2021-05-25T12:01:02.539 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:00:59.382184+0000 mon.c [INF] from='client.74704 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi181-32850-2","app": "rados","yes_i_really_mean_it": true}]': finished 2021-05-25T12:01:02.540 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:00:59.382251+0000 mon.c [INF] from='client.74701 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi181-33057-2","app": "rados","yes_i_really_mean_it": true}]': finished 2021-05-25T12:01:02.540 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:00:59.382532+0000 mon.c [INF] from='client.? v1:172.21.15.181:0/322065534' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi181-32930-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2021-05-25T12:01:02.540 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:00:59.385109+0000 mon.c [INF] from='client.? v1:172.21.15.181:0/322065534' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi181-32930-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi181-32930-10"}]: dispatch 2021-05-25T12:01:02.541 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:00:59.391215+0000 mon.b [INF] from='client.? v1:172.21.15.181:0/2689409629' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi181-33015-4", "tierpool": "test-rados-api-smithi181-33015-4-cache"}]: dispatch 2021-05-25T12:01:02.541 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:00:59.428349+0000 mon.c [INF] from='client.74530 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi181-33015-4", "tierpool": "test-rados-api-smithi181-33015-4-cache"}]: dispatch 2021-05-25T12:01:02.542 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:01:00.534447+0000 mon.c [INF] from='client.74728 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2021-05-25T12:01:02.542 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:01:01.368046+0000 mon.c [INF] from='client.? v1:172.21.15.181:0/322065534' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi181-32930-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi181-32930-10"}]': finished 2021-05-25T12:01:02.543 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:01:01.368106+0000 mon.c [INF] from='client.74704 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi181-32850-3","app": "rados","yes_i_really_mean_it": true}]': finished 2021-05-25T12:01:02.543 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:01:01.368156+0000 mon.c [INF] from='client.74530 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi181-33015-4", "overlaypool": "test-rados-api-smithi181-33015-4-cache"}]': finished 2021-05-25T12:01:02.544 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:01:01.369388+0000 mon.b [INF] from='client.? v1:172.21.15.181:0/2689409629' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi181-33015-4-cache", "mode": "writeback"}]: dispatch 2021-05-25T12:01:02.544 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:01:01.370030+0000 mon.b [INF] from='client.? v1:172.21.15.181:0/2399728201' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "test-rados-api-smithi181-32850-3", "field": "max_bytes", "val": "4096"}]: dispatch 2021-05-25T12:01:02.544 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:01:01.386451+0000 mon.c [INF] from='client.74743 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi181-32842-3","app": "rados","yes_i_really_mean_it": true}]': finished 2021-05-25T12:01:02.545 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:01:01.386525+0000 mon.c [INF] from='client.? v1:172.21.15.181:0/758857343' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi181-33204-1"}]': finished 2021-05-25T12:01:02.545 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:01:01.387442+0000 mon.b [INF] from='client.? v1:172.21.15.181:0/2768366830' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi181-32858-15"}]: dispatch 2021-05-25T12:01:02.546 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:01:01.387966+0000 mon.b [INF] from='client.? v1:172.21.15.181:0/2534779053' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi181-33057-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-05-25T12:01:02.546 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:01:01.388287+0000 mon.b [INF] from='client.? v1:172.21.15.181:0/2517665063' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi181-33335-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-05-25T12:01:02.547 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:01:01.389328+0000 mon.b [INF] from='client.74758 v1:172.21.15.181:0/141560728' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi181-32978-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-05-25T12:01:02.547 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:01:01.389899+0000 mon.c [INF] from='client.74530 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi181-33015-4-cache", "mode": "writeback"}]: dispatch 2021-05-25T12:01:02.548 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:01:01.389963+0000 mon.b [INF] from='client.74758 v1:172.21.15.181:0/141560728' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi181-32978-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-05-25T12:01:02.548 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:01:01.390140+0000 mon.c [INF] from='client.? v1:172.21.15.181:0/758857343' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi181-33204-1"}]: dispatch 2021-05-25T12:01:02.548 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:01:01.390790+0000 mon.c [INF] from='client.74704 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "test-rados-api-smithi181-32850-3", "field": "max_bytes", "val": "4096"}]: dispatch 2021-05-25T12:01:02.549 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:01:01.391123+0000 mon.c [INF] from='client.74656 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi181-32858-15"}]: dispatch 2021-05-25T12:01:02.549 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:01:01.391316+0000 mon.c [INF] from='client.74752 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi181-33057-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-05-25T12:01:02.550 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:01:01.391522+0000 mon.c [INF] from='client.74764 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi181-33335-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-05-25T12:01:02.550 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:01:01.391877+0000 mon.c [INF] from='client.74758 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi181-32978-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-05-25T12:01:02.550 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:01:01.392040+0000 mon.c [INF] from='client.? v1:172.21.15.181:0/3650250362' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi181-33095-6"}]: dispatch 2021-05-25T12:01:02.551 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:01:01.392218+0000 mon.c [INF] from='client.74758 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi181-32978-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-05-25T12:01:02.552 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:01:01.392409+0000 mon.c [INF] from='client.? v1:172.21.15.181:0/1206631197' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi181-32863-22"}]: dispatch 2021-05-25T12:01:02.553 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:01:02.367023+0000 mon.c [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2021-05-25T12:01:02.553 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:01:02.372936+0000 mon.c [INF] from='client.74749 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi181-32890-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi181-32890-10"}]': finished 2021-05-25T12:01:02.553 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:01:02.373005+0000 mon.c [INF] from='client.74530 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi181-33015-4-cache", "mode": "writeback"}]': finished 2021-05-25T12:01:02.554 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:01:02.373060+0000 mon.c [INF] from='client.? v1:172.21.15.181:0/758857343' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi181-33204-1"}]': finished 2021-05-25T12:01:02.554 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:01:02.373096+0000 mon.c [INF] from='client.74704 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "test-rados-api-smithi181-32850-3", "field": "max_bytes", "val": "4096"}]': finished 2021-05-25T12:01:02.555 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:01:02.373167+0000 mon.c [INF] from='client.74656 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi181-32858-15"}]': finished 2021-05-25T12:01:02.555 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:01:02.373212+0000 mon.c [INF] from='client.74752 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi181-33057-3","app": "rados","yes_i_really_mean_it": true}]': finished 2021-05-25T12:01:02.556 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:01:02.373259+0000 mon.c [INF] from='client.74764 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi181-33335-3","app": "rados","yes_i_really_mean_it": true}]': finished 2021-05-25T12:01:02.556 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:01:02.373307+0000 mon.c [INF] from='client.74758 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi181-32978-1","app": "rados","yes_i_really_mean_it": true}]': finished 2021-05-25T12:01:02.557 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:01:02.373349+0000 mon.c [INF] from='client.? v1:172.21.15.181:0/3650250362' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi181-33095-6"}]': finished 2021-05-25T12:01:02.557 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:01:02.373384+0000 mon.c [INF] from='client.74758 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi181-32978-1","app": "rados","yes_i_really_mean_it": true}]': finished 2021-05-25T12:01:02.557 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:01:02.373442+0000 mon.c [INF] from='client.? v1:172.21.15.181:0/1206631197' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi181-32863-22"}]': finished 2021-05-25T12:01:02.558 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:01:02.374827+0000 mon.c [INF] from='client.? v1:172.21.15.181:0/1206631197' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi181-32863-22"}]: dispatch 2021-05-25T12:01:02.558 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:01:02.375029+0000 mon.b [INF] from='client.? v1:172.21.15.181:0/2768366830' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi181-32858-15"}]: dispatch 2021-05-25T12:01:02.559 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:01:02.375128+0000 mon.c [INF] from='client.? v1:172.21.15.181:0/3650250362' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi181-33095-6"}]: dispatch 2021-05-25T12:01:02.559 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:01:02.375707+0000 mon.c [INF] from='client.74656 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi181-32858-15"}]: dispatch 2021-05-25T12:01:03.379 INFO:tasks.workunit.client.0.smithi181.stdout: ap ] LibRadosIoECPP.OverlappingWriteRoundTripPP 2021-05-25T12:01:03.379 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [ OK ] LibRadosIoECPP.OverlappingWriteRoundTripPP (12 ms) 2021-05-25T12:01:03.379 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP 2021-05-25T12:01:03.380 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP (12 ms) 2021-05-25T12:01:03.380 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP2 2021-05-25T12:01:03.380 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP2 (9 ms) 2021-05-25T12:01:03.380 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.AppendRoundTripPP 2021-05-25T12:01:03.380 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [ OK ] LibRadosIoECPP.AppendRoundTripPP (9 ms) 2021-05-25T12:01:03.381 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.TruncTestPP 2021-05-25T12:01:03.381 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [ OK ] LibRadosIoECPP.TruncTestPP (5 ms) 2021-05-25T12:01:03.381 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RemoveTestPP 2021-05-25T12:01:03.381 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RemoveTestPP (13 ms) 2021-05-25T12:01:03.381 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrsRoundTripPP 2021-05-25T12:01:03.382 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrsRoundTripPP (6 ms) 2021-05-25T12:01:03.382 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RmXattrPP 2021-05-25T12:01:03.382 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RmXattrPP (21 ms) 2021-05-25T12:01:03.382 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrListPP 2021-05-25T12:01:03.383 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrListPP (6 ms) 2021-05-25T12:01:03.383 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtPP 2021-05-25T12:01:03.383 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtPP (11 ms) 2021-05-25T12:01:03.383 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtDNEPP 2021-05-25T12:01:03.383 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtDNEPP (3 ms) 2021-05-25T12:01:03.384 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtMismatchPP 2021-05-25T12:01:03.384 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtMismatchPP (4 ms) 2021-05-25T12:01:03.384 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP (1236 ms total) 2021-05-25T12:01:03.384 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: 2021-05-25T12:01:03.384 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [----------] Global test environment tear-down 2021-05-25T12:01:03.385 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [==========] 37 tests from 2 test suites ran. (23686 ms total) 2021-05-25T12:01:03.385 INFO:tasks.workunit.client.0.smithi181.stdout: api_io_pp: [ PASSED ] 37 tests. 2021-05-25T12:01:03.385 INFO:tasks.workunit.client.0.smithi181.stdout: api_stat: Running main() from gmock_main.cc 2021-05-25T12:01:03.386 INFO:tasks.workunit.client.0.smithi181.stdout: api_stat: [==========] Running 8 tests from 2 test suites. 2021-05-25T12:01:03.386 INFO:tasks.workunit.client.0.smithi181.stdout: api_stat: [----------] Global test environment set-up. 2021-05-25T12:01:03.386 INFO:tasks.workunit.client.0.smithi181.stdout: api_stat: [----------] 4 tests from LibRadosStat 2021-05-25T12:01:03.386 INFO:tasks.workunit.client.0.smithi181.stdout: api_stat: [ RUN ] LibRadosStat.Stat 2021-05-25T12:01:03.387 INFO:tasks.workunit.client.0.smithi181.stdout: api_stat: [ OK ] LibRadosStat.Stat (12 ms) 2021-05-25T12:01:03.387 INFO:tasks.workunit.client.0.smithi181.stdout: api_stat: [ RUN ] LibRadosStat.StatNS 2021-05-25T12:01:03.387 INFO:tasks.workunit.client.0.smithi181.stdout: api_stat: [ OK ] LibRadosStat.StatNS (17 ms) 2021-05-25T12:01:03.387 INFO:tasks.workunit.client.0.smithi181.stdout: api_stat: [ RUN ] LibRadosStat.ClusterStat 2021-05-25T12:01:03.387 INFO:tasks.workunit.client.0.smithi181.stdout: api_stat: [ OK ] LibRadosStat.ClusterStat (9 ms) 2021-05-25T12:01:03.388 INFO:tasks.workunit.client.0.smithi181.stdout: api_stat: [ RUN ] LibRadosStat.PoolStat 2021-05-25T12:01:03.388 INFO:tasks.workunit.client.0.smithi181.stdout: api_stat: [ OK ] LibRadosStat.PoolStat (2 ms) 2021-05-25T12:01:03.388 INFO:tasks.workunit.client.0.smithi181.stdout: api_stat: [----------] 4 tests from LibRadosStat (40 ms total) 2021-05-25T12:01:03.388 INFO:tasks.workunit.client.0.smithi181.stdout: api_stat: 2021-05-25T12:01:03.388 INFO:tasks.workunit.client.0.smithi181.stdout: api_stat: [----------] 4 tests from LibRadosStatEC 2021-05-25T12:01:03.389 INFO:tasks.workunit.client.0.smithi181.stdout: api_stat: [ RUN ] LibRadosStatEC.Stat 2021-05-25T12:01:03.389 INFO:tasks.workunit.client.0.smithi181.stdout: api_stat: [ OK ] LibRadosStatEC.Stat (1087 ms) 2021-05-25T12:01:03.389 INFO:tasks.workunit.client.0.smithi181.stdout: api_stat: [ RUN ] LibRadosStatEC.StatNS 2021-05-25T12:01:03.389 INFO:tasks.workunit.client.0.smithi181.stdout: api_stat: [ OK ] LibRadosStatEC.StatNS (20 ms) 2021-05-25T12:01:03.390 INFO:tasks.workunit.client.0.smithi181.stdout: api_stat: [ RUN ] LibRadosStatEC.ClusterStat 2021-05-25T12:01:03.390 INFO:tasks.workunit.client.0.smithi181.stdout: api_stat: [ OK ] LibRadosStatEC.ClusterStat (33 ms) 2021-05-25T12:01:03.390 INFO:tasks.workunit.client.0.smithi181.stdout: api_stat: [ RUN ] LibRadosStatEC.PoolStat 2021-05-25T12:01:03.390 INFO:tasks.workunit.client.0.smithi181.stdout: api_stat: [ OK ] LibRadosStatEC.PoolStat (6 ms) 2021-05-25T12:01:03.390 INFO:tasks.workunit.client.0.smithi181.stdout: api_stat: [----------] 4 tests from LibRadosStatEC (1146 ms total) 2021-05-25T12:01:03.391 INFO:tasks.workunit.client.0.smithi181.stdout: api_stat: 2021-05-25T12:01:03.391 INFO:tasks.workunit.client.0.smithi181.stdout: api_stat: [----------] Global test environment tear-down 2021-05-25T12:01:03.391 INFO:tasks.workunit.client.0.smithi181.stdout: api_stat: [==========] 8 tests from 2 test suites ran. (23611 ms total) 2021-05-25T12:01:03.391 INFO:tasks.workunit.client.0.smithi181.stdout: api_stat: [ PASSED ] 8 tests. 2021-05-25T12:01:03.392 INFO:tasks.workunit.client.0.smithi181.stdout: api_io: Running main() from gmock_main.cc 2021-05-25T12:01:03.392 INFO:tasks.workunit.client.0.smithi181.stdout: api_io: [==========] Running 23 tests from 2 test suites. 2021-05-25T12:01:03.392 INFO:tasks.workunit.client.0.smithi181.stdout: api_io: [----------] Global test environment set-up. 2021-05-25T12:01:03.393 INFO:tasks.workunit.client.0.smithi181.stdout: api_io: [----------] 13 tests from LibRadosIo 2021-05-25T12:01:03.393 INFO:tasks.workunit.client.0.smithi181.stdout: api_io: [ RUN ] LibRadosIo.SimpleWrite 2021-05-25T12:01:03.393 INFO:tasks.workunit.client.0.smithi181.stdout: api_io: [ OK ] LibRadosIo.SimpleWrite (24 ms) 2021-05-25T12:01:03.393 INFO:tasks.workunit.client.0.smithi181.stdout: api_io: [ RUN ] LibRadosIo.TooBig 2021-05-25T12:01:03.393 INFO:tasks.workunit.client.0.smithi181.stdout: api_io: [ OK ] LibRadosIo.TooBig (0 ms) 2021-05-25T12:01:03.394 INFO:tasks.workunit.client.0.smithi181.stdout: api_io: [ RUN ] LibRadosIo.ReadTimeout 2021-05-25T12:01:03.394 INFO:tasks.workunit.client.0.smithi181.stdout: api_io: no timeout :/ 2021-05-25T12:01:03.394 INFO:tasks.workunit.client.0.smithi181.stdout: api_io: no timeout :/ 2021-05-25T12:01:03.394 INFO:tasks.workunit.client.0.smithi181.stdout: api_io: no timeout :/ 2021-05-25T12:01:03.394 INFO:tasks.workunit.client.0.smithi181.stdout: api_io: no timeout :/ 2021-05-25T12:01:03.395 INFO:tasks.workunit.client.0.smithi181.stdout: api_io: no timeout :/ 2021-05-25T12:01:03.395 INFO:tasks.workunit.client.0.smithi181.stdout: api_io: [ OK ] LibRadosIo.ReadTimeout (25 ms) 2021-05-25T12:01:03.395 INFO:tasks.workunit.client.0.smithi181.stdout: api_io: [ RUN ] LibRadosIo.RoundTrip 2021-05-25T12:01:03.395 INFO:tasks.workunit.client.0.smithi181.stdout: api_io: [ OK ] LibRadosIo.RoundTrip (15 ms) 2021-05-25T12:01:03.396 INFO:tasks.workunit.client.0.smithi181.stdout: api_io: [ RUN ] LibRadosIo.Checksum 2021-05-25T12:01:03.396 INFO:tasks.workunit.client.0.smithi181.stdout: api_io: [ OK ] LibRadosIo.Checksum (4 ms) 2021-05-25T12:01:03.396 INFO:tasks.workunit.client.0.smithi181.stdout: api_io: [ RUN ] LibRadosIo.OverlappingWriteRoundTrip 2021-05-25T12:01:03.396 INFO:tasks.workunit.client.0.smithi181.stdout: api_io: [ OK ] LibRadosIo.OverlappingWriteRoundTrip (8 ms) 2021-05-25T12:01:03.396 INFO:tasks.workunit.client.0.smithi181.stdout: api_io: [ RUN ] LibRadosIo.WriteFullRoundTrip 2021-05-25T12:01:03.397 INFO:tasks.workunit.client.0.smithi181.stdout: api_io: [ OK ] LibRadosIo.WriteFullRoundTrip (11 ms) 2021-05-25T12:01:03.397 INFO:tasks.workunit.client.0.smithi181.stdout: api_io: [ RUN ] LibRadosIo.AppendRoundTrip 2021-05-25T12:01:03.397 INFO:tasks.workunit.client.0.smithi181.stdout: api_io: [ OK ] LibRadosIo.AppendRoundTrip (11 ms) 2021-05-25T12:01:03.397 INFO:tasks.workunit.client.0.smithi181.stdout: api_io: [ RUN ] LibRadosIo.TruncTest 2021-05-25T12:01:03.397 INFO:tasks.workunit.client.0.smithi181.stdout: api_io: [ OK ] LibRadosIo.TruncTest (6 ms) 2021-05-25T12:01:03.398 INFO:tasks.workunit.client.0.smithi181.stdout: api_io: [ RUN ] LibRadosIo.RemoveTest 2021-05-25T12:01:03.398 INFO:tasks.workunit.client.0.smithi181.stdout: api_io: [ OK ] LibRadosIo.RemoveTest (7 ms) 2021-05-25T12:01:03.398 INFO:tasks.workunit.client.0.smithi181.stdout: api_io: [ RUN ] LibRadosIo.XattrsRoundTrip 2021-05-25T12:01:03.398 INFO:tasks.workunit.client.0.smithi181.stdout: api_io: [ OK ] LibRadosIo.XattrsRoundTrip (5 ms) 2021-05-25T12:01:03.399 INFO:tasks.workunit.client.0.smithi181.stdout: api_io: [ RUN ] LibRadosIo.RmXattr 2021-05-25T12:01:03.399 INFO:tasks.workunit.client.0.smithi181.stdout: api_io: [ OK ] LibRadosIo.RmXattr (24 ms) 2021-05-25T12:01:03.399 INFO:tasks.workunit.client.0.smithi181.stdout: api_io: [ RUN ] LibRadosIo.XattrIter 2021-05-25T12:01:03.399 INFO:tasks.workunit.client.0.smithi181.stdout: api_io: [ OK ] LibRadosIo.XattrIter (4 ms) 2021-05-25T12:01:03.399 INFO:tasks.workunit.client.0.smithi181.stdout: api_io: [----------] 13 tests from LibRadosIo (144 ms total) 2021-05-25T12:01:03.400 INFO:tasks.workunit.client.0.smithi181.stdout: api_io: 2021-05-25T12:01:03.400 INFO:tasks.workunit.client.0.smithi181.stdout: api_io: [----------] 10 tests from LibRadosIoEC 2021-05-25T12:01:03.400 INFO:tasks.workunit.client.0.smithi181.stdout: api_io: [ RUN ] LibRadosIoEC.SimpleWrite 2021-05-25T12:01:03.400 INFO:tasks.workunit.client.0.smithi181.stdout: api_io: [ OK ] LibRadosIoEC.SimpleWrite (1103 ms) 2021-05-25T12:01:03.401 INFO:tasks.workunit.client.0.smithi181.stdout: api_io: [ RUN ] LibRadosIoEC.RoundTrip 2021-05-25T12:01:03.401 INFO:tasks.workunit.client.0.smithi181.stdout: api_io: [ OK ] LibRadosIoEC.RoundTrip (13 ms) 2021-05-25T12:01:03.401 INFO:tasks.workunit.client.0.smithi181.stdout: api_io: [ RUN ] LibRadosIoEC.OverlappingWriteRoundTrip 2021-05-25T12:01:03.401 INFO:tasks.workunit.client.0.smithi181.stdout: api_io: [ OK ] LibRadosIoEC.OverlappingWriteRoundTrip (14 ms) 2021-05-25T12:01:03.401 INFO:tasks.workunit.client.0.smithi181.stdout: api_io: [ RUN ] LibRadosIoEC.WriteFullRoundTrip 2021-05-25T12:01:03.402 INFO:tasks.workunit.client.0.smithi181.stdout: api_io: [ OK ] LibRadosIoEC.WriteFullRoundTrip (8 ms) 2021-05-25T12:01:03.402 INFO:tasks.workunit.client.0.smithi181.stdout: api_io: [ RUN ] LibRadosIoEC.AppendRoundTrip 2021-05-25T12:01:03.402 INFO:tasks.workunit.client.0.smithi181.stdout: api_io: [ OK ] LibRadosIoEC.AppendRoundTrip (18 ms) 2021-05-25T12:01:03.402 INFO:tasks.workunit.client.0.smithi181.stdout: api_io: [ RUN ] LibRadosIoEC.TruncTest 2021-05-25T12:01:03.403 INFO:tasks.workunit.client.0.smithi181.stdout: api_io: [ OK ] LibRadosIoEC.TruncTest (12 ms) 2021-05-25T12:01:03.403 INFO:tasks.workunit.client.0.smithi181.stdout: api_io: [ RUN ] LibRadosIoEC.RemoveTest 2021-05-25T12:01:03.403 INFO:tasks.workunit.client.0.smithi181.stdout: api_io: [ OK ] LibRadosIoEC.RemoveTest (8 ms) 2021-05-25T12:01:03.403 INFO:tasks.workunit.client.0.smithi181.stdout: api_io: [ RUN ] LibRadosIoEC.XattrsRoundTrip 2021-05-25T12:01:03.404 INFO:tasks.workunit.client.0.smithi181.stdout: api_io: [ OK ] LibRadosIoEC.XattrsRoundTrip (9 ms) 2021-05-25T12:01:03.404 INFO:tasks.workunit.client.0.smithi181.stdout: api_io: [ RUN ] LibRadosIoEC.RmXattr 2021-05-25T12:01:03.404 INFO:tasks.workunit.client.0.smithi181.stdout: api_io: [ OK ] LibRadosIoEC.RmXattr (31 ms) 2021-05-25T12:01:03.404 INFO:tasks.workunit.client.0.smithi181.stdout: api_io: [ RUN ] LibRadosIoEC.XattrIter 2021-05-25T12:01:03.404 INFO:tasks.workunit.client.0.smithi181.stdout: api_io: [ OK ] LibRadosIoEC.XattrIter (11 ms) 2021-05-25T12:01:03.405 INFO:tasks.workunit.client.0.smithi181.stdout: api_io: [----------] 10 tests from LibRadosIoEC (1227 ms total) 2021-05-25T12:01:03.405 INFO:tasks.workunit.client.0.smithi181.stdout: api_io: 2021-05-25T12:01:03.405 INFO:tasks.workunit.client.0.smithi181.stdout: api_io: [----------] Global test environment tear-down 2021-05-25T12:01:03.405 INFO:tasks.workunit.client.0.smithi181.stdout: api_io: [==========] 23 tests from 2 test suites ran. (23700 ms total) 2021-05-25T12:01:03.405 INFO:tasks.workunit.client.0.smithi181.stdout: api_io: [ PASSED ] 23 tests. 2021-05-25T12:01:03.544 INFO:tasks.workunit.client.0.smithi181.stdout:i_cmd: got: 2021-05-25T12:01:02.377760+0000 mon.b [INF] from='client.? v1:172.21.15.181:0/2288382365' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi181-33162-7"}]: dispatch 2021-05-25T12:01:03.544 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:01:02.379319+0000 mon.c [INF] from='client.74698 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi181-33162-7"}]: dispatch 2021-05-25T12:01:03.544 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:01:02.453885+0000 mon.b [INF] from='client.? v1:172.21.15.181:0/2689409629' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi181-33015-4"}]: dispatch 2021-05-25T12:01:03.544 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:01:02.454594+0000 mon.b [INF] from='client.74758 v1:172.21.15.181:0/141560728' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi181-32978-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2021-05-25T12:01:03.545 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:01:02.486401+0000 mon.c [INF] from='client.74530 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi181-33015-4"}]: dispatch 2021-05-25T12:01:03.545 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:01:02.486757+0000 mon.c [INF] from='client.74758 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi181-32978-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2021-05-25T12:01:03.545 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:01:02.537526+0000 mon.b [INF] from='client.? v1:172.21.15.181:0/3023005921' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2021-05-25T12:01:03.545 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:01:02.542632+0000 client.admin [INF] threexx 2021-05-25T12:01:03.546 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:01:02.542763+0000 mon.c [INF] from='client.74728 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2021-05-25T12:01:03.546 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:01:03.372740+0000 mon.c [INF] from='client.? v1:172.21.15.181:0/1206631197' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi181-32863-22"}]': finished 2021-05-25T12:01:03.546 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:01:03.372790+0000 mon.c [INF] from='client.? v1:172.21.15.181:0/3650250362' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi181-33095-6"}]': finished 2021-05-25T12:01:03.546 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:01:03.372827+0000 mon.c [INF] from='client.74656 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi181-32858-15"}]': finished 2021-05-25T12:01:03.547 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:01:03.372897+0000 mon.c [INF] from='client.74698 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi181-33162-7"}]': finished 2021-05-25T12:01:03.547 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:01:03.373037+0000 mon.c [INF] from='client.74530 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi181-33015-4"}]': finished 2021-05-25T12:01:03.547 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:01:03.373079+0000 mon.c [INF] from='client.74758 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi181-32978-1","app": "app1","yes_i_really_mean_it": true}]': finished 2021-05-25T12:01:03.547 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:01:03.382724+0000 mon.b [INF] from='client.? v1:172.21.15.181:0/2288382365' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi181-33162-7"}]: dispatch 2021-05-25T12:01:03.548 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:01:03.385635+0000 mon.b [INF] from='client.? v1:172.21.15.181:0/2689409629' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi181-33015-4", "tierpool": "test-rados-api-smithi181-33015-4-cache"}]: dispatch 2021-05-25T12:01:03.548 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:01:03.386253+0000 mon.c [INF] from='client.74698 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi181-33162-7"}]: dispatch 2021-05-25T12:01:04.387 INFO:tasks.workunit.client.0.smithi181.stdout: api_stat_pp: Running main() from gmock_main.cc 2021-05-25T12:01:04.388 INFO:tasks.workunit.client.0.smithi181.stdout: api_stat_pp: [==========] Running 9 tests from 2 test suites. 2021-05-25T12:01:04.388 INFO:tasks.workunit.client.0.smithi181.stdout: api_stat_pp: [----------] Global test environment set-up. 2021-05-25T12:01:04.388 INFO:tasks.workunit.client.0.smithi181.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP 2021-05-25T12:01:04.388 INFO:tasks.workunit.client.0.smithi181.stdout: api_stat_pp: seed 33162 2021-05-25T12:01:04.388 INFO:tasks.workunit.client.0.smithi181.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPP 2021-05-25T12:01:04.389 INFO:tasks.workunit.client.0.smithi181.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPP (93 ms) 2021-05-25T12:01:04.389 INFO:tasks.workunit.client.0.smithi181.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.Stat2Mtime2PP 2021-05-25T12:01:04.389 INFO:tasks.workunit.client.0.smithi181.stdout: api_stat_pp: [ OK ] LibRadosStatPP.Stat2Mtime2PP (20 ms) 2021-05-25T12:01:04.389 INFO:tasks.workunit.client.0.smithi181.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.ClusterStatPP 2021-05-25T12:01:04.390 INFO:tasks.workunit.client.0.smithi181.stdout: api_stat_pp: [ OK ] LibRadosStatPP.ClusterStatPP (0 ms) 2021-05-25T12:01:04.390 INFO:tasks.workunit.client.0.smithi181.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.PoolStatPP 2021-05-25T12:01:04.390 INFO:tasks.workunit.client.0.smithi181.stdout: api_stat_pp: [ OK ] LibRadosStatPP.PoolStatPP (26 ms) 2021-05-25T12:01:04.390 INFO:tasks.workunit.client.0.smithi181.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPPNS 2021-05-25T12:01:04.390 INFO:tasks.workunit.client.0.smithi181.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPPNS (11 ms) 2021-05-25T12:01:04.391 INFO:tasks.workunit.client.0.smithi181.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP (150 ms total) 2021-05-25T12:01:04.391 INFO:tasks.workunit.client.0.smithi181.stdout: api_stat_pp: 2021-05-25T12:01:04.391 INFO:tasks.workunit.client.0.smithi181.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP 2021-05-25T12:01:04.391 INFO:tasks.workunit.client.0.smithi181.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPP 2021-05-25T12:01:04.391 INFO:tasks.workunit.client.0.smithi181.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPP (1072 ms) 2021-05-25T12:01:04.392 INFO:tasks.workunit.client.0.smithi181.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.ClusterStatPP 2021-05-25T12:01:04.392 INFO:tasks.workunit.client.0.smithi181.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.ClusterStatPP (0 ms) 2021-05-25T12:01:04.392 INFO:tasks.workunit.client.0.smithi181.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.PoolStatPP 2021-05-25T12:01:04.392 INFO:tasks.workunit.client.0.smithi181.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.PoolStatPP (6 ms) 2021-05-25T12:01:04.392 INFO:tasks.workunit.client.0.smithi181.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPPNS 2021-05-25T12:01:04.393 INFO:tasks.workunit.client.0.smithi181.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPPNS (15 ms) 2021-05-25T12:01:04.393 INFO:tasks.workunit.client.0.smithi181.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP (1093 ms total) 2021-05-25T12:01:04.393 INFO:tasks.workunit.client.0.smithi181.stdout: api_stat_pp: 2021-05-25T12:01:04.393 INFO:tasks.workunit.client.0.smithi181.stdout: api_stat_pp: [----------] Global test environment tear-down 2021-05-25T12:01:04.394 INFO:tasks.workunit.client.0.smithi181.stdout: api_stat_pp: [==========] 9 tests from 2 test suites ran. (24587 ms total) 2021-05-25T12:01:04.394 INFO:tasks.workunit.client.0.smithi181.stdout: api_stat_pp: [ PASSED ] 9 tests. 2021-05-25T12:01:06.555 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:01:03.386579+0000 mon.c [INF] from='client.74530 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi181-33015-4", "tierpool": "test-rados-api-smithi181-33015-4-cache"}]: dispatch 2021-05-25T12:01:06.555 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:01:03.389051+0000 mon.b [INF] from='client.74758 v1:172.21.15.181:0/141560728' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi181-32978-1","app": "app2"}]: dispatch 2021-05-25T12:01:06.556 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:01:03.389380+0000 mon.b [INF] from='client.? v1:172.21.15.181:0/307986492' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi181-33204-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-05-25T12:01:06.556 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:01:03.389756+0000 mon.b [INF] from='client.74758 v1:172.21.15.181:0/141560728' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi181-32978-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2021-05-25T12:01:06.556 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:01:03.391929+0000 mon.c [INF] from='client.74773 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi181-33204-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-05-25T12:01:06.556 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:01:03.392198+0000 mon.c [INF] from='client.74758 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi181-32978-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2021-05-25T12:01:06.556 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:01:03.392604+0000 mon.b [INF] from='client.? v1:172.21.15.181:0/4132951187' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi181-32842-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-05-25T12:01:06.557 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: got: 2021-05-25T12:01:03.392988+0000 mon.c [INF] from='client.74776 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi181-32842-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2021-05-25T12:01:06.557 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: [ OK ] LibRadosCmd.WatchLog (8125 ms) 2021-05-25T12:01:06.557 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: [----------] 4 tests from LibRadosCmd (26734 ms total) 2021-05-25T12:01:06.557 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: 2021-05-25T12:01:06.558 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: [----------] Global test environment tear-down 2021-05-25T12:01:06.558 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: [==========] 4 tests from 1 test suite ran. (26735 ms total) 2021-05-25T12:01:06.558 INFO:tasks.workunit.client.0.smithi181.stdout: api_cmd: [ PASSED ] 4 tests. 2021-05-25T12:01:07.045 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock_pp: Running main() from gmock_main.cc 2021-05-25T12:01:07.046 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock_pp: [==========] Running 16 tests from 2 test suites. 2021-05-25T12:01:07.046 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock_pp: [----------] Global test environment set-up. 2021-05-25T12:01:07.046 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP 2021-05-25T12:01:07.046 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock_pp: seed 32930 2021-05-25T12:01:07.046 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusivePP 2021-05-25T12:01:07.047 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusivePP (17 ms) 2021-05-25T12:01:07.047 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedPP 2021-05-25T12:01:07.047 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedPP (5 ms) 2021-05-25T12:01:07.047 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusiveDurPP 2021-05-25T12:01:07.048 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusiveDurPP (1085 ms) 2021-05-25T12:01:07.048 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedDurPP 2021-05-25T12:01:07.048 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedDurPP (1009 ms) 2021-05-25T12:01:07.048 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockMayRenewPP 2021-05-25T12:01:07.048 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockMayRenewPP (5 ms) 2021-05-25T12:01:07.049 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.UnlockPP 2021-05-25T12:01:07.049 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock_pp: [ OK ] LibRadosLockPP.UnlockPP (10 ms) 2021-05-25T12:01:07.049 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.ListLockersPP 2021-05-25T12:01:07.049 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock_pp: [ OK ] LibRadosLockPP.ListLockersPP (8 ms) 2021-05-25T12:01:07.049 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.BreakLockPP 2021-05-25T12:01:07.050 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock_pp: [ OK ] LibRadosLockPP.BreakLockPP (7 ms) 2021-05-25T12:01:07.050 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP (2146 ms total) 2021-05-25T12:01:07.050 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock_pp: 2021-05-25T12:01:07.050 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP 2021-05-25T12:01:07.050 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusivePP 2021-05-25T12:01:07.051 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusivePP (1050 ms) 2021-05-25T12:01:07.051 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedPP 2021-05-25T12:01:07.051 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedPP (8 ms) 2021-05-25T12:01:07.051 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusiveDurPP 2021-05-25T12:01:07.051 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusiveDurPP (1006 ms) 2021-05-25T12:01:07.052 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedDurPP 2021-05-25T12:01:07.052 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedDurPP (1005 ms) 2021-05-25T12:01:07.052 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockMayRenewPP 2021-05-25T12:01:07.052 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockMayRenewPP (5 ms) 2021-05-25T12:01:07.052 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.UnlockPP 2021-05-25T12:01:07.053 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.UnlockPP (15 ms) 2021-05-25T12:01:07.053 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.ListLockersPP 2021-05-25T12:01:07.053 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.ListLockersPP (5 ms) 2021-05-25T12:01:07.053 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.BreakLockPP 2021-05-25T12:01:07.053 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.BreakLockPP (7 ms) 2021-05-25T12:01:07.054 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP (3102 ms total) 2021-05-25T12:01:07.054 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock_pp: 2021-05-25T12:01:07.054 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock_pp: [----------] Global test environment tear-down 2021-05-25T12:01:07.054 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock_pp: [==========] 16 tests from 2 test suites ran. (27322 ms total) 2021-05-25T12:01:07.055 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock_pp: [ PASSED ] 16 tests. 2021-05-25T12:01:07.056 INFO:tasks.ceph.osd.0.smithi102.stderr:2021-05-25T12:01:07.054+0000 7f38dd34a700 -1 osd.0 pg_epoch: 29 pg[12.3( v 28'10 (0'0,28'10] local-lis/les=16/17 n=3 ec=16/16 lis/c=16/16 les/c/f=17/17/0 sis=16) [0,4] r=0 lpr=16 crt=28'10 lcod 27'9 mlcod 27'9 active+clean+snaptrim trimq=[9~1] ps=[5~1,7~1]] removing snap head 2021-05-25T12:01:08.062 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock: Running main() from gmock_main.cc 2021-05-25T12:01:08.063 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock: [==========] Running 16 tests from 2 test suites. 2021-05-25T12:01:08.063 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock: [----------] Global test environment set-up. 2021-05-25T12:01:08.064 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock: [----------] 8 tests from LibRadosLock 2021-05-25T12:01:08.064 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusive 2021-05-25T12:01:08.065 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock: [ OK ] LibRadosLock.LockExclusive (123 ms) 2021-05-25T12:01:08.065 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock: [ RUN ] LibRadosLock.LockShared 2021-05-25T12:01:08.065 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock: [ OK ] LibRadosLock.LockShared (22 ms) 2021-05-25T12:01:08.066 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusiveDur 2021-05-25T12:01:08.066 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock: [ OK ] LibRadosLock.LockExclusiveDur (1071 ms) 2021-05-25T12:01:08.067 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock: [ RUN ] LibRadosLock.LockSharedDur 2021-05-25T12:01:08.067 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock: [ OK ] LibRadosLock.LockSharedDur (1005 ms) 2021-05-25T12:01:08.067 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock: [ RUN ] LibRadosLock.LockMayRenew 2021-05-25T12:01:08.068 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock: [ OK ] LibRadosLock.LockMayRenew (4 ms) 2021-05-25T12:01:08.068 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock: [ RUN ] LibRadosLock.Unlock 2021-05-25T12:01:08.069 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock: [ OK ] LibRadosLock.Unlock (4 ms) 2021-05-25T12:01:08.069 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock: [ RUN ] LibRadosLock.ListLockers 2021-05-25T12:01:08.070 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock: [ OK ] LibRadosLock.ListLockers (5 ms) 2021-05-25T12:01:08.070 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock: [ RUN ] LibRadosLock.BreakLock 2021-05-25T12:01:08.070 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock: [ OK ] LibRadosLock.BreakLock (4 ms) 2021-05-25T12:01:08.071 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock: [----------] 8 tests from LibRadosLock (2238 ms total) 2021-05-25T12:01:08.071 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock: 2021-05-25T12:01:08.072 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock: [----------] 8 tests from LibRadosLockEC 2021-05-25T12:01:08.072 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusive 2021-05-25T12:01:08.072 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusive (1048 ms) 2021-05-25T12:01:08.073 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock: [ RUN ] LibRadosLockEC.LockShared 2021-05-25T12:01:08.073 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock: [ OK ] LibRadosLockEC.LockShared (11 ms) 2021-05-25T12:01:08.074 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusiveDur 2021-05-25T12:01:08.074 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusiveDur (1006 ms) 2021-05-25T12:01:08.075 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock: [ RUN ] LibRadosLockEC.LockSharedDur 2021-05-25T12:01:08.075 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock: [ OK ] LibRadosLockEC.LockSharedDur (1005 ms) 2021-05-25T12:01:08.075 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock: [ RUN ] LibRadosLockEC.LockMayRenew 2021-05-25T12:01:08.076 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock: [ OK ] LibRadosLockEC.LockMayRenew (11 ms) 2021-05-25T12:01:08.076 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock: [ RUN ] LibRadosLockEC.Unlock 2021-05-25T12:01:08.077 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock: [ OK ] LibRadosLockEC.Unlock (10 ms) 2021-05-25T12:01:08.077 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock: [ RUN ] LibRadosLockEC.ListLockers 2021-05-25T12:01:08.077 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock: [ OK ] LibRadosLockEC.ListLockers (9 ms) 2021-05-25T12:01:08.078 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock: [ RUN ] LibRadosLockEC.BreakLock 2021-05-25T12:01:08.078 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock: [ OK ] LibRadosLockEC.BreakLock (7 ms) 2021-05-25T12:01:08.079 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock: [----------] 8 tests from LibRadosLockEC (3108 ms total) 2021-05-25T12:01:08.079 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock: 2021-05-25T12:01:08.080 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock: [----------] Global test environment tear-down 2021-05-25T12:01:08.080 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock: [==========] 16 tests from 2 test suites ran. (28343 ms total) 2021-05-25T12:01:08.080 INFO:tasks.workunit.client.0.smithi181.stdout: api_lock: [ PASSED ] 16 tests. 2021-05-25T12:01:10.241 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: [==========] Running 4 tests from 1 test suite. 2021-05-25T12:01:10.241 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: [----------] Global test environment set-up. 2021-05-25T12:01:10.241 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP 2021-05-25T12:01:10.242 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterEarly 2021-05-25T12:01:10.242 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterEarly (5039 ms) 2021-05-25T12:01:10.242 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterLate 2021-05-25T12:01:10.242 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterLate (30 ms) 2021-05-25T12:01:10.242 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Status 2021-05-25T12:01:10.243 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: [ OK ] LibRadosServicePP.Status (20023 ms) 2021-05-25T12:01:10.243 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Close 2021-05-25T12:01:10.243 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: attempt 0 of 20 2021-05-25T12:01:10.243 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: failed to deregister: 2021-05-25T12:01:10.243 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: { 2021-05-25T12:01:10.244 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: "epoch": 8, 2021-05-25T12:01:10.244 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: "modified": "2021-05-25T12:01:05.573629+0000", 2021-05-25T12:01:10.244 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: "services": { 2021-05-25T12:01:10.244 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: "laundry": { 2021-05-25T12:01:10.244 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: "daemons": { 2021-05-25T12:01:10.245 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: "summary": "", 2021-05-25T12:01:10.245 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: "close-test-pid33313": { 2021-05-25T12:01:10.245 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: "start_epoch": 8, 2021-05-25T12:01:10.245 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: "start_stamp": "2021-05-25T12:01:04.973810+0000", 2021-05-25T12:01:10.245 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: "gid": 74794, 2021-05-25T12:01:10.246 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: "addr": "172.21.15.181:0/2128284453", 2021-05-25T12:01:10.246 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: "metadata": { 2021-05-25T12:01:10.246 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: "arch": "x86_64", 2021-05-25T12:01:10.247 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: "ceph_release": "quincy", 2021-05-25T12:01:10.247 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: "ceph_version": "ceph version 17.0.0-4502-gece5ed1a (ece5ed1ac99f43a50411c04b4ca92c148baf0cd1) quincy (dev)", 2021-05-25T12:01:10.247 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: "ceph_version_short": "17.0.0-4502-gece5ed1a", 2021-05-25T12:01:10.247 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-05-25T12:01:10.247 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: "distro": "centos", 2021-05-25T12:01:10.248 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: "distro_description": "CentOS Linux 8", 2021-05-25T12:01:10.248 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: "distro_version": "8", 2021-05-25T12:01:10.248 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: "foo": "bar", 2021-05-25T12:01:10.248 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: "hostname": "smithi181", 2021-05-25T12:01:10.249 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: "kernel_description": "#1 SMP Thu Apr 8 19:01:30 UTC 2021", 2021-05-25T12:01:10.249 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: "kernel_version": "4.18.0-240.22.1.el8_3.x86_64", 2021-05-25T12:01:10.249 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: "mem_swap_kb": "0", 2021-05-25T12:01:10.249 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: "mem_total_kb": "32611872", 2021-05-25T12:01:10.249 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: "os": "Linux", 2021-05-25T12:01:10.250 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: "this": "that" 2021-05-25T12:01:10.250 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: }, 2021-05-25T12:01:10.250 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: "task_status": {} 2021-05-25T12:01:10.250 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: }, 2021-05-25T12:01:10.250 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: "pid33294": { 2021-05-25T12:01:10.251 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: "start_epoch": 7, 2021-05-25T12:01:10.251 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: "start_stamp": "2021-05-25T12:00:57.544998+0000", 2021-05-25T12:01:10.251 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: "gid": 74713, 2021-05-25T12:01:10.251 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: "addr": "172.21.15.181:0/304196470", 2021-05-25T12:01:10.251 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: "metadata": { 2021-05-25T12:01:10.252 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: "arch": "x86_64", 2021-05-25T12:01:10.252 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: "ceph_release": "quincy", 2021-05-25T12:01:10.252 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: "ceph_version": "ceph version 17.0.0-4502-gece5ed1a (ece5ed1ac99f43a50411c04b4ca92c148baf0cd1) quincy (dev)", 2021-05-25T12:01:10.252 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: "ceph_version_short": "17.0.0-4502-gece5ed1a", 2021-05-25T12:01:10.252 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-05-25T12:01:11.209 INFO:tasks.workunit.client.0.smithi181.stdout: ap api_watch_notify_pp: Running main() from gmock_main.cc 2021-05-25T12:01:11.210 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify_pp: [==========] Running 16 tests from 2 test suites. 2021-05-25T12:01:11.210 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify_pp: [----------] Global test environment set-up. 2021-05-25T12:01:11.210 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP 2021-05-25T12:01:11.210 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotify 2021-05-25T12:01:11.211 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify_pp: notify 2021-05-25T12:01:11.211 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotify (1032 ms) 2021-05-25T12:01:11.211 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotifyTimeout 2021-05-25T12:01:11.211 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotifyTimeout (7 ms) 2021-05-25T12:01:11.211 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP (1039 ms total) 2021-05-25T12:01:11.212 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify_pp: 2021-05-25T12:01:11.212 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP 2021-05-25T12:01:11.212 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 2021-05-25T12:01:11.212 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify_pp: notify 2021-05-25T12:01:11.213 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 (42 ms) 2021-05-25T12:01:11.213 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 2021-05-25T12:01:11.213 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify_pp: notify 2021-05-25T12:01:11.213 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 (3710 ms) 2021-05-25T12:01:11.213 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 2021-05-25T12:01:11.214 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 (6 ms) 2021-05-25T12:01:11.214 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 2021-05-25T12:01:11.214 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 (10 ms) 2021-05-25T12:01:11.214 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 2021-05-25T12:01:11.215 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify_pp: handle_notify cookie 94792731065824 notify_id 128849018882 notifier_gid 74773 2021-05-25T12:01:11.215 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 (8 ms) 2021-05-25T12:01:11.215 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 2021-05-25T12:01:11.215 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify_pp: handle_notify cookie 94792742243984 notify_id 128849018883 notifier_gid 74773 2021-05-25T12:01:11.215 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 (9 ms) 2021-05-25T12:01:11.216 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 2021-05-25T12:01:11.216 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify_pp: handle_notify cookie 94792742243984 notify_id 128849018882 notifier_gid 74773 2021-05-25T12:01:11.216 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 (10 ms) 2021-05-25T12:01:11.216 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 2021-05-25T12:01:11.217 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify_pp: handle_notify cookie 94792742241200 notify_id 128849018883 notifier_gid 74773 2021-05-25T12:01:11.217 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 (5 ms) 2021-05-25T12:01:11.217 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 2021-05-25T12:01:11.217 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify_pp: handle_notify cookie 94792742241200 notify_id 128849018883 notifier_gid 74773 2021-05-25T12:01:11.218 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 (16 ms) 2021-05-25T12:01:11.218 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 2021-05-25T12:01:11.218 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify_pp: handle_notify cookie 94792742248176 notify_id 128849018884 notifier_gid 74773 2021-05-25T12:01:11.219 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 (7 ms) 2021-05-25T12:01:11.219 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 2021-05-25T12:01:11.219 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify_pp: trying... 2021-05-25T12:01:11.219 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify_pp: handle_notify cookie 94792742248176 notify_id 128849018884 notifier_gid 74773 2021-05-25T12:01:11.219 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify_pp: timed out 2021-05-25T12:01:11.220 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify_pp: flushing 2021-05-25T12:01:15.035 INFO:tasks.workunit.client.0.smithi181.stdout: api_wa api_c_write_operations: Running main() from gmock_main.cc 2021-05-25T12:01:15.036 INFO:tasks.workunit.client.0.smithi181.stdout: api_c_write_operations: [==========] Running 8 tests from 2 test suites. 2021-05-25T12:01:15.036 INFO:tasks.workunit.client.0.smithi181.stdout: api_c_write_operations: [----------] Global test environment set-up. 2021-05-25T12:01:15.036 INFO:tasks.workunit.client.0.smithi181.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps 2021-05-25T12:01:15.037 INFO:tasks.workunit.client.0.smithi181.stdout: api_c_write_operations: [ RUN ] LibradosCWriteOps.NewDelete 2021-05-25T12:01:15.037 INFO:tasks.workunit.client.0.smithi181.stdout: api_c_write_operations: [ OK ] LibradosCWriteOps.NewDelete (0 ms) 2021-05-25T12:01:15.037 INFO:tasks.workunit.client.0.smithi181.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps (1 ms total) 2021-05-25T12:01:15.037 INFO:tasks.workunit.client.0.smithi181.stdout: api_c_write_operations: 2021-05-25T12:01:15.038 INFO:tasks.workunit.client.0.smithi181.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps 2021-05-25T12:01:15.038 INFO:tasks.workunit.client.0.smithi181.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.assertExists 2021-05-25T12:01:15.039 INFO:tasks.workunit.client.0.smithi181.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.assertExists (17541 ms) 2021-05-25T12:01:15.039 INFO:tasks.workunit.client.0.smithi181.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteOpAssertVersion 2021-05-25T12:01:15.039 INFO:tasks.workunit.client.0.smithi181.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteOpAssertVersion (3005 ms) 2021-05-25T12:01:15.040 INFO:tasks.workunit.client.0.smithi181.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Xattrs 2021-05-25T12:01:15.040 INFO:tasks.workunit.client.0.smithi181.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Xattrs (2992 ms) 2021-05-25T12:01:15.040 INFO:tasks.workunit.client.0.smithi181.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Write 2021-05-25T12:01:15.040 INFO:tasks.workunit.client.0.smithi181.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Write (2651 ms) 2021-05-25T12:01:15.040 INFO:tasks.workunit.client.0.smithi181.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Exec 2021-05-25T12:01:15.041 INFO:tasks.workunit.client.0.smithi181.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Exec (3010 ms) 2021-05-25T12:01:15.041 INFO:tasks.workunit.client.0.smithi181.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteSame 2021-05-25T12:01:15.041 INFO:tasks.workunit.client.0.smithi181.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteSame (3039 ms) 2021-05-25T12:01:15.041 INFO:tasks.workunit.client.0.smithi181.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.CmpExt 2021-05-25T12:01:15.041 INFO:tasks.workunit.client.0.smithi181.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.CmpExt (2948 ms) 2021-05-25T12:01:15.042 INFO:tasks.workunit.client.0.smithi181.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps (35186 ms total) 2021-05-25T12:01:15.042 INFO:tasks.workunit.client.0.smithi181.stdout: api_c_write_operations: 2021-05-25T12:01:15.042 INFO:tasks.workunit.client.0.smithi181.stdout: api_c_write_operations: [----------] Global test environment tear-down 2021-05-25T12:01:15.042 INFO:tasks.workunit.client.0.smithi181.stdout: api_c_write_operations: [==========] 8 tests from 2 test suites ran. (35187 ms total) 2021-05-25T12:01:15.043 INFO:tasks.workunit.client.0.smithi181.stdout: api_c_write_operations: [ PASSED ] 8 tests. 2021-05-25T12:01:15.063 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify: Running main() from gmock_main.cc 2021-05-25T12:01:15.063 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify: [==========] Running 11 tests from 2 test suites. 2021-05-25T12:01:15.064 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify: [----------] Global test environment set-up. 2021-05-25T12:01:15.064 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify 2021-05-25T12:01:15.064 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify 2021-05-25T12:01:15.064 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify: watch_notify_test_cb 2021-05-25T12:01:15.065 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify (130 ms) 2021-05-25T12:01:15.065 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch2Delete 2021-05-25T12:01:15.065 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2021-05-25T12:01:15.065 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify: watch_notify2_test_errcb cookie 93861472241296 err -107 2021-05-25T12:01:15.065 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch2Delete (1017 ms) 2021-05-25T12:01:15.066 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete 2021-05-25T12:01:15.066 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2021-05-25T12:01:15.066 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify: watch_notify2_test_errcb cookie 93861472243504 err -107 2021-05-25T12:01:15.066 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete (1013 ms) 2021-05-25T12:01:15.067 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2 2021-05-25T12:01:15.067 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify: watch_notify2_test_cb from 4595 notify_id 85899345920 cookie 93861472264336 2021-05-25T12:01:15.067 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2 (4 ms) 2021-05-25T12:01:15.067 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchNotify2 2021-05-25T12:01:15.067 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify: watch_notify2_test_cb from 4595 notify_id 85899345920 cookie 93861472264336 2021-05-25T12:01:15.068 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchNotify2 (7 ms) 2021-05-25T12:01:15.068 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioNotify 2021-05-25T12:01:15.068 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify: watch_notify2_test_cb from 4595 notify_id 85899345920 cookie 93861472294896 2021-05-25T12:01:15.068 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioNotify (6 ms) 2021-05-25T12:01:15.069 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Multi 2021-05-25T12:01:15.069 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify: watch_notify2_test_cb from 4595 notify_id 85899345921 cookie 93861472290944 2021-05-25T12:01:15.069 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify: watch_notify2_test_cb from 4595 notify_id 85899345921 cookie 93861472294896 2021-05-25T12:01:15.069 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Multi (12 ms) 2021-05-25T12:01:15.069 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Timeout 2021-05-25T12:01:15.070 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify: watch_notify2_test_cb from 4595 notify_id 85899345921 cookie 93861472290944 2021-05-25T12:01:15.070 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify: watch_notify2_test_cb from 4595 notify_id 90194313218 cookie 93861472290944 2021-05-25T12:01:15.070 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Timeout (3006 ms) 2021-05-25T12:01:15.070 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch3Timeout 2021-05-25T12:01:15.071 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify: waiting up to 64 for osd to time us out ... 2021-05-25T12:01:15.071 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify: watch_notify2_test_errcb cookie 93861472294896 err -107 2021-05-25T12:01:15.071 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify: watch_notify2_test_cb from 4595 notify_id 120259084290 cookie 93861472294896 2021-05-25T12:01:15.071 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch3Timeout (5011 ms) 2021-05-25T12:01:15.071 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete2 2021-05-25T12:01:15.072 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify: waiting up to 30 for disconnect notification ... 2021-05-25T12:01:15.072 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify: watch_notify2_test_errcb cookie 93861472294896 err -107 2021-05-25T12:01:15.072 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete2 (1006 ms) 2021-05-25T12:01:15.073 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify (11213 ms total) 2021-05-25T12:01:15.073 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify: 2021-05-25T12:01:15.073 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC 2021-05-25T12:01:15.074 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotifyEC.WatchNotify 2021-05-25T12:01:15.074 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify: watch_notify_test_cb 2021-05-25T12:01:15.074 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify: [ OK ] LibRadosWatchNotifyEC.WatchNotify (1220 ms) 2021-05-25T12:01:15.075 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC (1220 ms total) 2021-05-25T12:01:15.075 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify: 2021-05-25T12:01:15.075 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify: [----------] Global test environment tear-down 2021-05-25T12:01:15.076 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify: [==========] 11 tests from 2 test suites ran. (35253 ms total) 2021-05-25T12:01:15.076 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify: [ PASSED ] 11 tests. 2021-05-25T12:01:15.444 INFO:tasks.workunit.client.0.smithi181.stdout:i_service_pp: "distro": "centos", 2021-05-25T12:01:15.445 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: "distro_description": "CentOS Linux 8", 2021-05-25T12:01:15.445 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: "distro_version": "8", 2021-05-25T12:01:15.445 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: "foo": "bar", 2021-05-25T12:01:15.445 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: "hostname": "smithi181", 2021-05-25T12:01:15.446 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: "kernel_description": "#1 SMP Thu Apr 8 19:01:30 UTC 2021", 2021-05-25T12:01:15.446 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: "kernel_version": "4.18.0-240.22.1.el8_3.x86_64", 2021-05-25T12:01:15.446 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: "mem_swap_kb": "0", 2021-05-25T12:01:15.446 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: "mem_total_kb": "32611872", 2021-05-25T12:01:15.447 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: "os": "Linux", 2021-05-25T12:01:15.447 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: "this": "that" 2021-05-25T12:01:15.447 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: }, 2021-05-25T12:01:15.447 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: "task_status": {} 2021-05-25T12:01:15.447 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: } 2021-05-25T12:01:15.448 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: } 2021-05-25T12:01:15.448 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: }, 2021-05-25T12:01:15.448 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: "mon": { 2021-05-25T12:01:15.448 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: "daemons": { 2021-05-25T12:01:15.448 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: "summary": "", 2021-05-25T12:01:15.449 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: "b": { 2021-05-25T12:01:15.449 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: "start_epoch": 0, 2021-05-25T12:01:15.449 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: "start_stamp": "0.000000", 2021-05-25T12:01:15.449 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: "gid": 0, 2021-05-25T12:01:15.450 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: "addr": "(unrecognized address family 0)/0", 2021-05-25T12:01:15.450 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: "metadata": {}, 2021-05-25T12:01:15.450 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: "task_status": {} 2021-05-25T12:01:15.450 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: } 2021-05-25T12:01:15.450 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: } 2021-05-25T12:01:15.451 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: } 2021-05-25T12:01:15.451 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: } 2021-05-25T12:01:15.451 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: } 2021-05-25T12:01:15.451 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: 2021-05-25T12:01:15.452 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: attempt 1 of 20 2021-05-25T12:01:15.452 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: [ OK ] LibRadosServicePP.Close (10496 ms) 2021-05-25T12:01:15.452 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP (35589 ms total) 2021-05-25T12:01:15.452 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: 2021-05-25T12:01:15.452 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: [----------] Global test environment tear-down 2021-05-25T12:01:15.453 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: [==========] 4 tests from 1 test suite ran. (35589 ms total) 2021-05-25T12:01:15.453 INFO:tasks.workunit.client.0.smithi181.stdout: api_service_pp: [ PASSED ] 4 tests. 2021-05-25T12:01:16.230 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2021-05-25T12:01:16.231 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2021-05-25T12:01:16.231 INFO:tasks.ceph.mon.a:Restarting daemon 2021-05-25T12:01:16.232 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2021-05-25T12:01:16.234 INFO:tasks.ceph.mon.a:Started 2021-05-25T12:01:16.235 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2021-05-25T12:01:16.235 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2021-05-25T12:01:16.359 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.358+0000 7f5937c07700 10 monclient: get_monmap_and_config 2021-05-25T12:01:16.360 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.358+0000 7f5937c07700 10 monclient: build_initial_monmap 2021-05-25T12:01:16.360 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.358+0000 7f5937c07700 10 monclient: monmap: 2021-05-25T12:01:16.360 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:01:16.360 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:16.361 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:01:16.359394+0000 2021-05-25T12:01:16.361 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:01:16.359394+0000 2021-05-25T12:01:16.361 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:01:16.361 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:16.361 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:01:16.362 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:01:16.362 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:01:16.362 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:16.364 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.358+0000 7f5937c07700 1 Processor -- start 2021-05-25T12:01:16.364 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.359+0000 7f5937c07700 1 -- start start 2021-05-25T12:01:16.364 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.359+0000 7f5937c07700 10 monclient: init 2021-05-25T12:01:16.365 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.359+0000 7f5937c07700 10 monclient: _reopen_session rank -1 2021-05-25T12:01:16.365 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.359+0000 7f5937c07700 10 monclient: _add_conns ranks=[2,1,0] 2021-05-25T12:01:16.365 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.359+0000 7f5937c07700 10 monclient(hunting): picked mon.noname-b con 0x7f5930090990 addr v1:172.21.15.102:6790/0 2021-05-25T12:01:16.365 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.359+0000 7f5937c07700 10 monclient(hunting): picked mon.noname-c con 0x7f5930090d70 addr v1:172.21.15.181:6789/0 2021-05-25T12:01:16.365 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.359+0000 7f5937c07700 10 monclient(hunting): picked mon.noname-a con 0x7f593012e330 addr v1:172.21.15.102:6789/0 2021-05-25T12:01:16.366 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.359+0000 7f5937c07700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5930096fa0 con 0x7f593012e330 2021-05-25T12:01:16.366 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.359+0000 7f5937c07700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59300926a0 con 0x7f5930090d70 2021-05-25T12:01:16.366 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.359+0000 7f5937c07700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5930092940 con 0x7f5930090990 2021-05-25T12:01:16.366 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.359+0000 7f5937c07700 10 monclient(hunting): _renew_subs 2021-05-25T12:01:16.367 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.359+0000 7f5937c07700 10 monclient(hunting): authenticate will time out at 876.956687s 2021-05-25T12:01:16.367 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.359+0000 7f59359a3700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f5930090990 0x7f5930093240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:54544/0 (socket says 172.21.15.102:54544) 2021-05-25T12:01:16.367 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.359+0000 7f59359a3700 1 -- 172.21.15.102:0/2063260321 learned_addr learned my addr 172.21.15.102:0/2063260321 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:01:16.367 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.359+0000 7f59361a4700 1 --1- 172.21.15.102:0/2063260321 >> v1:172.21.15.102:6789/0 conn(0x7f593012e330 0x7f5930130720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:59670/0 (socket says 172.21.15.102:59670) 2021-05-25T12:01:16.370 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.369+0000 7f59349a1700 1 -- 172.21.15.102:0/2063260321 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3033855904 0 0) 0x7f59300926a0 con 0x7f5930090d70 2021-05-25T12:01:16.370 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.369+0000 7f59349a1700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:01:16.371 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.369+0000 7f59349a1700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:01:16.371 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.369+0000 7f59349a1700 10 monclient(hunting): my global_id is 81913 2021-05-25T12:01:16.371 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.369+0000 7f59349a1700 1 -- 172.21.15.102:0/2063260321 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5918004380 con 0x7f5930090d70 2021-05-25T12:01:16.398 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.397+0000 7f59349a1700 1 -- 172.21.15.102:0/2063260321 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 554467780 0 0) 0x7f5930092940 con 0x7f5930090990 2021-05-25T12:01:16.398 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.397+0000 7f59349a1700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:01:16.399 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.397+0000 7f59349a1700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:01:16.399 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.397+0000 7f59349a1700 10 monclient(hunting): my global_id is 13271 2021-05-25T12:01:16.399 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.397+0000 7f59349a1700 1 -- 172.21.15.102:0/2063260321 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59300926a0 con 0x7f5930090990 2021-05-25T12:01:16.402 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.400+0000 7f59349a1700 1 -- 172.21.15.102:0/2063260321 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1983204220 0 0) 0x7f59300926a0 con 0x7f5930090990 2021-05-25T12:01:16.403 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.400+0000 7f59349a1700 1 -- 172.21.15.102:0/2063260321 >> v1:172.21.15.181:6789/0 conn(0x7f5930090d70 legacy=0x7f5930091200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:16.403 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.400+0000 7f59349a1700 1 -- 172.21.15.102:0/2063260321 >> v1:172.21.15.102:6789/0 conn(0x7f593012e330 legacy=0x7f5930130720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:16.403 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.400+0000 7f59349a1700 10 monclient: _finish_hunting 0 2021-05-25T12:01:16.403 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.400+0000 7f59349a1700 1 monclient: found mon.noname-b 2021-05-25T12:01:16.404 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.400+0000 7f59349a1700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:01:16.404 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.400+0000 7f59349a1700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T12:01:16.404 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.400+0000 7f59349a1700 1 -- 172.21.15.102:0/2063260321 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5930092be0 con 0x7f5930090990 2021-05-25T12:01:16.404 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.400+0000 7f59349a1700 10 monclient: _finish_auth 0 2021-05-25T12:01:16.404 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.400+0000 7f59349a1700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:01:16.405 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.400+0000 7f59349a1700 1 -- 172.21.15.102:0/2063260321 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f59200015f0 con 0x7f5930090990 2021-05-25T12:01:16.405 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.400+0000 7f59349a1700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:01:16.405 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.400+0000 7f59349a1700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:01:16.405 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.400+0000 7f59349a1700 10 monclient: dump: 2021-05-25T12:01:16.406 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:01:16.406 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:16.406 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:16.406 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:16.406 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:01:16.407 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:16.407 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:01:16.407 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:01:16.407 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:01:16.407 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:16.408 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.400+0000 7f5937c07700 5 monclient: authenticate success, global_id 13271 2021-05-25T12:01:16.408 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.400+0000 7f5937c07700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:01:16.408 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.401+0000 7f59349a1700 1 -- 172.21.15.102:0/2063260321 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f59200018a0 con 0x7f5930090990 2021-05-25T12:01:16.408 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.401+0000 7f59349a1700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:01:16.409 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.401+0000 7f5937c07700 10 monclient: get_monmap_and_config success 2021-05-25T12:01:16.409 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.401+0000 7f59349a1700 1 -- 172.21.15.102:0/2063260321 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f5920001bf0 con 0x7f5930090990 2021-05-25T12:01:16.409 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.401+0000 7f59349a1700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:01:16.409 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.401+0000 7f5937c07700 4 set_mon_vals no callback set 2021-05-25T12:01:16.409 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.401+0000 7f59349a1700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:01:16.410 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.401+0000 7f59349a1700 10 monclient: dump: 2021-05-25T12:01:16.410 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:01:16.410 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:16.410 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:16.410 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:16.411 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:01:16.411 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:16.411 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:01:16.411 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:01:16.411 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:01:16.412 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:16.418 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.402+0000 7f5937c07700 10 monclient: shutdown 2021-05-25T12:01:16.418 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.402+0000 7f5937c07700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:01:16.419 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.402+0000 7f5937c07700 1 -- 172.21.15.102:0/2063260321 >> v1:172.21.15.102:6790/0 conn(0x7f5930090990 legacy=0x7f5930093240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:16.419 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.402+0000 7f5937c07700 1 -- 172.21.15.102:0/2063260321 shutdown_connections 2021-05-25T12:01:16.419 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.402+0000 7f5937c07700 1 -- 172.21.15.102:0/2063260321 >> 172.21.15.102:0/2063260321 conn(0x7f593009b370 msgr2=0x7f5930097430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:01:16.419 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.402+0000 7f5937c07700 1 -- 172.21.15.102:0/2063260321 shutdown_connections 2021-05-25T12:01:16.419 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.402+0000 7f5937c07700 1 -- 172.21.15.102:0/2063260321 wait complete. 2021-05-25T12:01:16.420 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.402+0000 7f5937c07700 10 monclient: build_initial_monmap 2021-05-25T12:01:16.420 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.402+0000 7f5937c07700 10 monclient: monmap: 2021-05-25T12:01:16.420 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:01:16.420 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:01:16.421 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:01:16.421 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:01:16.421 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:01:16.421 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:01:16.421 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:01:16.422 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:01:16.422 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:01:16.422 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:16.422 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.403+0000 7f5937c07700 1 librados: starting msgr at 2021-05-25T12:01:16.423 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.403+0000 7f5937c07700 1 librados: starting objecter 2021-05-25T12:01:16.423 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.403+0000 7f5937c07700 1 Processor -- start 2021-05-25T12:01:16.423 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.403+0000 7f5937c07700 1 -- start start 2021-05-25T12:01:16.423 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.403+0000 7f5937c07700 1 librados: setting wanted keys 2021-05-25T12:01:16.423 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.403+0000 7f5937c07700 1 librados: calling monclient init 2021-05-25T12:01:16.424 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.403+0000 7f5937c07700 10 monclient: init 2021-05-25T12:01:16.424 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.404+0000 7f5937c07700 10 monclient: _reopen_session rank -1 2021-05-25T12:01:16.424 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.404+0000 7f5937c07700 10 monclient: _add_conns ranks=[2,1,0] 2021-05-25T12:01:16.424 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.404+0000 7f5937c07700 10 monclient(hunting): picked mon.noname-c con 0x7f5930090990 addr v1:172.21.15.102:6790/0 2021-05-25T12:01:16.424 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.404+0000 7f5937c07700 10 monclient(hunting): picked mon.noname-b con 0x7f5930090d70 addr v1:172.21.15.181:6789/0 2021-05-25T12:01:16.425 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.404+0000 7f5937c07700 10 monclient(hunting): picked mon.noname-a con 0x7f593012e330 addr v1:172.21.15.102:6789/0 2021-05-25T12:01:16.425 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.404+0000 7f59359a3700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f5930090990 0x7f5930064d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:54550/0 (socket says 172.21.15.102:54550) 2021-05-25T12:01:16.425 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.404+0000 7f5937c07700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5930068270 con 0x7f593012e330 2021-05-25T12:01:16.425 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.404+0000 7f59359a3700 1 -- 172.21.15.102:0/617085543 learned_addr learned my addr 172.21.15.102:0/617085543 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:01:16.425 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.404+0000 7f5937c07700 1 -- 172.21.15.102:0/617085543 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5930068510 con 0x7f5930090d70 2021-05-25T12:01:16.426 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.404+0000 7f5937c07700 1 -- 172.21.15.102:0/617085543 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59300687b0 con 0x7f5930090990 2021-05-25T12:01:16.426 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.404+0000 7f5937c07700 10 monclient(hunting): _renew_subs 2021-05-25T12:01:16.426 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.404+0000 7f5937c07700 10 monclient(hunting): authenticate will time out at 877.001305s 2021-05-25T12:01:16.426 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.404+0000 7f591dffb700 10 client.?.objecter ms_handle_connect 0x7f5930090990 2021-05-25T12:01:16.427 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.404+0000 7f591dffb700 10 client.?.objecter resend_mon_ops 2021-05-25T12:01:16.427 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.404+0000 7f591dffb700 10 client.?.objecter ms_handle_connect 0x7f5930090d70 2021-05-25T12:01:16.427 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.404+0000 7f591dffb700 10 client.?.objecter resend_mon_ops 2021-05-25T12:01:16.427 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.404+0000 7f591dffb700 10 client.?.objecter ms_handle_connect 0x7f593012e330 2021-05-25T12:01:16.427 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.404+0000 7f591dffb700 10 client.?.objecter resend_mon_ops 2021-05-25T12:01:16.428 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.404+0000 7f591dffb700 1 -- 172.21.15.102:0/617085543 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2586949839 0 0) 0x7f59300687b0 con 0x7f5930090990 2021-05-25T12:01:16.428 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.404+0000 7f591dffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:01:16.428 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.404+0000 7f591dffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:01:16.428 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.404+0000 7f591dffb700 10 monclient(hunting): my global_id is 13283 2021-05-25T12:01:16.428 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.404+0000 7f591dffb700 1 -- 172.21.15.102:0/617085543 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5910004c10 con 0x7f5930090990 2021-05-25T12:01:16.429 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.404+0000 7f591dffb700 1 -- 172.21.15.102:0/617085543 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2317860791 0 0) 0x7f5910004c10 con 0x7f5930090990 2021-05-25T12:01:16.429 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.404+0000 7f591dffb700 1 -- 172.21.15.102:0/617085543 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f59300687b0 con 0x7f5930090990 2021-05-25T12:01:16.429 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.404+0000 7f591dffb700 1 -- 172.21.15.102:0/617085543 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f59200016b0 con 0x7f5930090990 2021-05-25T12:01:16.429 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.404+0000 7f591dffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:01:16.430 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.404+0000 7f591dffb700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:01:16.430 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.404+0000 7f591dffb700 10 monclient(hunting): dump: 2021-05-25T12:01:16.430 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:01:16.430 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:16.430 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:16.431 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:16.431 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:01:16.431 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:16.431 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:01:16.431 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:01:16.432 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:01:16.432 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:16.432 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.404+0000 7f591dffb700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:01:16.432 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.404+0000 7f591dffb700 1 -- 172.21.15.102:0/617085543 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3612115105 0 0) 0x7f59300687b0 con 0x7f5930090990 2021-05-25T12:01:16.432 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.404+0000 7f591dffb700 1 -- 172.21.15.102:0/617085543 >> v1:172.21.15.181:6789/0 conn(0x7f5930090d70 legacy=0x7f5930065440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:16.433 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.404+0000 7f591dffb700 1 -- 172.21.15.102:0/617085543 >> v1:172.21.15.102:6789/0 conn(0x7f593012e330 legacy=0x7f5930065b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:16.433 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.404+0000 7f591dffb700 10 monclient: _finish_hunting 0 2021-05-25T12:01:16.433 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.404+0000 7f591dffb700 1 monclient: found mon.c 2021-05-25T12:01:16.433 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.404+0000 7f591dffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:01:16.433 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.405+0000 7f591dffb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:01:16.434 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.405+0000 7f591dffb700 1 -- 172.21.15.102:0/617085543 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59300673c0 con 0x7f5930090990 2021-05-25T12:01:16.434 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.405+0000 7f591dffb700 10 monclient: _finish_auth 0 2021-05-25T12:01:16.434 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.405+0000 7f591dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:01:16.434 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.405+0000 7f5937c07700 5 monclient: authenticate success, global_id 13283 2021-05-25T12:01:16.434 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.405+0000 7f5937c07700 10 monclient: _renew_subs 2021-05-25T12:01:16.435 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.405+0000 7f5937c07700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:01:16.435 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.405+0000 7f5937c07700 1 -- 172.21.15.102:0/617085543 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59300668e0 con 0x7f5930090990 2021-05-25T12:01:16.435 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.405+0000 7f5937c07700 10 client.13283.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:01:16.435 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.405+0000 7f5937c07700 10 monclient: _renew_subs 2021-05-25T12:01:16.435 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.405+0000 7f5937c07700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:01:16.436 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.405+0000 7f5937c07700 1 -- 172.21.15.102:0/617085543 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5930148880 con 0x7f5930090990 2021-05-25T12:01:16.436 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.405+0000 7f5937c07700 1 librados: init done 2021-05-25T12:01:16.436 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.406+0000 7f5937c07700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:01:16.436 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.406+0000 7f5937c07700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:01:16.437 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.406+0000 7f5937c07700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:01:16.437 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.406+0000 7f5937c07700 1 -- 172.21.15.102:0/617085543 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5930068940 con 0x7f5930090990 2021-05-25T12:01:16.437 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.407+0000 7f591dffb700 1 -- 172.21.15.102:0/617085543 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f59200049f0 con 0x7f5930090990 2021-05-25T12:01:16.437 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.407+0000 7f591dffb700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:01:16.438 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.407+0000 7f591dffb700 1 -- 172.21.15.102:0/617085543 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f5920004be0 con 0x7f5930090990 2021-05-25T12:01:16.438 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.407+0000 7f591dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:01:16.438 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.407+0000 7f591dffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:01:16.438 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.407+0000 7f591dffb700 10 monclient: dump: 2021-05-25T12:01:16.438 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:01:16.439 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:16.439 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:16.439 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:16.439 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:01:16.439 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:16.440 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:01:16.440 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:01:16.440 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:01:16.440 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:16.440 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.407+0000 7f591ffff700 4 set_mon_vals no callback set 2021-05-25T12:01:16.441 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.408+0000 7f591dffb700 1 -- 172.21.15.102:0/617085543 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f5920004dc0 con 0x7f5930090990 2021-05-25T12:01:16.441 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.408+0000 7f591dffb700 1 -- 172.21.15.102:0/617085543 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(38..38 src has 1..38) v4 ==== 8377+0+0 (unknown 2583901202 0 0) 0x7f592003e290 con 0x7f5930090990 2021-05-25T12:01:16.441 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.408+0000 7f591dffb700 10 client.13283.objecter ms_dispatch 0x7f5930001550 osd_map(38..38 src has 1..38) v4 2021-05-25T12:01:16.441 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.408+0000 7f591dffb700 3 client.13283.objecter handle_osd_map got epochs [38,38] > 0 2021-05-25T12:01:16.441 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.408+0000 7f591dffb700 3 client.13283.objecter handle_osd_map decoding full epoch 38 2021-05-25T12:01:16.442 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.408+0000 7f591dffb700 20 client.13283.objecter dump_active .. 0 homeless 2021-05-25T12:01:16.442 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.408+0000 7f59351a2700 10 monclient: get_auth_request con 0x7f5910033140 auth_method 0 2021-05-25T12:01:16.442 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.408+0000 7f591dffb700 10 client.13283.objecter ms_handle_connect 0x7f5910033140 2021-05-25T12:01:16.442 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.412+0000 7f591dffb700 1 -- 172.21.15.102:0/617085543 <== mon.2 v1:172.21.15.102:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7f5920012c90 con 0x7f5930090990 2021-05-25T12:01:16.442 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.412+0000 7f591dffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:01:16.443 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.412+0000 7f591dffb700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:01:16.655 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.653+0000 7f5937c07700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2021-05-25T12:01:16.655 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.653+0000 7f5937c07700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2021-05-25T12:01:16.655 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.653+0000 7f5937c07700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:01:16.655 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.653+0000 7f5937c07700 1 -- 172.21.15.102:0/617085543 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f5930094b50 con 0x7f5930090990 2021-05-25T12:01:16.656 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.653+0000 7f591dffb700 1 -- 172.21.15.102:0/617085543 <== mon.2 v1:172.21.15.102:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1342 (unknown 254746657 0 2576947) 0x7f5920012c90 con 0x7f5930090990 2021-05-25T12:01:16.656 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.653+0000 7f591dffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2021-05-25T12:01:16.656 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.653+0000 7f591dffb700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:01:16.657 INFO:teuthology.orchestra.run.smithi102.stdout:{"election_epoch":20,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138303579357183","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"20ee91b6-8935-4b98-b868-85c03789ccc4","modified":"2021-05-25T11:59:11.011522Z","created":"2021-05-25T11:59:11.011522Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6790","nonce":0}]},"addr":"172.21.15.102:6790/0","public_addr":"172.21.15.102:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2021-05-25T12:01:16.658 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.655+0000 7f5937c07700 10 librados: watch_flush enter 2021-05-25T12:01:16.658 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.655+0000 7f5937c07700 10 librados: watch_flush exit 2021-05-25T12:01:16.659 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.655+0000 7f5937c07700 20 client.13283.objecter shutdown clearing up homeless session... 2021-05-25T12:01:16.659 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.655+0000 7f5937c07700 10 client.13283.objecter successfully canceled tick 2021-05-25T12:01:16.659 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.655+0000 7f5937c07700 1 -- 172.21.15.102:0/617085543 >> v1:172.21.15.181:6812/29582 conn(0x7f5910033140 legacy=0x7f5910035600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:16.659 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.655+0000 7f5937c07700 10 monclient: shutdown 2021-05-25T12:01:16.660 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.655+0000 7f5937c07700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:01:16.660 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.655+0000 7f5937c07700 1 -- 172.21.15.102:0/617085543 >> v1:172.21.15.102:6790/0 conn(0x7f5930090990 legacy=0x7f5930064d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:16.660 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.655+0000 7f5937c07700 1 -- 172.21.15.102:0/617085543 shutdown_connections 2021-05-25T12:01:16.660 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.655+0000 7f5937c07700 1 -- 172.21.15.102:0/617085543 >> 172.21.15.102:0/617085543 conn(0x7f593009b370 msgr2=0x7f5930129390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:01:16.660 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.655+0000 7f5937c07700 1 -- 172.21.15.102:0/617085543 shutdown_connections 2021-05-25T12:01:16.661 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.655+0000 7f5937c07700 1 -- 172.21.15.102:0/617085543 wait complete. 2021-05-25T12:01:16.661 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.655+0000 7f5937c07700 1 librados: shutdown 2021-05-25T12:01:16.668 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2021-05-25T12:01:16.668 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2021-05-25T12:01:16.790 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.789+0000 7fe764a9a700 10 monclient: get_monmap_and_config 2021-05-25T12:01:16.790 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.789+0000 7fe764a9a700 10 monclient: build_initial_monmap 2021-05-25T12:01:16.790 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.789+0000 7fe764a9a700 10 monclient: monmap: 2021-05-25T12:01:16.791 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:01:16.791 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:16.791 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:01:16.790054+0000 2021-05-25T12:01:16.791 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:01:16.790054+0000 2021-05-25T12:01:16.792 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:01:16.792 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:16.792 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:01:16.792 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:01:16.792 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:01:16.792 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:16.805 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.789+0000 7fe764a9a700 1 Processor -- start 2021-05-25T12:01:16.805 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.789+0000 7fe764a9a700 1 -- start start 2021-05-25T12:01:16.805 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.789+0000 7fe764a9a700 10 monclient: init 2021-05-25T12:01:16.806 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.790+0000 7fe764a9a700 10 monclient: _reopen_session rank -1 2021-05-25T12:01:16.806 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.790+0000 7fe764a9a700 10 monclient: _add_conns ranks=[2,1,0] 2021-05-25T12:01:16.806 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.790+0000 7fe764a9a700 10 monclient(hunting): picked mon.noname-b con 0x7fe760098930 addr v1:172.21.15.102:6790/0 2021-05-25T12:01:16.806 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.790+0000 7fe764a9a700 10 monclient(hunting): picked mon.noname-c con 0x7fe76012bca0 addr v1:172.21.15.181:6789/0 2021-05-25T12:01:16.806 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.790+0000 7fe764a9a700 10 monclient(hunting): picked mon.noname-a con 0x7fe76012c150 addr v1:172.21.15.102:6789/0 2021-05-25T12:01:16.807 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.790+0000 7fe764a9a700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe760099b20 con 0x7fe76012c150 2021-05-25T12:01:16.807 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.790+0000 7fe764a9a700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe76009a3a0 con 0x7fe76012bca0 2021-05-25T12:01:16.807 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.790+0000 7fe764a9a700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe76009ac60 con 0x7fe760098930 2021-05-25T12:01:16.807 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.790+0000 7fe764a9a700 10 monclient(hunting): _renew_subs 2021-05-25T12:01:16.808 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.790+0000 7fe764a9a700 10 monclient(hunting): authenticate will time out at 877.387357s 2021-05-25T12:01:16.808 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.790+0000 7fe75f7fe700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fe760098930 0x7fe76012b590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:54562/0 (socket says 172.21.15.102:54562) 2021-05-25T12:01:16.808 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.790+0000 7fe75f7fe700 1 -- 172.21.15.102:0/1008375994 learned_addr learned my addr 172.21.15.102:0/1008375994 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:01:16.808 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.790+0000 7fe75ffff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fe76012c150 0x7fe760133ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:59686/0 (socket says 172.21.15.102:59686) 2021-05-25T12:01:16.808 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.790+0000 7fe75e7fc700 1 -- 172.21.15.102:0/1008375994 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3354636101 0 0) 0x7fe760099b20 con 0x7fe76012c150 2021-05-25T12:01:16.809 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.790+0000 7fe75e7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:01:16.809 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.790+0000 7fe75e7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:01:16.809 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.790+0000 7fe75e7fc700 10 monclient(hunting): my global_id is 84510 2021-05-25T12:01:16.809 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.790+0000 7fe75e7fc700 1 -- 172.21.15.102:0/1008375994 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe744003270 con 0x7fe76012c150 2021-05-25T12:01:16.810 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.790+0000 7fe75e7fc700 1 -- 172.21.15.102:0/1008375994 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3547584129 0 0) 0x7fe76009ac60 con 0x7fe760098930 2021-05-25T12:01:16.810 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.790+0000 7fe75e7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:01:16.810 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.790+0000 7fe75e7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:01:16.810 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.790+0000 7fe75e7fc700 10 monclient(hunting): my global_id is 13706 2021-05-25T12:01:16.810 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.790+0000 7fe75e7fc700 1 -- 172.21.15.102:0/1008375994 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe760099b20 con 0x7fe760098930 2021-05-25T12:01:16.811 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.790+0000 7fe75e7fc700 1 -- 172.21.15.102:0/1008375994 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 594984162 0 0) 0x7fe744003270 con 0x7fe76012c150 2021-05-25T12:01:16.811 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.790+0000 7fe75e7fc700 1 -- 172.21.15.102:0/1008375994 >> v1:172.21.15.102:6790/0 conn(0x7fe760098930 legacy=0x7fe76012b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:16.811 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.790+0000 7fe75e7fc700 1 -- 172.21.15.102:0/1008375994 >> v1:172.21.15.181:6789/0 conn(0x7fe76012bca0 legacy=0x7fe7601308c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:16.811 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.790+0000 7fe75e7fc700 10 monclient: _finish_hunting 0 2021-05-25T12:01:16.811 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.790+0000 7fe75e7fc700 1 monclient: found mon.noname-a 2021-05-25T12:01:16.812 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.790+0000 7fe75e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:01:16.812 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.790+0000 7fe75e7fc700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T12:01:16.812 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.790+0000 7fe75e7fc700 1 -- 172.21.15.102:0/1008375994 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe76009af00 con 0x7fe76012c150 2021-05-25T12:01:16.812 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.790+0000 7fe75e7fc700 10 monclient: _finish_auth 0 2021-05-25T12:01:16.813 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.790+0000 7fe75e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:01:16.813 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.790+0000 7fe764a9a700 5 monclient: authenticate success, global_id 84510 2021-05-25T12:01:16.813 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.790+0000 7fe75e7fc700 1 -- 172.21.15.102:0/1008375994 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fe754001990 con 0x7fe76012c150 2021-05-25T12:01:16.813 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.790+0000 7fe764a9a700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:01:16.813 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.790+0000 7fe75e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:01:16.814 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.790+0000 7fe75e7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:01:16.814 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.790+0000 7fe75e7fc700 10 monclient: dump: 2021-05-25T12:01:16.814 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:01:16.814 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:16.815 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:16.815 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:16.815 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:01:16.815 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:16.815 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:01:16.816 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:01:16.816 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:01:16.816 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:16.816 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.790+0000 7fe764a9a700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:01:16.816 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.791+0000 7fe75e7fc700 1 -- 172.21.15.102:0/1008375994 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe754001c00 con 0x7fe76012c150 2021-05-25T12:01:16.816 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.791+0000 7fe75e7fc700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:01:16.817 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.791+0000 7fe764a9a700 10 monclient: get_monmap_and_config success 2021-05-25T12:01:16.817 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.791+0000 7fe75e7fc700 1 -- 172.21.15.102:0/1008375994 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fe754001f90 con 0x7fe76012c150 2021-05-25T12:01:16.817 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.791+0000 7fe764a9a700 4 set_mon_vals no callback set 2021-05-25T12:01:16.817 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.791+0000 7fe75e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:01:16.817 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.791+0000 7fe75e7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:01:16.818 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.791+0000 7fe75e7fc700 10 monclient: dump: 2021-05-25T12:01:16.818 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:01:16.818 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:16.818 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:16.819 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:16.819 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:01:16.819 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:16.819 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:01:16.819 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:01:16.820 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:01:16.820 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:16.820 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.791+0000 7fe764a9a700 10 monclient: shutdown 2021-05-25T12:01:16.820 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.791+0000 7fe764a9a700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:01:16.820 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.791+0000 7fe764a9a700 1 -- 172.21.15.102:0/1008375994 >> v1:172.21.15.102:6789/0 conn(0x7fe76012c150 legacy=0x7fe760133ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:16.820 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.791+0000 7fe764a9a700 1 -- 172.21.15.102:0/1008375994 shutdown_connections 2021-05-25T12:01:16.821 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.791+0000 7fe764a9a700 1 -- 172.21.15.102:0/1008375994 >> 172.21.15.102:0/1008375994 conn(0x7fe76009e9d0 msgr2=0x7fe76009ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:01:16.821 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.791+0000 7fe764a9a700 1 -- 172.21.15.102:0/1008375994 shutdown_connections 2021-05-25T12:01:16.821 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.791+0000 7fe764a9a700 1 -- 172.21.15.102:0/1008375994 wait complete. 2021-05-25T12:01:16.821 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.792+0000 7fe764a9a700 10 monclient: build_initial_monmap 2021-05-25T12:01:16.821 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.792+0000 7fe764a9a700 10 monclient: monmap: 2021-05-25T12:01:16.822 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:01:16.822 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:01:16.822 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:01:16.822 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:01:16.822 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:01:16.823 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:01:16.823 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:01:16.823 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:01:16.823 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:01:16.824 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:16.824 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.792+0000 7fe764a9a700 1 librados: starting msgr at 2021-05-25T12:01:16.824 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.792+0000 7fe764a9a700 1 librados: starting objecter 2021-05-25T12:01:16.824 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.792+0000 7fe764a9a700 1 Processor -- start 2021-05-25T12:01:16.824 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.792+0000 7fe764a9a700 1 -- start start 2021-05-25T12:01:16.825 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.792+0000 7fe764a9a700 1 librados: setting wanted keys 2021-05-25T12:01:16.825 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.792+0000 7fe764a9a700 1 librados: calling monclient init 2021-05-25T12:01:16.825 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.792+0000 7fe764a9a700 10 monclient: init 2021-05-25T12:01:16.825 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.792+0000 7fe764a9a700 10 monclient: _reopen_session rank -1 2021-05-25T12:01:16.825 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.792+0000 7fe764a9a700 10 monclient: _add_conns ranks=[0,2,1] 2021-05-25T12:01:16.826 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.792+0000 7fe764a9a700 10 monclient(hunting): picked mon.noname-a con 0x7fe760098930 addr v1:172.21.15.102:6789/0 2021-05-25T12:01:16.826 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.792+0000 7fe764a9a700 10 monclient(hunting): picked mon.noname-c con 0x7fe76012bca0 addr v1:172.21.15.102:6790/0 2021-05-25T12:01:16.826 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.793+0000 7fe764a9a700 10 monclient(hunting): picked mon.noname-b con 0x7fe76012c150 addr v1:172.21.15.181:6789/0 2021-05-25T12:01:16.826 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.793+0000 7fe764a9a700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7600937b0 con 0x7fe760098930 2021-05-25T12:01:16.826 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.793+0000 7fe764a9a700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe760092db0 con 0x7fe76012c150 2021-05-25T12:01:16.827 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.793+0000 7fe764a9a700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe76009a190 con 0x7fe76012bca0 2021-05-25T12:01:16.827 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.793+0000 7fe764a9a700 10 monclient(hunting): _renew_subs 2021-05-25T12:01:16.827 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.793+0000 7fe764a9a700 10 monclient(hunting): authenticate will time out at 877.390032s 2021-05-25T12:01:16.827 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.793+0000 7fe75f7fe700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fe760098930 0x7fe7600972e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:59690/0 (socket says 172.21.15.102:59690) 2021-05-25T12:01:16.828 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.793+0000 7fe75f7fe700 1 -- 172.21.15.102:0/3330467199 learned_addr learned my addr 172.21.15.102:0/3330467199 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:01:16.828 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.793+0000 7fe74b7fe700 10 client.?.objecter ms_handle_connect 0x7fe760098930 2021-05-25T12:01:16.828 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.793+0000 7fe74b7fe700 10 client.?.objecter resend_mon_ops 2021-05-25T12:01:16.828 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.793+0000 7fe74b7fe700 10 client.?.objecter ms_handle_connect 0x7fe76012bca0 2021-05-25T12:01:16.828 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.793+0000 7fe74b7fe700 10 client.?.objecter resend_mon_ops 2021-05-25T12:01:16.829 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.793+0000 7fe74b7fe700 1 -- 172.21.15.102:0/3330467199 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2414512999 0 0) 0x7fe7600937b0 con 0x7fe760098930 2021-05-25T12:01:16.829 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.793+0000 7fe74b7fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:01:16.829 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.793+0000 7fe74b7fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:01:16.829 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.793+0000 7fe74b7fe700 10 monclient(hunting): my global_id is 84513 2021-05-25T12:01:16.830 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.793+0000 7fe74b7fe700 1 -- 172.21.15.102:0/3330467199 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe740004c10 con 0x7fe760098930 2021-05-25T12:01:16.830 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.793+0000 7fe74b7fe700 10 client.?.objecter ms_handle_connect 0x7fe76012c150 2021-05-25T12:01:16.830 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.793+0000 7fe74b7fe700 10 client.?.objecter resend_mon_ops 2021-05-25T12:01:16.830 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.793+0000 7fe74b7fe700 1 -- 172.21.15.102:0/3330467199 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2764101651 0 0) 0x7fe76009a190 con 0x7fe76012bca0 2021-05-25T12:01:16.830 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.793+0000 7fe74b7fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:01:16.831 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.793+0000 7fe74b7fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:01:16.831 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.793+0000 7fe74b7fe700 10 monclient(hunting): my global_id is 13709 2021-05-25T12:01:16.832 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.793+0000 7fe74b7fe700 1 -- 172.21.15.102:0/3330467199 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7600937b0 con 0x7fe76012bca0 2021-05-25T12:01:16.832 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.793+0000 7fe74b7fe700 1 -- 172.21.15.102:0/3330467199 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1685385975 0 0) 0x7fe760092db0 con 0x7fe76012c150 2021-05-25T12:01:16.832 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.793+0000 7fe74b7fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:01:16.833 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.793+0000 7fe74b7fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:01:16.833 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.793+0000 7fe74b7fe700 10 monclient(hunting): my global_id is 82243 2021-05-25T12:01:16.833 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.793+0000 7fe74b7fe700 1 -- 172.21.15.102:0/3330467199 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe76009a190 con 0x7fe76012c150 2021-05-25T12:01:16.833 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.793+0000 7fe74b7fe700 1 -- 172.21.15.102:0/3330467199 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2188488735 0 0) 0x7fe740004c10 con 0x7fe760098930 2021-05-25T12:01:16.833 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.793+0000 7fe74b7fe700 1 -- 172.21.15.102:0/3330467199 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe760092db0 con 0x7fe760098930 2021-05-25T12:01:16.834 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.793+0000 7fe74b7fe700 1 -- 172.21.15.102:0/3330467199 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fe74c0018c0 con 0x7fe760098930 2021-05-25T12:01:16.834 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.793+0000 7fe74b7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:01:16.834 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.793+0000 7fe74b7fe700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:01:16.834 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.793+0000 7fe74b7fe700 10 monclient(hunting): dump: 2021-05-25T12:01:16.835 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:01:16.835 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:16.835 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:16.835 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:16.835 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:01:16.836 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:16.836 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:01:16.836 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:01:16.836 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:01:16.836 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:16.837 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.793+0000 7fe74b7fe700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:01:16.837 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.793+0000 7fe74b7fe700 1 -- 172.21.15.102:0/3330467199 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 804988939 0 0) 0x7fe7600937b0 con 0x7fe76012bca0 2021-05-25T12:01:16.837 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.793+0000 7fe74b7fe700 1 -- 172.21.15.102:0/3330467199 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe740004c10 con 0x7fe76012bca0 2021-05-25T12:01:16.837 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.793+0000 7fe74b7fe700 1 -- 172.21.15.102:0/3330467199 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fe7500017f0 con 0x7fe76012bca0 2021-05-25T12:01:16.838 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.793+0000 7fe74b7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:01:16.838 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.793+0000 7fe74b7fe700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2021-05-25T12:01:16.838 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.793+0000 7fe74b7fe700 10 monclient(hunting): dump: 2021-05-25T12:01:16.838 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:01:16.838 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:16.839 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:16.839 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:16.839 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:01:16.839 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:16.839 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:01:16.840 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:01:16.840 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:01:16.840 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:16.840 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.793+0000 7fe74b7fe700 1 -- 172.21.15.102:0/3330467199 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2157012664 0 0) 0x7fe760092db0 con 0x7fe760098930 2021-05-25T12:01:16.840 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.793+0000 7fe74b7fe700 1 -- 172.21.15.102:0/3330467199 >> v1:172.21.15.102:6790/0 conn(0x7fe76012bca0 legacy=0x7fe76013abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:16.841 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.793+0000 7fe74b7fe700 1 -- 172.21.15.102:0/3330467199 >> v1:172.21.15.181:6789/0 conn(0x7fe76012c150 legacy=0x7fe760142f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:16.841 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.793+0000 7fe74b7fe700 10 monclient: _finish_hunting 0 2021-05-25T12:01:16.841 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.793+0000 7fe74b7fe700 1 monclient: found mon.a 2021-05-25T12:01:16.841 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.793+0000 7fe74b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:01:16.842 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.793+0000 7fe74b7fe700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:01:16.842 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.793+0000 7fe74b7fe700 1 -- 172.21.15.102:0/3330467199 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe760094bc0 con 0x7fe760098930 2021-05-25T12:01:16.842 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.793+0000 7fe74b7fe700 10 monclient: _finish_auth 0 2021-05-25T12:01:16.842 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.793+0000 7fe74b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:01:16.842 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.793+0000 7fe764a9a700 5 monclient: authenticate success, global_id 84513 2021-05-25T12:01:16.843 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.793+0000 7fe764a9a700 10 monclient: _renew_subs 2021-05-25T12:01:16.843 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.793+0000 7fe764a9a700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:01:16.843 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.793+0000 7fe764a9a700 1 -- 172.21.15.102:0/3330467199 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7600979f0 con 0x7fe760098930 2021-05-25T12:01:16.843 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.793+0000 7fe764a9a700 10 client.84513.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:01:16.843 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.793+0000 7fe764a9a700 10 monclient: _renew_subs 2021-05-25T12:01:16.844 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.793+0000 7fe764a9a700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:01:16.844 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.793+0000 7fe764a9a700 1 -- 172.21.15.102:0/3330467199 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe76013b540 con 0x7fe760098930 2021-05-25T12:01:16.844 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.793+0000 7fe764a9a700 1 librados: init done 2021-05-25T12:01:16.844 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.793+0000 7fe74b7fe700 1 -- 172.21.15.102:0/3330467199 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe74c001c70 con 0x7fe760098930 2021-05-25T12:01:16.844 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.793+0000 7fe74b7fe700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:01:16.845 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.793+0000 7fe74b7fe700 1 -- 172.21.15.102:0/3330467199 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fe74c004980 con 0x7fe760098930 2021-05-25T12:01:16.845 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.793+0000 7fe74b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:01:16.845 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.793+0000 7fe74b7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:01:16.845 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.793+0000 7fe75dffb700 4 set_mon_vals no callback set 2021-05-25T12:01:16.845 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.793+0000 7fe74b7fe700 10 monclient: dump: 2021-05-25T12:01:16.846 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:01:16.846 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:16.846 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:16.846 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:16.846 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:01:16.847 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:16.847 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:01:16.847 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:01:16.847 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:01:16.847 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:16.848 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.794+0000 7fe74b7fe700 1 -- 172.21.15.102:0/3330467199 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7fe74c00dca0 con 0x7fe760098930 2021-05-25T12:01:16.848 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.794+0000 7fe74b7fe700 1 -- 172.21.15.102:0/3330467199 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(38..38 src has 1..38) v4 ==== 8377+0+0 (unknown 2583901202 0 0) 0x7fe74c03e030 con 0x7fe760098930 2021-05-25T12:01:16.848 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.794+0000 7fe74b7fe700 10 client.84513.objecter ms_dispatch 0x7fe760001550 osd_map(38..38 src has 1..38) v4 2021-05-25T12:01:16.848 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.794+0000 7fe74b7fe700 3 client.84513.objecter handle_osd_map got epochs [38,38] > 0 2021-05-25T12:01:16.848 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.794+0000 7fe74b7fe700 3 client.84513.objecter handle_osd_map decoding full epoch 38 2021-05-25T12:01:16.849 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.794+0000 7fe764a9a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:01:16.849 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.794+0000 7fe764a9a700 1 -- 172.21.15.102:0/3330467199 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fe7380065a0 con 0x7fe738002840 2021-05-25T12:01:16.849 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.794+0000 7fe74b7fe700 20 client.84513.objecter dump_active .. 0 homeless 2021-05-25T12:01:16.849 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.794+0000 7fe74b7fe700 10 client.84513.objecter ms_handle_connect 0x7fe738002840 2021-05-25T12:01:16.850 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.794+0000 7fe74b7fe700 10 client.84513.objecter resend_mon_ops 2021-05-25T12:01:16.850 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.795+0000 7fe74b7fe700 1 -- 172.21.15.102:0/3330467199 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1514054263 0 0) 0x7fe7380065a0 con 0x7fe738002840 2021-05-25T12:01:16.850 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.795+0000 7fe74b7fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:01:16.850 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.795+0000 7fe74b7fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:01:16.850 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.795+0000 7fe74b7fe700 10 monclient(hunting): my global_id is 84516 2021-05-25T12:01:16.851 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.795+0000 7fe74b7fe700 1 -- 172.21.15.102:0/3330467199 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe760092db0 con 0x7fe738002840 2021-05-25T12:01:16.851 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.795+0000 7fe74b7fe700 1 -- 172.21.15.102:0/3330467199 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2909067064 0 0) 0x7fe760092db0 con 0x7fe738002840 2021-05-25T12:01:16.851 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.795+0000 7fe74b7fe700 1 -- 172.21.15.102:0/3330467199 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe7380065a0 con 0x7fe738002840 2021-05-25T12:01:16.851 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.795+0000 7fe74b7fe700 1 -- 172.21.15.102:0/3330467199 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fe754004c40 con 0x7fe738002840 2021-05-25T12:01:16.851 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.795+0000 7fe74b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:01:16.852 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.795+0000 7fe74b7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:01:16.852 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.795+0000 7fe74b7fe700 10 monclient: dump: 2021-05-25T12:01:16.852 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:01:16.852 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:16.852 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:16.853 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:16.853 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:01:16.853 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:16.853 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:01:16.853 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:01:16.853 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:01:16.854 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:16.854 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.795+0000 7fe74b7fe700 1 -- 172.21.15.102:0/3330467199 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2334393930 0 0) 0x7fe7380065a0 con 0x7fe738002840 2021-05-25T12:01:16.854 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.795+0000 7fe74b7fe700 1 -- 172.21.15.102:0/3330467199 --> v1:172.21.15.102:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe738009e40 con 0x7fe738002840 2021-05-25T12:01:16.854 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.795+0000 7fe75effd700 10 monclient: get_auth_request con 0x7fe740033220 auth_method 0 2021-05-25T12:01:16.855 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.795+0000 7fe74b7fe700 10 client.84513.objecter ms_handle_connect 0x7fe740033220 2021-05-25T12:01:16.855 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.795+0000 7fe74b7fe700 1 -- 172.21.15.102:0/3330467199 <== mon.0 v1:172.21.15.102:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7fe754004fa0 con 0x7fe738002840 2021-05-25T12:01:16.855 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.795+0000 7fe74b7fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:01:16.855 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.795+0000 7fe74b7fe700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:01:16.855 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.795+0000 7fe74b7fe700 1 -- 172.21.15.102:0/3330467199 >> v1:172.21.15.102:6789/0 conn(0x7fe738002840 legacy=0x7fe738004ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:16.856 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.795+0000 7fe74b7fe700 1 -- 172.21.15.102:0/3330467199 >> v1:172.21.15.102:6789/0 conn(0x7fe738002840 legacy=0x7fe738004ce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:01:16.856 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.802+0000 7fe764a9a700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:01:16.856 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.802+0000 7fe764a9a700 1 -- 172.21.15.102:0/3330467199 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fe738009ab0 con 0x7fe738008d70 2021-05-25T12:01:16.856 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.802+0000 7fe74b7fe700 10 client.84513.objecter ms_handle_connect 0x7fe738008d70 2021-05-25T12:01:16.856 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.802+0000 7fe74b7fe700 10 client.84513.objecter resend_mon_ops 2021-05-25T12:01:16.857 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.802+0000 7fe74b7fe700 1 -- 172.21.15.102:0/3330467199 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 67984031 0 0) 0x7fe738009ab0 con 0x7fe738008d70 2021-05-25T12:01:16.857 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.802+0000 7fe74b7fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:01:16.857 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.802+0000 7fe74b7fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:01:16.857 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.802+0000 7fe74b7fe700 10 monclient(hunting): my global_id is 84528 2021-05-25T12:01:16.857 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.802+0000 7fe74b7fe700 1 -- 172.21.15.102:0/3330467199 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7380065a0 con 0x7fe738008d70 2021-05-25T12:01:16.858 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.803+0000 7fe74b7fe700 1 -- 172.21.15.102:0/3330467199 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2580402033 0 0) 0x7fe7380065a0 con 0x7fe738008d70 2021-05-25T12:01:16.858 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.803+0000 7fe74b7fe700 1 -- 172.21.15.102:0/3330467199 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe738009ab0 con 0x7fe738008d70 2021-05-25T12:01:16.858 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.803+0000 7fe74b7fe700 1 -- 172.21.15.102:0/3330467199 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fe7540055c0 con 0x7fe738008d70 2021-05-25T12:01:16.858 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.803+0000 7fe74b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:01:16.859 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.803+0000 7fe74b7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:01:16.859 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.803+0000 7fe74b7fe700 10 monclient: dump: 2021-05-25T12:01:16.859 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:01:16.859 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:16.859 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:16.860 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:16.860 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:01:16.860 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:16.860 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:01:16.860 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:01:16.861 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:01:16.861 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:16.861 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.803+0000 7fe74b7fe700 1 -- 172.21.15.102:0/3330467199 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3329242036 0 0) 0x7fe738009ab0 con 0x7fe738008d70 2021-05-25T12:01:16.861 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.803+0000 7fe74b7fe700 1 -- 172.21.15.102:0/3330467199 --> v1:172.21.15.102:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe738008740 con 0x7fe738008d70 2021-05-25T12:01:16.861 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.803+0000 7fe74b7fe700 1 -- 172.21.15.102:0/3330467199 <== mon.0 v1:172.21.15.102:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3771 (unknown 0 0 1085389743) 0x7fe754006910 con 0x7fe738008d70 2021-05-25T12:01:16.862 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.803+0000 7fe74b7fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:01:16.862 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.803+0000 7fe74b7fe700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:01:16.862 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.803+0000 7fe74b7fe700 1 -- 172.21.15.102:0/3330467199 >> v1:172.21.15.102:6789/0 conn(0x7fe738008d70 legacy=0x7fe738009200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:16.862 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.803+0000 7fe74b7fe700 1 -- 172.21.15.102:0/3330467199 >> v1:172.21.15.102:6789/0 conn(0x7fe738008d70 legacy=0x7fe738009200 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:01:16.863 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:01:16.863 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:01:16.863 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:01:16.863 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "peon", 2021-05-25T12:01:16.864 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 20, 2021-05-25T12:01:16.864 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:01:16.864 INFO:teuthology.orchestra.run.smithi102.stdout: 0, 2021-05-25T12:01:16.864 INFO:teuthology.orchestra.run.smithi102.stdout: 1, 2021-05-25T12:01:16.864 INFO:teuthology.orchestra.run.smithi102.stdout: 2 2021-05-25T12:01:16.865 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:01:16.865 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 0, 2021-05-25T12:01:16.865 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:01:16.865 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:01:16.865 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:01:16.866 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:01:16.866 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:01:16.866 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:01:16.866 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:01:16.867 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:01:16.867 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:01:16.867 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:01:16.867 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:01:16.867 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:01:16.868 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:01:16.868 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:01:16.868 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:01:16.868 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:01:16.868 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:01:16.869 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:01:16.869 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:01:16.869 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:01:16.869 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:01:16.870 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:01:16.870 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:01:16.870 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:01:16.870 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:01:16.870 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:01:16.871 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:01:16.871 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:01:16.871 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:01:16.871 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:01:16.871 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:01:16.872 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:01:16.872 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:01:16.872 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:01:16.872 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:01:16.872 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:01:16.873 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:01:16.873 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:01:16.873 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:01:16.873 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:01:16.873 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:01:16.874 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:01:16.874 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:01:16.874 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:01:16.874 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:01:16.874 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:01:16.875 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:01:16.875 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:01:16.875 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:01:16.875 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:01:16.875 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:01:16.876 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:01:16.876 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:01:16.876 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:01:16.876 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:01:16.876 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:01:16.877 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:01:16.877 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:01:16.877 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:01:16.877 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:01:16.877 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:01:16.878 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:01:16.878 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:01:16.878 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:01:16.878 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:01:16.878 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:01:16.879 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:01:16.879 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:01:16.879 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:01:16.879 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:01:16.879 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:01:16.880 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:01:16.880 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:01:16.880 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:01:16.880 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:01:16.880 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:01:16.881 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:01:16.881 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:01:16.881 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:01:16.881 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:01:16.881 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:01:16.882 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:01:16.882 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:01:16.882 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:01:16.882 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:01:16.882 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:01:16.883 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:01:16.883 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:01:16.883 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:01:16.883 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:01:16.883 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:01:16.884 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:01:16.884 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:01:16.884 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:01:16.884 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:01:16.884 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:01:16.885 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:01:16.885 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:01:16.885 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:01:16.885 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:01:16.886 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:01:16.886 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:01:16.886 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:01:16.886 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:01:16.886 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:01:16.887 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:01:16.887 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:01:16.887 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:01:16.887 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:01:16.887 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:01:16.888 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:01:16.888 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:01:16.888 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:01:16.888 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:01:16.888 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:01:16.889 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:01:16.889 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:01:16.889 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:01:16.889 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:01:16.889 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:01:16.890 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:01:16.890 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:01:16.890 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 12 2021-05-25T12:01:16.890 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:01:16.890 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:01:16.891 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:01:16.891 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:01:16.891 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:01:16.893 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.803+0000 7fe764a9a700 10 librados: watch_flush enter 2021-05-25T12:01:16.893 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.803+0000 7fe764a9a700 10 librados: watch_flush exit 2021-05-25T12:01:16.894 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.803+0000 7fe764a9a700 20 client.84513.objecter shutdown clearing up homeless session... 2021-05-25T12:01:16.894 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.803+0000 7fe764a9a700 10 client.84513.objecter successfully canceled tick 2021-05-25T12:01:16.894 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.803+0000 7fe764a9a700 1 -- 172.21.15.102:0/3330467199 >> v1:172.21.15.181:6812/29582 conn(0x7fe740033220 legacy=0x7fe7400356e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:16.894 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.803+0000 7fe764a9a700 10 monclient: shutdown 2021-05-25T12:01:16.894 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.803+0000 7fe764a9a700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:01:16.895 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.803+0000 7fe764a9a700 1 -- 172.21.15.102:0/3330467199 >> v1:172.21.15.102:6789/0 conn(0x7fe760098930 legacy=0x7fe7600972e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:16.895 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.803+0000 7fe75ffff700 1 -- 172.21.15.102:0/3330467199 reap_dead start 2021-05-25T12:01:16.895 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.803+0000 7fe764a9a700 1 -- 172.21.15.102:0/3330467199 shutdown_connections 2021-05-25T12:01:16.895 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.803+0000 7fe764a9a700 1 -- 172.21.15.102:0/3330467199 >> 172.21.15.102:0/3330467199 conn(0x7fe76009e9d0 msgr2=0x7fe76009bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:01:16.895 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.803+0000 7fe764a9a700 1 -- 172.21.15.102:0/3330467199 shutdown_connections 2021-05-25T12:01:16.896 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.803+0000 7fe764a9a700 1 -- 172.21.15.102:0/3330467199 wait complete. 2021-05-25T12:01:16.896 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:16.804+0000 7fe764a9a700 1 librados: shutdown 2021-05-25T12:01:16.896 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2021-05-25T12:01:17.017 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.015+0000 7fca860f5700 10 monclient: get_monmap_and_config 2021-05-25T12:01:17.017 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.015+0000 7fca860f5700 10 monclient: build_initial_monmap 2021-05-25T12:01:17.017 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.015+0000 7fca860f5700 10 monclient: monmap: 2021-05-25T12:01:17.018 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:01:17.018 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:17.018 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:01:17.016826+0000 2021-05-25T12:01:17.018 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:01:17.016826+0000 2021-05-25T12:01:17.018 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:01:17.019 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:17.019 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:01:17.019 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:01:17.019 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:01:17.019 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:17.030 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.016+0000 7fca860f5700 1 Processor -- start 2021-05-25T12:01:17.030 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.016+0000 7fca860f5700 1 -- start start 2021-05-25T12:01:17.030 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.016+0000 7fca860f5700 10 monclient: init 2021-05-25T12:01:17.030 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.017+0000 7fca860f5700 10 monclient: _reopen_session rank -1 2021-05-25T12:01:17.031 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.017+0000 7fca860f5700 10 monclient: _add_conns ranks=[1,2,0] 2021-05-25T12:01:17.031 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.017+0000 7fca860f5700 10 monclient(hunting): picked mon.noname-c con 0x7fca80069070 addr v1:172.21.15.181:6789/0 2021-05-25T12:01:17.031 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.017+0000 7fca860f5700 10 monclient(hunting): picked mon.noname-b con 0x7fca800954b0 addr v1:172.21.15.102:6790/0 2021-05-25T12:01:17.031 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.017+0000 7fca860f5700 10 monclient(hunting): picked mon.noname-a con 0x7fca80095930 addr v1:172.21.15.102:6789/0 2021-05-25T12:01:17.031 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.017+0000 7fca860f5700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca80096820 con 0x7fca80095930 2021-05-25T12:01:17.032 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.017+0000 7fca860f5700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca80069450 con 0x7fca80069070 2021-05-25T12:01:17.032 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.017+0000 7fca860f5700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca80097bf0 con 0x7fca800954b0 2021-05-25T12:01:17.032 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.017+0000 7fca860f5700 10 monclient(hunting): _renew_subs 2021-05-25T12:01:17.032 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.017+0000 7fca860f5700 10 monclient(hunting): authenticate will time out at 877.614160s 2021-05-25T12:01:17.032 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.017+0000 7fca7effd700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fca800954b0 0x7fca80130aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:54582/0 (socket says 172.21.15.102:54582) 2021-05-25T12:01:17.033 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.017+0000 7fca7effd700 1 -- 172.21.15.102:0/3392700098 learned_addr learned my addr 172.21.15.102:0/3392700098 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:01:17.033 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.017+0000 7fca7ffff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fca80095930 0x7fca801341d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:59706/0 (socket says 172.21.15.102:59706) 2021-05-25T12:01:17.033 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.017+0000 7fca7e7fc700 1 -- 172.21.15.102:0/3392700098 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3102764431 0 0) 0x7fca80096820 con 0x7fca80095930 2021-05-25T12:01:17.033 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.017+0000 7fca7e7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:01:17.034 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.017+0000 7fca7e7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:01:17.034 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.017+0000 7fca7e7fc700 10 monclient(hunting): my global_id is 84711 2021-05-25T12:01:17.034 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.017+0000 7fca7e7fc700 1 -- 172.21.15.102:0/3392700098 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca64003270 con 0x7fca80095930 2021-05-25T12:01:17.034 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.017+0000 7fca7e7fc700 1 -- 172.21.15.102:0/3392700098 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2039236373 0 0) 0x7fca80097bf0 con 0x7fca800954b0 2021-05-25T12:01:17.034 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.017+0000 7fca7e7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:01:17.035 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.017+0000 7fca7e7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:01:17.035 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.017+0000 7fca7e7fc700 10 monclient(hunting): my global_id is 13910 2021-05-25T12:01:17.035 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.017+0000 7fca7e7fc700 1 -- 172.21.15.102:0/3392700098 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca80096820 con 0x7fca800954b0 2021-05-25T12:01:17.035 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.017+0000 7fca7e7fc700 1 -- 172.21.15.102:0/3392700098 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2547637828 0 0) 0x7fca80096820 con 0x7fca800954b0 2021-05-25T12:01:17.035 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.017+0000 7fca7e7fc700 1 -- 172.21.15.102:0/3392700098 >> v1:172.21.15.181:6789/0 conn(0x7fca80069070 legacy=0x7fca80094da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-05-25T12:01:17.036 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.017+0000 7fca7e7fc700 1 -- 172.21.15.102:0/3392700098 >> v1:172.21.15.102:6789/0 conn(0x7fca80095930 legacy=0x7fca801341d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:17.036 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.017+0000 7fca7e7fc700 10 monclient: _finish_hunting 0 2021-05-25T12:01:17.036 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.017+0000 7fca7e7fc700 1 monclient: found mon.noname-b 2021-05-25T12:01:17.036 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.017+0000 7fca7e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:01:17.037 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.017+0000 7fca7e7fc700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T12:01:17.037 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.017+0000 7fca7e7fc700 1 -- 172.21.15.102:0/3392700098 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca80097e90 con 0x7fca800954b0 2021-05-25T12:01:17.037 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.017+0000 7fca7e7fc700 10 monclient: _finish_auth 0 2021-05-25T12:01:17.037 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.017+0000 7fca7e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:01:17.037 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.017+0000 7fca860f5700 5 monclient: authenticate success, global_id 13910 2021-05-25T12:01:17.037 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.017+0000 7fca7e7fc700 1 -- 172.21.15.102:0/3392700098 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fca70001650 con 0x7fca800954b0 2021-05-25T12:01:17.038 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.017+0000 7fca860f5700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:01:17.038 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.017+0000 7fca7e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:01:17.038 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.017+0000 7fca7e7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:01:17.038 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.017+0000 7fca7e7fc700 10 monclient: dump: 2021-05-25T12:01:17.038 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:01:17.039 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:17.039 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:17.039 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:17.039 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:01:17.039 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:17.040 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:01:17.040 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:01:17.040 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:01:17.040 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:17.040 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.017+0000 7fca860f5700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:01:17.041 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.017+0000 7fca7e7fc700 1 -- 172.21.15.102:0/3392700098 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fca700017f0 con 0x7fca800954b0 2021-05-25T12:01:17.041 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.017+0000 7fca7e7fc700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:01:17.041 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.017+0000 7fca7e7fc700 1 -- 172.21.15.102:0/3392700098 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fca70001b80 con 0x7fca800954b0 2021-05-25T12:01:17.041 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.017+0000 7fca860f5700 10 monclient: get_monmap_and_config success 2021-05-25T12:01:17.041 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.017+0000 7fca860f5700 4 set_mon_vals no callback set 2021-05-25T12:01:17.042 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.017+0000 7fca7e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:01:17.042 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.017+0000 7fca7e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:01:17.042 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.017+0000 7fca7e7fc700 10 monclient: dump: 2021-05-25T12:01:17.042 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:01:17.042 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:17.043 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:17.043 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:17.043 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:01:17.043 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:17.043 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:01:17.043 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:01:17.044 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:01:17.044 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:17.044 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.018+0000 7fca860f5700 10 monclient: shutdown 2021-05-25T12:01:17.044 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.018+0000 7fca860f5700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:01:17.044 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.018+0000 7fca860f5700 1 -- 172.21.15.102:0/3392700098 >> v1:172.21.15.102:6790/0 conn(0x7fca800954b0 legacy=0x7fca80130aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:17.045 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.018+0000 7fca860f5700 1 -- 172.21.15.102:0/3392700098 shutdown_connections 2021-05-25T12:01:17.045 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.018+0000 7fca860f5700 1 -- 172.21.15.102:0/3392700098 >> 172.21.15.102:0/3392700098 conn(0x7fca800984c0 msgr2=0x7fca800988d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:01:17.045 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.018+0000 7fca860f5700 1 -- 172.21.15.102:0/3392700098 shutdown_connections 2021-05-25T12:01:17.045 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.018+0000 7fca860f5700 1 -- 172.21.15.102:0/3392700098 wait complete. 2021-05-25T12:01:17.045 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.018+0000 7fca860f5700 10 monclient: build_initial_monmap 2021-05-25T12:01:17.046 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.018+0000 7fca860f5700 10 monclient: monmap: 2021-05-25T12:01:17.046 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:01:17.046 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:01:17.046 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:01:17.047 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:01:17.047 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:01:17.047 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:01:17.047 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:01:17.047 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:01:17.048 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:01:17.048 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:17.048 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.019+0000 7fca860f5700 1 librados: starting msgr at 2021-05-25T12:01:17.048 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.019+0000 7fca860f5700 1 librados: starting objecter 2021-05-25T12:01:17.048 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.019+0000 7fca860f5700 1 Processor -- start 2021-05-25T12:01:17.049 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.019+0000 7fca860f5700 1 -- start start 2021-05-25T12:01:17.049 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.019+0000 7fca860f5700 1 librados: setting wanted keys 2021-05-25T12:01:17.049 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.019+0000 7fca860f5700 1 librados: calling monclient init 2021-05-25T12:01:17.049 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.019+0000 7fca860f5700 10 monclient: init 2021-05-25T12:01:17.049 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.020+0000 7fca860f5700 10 monclient: _reopen_session rank -1 2021-05-25T12:01:17.050 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.020+0000 7fca860f5700 10 monclient: _add_conns ranks=[1,0,2] 2021-05-25T12:01:17.050 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.020+0000 7fca860f5700 10 monclient(hunting): picked mon.noname-b con 0x7fca80069070 addr v1:172.21.15.181:6789/0 2021-05-25T12:01:17.050 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.020+0000 7fca860f5700 10 monclient(hunting): picked mon.noname-a con 0x7fca800954b0 addr v1:172.21.15.102:6789/0 2021-05-25T12:01:17.050 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.020+0000 7fca860f5700 10 monclient(hunting): picked mon.noname-c con 0x7fca80095930 addr v1:172.21.15.102:6790/0 2021-05-25T12:01:17.050 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.020+0000 7fca7effd700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fca800954b0 0x7fca8013fc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:59708/0 (socket says 172.21.15.102:59708) 2021-05-25T12:01:17.051 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.020+0000 7fca7effd700 1 -- 172.21.15.102:0/3373107661 learned_addr learned my addr 172.21.15.102:0/3373107661 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:01:17.051 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.020+0000 7fca860f5700 1 -- 172.21.15.102:0/3373107661 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca80128730 con 0x7fca800954b0 2021-05-25T12:01:17.051 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.020+0000 7fca860f5700 1 -- 172.21.15.102:0/3373107661 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca801289d0 con 0x7fca80069070 2021-05-25T12:01:17.051 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.020+0000 7fca860f5700 1 -- 172.21.15.102:0/3373107661 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca80128c70 con 0x7fca80095930 2021-05-25T12:01:17.051 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.020+0000 7fca860f5700 10 monclient(hunting): _renew_subs 2021-05-25T12:01:17.052 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.020+0000 7fca860f5700 10 monclient(hunting): authenticate will time out at 877.617284s 2021-05-25T12:01:17.052 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.020+0000 7fca6b7fe700 10 client.?.objecter ms_handle_connect 0x7fca800954b0 2021-05-25T12:01:17.052 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.020+0000 7fca6b7fe700 10 client.?.objecter resend_mon_ops 2021-05-25T12:01:17.052 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.020+0000 7fca6b7fe700 10 client.?.objecter ms_handle_connect 0x7fca80095930 2021-05-25T12:01:17.052 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.020+0000 7fca6b7fe700 10 client.?.objecter resend_mon_ops 2021-05-25T12:01:17.053 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.020+0000 7fca6b7fe700 1 -- 172.21.15.102:0/3373107661 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1665480741 0 0) 0x7fca80128c70 con 0x7fca80095930 2021-05-25T12:01:17.053 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.020+0000 7fca6b7fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:01:17.053 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.020+0000 7fca6b7fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:01:17.053 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.020+0000 7fca6b7fe700 10 monclient(hunting): my global_id is 13925 2021-05-25T12:01:17.054 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.020+0000 7fca6b7fe700 1 -- 172.21.15.102:0/3373107661 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca60004c10 con 0x7fca80095930 2021-05-25T12:01:17.054 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.020+0000 7fca6b7fe700 1 -- 172.21.15.102:0/3373107661 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2056406514 0 0) 0x7fca80128730 con 0x7fca800954b0 2021-05-25T12:01:17.054 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.020+0000 7fca6b7fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:01:17.054 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.020+0000 7fca6b7fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:01:17.054 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.020+0000 7fca6b7fe700 10 monclient(hunting): my global_id is 84729 2021-05-25T12:01:17.055 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.020+0000 7fca6b7fe700 1 -- 172.21.15.102:0/3373107661 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca80128c70 con 0x7fca800954b0 2021-05-25T12:01:17.055 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.020+0000 7fca6b7fe700 1 -- 172.21.15.102:0/3373107661 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 760192769 0 0) 0x7fca60004c10 con 0x7fca80095930 2021-05-25T12:01:17.055 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.020+0000 7fca6b7fe700 1 -- 172.21.15.102:0/3373107661 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca80128730 con 0x7fca80095930 2021-05-25T12:01:17.055 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.020+0000 7fca6b7fe700 1 -- 172.21.15.102:0/3373107661 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fca74001d70 con 0x7fca80095930 2021-05-25T12:01:17.055 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.020+0000 7fca6b7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:01:17.056 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.020+0000 7fca6b7fe700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:01:17.056 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.020+0000 7fca6b7fe700 10 monclient(hunting): dump: 2021-05-25T12:01:17.056 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:01:17.056 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:17.056 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:17.057 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:17.057 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:01:17.057 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:17.057 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:01:17.058 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:01:17.058 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:01:17.058 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:17.058 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.020+0000 7fca6b7fe700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:01:17.058 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.020+0000 7fca6b7fe700 1 -- 172.21.15.102:0/3373107661 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 433422091 0 0) 0x7fca80128c70 con 0x7fca800954b0 2021-05-25T12:01:17.059 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.020+0000 7fca6b7fe700 1 -- 172.21.15.102:0/3373107661 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca60004c10 con 0x7fca800954b0 2021-05-25T12:01:17.059 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.020+0000 7fca6b7fe700 1 -- 172.21.15.102:0/3373107661 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fca70001610 con 0x7fca800954b0 2021-05-25T12:01:17.059 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.020+0000 7fca6b7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:01:17.059 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.020+0000 7fca6b7fe700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2021-05-25T12:01:17.059 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.020+0000 7fca6b7fe700 10 monclient(hunting): dump: 2021-05-25T12:01:17.060 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:01:17.060 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:17.060 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:17.060 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:17.060 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:01:17.061 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:17.061 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:01:17.061 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:01:17.061 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:01:17.061 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:17.062 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.020+0000 7fca6b7fe700 1 -- 172.21.15.102:0/3373107661 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3286633849 0 0) 0x7fca80128730 con 0x7fca80095930 2021-05-25T12:01:17.062 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.020+0000 7fca6b7fe700 1 -- 172.21.15.102:0/3373107661 >> v1:172.21.15.181:6789/0 conn(0x7fca80069070 legacy=0x7fca80134850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:17.062 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.020+0000 7fca6b7fe700 1 -- 172.21.15.102:0/3373107661 >> v1:172.21.15.102:6789/0 conn(0x7fca800954b0 legacy=0x7fca8013fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:17.062 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.020+0000 7fca6b7fe700 10 monclient: _finish_hunting 0 2021-05-25T12:01:17.063 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.020+0000 7fca6b7fe700 1 monclient: found mon.c 2021-05-25T12:01:17.063 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.020+0000 7fca6b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:01:17.063 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.020+0000 7fca6b7fe700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:01:17.063 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.020+0000 7fca6b7fe700 1 -- 172.21.15.102:0/3373107661 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca80126b00 con 0x7fca80095930 2021-05-25T12:01:17.063 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.020+0000 7fca6b7fe700 10 monclient: _finish_auth 0 2021-05-25T12:01:17.064 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.020+0000 7fca6b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:01:17.064 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.021+0000 7fca860f5700 5 monclient: authenticate success, global_id 13925 2021-05-25T12:01:17.064 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.021+0000 7fca860f5700 10 monclient: _renew_subs 2021-05-25T12:01:17.064 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.021+0000 7fca860f5700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:01:17.064 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.021+0000 7fca860f5700 1 -- 172.21.15.102:0/3373107661 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca80134f60 con 0x7fca80095930 2021-05-25T12:01:17.065 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.021+0000 7fca860f5700 10 client.13925.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:01:17.065 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.021+0000 7fca860f5700 10 monclient: _renew_subs 2021-05-25T12:01:17.065 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.021+0000 7fca860f5700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:01:17.065 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.021+0000 7fca860f5700 1 -- 172.21.15.102:0/3373107661 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fca8013b800 con 0x7fca80095930 2021-05-25T12:01:17.065 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.021+0000 7fca860f5700 1 librados: init done 2021-05-25T12:01:17.066 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.021+0000 7fca6b7fe700 1 -- 172.21.15.102:0/3373107661 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fca74002050 con 0x7fca80095930 2021-05-25T12:01:17.066 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.021+0000 7fca6b7fe700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:01:17.066 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.021+0000 7fca7dffb700 4 set_mon_vals no callback set 2021-05-25T12:01:17.066 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.021+0000 7fca6b7fe700 1 -- 172.21.15.102:0/3373107661 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fca74004d00 con 0x7fca80095930 2021-05-25T12:01:17.067 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.021+0000 7fca6b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:01:17.067 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.021+0000 7fca6b7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:01:17.067 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.021+0000 7fca6b7fe700 10 monclient: dump: 2021-05-25T12:01:17.067 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:01:17.067 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:17.068 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:17.068 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:17.068 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:01:17.068 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:17.068 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:01:17.069 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:01:17.069 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:01:17.069 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:17.069 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.021+0000 7fca860f5700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:01:17.069 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.021+0000 7fca6b7fe700 1 -- 172.21.15.102:0/3373107661 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7fca7400e020 con 0x7fca80095930 2021-05-25T12:01:17.069 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.021+0000 7fca860f5700 1 -- 172.21.15.102:0/3373107661 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fca580065a0 con 0x7fca58002840 2021-05-25T12:01:17.070 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.022+0000 7fca6b7fe700 1 -- 172.21.15.102:0/3373107661 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(38..38 src has 1..38) v4 ==== 8377+0+0 (unknown 2583901202 0 0) 0x7fca7403e3a0 con 0x7fca80095930 2021-05-25T12:01:17.070 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.022+0000 7fca6b7fe700 10 client.13925.objecter ms_dispatch 0x7fca80001550 osd_map(38..38 src has 1..38) v4 2021-05-25T12:01:17.070 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.022+0000 7fca6b7fe700 3 client.13925.objecter handle_osd_map got epochs [38,38] > 0 2021-05-25T12:01:17.070 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.022+0000 7fca6b7fe700 3 client.13925.objecter handle_osd_map decoding full epoch 38 2021-05-25T12:01:17.071 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.022+0000 7fca6b7fe700 20 client.13925.objecter dump_active .. 0 homeless 2021-05-25T12:01:17.071 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.022+0000 7fca6b7fe700 10 client.13925.objecter ms_handle_connect 0x7fca58002840 2021-05-25T12:01:17.071 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.022+0000 7fca6b7fe700 10 client.13925.objecter resend_mon_ops 2021-05-25T12:01:17.071 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.022+0000 7fca6b7fe700 1 -- 172.21.15.102:0/3373107661 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1154323928 0 0) 0x7fca580065a0 con 0x7fca58002840 2021-05-25T12:01:17.071 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.022+0000 7fca6b7fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:01:17.072 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.022+0000 7fca6b7fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:01:17.072 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.022+0000 7fca6b7fe700 10 monclient(hunting): my global_id is 13934 2021-05-25T12:01:17.072 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.022+0000 7fca6b7fe700 1 -- 172.21.15.102:0/3373107661 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca80128730 con 0x7fca58002840 2021-05-25T12:01:17.072 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.022+0000 7fca6b7fe700 1 -- 172.21.15.102:0/3373107661 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1011009012 0 0) 0x7fca80128730 con 0x7fca58002840 2021-05-25T12:01:17.072 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.022+0000 7fca6b7fe700 1 -- 172.21.15.102:0/3373107661 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca580065a0 con 0x7fca58002840 2021-05-25T12:01:17.072 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.022+0000 7fca6b7fe700 1 -- 172.21.15.102:0/3373107661 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fca6c001520 con 0x7fca58002840 2021-05-25T12:01:17.073 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.022+0000 7fca6b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:01:17.073 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.022+0000 7fca6b7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:01:17.073 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.022+0000 7fca6b7fe700 10 monclient: dump: 2021-05-25T12:01:17.073 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:01:17.073 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:17.074 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:17.074 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:17.074 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:01:17.074 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:17.074 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:01:17.075 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:01:17.075 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:01:17.075 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:17.075 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.022+0000 7fca6b7fe700 1 -- 172.21.15.102:0/3373107661 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2558438175 0 0) 0x7fca580065a0 con 0x7fca58002840 2021-05-25T12:01:17.075 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.022+0000 7fca6b7fe700 1 -- 172.21.15.102:0/3373107661 --> v1:172.21.15.102:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fca58009e40 con 0x7fca58002840 2021-05-25T12:01:17.076 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.023+0000 7fca6b7fe700 1 -- 172.21.15.102:0/3373107661 <== mon.2 v1:172.21.15.102:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7fca6c002a40 con 0x7fca58002840 2021-05-25T12:01:17.076 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.023+0000 7fca6b7fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:01:17.076 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.023+0000 7fca6b7fe700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:01:17.076 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.023+0000 7fca6b7fe700 1 -- 172.21.15.102:0/3373107661 >> v1:172.21.15.102:6790/0 conn(0x7fca58002840 legacy=0x7fca58004ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:17.077 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.023+0000 7fca6b7fe700 1 -- 172.21.15.102:0/3373107661 >> v1:172.21.15.102:6790/0 conn(0x7fca58002840 legacy=0x7fca58004ce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:01:17.077 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.023+0000 7fca7effd700 10 monclient: get_auth_request con 0x7fca600331f0 auth_method 0 2021-05-25T12:01:17.077 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.023+0000 7fca6b7fe700 10 client.13925.objecter ms_handle_connect 0x7fca600331f0 2021-05-25T12:01:17.079 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.029+0000 7fca860f5700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:01:17.079 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.029+0000 7fca860f5700 1 -- 172.21.15.102:0/3373107661 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fca58009ad0 con 0x7fca58008d90 2021-05-25T12:01:17.079 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.030+0000 7fca6b7fe700 10 client.13925.objecter ms_handle_connect 0x7fca58008d90 2021-05-25T12:01:17.080 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.030+0000 7fca6b7fe700 10 client.13925.objecter resend_mon_ops 2021-05-25T12:01:17.080 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.030+0000 7fca6b7fe700 1 -- 172.21.15.102:0/3373107661 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1440054568 0 0) 0x7fca58009ad0 con 0x7fca58008d90 2021-05-25T12:01:17.080 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.030+0000 7fca6b7fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:01:17.080 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.030+0000 7fca6b7fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:01:17.080 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.030+0000 7fca6b7fe700 10 monclient(hunting): my global_id is 13940 2021-05-25T12:01:17.081 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.030+0000 7fca6b7fe700 1 -- 172.21.15.102:0/3373107661 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca580065a0 con 0x7fca58008d90 2021-05-25T12:01:17.081 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.030+0000 7fca6b7fe700 1 -- 172.21.15.102:0/3373107661 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 529008014 0 0) 0x7fca580065a0 con 0x7fca58008d90 2021-05-25T12:01:17.081 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.030+0000 7fca6b7fe700 1 -- 172.21.15.102:0/3373107661 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca58009ad0 con 0x7fca58008d90 2021-05-25T12:01:17.081 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.030+0000 7fca6b7fe700 1 -- 172.21.15.102:0/3373107661 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fca6c001c10 con 0x7fca58008d90 2021-05-25T12:01:17.081 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.030+0000 7fca6b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:01:17.082 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.030+0000 7fca6b7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:01:17.082 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.030+0000 7fca6b7fe700 10 monclient: dump: 2021-05-25T12:01:17.082 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:01:17.082 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:17.083 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:17.083 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:17.083 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:01:17.083 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:17.083 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:01:17.084 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:01:17.084 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:01:17.084 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:17.084 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.030+0000 7fca6b7fe700 1 -- 172.21.15.102:0/3373107661 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 15294851 0 0) 0x7fca58009ad0 con 0x7fca58008d90 2021-05-25T12:01:17.084 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.030+0000 7fca6b7fe700 1 -- 172.21.15.102:0/3373107661 --> v1:172.21.15.102:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fca58008740 con 0x7fca58008d90 2021-05-25T12:01:17.085 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.030+0000 7fca6b7fe700 1 -- 172.21.15.102:0/3373107661 <== mon.2 v1:172.21.15.102:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4113 (unknown 0 0 576805600) 0x7fca6c001f50 con 0x7fca58008d90 2021-05-25T12:01:17.085 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.030+0000 7fca6b7fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:01:17.085 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.030+0000 7fca6b7fe700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:01:17.085 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.030+0000 7fca6b7fe700 1 -- 172.21.15.102:0/3373107661 >> v1:172.21.15.102:6790/0 conn(0x7fca58008d90 legacy=0x7fca58009220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:17.086 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.030+0000 7fca6b7fe700 1 -- 172.21.15.102:0/3373107661 >> v1:172.21.15.102:6790/0 conn(0x7fca58008d90 legacy=0x7fca58009220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:01:17.086 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:01:17.086 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:01:17.086 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:01:17.087 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "leader", 2021-05-25T12:01:17.087 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 20, 2021-05-25T12:01:17.087 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:01:17.087 INFO:teuthology.orchestra.run.smithi102.stdout: 0, 2021-05-25T12:01:17.088 INFO:teuthology.orchestra.run.smithi102.stdout: 1, 2021-05-25T12:01:17.088 INFO:teuthology.orchestra.run.smithi102.stdout: 2 2021-05-25T12:01:17.088 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:01:17.088 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 0, 2021-05-25T12:01:17.088 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:01:17.089 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:01:17.089 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:01:17.089 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:01:17.089 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:01:17.089 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:01:17.090 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:01:17.090 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:01:17.090 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:01:17.090 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:01:17.090 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:01:17.091 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:01:17.091 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:01:17.091 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:01:17.091 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:01:17.092 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:01:17.092 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:01:17.092 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:01:17.092 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:01:17.092 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:01:17.093 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:01:17.093 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:01:17.093 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:01:17.093 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:01:17.093 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:01:17.093 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:01:17.094 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:01:17.094 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:01:17.094 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:01:17.094 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:01:17.095 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:01:17.095 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:01:17.095 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:01:17.095 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:01:17.095 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:01:17.096 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:01:17.096 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:01:17.096 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:01:17.096 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:01:17.097 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:01:17.097 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:01:17.097 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:01:17.097 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:01:17.097 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:01:17.098 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:01:17.098 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:01:17.098 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:01:17.098 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:01:17.099 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:01:17.099 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:01:17.099 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:01:17.099 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:01:17.099 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:01:17.100 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:01:17.100 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:01:17.100 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:01:17.100 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:01:17.100 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:01:17.101 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:01:17.101 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:01:17.101 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:01:17.101 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:01:17.101 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:01:17.102 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:01:17.102 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:01:17.102 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:01:17.102 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:01:17.102 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:01:17.103 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:01:17.103 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:01:17.103 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:01:17.103 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:01:17.103 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:01:17.104 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:01:17.104 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:01:17.104 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:01:17.104 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:01:17.104 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:01:17.105 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:01:17.105 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:01:17.105 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:01:17.105 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:01:17.105 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:01:17.106 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:01:17.106 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:01:17.106 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:01:17.106 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:01:17.106 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:01:17.107 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:01:17.107 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:01:17.107 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:01:17.107 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:01:17.108 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:01:17.108 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:01:17.108 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:01:17.108 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:01:17.108 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:01:17.109 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:01:17.109 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:01:17.109 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:01:17.109 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:01:17.109 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:01:17.110 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:01:17.110 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:01:17.110 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:01:17.110 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:01:17.110 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:01:17.111 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:01:17.111 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:01:17.111 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:01:17.111 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:01:17.111 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:01:17.112 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:01:17.112 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:01:17.112 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:01:17.112 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:01:17.113 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:01:17.113 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:01:17.113 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": [ 2021-05-25T12:01:17.113 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:01:17.113 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:01:17.114 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:01:17.114 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 6 2021-05-25T12:01:17.114 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:01:17.114 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:01:17.114 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:01:17.115 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:01:17.115 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:01:17.115 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:01:17.115 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 11 2021-05-25T12:01:17.115 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:01:17.116 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:01:17.116 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": [ 2021-05-25T12:01:17.116 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:01:17.116 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:01:17.117 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:01:17.117 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:01:17.117 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:01:17.117 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:01:17.117 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:01:17.118 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:01:17.118 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:01:17.120 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.030+0000 7fca860f5700 10 librados: watch_flush enter 2021-05-25T12:01:17.120 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.031+0000 7fca860f5700 10 librados: watch_flush exit 2021-05-25T12:01:17.120 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.031+0000 7fca860f5700 20 client.13925.objecter shutdown clearing up homeless session... 2021-05-25T12:01:17.121 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.031+0000 7fca860f5700 10 client.13925.objecter successfully canceled tick 2021-05-25T12:01:17.121 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.031+0000 7fca860f5700 1 -- 172.21.15.102:0/3373107661 >> v1:172.21.15.181:6812/29582 conn(0x7fca600331f0 legacy=0x7fca600356b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:17.121 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.031+0000 7fca860f5700 10 monclient: shutdown 2021-05-25T12:01:17.121 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.031+0000 7fca860f5700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:01:17.121 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.031+0000 7fca860f5700 1 -- 172.21.15.102:0/3373107661 >> v1:172.21.15.102:6790/0 conn(0x7fca80095930 legacy=0x7fca80143330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:17.122 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.031+0000 7fca7ffff700 1 -- 172.21.15.102:0/3373107661 reap_dead start 2021-05-25T12:01:17.122 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.031+0000 7fca7ffff700 1 -- 172.21.15.102:0/3373107661 reap_dead start 2021-05-25T12:01:17.122 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.031+0000 7fca860f5700 1 -- 172.21.15.102:0/3373107661 shutdown_connections 2021-05-25T12:01:17.122 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.031+0000 7fca860f5700 1 -- 172.21.15.102:0/3373107661 >> 172.21.15.102:0/3373107661 conn(0x7fca800984c0 msgr2=0x7fca80129de0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:01:17.123 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.031+0000 7fca860f5700 1 -- 172.21.15.102:0/3373107661 shutdown_connections 2021-05-25T12:01:17.123 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.031+0000 7fca860f5700 1 -- 172.21.15.102:0/3373107661 wait complete. 2021-05-25T12:01:17.123 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.031+0000 7fca860f5700 1 librados: shutdown 2021-05-25T12:01:17.123 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2021-05-25T12:01:17.247 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.245+0000 7f7cfe492700 10 monclient: get_monmap_and_config 2021-05-25T12:01:17.247 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.245+0000 7f7cfe492700 10 monclient: build_initial_monmap 2021-05-25T12:01:17.247 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.245+0000 7f7cfe492700 10 monclient: monmap: 2021-05-25T12:01:17.247 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:01:17.248 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:17.248 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:01:17.246571+0000 2021-05-25T12:01:17.248 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:01:17.246571+0000 2021-05-25T12:01:17.248 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:01:17.248 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:17.249 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:01:17.249 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:01:17.249 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:01:17.249 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:17.260 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.246+0000 7f7cfe492700 1 Processor -- start 2021-05-25T12:01:17.260 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.246+0000 7f7cfe492700 1 -- start start 2021-05-25T12:01:17.260 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.246+0000 7f7cfe492700 10 monclient: init 2021-05-25T12:01:17.260 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.246+0000 7f7cfe492700 10 monclient: _reopen_session rank -1 2021-05-25T12:01:17.260 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.246+0000 7f7cfe492700 10 monclient: _add_conns ranks=[0,1,2] 2021-05-25T12:01:17.261 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.246+0000 7f7cfe492700 10 monclient(hunting): picked mon.noname-a con 0x7f7cf8098930 addr v1:172.21.15.102:6789/0 2021-05-25T12:01:17.261 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.246+0000 7f7cfe492700 10 monclient(hunting): picked mon.noname-c con 0x7f7cf812bca0 addr v1:172.21.15.181:6789/0 2021-05-25T12:01:17.261 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.246+0000 7f7cfe492700 10 monclient(hunting): picked mon.noname-b con 0x7f7cf812c150 addr v1:172.21.15.102:6790/0 2021-05-25T12:01:17.261 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.246+0000 7f7cfe492700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cf8099b20 con 0x7f7cf8098930 2021-05-25T12:01:17.261 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.246+0000 7f7cfe492700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cf809a3a0 con 0x7f7cf812bca0 2021-05-25T12:01:17.262 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.246+0000 7f7cfe492700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cf809ac60 con 0x7f7cf812c150 2021-05-25T12:01:17.262 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.246+0000 7f7cfe492700 10 monclient(hunting): _renew_subs 2021-05-25T12:01:17.262 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.246+0000 7f7cfe492700 10 monclient(hunting): authenticate will time out at 877.843874s 2021-05-25T12:01:17.262 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.246+0000 7f7cfd490700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f7cf8098930 0x7f7cf812b590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:59726/0 (socket says 172.21.15.102:59726) 2021-05-25T12:01:17.263 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.246+0000 7f7cfd490700 1 -- 172.21.15.102:0/536010330 learned_addr learned my addr 172.21.15.102:0/536010330 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:01:17.263 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.246+0000 7f7cfdc91700 1 --1- 172.21.15.102:0/536010330 >> v1:172.21.15.102:6790/0 conn(0x7f7cf812c150 0x7f7cf8133ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:54608/0 (socket says 172.21.15.102:54608) 2021-05-25T12:01:17.263 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.247+0000 7f7ce7fff700 1 -- 172.21.15.102:0/536010330 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1996629439 0 0) 0x7f7cf809ac60 con 0x7f7cf812c150 2021-05-25T12:01:17.263 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.247+0000 7f7ce7fff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:01:17.264 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.247+0000 7f7ce7fff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:01:17.264 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.247+0000 7f7ce7fff700 10 monclient(hunting): my global_id is 14084 2021-05-25T12:01:17.264 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.247+0000 7f7ce7fff700 1 -- 172.21.15.102:0/536010330 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ce0003270 con 0x7f7cf812c150 2021-05-25T12:01:17.264 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.247+0000 7f7ce7fff700 1 -- 172.21.15.102:0/536010330 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 578779019 0 0) 0x7f7cf8099b20 con 0x7f7cf8098930 2021-05-25T12:01:17.264 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.247+0000 7f7ce7fff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:01:17.265 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.247+0000 7f7ce7fff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:01:17.265 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.247+0000 7f7ce7fff700 10 monclient(hunting): my global_id is 84939 2021-05-25T12:01:17.265 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.247+0000 7f7ce7fff700 1 -- 172.21.15.102:0/536010330 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cf809ac60 con 0x7f7cf8098930 2021-05-25T12:01:17.265 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.247+0000 7f7ce7fff700 1 -- 172.21.15.102:0/536010330 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1301337904 0 0) 0x7f7ce0003270 con 0x7f7cf812c150 2021-05-25T12:01:17.265 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.247+0000 7f7ce7fff700 1 -- 172.21.15.102:0/536010330 >> v1:172.21.15.181:6789/0 conn(0x7f7cf812bca0 legacy=0x7f7cf81308c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-05-25T12:01:17.266 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.247+0000 7f7ce7fff700 1 -- 172.21.15.102:0/536010330 >> v1:172.21.15.102:6789/0 conn(0x7f7cf8098930 legacy=0x7f7cf812b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:17.266 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.247+0000 7f7ce7fff700 10 monclient: _finish_hunting 0 2021-05-25T12:01:17.266 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.247+0000 7f7ce7fff700 1 monclient: found mon.noname-b 2021-05-25T12:01:17.266 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.247+0000 7f7ce7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:01:17.267 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.247+0000 7f7ce7fff700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T12:01:17.267 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.247+0000 7f7ce7fff700 1 -- 172.21.15.102:0/536010330 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cf809af00 con 0x7f7cf812c150 2021-05-25T12:01:17.267 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.247+0000 7f7ce7fff700 10 monclient: _finish_auth 0 2021-05-25T12:01:17.267 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.247+0000 7f7ce7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:01:17.267 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.247+0000 7f7ce7fff700 1 -- 172.21.15.102:0/536010330 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f7cf4001990 con 0x7f7cf812c150 2021-05-25T12:01:17.268 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.247+0000 7f7cfe492700 5 monclient: authenticate success, global_id 14084 2021-05-25T12:01:17.268 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.247+0000 7f7cfe492700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:01:17.268 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.247+0000 7f7ce7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:01:17.268 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.247+0000 7f7ce7fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:01:17.268 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.247+0000 7f7ce7fff700 10 monclient: dump: 2021-05-25T12:01:17.269 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:01:17.269 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:17.269 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:17.269 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:17.269 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:01:17.270 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:17.270 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:01:17.270 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:01:17.270 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:01:17.271 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:17.271 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.247+0000 7f7cfe492700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:01:17.271 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.247+0000 7f7ce7fff700 1 -- 172.21.15.102:0/536010330 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7cf4001c00 con 0x7f7cf812c150 2021-05-25T12:01:17.271 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.247+0000 7f7ce7fff700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:01:17.271 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.247+0000 7f7ce7fff700 1 -- 172.21.15.102:0/536010330 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f7cf4001f90 con 0x7f7cf812c150 2021-05-25T12:01:17.272 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.247+0000 7f7cfe492700 10 monclient: get_monmap_and_config success 2021-05-25T12:01:17.272 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.247+0000 7f7cfe492700 4 set_mon_vals no callback set 2021-05-25T12:01:17.272 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.247+0000 7f7ce7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:01:17.272 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.247+0000 7f7ce7fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:01:17.272 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.247+0000 7f7ce7fff700 10 monclient: dump: 2021-05-25T12:01:17.273 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:01:17.273 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:17.273 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:17.273 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:17.273 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:01:17.274 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:17.274 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:01:17.274 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:01:17.274 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:01:17.274 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:17.275 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.248+0000 7f7cfe492700 10 monclient: shutdown 2021-05-25T12:01:17.275 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.248+0000 7f7cfe492700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:01:17.275 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.248+0000 7f7cfe492700 1 -- 172.21.15.102:0/536010330 >> v1:172.21.15.102:6790/0 conn(0x7f7cf812c150 legacy=0x7f7cf8133ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:17.275 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.248+0000 7f7cfe492700 1 -- 172.21.15.102:0/536010330 shutdown_connections 2021-05-25T12:01:17.275 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.248+0000 7f7cfe492700 1 -- 172.21.15.102:0/536010330 >> 172.21.15.102:0/536010330 conn(0x7f7cf809e9d0 msgr2=0x7f7cf809ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:01:17.276 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.248+0000 7f7cfe492700 1 -- 172.21.15.102:0/536010330 shutdown_connections 2021-05-25T12:01:17.276 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.248+0000 7f7cfe492700 1 -- 172.21.15.102:0/536010330 wait complete. 2021-05-25T12:01:17.276 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.248+0000 7f7cfe492700 10 monclient: build_initial_monmap 2021-05-25T12:01:17.276 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.248+0000 7f7cfe492700 10 monclient: monmap: 2021-05-25T12:01:17.276 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:01:17.277 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:01:17.277 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:01:17.277 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:01:17.277 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:01:17.277 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:01:17.278 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:01:17.278 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:01:17.278 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:01:17.278 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:17.278 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.248+0000 7f7cfe492700 1 librados: starting msgr at 2021-05-25T12:01:17.279 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.248+0000 7f7cfe492700 1 librados: starting objecter 2021-05-25T12:01:17.279 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.249+0000 7f7cfe492700 1 Processor -- start 2021-05-25T12:01:17.279 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.249+0000 7f7cfe492700 1 -- start start 2021-05-25T12:01:17.279 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.249+0000 7f7cfe492700 1 librados: setting wanted keys 2021-05-25T12:01:17.280 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.249+0000 7f7cfe492700 1 librados: calling monclient init 2021-05-25T12:01:17.280 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.249+0000 7f7cfe492700 10 monclient: init 2021-05-25T12:01:17.280 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.249+0000 7f7cfe492700 10 monclient: _reopen_session rank -1 2021-05-25T12:01:17.280 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.249+0000 7f7cfe492700 10 monclient: _add_conns ranks=[2,0,1] 2021-05-25T12:01:17.280 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.249+0000 7f7cfe492700 10 monclient(hunting): picked mon.noname-c con 0x7f7cf8098930 addr v1:172.21.15.102:6790/0 2021-05-25T12:01:17.281 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.249+0000 7f7cfe492700 10 monclient(hunting): picked mon.noname-a con 0x7f7cf812bca0 addr v1:172.21.15.102:6789/0 2021-05-25T12:01:17.281 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.249+0000 7f7cfe492700 10 monclient(hunting): picked mon.noname-b con 0x7f7cf812c150 addr v1:172.21.15.181:6789/0 2021-05-25T12:01:17.281 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.249+0000 7f7cfe492700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cf8130020 con 0x7f7cf812bca0 2021-05-25T12:01:17.281 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.249+0000 7f7cfe492700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cf812e150 con 0x7f7cf812c150 2021-05-25T12:01:17.281 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.249+0000 7f7cfe492700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cf8130250 con 0x7f7cf8098930 2021-05-25T12:01:17.282 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.249+0000 7f7cfe492700 10 monclient(hunting): _renew_subs 2021-05-25T12:01:17.282 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.249+0000 7f7cfe492700 10 monclient(hunting): authenticate will time out at 877.846777s 2021-05-25T12:01:17.282 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.249+0000 7f7cfd490700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f7cf8098930 0x7f7cf8128360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:54610/0 (socket says 172.21.15.102:54610) 2021-05-25T12:01:17.282 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.249+0000 7f7cfcc8f700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f7cf812bca0 0x7f7cf8128a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:59734/0 (socket says 172.21.15.102:59734) 2021-05-25T12:01:17.282 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.249+0000 7f7cfd490700 1 -- 172.21.15.102:0/2026322702 learned_addr learned my addr 172.21.15.102:0/2026322702 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:01:17.283 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.249+0000 7f7ce57fa700 10 client.?.objecter ms_handle_connect 0x7f7cf812bca0 2021-05-25T12:01:17.283 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.249+0000 7f7ce57fa700 10 client.?.objecter resend_mon_ops 2021-05-25T12:01:17.283 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.249+0000 7f7ce57fa700 10 client.?.objecter ms_handle_connect 0x7f7cf8098930 2021-05-25T12:01:17.283 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.249+0000 7f7ce57fa700 10 client.?.objecter resend_mon_ops 2021-05-25T12:01:17.283 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.250+0000 7f7ce57fa700 10 client.?.objecter ms_handle_connect 0x7f7cf812c150 2021-05-25T12:01:17.284 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.250+0000 7f7ce57fa700 10 client.?.objecter resend_mon_ops 2021-05-25T12:01:17.284 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.250+0000 7f7ce57fa700 1 -- 172.21.15.102:0/2026322702 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4199020316 0 0) 0x7f7cf8130020 con 0x7f7cf812bca0 2021-05-25T12:01:17.284 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.250+0000 7f7ce57fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:01:17.284 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.250+0000 7f7ce57fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:01:17.284 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.250+0000 7f7ce57fa700 10 monclient(hunting): my global_id is 84945 2021-05-25T12:01:17.285 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.250+0000 7f7ce57fa700 1 -- 172.21.15.102:0/2026322702 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cd8004c10 con 0x7f7cf812bca0 2021-05-25T12:01:17.285 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.250+0000 7f7ce57fa700 1 -- 172.21.15.102:0/2026322702 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 733183203 0 0) 0x7f7cf8130250 con 0x7f7cf8098930 2021-05-25T12:01:17.285 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.250+0000 7f7ce57fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:01:17.285 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.250+0000 7f7ce57fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:01:17.286 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.250+0000 7f7ce57fa700 10 monclient(hunting): my global_id is 14087 2021-05-25T12:01:17.286 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.250+0000 7f7ce57fa700 1 -- 172.21.15.102:0/2026322702 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cf8130020 con 0x7f7cf8098930 2021-05-25T12:01:17.286 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.250+0000 7f7ce57fa700 1 -- 172.21.15.102:0/2026322702 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2354683004 0 0) 0x7f7cf812e150 con 0x7f7cf812c150 2021-05-25T12:01:17.286 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.250+0000 7f7ce57fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:01:17.286 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.250+0000 7f7ce57fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:01:17.287 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.250+0000 7f7ce57fa700 10 monclient(hunting): my global_id is 82567 2021-05-25T12:01:17.287 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.250+0000 7f7ce57fa700 1 -- 172.21.15.102:0/2026322702 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cf8130250 con 0x7f7cf812c150 2021-05-25T12:01:17.287 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.250+0000 7f7ce57fa700 1 -- 172.21.15.102:0/2026322702 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1929132299 0 0) 0x7f7cf8130020 con 0x7f7cf8098930 2021-05-25T12:01:17.287 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.250+0000 7f7ce57fa700 1 -- 172.21.15.102:0/2026322702 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7cf812e150 con 0x7f7cf8098930 2021-05-25T12:01:17.287 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.250+0000 7f7ce57fa700 1 -- 172.21.15.102:0/2026322702 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f7ce80019f0 con 0x7f7cf8098930 2021-05-25T12:01:17.288 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.250+0000 7f7ce57fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:01:17.288 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.250+0000 7f7ce57fa700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:01:17.288 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.250+0000 7f7ce57fa700 10 monclient(hunting): dump: 2021-05-25T12:01:17.288 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:01:17.288 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:17.289 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:17.289 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:17.289 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:01:17.289 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:17.289 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:01:17.290 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:01:17.290 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:01:17.290 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:17.290 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.250+0000 7f7ce57fa700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:01:17.291 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.250+0000 7f7ce57fa700 1 -- 172.21.15.102:0/2026322702 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1817224218 0 0) 0x7f7cd8004c10 con 0x7f7cf812bca0 2021-05-25T12:01:17.291 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.250+0000 7f7ce57fa700 1 -- 172.21.15.102:0/2026322702 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7cf8130020 con 0x7f7cf812bca0 2021-05-25T12:01:17.291 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.250+0000 7f7ce57fa700 1 -- 172.21.15.102:0/2026322702 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f7cec001780 con 0x7f7cf812bca0 2021-05-25T12:01:17.291 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.250+0000 7f7ce57fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:01:17.291 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.250+0000 7f7ce57fa700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2021-05-25T12:01:17.292 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.250+0000 7f7ce57fa700 10 monclient(hunting): dump: 2021-05-25T12:01:17.292 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:01:17.292 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:17.292 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:17.292 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:17.293 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:01:17.293 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:17.293 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:01:17.293 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:01:17.293 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:01:17.294 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:17.294 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.250+0000 7f7ce57fa700 1 -- 172.21.15.102:0/2026322702 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2185295251 0 0) 0x7f7cf812e150 con 0x7f7cf8098930 2021-05-25T12:01:17.294 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.250+0000 7f7ce57fa700 1 -- 172.21.15.102:0/2026322702 >> v1:172.21.15.181:6789/0 conn(0x7f7cf812c150 legacy=0x7f7cf8142d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:17.294 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.250+0000 7f7ce57fa700 1 -- 172.21.15.102:0/2026322702 >> v1:172.21.15.102:6789/0 conn(0x7f7cf812bca0 legacy=0x7f7cf8128a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:17.294 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.250+0000 7f7ce57fa700 10 monclient: _finish_hunting 0 2021-05-25T12:01:17.295 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.250+0000 7f7ce57fa700 1 monclient: found mon.c 2021-05-25T12:01:17.295 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.250+0000 7f7ce57fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:01:17.295 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.250+0000 7f7ce57fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:01:17.295 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.250+0000 7f7ce57fa700 1 -- 172.21.15.102:0/2026322702 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cf809a1d0 con 0x7f7cf8098930 2021-05-25T12:01:17.296 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.250+0000 7f7ce57fa700 10 monclient: _finish_auth 0 2021-05-25T12:01:17.296 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.250+0000 7f7ce57fa700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:01:17.296 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.250+0000 7f7cfe492700 5 monclient: authenticate success, global_id 14087 2021-05-25T12:01:17.296 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.250+0000 7f7cfe492700 10 monclient: _renew_subs 2021-05-25T12:01:17.296 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.250+0000 7f7cfe492700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:01:17.297 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.250+0000 7f7cfe492700 1 -- 172.21.15.102:0/2026322702 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cf812eb50 con 0x7f7cf8098930 2021-05-25T12:01:17.297 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.250+0000 7f7cfe492700 10 client.14087.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:01:17.297 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.250+0000 7f7cfe492700 10 monclient: _renew_subs 2021-05-25T12:01:17.297 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.250+0000 7f7cfe492700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:01:17.297 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.250+0000 7f7cfe492700 1 -- 172.21.15.102:0/2026322702 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cf8130de0 con 0x7f7cf8098930 2021-05-25T12:01:17.298 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.250+0000 7f7cfe492700 1 librados: init done 2021-05-25T12:01:17.298 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.250+0000 7f7ce57fa700 1 -- 172.21.15.102:0/2026322702 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7ce80047e0 con 0x7f7cf8098930 2021-05-25T12:01:17.298 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.250+0000 7f7ce57fa700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:01:17.298 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.250+0000 7f7ce57fa700 1 -- 172.21.15.102:0/2026322702 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f7ce8004af0 con 0x7f7cf8098930 2021-05-25T12:01:17.298 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.250+0000 7f7ce57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:01:17.299 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.250+0000 7f7ce77fe700 4 set_mon_vals no callback set 2021-05-25T12:01:17.299 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.250+0000 7f7ce57fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:01:17.299 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.250+0000 7f7ce57fa700 10 monclient: dump: 2021-05-25T12:01:17.299 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:01:17.300 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:17.300 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:17.300 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:17.300 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:01:17.300 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:17.300 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:01:17.301 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:01:17.301 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:01:17.301 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:17.301 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.251+0000 7f7cfe492700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:01:17.301 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.251+0000 7f7cfe492700 1 -- 172.21.15.102:0/2026322702 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f7cd00065a0 con 0x7f7cd0002840 2021-05-25T12:01:17.302 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.251+0000 7f7ce57fa700 1 -- 172.21.15.102:0/2026322702 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f7ce800de10 con 0x7f7cf8098930 2021-05-25T12:01:17.302 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.251+0000 7f7ce57fa700 1 -- 172.21.15.102:0/2026322702 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(39..39 src has 1..39) v4 ==== 7898+0+0 (unknown 743688199 0 0) 0x7f7ce803e020 con 0x7f7cf8098930 2021-05-25T12:01:17.302 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.251+0000 7f7ce57fa700 10 client.14087.objecter ms_dispatch 0x7f7cf8001550 osd_map(39..39 src has 1..39) v4 2021-05-25T12:01:17.302 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.251+0000 7f7ce57fa700 3 client.14087.objecter handle_osd_map got epochs [39,39] > 0 2021-05-25T12:01:17.303 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.251+0000 7f7ce57fa700 3 client.14087.objecter handle_osd_map decoding full epoch 39 2021-05-25T12:01:17.303 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.251+0000 7f7ce57fa700 20 client.14087.objecter dump_active .. 0 homeless 2021-05-25T12:01:17.303 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.257+0000 7f7cfdc91700 10 monclient: get_auth_request con 0x7f7cd8033450 auth_method 0 2021-05-25T12:01:17.303 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.257+0000 7f7ce57fa700 10 client.14087.objecter ms_handle_connect 0x7f7cd8033450 2021-05-25T12:01:17.303 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.258+0000 7f7ce57fa700 10 client.14087.objecter ms_handle_connect 0x7f7cd0002840 2021-05-25T12:01:17.304 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.258+0000 7f7ce57fa700 10 client.14087.objecter resend_mon_ops 2021-05-25T12:01:17.307 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.259+0000 7f7ce57fa700 1 -- 172.21.15.102:0/2026322702 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1986600243 0 0) 0x7f7cd00065a0 con 0x7f7cd0002840 2021-05-25T12:01:17.307 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.259+0000 7f7ce57fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:01:17.307 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.259+0000 7f7ce57fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:01:17.307 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.259+0000 7f7ce57fa700 10 monclient(hunting): my global_id is 82579 2021-05-25T12:01:17.307 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.259+0000 7f7ce57fa700 1 -- 172.21.15.102:0/2026322702 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cf812e150 con 0x7f7cd0002840 2021-05-25T12:01:17.308 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.259+0000 7f7ce57fa700 1 -- 172.21.15.102:0/2026322702 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2410709386 0 0) 0x7f7cf812e150 con 0x7f7cd0002840 2021-05-25T12:01:17.308 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.259+0000 7f7ce57fa700 1 -- 172.21.15.102:0/2026322702 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7cd00065a0 con 0x7f7cd0002840 2021-05-25T12:01:17.308 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.259+0000 7f7ce57fa700 1 -- 172.21.15.102:0/2026322702 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f7cec001d70 con 0x7f7cd0002840 2021-05-25T12:01:17.308 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.259+0000 7f7ce57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:01:17.308 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.259+0000 7f7ce57fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T12:01:17.309 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.259+0000 7f7ce57fa700 10 monclient: dump: 2021-05-25T12:01:17.309 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:01:17.309 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:17.309 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:17.310 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:17.310 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:01:17.310 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:17.310 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:01:17.310 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:01:17.311 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:01:17.311 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:17.311 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.260+0000 7f7ce57fa700 1 -- 172.21.15.102:0/2026322702 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3820730713 0 0) 0x7f7cd00065a0 con 0x7f7cd0002840 2021-05-25T12:01:17.311 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.260+0000 7f7ce57fa700 1 -- 172.21.15.102:0/2026322702 --> v1:172.21.15.181:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7cd0009e40 con 0x7f7cd0002840 2021-05-25T12:01:17.311 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.260+0000 7f7ce57fa700 1 -- 172.21.15.102:0/2026322702 <== mon.1 v1:172.21.15.181:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7f7cec0067a0 con 0x7f7cd0002840 2021-05-25T12:01:17.312 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.260+0000 7f7ce57fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:01:17.312 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.260+0000 7f7ce57fa700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:01:17.312 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.260+0000 7f7ce57fa700 1 -- 172.21.15.102:0/2026322702 >> v1:172.21.15.181:6789/0 conn(0x7f7cd0002840 legacy=0x7f7cd0004ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:17.312 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.260+0000 7f7ce57fa700 1 -- 172.21.15.102:0/2026322702 >> v1:172.21.15.181:6789/0 conn(0x7f7cd0002840 legacy=0x7f7cd0004ce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:01:17.313 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.267+0000 7f7cfe492700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:01:17.313 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.267+0000 7f7cfe492700 1 -- 172.21.15.102:0/2026322702 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f7cd0007510 con 0x7f7cd00068b0 2021-05-25T12:01:17.313 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.268+0000 7f7ce57fa700 10 client.14087.objecter ms_handle_connect 0x7f7cd00068b0 2021-05-25T12:01:17.313 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.268+0000 7f7ce57fa700 10 client.14087.objecter resend_mon_ops 2021-05-25T12:01:17.313 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.268+0000 7f7ce57fa700 1 -- 172.21.15.102:0/2026322702 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1237736418 0 0) 0x7f7cd0007510 con 0x7f7cd00068b0 2021-05-25T12:01:17.314 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.268+0000 7f7ce57fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:01:17.314 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.268+0000 7f7ce57fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:01:17.314 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.268+0000 7f7ce57fa700 10 monclient(hunting): my global_id is 82591 2021-05-25T12:01:17.314 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.268+0000 7f7ce57fa700 1 -- 172.21.15.102:0/2026322702 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cd00065a0 con 0x7f7cd00068b0 2021-05-25T12:01:17.314 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.268+0000 7f7ce57fa700 1 -- 172.21.15.102:0/2026322702 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1611756170 0 0) 0x7f7cd00065a0 con 0x7f7cd00068b0 2021-05-25T12:01:17.315 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.268+0000 7f7ce57fa700 1 -- 172.21.15.102:0/2026322702 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7cd0007510 con 0x7f7cd00068b0 2021-05-25T12:01:17.315 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.268+0000 7f7ce57fa700 1 -- 172.21.15.102:0/2026322702 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f7cec004ba0 con 0x7f7cd00068b0 2021-05-25T12:01:17.315 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.268+0000 7f7ce57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:01:17.315 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.268+0000 7f7ce57fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T12:01:17.315 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.268+0000 7f7ce57fa700 10 monclient: dump: 2021-05-25T12:01:17.316 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:01:17.316 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:17.316 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:17.316 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:17.316 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:01:17.317 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:17.317 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:01:17.317 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:01:17.317 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:01:17.317 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:17.318 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.268+0000 7f7ce57fa700 1 -- 172.21.15.102:0/2026322702 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1038350094 0 0) 0x7f7cd0007510 con 0x7f7cd00068b0 2021-05-25T12:01:17.318 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.268+0000 7f7ce57fa700 1 -- 172.21.15.102:0/2026322702 --> v1:172.21.15.181:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f7cd0008770 con 0x7f7cd00068b0 2021-05-25T12:01:17.318 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.269+0000 7f7ce57fa700 1 -- 172.21.15.102:0/2026322702 <== mon.1 v1:172.21.15.181:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3941 (unknown 0 0 2951157899) 0x7f7cec006a50 con 0x7f7cd00068b0 2021-05-25T12:01:17.318 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.269+0000 7f7ce57fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:01:17.319 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.269+0000 7f7ce57fa700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:01:17.319 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.269+0000 7f7ce57fa700 1 -- 172.21.15.102:0/2026322702 >> v1:172.21.15.181:6789/0 conn(0x7f7cd00068b0 legacy=0x7f7cd0006d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:17.319 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.269+0000 7f7ce57fa700 1 -- 172.21.15.102:0/2026322702 >> v1:172.21.15.181:6789/0 conn(0x7f7cd00068b0 legacy=0x7f7cd0006d40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:01:17.319 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:01:17.320 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:01:17.320 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:01:17.320 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "peon", 2021-05-25T12:01:17.320 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 20, 2021-05-25T12:01:17.320 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:01:17.321 INFO:teuthology.orchestra.run.smithi102.stdout: 0, 2021-05-25T12:01:17.321 INFO:teuthology.orchestra.run.smithi102.stdout: 1, 2021-05-25T12:01:17.321 INFO:teuthology.orchestra.run.smithi102.stdout: 2 2021-05-25T12:01:17.321 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:01:17.321 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 0, 2021-05-25T12:01:17.322 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:01:17.322 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:01:17.322 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:01:17.322 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:01:17.323 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:01:17.323 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:01:17.323 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:01:17.323 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:01:17.323 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:01:17.324 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:01:17.324 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:01:17.324 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:01:17.324 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:01:17.324 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:01:17.325 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:01:17.325 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:01:17.325 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:01:17.325 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:01:17.325 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:01:17.326 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:01:17.326 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:01:17.326 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:01:17.326 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:01:17.327 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:01:17.327 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:01:17.327 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:01:17.327 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:01:17.327 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:01:17.328 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:01:17.328 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:01:17.328 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:01:17.328 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:01:17.328 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:01:17.329 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:01:17.329 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:01:17.329 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:01:17.329 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:01:17.329 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:01:17.330 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:01:17.330 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:01:17.330 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:01:17.330 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:01:17.330 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:01:17.331 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:01:17.331 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:01:17.331 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:01:17.331 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:01:17.331 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:01:17.332 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:01:17.332 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:01:17.332 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:01:17.332 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:01:17.332 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:01:17.333 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:01:17.333 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:01:17.333 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:01:17.333 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:01:17.334 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:01:17.334 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:01:17.334 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:01:17.334 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:01:17.334 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:01:17.335 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:01:17.335 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:01:17.335 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:01:17.335 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:01:17.335 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:01:17.336 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:01:17.336 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:01:17.336 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:01:17.336 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:01:17.336 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:01:17.337 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:01:17.337 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:01:17.337 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:01:17.337 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:01:17.337 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:01:17.338 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:01:17.338 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:01:17.338 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:01:17.338 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:01:17.338 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:01:17.339 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:01:17.339 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:01:17.339 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:01:17.339 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:01:17.339 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:01:17.340 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:01:17.340 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:01:17.340 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:01:17.340 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:01:17.340 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:01:17.341 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:01:17.341 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:01:17.341 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:01:17.341 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:01:17.341 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:01:17.342 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:01:17.342 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:01:17.342 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:01:17.342 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:01:17.342 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:01:17.343 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:01:17.343 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:01:17.343 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:01:17.343 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:01:17.343 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:01:17.344 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:01:17.344 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:01:17.344 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:01:17.344 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:01:17.344 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:01:17.345 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:01:17.345 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:01:17.345 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:01:17.345 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:01:17.346 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:01:17.346 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:01:17.346 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": [ 2021-05-25T12:01:17.346 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:01:17.346 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:01:17.347 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:01:17.347 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:01:17.347 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:01:17.347 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:01:17.347 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:01:17.348 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:01:17.348 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:01:17.348 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:01:17.348 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 30 2021-05-25T12:01:17.348 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:01:17.349 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:01:17.349 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:01:17.349 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:01:17.349 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:01:17.351 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.269+0000 7f7cfe492700 10 librados: watch_flush enter 2021-05-25T12:01:17.351 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.269+0000 7f7cfe492700 10 librados: watch_flush exit 2021-05-25T12:01:17.352 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.269+0000 7f7cfe492700 20 client.14087.objecter shutdown clearing up homeless session... 2021-05-25T12:01:17.352 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.269+0000 7f7cfe492700 10 client.14087.objecter successfully canceled tick 2021-05-25T12:01:17.352 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.269+0000 7f7cfe492700 1 -- 172.21.15.102:0/2026322702 >> v1:172.21.15.181:6812/29582 conn(0x7f7cd8033450 legacy=0x7f7cd8035910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:17.352 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.269+0000 7f7cfe492700 10 monclient: shutdown 2021-05-25T12:01:17.352 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.269+0000 7f7cfe492700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:01:17.353 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.269+0000 7f7cfe492700 1 -- 172.21.15.102:0/2026322702 >> v1:172.21.15.102:6790/0 conn(0x7f7cf8098930 legacy=0x7f7cf8128360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:17.353 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.269+0000 7f7cfdc91700 1 -- 172.21.15.102:0/2026322702 reap_dead start 2021-05-25T12:01:17.353 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.269+0000 7f7cfdc91700 1 -- 172.21.15.102:0/2026322702 reap_dead start 2021-05-25T12:01:17.353 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.269+0000 7f7cfe492700 1 -- 172.21.15.102:0/2026322702 shutdown_connections 2021-05-25T12:01:17.354 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.269+0000 7f7cfe492700 1 -- 172.21.15.102:0/2026322702 >> 172.21.15.102:0/2026322702 conn(0x7f7cf809e9d0 msgr2=0x7f7cf8129f90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:01:17.354 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.269+0000 7f7cfe492700 1 -- 172.21.15.102:0/2026322702 shutdown_connections 2021-05-25T12:01:17.354 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.269+0000 7f7cfe492700 1 -- 172.21.15.102:0/2026322702 wait complete. 2021-05-25T12:01:17.354 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.269+0000 7f7cfe492700 1 librados: shutdown 2021-05-25T12:01:17.355 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2021-05-25T12:01:17.355 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2021-05-25T12:01:17.471 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.470+0000 7f02cb00a700 10 monclient: get_monmap_and_config 2021-05-25T12:01:17.472 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.470+0000 7f02cb00a700 10 monclient: build_initial_monmap 2021-05-25T12:01:17.472 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.470+0000 7f02cb00a700 10 monclient: monmap: 2021-05-25T12:01:17.472 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:01:17.472 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:17.473 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:01:17.471303+0000 2021-05-25T12:01:17.473 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:01:17.471303+0000 2021-05-25T12:01:17.473 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:01:17.473 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:17.473 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:01:17.474 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:01:17.474 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:01:17.474 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:17.483 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.471+0000 7f02cb00a700 1 Processor -- start 2021-05-25T12:01:17.483 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.471+0000 7f02cb00a700 1 -- start start 2021-05-25T12:01:17.483 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.471+0000 7f02cb00a700 10 monclient: init 2021-05-25T12:01:17.484 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.471+0000 7f02cb00a700 10 monclient: _reopen_session rank -1 2021-05-25T12:01:17.484 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.471+0000 7f02cb00a700 10 monclient: _add_conns ranks=[1,2,0] 2021-05-25T12:01:17.484 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.471+0000 7f02cb00a700 10 monclient(hunting): picked mon.noname-c con 0x7f02c4098d40 addr v1:172.21.15.181:6789/0 2021-05-25T12:01:17.484 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.471+0000 7f02cb00a700 10 monclient(hunting): picked mon.noname-b con 0x7f02c409c250 addr v1:172.21.15.102:6790/0 2021-05-25T12:01:17.485 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.471+0000 7f02cb00a700 10 monclient(hunting): picked mon.noname-a con 0x7f02c4095bf0 addr v1:172.21.15.102:6789/0 2021-05-25T12:01:17.485 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.471+0000 7f02cb00a700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02c409b310 con 0x7f02c4095bf0 2021-05-25T12:01:17.485 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.471+0000 7f02cb00a700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02c409b5b0 con 0x7f02c4098d40 2021-05-25T12:01:17.485 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.471+0000 7f02cb00a700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02c4096340 con 0x7f02c409c250 2021-05-25T12:01:17.486 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.471+0000 7f02cb00a700 10 monclient(hunting): _renew_subs 2021-05-25T12:01:17.486 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.471+0000 7f02cb00a700 10 monclient(hunting): authenticate will time out at 878.068555s 2021-05-25T12:01:17.486 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.471+0000 7f02c95a7700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f02c4095bf0 0x7f02c41344c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:59748/0 (socket says 172.21.15.102:59748) 2021-05-25T12:01:17.486 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.471+0000 7f02bbfff700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f02c409c250 0x7f02c4130d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:54624/0 (socket says 172.21.15.102:54624) 2021-05-25T12:01:17.486 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.471+0000 7f02c95a7700 1 -- 172.21.15.102:0/1950351263 learned_addr learned my addr 172.21.15.102:0/1950351263 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:01:17.487 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.471+0000 7f02bb7fe700 1 -- 172.21.15.102:0/1950351263 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 288174220 0 0) 0x7f02c4096340 con 0x7f02c409c250 2021-05-25T12:01:17.487 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.471+0000 7f02bb7fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:01:17.487 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.471+0000 7f02bb7fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:01:17.487 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.471+0000 7f02bb7fe700 10 monclient(hunting): my global_id is 14231 2021-05-25T12:01:17.488 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.471+0000 7f02bb7fe700 1 -- 172.21.15.102:0/1950351263 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02b0003270 con 0x7f02c409c250 2021-05-25T12:01:17.488 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.471+0000 7f02bb7fe700 1 -- 172.21.15.102:0/1950351263 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1415888990 0 0) 0x7f02c409b310 con 0x7f02c4095bf0 2021-05-25T12:01:17.488 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.471+0000 7f02bb7fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:01:17.488 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.471+0000 7f02bb7fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:01:17.488 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.471+0000 7f02bb7fe700 10 monclient(hunting): my global_id is 85089 2021-05-25T12:01:17.489 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.471+0000 7f02bb7fe700 1 -- 172.21.15.102:0/1950351263 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02c4096340 con 0x7f02c4095bf0 2021-05-25T12:01:17.489 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.472+0000 7f02bb7fe700 1 -- 172.21.15.102:0/1950351263 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 705917437 0 0) 0x7f02b0003270 con 0x7f02c409c250 2021-05-25T12:01:17.489 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.472+0000 7f02bb7fe700 1 -- 172.21.15.102:0/1950351263 >> v1:172.21.15.181:6789/0 conn(0x7f02c4098d40 legacy=0x7f02c412bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:17.489 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.472+0000 7f02bb7fe700 1 -- 172.21.15.102:0/1950351263 >> v1:172.21.15.102:6789/0 conn(0x7f02c4095bf0 legacy=0x7f02c41344c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:17.490 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.472+0000 7f02bb7fe700 10 monclient: _finish_hunting 0 2021-05-25T12:01:17.490 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.472+0000 7f02bb7fe700 1 monclient: found mon.noname-b 2021-05-25T12:01:17.490 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.472+0000 7f02bb7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:01:17.490 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.472+0000 7f02bb7fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T12:01:17.490 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.472+0000 7f02bb7fe700 1 -- 172.21.15.102:0/1950351263 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02c4099120 con 0x7f02c409c250 2021-05-25T12:01:17.491 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.472+0000 7f02bb7fe700 10 monclient: _finish_auth 0 2021-05-25T12:01:17.491 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.472+0000 7f02bb7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:01:17.491 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.472+0000 7f02bb7fe700 1 -- 172.21.15.102:0/1950351263 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f02ac001610 con 0x7f02c409c250 2021-05-25T12:01:17.491 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.472+0000 7f02bb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:01:17.491 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.472+0000 7f02bb7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:01:17.492 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.472+0000 7f02bb7fe700 10 monclient: dump: 2021-05-25T12:01:17.492 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:01:17.492 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:17.492 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:17.493 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:17.493 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:01:17.493 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:17.493 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:01:17.493 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:01:17.494 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:01:17.494 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:17.494 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.472+0000 7f02bb7fe700 1 -- 172.21.15.102:0/1950351263 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f02ac001880 con 0x7f02c409c250 2021-05-25T12:01:17.494 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.472+0000 7f02bb7fe700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:01:17.495 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.472+0000 7f02bb7fe700 1 -- 172.21.15.102:0/1950351263 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f02ac001c10 con 0x7f02c409c250 2021-05-25T12:01:17.495 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.472+0000 7f02bb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:01:17.495 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.472+0000 7f02bb7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:01:17.495 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.472+0000 7f02bb7fe700 10 monclient: dump: 2021-05-25T12:01:17.495 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:01:17.496 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:17.496 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:17.496 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:17.496 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:01:17.496 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:17.497 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:01:17.497 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:01:17.497 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:01:17.497 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:17.498 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.472+0000 7f02cb00a700 5 monclient: authenticate success, global_id 14231 2021-05-25T12:01:17.498 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.472+0000 7f02cb00a700 10 monclient: get_monmap_and_config success 2021-05-25T12:01:17.498 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.472+0000 7f02cb00a700 4 set_mon_vals no callback set 2021-05-25T12:01:17.498 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.472+0000 7f02cb00a700 10 monclient: shutdown 2021-05-25T12:01:17.498 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.472+0000 7f02cb00a700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:01:17.499 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.472+0000 7f02cb00a700 1 -- 172.21.15.102:0/1950351263 >> v1:172.21.15.102:6790/0 conn(0x7f02c409c250 legacy=0x7f02c4130d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:17.499 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.472+0000 7f02cb00a700 1 -- 172.21.15.102:0/1950351263 shutdown_connections 2021-05-25T12:01:17.499 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.472+0000 7f02cb00a700 1 -- 172.21.15.102:0/1950351263 >> 172.21.15.102:0/1950351263 conn(0x7f02c4069630 msgr2=0x7f02c4069a40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:01:17.499 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.472+0000 7f02cb00a700 1 -- 172.21.15.102:0/1950351263 shutdown_connections 2021-05-25T12:01:17.499 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.472+0000 7f02cb00a700 1 -- 172.21.15.102:0/1950351263 wait complete. 2021-05-25T12:01:17.500 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.473+0000 7f02cb00a700 10 monclient: build_initial_monmap 2021-05-25T12:01:17.500 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.473+0000 7f02cb00a700 10 monclient: monmap: 2021-05-25T12:01:17.500 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:01:17.500 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:01:17.500 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:01:17.501 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:01:17.501 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:01:17.501 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:01:17.501 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:01:17.502 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:01:17.502 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:01:17.502 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:17.502 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.473+0000 7f02cb00a700 1 librados: starting msgr at 2021-05-25T12:01:17.502 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.473+0000 7f02cb00a700 1 librados: starting objecter 2021-05-25T12:01:17.503 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.473+0000 7f02cb00a700 1 Processor -- start 2021-05-25T12:01:17.503 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.473+0000 7f02cb00a700 1 -- start start 2021-05-25T12:01:17.503 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.473+0000 7f02cb00a700 1 librados: setting wanted keys 2021-05-25T12:01:17.503 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.473+0000 7f02cb00a700 1 librados: calling monclient init 2021-05-25T12:01:17.503 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.473+0000 7f02cb00a700 10 monclient: init 2021-05-25T12:01:17.504 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.474+0000 7f02cb00a700 10 monclient: _reopen_session rank -1 2021-05-25T12:01:17.504 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.474+0000 7f02cb00a700 10 monclient: _add_conns ranks=[1,0,2] 2021-05-25T12:01:17.504 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.474+0000 7f02cb00a700 10 monclient(hunting): picked mon.noname-b con 0x7f02c4098d40 addr v1:172.21.15.181:6789/0 2021-05-25T12:01:17.504 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.474+0000 7f02cb00a700 10 monclient(hunting): picked mon.noname-a con 0x7f02c409c250 addr v1:172.21.15.102:6789/0 2021-05-25T12:01:17.505 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.474+0000 7f02cb00a700 10 monclient(hunting): picked mon.noname-c con 0x7f02c4141240 addr v1:172.21.15.102:6790/0 2021-05-25T12:01:17.505 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.474+0000 7f02cb00a700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02c41250a0 con 0x7f02c409c250 2021-05-25T12:01:17.505 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.474+0000 7f02cb00a700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02c4128140 con 0x7f02c4098d40 2021-05-25T12:01:17.505 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.474+0000 7f02cb00a700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02c409aa00 con 0x7f02c4141240 2021-05-25T12:01:17.505 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.474+0000 7f02cb00a700 10 monclient(hunting): _renew_subs 2021-05-25T12:01:17.506 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.474+0000 7f02cb00a700 10 monclient(hunting): authenticate will time out at 878.071242s 2021-05-25T12:01:17.506 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.474+0000 7f02bbfff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f02c409c250 0x7f02c41392e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:59750/0 (socket says 172.21.15.102:59750) 2021-05-25T12:01:17.506 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.474+0000 7f02bbfff700 1 -- 172.21.15.102:0/2805280689 learned_addr learned my addr 172.21.15.102:0/2805280689 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:01:17.506 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.474+0000 7f02c95a7700 1 --1- 172.21.15.102:0/2805280689 >> v1:172.21.15.102:6790/0 conn(0x7f02c4141240 0x7f02c4143630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:54630/0 (socket says 172.21.15.102:54630) 2021-05-25T12:01:17.506 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.474+0000 7f02b8ff9700 10 client.?.objecter ms_handle_connect 0x7f02c409c250 2021-05-25T12:01:17.507 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.474+0000 7f02b8ff9700 10 client.?.objecter resend_mon_ops 2021-05-25T12:01:17.507 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.474+0000 7f02b8ff9700 10 client.?.objecter ms_handle_connect 0x7f02c4141240 2021-05-25T12:01:17.507 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.474+0000 7f02b8ff9700 10 client.?.objecter resend_mon_ops 2021-05-25T12:01:17.507 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.474+0000 7f02b8ff9700 1 -- 172.21.15.102:0/2805280689 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 545222786 0 0) 0x7f02c41250a0 con 0x7f02c409c250 2021-05-25T12:01:17.507 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.474+0000 7f02b8ff9700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:01:17.508 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.474+0000 7f02b8ff9700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:01:17.508 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.474+0000 7f02b8ff9700 10 monclient(hunting): my global_id is 85092 2021-05-25T12:01:17.508 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.474+0000 7f02b8ff9700 1 -- 172.21.15.102:0/2805280689 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02a4004c10 con 0x7f02c409c250 2021-05-25T12:01:17.508 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.474+0000 7f02b8ff9700 10 client.?.objecter ms_handle_connect 0x7f02c4098d40 2021-05-25T12:01:17.509 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.474+0000 7f02b8ff9700 10 client.?.objecter resend_mon_ops 2021-05-25T12:01:17.509 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.474+0000 7f02b8ff9700 1 -- 172.21.15.102:0/2805280689 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 13625449 0 0) 0x7f02c409aa00 con 0x7f02c4141240 2021-05-25T12:01:17.509 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.474+0000 7f02b8ff9700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:01:17.509 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.474+0000 7f02b8ff9700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:01:17.509 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.474+0000 7f02b8ff9700 10 monclient(hunting): my global_id is 14237 2021-05-25T12:01:17.510 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.474+0000 7f02b8ff9700 1 -- 172.21.15.102:0/2805280689 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02c41250a0 con 0x7f02c4141240 2021-05-25T12:01:17.510 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.474+0000 7f02b8ff9700 1 -- 172.21.15.102:0/2805280689 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3824838689 0 0) 0x7f02a4004c10 con 0x7f02c409c250 2021-05-25T12:01:17.510 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.474+0000 7f02b8ff9700 1 -- 172.21.15.102:0/2805280689 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f02c409aa00 con 0x7f02c409c250 2021-05-25T12:01:17.510 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.474+0000 7f02b8ff9700 1 -- 172.21.15.102:0/2805280689 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f02ac001690 con 0x7f02c409c250 2021-05-25T12:01:17.510 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.474+0000 7f02b8ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:01:17.511 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.474+0000 7f02b8ff9700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:01:17.511 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.474+0000 7f02b8ff9700 10 monclient(hunting): dump: 2021-05-25T12:01:17.511 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:01:17.511 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:17.512 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:17.512 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:17.512 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:01:17.512 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:17.512 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:01:17.513 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:01:17.513 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:01:17.513 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:17.513 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.474+0000 7f02b8ff9700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:01:17.513 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.474+0000 7f02b8ff9700 1 -- 172.21.15.102:0/2805280689 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2555340018 0 0) 0x7f02c41250a0 con 0x7f02c4141240 2021-05-25T12:01:17.514 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.474+0000 7f02b8ff9700 1 -- 172.21.15.102:0/2805280689 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f02a4004c10 con 0x7f02c4141240 2021-05-25T12:01:17.514 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.474+0000 7f02b8ff9700 1 -- 172.21.15.102:0/2805280689 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f02c0001b40 con 0x7f02c4141240 2021-05-25T12:01:17.514 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.474+0000 7f02b8ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:01:17.514 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.474+0000 7f02b8ff9700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2021-05-25T12:01:17.515 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.474+0000 7f02b8ff9700 10 monclient(hunting): dump: 2021-05-25T12:01:17.515 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:01:17.515 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:17.515 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:17.515 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:17.516 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:01:17.516 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:17.516 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:01:17.516 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:01:17.517 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:01:17.517 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:17.517 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.475+0000 7f02b8ff9700 1 -- 172.21.15.102:0/2805280689 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1823444750 0 0) 0x7f02c409aa00 con 0x7f02c409c250 2021-05-25T12:01:17.517 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.475+0000 7f02b8ff9700 1 -- 172.21.15.102:0/2805280689 >> v1:172.21.15.102:6790/0 conn(0x7f02c4141240 legacy=0x7f02c4143630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:17.517 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.475+0000 7f02b8ff9700 1 -- 172.21.15.102:0/2805280689 >> v1:172.21.15.181:6789/0 conn(0x7f02c4098d40 legacy=0x7f02c4127a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:17.518 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.475+0000 7f02b8ff9700 10 monclient: _finish_hunting 0 2021-05-25T12:01:17.518 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.475+0000 7f02b8ff9700 1 monclient: found mon.a 2021-05-25T12:01:17.518 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.475+0000 7f02b8ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:01:17.518 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.475+0000 7f02b8ff9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:01:17.518 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.475+0000 7f02b8ff9700 1 -- 172.21.15.102:0/2805280689 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02c41399f0 con 0x7f02c409c250 2021-05-25T12:01:17.519 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.475+0000 7f02b8ff9700 10 monclient: _finish_auth 0 2021-05-25T12:01:17.519 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.475+0000 7f02b8ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:01:17.519 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.475+0000 7f02cb00a700 5 monclient: authenticate success, global_id 85092 2021-05-25T12:01:17.519 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.475+0000 7f02cb00a700 10 monclient: _renew_subs 2021-05-25T12:01:17.520 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.475+0000 7f02cb00a700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:01:17.520 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.475+0000 7f02cb00a700 1 -- 172.21.15.102:0/2805280689 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02c41252d0 con 0x7f02c409c250 2021-05-25T12:01:17.520 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.475+0000 7f02cb00a700 10 client.85092.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:01:17.520 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.475+0000 7f02cb00a700 10 monclient: _renew_subs 2021-05-25T12:01:17.520 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.475+0000 7f02cb00a700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:01:17.521 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.475+0000 7f02cb00a700 1 -- 172.21.15.102:0/2805280689 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f02c4144d50 con 0x7f02c409c250 2021-05-25T12:01:17.521 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.475+0000 7f02cb00a700 1 librados: init done 2021-05-25T12:01:17.521 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.475+0000 7f02b8ff9700 1 -- 172.21.15.102:0/2805280689 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f02ac004a00 con 0x7f02c409c250 2021-05-25T12:01:17.521 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.475+0000 7f02b8ff9700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:01:17.521 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.475+0000 7f02b8ff9700 1 -- 172.21.15.102:0/2805280689 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f02ac004d90 con 0x7f02c409c250 2021-05-25T12:01:17.522 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.475+0000 7f02b8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:01:17.522 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.475+0000 7f02b8ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:01:17.522 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.475+0000 7f02baffd700 4 set_mon_vals no callback set 2021-05-25T12:01:17.522 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.475+0000 7f02b8ff9700 10 monclient: dump: 2021-05-25T12:01:17.523 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:01:17.523 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:17.523 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:17.523 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:17.523 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:01:17.524 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:17.524 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:01:17.524 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:01:17.524 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:01:17.524 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:17.525 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.475+0000 7f02b8ff9700 1 -- 172.21.15.102:0/2805280689 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f02ac004f70 con 0x7f02c409c250 2021-05-25T12:01:17.525 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.475+0000 7f02cb00a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:01:17.525 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.475+0000 7f02cb00a700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:01:17.525 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.475+0000 7f02cb00a700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:01:17.525 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.475+0000 7f02cb00a700 1 -- 172.21.15.102:0/2805280689 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f029c004db0 con 0x7f02c409c250 2021-05-25T12:01:17.526 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.476+0000 7f02b8ff9700 1 -- 172.21.15.102:0/2805280689 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(39..39 src has 1..39) v4 ==== 7898+0+0 (unknown 743688199 0 0) 0x7f02ac03e200 con 0x7f02c409c250 2021-05-25T12:01:17.526 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.476+0000 7f02b8ff9700 10 client.85092.objecter ms_dispatch 0x7f02c4001550 osd_map(39..39 src has 1..39) v4 2021-05-25T12:01:17.526 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.476+0000 7f02b8ff9700 3 client.85092.objecter handle_osd_map got epochs [39,39] > 0 2021-05-25T12:01:17.526 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.476+0000 7f02b8ff9700 3 client.85092.objecter handle_osd_map decoding full epoch 39 2021-05-25T12:01:17.527 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.476+0000 7f02b8ff9700 20 client.85092.objecter dump_active .. 0 homeless 2021-05-25T12:01:17.527 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.476+0000 7f02c8da6700 10 monclient: get_auth_request con 0x7f02a4033200 auth_method 0 2021-05-25T12:01:17.527 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.477+0000 7f02b8ff9700 10 client.85092.objecter ms_handle_connect 0x7f02a4033200 2021-05-25T12:01:17.527 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.480+0000 7f02b8ff9700 1 -- 172.21.15.102:0/2805280689 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7f02ac013100 con 0x7f02c409c250 2021-05-25T12:01:17.527 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.480+0000 7f02b8ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:01:17.528 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.480+0000 7f02b8ff9700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:01:17.564 INFO:tasks.workunit.client.0.smithi181.stdout: api_service: [==========] Running 4 tests from 1 test suite. 2021-05-25T12:01:17.564 INFO:tasks.workunit.client.0.smithi181.stdout: api_service: [----------] Global test environment set-up. 2021-05-25T12:01:17.564 INFO:tasks.workunit.client.0.smithi181.stdout: api_service: [----------] 4 tests from LibRadosService 2021-05-25T12:01:17.564 INFO:tasks.workunit.client.0.smithi181.stdout: api_service: [ RUN ] LibRadosService.RegisterEarly 2021-05-25T12:01:17.565 INFO:tasks.workunit.client.0.smithi181.stdout: api_service: [ OK ] LibRadosService.RegisterEarly (5046 ms) 2021-05-25T12:01:17.565 INFO:tasks.workunit.client.0.smithi181.stdout: api_service: [ RUN ] LibRadosService.RegisterLate 2021-05-25T12:01:17.565 INFO:tasks.workunit.client.0.smithi181.stdout: api_service: [ OK ] LibRadosService.RegisterLate (30 ms) 2021-05-25T12:01:17.565 INFO:tasks.workunit.client.0.smithi181.stdout: api_service: [ RUN ] LibRadosService.StatusFormat 2021-05-25T12:01:17.566 INFO:tasks.workunit.client.0.smithi181.stdout: api_service: cluster: 2021-05-25T12:01:17.566 INFO:tasks.workunit.client.0.smithi181.stdout: api_service: id: 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:17.566 INFO:tasks.workunit.client.0.smithi181.stdout: api_service: health: HEALTH_WARN 2021-05-25T12:01:17.566 INFO:tasks.workunit.client.0.smithi181.stdout: api_service: 1/3 mons down, quorum b,c 2021-05-25T12:01:17.567 INFO:tasks.workunit.client.0.smithi181.stdout: api_service: 2021-05-25T12:01:17.567 INFO:tasks.workunit.client.0.smithi181.stdout: api_service: services: 2021-05-25T12:01:17.567 INFO:tasks.workunit.client.0.smithi181.stdout: api_service: mon: 3 daemons, quorum b,c (age 0.22044s), out of quorum: a 2021-05-25T12:01:17.567 INFO:tasks.workunit.client.0.smithi181.stdout: api_service: mgr: x(active, since 69s) 2021-05-25T12:01:17.567 INFO:tasks.workunit.client.0.smithi181.stdout: api_service: osd: 6 osds: 6 up (since 65s), 6 in (since 72s) 2021-05-25T12:01:17.568 INFO:tasks.workunit.client.0.smithi181.stdout: api_service: laundry: 2 daemons active (1 hosts) 2021-05-25T12:01:17.568 INFO:tasks.workunit.client.0.smithi181.stdout: api_service: 2021-05-25T12:01:17.568 INFO:tasks.workunit.client.0.smithi181.stdout: api_service: data: 2021-05-25T12:01:17.568 INFO:tasks.workunit.client.0.smithi181.stdout: api_service: pools: 17 pools, 489 pgs 2021-05-25T12:01:17.569 INFO:tasks.workunit.client.0.smithi181.stdout: api_service: objects: 0 objects, 0 B 2021-05-25T12:01:17.569 INFO:tasks.workunit.client.0.smithi181.stdout: api_service: usage: 29 MiB used, 540 GiB / 540 GiB avail 2021-05-25T12:01:17.569 INFO:tasks.workunit.client.0.smithi181.stdout: api_service: pgs: 98.160% pgs not active 2021-05-25T12:01:17.569 INFO:tasks.workunit.client.0.smithi181.stdout: api_service: 480 creating+peering 2021-05-25T12:01:17.569 INFO:tasks.workunit.client.0.smithi181.stdout: api_service: 9 active+clean 2021-05-25T12:01:17.570 INFO:tasks.workunit.client.0.smithi181.stdout: api_service: 2021-05-25T12:01:17.570 INFO:tasks.workunit.client.0.smithi181.stdout: api_service: 2021-05-25T12:01:17.570 INFO:tasks.workunit.client.0.smithi181.stdout: api_service: cluster: 2021-05-25T12:01:17.570 INFO:tasks.workunit.client.0.smithi181.stdout: api_service: id: 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:17.571 INFO:tasks.workunit.client.0.smithi181.stdout: api_service: health: HEALTH_WARN 2021-05-25T12:01:17.571 INFO:tasks.workunit.client.0.smithi181.stdout: api_service: 1/3 mons down, quorum b,c 2021-05-25T12:01:17.571 INFO:tasks.workunit.client.0.smithi181.stdout: api_service: 2021-05-25T12:01:17.571 INFO:tasks.workunit.client.0.smithi181.stdout: api_service: services: 2021-05-25T12:01:17.571 INFO:tasks.workunit.client.0.smithi181.stdout: api_service: mon: 3 daemons, quorum b,c (age 2s), out of quorum: a 2021-05-25T12:01:17.572 INFO:tasks.workunit.client.0.smithi181.stdout: api_service: mgr: x(active, since 71s) 2021-05-25T12:01:17.572 INFO:tasks.workunit.client.0.smithi181.stdout: api_service: osd: 6 osds: 6 up (since 67s), 6 in (since 74s) 2021-05-25T12:01:17.572 INFO:tasks.workunit.client.0.smithi181.stdout: api_service: foo: 16 portals active (1 hosts, 3 zones) 2021-05-25T12:01:17.572 INFO:tasks.workunit.client.0.smithi181.stdout: api_service: laundry: 1 daemon active (1 hosts) 2021-05-25T12:01:17.573 INFO:tasks.workunit.client.0.smithi181.stdout: api_service: 2021-05-25T12:01:17.573 INFO:tasks.workunit.client.0.smithi181.stdout: api_service: data: 2021-05-25T12:01:17.573 INFO:tasks.workunit.client.0.smithi181.stdout: api_service: pools: 24 pools, 713 pgs 2021-05-25T12:01:17.573 INFO:tasks.workunit.client.0.smithi181.stdout: api_service: objects: 0 objects, 0 B 2021-05-25T12:01:17.573 INFO:tasks.workunit.client.0.smithi181.stdout: api_service: usage: 29 MiB used, 540 GiB / 540 GiB avail 2021-05-25T12:01:17.574 INFO:tasks.workunit.client.0.smithi181.stdout: api_service: pgs: 31.417% pgs unknown 2021-05-25T12:01:17.574 INFO:tasks.workunit.client.0.smithi181.stdout: api_service: 67.321% pgs not active 2021-05-25T12:01:17.574 INFO:tasks.workunit.client.0.smithi181.stdout: api_service: 480 creating+peering 2021-05-25T12:01:17.574 INFO:tasks.workunit.client.0.smithi181.stdout: api_service: 224 unknown 2021-05-25T12:01:17.574 INFO:tasks.workunit.client.0.smithi181.stdout: api_service: 9 active+clean 2021-05-25T12:01:17.575 INFO:tasks.workunit.client.0.smithi181.stdout: api_service: 2021-05-25T12:01:17.575 INFO:tasks.workunit.client.0.smithi181.stdout: api_service: 2021-05-25T12:01:17.575 INFO:tasks.workunit.client.0.smithi181.stdout: api_service: [ OK ] LibRadosService.StatusFormat (12575 ms) 2021-05-25T12:01:17.575 INFO:tasks.workunit.client.0.smithi181.stdout: api_service: [ RUN ] LibRadosService.Status 2021-05-25T12:01:17.575 INFO:tasks.workunit.client.0.smithi181.stdout: api_service: [ OK ] LibRadosService.Status (20054 ms) 2021-05-25T12:01:17.576 INFO:tasks.workunit.client.0.smithi181.stdout: api_service: [----------] 4 tests from LibRadosService (37706 ms total) 2021-05-25T12:01:17.576 INFO:tasks.workunit.client.0.smithi181.stdout: api_service: 2021-05-25T12:01:17.576 INFO:tasks.workunit.client.0.smithi181.stdout: api_service: [----------] Global test environment tear-down 2021-05-25T12:01:17.576 INFO:tasks.workunit.client.0.smithi181.stdout: api_service: [==========] 4 tests from 1 test suite ran. (37706 ms total) 2021-05-25T12:01:17.576 INFO:tasks.workunit.client.0.smithi181.stdout: api_service: [ PASSED ] 4 tests. 2021-05-25T12:01:17.715 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.714+0000 7f02cb00a700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2021-05-25T12:01:17.715 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.714+0000 7f02cb00a700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2021-05-25T12:01:17.716 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.714+0000 7f02cb00a700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:01:17.716 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.714+0000 7f02cb00a700 1 -- 172.21.15.102:0/2805280689 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f029c005870 con 0x7f02c409c250 2021-05-25T12:01:17.716 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.715+0000 7f02b8ff9700 1 -- 172.21.15.102:0/2805280689 <== mon.0 v1:172.21.15.102:6789/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7f02ac012cf0 con 0x7f02c409c250 2021-05-25T12:01:17.717 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:01:17.715+0000 7fd738a30700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:01:17.717 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:01:17.715+0000 7fd738a30700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=22,config=2,health=10,kv=6,logm=60} crc {auth=1675370530,config=3140514009,health=585578440,kv=1184736763,logm=187085861}) 2021-05-25T12:01:17.718 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.715+0000 7f02b8ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2021-05-25T12:01:17.718 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.715+0000 7f02b8ff9700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:01:17.719 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:01:17.715+0000 7fd738a30700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=24,config=2,health=10,kv=6,logm=58} crc {auth=3075985050,config=3140514009,health=2548172754,kv=1184736763,logm=971146845}) 2021-05-25T12:01:17.719 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:01:17.715+0000 7fd738a30700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:01:17.719 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:01:17.715+0000 7fd738a30700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=22,config=2,health=10,kv=6,logm=60} crc {auth=1675370530,config=3140514009,health=585578440,kv=1184736763,logm=187085861}) 2021-05-25T12:01:17.719 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:01:17.715+0000 7fd738a30700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=22,config=2,health=12,kv=6,logm=58} crc {auth=1737243312,config=3140514009,health=942569099,kv=1184736763,logm=2382775972}) 2021-05-25T12:01:17.721 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.717+0000 7f02cb00a700 10 librados: watch_flush enter 2021-05-25T12:01:17.721 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.717+0000 7f02cb00a700 10 librados: watch_flush exit 2021-05-25T12:01:17.721 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.717+0000 7f02cb00a700 20 client.85092.objecter shutdown clearing up homeless session... 2021-05-25T12:01:17.721 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.717+0000 7f02cb00a700 10 client.85092.objecter successfully canceled tick 2021-05-25T12:01:17.722 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.717+0000 7f02cb00a700 1 -- 172.21.15.102:0/2805280689 >> v1:172.21.15.181:6812/29582 conn(0x7f02a4033200 legacy=0x7f02a40356c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:17.722 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.717+0000 7f02cb00a700 10 monclient: shutdown 2021-05-25T12:01:17.722 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.717+0000 7f02cb00a700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:01:17.722 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.717+0000 7f02cb00a700 1 -- 172.21.15.102:0/2805280689 >> v1:172.21.15.102:6789/0 conn(0x7f02c409c250 legacy=0x7f02c41392e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:17.723 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.717+0000 7f02cb00a700 1 -- 172.21.15.102:0/2805280689 shutdown_connections 2021-05-25T12:01:17.723 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.717+0000 7f02cb00a700 1 -- 172.21.15.102:0/2805280689 >> 172.21.15.102:0/2805280689 conn(0x7f02c4069630 msgr2=0x7f02c4128be0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:01:17.723 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.717+0000 7f02cb00a700 1 -- 172.21.15.102:0/2805280689 shutdown_connections 2021-05-25T12:01:17.723 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.717+0000 7f02cb00a700 1 -- 172.21.15.102:0/2805280689 wait complete. 2021-05-25T12:01:17.723 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:17.717+0000 7f02cb00a700 1 librados: shutdown 2021-05-25T12:01:17.724 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:01:17.717+0000 7fd738a30700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:01:17.725 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:01:17.717+0000 7fd738a30700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=47,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=4,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=9,osdmap=9} crc {logm=243401091,mdsmap=1531151339,mgr=2996579194,mgr_command_descs=3824783157,mgr_metadata=1267308364,mgrstat=2522199143,mon_config_key=4118394475,monmap=3809614708,osd_metadata=725478305,osd_pg_creating=1682852622,osd_snap=1531680909,osdmap=2539694350}) 2021-05-25T12:01:17.725 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:01:17.717+0000 7fd738a30700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=49,mdsmap=2,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=4,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=9,osdmap=8} crc {logm=4275202934,mdsmap=3585761405,mgr=1328532805,mgr_command_descs=3824783157,mgr_metadata=1267308364,mgrstat=1173851305,mon_config_key=4118394475,monmap=3809614708,osd_metadata=927457006,osd_pg_creating=1682852622,osd_snap=1531680909,osdmap=974458103}) 2021-05-25T12:01:17.725 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:01:17.717+0000 7fd738a30700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:01:17.725 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:01:17.717+0000 7fd738a30700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=47,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=4,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=9,osdmap=9} crc {logm=243401091,mdsmap=1531151339,mgr=2996579194,mgr_command_descs=3824783157,mgr_metadata=1267308364,mgrstat=2522199143,mon_config_key=4118394475,monmap=3809614708,osd_metadata=725478305,osd_pg_creating=1682852622,osd_snap=1531680909,osdmap=2539694350}) 2021-05-25T12:01:17.726 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:01:17.717+0000 7fd738a30700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=48,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=4,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=9,osdmap=6} crc {logm=4054755406,mdsmap=1531151339,mgr=1723869108,mgr_command_descs=3824783157,mgr_metadata=1267308364,mgrstat=2867525108,mon_config_key=4118394475,monmap=3809614708,osd_metadata=725478305,osd_pg_creating=1682852622,osd_snap=1531680909,osdmap=3984048057}) 2021-05-25T12:01:17.726 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:01:17.720+0000 7fd738a30700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:01:17.726 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:01:17.720+0000 7fd738a30700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=66} crc {osdmap=245425395}) 2021-05-25T12:01:17.726 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:01:17.720+0000 7fd738a30700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=68} crc {osdmap=3091022613}) 2021-05-25T12:01:17.727 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:01:17.720+0000 7fd738a30700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:01:17.727 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:01:17.720+0000 7fd738a30700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=66} crc {osdmap=245425395}) 2021-05-25T12:01:17.727 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:01:17.720+0000 7fd738a30700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=68} crc {osdmap=324837357}) 2021-05-25T12:01:17.731 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2021-05-25T12:01:18.732 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2021-05-25T12:01:18.733 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2021-05-25T12:01:18.847 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.846+0000 7f604546c700 10 monclient: get_monmap_and_config 2021-05-25T12:01:18.848 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.846+0000 7f604546c700 10 monclient: build_initial_monmap 2021-05-25T12:01:18.848 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.846+0000 7f604546c700 10 monclient: monmap: 2021-05-25T12:01:18.848 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:01:18.848 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:18.849 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:01:18.847177+0000 2021-05-25T12:01:18.849 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:01:18.847177+0000 2021-05-25T12:01:18.849 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:01:18.849 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:18.849 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:01:18.850 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:01:18.850 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:01:18.850 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:18.861 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.846+0000 7f604546c700 1 Processor -- start 2021-05-25T12:01:18.862 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.847+0000 7f604546c700 1 -- start start 2021-05-25T12:01:18.862 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.847+0000 7f604546c700 10 monclient: init 2021-05-25T12:01:18.862 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.847+0000 7f604546c700 10 monclient: _reopen_session rank -1 2021-05-25T12:01:18.862 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.847+0000 7f604546c700 10 monclient: _add_conns ranks=[0,2,1] 2021-05-25T12:01:18.863 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.847+0000 7f604546c700 10 monclient(hunting): picked mon.noname-a con 0x7f6040098930 addr v1:172.21.15.102:6789/0 2021-05-25T12:01:18.863 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.847+0000 7f604546c700 10 monclient(hunting): picked mon.noname-b con 0x7f604012bca0 addr v1:172.21.15.102:6790/0 2021-05-25T12:01:18.863 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.847+0000 7f604546c700 10 monclient(hunting): picked mon.noname-c con 0x7f604012c150 addr v1:172.21.15.181:6789/0 2021-05-25T12:01:18.863 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.847+0000 7f604546c700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6040099b20 con 0x7f6040098930 2021-05-25T12:01:18.863 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.847+0000 7f604546c700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f604009a3a0 con 0x7f604012c150 2021-05-25T12:01:18.864 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.847+0000 7f604546c700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f604009ac60 con 0x7f604012bca0 2021-05-25T12:01:18.864 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.847+0000 7f604546c700 10 monclient(hunting): _renew_subs 2021-05-25T12:01:18.864 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.847+0000 7f604546c700 10 monclient(hunting): authenticate will time out at 879.444684s 2021-05-25T12:01:18.864 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.847+0000 7f603f7fe700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f604012bca0 0x7f60401308c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:54642/0 (socket says 172.21.15.102:54642) 2021-05-25T12:01:18.865 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.847+0000 7f603f7fe700 1 -- 172.21.15.102:0/3616148533 learned_addr learned my addr 172.21.15.102:0/3616148533 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:01:18.865 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.848+0000 7f603effd700 1 -- 172.21.15.102:0/3616148533 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 425020804 0 0) 0x7f604009ac60 con 0x7f604012bca0 2021-05-25T12:01:18.865 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.848+0000 7f603effd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:01:18.865 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.848+0000 7f603effd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:01:18.865 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.848+0000 7f603effd700 10 monclient(hunting): my global_id is 15575 2021-05-25T12:01:18.866 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.848+0000 7f603effd700 1 -- 172.21.15.102:0/3616148533 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6024003270 con 0x7f604012bca0 2021-05-25T12:01:18.866 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.848+0000 7f603effd700 1 -- 172.21.15.102:0/3616148533 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2456207819 0 0) 0x7f6040099b20 con 0x7f6040098930 2021-05-25T12:01:18.866 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.848+0000 7f603effd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:01:18.866 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.848+0000 7f603effd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:01:18.866 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.848+0000 7f603effd700 10 monclient(hunting): my global_id is 86439 2021-05-25T12:01:18.867 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.848+0000 7f603effd700 1 -- 172.21.15.102:0/3616148533 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f604009ac60 con 0x7f6040098930 2021-05-25T12:01:18.867 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.848+0000 7f603effd700 1 -- 172.21.15.102:0/3616148533 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3441074995 0 0) 0x7f6024003270 con 0x7f604012bca0 2021-05-25T12:01:18.867 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.848+0000 7f603effd700 1 -- 172.21.15.102:0/3616148533 >> v1:172.21.15.181:6789/0 conn(0x7f604012c150 legacy=0x7f6040133ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-05-25T12:01:18.867 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.848+0000 7f603effd700 1 -- 172.21.15.102:0/3616148533 >> v1:172.21.15.102:6789/0 conn(0x7f6040098930 legacy=0x7f604012b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:18.867 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.848+0000 7f603effd700 10 monclient: _finish_hunting 0 2021-05-25T12:01:18.868 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.848+0000 7f603effd700 1 monclient: found mon.noname-b 2021-05-25T12:01:18.868 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.848+0000 7f603effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:01:18.868 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.848+0000 7f603effd700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T12:01:18.868 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.848+0000 7f603effd700 1 -- 172.21.15.102:0/3616148533 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f604009af00 con 0x7f604012bca0 2021-05-25T12:01:18.868 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.848+0000 7f603effd700 10 monclient: _finish_auth 0 2021-05-25T12:01:18.869 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.848+0000 7f603effd700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:01:18.869 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.848+0000 7f603effd700 1 -- 172.21.15.102:0/3616148533 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f6030001610 con 0x7f604012bca0 2021-05-25T12:01:18.869 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.848+0000 7f603effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:01:18.869 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.848+0000 7f603effd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:01:18.869 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.848+0000 7f603effd700 10 monclient: dump: 2021-05-25T12:01:18.870 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:01:18.870 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:18.870 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:18.870 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:18.870 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:01:18.871 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:18.871 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:01:18.871 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:01:18.871 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:01:18.871 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:18.872 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.848+0000 7f604546c700 5 monclient: authenticate success, global_id 15575 2021-05-25T12:01:18.872 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.848+0000 7f604546c700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:01:18.872 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.848+0000 7f603effd700 1 -- 172.21.15.102:0/3616148533 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6030001880 con 0x7f604012bca0 2021-05-25T12:01:18.872 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.848+0000 7f603effd700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:01:18.873 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.848+0000 7f604546c700 10 monclient: get_monmap_and_config success 2021-05-25T12:01:18.873 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.848+0000 7f603effd700 1 -- 172.21.15.102:0/3616148533 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f6030001c10 con 0x7f604012bca0 2021-05-25T12:01:18.873 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.848+0000 7f603effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:01:18.873 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.848+0000 7f604546c700 4 set_mon_vals no callback set 2021-05-25T12:01:18.873 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.848+0000 7f603effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:01:18.874 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.848+0000 7f603effd700 10 monclient: dump: 2021-05-25T12:01:18.874 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:01:18.874 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:18.874 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:18.874 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:18.875 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:01:18.875 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:18.875 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:01:18.875 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:01:18.875 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:01:18.876 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:18.876 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.848+0000 7f604546c700 10 monclient: shutdown 2021-05-25T12:01:18.876 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.848+0000 7f604546c700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:01:18.876 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.848+0000 7f604546c700 1 -- 172.21.15.102:0/3616148533 >> v1:172.21.15.102:6790/0 conn(0x7f604012bca0 legacy=0x7f60401308c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:18.876 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.848+0000 7f604546c700 1 -- 172.21.15.102:0/3616148533 shutdown_connections 2021-05-25T12:01:18.877 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.848+0000 7f604546c700 1 -- 172.21.15.102:0/3616148533 >> 172.21.15.102:0/3616148533 conn(0x7f604009e9d0 msgr2=0x7f604009ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:01:18.877 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.848+0000 7f604546c700 1 -- 172.21.15.102:0/3616148533 shutdown_connections 2021-05-25T12:01:18.877 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.848+0000 7f604546c700 1 -- 172.21.15.102:0/3616148533 wait complete. 2021-05-25T12:01:18.877 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.849+0000 7f604546c700 10 monclient: build_initial_monmap 2021-05-25T12:01:18.878 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.849+0000 7f604546c700 10 monclient: monmap: 2021-05-25T12:01:18.878 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:01:18.878 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:01:18.878 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:01:18.878 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:01:18.879 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:01:18.879 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:01:18.879 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:01:18.879 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:01:18.880 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:01:18.880 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:18.880 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.849+0000 7f604546c700 1 librados: starting msgr at 2021-05-25T12:01:18.880 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.849+0000 7f604546c700 1 librados: starting objecter 2021-05-25T12:01:18.881 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.849+0000 7f604546c700 1 Processor -- start 2021-05-25T12:01:18.881 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.849+0000 7f604546c700 1 -- start start 2021-05-25T12:01:18.881 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.849+0000 7f604546c700 1 librados: setting wanted keys 2021-05-25T12:01:18.881 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.849+0000 7f604546c700 1 librados: calling monclient init 2021-05-25T12:01:18.882 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.849+0000 7f604546c700 10 monclient: init 2021-05-25T12:01:18.882 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.850+0000 7f604546c700 10 monclient: _reopen_session rank -1 2021-05-25T12:01:18.882 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.850+0000 7f604546c700 10 monclient: _add_conns ranks=[1,2,0] 2021-05-25T12:01:18.882 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.850+0000 7f604546c700 10 monclient(hunting): picked mon.noname-b con 0x7f6040098930 addr v1:172.21.15.181:6789/0 2021-05-25T12:01:18.882 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.850+0000 7f604546c700 10 monclient(hunting): picked mon.noname-c con 0x7f604012bca0 addr v1:172.21.15.102:6790/0 2021-05-25T12:01:18.883 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.850+0000 7f604546c700 10 monclient(hunting): picked mon.noname-a con 0x7f604012c150 addr v1:172.21.15.102:6789/0 2021-05-25T12:01:18.883 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.850+0000 7f604546c700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f604009a190 con 0x7f604012c150 2021-05-25T12:01:18.883 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.850+0000 7f604546c700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6040095680 con 0x7f6040098930 2021-05-25T12:01:18.883 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.850+0000 7f604546c700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6040095920 con 0x7f604012bca0 2021-05-25T12:01:18.884 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.850+0000 7f604546c700 10 monclient(hunting): _renew_subs 2021-05-25T12:01:18.884 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.850+0000 7f604546c700 10 monclient(hunting): authenticate will time out at 879.447428s 2021-05-25T12:01:18.884 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.850+0000 7f603f7fe700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f604012bca0 0x7f6040122a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:54648/0 (socket says 172.21.15.102:54648) 2021-05-25T12:01:18.884 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.850+0000 7f603f7fe700 1 -- 172.21.15.102:0/725246391 learned_addr learned my addr 172.21.15.102:0/725246391 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:01:18.884 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.850+0000 7f602bfff700 10 client.?.objecter ms_handle_connect 0x7f604012c150 2021-05-25T12:01:18.885 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.850+0000 7f602bfff700 10 client.?.objecter resend_mon_ops 2021-05-25T12:01:18.885 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.850+0000 7f602bfff700 10 client.?.objecter ms_handle_connect 0x7f604012bca0 2021-05-25T12:01:18.885 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.850+0000 7f602bfff700 10 client.?.objecter resend_mon_ops 2021-05-25T12:01:18.885 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.850+0000 7f602bfff700 10 client.?.objecter ms_handle_connect 0x7f6040098930 2021-05-25T12:01:18.885 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.850+0000 7f602bfff700 10 client.?.objecter resend_mon_ops 2021-05-25T12:01:18.886 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.850+0000 7f602bfff700 1 -- 172.21.15.102:0/725246391 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 396578825 0 0) 0x7f6040095920 con 0x7f604012bca0 2021-05-25T12:01:18.886 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.850+0000 7f602bfff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:01:18.886 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.850+0000 7f602bfff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:01:18.886 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.850+0000 7f602bfff700 10 monclient(hunting): my global_id is 15581 2021-05-25T12:01:18.887 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.850+0000 7f602bfff700 1 -- 172.21.15.102:0/725246391 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6020004c10 con 0x7f604012bca0 2021-05-25T12:01:18.887 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.850+0000 7f602bfff700 1 -- 172.21.15.102:0/725246391 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 436143805 0 0) 0x7f604009a190 con 0x7f604012c150 2021-05-25T12:01:18.887 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.850+0000 7f602bfff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:01:18.887 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.850+0000 7f602bfff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:01:18.887 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.850+0000 7f602bfff700 10 monclient(hunting): my global_id is 86445 2021-05-25T12:01:18.888 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.850+0000 7f602bfff700 1 -- 172.21.15.102:0/725246391 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6040095920 con 0x7f604012c150 2021-05-25T12:01:18.888 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.850+0000 7f602bfff700 1 -- 172.21.15.102:0/725246391 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 705801725 0 0) 0x7f6040095680 con 0x7f6040098930 2021-05-25T12:01:18.888 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.850+0000 7f602bfff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:01:18.888 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.850+0000 7f602bfff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:01:18.888 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.850+0000 7f602bfff700 10 monclient(hunting): my global_id is 83902 2021-05-25T12:01:18.889 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.850+0000 7f602bfff700 1 -- 172.21.15.102:0/725246391 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f604009a190 con 0x7f6040098930 2021-05-25T12:01:18.889 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.850+0000 7f602bfff700 1 -- 172.21.15.102:0/725246391 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 387215761 0 0) 0x7f6040095920 con 0x7f604012c150 2021-05-25T12:01:18.889 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.850+0000 7f602bfff700 1 -- 172.21.15.102:0/725246391 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6040095680 con 0x7f604012c150 2021-05-25T12:01:18.889 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.850+0000 7f602bfff700 1 -- 172.21.15.102:0/725246391 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f6034001aa0 con 0x7f604012c150 2021-05-25T12:01:18.889 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.850+0000 7f602bfff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:01:18.890 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.850+0000 7f602bfff700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:01:18.890 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.850+0000 7f602bfff700 10 monclient(hunting): dump: 2021-05-25T12:01:18.890 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:01:18.890 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:18.891 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:18.891 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:18.891 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:01:18.891 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:18.891 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:01:18.892 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:01:18.892 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:01:18.892 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:18.892 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.850+0000 7f602bfff700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:01:18.892 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.850+0000 7f602bfff700 1 -- 172.21.15.102:0/725246391 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3997596650 0 0) 0x7f6020004c10 con 0x7f604012bca0 2021-05-25T12:01:18.893 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.850+0000 7f602bfff700 1 -- 172.21.15.102:0/725246391 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6040095920 con 0x7f604012bca0 2021-05-25T12:01:18.893 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.851+0000 7f602bfff700 1 -- 172.21.15.102:0/725246391 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f6030001c50 con 0x7f604012bca0 2021-05-25T12:01:18.893 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.851+0000 7f602bfff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:01:18.893 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.851+0000 7f602bfff700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2021-05-25T12:01:18.894 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.851+0000 7f602bfff700 10 monclient(hunting): dump: 2021-05-25T12:01:18.894 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:01:18.894 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:18.894 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:18.894 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:18.895 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:01:18.895 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:18.895 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:01:18.895 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:01:18.895 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:01:18.896 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:18.896 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.851+0000 7f602bfff700 1 -- 172.21.15.102:0/725246391 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 554953254 0 0) 0x7f6040095680 con 0x7f604012c150 2021-05-25T12:01:18.896 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.851+0000 7f602bfff700 1 -- 172.21.15.102:0/725246391 >> v1:172.21.15.102:6790/0 conn(0x7f604012bca0 legacy=0x7f6040122a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:18.896 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.851+0000 7f602bfff700 1 -- 172.21.15.102:0/725246391 >> v1:172.21.15.181:6789/0 conn(0x7f6040098930 legacy=0x7f6040122300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:18.896 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.851+0000 7f602bfff700 10 monclient: _finish_hunting 0 2021-05-25T12:01:18.897 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.851+0000 7f602bfff700 1 monclient: found mon.a 2021-05-25T12:01:18.897 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.851+0000 7f602bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:01:18.897 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.851+0000 7f602bfff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:01:18.897 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.851+0000 7f602bfff700 1 -- 172.21.15.102:0/725246391 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60400937f0 con 0x7f604012c150 2021-05-25T12:01:18.897 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.851+0000 7f602bfff700 10 monclient: _finish_auth 0 2021-05-25T12:01:18.898 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.851+0000 7f602bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:01:18.898 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.851+0000 7f604546c700 5 monclient: authenticate success, global_id 86445 2021-05-25T12:01:18.898 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.851+0000 7f604546c700 10 monclient: _renew_subs 2021-05-25T12:01:18.898 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.851+0000 7f604546c700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:01:18.899 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.851+0000 7f604546c700 1 -- 172.21.15.102:0/725246391 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6040092d80 con 0x7f604012c150 2021-05-25T12:01:18.899 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.851+0000 7f604546c700 10 client.86445.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:01:18.899 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.851+0000 7f604546c700 10 monclient: _renew_subs 2021-05-25T12:01:18.899 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.851+0000 7f604546c700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:01:18.899 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.851+0000 7f604546c700 1 -- 172.21.15.102:0/725246391 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f60400961b0 con 0x7f604012c150 2021-05-25T12:01:18.900 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.851+0000 7f604546c700 1 librados: init done 2021-05-25T12:01:18.900 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.851+0000 7f602bfff700 1 -- 172.21.15.102:0/725246391 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6034002130 con 0x7f604012c150 2021-05-25T12:01:18.900 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.851+0000 7f602bfff700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:01:18.900 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.851+0000 7f602bfff700 1 -- 172.21.15.102:0/725246391 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f60340024c0 con 0x7f604012c150 2021-05-25T12:01:18.901 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.851+0000 7f602bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:01:18.901 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.851+0000 7f603e7fc700 4 set_mon_vals no callback set 2021-05-25T12:01:18.901 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.851+0000 7f602bfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:01:18.901 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.851+0000 7f602bfff700 10 monclient: dump: 2021-05-25T12:01:18.901 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:01:18.902 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:18.902 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:18.902 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:18.902 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:01:18.902 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:18.903 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:01:18.903 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:01:18.903 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:01:18.903 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:18.904 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.851+0000 7f602bfff700 1 -- 172.21.15.102:0/725246391 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f60340026a0 con 0x7f604012c150 2021-05-25T12:01:18.904 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.852+0000 7f604546c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:01:18.904 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.852+0000 7f604546c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:01:18.904 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.852+0000 7f604546c700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:01:18.905 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.852+0000 7f604546c700 1 -- 172.21.15.102:0/725246391 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6018004db0 con 0x7f604012c150 2021-05-25T12:01:18.905 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.852+0000 7f602bfff700 1 -- 172.21.15.102:0/725246391 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(40..40 src has 1..40) v4 ==== 8421+0+0 (unknown 640646119 0 0) 0x7f603403bdd0 con 0x7f604012c150 2021-05-25T12:01:18.905 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.852+0000 7f602bfff700 10 client.86445.objecter ms_dispatch 0x7f6040001550 osd_map(40..40 src has 1..40) v4 2021-05-25T12:01:18.905 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.852+0000 7f602bfff700 3 client.86445.objecter handle_osd_map got epochs [40,40] > 0 2021-05-25T12:01:18.906 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.852+0000 7f602bfff700 3 client.86445.objecter handle_osd_map decoding full epoch 40 2021-05-25T12:01:18.906 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.852+0000 7f602bfff700 20 client.86445.objecter dump_active .. 0 homeless 2021-05-25T12:01:18.906 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.852+0000 7f603ffff700 10 monclient: get_auth_request con 0x7f6020033200 auth_method 0 2021-05-25T12:01:18.906 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.852+0000 7f602bfff700 10 client.86445.objecter ms_handle_connect 0x7f6020033200 2021-05-25T12:01:18.907 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.856+0000 7f602bfff700 1 -- 172.21.15.102:0/725246391 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7f60340107b0 con 0x7f604012c150 2021-05-25T12:01:18.907 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.856+0000 7f602bfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:01:18.907 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:18.856+0000 7f602bfff700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:01:19.087 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.085+0000 7f604546c700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2021-05-25T12:01:19.088 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.085+0000 7f604546c700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2021-05-25T12:01:19.088 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.085+0000 7f604546c700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:01:19.088 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.085+0000 7f604546c700 1 -- 172.21.15.102:0/725246391 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f60180055e0 con 0x7f604012c150 2021-05-25T12:01:19.088 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.086+0000 7f602bfff700 1 -- 172.21.15.102:0/725246391 <== mon.0 v1:172.21.15.102:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1342 (unknown 254746657 0 3031887315) 0x7f60340105d0 con 0x7f604012c150 2021-05-25T12:01:19.088 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.086+0000 7f602bfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2021-05-25T12:01:19.089 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.086+0000 7f602bfff700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:01:19.089 INFO:teuthology.orchestra.run.smithi102.stdout:{"election_epoch":20,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":2,"features":{"quorum_con":"4540138303579357183","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"20ee91b6-8935-4b98-b868-85c03789ccc4","modified":"2021-05-25T11:59:11.011522Z","created":"2021-05-25T11:59:11.011522Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6790","nonce":0}]},"addr":"172.21.15.102:6790/0","public_addr":"172.21.15.102:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2021-05-25T12:01:19.091 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.087+0000 7f604546c700 10 librados: watch_flush enter 2021-05-25T12:01:19.091 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.087+0000 7f604546c700 10 librados: watch_flush exit 2021-05-25T12:01:19.091 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.087+0000 7f604546c700 20 client.86445.objecter shutdown clearing up homeless session... 2021-05-25T12:01:19.091 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.087+0000 7f604546c700 10 client.86445.objecter successfully canceled tick 2021-05-25T12:01:19.092 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.087+0000 7f604546c700 1 -- 172.21.15.102:0/725246391 >> v1:172.21.15.181:6812/29582 conn(0x7f6020033200 legacy=0x7f60200356c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:19.092 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.088+0000 7f604546c700 10 monclient: shutdown 2021-05-25T12:01:19.092 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.088+0000 7f604546c700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:01:19.092 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.088+0000 7f604546c700 1 -- 172.21.15.102:0/725246391 >> v1:172.21.15.102:6789/0 conn(0x7f604012c150 legacy=0x7f6040142d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:19.093 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.088+0000 7f604546c700 1 -- 172.21.15.102:0/725246391 shutdown_connections 2021-05-25T12:01:19.093 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.088+0000 7f604546c700 1 -- 172.21.15.102:0/725246391 >> 172.21.15.102:0/725246391 conn(0x7f604009e9d0 msgr2=0x7f6040129f20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:01:19.093 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.088+0000 7f604546c700 1 -- 172.21.15.102:0/725246391 shutdown_connections 2021-05-25T12:01:19.093 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.088+0000 7f604546c700 1 -- 172.21.15.102:0/725246391 wait complete. 2021-05-25T12:01:19.093 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.088+0000 7f604546c700 1 librados: shutdown 2021-05-25T12:01:19.100 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2021-05-25T12:01:19.101 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2021-05-25T12:01:19.101 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2021-05-25T12:01:19.236 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.234+0000 7f3642f10700 10 monclient: get_monmap_and_config 2021-05-25T12:01:19.236 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.234+0000 7f3642f10700 10 monclient: build_initial_monmap 2021-05-25T12:01:19.236 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.234+0000 7f3642f10700 10 monclient: monmap: 2021-05-25T12:01:19.237 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:01:19.237 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:19.237 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:01:19.235852+0000 2021-05-25T12:01:19.237 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:01:19.235852+0000 2021-05-25T12:01:19.238 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:01:19.238 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:19.238 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:01:19.238 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:01:19.238 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:01:19.239 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:19.251 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.235+0000 7f3642f10700 1 Processor -- start 2021-05-25T12:01:19.251 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.235+0000 7f3642f10700 1 -- start start 2021-05-25T12:01:19.252 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.235+0000 7f3642f10700 10 monclient: init 2021-05-25T12:01:19.252 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.236+0000 7f3642f10700 10 monclient: _reopen_session rank -1 2021-05-25T12:01:19.252 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.236+0000 7f3642f10700 10 monclient: _add_conns ranks=[1,0,2] 2021-05-25T12:01:19.252 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.236+0000 7f3642f10700 10 monclient(hunting): picked mon.noname-c con 0x7f363c098930 addr v1:172.21.15.181:6789/0 2021-05-25T12:01:19.252 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.236+0000 7f3642f10700 10 monclient(hunting): picked mon.noname-a con 0x7f363c12bca0 addr v1:172.21.15.102:6789/0 2021-05-25T12:01:19.253 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.236+0000 7f3642f10700 10 monclient(hunting): picked mon.noname-b con 0x7f363c12c150 addr v1:172.21.15.102:6790/0 2021-05-25T12:01:19.253 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.236+0000 7f3642f10700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f363c099b20 con 0x7f363c12bca0 2021-05-25T12:01:19.253 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.236+0000 7f3642f10700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f363c09a3a0 con 0x7f363c098930 2021-05-25T12:01:19.253 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.236+0000 7f3642f10700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f363c09ac60 con 0x7f363c12c150 2021-05-25T12:01:19.253 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.236+0000 7f3642f10700 10 monclient(hunting): _renew_subs 2021-05-25T12:01:19.254 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.236+0000 7f3642f10700 10 monclient(hunting): authenticate will time out at 879.833214s 2021-05-25T12:01:19.254 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.236+0000 7f364170d700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f363c12bca0 0x7f363c1308c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:59778/0 (socket says 172.21.15.102:59778) 2021-05-25T12:01:19.254 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.236+0000 7f364170d700 1 -- 172.21.15.102:0/2539009645 learned_addr learned my addr 172.21.15.102:0/2539009645 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:01:19.254 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.236+0000 7f3640f0c700 1 -- 172.21.15.102:0/2539009645 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3141809263 0 0) 0x7f363c099b20 con 0x7f363c12bca0 2021-05-25T12:01:19.255 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.236+0000 7f3640f0c700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:01:19.255 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.236+0000 7f3640f0c700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:01:19.255 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.236+0000 7f3640f0c700 10 monclient(hunting): my global_id is 86730 2021-05-25T12:01:19.255 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.236+0000 7f3640f0c700 1 -- 172.21.15.102:0/2539009645 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3624003270 con 0x7f363c12bca0 2021-05-25T12:01:19.255 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.236+0000 7f3640f0c700 1 -- 172.21.15.102:0/2539009645 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3550230173 0 0) 0x7f363c09ac60 con 0x7f363c12c150 2021-05-25T12:01:19.256 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.236+0000 7f3640f0c700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:01:19.256 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.236+0000 7f3640f0c700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:01:19.256 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.236+0000 7f3640f0c700 10 monclient(hunting): my global_id is 15842 2021-05-25T12:01:19.256 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.236+0000 7f3640f0c700 1 -- 172.21.15.102:0/2539009645 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f363c099b20 con 0x7f363c12c150 2021-05-25T12:01:19.256 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.236+0000 7f3640f0c700 1 -- 172.21.15.102:0/2539009645 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2297799347 0 0) 0x7f363c09a3a0 con 0x7f363c098930 2021-05-25T12:01:19.257 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.236+0000 7f3640f0c700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:01:19.257 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.236+0000 7f3640f0c700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:01:19.257 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.236+0000 7f3640f0c700 10 monclient(hunting): my global_id is 84115 2021-05-25T12:01:19.257 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.236+0000 7f3640f0c700 1 -- 172.21.15.102:0/2539009645 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f363c09ac60 con 0x7f363c098930 2021-05-25T12:01:19.257 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.236+0000 7f3640f0c700 1 -- 172.21.15.102:0/2539009645 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2949603823 0 0) 0x7f3624003270 con 0x7f363c12bca0 2021-05-25T12:01:19.258 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.236+0000 7f3640f0c700 1 -- 172.21.15.102:0/2539009645 >> v1:172.21.15.102:6790/0 conn(0x7f363c12c150 legacy=0x7f363c133ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:19.258 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.236+0000 7f3640f0c700 1 -- 172.21.15.102:0/2539009645 >> v1:172.21.15.181:6789/0 conn(0x7f363c098930 legacy=0x7f363c12b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:19.258 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.236+0000 7f3640f0c700 10 monclient: _finish_hunting 0 2021-05-25T12:01:19.258 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.236+0000 7f3640f0c700 1 monclient: found mon.noname-a 2021-05-25T12:01:19.258 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.236+0000 7f3640f0c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:01:19.259 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.236+0000 7f3640f0c700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T12:01:19.259 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.236+0000 7f3640f0c700 1 -- 172.21.15.102:0/2539009645 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f363c09af00 con 0x7f363c12bca0 2021-05-25T12:01:19.259 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.236+0000 7f3640f0c700 10 monclient: _finish_auth 0 2021-05-25T12:01:19.259 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.236+0000 7f3640f0c700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:01:19.259 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.236+0000 7f3642f10700 5 monclient: authenticate success, global_id 86730 2021-05-25T12:01:19.260 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.236+0000 7f3640f0c700 1 -- 172.21.15.102:0/2539009645 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f362c001610 con 0x7f363c12bca0 2021-05-25T12:01:19.260 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.236+0000 7f3642f10700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:01:19.260 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.236+0000 7f3640f0c700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:01:19.260 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.236+0000 7f3640f0c700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:01:19.261 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.236+0000 7f3640f0c700 10 monclient: dump: 2021-05-25T12:01:19.261 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:01:19.261 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:19.261 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:19.261 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:19.261 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:01:19.262 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:19.262 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:01:19.262 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:01:19.262 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:01:19.262 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:19.263 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.236+0000 7f3642f10700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:01:19.263 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.236+0000 7f3640f0c700 1 -- 172.21.15.102:0/2539009645 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f362c001880 con 0x7f363c12bca0 2021-05-25T12:01:19.263 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.236+0000 7f3640f0c700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:01:19.263 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.236+0000 7f3640f0c700 1 -- 172.21.15.102:0/2539009645 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f362c001c10 con 0x7f363c12bca0 2021-05-25T12:01:19.264 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.236+0000 7f3642f10700 10 monclient: get_monmap_and_config success 2021-05-25T12:01:19.264 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.236+0000 7f3642f10700 4 set_mon_vals no callback set 2021-05-25T12:01:19.264 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.236+0000 7f3640f0c700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:01:19.265 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.236+0000 7f3640f0c700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:01:19.265 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.236+0000 7f3640f0c700 10 monclient: dump: 2021-05-25T12:01:19.265 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:01:19.265 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:19.265 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:19.266 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:19.266 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:01:19.266 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:19.266 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:01:19.266 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:01:19.267 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:01:19.267 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:19.267 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.237+0000 7f3642f10700 10 monclient: shutdown 2021-05-25T12:01:19.267 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.237+0000 7f3642f10700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:01:19.267 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.237+0000 7f3642f10700 1 -- 172.21.15.102:0/2539009645 >> v1:172.21.15.102:6789/0 conn(0x7f363c12bca0 legacy=0x7f363c1308c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:19.268 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.237+0000 7f3642f10700 1 -- 172.21.15.102:0/2539009645 shutdown_connections 2021-05-25T12:01:19.268 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.237+0000 7f3642f10700 1 -- 172.21.15.102:0/2539009645 >> 172.21.15.102:0/2539009645 conn(0x7f363c09e9d0 msgr2=0x7f363c09ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:01:19.268 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.237+0000 7f3642f10700 1 -- 172.21.15.102:0/2539009645 shutdown_connections 2021-05-25T12:01:19.268 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.237+0000 7f3642f10700 1 -- 172.21.15.102:0/2539009645 wait complete. 2021-05-25T12:01:19.268 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.237+0000 7f3642f10700 10 monclient: build_initial_monmap 2021-05-25T12:01:19.269 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.237+0000 7f3642f10700 10 monclient: monmap: 2021-05-25T12:01:19.269 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:01:19.269 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:01:19.269 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:01:19.269 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:01:19.270 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:01:19.270 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:01:19.270 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:01:19.270 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:01:19.270 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:01:19.271 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:19.271 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.238+0000 7f3642f10700 1 librados: starting msgr at 2021-05-25T12:01:19.271 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.238+0000 7f3642f10700 1 librados: starting objecter 2021-05-25T12:01:19.271 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.238+0000 7f3642f10700 1 Processor -- start 2021-05-25T12:01:19.271 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.238+0000 7f3642f10700 1 -- start start 2021-05-25T12:01:19.272 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.238+0000 7f3642f10700 1 librados: setting wanted keys 2021-05-25T12:01:19.272 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.238+0000 7f3642f10700 1 librados: calling monclient init 2021-05-25T12:01:19.272 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.238+0000 7f3642f10700 10 monclient: init 2021-05-25T12:01:19.272 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.238+0000 7f3642f10700 10 monclient: _reopen_session rank -1 2021-05-25T12:01:19.272 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.238+0000 7f3642f10700 10 monclient: _add_conns ranks=[2,0,1] 2021-05-25T12:01:19.273 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.238+0000 7f3642f10700 10 monclient(hunting): picked mon.noname-c con 0x7f363c098930 addr v1:172.21.15.102:6790/0 2021-05-25T12:01:19.273 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.238+0000 7f3642f10700 10 monclient(hunting): picked mon.noname-a con 0x7f363c12bca0 addr v1:172.21.15.102:6789/0 2021-05-25T12:01:19.273 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.238+0000 7f3642f10700 10 monclient(hunting): picked mon.noname-b con 0x7f363c12c150 addr v1:172.21.15.181:6789/0 2021-05-25T12:01:19.273 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.238+0000 7f3642f10700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f363c09a190 con 0x7f363c12bca0 2021-05-25T12:01:19.273 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.238+0000 7f3642f10700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f363c092b70 con 0x7f363c12c150 2021-05-25T12:01:19.274 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.238+0000 7f3642f10700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f363c094c60 con 0x7f363c098930 2021-05-25T12:01:19.274 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.238+0000 7f3642f10700 10 monclient(hunting): _renew_subs 2021-05-25T12:01:19.274 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.238+0000 7f3642f10700 10 monclient(hunting): authenticate will time out at 879.835762s 2021-05-25T12:01:19.274 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.238+0000 7f3641f0e700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f363c098930 0x7f363c092460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:54660/0 (socket says 172.21.15.102:54660) 2021-05-25T12:01:19.274 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.238+0000 7f3641f0e700 1 -- 172.21.15.102:0/1764542994 learned_addr learned my addr 172.21.15.102:0/1764542994 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:01:19.275 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.238+0000 7f364170d700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f363c12bca0 0x7f363c0973c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:59784/0 (socket says 172.21.15.102:59784) 2021-05-25T12:01:19.275 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.238+0000 7f3629ffb700 10 client.?.objecter ms_handle_connect 0x7f363c098930 2021-05-25T12:01:19.275 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.238+0000 7f3629ffb700 10 client.?.objecter resend_mon_ops 2021-05-25T12:01:19.275 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.238+0000 7f3629ffb700 10 client.?.objecter ms_handle_connect 0x7f363c12bca0 2021-05-25T12:01:19.275 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.238+0000 7f3629ffb700 10 client.?.objecter resend_mon_ops 2021-05-25T12:01:19.276 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.239+0000 7f3629ffb700 10 client.?.objecter ms_handle_connect 0x7f363c12c150 2021-05-25T12:01:19.276 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.239+0000 7f3629ffb700 10 client.?.objecter resend_mon_ops 2021-05-25T12:01:19.276 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.239+0000 7f3629ffb700 1 -- 172.21.15.102:0/1764542994 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2635811364 0 0) 0x7f363c094c60 con 0x7f363c098930 2021-05-25T12:01:19.276 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.239+0000 7f3629ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:01:19.276 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.239+0000 7f3629ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:01:19.277 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.239+0000 7f3629ffb700 10 monclient(hunting): my global_id is 15851 2021-05-25T12:01:19.277 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.239+0000 7f3629ffb700 1 -- 172.21.15.102:0/1764542994 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f361c004c10 con 0x7f363c098930 2021-05-25T12:01:19.277 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.239+0000 7f3629ffb700 1 -- 172.21.15.102:0/1764542994 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3378654343 0 0) 0x7f363c09a190 con 0x7f363c12bca0 2021-05-25T12:01:19.277 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.239+0000 7f3629ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:01:19.277 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.239+0000 7f3629ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:01:19.278 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.239+0000 7f3629ffb700 10 monclient(hunting): my global_id is 86739 2021-05-25T12:01:19.278 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.239+0000 7f3629ffb700 1 -- 172.21.15.102:0/1764542994 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f363c094c60 con 0x7f363c12bca0 2021-05-25T12:01:19.278 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.239+0000 7f3629ffb700 1 -- 172.21.15.102:0/1764542994 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2416521287 0 0) 0x7f361c004c10 con 0x7f363c098930 2021-05-25T12:01:19.278 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.239+0000 7f3629ffb700 1 -- 172.21.15.102:0/1764542994 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f363c09a190 con 0x7f363c098930 2021-05-25T12:01:19.278 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.239+0000 7f3629ffb700 1 -- 172.21.15.102:0/1764542994 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f3630002970 con 0x7f363c098930 2021-05-25T12:01:19.279 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.239+0000 7f3629ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:01:19.279 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.239+0000 7f3629ffb700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:01:19.279 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.239+0000 7f3629ffb700 10 monclient(hunting): dump: 2021-05-25T12:01:19.279 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:01:19.279 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:19.280 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:19.280 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:19.280 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:01:19.280 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:19.280 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:01:19.280 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:01:19.281 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:01:19.281 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:19.281 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.239+0000 7f3629ffb700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:01:19.281 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.239+0000 7f3629ffb700 1 -- 172.21.15.102:0/1764542994 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1249833659 0 0) 0x7f363c094c60 con 0x7f363c12bca0 2021-05-25T12:01:19.281 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.239+0000 7f3629ffb700 1 -- 172.21.15.102:0/1764542994 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f361c004c10 con 0x7f363c12bca0 2021-05-25T12:01:19.282 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.239+0000 7f3629ffb700 1 -- 172.21.15.102:0/1764542994 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f362c001430 con 0x7f363c12bca0 2021-05-25T12:01:19.282 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.239+0000 7f3629ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:01:19.282 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.239+0000 7f3629ffb700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2021-05-25T12:01:19.282 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.239+0000 7f3629ffb700 10 monclient(hunting): dump: 2021-05-25T12:01:19.282 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:01:19.283 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:19.283 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:19.283 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:19.283 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:01:19.283 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:19.284 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:01:19.284 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:01:19.284 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:01:19.284 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:19.284 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.239+0000 7f3629ffb700 1 -- 172.21.15.102:0/1764542994 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4228140090 0 0) 0x7f363c09a190 con 0x7f363c098930 2021-05-25T12:01:19.285 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.239+0000 7f3629ffb700 1 -- 172.21.15.102:0/1764542994 >> v1:172.21.15.181:6789/0 conn(0x7f363c12c150 legacy=0x7f363c143050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:19.285 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.239+0000 7f3629ffb700 1 -- 172.21.15.102:0/1764542994 >> v1:172.21.15.102:6789/0 conn(0x7f363c12bca0 legacy=0x7f363c0973c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:19.285 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.239+0000 7f3629ffb700 10 monclient: _finish_hunting 0 2021-05-25T12:01:19.285 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.239+0000 7f3629ffb700 1 monclient: found mon.c 2021-05-25T12:01:19.286 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.239+0000 7f3629ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:01:19.286 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.239+0000 7f3629ffb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:01:19.286 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.239+0000 7f3629ffb700 1 -- 172.21.15.102:0/1764542994 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f363c094f00 con 0x7f363c098930 2021-05-25T12:01:19.287 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.239+0000 7f3629ffb700 10 monclient: _finish_auth 0 2021-05-25T12:01:19.287 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.239+0000 7f3629ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:01:19.287 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.239+0000 7f3642f10700 5 monclient: authenticate success, global_id 15851 2021-05-25T12:01:19.287 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.239+0000 7f3642f10700 10 monclient: _renew_subs 2021-05-25T12:01:19.287 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.239+0000 7f3642f10700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:01:19.288 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.239+0000 7f3642f10700 1 -- 172.21.15.102:0/1764542994 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f363c097ad0 con 0x7f363c098930 2021-05-25T12:01:19.288 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.239+0000 7f3629ffb700 1 -- 172.21.15.102:0/1764542994 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3630002cb0 con 0x7f363c098930 2021-05-25T12:01:19.288 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.239+0000 7f3629ffb700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:01:19.288 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.239+0000 7f3629ffb700 1 -- 172.21.15.102:0/1764542994 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f3630001820 con 0x7f363c098930 2021-05-25T12:01:19.288 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.239+0000 7f3629ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:01:19.289 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.239+0000 7f3629ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:01:19.289 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.239+0000 7f3629ffb700 10 monclient: dump: 2021-05-25T12:01:19.289 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:01:19.289 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:19.290 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:19.290 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:19.290 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:01:19.290 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:19.290 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:01:19.291 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:01:19.291 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:01:19.291 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:19.291 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.239+0000 7f3642f10700 10 client.15851.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:01:19.291 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.239+0000 7f362bfff700 4 set_mon_vals no callback set 2021-05-25T12:01:19.292 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.239+0000 7f3642f10700 10 monclient: _renew_subs 2021-05-25T12:01:19.292 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.239+0000 7f3642f10700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:01:19.292 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.239+0000 7f3642f10700 1 -- 172.21.15.102:0/1764542994 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f363c127a00 con 0x7f363c098930 2021-05-25T12:01:19.292 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.239+0000 7f3642f10700 1 librados: init done 2021-05-25T12:01:19.292 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.240+0000 7f3629ffb700 1 -- 172.21.15.102:0/1764542994 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f3630001a00 con 0x7f363c098930 2021-05-25T12:01:19.293 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.240+0000 7f36237fe700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:01:19.293 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.240+0000 7f36237fe700 1 -- 172.21.15.102:0/1764542994 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f36100065a0 con 0x7f3610002840 2021-05-25T12:01:19.293 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.240+0000 7f3629ffb700 1 -- 172.21.15.102:0/1764542994 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(41..41 src has 1..41) v4 ==== 9241+0+0 (unknown 837274651 0 0) 0x7f363003e500 con 0x7f363c098930 2021-05-25T12:01:19.293 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.240+0000 7f3629ffb700 10 client.15851.objecter ms_dispatch 0x7f363c001550 osd_map(41..41 src has 1..41) v4 2021-05-25T12:01:19.293 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.240+0000 7f3629ffb700 3 client.15851.objecter handle_osd_map got epochs [41,41] > 0 2021-05-25T12:01:19.294 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.240+0000 7f3629ffb700 3 client.15851.objecter handle_osd_map decoding full epoch 41 2021-05-25T12:01:19.294 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.240+0000 7f364270f700 10 monclient: get_auth_request con 0x7f361c033320 auth_method 0 2021-05-25T12:01:19.294 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.240+0000 7f3629ffb700 20 client.15851.objecter dump_active .. 0 homeless 2021-05-25T12:01:19.294 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.240+0000 7f3629ffb700 10 client.15851.objecter ms_handle_connect 0x7f3610002840 2021-05-25T12:01:19.294 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.240+0000 7f3629ffb700 10 client.15851.objecter resend_mon_ops 2021-05-25T12:01:19.295 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.240+0000 7f3629ffb700 1 -- 172.21.15.102:0/1764542994 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1418508754 0 0) 0x7f36100065a0 con 0x7f3610002840 2021-05-25T12:01:19.295 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.240+0000 7f3629ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:01:19.295 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.240+0000 7f3629ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:01:19.295 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.240+0000 7f3629ffb700 10 monclient(hunting): my global_id is 86742 2021-05-25T12:01:19.295 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.240+0000 7f3629ffb700 1 -- 172.21.15.102:0/1764542994 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f363c09a190 con 0x7f3610002840 2021-05-25T12:01:19.296 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.240+0000 7f3629ffb700 1 -- 172.21.15.102:0/1764542994 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 112479645 0 0) 0x7f363c09a190 con 0x7f3610002840 2021-05-25T12:01:19.296 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.241+0000 7f3629ffb700 1 -- 172.21.15.102:0/1764542994 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f36100065a0 con 0x7f3610002840 2021-05-25T12:01:19.296 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.241+0000 7f3629ffb700 10 client.15851.objecter ms_handle_connect 0x7f361c033320 2021-05-25T12:01:19.296 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.241+0000 7f3629ffb700 1 -- 172.21.15.102:0/1764542994 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f362c001230 con 0x7f3610002840 2021-05-25T12:01:19.297 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.241+0000 7f3629ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:01:19.297 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.241+0000 7f3629ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:01:19.297 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.241+0000 7f3629ffb700 10 monclient: dump: 2021-05-25T12:01:19.297 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:01:19.298 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:19.298 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:19.298 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:19.298 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:01:19.298 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:19.299 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:01:19.299 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:01:19.299 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:01:19.300 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:19.300 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.241+0000 7f3629ffb700 1 -- 172.21.15.102:0/1764542994 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2924058675 0 0) 0x7f36100065a0 con 0x7f3610002840 2021-05-25T12:01:19.300 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.241+0000 7f3629ffb700 1 -- 172.21.15.102:0/1764542994 --> v1:172.21.15.102:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3610009e40 con 0x7f3610002840 2021-05-25T12:01:19.300 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.241+0000 7f3629ffb700 1 -- 172.21.15.102:0/1764542994 <== mon.0 v1:172.21.15.102:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7f362c004a10 con 0x7f3610002840 2021-05-25T12:01:19.300 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.241+0000 7f3629ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:01:19.301 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.241+0000 7f3629ffb700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:01:19.301 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.241+0000 7f3629ffb700 1 -- 172.21.15.102:0/1764542994 >> v1:172.21.15.102:6789/0 conn(0x7f3610002840 legacy=0x7f3610004ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:19.301 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.241+0000 7f3629ffb700 1 -- 172.21.15.102:0/1764542994 >> v1:172.21.15.102:6789/0 conn(0x7f3610002840 legacy=0x7f3610004ce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:01:19.301 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.248+0000 7f36237fe700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:01:19.302 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.248+0000 7f36237fe700 1 -- 172.21.15.102:0/1764542994 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f3610009b00 con 0x7f3610008dc0 2021-05-25T12:01:19.302 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.248+0000 7f3629ffb700 10 client.15851.objecter ms_handle_connect 0x7f3610008dc0 2021-05-25T12:01:19.302 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.248+0000 7f3629ffb700 10 client.15851.objecter resend_mon_ops 2021-05-25T12:01:19.302 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.248+0000 7f3629ffb700 1 -- 172.21.15.102:0/1764542994 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2878802532 0 0) 0x7f3610009b00 con 0x7f3610008dc0 2021-05-25T12:01:19.302 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.248+0000 7f3629ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:01:19.303 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.248+0000 7f3629ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:01:19.303 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.248+0000 7f3629ffb700 10 monclient(hunting): my global_id is 86751 2021-05-25T12:01:19.303 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.248+0000 7f3629ffb700 1 -- 172.21.15.102:0/1764542994 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36100065a0 con 0x7f3610008dc0 2021-05-25T12:01:19.303 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.249+0000 7f3629ffb700 1 -- 172.21.15.102:0/1764542994 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1387006643 0 0) 0x7f36100065a0 con 0x7f3610008dc0 2021-05-25T12:01:19.303 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.249+0000 7f3629ffb700 1 -- 172.21.15.102:0/1764542994 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3610009b00 con 0x7f3610008dc0 2021-05-25T12:01:19.304 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.249+0000 7f3629ffb700 1 -- 172.21.15.102:0/1764542994 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f362c005030 con 0x7f3610008dc0 2021-05-25T12:01:19.304 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.249+0000 7f3629ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:01:19.304 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.249+0000 7f3629ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:01:19.304 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.249+0000 7f3629ffb700 10 monclient: dump: 2021-05-25T12:01:19.304 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:01:19.305 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:19.305 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:19.305 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:19.305 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:01:19.305 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:19.306 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:01:19.306 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:01:19.306 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:01:19.306 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:19.306 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.249+0000 7f3629ffb700 1 -- 172.21.15.102:0/1764542994 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3756679571 0 0) 0x7f3610009b00 con 0x7f3610008dc0 2021-05-25T12:01:19.307 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.249+0000 7f3629ffb700 1 -- 172.21.15.102:0/1764542994 --> v1:172.21.15.102:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f3610008740 con 0x7f3610008dc0 2021-05-25T12:01:19.307 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.249+0000 7f3629ffb700 1 -- 172.21.15.102:0/1764542994 <== mon.0 v1:172.21.15.102:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3771 (unknown 0 0 2205592335) 0x7f362c005360 con 0x7f3610008dc0 2021-05-25T12:01:19.307 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.249+0000 7f3629ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:01:19.307 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.249+0000 7f3629ffb700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:01:19.307 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.249+0000 7f3629ffb700 1 -- 172.21.15.102:0/1764542994 >> v1:172.21.15.102:6789/0 conn(0x7f3610008dc0 legacy=0x7f3610009250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:19.308 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.249+0000 7f3629ffb700 1 -- 172.21.15.102:0/1764542994 >> v1:172.21.15.102:6789/0 conn(0x7f3610008dc0 legacy=0x7f3610009250 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:01:19.308 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:01:19.308 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:01:19.309 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:01:19.309 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "peon", 2021-05-25T12:01:19.309 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 20, 2021-05-25T12:01:19.310 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:01:19.310 INFO:teuthology.orchestra.run.smithi102.stdout: 0, 2021-05-25T12:01:19.310 INFO:teuthology.orchestra.run.smithi102.stdout: 1, 2021-05-25T12:01:19.310 INFO:teuthology.orchestra.run.smithi102.stdout: 2 2021-05-25T12:01:19.311 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:01:19.311 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 2, 2021-05-25T12:01:19.311 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:01:19.311 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:01:19.311 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:01:19.312 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:01:19.312 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:01:19.312 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:01:19.312 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:01:19.312 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:01:19.313 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:01:19.313 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:01:19.313 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:01:19.313 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:01:19.313 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:01:19.314 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:01:19.314 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:01:19.314 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:01:19.314 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:01:19.314 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:01:19.315 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:01:19.315 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:01:19.315 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:01:19.315 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:01:19.315 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:01:19.316 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:01:19.316 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:01:19.316 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:01:19.316 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:01:19.316 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:01:19.317 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:01:19.317 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:01:19.317 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:01:19.317 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:01:19.317 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:01:19.318 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:01:19.318 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:01:19.318 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:01:19.318 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:01:19.318 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:01:19.319 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:01:19.319 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:01:19.319 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:01:19.319 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:01:19.319 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:01:19.320 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:01:19.320 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:01:19.320 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:01:19.320 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:01:19.320 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:01:19.321 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:01:19.321 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:01:19.321 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:01:19.321 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:01:19.321 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:01:19.322 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:01:19.322 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:01:19.322 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:01:19.322 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:01:19.323 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:01:19.323 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:01:19.323 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:01:19.323 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:01:19.323 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:01:19.324 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:01:19.324 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:01:19.324 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:01:19.324 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:01:19.324 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:01:19.325 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:01:19.325 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:01:19.325 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:01:19.325 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:01:19.325 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:01:19.326 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:01:19.326 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:01:19.326 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:01:19.326 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:01:19.327 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:01:19.327 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:01:19.327 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:01:19.327 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:01:19.327 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:01:19.328 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:01:19.328 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:01:19.328 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:01:19.328 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:01:19.328 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:01:19.328 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:01:19.329 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:01:19.329 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:01:19.329 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:01:19.329 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:01:19.329 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:01:19.330 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:01:19.330 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:01:19.330 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:01:19.330 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:01:19.331 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:01:19.331 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:01:19.331 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:01:19.331 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:01:19.332 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:01:19.332 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:01:19.332 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:01:19.332 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:01:19.333 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:01:19.333 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:01:19.333 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:01:19.333 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:01:19.334 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:01:19.334 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:01:19.334 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:01:19.334 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:01:19.334 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:01:19.335 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:01:19.335 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:01:19.335 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:01:19.335 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:01:19.335 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:01:19.336 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:01:19.336 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:01:19.336 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:01:19.336 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:01:19.336 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 16 2021-05-25T12:01:19.337 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:01:19.337 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:01:19.337 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:01:19.337 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:01:19.337 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:01:19.340 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.249+0000 7f36237fe700 10 librados: watch_flush enter 2021-05-25T12:01:19.340 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.249+0000 7f36237fe700 10 librados: watch_flush exit 2021-05-25T12:01:19.340 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.249+0000 7f36237fe700 20 client.15851.objecter shutdown clearing up homeless session... 2021-05-25T12:01:19.340 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.249+0000 7f36237fe700 10 client.15851.objecter successfully canceled tick 2021-05-25T12:01:19.341 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.250+0000 7f36237fe700 1 -- 172.21.15.102:0/1764542994 >> v1:172.21.15.181:6812/29582 conn(0x7f361c033320 legacy=0x7f361c0357e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:19.341 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.250+0000 7f36237fe700 10 monclient: shutdown 2021-05-25T12:01:19.341 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.250+0000 7f36237fe700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:01:19.341 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.250+0000 7f36237fe700 1 -- 172.21.15.102:0/1764542994 >> v1:172.21.15.102:6790/0 conn(0x7f363c098930 legacy=0x7f363c092460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:19.341 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.250+0000 7f364270f700 1 -- 172.21.15.102:0/1764542994 reap_dead start 2021-05-25T12:01:19.342 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.250+0000 7f364270f700 1 -- 172.21.15.102:0/1764542994 reap_dead start 2021-05-25T12:01:19.342 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.250+0000 7f36237fe700 1 -- 172.21.15.102:0/1764542994 shutdown_connections 2021-05-25T12:01:19.342 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.250+0000 7f36237fe700 1 -- 172.21.15.102:0/1764542994 >> 172.21.15.102:0/1764542994 conn(0x7f363c09e9d0 msgr2=0x7f363c128640 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:01:19.342 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.250+0000 7f36237fe700 1 -- 172.21.15.102:0/1764542994 shutdown_connections 2021-05-25T12:01:19.342 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.250+0000 7f36237fe700 1 -- 172.21.15.102:0/1764542994 wait complete. 2021-05-25T12:01:19.343 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.250+0000 7f36237fe700 1 librados: shutdown 2021-05-25T12:01:19.343 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2021-05-25T12:01:19.466 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.465+0000 7f5bb8093700 10 monclient: get_monmap_and_config 2021-05-25T12:01:19.467 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.465+0000 7f5bb8093700 10 monclient: build_initial_monmap 2021-05-25T12:01:19.467 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.465+0000 7f5bb8093700 10 monclient: monmap: 2021-05-25T12:01:19.467 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:01:19.467 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:19.468 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:01:19.465973+0000 2021-05-25T12:01:19.468 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:01:19.465973+0000 2021-05-25T12:01:19.468 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:01:19.468 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:19.468 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:01:19.469 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:01:19.469 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:01:19.469 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:19.469 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.465+0000 7f5bb8093700 1 Processor -- start 2021-05-25T12:01:19.469 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.465+0000 7f5bb8093700 1 -- start start 2021-05-25T12:01:19.470 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.465+0000 7f5bb8093700 10 monclient: init 2021-05-25T12:01:19.483 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.466+0000 7f5bb8093700 10 monclient: _reopen_session rank -1 2021-05-25T12:01:19.483 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.466+0000 7f5bb8093700 10 monclient: _add_conns ranks=[2,1,0] 2021-05-25T12:01:19.483 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.466+0000 7f5bb8093700 10 monclient(hunting): picked mon.noname-b con 0x7f5bb0097fa0 addr v1:172.21.15.102:6790/0 2021-05-25T12:01:19.483 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.466+0000 7f5bb8093700 10 monclient(hunting): picked mon.noname-c con 0x7f5bb0063b70 addr v1:172.21.15.181:6789/0 2021-05-25T12:01:19.484 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.466+0000 7f5bb8093700 10 monclient(hunting): picked mon.noname-a con 0x7f5bb01325c0 addr v1:172.21.15.102:6789/0 2021-05-25T12:01:19.484 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.466+0000 7f5bb8093700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bb0096b20 con 0x7f5bb01325c0 2021-05-25T12:01:19.484 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.466+0000 7f5bb8093700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bb009b460 con 0x7f5bb0063b70 2021-05-25T12:01:19.484 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.466+0000 7f5bb8093700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bb00964c0 con 0x7f5bb0097fa0 2021-05-25T12:01:19.485 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.466+0000 7f5bb8093700 10 monclient(hunting): _renew_subs 2021-05-25T12:01:19.485 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.466+0000 7f5bb8093700 10 monclient(hunting): authenticate will time out at 880.063231s 2021-05-25T12:01:19.485 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.466+0000 7f5bb5e2f700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f5bb0097fa0 0x7f5bb0063460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:54674/0 (socket says 172.21.15.102:54674) 2021-05-25T12:01:19.485 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.466+0000 7f5bb6630700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f5bb01325c0 0x7f5bb01349b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:59796/0 (socket says 172.21.15.102:59796) 2021-05-25T12:01:19.486 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.466+0000 7f5bb5e2f700 1 -- 172.21.15.102:0/605142792 learned_addr learned my addr 172.21.15.102:0/605142792 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:01:19.486 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.466+0000 7f5bb4e2d700 1 -- 172.21.15.102:0/605142792 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2726394470 0 0) 0x7f5bb00964c0 con 0x7f5bb0097fa0 2021-05-25T12:01:19.486 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.466+0000 7f5bb4e2d700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:01:19.486 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.466+0000 7f5bb4e2d700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:01:19.487 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.466+0000 7f5bb4e2d700 10 monclient(hunting): my global_id is 16151 2021-05-25T12:01:19.487 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.466+0000 7f5bb4e2d700 1 -- 172.21.15.102:0/605142792 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b98003270 con 0x7f5bb0097fa0 2021-05-25T12:01:19.487 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.466+0000 7f5bb4e2d700 1 -- 172.21.15.102:0/605142792 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2255554259 0 0) 0x7f5bb0096b20 con 0x7f5bb01325c0 2021-05-25T12:01:19.487 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.466+0000 7f5bb4e2d700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:01:19.487 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.466+0000 7f5bb4e2d700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:01:19.488 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.466+0000 7f5bb4e2d700 10 monclient(hunting): my global_id is 87027 2021-05-25T12:01:19.488 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.466+0000 7f5bb4e2d700 1 -- 172.21.15.102:0/605142792 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bb00964c0 con 0x7f5bb01325c0 2021-05-25T12:01:19.488 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.466+0000 7f5bb4e2d700 1 -- 172.21.15.102:0/605142792 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1638797631 0 0) 0x7f5bb009b460 con 0x7f5bb0063b70 2021-05-25T12:01:19.488 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.466+0000 7f5bb4e2d700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:01:19.489 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.466+0000 7f5bb4e2d700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:01:19.489 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.466+0000 7f5bb4e2d700 10 monclient(hunting): my global_id is 84355 2021-05-25T12:01:19.489 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.466+0000 7f5bb4e2d700 1 -- 172.21.15.102:0/605142792 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bb0096b20 con 0x7f5bb0063b70 2021-05-25T12:01:19.489 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.466+0000 7f5bb4e2d700 1 -- 172.21.15.102:0/605142792 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4008228144 0 0) 0x7f5b98003270 con 0x7f5bb0097fa0 2021-05-25T12:01:19.489 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.466+0000 7f5bb4e2d700 1 -- 172.21.15.102:0/605142792 >> v1:172.21.15.181:6789/0 conn(0x7f5bb0063b70 legacy=0x7f5bb0064020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:19.490 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.466+0000 7f5bb4e2d700 1 -- 172.21.15.102:0/605142792 >> v1:172.21.15.102:6789/0 conn(0x7f5bb01325c0 legacy=0x7f5bb01349b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:19.490 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.466+0000 7f5bb4e2d700 10 monclient: _finish_hunting 0 2021-05-25T12:01:19.490 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.466+0000 7f5bb4e2d700 1 monclient: found mon.noname-b 2021-05-25T12:01:19.490 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.466+0000 7f5bb4e2d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:01:19.490 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.466+0000 7f5bb4e2d700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T12:01:19.491 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.466+0000 7f5bb4e2d700 1 -- 172.21.15.102:0/605142792 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bb0096760 con 0x7f5bb0097fa0 2021-05-25T12:01:19.491 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.466+0000 7f5bb4e2d700 10 monclient: _finish_auth 0 2021-05-25T12:01:19.491 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.466+0000 7f5bb4e2d700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:01:19.491 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.466+0000 7f5bb4e2d700 1 -- 172.21.15.102:0/605142792 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f5ba4001610 con 0x7f5bb0097fa0 2021-05-25T12:01:19.491 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.466+0000 7f5bb4e2d700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:01:19.492 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.466+0000 7f5bb4e2d700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:01:19.492 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.466+0000 7f5bb4e2d700 10 monclient: dump: 2021-05-25T12:01:19.492 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:01:19.492 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:19.493 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:19.493 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:19.493 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:01:19.493 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:19.493 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:01:19.494 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:01:19.494 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:01:19.494 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:19.494 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.466+0000 7f5bb8093700 5 monclient: authenticate success, global_id 16151 2021-05-25T12:01:19.494 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.466+0000 7f5bb8093700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:01:19.495 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.466+0000 7f5bb4e2d700 1 -- 172.21.15.102:0/605142792 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5ba4001880 con 0x7f5bb0097fa0 2021-05-25T12:01:19.495 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.466+0000 7f5bb4e2d700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:01:19.495 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.466+0000 7f5bb8093700 10 monclient: get_monmap_and_config success 2021-05-25T12:01:19.495 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.466+0000 7f5bb4e2d700 1 -- 172.21.15.102:0/605142792 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f5ba4001c10 con 0x7f5bb0097fa0 2021-05-25T12:01:19.496 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.466+0000 7f5bb8093700 4 set_mon_vals no callback set 2021-05-25T12:01:19.496 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.466+0000 7f5bb4e2d700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:01:19.496 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.466+0000 7f5bb4e2d700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:01:19.496 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.466+0000 7f5bb4e2d700 10 monclient: dump: 2021-05-25T12:01:19.496 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:01:19.497 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:19.497 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:19.497 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:19.497 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:01:19.497 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:19.498 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:01:19.498 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:01:19.498 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:01:19.498 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:19.498 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.467+0000 7f5bb8093700 10 monclient: shutdown 2021-05-25T12:01:19.499 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.467+0000 7f5bb8093700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:01:19.499 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.467+0000 7f5bb8093700 1 -- 172.21.15.102:0/605142792 >> v1:172.21.15.102:6790/0 conn(0x7f5bb0097fa0 legacy=0x7f5bb0063460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:19.499 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.467+0000 7f5bb8093700 1 -- 172.21.15.102:0/605142792 shutdown_connections 2021-05-25T12:01:19.499 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.467+0000 7f5bb8093700 1 -- 172.21.15.102:0/605142792 >> 172.21.15.102:0/605142792 conn(0x7f5bb009ef50 msgr2=0x7f5bb009f360 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:01:19.500 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.467+0000 7f5bb8093700 1 -- 172.21.15.102:0/605142792 shutdown_connections 2021-05-25T12:01:19.500 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.467+0000 7f5bb8093700 1 -- 172.21.15.102:0/605142792 wait complete. 2021-05-25T12:01:19.500 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.467+0000 7f5bb8093700 10 monclient: build_initial_monmap 2021-05-25T12:01:19.500 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.467+0000 7f5bb8093700 10 monclient: monmap: 2021-05-25T12:01:19.500 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:01:19.501 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:01:19.501 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:01:19.501 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:01:19.501 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:01:19.501 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:01:19.502 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:01:19.502 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:01:19.502 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:01:19.502 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:19.502 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.468+0000 7f5bb8093700 1 librados: starting msgr at 2021-05-25T12:01:19.503 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.468+0000 7f5bb8093700 1 librados: starting objecter 2021-05-25T12:01:19.503 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.468+0000 7f5bb8093700 1 Processor -- start 2021-05-25T12:01:19.503 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.468+0000 7f5bb8093700 1 -- start start 2021-05-25T12:01:19.503 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.468+0000 7f5bb8093700 1 librados: setting wanted keys 2021-05-25T12:01:19.504 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.468+0000 7f5bb8093700 1 librados: calling monclient init 2021-05-25T12:01:19.504 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.468+0000 7f5bb8093700 10 monclient: init 2021-05-25T12:01:19.504 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.468+0000 7f5bb8093700 10 monclient: _reopen_session rank -1 2021-05-25T12:01:19.504 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.468+0000 7f5bb8093700 10 monclient: _add_conns ranks=[1,2,0] 2021-05-25T12:01:19.504 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.468+0000 7f5bb8093700 10 monclient(hunting): picked mon.noname-b con 0x7f5bb0063b70 addr v1:172.21.15.181:6789/0 2021-05-25T12:01:19.505 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.468+0000 7f5bb8093700 10 monclient(hunting): picked mon.noname-c con 0x7f5bb0097fa0 addr v1:172.21.15.102:6790/0 2021-05-25T12:01:19.505 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.468+0000 7f5bb8093700 10 monclient(hunting): picked mon.noname-a con 0x7f5bb01325c0 addr v1:172.21.15.102:6789/0 2021-05-25T12:01:19.505 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.468+0000 7f5bb8093700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bb009ab70 con 0x7f5bb01325c0 2021-05-25T12:01:19.505 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.468+0000 7f5bb8093700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bb012c100 con 0x7f5bb0063b70 2021-05-25T12:01:19.505 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.468+0000 7f5bb8093700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bb0116160 con 0x7f5bb0097fa0 2021-05-25T12:01:19.506 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.468+0000 7f5bb8093700 10 monclient(hunting): _renew_subs 2021-05-25T12:01:19.506 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.468+0000 7f5bb8093700 10 monclient(hunting): authenticate will time out at 880.065874s 2021-05-25T12:01:19.506 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.468+0000 7f5bb6630700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f5bb01325c0 0x7f5bb0143240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:59804/0 (socket says 172.21.15.102:59804) 2021-05-25T12:01:19.506 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.468+0000 7f5bb6630700 1 -- 172.21.15.102:0/3221854232 learned_addr learned my addr 172.21.15.102:0/3221854232 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:01:19.507 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.468+0000 7f5bb562e700 1 --1- 172.21.15.102:0/3221854232 >> v1:172.21.15.102:6790/0 conn(0x7f5bb0097fa0 0x7f5bb012b9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:54680/0 (socket says 172.21.15.102:54680) 2021-05-25T12:01:19.507 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.469+0000 7f5b9dffb700 10 client.?.objecter ms_handle_connect 0x7f5bb01325c0 2021-05-25T12:01:19.507 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.469+0000 7f5b9dffb700 10 client.?.objecter resend_mon_ops 2021-05-25T12:01:19.507 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.469+0000 7f5b9dffb700 10 client.?.objecter ms_handle_connect 0x7f5bb0097fa0 2021-05-25T12:01:19.508 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.469+0000 7f5b9dffb700 10 client.?.objecter resend_mon_ops 2021-05-25T12:01:19.508 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.469+0000 7f5b9dffb700 1 -- 172.21.15.102:0/3221854232 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3355992404 0 0) 0x7f5bb009ab70 con 0x7f5bb01325c0 2021-05-25T12:01:19.508 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.469+0000 7f5b9dffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:01:19.508 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.469+0000 7f5b9dffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:01:19.509 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.469+0000 7f5b9dffb700 10 monclient(hunting): my global_id is 87030 2021-05-25T12:01:19.509 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.469+0000 7f5b9dffb700 1 -- 172.21.15.102:0/3221854232 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b90004c10 con 0x7f5bb01325c0 2021-05-25T12:01:19.509 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.469+0000 7f5b9dffb700 1 -- 172.21.15.102:0/3221854232 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1171230204 0 0) 0x7f5bb0116160 con 0x7f5bb0097fa0 2021-05-25T12:01:19.509 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.469+0000 7f5b9dffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:01:19.509 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.469+0000 7f5b9dffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:01:19.510 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.469+0000 7f5b9dffb700 10 monclient(hunting): my global_id is 16154 2021-05-25T12:01:19.510 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.469+0000 7f5b9dffb700 1 -- 172.21.15.102:0/3221854232 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bb009ab70 con 0x7f5bb0097fa0 2021-05-25T12:01:19.510 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.469+0000 7f5b9dffb700 10 client.?.objecter ms_handle_connect 0x7f5bb0063b70 2021-05-25T12:01:19.510 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.469+0000 7f5b9dffb700 10 client.?.objecter resend_mon_ops 2021-05-25T12:01:19.510 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.469+0000 7f5b9dffb700 1 -- 172.21.15.102:0/3221854232 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 64339458 0 0) 0x7f5b90004c10 con 0x7f5bb01325c0 2021-05-25T12:01:19.511 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.469+0000 7f5b9dffb700 1 -- 172.21.15.102:0/3221854232 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5bb0116160 con 0x7f5bb01325c0 2021-05-25T12:01:19.511 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.469+0000 7f5b9dffb700 1 -- 172.21.15.102:0/3221854232 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f5bac001cd0 con 0x7f5bb01325c0 2021-05-25T12:01:19.511 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.469+0000 7f5b9dffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:01:19.511 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.469+0000 7f5b9dffb700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:01:19.512 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.469+0000 7f5b9dffb700 10 monclient(hunting): dump: 2021-05-25T12:01:19.512 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:01:19.512 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:19.512 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:19.512 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:19.513 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:01:19.513 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:19.513 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:01:19.513 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:01:19.513 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:01:19.514 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:19.514 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.469+0000 7f5b9dffb700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:01:19.514 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.469+0000 7f5b9dffb700 1 -- 172.21.15.102:0/3221854232 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3076795840 0 0) 0x7f5bb009ab70 con 0x7f5bb0097fa0 2021-05-25T12:01:19.514 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.469+0000 7f5b9dffb700 1 -- 172.21.15.102:0/3221854232 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5b90004c10 con 0x7f5bb0097fa0 2021-05-25T12:01:19.514 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.469+0000 7f5b9dffb700 1 -- 172.21.15.102:0/3221854232 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f5ba0002890 con 0x7f5bb0097fa0 2021-05-25T12:01:19.515 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.469+0000 7f5b9dffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:01:19.515 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.469+0000 7f5b9dffb700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2021-05-25T12:01:19.515 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.469+0000 7f5b9dffb700 10 monclient(hunting): dump: 2021-05-25T12:01:19.515 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:01:19.515 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:19.516 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:19.516 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:19.516 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:01:19.516 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:19.516 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:01:19.517 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:01:19.517 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:01:19.517 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:19.517 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.469+0000 7f5b9dffb700 1 -- 172.21.15.102:0/3221854232 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2336131671 0 0) 0x7f5bb012c100 con 0x7f5bb0063b70 2021-05-25T12:01:19.518 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.469+0000 7f5b9dffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:01:19.518 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.469+0000 7f5b9dffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:01:19.518 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.469+0000 7f5b9dffb700 10 monclient(hunting): my global_id is 84358 2021-05-25T12:01:19.518 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.469+0000 7f5b9dffb700 1 -- 172.21.15.102:0/3221854232 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bb009ab70 con 0x7f5bb0063b70 2021-05-25T12:01:19.518 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.469+0000 7f5b9dffb700 1 -- 172.21.15.102:0/3221854232 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3195888341 0 0) 0x7f5bb0116160 con 0x7f5bb01325c0 2021-05-25T12:01:19.519 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.469+0000 7f5b9dffb700 1 -- 172.21.15.102:0/3221854232 >> v1:172.21.15.102:6790/0 conn(0x7f5bb0097fa0 legacy=0x7f5bb012b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:19.519 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.469+0000 7f5b9dffb700 1 -- 172.21.15.102:0/3221854232 >> v1:172.21.15.181:6789/0 conn(0x7f5bb0063b70 legacy=0x7f5bb00924f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:19.519 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.469+0000 7f5b9dffb700 10 monclient: _finish_hunting 0 2021-05-25T12:01:19.519 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.469+0000 7f5b9dffb700 1 monclient: found mon.a 2021-05-25T12:01:19.519 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.469+0000 7f5b9dffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:01:19.520 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.469+0000 7f5b9dffb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:01:19.520 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.469+0000 7f5b9dffb700 1 -- 172.21.15.102:0/3221854232 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bb0116400 con 0x7f5bb01325c0 2021-05-25T12:01:19.520 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.469+0000 7f5b9dffb700 10 monclient: _finish_auth 0 2021-05-25T12:01:19.520 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.469+0000 7f5b9dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:01:19.521 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.469+0000 7f5bb8093700 5 monclient: authenticate success, global_id 87030 2021-05-25T12:01:19.521 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.469+0000 7f5bb8093700 10 monclient: _renew_subs 2021-05-25T12:01:19.521 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.469+0000 7f5bb8093700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:01:19.521 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.469+0000 7f5bb8093700 1 -- 172.21.15.102:0/3221854232 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bb00947f0 con 0x7f5bb01325c0 2021-05-25T12:01:19.521 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.469+0000 7f5b9dffb700 1 -- 172.21.15.102:0/3221854232 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5bac002000 con 0x7f5bb01325c0 2021-05-25T12:01:19.522 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.469+0000 7f5b9dffb700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:01:19.522 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.469+0000 7f5b9dffb700 1 -- 172.21.15.102:0/3221854232 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f5bac004d00 con 0x7f5bb01325c0 2021-05-25T12:01:19.522 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.469+0000 7f5b9dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:01:19.522 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.469+0000 7f5b9dffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:01:19.522 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.469+0000 7f5b9dffb700 10 monclient: dump: 2021-05-25T12:01:19.523 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:01:19.523 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:19.523 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:19.523 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:19.524 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:01:19.524 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:19.524 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:01:19.524 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:01:19.524 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:01:19.525 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:19.525 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.469+0000 7f5bb8093700 10 client.87030.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:01:19.525 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.469+0000 7f5b9ffff700 4 set_mon_vals no callback set 2021-05-25T12:01:19.525 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.469+0000 7f5bb8093700 10 monclient: _renew_subs 2021-05-25T12:01:19.525 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.469+0000 7f5bb8093700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:01:19.526 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.469+0000 7f5bb8093700 1 -- 172.21.15.102:0/3221854232 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bb00931c0 con 0x7f5bb01325c0 2021-05-25T12:01:19.526 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.469+0000 7f5bb8093700 1 librados: init done 2021-05-25T12:01:19.526 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.470+0000 7f5b9dffb700 1 -- 172.21.15.102:0/3221854232 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f5bac00e020 con 0x7f5bb01325c0 2021-05-25T12:01:19.526 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.470+0000 7f5bb8093700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:01:19.526 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.470+0000 7f5bb8093700 1 -- 172.21.15.102:0/3221854232 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f5bb0059730 con 0x7f5bb012fbd0 2021-05-25T12:01:19.527 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.470+0000 7f5b9dffb700 1 -- 172.21.15.102:0/3221854232 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(41..41 src has 1..41) v4 ==== 9241+0+0 (unknown 837274651 0 0) 0x7f5bac03e8b0 con 0x7f5bb01325c0 2021-05-25T12:01:19.527 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.470+0000 7f5b9dffb700 10 client.87030.objecter ms_dispatch 0x7f5bb0001550 osd_map(41..41 src has 1..41) v4 2021-05-25T12:01:19.527 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.470+0000 7f5b9dffb700 3 client.87030.objecter handle_osd_map got epochs [41,41] > 0 2021-05-25T12:01:19.527 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.470+0000 7f5b9dffb700 3 client.87030.objecter handle_osd_map decoding full epoch 41 2021-05-25T12:01:19.528 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.470+0000 7f5b9dffb700 20 client.87030.objecter dump_active .. 0 homeless 2021-05-25T12:01:19.528 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.470+0000 7f5bb5e2f700 10 monclient: get_auth_request con 0x7f5b90033400 auth_method 0 2021-05-25T12:01:19.528 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.470+0000 7f5b9dffb700 10 client.87030.objecter ms_handle_connect 0x7f5bb012fbd0 2021-05-25T12:01:19.528 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.470+0000 7f5b9dffb700 10 client.87030.objecter resend_mon_ops 2021-05-25T12:01:19.529 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.470+0000 7f5b9dffb700 1 -- 172.21.15.102:0/3221854232 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3568443576 0 0) 0x7f5bb0059730 con 0x7f5bb012fbd0 2021-05-25T12:01:19.529 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.470+0000 7f5b9dffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:01:19.529 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.470+0000 7f5b9dffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:01:19.529 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.470+0000 7f5b9dffb700 10 monclient(hunting): my global_id is 16157 2021-05-25T12:01:19.530 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.470+0000 7f5b9dffb700 1 -- 172.21.15.102:0/3221854232 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bb0116160 con 0x7f5bb012fbd0 2021-05-25T12:01:19.530 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.470+0000 7f5b9dffb700 1 -- 172.21.15.102:0/3221854232 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3039099655 0 0) 0x7f5bb0116160 con 0x7f5bb012fbd0 2021-05-25T12:01:19.530 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.470+0000 7f5b9dffb700 1 -- 172.21.15.102:0/3221854232 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5bb0059730 con 0x7f5bb012fbd0 2021-05-25T12:01:19.530 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.470+0000 7f5b9dffb700 10 client.87030.objecter ms_handle_connect 0x7f5b90033400 2021-05-25T12:01:19.531 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.470+0000 7f5b9dffb700 1 -- 172.21.15.102:0/3221854232 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f5ba00012b0 con 0x7f5bb012fbd0 2021-05-25T12:01:19.531 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.470+0000 7f5b9dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:01:19.531 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.470+0000 7f5b9dffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:01:19.531 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.470+0000 7f5b9dffb700 10 monclient: dump: 2021-05-25T12:01:19.532 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:01:19.532 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:19.532 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:19.532 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:19.532 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:01:19.533 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:19.533 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:01:19.533 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:01:19.533 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:01:19.534 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:19.534 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.471+0000 7f5b9dffb700 1 -- 172.21.15.102:0/3221854232 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1819577152 0 0) 0x7f5bb0059730 con 0x7f5bb012fbd0 2021-05-25T12:01:19.534 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.471+0000 7f5b9dffb700 1 -- 172.21.15.102:0/3221854232 --> v1:172.21.15.102:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5bb0115f50 con 0x7f5bb012fbd0 2021-05-25T12:01:19.534 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.471+0000 7f5b9dffb700 1 -- 172.21.15.102:0/3221854232 <== mon.2 v1:172.21.15.102:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7f5ba0001880 con 0x7f5bb012fbd0 2021-05-25T12:01:19.535 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.471+0000 7f5b9dffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:01:19.535 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.471+0000 7f5b9dffb700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:01:19.535 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.471+0000 7f5b9dffb700 1 -- 172.21.15.102:0/3221854232 >> v1:172.21.15.102:6790/0 conn(0x7f5bb012fbd0 legacy=0x7f5bb012ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:19.535 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.471+0000 7f5b9dffb700 1 -- 172.21.15.102:0/3221854232 >> v1:172.21.15.102:6790/0 conn(0x7f5bb012fbd0 legacy=0x7f5bb012ffb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:01:19.535 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.478+0000 7f5bb8093700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:01:19.536 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.479+0000 7f5bb8093700 1 -- 172.21.15.102:0/3221854232 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f5bb0095df0 con 0x7f5bb0027ce0 2021-05-25T12:01:19.536 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.479+0000 7f5b9dffb700 10 client.87030.objecter ms_handle_connect 0x7f5bb0027ce0 2021-05-25T12:01:19.536 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.479+0000 7f5b9dffb700 10 client.87030.objecter resend_mon_ops 2021-05-25T12:01:19.536 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.479+0000 7f5b9dffb700 1 -- 172.21.15.102:0/3221854232 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 217594263 0 0) 0x7f5bb0095df0 con 0x7f5bb0027ce0 2021-05-25T12:01:19.536 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.479+0000 7f5b9dffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:01:19.537 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.479+0000 7f5b9dffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:01:19.537 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.479+0000 7f5b9dffb700 10 monclient(hunting): my global_id is 16187 2021-05-25T12:01:19.537 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.479+0000 7f5b9dffb700 1 -- 172.21.15.102:0/3221854232 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bb0059730 con 0x7f5bb0027ce0 2021-05-25T12:01:19.537 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.479+0000 7f5b9dffb700 1 -- 172.21.15.102:0/3221854232 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2301089314 0 0) 0x7f5bb0059730 con 0x7f5bb0027ce0 2021-05-25T12:01:19.537 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.479+0000 7f5b9dffb700 1 -- 172.21.15.102:0/3221854232 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5bb0095df0 con 0x7f5bb0027ce0 2021-05-25T12:01:19.538 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.479+0000 7f5b9dffb700 1 -- 172.21.15.102:0/3221854232 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f5ba0001f70 con 0x7f5bb0027ce0 2021-05-25T12:01:19.538 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.479+0000 7f5b9dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:01:19.538 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.479+0000 7f5b9dffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:01:19.538 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.479+0000 7f5b9dffb700 10 monclient: dump: 2021-05-25T12:01:19.539 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:01:19.539 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:19.539 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:19.539 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:19.539 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:01:19.540 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:19.540 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:01:19.540 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:01:19.540 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:01:19.540 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:19.541 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.479+0000 7f5b9dffb700 1 -- 172.21.15.102:0/3221854232 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 816141427 0 0) 0x7f5bb0095df0 con 0x7f5bb0027ce0 2021-05-25T12:01:19.541 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.479+0000 7f5b9dffb700 1 -- 172.21.15.102:0/3221854232 --> v1:172.21.15.102:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f5bb006c980 con 0x7f5bb0027ce0 2021-05-25T12:01:19.541 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.479+0000 7f5b9dffb700 1 -- 172.21.15.102:0/3221854232 <== mon.2 v1:172.21.15.102:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4113 (unknown 0 0 4063513884) 0x7f5ba0005510 con 0x7f5bb0027ce0 2021-05-25T12:01:19.541 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.479+0000 7f5b9dffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:01:19.541 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.479+0000 7f5b9dffb700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:01:19.542 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.479+0000 7f5b9dffb700 1 -- 172.21.15.102:0/3221854232 >> v1:172.21.15.102:6790/0 conn(0x7f5bb0027ce0 legacy=0x7f5bb0148980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:19.542 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.479+0000 7f5b9dffb700 1 -- 172.21.15.102:0/3221854232 >> v1:172.21.15.102:6790/0 conn(0x7f5bb0027ce0 legacy=0x7f5bb0148980 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:01:19.542 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:01:19.543 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:01:19.543 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:01:19.543 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "leader", 2021-05-25T12:01:19.543 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 20, 2021-05-25T12:01:19.544 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:01:19.544 INFO:teuthology.orchestra.run.smithi102.stdout: 0, 2021-05-25T12:01:19.544 INFO:teuthology.orchestra.run.smithi102.stdout: 1, 2021-05-25T12:01:19.544 INFO:teuthology.orchestra.run.smithi102.stdout: 2 2021-05-25T12:01:19.544 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:01:19.545 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 3, 2021-05-25T12:01:19.545 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:01:19.545 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:01:19.545 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:01:19.545 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:01:19.546 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:01:19.546 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:01:19.546 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:01:19.546 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:01:19.546 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:01:19.547 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:01:19.547 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:01:19.547 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:01:19.547 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:01:19.548 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:01:19.548 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:01:19.548 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:01:19.548 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:01:19.548 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:01:19.549 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:01:19.549 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:01:19.549 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:01:19.549 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:01:19.550 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:01:19.550 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:01:19.550 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:01:19.550 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:01:19.551 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:01:19.551 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:01:19.551 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:01:19.551 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:01:19.552 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:01:19.552 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:01:19.552 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:01:19.552 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:01:19.552 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:01:19.553 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:01:19.553 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:01:19.553 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:01:19.553 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:01:19.554 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:01:19.554 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:01:19.554 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:01:19.554 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:01:19.554 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:01:19.555 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:01:19.555 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:01:19.555 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:01:19.555 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:01:19.555 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:01:19.556 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:01:19.556 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:01:19.556 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:01:19.556 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:01:19.556 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:01:19.557 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:01:19.557 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:01:19.557 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:01:19.557 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:01:19.557 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:01:19.558 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:01:19.558 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:01:19.558 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:01:19.558 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:01:19.559 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:01:19.559 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:01:19.559 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:01:19.559 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:01:19.559 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:01:19.560 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:01:19.560 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:01:19.560 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:01:19.560 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:01:19.560 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:01:19.561 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:01:19.561 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:01:19.561 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:01:19.561 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:01:19.561 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:01:19.562 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:01:19.562 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:01:19.562 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:01:19.562 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:01:19.562 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:01:19.563 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:01:19.563 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:01:19.563 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:01:19.563 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:01:19.563 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:01:19.564 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:01:19.564 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:01:19.564 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:01:19.564 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:01:19.564 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:01:19.565 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:01:19.565 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:01:19.565 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:01:19.565 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:01:19.565 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:01:19.566 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:01:19.566 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:01:19.566 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:01:19.566 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:01:19.566 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:01:19.567 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:01:19.567 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:01:19.567 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:01:19.567 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:01:19.568 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:01:19.568 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:01:19.568 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:01:19.568 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:01:19.568 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:01:19.569 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:01:19.569 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:01:19.569 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:01:19.569 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:01:19.569 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:01:19.570 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:01:19.570 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": [ 2021-05-25T12:01:19.570 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:01:19.570 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:01:19.570 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:01:19.571 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 5 2021-05-25T12:01:19.571 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:01:19.571 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:01:19.571 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:01:19.572 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:01:19.572 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:01:19.572 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:01:19.572 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 25 2021-05-25T12:01:19.573 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:01:19.573 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:01:19.573 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": [ 2021-05-25T12:01:19.573 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:01:19.573 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:01:19.574 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:01:19.574 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:01:19.574 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:01:19.574 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:01:19.575 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:01:19.575 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:01:19.575 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:01:19.577 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.480+0000 7f5bb8093700 10 librados: watch_flush enter 2021-05-25T12:01:19.577 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.480+0000 7f5bb8093700 10 librados: watch_flush exit 2021-05-25T12:01:19.577 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.480+0000 7f5bb8093700 20 client.87030.objecter shutdown clearing up homeless session... 2021-05-25T12:01:19.578 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.480+0000 7f5bb8093700 10 client.87030.objecter successfully canceled tick 2021-05-25T12:01:19.578 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.480+0000 7f5bb8093700 1 -- 172.21.15.102:0/3221854232 >> v1:172.21.15.181:6812/29582 conn(0x7f5b90033400 legacy=0x7f5b900358c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:19.578 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.480+0000 7f5bb8093700 10 monclient: shutdown 2021-05-25T12:01:19.578 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.480+0000 7f5bb8093700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:01:19.579 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.480+0000 7f5bb8093700 1 -- 172.21.15.102:0/3221854232 >> v1:172.21.15.102:6789/0 conn(0x7f5bb01325c0 legacy=0x7f5bb0143240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:19.579 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.480+0000 7f5bb6630700 1 -- 172.21.15.102:0/3221854232 reap_dead start 2021-05-25T12:01:19.579 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.480+0000 7f5bb6630700 1 -- 172.21.15.102:0/3221854232 reap_dead start 2021-05-25T12:01:19.579 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.480+0000 7f5bb8093700 1 -- 172.21.15.102:0/3221854232 shutdown_connections 2021-05-25T12:01:19.579 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.480+0000 7f5bb8093700 1 -- 172.21.15.102:0/3221854232 >> 172.21.15.102:0/3221854232 conn(0x7f5bb009ef50 msgr2=0x7f5bb009db10 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:01:19.580 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.480+0000 7f5bb8093700 1 -- 172.21.15.102:0/3221854232 shutdown_connections 2021-05-25T12:01:19.580 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.480+0000 7f5bb8093700 1 -- 172.21.15.102:0/3221854232 wait complete. 2021-05-25T12:01:19.580 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.480+0000 7f5bb8093700 1 librados: shutdown 2021-05-25T12:01:19.580 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2021-05-25T12:01:19.699 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.697+0000 7f6e3a2fb700 10 monclient: get_monmap_and_config 2021-05-25T12:01:19.699 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.697+0000 7f6e3a2fb700 10 monclient: build_initial_monmap 2021-05-25T12:01:19.699 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.697+0000 7f6e3a2fb700 10 monclient: monmap: 2021-05-25T12:01:19.700 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:01:19.700 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:19.700 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:01:19.698655+0000 2021-05-25T12:01:19.700 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:01:19.698655+0000 2021-05-25T12:01:19.701 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:01:19.701 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:19.701 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:01:19.701 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:01:19.702 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:01:19.702 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:19.713 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.698+0000 7f6e3a2fb700 1 Processor -- start 2021-05-25T12:01:19.713 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.698+0000 7f6e3a2fb700 1 -- start start 2021-05-25T12:01:19.713 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.698+0000 7f6e3a2fb700 10 monclient: init 2021-05-25T12:01:19.713 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.698+0000 7f6e3a2fb700 10 monclient: _reopen_session rank -1 2021-05-25T12:01:19.713 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.698+0000 7f6e3a2fb700 10 monclient: _add_conns ranks=[0,2,1] 2021-05-25T12:01:19.714 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.698+0000 7f6e3a2fb700 10 monclient(hunting): picked mon.noname-a con 0x7f6e34095c40 addr v1:172.21.15.102:6789/0 2021-05-25T12:01:19.714 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.698+0000 7f6e3a2fb700 10 monclient(hunting): picked mon.noname-b con 0x7f6e34096020 addr v1:172.21.15.102:6790/0 2021-05-25T12:01:19.714 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.698+0000 7f6e3a2fb700 10 monclient(hunting): picked mon.noname-c con 0x7f6e3412e5c0 addr v1:172.21.15.181:6789/0 2021-05-25T12:01:19.714 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.698+0000 7f6e3a2fb700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e34092990 con 0x7f6e34095c40 2021-05-25T12:01:19.715 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.698+0000 7f6e3a2fb700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e34092c30 con 0x7f6e3412e5c0 2021-05-25T12:01:19.715 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.698+0000 7f6e3a2fb700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e34092ed0 con 0x7f6e34096020 2021-05-25T12:01:19.715 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.699+0000 7f6e3a2fb700 10 monclient(hunting): _renew_subs 2021-05-25T12:01:19.715 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.699+0000 7f6e3a2fb700 10 monclient(hunting): authenticate will time out at 880.296010s 2021-05-25T12:01:19.715 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.699+0000 7f6e392f9700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f6e34095c40 0x7f6e34093290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:59812/0 (socket says 172.21.15.102:59812) 2021-05-25T12:01:19.716 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.699+0000 7f6e392f9700 1 -- 172.21.15.102:0/2061237552 learned_addr learned my addr 172.21.15.102:0/2061237552 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:01:19.716 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.699+0000 7f6e38af8700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f6e34096020 0x7f6e340964d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:54690/0 (socket says 172.21.15.102:54690) 2021-05-25T12:01:19.716 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.699+0000 7f6e23fff700 1 -- 172.21.15.102:0/2061237552 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3879039073 0 0) 0x7f6e34092990 con 0x7f6e34095c40 2021-05-25T12:01:19.716 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.699+0000 7f6e23fff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:01:19.716 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.699+0000 7f6e23fff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:01:19.717 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.699+0000 7f6e23fff700 10 monclient(hunting): my global_id is 87243 2021-05-25T12:01:19.717 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.699+0000 7f6e23fff700 1 -- 172.21.15.102:0/2061237552 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e1c004380 con 0x7f6e34095c40 2021-05-25T12:01:19.717 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.699+0000 7f6e23fff700 1 -- 172.21.15.102:0/2061237552 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1470002230 0 0) 0x7f6e34092ed0 con 0x7f6e34096020 2021-05-25T12:01:19.717 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.699+0000 7f6e23fff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:01:19.717 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.699+0000 7f6e23fff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:01:19.718 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.699+0000 7f6e23fff700 10 monclient(hunting): my global_id is 16376 2021-05-25T12:01:19.718 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.699+0000 7f6e23fff700 1 -- 172.21.15.102:0/2061237552 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e34092990 con 0x7f6e34096020 2021-05-25T12:01:19.718 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.699+0000 7f6e23fff700 1 -- 172.21.15.102:0/2061237552 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1648967314 0 0) 0x7f6e34092990 con 0x7f6e34096020 2021-05-25T12:01:19.718 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.699+0000 7f6e23fff700 1 -- 172.21.15.102:0/2061237552 >> v1:172.21.15.181:6789/0 conn(0x7f6e3412e5c0 legacy=0x7f6e341309b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-05-25T12:01:19.719 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.699+0000 7f6e23fff700 1 -- 172.21.15.102:0/2061237552 >> v1:172.21.15.102:6789/0 conn(0x7f6e34095c40 legacy=0x7f6e34093290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:19.719 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.699+0000 7f6e23fff700 10 monclient: _finish_hunting 0 2021-05-25T12:01:19.719 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.699+0000 7f6e23fff700 1 monclient: found mon.noname-b 2021-05-25T12:01:19.719 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.699+0000 7f6e23fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:01:19.719 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.699+0000 7f6e23fff700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T12:01:19.720 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.699+0000 7f6e23fff700 1 -- 172.21.15.102:0/2061237552 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e34092430 con 0x7f6e34096020 2021-05-25T12:01:19.720 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.699+0000 7f6e23fff700 10 monclient: _finish_auth 0 2021-05-25T12:01:19.720 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.699+0000 7f6e23fff700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:01:19.720 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.699+0000 7f6e3a2fb700 5 monclient: authenticate success, global_id 16376 2021-05-25T12:01:19.721 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.699+0000 7f6e23fff700 1 -- 172.21.15.102:0/2061237552 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f6e28001610 con 0x7f6e34096020 2021-05-25T12:01:19.721 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.699+0000 7f6e3a2fb700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:01:19.721 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.699+0000 7f6e23fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:01:19.721 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.699+0000 7f6e23fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:01:19.722 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.699+0000 7f6e23fff700 10 monclient: dump: 2021-05-25T12:01:19.722 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:01:19.722 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:19.722 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:19.723 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:19.723 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:01:19.723 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:19.723 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:01:19.723 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:01:19.724 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:01:19.724 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:19.724 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.699+0000 7f6e3a2fb700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:01:19.724 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.699+0000 7f6e23fff700 1 -- 172.21.15.102:0/2061237552 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6e28001800 con 0x7f6e34096020 2021-05-25T12:01:19.724 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.699+0000 7f6e23fff700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:01:19.725 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.699+0000 7f6e3a2fb700 10 monclient: get_monmap_and_config success 2021-05-25T12:01:19.725 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.699+0000 7f6e23fff700 1 -- 172.21.15.102:0/2061237552 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f6e28001b90 con 0x7f6e34096020 2021-05-25T12:01:19.725 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.699+0000 7f6e23fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:01:19.725 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.699+0000 7f6e3a2fb700 4 set_mon_vals no callback set 2021-05-25T12:01:19.726 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.699+0000 7f6e23fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:01:19.726 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.699+0000 7f6e23fff700 10 monclient: dump: 2021-05-25T12:01:19.726 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:01:19.726 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:19.726 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:19.727 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:19.727 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:01:19.727 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:19.727 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:01:19.727 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:01:19.728 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:01:19.728 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:19.728 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.700+0000 7f6e3a2fb700 10 monclient: shutdown 2021-05-25T12:01:19.728 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.700+0000 7f6e3a2fb700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:01:19.728 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.700+0000 7f6e3a2fb700 1 -- 172.21.15.102:0/2061237552 >> v1:172.21.15.102:6790/0 conn(0x7f6e34096020 legacy=0x7f6e340964d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:19.729 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.700+0000 7f6e3a2fb700 1 -- 172.21.15.102:0/2061237552 shutdown_connections 2021-05-25T12:01:19.729 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.700+0000 7f6e3a2fb700 1 -- 172.21.15.102:0/2061237552 >> 172.21.15.102:0/2061237552 conn(0x7f6e340974b0 msgr2=0x7f6e340978c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:01:19.729 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.700+0000 7f6e3a2fb700 1 -- 172.21.15.102:0/2061237552 shutdown_connections 2021-05-25T12:01:19.729 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.700+0000 7f6e3a2fb700 1 -- 172.21.15.102:0/2061237552 wait complete. 2021-05-25T12:01:19.729 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.700+0000 7f6e3a2fb700 10 monclient: build_initial_monmap 2021-05-25T12:01:19.730 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.700+0000 7f6e3a2fb700 10 monclient: monmap: 2021-05-25T12:01:19.730 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:01:19.730 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:01:19.730 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:01:19.731 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:01:19.731 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:01:19.731 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:01:19.731 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:01:19.731 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:01:19.732 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:01:19.732 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:19.732 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.701+0000 7f6e3a2fb700 1 librados: starting msgr at 2021-05-25T12:01:19.732 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.701+0000 7f6e3a2fb700 1 librados: starting objecter 2021-05-25T12:01:19.733 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.701+0000 7f6e3a2fb700 1 Processor -- start 2021-05-25T12:01:19.733 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.701+0000 7f6e3a2fb700 1 -- start start 2021-05-25T12:01:19.733 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.701+0000 7f6e3a2fb700 1 librados: setting wanted keys 2021-05-25T12:01:19.733 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.701+0000 7f6e3a2fb700 1 librados: calling monclient init 2021-05-25T12:01:19.733 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.701+0000 7f6e3a2fb700 10 monclient: init 2021-05-25T12:01:19.734 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.701+0000 7f6e3a2fb700 10 monclient: _reopen_session rank -1 2021-05-25T12:01:19.734 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.701+0000 7f6e3a2fb700 10 monclient: _add_conns ranks=[1,2,0] 2021-05-25T12:01:19.734 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.701+0000 7f6e3a2fb700 10 monclient(hunting): picked mon.noname-b con 0x7f6e34095c40 addr v1:172.21.15.181:6789/0 2021-05-25T12:01:19.734 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.701+0000 7f6e3a2fb700 10 monclient(hunting): picked mon.noname-c con 0x7f6e34096020 addr v1:172.21.15.102:6790/0 2021-05-25T12:01:19.734 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.701+0000 7f6e3a2fb700 10 monclient(hunting): picked mon.noname-a con 0x7f6e3412e5c0 addr v1:172.21.15.102:6789/0 2021-05-25T12:01:19.735 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.701+0000 7f6e3a2fb700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e34129350 con 0x7f6e3412e5c0 2021-05-25T12:01:19.735 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.701+0000 7f6e3a2fb700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e34064c40 con 0x7f6e34095c40 2021-05-25T12:01:19.735 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.701+0000 7f6e38af8700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f6e34096020 0x7f6e34064530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:54698/0 (socket says 172.21.15.102:54698) 2021-05-25T12:01:19.735 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.702+0000 7f6e38af8700 1 -- 172.21.15.102:0/3048552851 learned_addr learned my addr 172.21.15.102:0/3048552851 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:01:19.736 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.702+0000 7f6e3a2fb700 1 -- 172.21.15.102:0/3048552851 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e34067260 con 0x7f6e34096020 2021-05-25T12:01:19.736 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.702+0000 7f6e3a2fb700 10 monclient(hunting): _renew_subs 2021-05-25T12:01:19.736 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.702+0000 7f6e3a2fb700 10 monclient(hunting): authenticate will time out at 880.299034s 2021-05-25T12:01:19.736 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.702+0000 7f6e217fa700 10 client.?.objecter ms_handle_connect 0x7f6e34096020 2021-05-25T12:01:19.736 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.702+0000 7f6e217fa700 10 client.?.objecter resend_mon_ops 2021-05-25T12:01:19.736 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.702+0000 7f6e217fa700 1 -- 172.21.15.102:0/3048552851 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1933387029 0 0) 0x7f6e34067260 con 0x7f6e34096020 2021-05-25T12:01:19.737 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.702+0000 7f6e217fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:01:19.737 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.702+0000 7f6e217fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:01:19.737 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.702+0000 7f6e217fa700 10 monclient(hunting): my global_id is 16385 2021-05-25T12:01:19.737 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.702+0000 7f6e217fa700 1 -- 172.21.15.102:0/3048552851 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e14004c10 con 0x7f6e34096020 2021-05-25T12:01:19.738 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.702+0000 7f6e217fa700 10 client.?.objecter ms_handle_connect 0x7f6e3412e5c0 2021-05-25T12:01:19.738 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.702+0000 7f6e217fa700 10 client.?.objecter resend_mon_ops 2021-05-25T12:01:19.738 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.702+0000 7f6e217fa700 1 -- 172.21.15.102:0/3048552851 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3192388171 0 0) 0x7f6e34129350 con 0x7f6e3412e5c0 2021-05-25T12:01:19.738 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.702+0000 7f6e217fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:01:19.738 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.702+0000 7f6e217fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:01:19.739 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.702+0000 7f6e217fa700 10 monclient(hunting): my global_id is 87255 2021-05-25T12:01:19.739 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.702+0000 7f6e217fa700 1 -- 172.21.15.102:0/3048552851 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e34067260 con 0x7f6e3412e5c0 2021-05-25T12:01:19.739 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.702+0000 7f6e217fa700 1 -- 172.21.15.102:0/3048552851 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1102636972 0 0) 0x7f6e14004c10 con 0x7f6e34096020 2021-05-25T12:01:19.739 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.702+0000 7f6e217fa700 1 -- 172.21.15.102:0/3048552851 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e34129350 con 0x7f6e34096020 2021-05-25T12:01:19.739 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.702+0000 7f6e217fa700 1 -- 172.21.15.102:0/3048552851 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f6e280015d0 con 0x7f6e34096020 2021-05-25T12:01:19.740 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.702+0000 7f6e217fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:01:19.740 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.702+0000 7f6e217fa700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:01:19.740 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.702+0000 7f6e217fa700 10 monclient(hunting): dump: 2021-05-25T12:01:19.740 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:01:19.740 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:19.741 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:19.741 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:19.741 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:01:19.741 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:19.741 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:01:19.742 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:01:19.742 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:01:19.742 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:19.743 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.702+0000 7f6e217fa700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:01:19.743 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.702+0000 7f6e217fa700 1 -- 172.21.15.102:0/3048552851 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4116244342 0 0) 0x7f6e34067260 con 0x7f6e3412e5c0 2021-05-25T12:01:19.743 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.702+0000 7f6e217fa700 1 -- 172.21.15.102:0/3048552851 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e14004c10 con 0x7f6e3412e5c0 2021-05-25T12:01:19.743 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.702+0000 7f6e217fa700 1 -- 172.21.15.102:0/3048552851 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f6e30001af0 con 0x7f6e3412e5c0 2021-05-25T12:01:19.744 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.702+0000 7f6e217fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:01:19.744 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.702+0000 7f6e217fa700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2021-05-25T12:01:19.744 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.702+0000 7f6e217fa700 10 monclient(hunting): dump: 2021-05-25T12:01:19.744 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:01:19.744 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:19.745 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:19.745 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:19.745 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:01:19.745 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:19.745 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:01:19.746 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:01:19.746 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:01:19.746 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:19.746 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.702+0000 7f6e217fa700 1 -- 172.21.15.102:0/3048552851 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4236866160 0 0) 0x7f6e34129350 con 0x7f6e34096020 2021-05-25T12:01:19.746 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.702+0000 7f6e217fa700 1 -- 172.21.15.102:0/3048552851 >> v1:172.21.15.181:6789/0 conn(0x7f6e34095c40 legacy=0x7f6e34063e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:19.747 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.702+0000 7f6e217fa700 1 -- 172.21.15.102:0/3048552851 >> v1:172.21.15.102:6789/0 conn(0x7f6e3412e5c0 legacy=0x7f6e34141380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:19.747 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.702+0000 7f6e217fa700 10 monclient: _finish_hunting 0 2021-05-25T12:01:19.747 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.702+0000 7f6e217fa700 1 monclient: found mon.c 2021-05-25T12:01:19.747 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.702+0000 7f6e217fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:01:19.747 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.702+0000 7f6e217fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:01:19.748 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.702+0000 7f6e217fa700 1 -- 172.21.15.102:0/3048552851 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e34067500 con 0x7f6e34096020 2021-05-25T12:01:19.748 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.702+0000 7f6e217fa700 10 monclient: _finish_auth 0 2021-05-25T12:01:19.748 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.702+0000 7f6e217fa700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:01:19.748 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.702+0000 7f6e217fa700 10 client.?.objecter ms_handle_connect 0x7f6e34095c40 2021-05-25T12:01:19.749 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.702+0000 7f6e217fa700 10 client.?.objecter resend_mon_ops 2021-05-25T12:01:19.749 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.702+0000 7f6e3a2fb700 5 monclient: authenticate success, global_id 16385 2021-05-25T12:01:19.749 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.702+0000 7f6e3a2fb700 10 monclient: _renew_subs 2021-05-25T12:01:19.749 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.702+0000 7f6e3a2fb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:01:19.749 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.702+0000 7f6e3a2fb700 1 -- 172.21.15.102:0/3048552851 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e34098d80 con 0x7f6e34096020 2021-05-25T12:01:19.750 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.702+0000 7f6e3a2fb700 10 client.16385.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:01:19.750 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.702+0000 7f6e3a2fb700 10 monclient: _renew_subs 2021-05-25T12:01:19.750 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.702+0000 7f6e3a2fb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:01:19.750 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.702+0000 7f6e3a2fb700 1 -- 172.21.15.102:0/3048552851 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e340678c0 con 0x7f6e34096020 2021-05-25T12:01:19.750 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.702+0000 7f6e3a2fb700 1 librados: init done 2021-05-25T12:01:19.751 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.702+0000 7f6e217fa700 1 -- 172.21.15.102:0/3048552851 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6e28004970 con 0x7f6e34096020 2021-05-25T12:01:19.751 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.702+0000 7f6e217fa700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:01:19.751 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.702+0000 7f6e217fa700 1 -- 172.21.15.102:0/3048552851 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f6e28004d00 con 0x7f6e34096020 2021-05-25T12:01:19.751 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.702+0000 7f6e217fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:01:19.752 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.702+0000 7f6e217fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:01:19.752 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.702+0000 7f6e237fe700 4 set_mon_vals no callback set 2021-05-25T12:01:19.752 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.702+0000 7f6e217fa700 10 monclient: dump: 2021-05-25T12:01:19.752 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:01:19.752 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:19.753 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:19.753 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:19.753 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:01:19.753 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:19.753 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:01:19.754 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:01:19.754 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:01:19.754 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:19.754 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.703+0000 7f6e217fa700 1 -- 172.21.15.102:0/3048552851 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f6e28004ee0 con 0x7f6e34096020 2021-05-25T12:01:19.755 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.703+0000 7f6e3a2fb700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:01:19.755 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.703+0000 7f6e3a2fb700 1 -- 172.21.15.102:0/3048552851 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f6e0c0065a0 con 0x7f6e0c002840 2021-05-25T12:01:19.755 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.703+0000 7f6e217fa700 1 -- 172.21.15.102:0/3048552851 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(41..41 src has 1..41) v4 ==== 9241+0+0 (unknown 837274651 0 0) 0x7f6e2803e8a0 con 0x7f6e34096020 2021-05-25T12:01:19.755 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.703+0000 7f6e217fa700 10 client.16385.objecter ms_dispatch 0x7f6e34001550 osd_map(41..41 src has 1..41) v4 2021-05-25T12:01:19.755 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.703+0000 7f6e217fa700 3 client.16385.objecter handle_osd_map got epochs [41,41] > 0 2021-05-25T12:01:19.756 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.703+0000 7f6e217fa700 3 client.16385.objecter handle_osd_map decoding full epoch 41 2021-05-25T12:01:19.756 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.703+0000 7f6e217fa700 20 client.16385.objecter dump_active .. 0 homeless 2021-05-25T12:01:19.756 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.703+0000 7f6e217fa700 10 client.16385.objecter ms_handle_connect 0x7f6e0c002840 2021-05-25T12:01:19.756 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.703+0000 7f6e217fa700 10 client.16385.objecter resend_mon_ops 2021-05-25T12:01:19.756 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.704+0000 7f6e217fa700 1 -- 172.21.15.102:0/3048552851 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 489042601 0 0) 0x7f6e0c0065a0 con 0x7f6e0c002840 2021-05-25T12:01:19.757 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.704+0000 7f6e39afa700 10 monclient: get_auth_request con 0x7f6e140332a0 auth_method 0 2021-05-25T12:01:19.757 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.704+0000 7f6e217fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:01:19.757 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.704+0000 7f6e217fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:01:19.757 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.704+0000 7f6e217fa700 10 monclient(hunting): my global_id is 84520 2021-05-25T12:01:19.758 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.704+0000 7f6e217fa700 1 -- 172.21.15.102:0/3048552851 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e34129350 con 0x7f6e0c002840 2021-05-25T12:01:19.758 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.705+0000 7f6e217fa700 1 -- 172.21.15.102:0/3048552851 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 459914727 0 0) 0x7f6e34129350 con 0x7f6e0c002840 2021-05-25T12:01:19.758 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.705+0000 7f6e217fa700 1 -- 172.21.15.102:0/3048552851 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e0c0065a0 con 0x7f6e0c002840 2021-05-25T12:01:19.758 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.705+0000 7f6e217fa700 1 -- 172.21.15.102:0/3048552851 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f6e24001850 con 0x7f6e0c002840 2021-05-25T12:01:19.758 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.705+0000 7f6e217fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:01:19.759 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.705+0000 7f6e217fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T12:01:19.759 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.705+0000 7f6e217fa700 10 monclient: dump: 2021-05-25T12:01:19.759 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:01:19.759 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:19.759 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:19.760 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:19.760 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:01:19.760 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:19.760 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:01:19.760 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:01:19.761 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:01:19.761 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:19.761 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.706+0000 7f6e217fa700 1 -- 172.21.15.102:0/3048552851 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4043648471 0 0) 0x7f6e0c0065a0 con 0x7f6e0c002840 2021-05-25T12:01:19.761 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.706+0000 7f6e217fa700 1 -- 172.21.15.102:0/3048552851 --> v1:172.21.15.181:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6e0c009e40 con 0x7f6e0c002840 2021-05-25T12:01:19.761 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.706+0000 7f6e217fa700 1 -- 172.21.15.102:0/3048552851 <== mon.1 v1:172.21.15.181:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7f6e24001ad0 con 0x7f6e0c002840 2021-05-25T12:01:19.762 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.706+0000 7f6e217fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:01:19.762 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.706+0000 7f6e217fa700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:01:19.762 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.706+0000 7f6e217fa700 1 -- 172.21.15.102:0/3048552851 >> v1:172.21.15.181:6789/0 conn(0x7f6e0c002840 legacy=0x7f6e0c004ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:19.762 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.706+0000 7f6e217fa700 1 -- 172.21.15.102:0/3048552851 >> v1:172.21.15.181:6789/0 conn(0x7f6e0c002840 legacy=0x7f6e0c004ce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:01:19.763 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.708+0000 7f6e217fa700 10 client.16385.objecter ms_handle_connect 0x7f6e140332a0 2021-05-25T12:01:19.765 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.713+0000 7f6e3a2fb700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:01:19.766 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.713+0000 7f6e3a2fb700 1 -- 172.21.15.102:0/3048552851 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f6e0c009ad0 con 0x7f6e0c008d90 2021-05-25T12:01:19.766 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.713+0000 7f6e217fa700 10 client.16385.objecter ms_handle_connect 0x7f6e0c008d90 2021-05-25T12:01:19.766 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.713+0000 7f6e217fa700 10 client.16385.objecter resend_mon_ops 2021-05-25T12:01:19.766 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.714+0000 7f6e217fa700 1 -- 172.21.15.102:0/3048552851 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1335448192 0 0) 0x7f6e0c009ad0 con 0x7f6e0c008d90 2021-05-25T12:01:19.767 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.714+0000 7f6e217fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:01:19.767 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.714+0000 7f6e217fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:01:19.767 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.714+0000 7f6e217fa700 10 monclient(hunting): my global_id is 84532 2021-05-25T12:01:19.767 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.714+0000 7f6e217fa700 1 -- 172.21.15.102:0/3048552851 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e0c0065a0 con 0x7f6e0c008d90 2021-05-25T12:01:19.767 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.714+0000 7f6e217fa700 1 -- 172.21.15.102:0/3048552851 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1793741620 0 0) 0x7f6e0c0065a0 con 0x7f6e0c008d90 2021-05-25T12:01:19.768 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.714+0000 7f6e217fa700 1 -- 172.21.15.102:0/3048552851 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e0c009ad0 con 0x7f6e0c008d90 2021-05-25T12:01:19.768 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.714+0000 7f6e217fa700 1 -- 172.21.15.102:0/3048552851 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f6e24001da0 con 0x7f6e0c008d90 2021-05-25T12:01:19.768 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.714+0000 7f6e217fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:01:19.768 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.714+0000 7f6e217fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T12:01:19.768 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.714+0000 7f6e217fa700 10 monclient: dump: 2021-05-25T12:01:19.769 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:01:19.769 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:19.769 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:19.769 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:19.770 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:01:19.770 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:19.770 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:01:19.770 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:01:19.770 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:01:19.771 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:19.771 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.714+0000 7f6e217fa700 1 -- 172.21.15.102:0/3048552851 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3740670770 0 0) 0x7f6e0c009ad0 con 0x7f6e0c008d90 2021-05-25T12:01:19.771 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.714+0000 7f6e217fa700 1 -- 172.21.15.102:0/3048552851 --> v1:172.21.15.181:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f6e0c008740 con 0x7f6e0c008d90 2021-05-25T12:01:19.771 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.722+0000 7f6e217fa700 1 -- 172.21.15.102:0/3048552851 <== mon.1 v1:172.21.15.181:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3941 (unknown 0 0 252649743) 0x7f6e24004a00 con 0x7f6e0c008d90 2021-05-25T12:01:19.771 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.722+0000 7f6e217fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:01:19.772 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.722+0000 7f6e217fa700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:01:19.772 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.722+0000 7f6e217fa700 1 -- 172.21.15.102:0/3048552851 >> v1:172.21.15.181:6789/0 conn(0x7f6e0c008d90 legacy=0x7f6e0c009220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:19.772 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.722+0000 7f6e217fa700 1 -- 172.21.15.102:0/3048552851 >> v1:172.21.15.181:6789/0 conn(0x7f6e0c008d90 legacy=0x7f6e0c009220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:01:19.772 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:01:19.773 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:01:19.773 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:01:19.773 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "peon", 2021-05-25T12:01:19.773 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 20, 2021-05-25T12:01:19.773 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:01:19.774 INFO:teuthology.orchestra.run.smithi102.stdout: 0, 2021-05-25T12:01:19.774 INFO:teuthology.orchestra.run.smithi102.stdout: 1, 2021-05-25T12:01:19.774 INFO:teuthology.orchestra.run.smithi102.stdout: 2 2021-05-25T12:01:19.774 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:01:19.774 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 3, 2021-05-25T12:01:19.775 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:01:19.775 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:01:19.775 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:01:19.775 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:01:19.775 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:01:19.776 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:01:19.776 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:01:19.776 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:01:19.776 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:01:19.776 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:01:19.777 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:01:19.777 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:01:19.777 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:01:19.777 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:01:19.777 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:01:19.778 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:01:19.778 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:01:19.778 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:01:19.778 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:01:19.778 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:01:19.779 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:01:19.779 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:01:19.779 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:01:19.779 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:01:19.779 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:01:19.780 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:01:19.780 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:01:19.780 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:01:19.780 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:01:19.780 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:01:19.781 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:01:19.781 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:01:19.781 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:01:19.781 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:01:19.781 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:01:19.782 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:01:19.782 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:01:19.782 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:01:19.782 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:01:19.782 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:01:19.783 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:01:19.783 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:01:19.783 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:01:19.783 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:01:19.784 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:01:19.784 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:01:19.784 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:01:19.784 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:01:19.785 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:01:19.785 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:01:19.785 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:01:19.785 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:01:19.786 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:01:19.786 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:01:19.786 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:01:19.786 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:01:19.786 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:01:19.787 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:01:19.787 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:01:19.787 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:01:19.787 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:01:19.787 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:01:19.788 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:01:19.788 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:01:19.788 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:01:19.788 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:01:19.789 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:01:19.789 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:01:19.789 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:01:19.789 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:01:19.789 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:01:19.790 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:01:19.790 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:01:19.790 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:01:19.790 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:01:19.790 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:01:19.791 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:01:19.791 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:01:19.791 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:01:19.791 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:01:19.791 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:01:19.792 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:01:19.792 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:01:19.792 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:01:19.792 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:01:19.793 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:01:19.793 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:01:19.793 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:01:19.793 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:01:19.793 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:01:19.794 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:01:19.794 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:01:19.794 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:01:19.794 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:01:19.794 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:01:19.795 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:01:19.795 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:01:19.795 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:01:19.795 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:01:19.796 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:01:19.796 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:01:19.796 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:01:19.796 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:01:19.796 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:01:19.797 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:01:19.797 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:01:19.797 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:01:19.797 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:01:19.797 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:01:19.798 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:01:19.798 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:01:19.798 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:01:19.798 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:01:19.798 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:01:19.799 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:01:19.799 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:01:19.799 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:01:19.799 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:01:19.799 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": [ 2021-05-25T12:01:19.800 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:01:19.800 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:01:19.800 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:01:19.800 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:01:19.800 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:01:19.801 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:01:19.801 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:01:19.801 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:01:19.801 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:01:19.801 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:01:19.802 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 18 2021-05-25T12:01:19.802 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:01:19.802 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:01:19.802 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:01:19.803 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:01:19.803 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:01:19.805 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.722+0000 7f6e3a2fb700 10 librados: watch_flush enter 2021-05-25T12:01:19.805 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.722+0000 7f6e3a2fb700 10 librados: watch_flush exit 2021-05-25T12:01:19.805 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.722+0000 7f6e3a2fb700 20 client.16385.objecter shutdown clearing up homeless session... 2021-05-25T12:01:19.805 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.722+0000 7f6e3a2fb700 10 client.16385.objecter successfully canceled tick 2021-05-25T12:01:19.806 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.722+0000 7f6e3a2fb700 1 -- 172.21.15.102:0/3048552851 >> v1:172.21.15.181:6812/29582 conn(0x7f6e140332a0 legacy=0x7f6e14035760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:19.806 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.722+0000 7f6e3a2fb700 10 monclient: shutdown 2021-05-25T12:01:19.806 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.722+0000 7f6e3a2fb700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:01:19.806 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.722+0000 7f6e3a2fb700 1 -- 172.21.15.102:0/3048552851 >> v1:172.21.15.102:6790/0 conn(0x7f6e34096020 legacy=0x7f6e34064530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:19.807 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.722+0000 7f6e39afa700 1 -- 172.21.15.102:0/3048552851 reap_dead start 2021-05-25T12:01:19.807 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.722+0000 7f6e39afa700 1 -- 172.21.15.102:0/3048552851 reap_dead start 2021-05-25T12:01:19.807 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.722+0000 7f6e3a2fb700 1 -- 172.21.15.102:0/3048552851 shutdown_connections 2021-05-25T12:01:19.807 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.722+0000 7f6e3a2fb700 1 -- 172.21.15.102:0/3048552851 >> 172.21.15.102:0/3048552851 conn(0x7f6e340974b0 msgr2=0x7f6e34093bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:01:19.808 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.722+0000 7f6e3a2fb700 1 -- 172.21.15.102:0/3048552851 shutdown_connections 2021-05-25T12:01:19.808 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.722+0000 7f6e3a2fb700 1 -- 172.21.15.102:0/3048552851 wait complete. 2021-05-25T12:01:19.808 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:19.722+0000 7f6e3a2fb700 1 librados: shutdown 2021-05-25T12:01:19.809 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2021-05-25T12:01:19.809 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['c'] 2021-05-25T12:01:19.809 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2021-05-25T12:01:19.810 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2021-05-25T12:01:19.810 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2021-05-25T12:01:19.810 INFO:teuthology.orchestra.run:waiting for 300 2021-05-25T12:01:19.819 INFO:tasks.ceph.mon.a:Stopped 2021-05-25T12:01:19.819 INFO:tasks.mon_thrash:Sending STOP to mon c 2021-05-25T12:01:19.819 INFO:tasks.ceph.mon.c:Sent signal 19 2021-05-25T12:01:19.820 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs to unfreeze mons 2021-05-25T12:01:39.821 INFO:tasks.mon_thrash:Sending CONT to mon c 2021-05-25T12:01:39.822 INFO:tasks.ceph.mon.c:Sent signal 18 2021-05-25T12:01:39.823 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2021-05-25T12:01:39.824 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2021-05-25T12:01:39.949 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:39.947+0000 7f823c919700 10 monclient: get_monmap_and_config 2021-05-25T12:01:39.949 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:39.947+0000 7f823c919700 10 monclient: build_initial_monmap 2021-05-25T12:01:39.949 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:39.947+0000 7f823c919700 10 monclient: monmap: 2021-05-25T12:01:39.950 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:01:39.950 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:39.950 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:01:39.948387+0000 2021-05-25T12:01:39.950 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:01:39.948387+0000 2021-05-25T12:01:39.951 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:01:39.951 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:39.951 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:01:39.951 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:01:39.951 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:01:39.952 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:39.952 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:39.947+0000 7f823c919700 1 Processor -- start 2021-05-25T12:01:39.952 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:39.947+0000 7f823c919700 1 -- start start 2021-05-25T12:01:39.952 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:39.947+0000 7f823c919700 10 monclient: init 2021-05-25T12:01:39.954 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:39.947+0000 7f823c919700 10 monclient: _reopen_session rank -1 2021-05-25T12:01:39.954 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:39.947+0000 7f823c919700 10 monclient: _add_conns ranks=[2,0,1] 2021-05-25T12:01:39.954 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:39.947+0000 7f823c919700 10 monclient(hunting): picked mon.noname-b con 0x7f8238090990 addr v1:172.21.15.102:6790/0 2021-05-25T12:01:39.954 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:39.947+0000 7f823c919700 10 monclient(hunting): picked mon.noname-a con 0x7f8238090d70 addr v1:172.21.15.102:6789/0 2021-05-25T12:01:39.955 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:39.947+0000 7f823c919700 10 monclient(hunting): picked mon.noname-c con 0x7f823812e330 addr v1:172.21.15.181:6789/0 2021-05-25T12:01:39.955 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:39.947+0000 7f823c919700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8238096fa0 con 0x7f8238090d70 2021-05-25T12:01:39.955 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:39.947+0000 7f823c919700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82380926a0 con 0x7f823812e330 2021-05-25T12:01:39.955 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:39.947+0000 7f823c919700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8238092940 con 0x7f8238090990 2021-05-25T12:01:39.955 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:39.947+0000 7f823c919700 10 monclient(hunting): _renew_subs 2021-05-25T12:01:39.956 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:39.947+0000 7f823c919700 10 monclient(hunting): authenticate will time out at 900.545712s 2021-05-25T12:01:39.956 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:39.947+0000 7f8235d9b700 1 -- >> v1:172.21.15.102:6789/0 conn(0x7f8238090d70 legacy=0x7f8238091200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:01:39.956 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:39.947+0000 7f823659c700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f8238090990 0x7f8238093240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:54776/0 (socket says 172.21.15.102:54776) 2021-05-25T12:01:39.956 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:39.947+0000 7f823659c700 1 -- 172.21.15.102:0/4016677765 learned_addr learned my addr 172.21.15.102:0/4016677765 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:01:40.150 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:40.149+0000 7f8235d9b700 1 -- 172.21.15.102:0/4016677765 >> v1:172.21.15.102:6789/0 conn(0x7f8238090d70 legacy=0x7f8238091200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:01:40.552 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:40.550+0000 7f8235d9b700 1 -- 172.21.15.102:0/4016677765 >> v1:172.21.15.102:6789/0 conn(0x7f8238090d70 legacy=0x7f8238091200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:01:41.353 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:41.352+0000 7f8235d9b700 1 -- 172.21.15.102:0/4016677765 >> v1:172.21.15.102:6789/0 conn(0x7f8238090d70 legacy=0x7f8238091200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:01:41.882 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:41.880+0000 7f8236d9d700 1 -- 172.21.15.102:0/4016677765 >> v1:172.21.15.181:6789/0 conn(0x7f823812e330 legacy=0x7f8238130720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2021-05-25T12:01:41.882 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:41.880+0000 7f8236d9d700 1 -- 172.21.15.102:0/4016677765 >> v1:172.21.15.181:6789/0 conn(0x7f823812e330 legacy=0x7f8238130720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-05-25T12:01:41.883 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:41.880+0000 7f8236d9d700 1 --1- 172.21.15.102:0/4016677765 >> v1:172.21.15.181:6789/0 conn(0x7f823812e330 0x7f8238130720 :-1 s=OPENED pgs=6522 cs=1 l=1).handle_message read tag failed 2021-05-25T12:01:41.883 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:41.880+0000 7f8236d9d700 1 --1- 172.21.15.102:0/4016677765 >> v1:172.21.15.181:6789/0 conn(0x7f823812e330 0x7f8238130720 :-1 s=OPENED pgs=6522 cs=1 l=1).fault on lossy channel, failing 2021-05-25T12:01:41.883 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:41.880+0000 7f823559a700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.181:6789/0 2021-05-25T12:01:42.950 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:42.948+0000 7f8227fff700 10 monclient(hunting): tick 2021-05-25T12:01:42.951 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:42.948+0000 7f8227fff700 1 monclient(hunting): continuing hunt 2021-05-25T12:01:42.951 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:42.948+0000 7f8227fff700 10 monclient(hunting): _reopen_session rank -1 2021-05-25T12:01:42.951 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:42.948+0000 7f8227fff700 1 -- 172.21.15.102:0/4016677765 >> v1:172.21.15.102:6790/0 conn(0x7f8238090990 legacy=0x7f8238093240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:42.951 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:42.948+0000 7f8227fff700 1 -- 172.21.15.102:0/4016677765 >> v1:172.21.15.181:6789/0 conn(0x7f823812e330 legacy=0x7f8238130720 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:01:42.952 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:42.948+0000 7f8227fff700 1 -- 172.21.15.102:0/4016677765 >> v1:172.21.15.102:6789/0 conn(0x7f8238090d70 legacy=0x7f8238091200 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-05-25T12:01:42.952 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:42.948+0000 7f8227fff700 10 monclient: _add_conns ranks=[1,0,2] 2021-05-25T12:01:42.952 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:42.948+0000 7f8227fff700 10 monclient(hunting): picked mon.noname-c con 0x7f823812e330 addr v1:172.21.15.181:6789/0 2021-05-25T12:01:42.952 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:42.948+0000 7f8227fff700 10 monclient(hunting): picked mon.noname-a con 0x7f8238090d70 addr v1:172.21.15.102:6789/0 2021-05-25T12:01:42.953 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:42.948+0000 7f8227fff700 10 monclient(hunting): picked mon.noname-b con 0x7f8238090990 addr v1:172.21.15.102:6790/0 2021-05-25T12:01:42.953 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:42.948+0000 7f8227fff700 1 -- 172.21.15.102:0/4016677765 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8238092be0 con 0x7f8238090d70 2021-05-25T12:01:42.953 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:42.949+0000 7f8227fff700 1 -- 172.21.15.102:0/4016677765 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8238096fa0 con 0x7f823812e330 2021-05-25T12:01:42.953 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:42.949+0000 7f8227fff700 1 -- 172.21.15.102:0/4016677765 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8214011d50 con 0x7f8238090990 2021-05-25T12:01:42.953 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:42.949+0000 7f8227fff700 10 monclient(hunting): _renew_subs 2021-05-25T12:01:42.954 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:42.949+0000 7f8235d9b700 1 -- 172.21.15.102:0/4016677765 >> v1:172.21.15.102:6789/0 conn(0x7f8238090d70 legacy=0x7f821400ad70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:01:43.151 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:43.150+0000 7f8235d9b700 1 -- 172.21.15.102:0/4016677765 >> v1:172.21.15.102:6789/0 conn(0x7f8238090d70 legacy=0x7f821400ad70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:01:43.552 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:43.551+0000 7f8235d9b700 1 -- 172.21.15.102:0/4016677765 >> v1:172.21.15.102:6789/0 conn(0x7f8238090d70 legacy=0x7f821400ad70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:01:44.353 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.352+0000 7f8235d9b700 1 -- 172.21.15.102:0/4016677765 >> v1:172.21.15.102:6789/0 conn(0x7f8238090d70 legacy=0x7f821400ad70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:01:44.862 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.860+0000 7f823559a700 1 -- 172.21.15.102:0/4016677765 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 128689948 0 0) 0x7f8238096fa0 con 0x7f823812e330 2021-05-25T12:01:44.862 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.860+0000 7f823559a700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:01:44.863 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.860+0000 7f823559a700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:01:44.863 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.860+0000 7f823559a700 10 monclient(hunting): my global_id is 89647 2021-05-25T12:01:44.863 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.860+0000 7f823559a700 1 -- 172.21.15.102:0/4016677765 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f821c003270 con 0x7f823812e330 2021-05-25T12:01:44.880 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.879+0000 7f823559a700 1 -- 172.21.15.102:0/4016677765 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3439722389 0 0) 0x7f8214011d50 con 0x7f8238090990 2021-05-25T12:01:44.880 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.879+0000 7f823559a700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:01:44.880 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.879+0000 7f823559a700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:01:44.881 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.879+0000 7f823559a700 10 monclient(hunting): my global_id is 16886 2021-05-25T12:01:44.881 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.879+0000 7f823559a700 1 -- 172.21.15.102:0/4016677765 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8238096fa0 con 0x7f8238090990 2021-05-25T12:01:44.897 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.895+0000 7f823559a700 1 -- 172.21.15.102:0/4016677765 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1460923740 0 0) 0x7f8238096fa0 con 0x7f8238090990 2021-05-25T12:01:44.898 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.895+0000 7f823559a700 1 -- 172.21.15.102:0/4016677765 >> v1:172.21.15.181:6789/0 conn(0x7f823812e330 legacy=0x7f82140075a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:44.898 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.895+0000 7f823559a700 1 -- 172.21.15.102:0/4016677765 >> v1:172.21.15.102:6789/0 conn(0x7f8238090d70 legacy=0x7f821400ad70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-05-25T12:01:44.898 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.895+0000 7f823559a700 10 monclient: _finish_hunting 0 2021-05-25T12:01:44.899 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.895+0000 7f823559a700 1 monclient: found mon.noname-b 2021-05-25T12:01:44.899 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.895+0000 7f823559a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:01:44.899 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.895+0000 7f823559a700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T12:01:44.899 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.895+0000 7f823559a700 1 -- 172.21.15.102:0/4016677765 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8214012fc0 con 0x7f8238090990 2021-05-25T12:01:44.899 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.895+0000 7f823559a700 10 monclient: _finish_auth 0 2021-05-25T12:01:44.900 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.895+0000 7f823559a700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:01:44.900 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.895+0000 7f823559a700 1 -- 172.21.15.102:0/4016677765 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f8228001a50 con 0x7f8238090990 2021-05-25T12:01:44.900 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.895+0000 7f823c919700 5 monclient: authenticate success, global_id 16886 2021-05-25T12:01:44.900 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.895+0000 7f823c919700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:01:44.901 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.895+0000 7f823559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:01:44.901 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.895+0000 7f823559a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:01:44.901 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.895+0000 7f823559a700 10 monclient: dump: 2021-05-25T12:01:44.901 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:01:44.902 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:44.902 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:44.902 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:44.902 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:01:44.902 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:44.903 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:01:44.903 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:01:44.903 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:01:44.903 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:44.903 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.895+0000 7f823c919700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:01:44.905 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.904+0000 7f823559a700 1 -- 172.21.15.102:0/4016677765 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8228001cb0 con 0x7f8238090990 2021-05-25T12:01:44.905 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.904+0000 7f823559a700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:01:44.906 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.904+0000 7f823559a700 1 -- 172.21.15.102:0/4016677765 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f8228002080 con 0x7f8238090990 2021-05-25T12:01:44.906 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.904+0000 7f823559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:01:44.906 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.904+0000 7f823559a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:01:44.906 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.904+0000 7f823559a700 10 monclient: dump: 2021-05-25T12:01:44.907 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:01:44.907 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:44.907 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:44.907 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:44.907 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:01:44.908 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:44.908 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:01:44.908 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:01:44.908 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:01:44.909 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:44.909 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.904+0000 7f823c919700 10 monclient: get_monmap_and_config success 2021-05-25T12:01:44.909 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.904+0000 7f823c919700 4 set_mon_vals no callback set 2021-05-25T12:01:44.911 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.905+0000 7f823c919700 10 monclient: shutdown 2021-05-25T12:01:44.911 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.905+0000 7f823c919700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:01:44.911 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.905+0000 7f823c919700 1 -- 172.21.15.102:0/4016677765 >> v1:172.21.15.102:6790/0 conn(0x7f8238090990 legacy=0x7f821400e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:44.911 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.905+0000 7f823c919700 1 -- 172.21.15.102:0/4016677765 shutdown_connections 2021-05-25T12:01:44.912 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.905+0000 7f823c919700 1 -- 172.21.15.102:0/4016677765 >> 172.21.15.102:0/4016677765 conn(0x7f823809b370 msgr2=0x7f8238097430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:01:44.912 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.905+0000 7f823c919700 1 -- 172.21.15.102:0/4016677765 shutdown_connections 2021-05-25T12:01:44.912 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.905+0000 7f823c919700 1 -- 172.21.15.102:0/4016677765 wait complete. 2021-05-25T12:01:44.912 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.906+0000 7f823c919700 10 monclient: build_initial_monmap 2021-05-25T12:01:44.912 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.906+0000 7f823c919700 10 monclient: monmap: 2021-05-25T12:01:44.913 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:01:44.913 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:01:44.913 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:01:44.913 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:01:44.913 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:01:44.914 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:01:44.914 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:01:44.914 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:01:44.914 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:01:44.915 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:44.915 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.906+0000 7f823c919700 1 librados: starting msgr at 2021-05-25T12:01:44.915 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.906+0000 7f823c919700 1 librados: starting objecter 2021-05-25T12:01:44.915 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.906+0000 7f823c919700 1 Processor -- start 2021-05-25T12:01:44.915 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.906+0000 7f823c919700 1 -- start start 2021-05-25T12:01:44.916 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.906+0000 7f823c919700 1 librados: setting wanted keys 2021-05-25T12:01:44.916 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.906+0000 7f823c919700 1 librados: calling monclient init 2021-05-25T12:01:44.916 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.906+0000 7f823c919700 10 monclient: init 2021-05-25T12:01:44.916 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.907+0000 7f823c919700 10 monclient: _reopen_session rank -1 2021-05-25T12:01:44.916 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.907+0000 7f823c919700 10 monclient: _add_conns ranks=[1,2,0] 2021-05-25T12:01:44.917 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.907+0000 7f823c919700 10 monclient(hunting): picked mon.noname-b con 0x7f8238090990 addr v1:172.21.15.181:6789/0 2021-05-25T12:01:44.917 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.907+0000 7f823c919700 10 monclient(hunting): picked mon.noname-c con 0x7f8238090d70 addr v1:172.21.15.102:6790/0 2021-05-25T12:01:44.917 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.907+0000 7f823c919700 10 monclient(hunting): picked mon.noname-a con 0x7f823812e330 addr v1:172.21.15.102:6789/0 2021-05-25T12:01:44.917 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.907+0000 7f823c919700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8238091fe0 con 0x7f823812e330 2021-05-25T12:01:44.917 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.907+0000 7f823c919700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8238117ab0 con 0x7f8238090990 2021-05-25T12:01:44.918 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.907+0000 7f823c919700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8238117d90 con 0x7f8238090d70 2021-05-25T12:01:44.918 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.907+0000 7f823c919700 10 monclient(hunting): _renew_subs 2021-05-25T12:01:44.918 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.907+0000 7f823c919700 10 monclient(hunting): authenticate will time out at 905.504523s 2021-05-25T12:01:44.918 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.907+0000 7f8236d9d700 1 -- >> v1:172.21.15.102:6789/0 conn(0x7f823812e330 legacy=0x7f8238142da0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:01:44.919 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.907+0000 7f8235d9b700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f8238090d70 0x7f8238129300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:54866/0 (socket says 172.21.15.102:54866) 2021-05-25T12:01:44.919 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.907+0000 7f8235d9b700 1 -- 172.21.15.102:0/3766750045 learned_addr learned my addr 172.21.15.102:0/3766750045 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:01:44.919 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.907+0000 7f82267fc700 10 client.?.objecter ms_handle_connect 0x7f8238090d70 2021-05-25T12:01:44.919 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.907+0000 7f82267fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:01:44.922 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.910+0000 7f82267fc700 1 -- 172.21.15.102:0/3766750045 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3785748829 0 0) 0x7f8238117d90 con 0x7f8238090d70 2021-05-25T12:01:44.922 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.910+0000 7f82267fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:01:44.922 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.910+0000 7f82267fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:01:44.923 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.910+0000 7f82267fc700 10 monclient(hunting): my global_id is 16913 2021-05-25T12:01:44.923 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.910+0000 7f82267fc700 1 -- 172.21.15.102:0/3766750045 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f821400c040 con 0x7f8238090d70 2021-05-25T12:01:44.923 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.912+0000 7f82267fc700 1 -- 172.21.15.102:0/3766750045 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2712879904 0 0) 0x7f821400c040 con 0x7f8238090d70 2021-05-25T12:01:44.923 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.912+0000 7f82267fc700 1 -- 172.21.15.102:0/3766750045 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8238117d90 con 0x7f8238090d70 2021-05-25T12:01:44.924 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.912+0000 7f82267fc700 1 -- 172.21.15.102:0/3766750045 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f8220004150 con 0x7f8238090d70 2021-05-25T12:01:44.924 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.912+0000 7f82267fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:01:44.924 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.912+0000 7f82267fc700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:01:44.924 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.912+0000 7f82267fc700 10 monclient(hunting): dump: 2021-05-25T12:01:44.924 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:01:44.925 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:44.925 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:44.925 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:44.925 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:01:44.926 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:44.926 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:01:44.926 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:01:44.926 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:01:44.926 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:44.927 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.912+0000 7f82267fc700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:01:44.927 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.913+0000 7f82267fc700 1 -- 172.21.15.102:0/3766750045 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2787032415 0 0) 0x7f8238117d90 con 0x7f8238090d70 2021-05-25T12:01:44.927 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.913+0000 7f82267fc700 1 -- 172.21.15.102:0/3766750045 >> v1:172.21.15.181:6789/0 conn(0x7f8238090990 legacy=0x7f823813aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:44.927 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.913+0000 7f82267fc700 1 -- 172.21.15.102:0/3766750045 >> v1:172.21.15.102:6789/0 conn(0x7f823812e330 legacy=0x7f8238142da0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-05-25T12:01:44.927 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.913+0000 7f82267fc700 10 monclient: _finish_hunting 0 2021-05-25T12:01:44.928 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.913+0000 7f82267fc700 1 monclient: found mon.c 2021-05-25T12:01:44.928 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.913+0000 7f82267fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:01:44.928 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.913+0000 7f82267fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:01:44.928 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.913+0000 7f82267fc700 1 -- 172.21.15.102:0/3766750045 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f823812a110 con 0x7f8238090d70 2021-05-25T12:01:44.929 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.913+0000 7f82267fc700 10 monclient: _finish_auth 0 2021-05-25T12:01:44.929 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.913+0000 7f82267fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:01:44.929 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.914+0000 7f823c919700 5 monclient: authenticate success, global_id 16913 2021-05-25T12:01:44.929 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.914+0000 7f823c919700 10 monclient: _renew_subs 2021-05-25T12:01:44.929 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.914+0000 7f823c919700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:01:44.930 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.914+0000 7f823c919700 1 -- 172.21.15.102:0/3766750045 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8238115d00 con 0x7f8238090d70 2021-05-25T12:01:44.930 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.914+0000 7f823c919700 10 client.16913.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:01:44.930 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.914+0000 7f823c919700 10 monclient: _renew_subs 2021-05-25T12:01:44.930 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.914+0000 7f823c919700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:01:44.930 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.914+0000 7f823c919700 1 -- 172.21.15.102:0/3766750045 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8238129a10 con 0x7f8238090d70 2021-05-25T12:01:44.931 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.914+0000 7f823c919700 1 librados: init done 2021-05-25T12:01:44.931 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.915+0000 7f823c919700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:01:44.931 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.915+0000 7f823c919700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:01:44.931 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.915+0000 7f823c919700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:01:44.931 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.915+0000 7f823c919700 1 -- 172.21.15.102:0/3766750045 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8238129cc0 con 0x7f8238090d70 2021-05-25T12:01:44.932 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.915+0000 7f82267fc700 1 -- 172.21.15.102:0/3766750045 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8220004370 con 0x7f8238090d70 2021-05-25T12:01:44.932 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.915+0000 7f82267fc700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:01:44.932 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.915+0000 7f82267fc700 1 -- 172.21.15.102:0/3766750045 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f8220004680 con 0x7f8238090d70 2021-05-25T12:01:44.932 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.915+0000 7f82267fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:01:44.932 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.915+0000 7f82267fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:01:44.933 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.915+0000 7f82267fc700 10 monclient: dump: 2021-05-25T12:01:44.933 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:01:44.933 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:44.933 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:44.933 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:44.934 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:01:44.934 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:44.934 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:01:44.934 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:01:44.934 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:01:44.935 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:44.935 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.915+0000 7f8234d99700 4 set_mon_vals no callback set 2021-05-25T12:01:44.935 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.917+0000 7f82267fc700 1 -- 172.21.15.102:0/3766750045 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f8220004840 con 0x7f8238090d70 2021-05-25T12:01:44.935 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.917+0000 7f82267fc700 1 -- 172.21.15.102:0/3766750045 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(42..42 src has 1..42) v4 ==== 8458+0+0 (unknown 4079512998 0 0) 0x7f8220012f30 con 0x7f8238090d70 2021-05-25T12:01:44.935 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.917+0000 7f82267fc700 10 client.16913.objecter ms_dispatch 0x7f8238001550 osd_map(42..42 src has 1..42) v4 2021-05-25T12:01:44.936 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.917+0000 7f82267fc700 3 client.16913.objecter handle_osd_map got epochs [42,42] > 0 2021-05-25T12:01:44.936 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.917+0000 7f82267fc700 3 client.16913.objecter handle_osd_map decoding full epoch 42 2021-05-25T12:01:44.936 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.917+0000 7f82267fc700 20 client.16913.objecter dump_active .. 0 homeless 2021-05-25T12:01:44.936 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.919+0000 7f823659c700 10 monclient: get_auth_request con 0x7f8214033650 auth_method 0 2021-05-25T12:01:44.937 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.923+0000 7f82267fc700 10 client.16913.objecter ms_handle_connect 0x7f8214033650 2021-05-25T12:01:44.937 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.924+0000 7f82267fc700 1 -- 172.21.15.102:0/3766750045 <== mon.2 v1:172.21.15.102:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7f8220012ad0 con 0x7f8238090d70 2021-05-25T12:01:44.937 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.924+0000 7f82267fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:01:44.937 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:44.924+0000 7f82267fc700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:01:45.169 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.167+0000 7f823c919700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2021-05-25T12:01:45.170 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.167+0000 7f823c919700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2021-05-25T12:01:45.170 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.167+0000 7f823c919700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:01:45.170 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.167+0000 7f823c919700 1 -- 172.21.15.102:0/3766750045 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f8238116730 con 0x7f8238090d70 2021-05-25T12:01:45.170 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.168+0000 7f82267fc700 1 -- 172.21.15.102:0/3766750045 <== mon.2 v1:172.21.15.102:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1336 (unknown 254746657 0 1774386550) 0x7f8220012ad0 con 0x7f8238090d70 2021-05-25T12:01:45.171 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.168+0000 7f82267fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2021-05-25T12:01:45.171 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.168+0000 7f82267fc700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:01:45.171 INFO:teuthology.orchestra.run.smithi102.stdout:{"election_epoch":24,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138303579357183","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"20ee91b6-8935-4b98-b868-85c03789ccc4","modified":"2021-05-25T11:59:11.011522Z","created":"2021-05-25T11:59:11.011522Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6790","nonce":0}]},"addr":"172.21.15.102:6790/0","public_addr":"172.21.15.102:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2021-05-25T12:01:45.173 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.170+0000 7f823c919700 10 librados: watch_flush enter 2021-05-25T12:01:45.173 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.170+0000 7f823c919700 10 librados: watch_flush exit 2021-05-25T12:01:45.173 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.170+0000 7f823c919700 20 client.16913.objecter shutdown clearing up homeless session... 2021-05-25T12:01:45.174 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.170+0000 7f823c919700 10 client.16913.objecter successfully canceled tick 2021-05-25T12:01:45.174 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.170+0000 7f823c919700 1 -- 172.21.15.102:0/3766750045 >> v1:172.21.15.181:6812/29582 conn(0x7f8214033650 legacy=0x7f8214035b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:45.174 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.170+0000 7f823c919700 10 monclient: shutdown 2021-05-25T12:01:45.174 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.170+0000 7f823c919700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:01:45.174 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.170+0000 7f823c919700 1 -- 172.21.15.102:0/3766750045 >> v1:172.21.15.102:6790/0 conn(0x7f8238090d70 legacy=0x7f8238129300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:45.175 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.170+0000 7f823c919700 1 -- 172.21.15.102:0/3766750045 shutdown_connections 2021-05-25T12:01:45.175 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.170+0000 7f823c919700 1 -- 172.21.15.102:0/3766750045 >> 172.21.15.102:0/3766750045 conn(0x7f823809b370 msgr2=0x7f8238097360 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:01:45.175 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.170+0000 7f823c919700 1 -- 172.21.15.102:0/3766750045 shutdown_connections 2021-05-25T12:01:45.175 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.170+0000 7f823c919700 1 -- 172.21.15.102:0/3766750045 wait complete. 2021-05-25T12:01:45.175 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.170+0000 7f823c919700 1 librados: shutdown 2021-05-25T12:01:45.183 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2021-05-25T12:01:45.184 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2021-05-25T12:01:45.299 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.297+0000 7f1624f0e700 10 monclient: get_monmap_and_config 2021-05-25T12:01:45.299 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.297+0000 7f1624f0e700 10 monclient: build_initial_monmap 2021-05-25T12:01:45.299 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.297+0000 7f1624f0e700 10 monclient: monmap: 2021-05-25T12:01:45.300 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:01:45.300 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:45.300 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:01:45.298808+0000 2021-05-25T12:01:45.300 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:01:45.298808+0000 2021-05-25T12:01:45.301 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:01:45.301 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:45.301 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:01:45.301 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:01:45.301 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:01:45.302 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:45.315 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.298+0000 7f1624f0e700 1 Processor -- start 2021-05-25T12:01:45.315 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.298+0000 7f1624f0e700 1 -- start start 2021-05-25T12:01:45.315 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.298+0000 7f1624f0e700 10 monclient: init 2021-05-25T12:01:45.315 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.298+0000 7f1624f0e700 10 monclient: _reopen_session rank -1 2021-05-25T12:01:45.316 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.299+0000 7f1624f0e700 10 monclient: _add_conns ranks=[2,1,0] 2021-05-25T12:01:45.316 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.299+0000 7f1624f0e700 10 monclient(hunting): picked mon.noname-b con 0x7f1620063f40 addr v1:172.21.15.102:6790/0 2021-05-25T12:01:45.316 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.299+0000 7f1624f0e700 10 monclient(hunting): picked mon.noname-c con 0x7f1620098020 addr v1:172.21.15.181:6789/0 2021-05-25T12:01:45.316 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.299+0000 7f1624f0e700 10 monclient(hunting): picked mon.noname-a con 0x7f16200984a0 addr v1:172.21.15.102:6789/0 2021-05-25T12:01:45.317 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.299+0000 7f1624f0e700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1620128e60 con 0x7f16200984a0 2021-05-25T12:01:45.317 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.299+0000 7f1624f0e700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f162009ae90 con 0x7f1620098020 2021-05-25T12:01:45.317 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.299+0000 7f1624f0e700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f162009b130 con 0x7f1620063f40 2021-05-25T12:01:45.317 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.299+0000 7f1624f0e700 10 monclient(hunting): _renew_subs 2021-05-25T12:01:45.318 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.299+0000 7f1624f0e700 10 monclient(hunting): authenticate will time out at 905.896123s 2021-05-25T12:01:45.318 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.299+0000 7f161ffff700 1 -- >> v1:172.21.15.102:6789/0 conn(0x7f16200984a0 legacy=0x7f1620134180 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:01:45.318 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.299+0000 7f161f7fe700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f1620063f40 0x7f1620128750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:54872/0 (socket says 172.21.15.102:54872) 2021-05-25T12:01:45.318 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.299+0000 7f161f7fe700 1 -- 172.21.15.102:0/3549378622 learned_addr learned my addr 172.21.15.102:0/3549378622 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:01:45.319 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.299+0000 7f161e7fc700 1 -- 172.21.15.102:0/3549378622 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3654603571 0 0) 0x7f162009b130 con 0x7f1620063f40 2021-05-25T12:01:45.319 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.299+0000 7f161e7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:01:45.319 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.299+0000 7f161e7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:01:45.319 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.299+0000 7f161e7fc700 10 monclient(hunting): my global_id is 17192 2021-05-25T12:01:45.319 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.299+0000 7f161e7fc700 1 -- 172.21.15.102:0/3549378622 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1604003270 con 0x7f1620063f40 2021-05-25T12:01:45.320 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.299+0000 7f161e7fc700 1 -- 172.21.15.102:0/3549378622 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2157926570 0 0) 0x7f162009ae90 con 0x7f1620098020 2021-05-25T12:01:45.320 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.299+0000 7f161e7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:01:45.320 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.299+0000 7f161e7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:01:45.320 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.299+0000 7f161e7fc700 10 monclient(hunting): my global_id is 89971 2021-05-25T12:01:45.321 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.299+0000 7f161e7fc700 1 -- 172.21.15.102:0/3549378622 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f162009b130 con 0x7f1620098020 2021-05-25T12:01:45.321 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.299+0000 7f161e7fc700 1 -- 172.21.15.102:0/3549378622 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1647202987 0 0) 0x7f1604003270 con 0x7f1620063f40 2021-05-25T12:01:45.321 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.299+0000 7f161e7fc700 1 -- 172.21.15.102:0/3549378622 >> v1:172.21.15.181:6789/0 conn(0x7f1620098020 legacy=0x7f1620130a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:45.321 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.299+0000 7f161e7fc700 1 -- 172.21.15.102:0/3549378622 >> v1:172.21.15.102:6789/0 conn(0x7f16200984a0 legacy=0x7f1620134180 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-05-25T12:01:45.321 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.299+0000 7f161e7fc700 10 monclient: _finish_hunting 0 2021-05-25T12:01:45.322 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.299+0000 7f161e7fc700 1 monclient: found mon.noname-b 2021-05-25T12:01:45.322 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.299+0000 7f161e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:01:45.322 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.299+0000 7f161e7fc700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T12:01:45.322 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.299+0000 7f161e7fc700 1 -- 172.21.15.102:0/3549378622 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1620090b10 con 0x7f1620063f40 2021-05-25T12:01:45.323 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.299+0000 7f161e7fc700 10 monclient: _finish_auth 0 2021-05-25T12:01:45.323 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.299+0000 7f161e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:01:45.323 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.299+0000 7f161e7fc700 1 -- 172.21.15.102:0/3549378622 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f160c001610 con 0x7f1620063f40 2021-05-25T12:01:45.323 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.299+0000 7f1624f0e700 5 monclient: authenticate success, global_id 17192 2021-05-25T12:01:45.323 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.299+0000 7f1624f0e700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:01:45.324 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.299+0000 7f161e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:01:45.324 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.299+0000 7f161e7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:01:45.324 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.299+0000 7f161e7fc700 10 monclient: dump: 2021-05-25T12:01:45.324 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:01:45.324 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:45.325 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:45.325 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:45.325 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:01:45.325 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:45.325 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:01:45.326 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:01:45.326 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:01:45.326 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:45.326 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.299+0000 7f1624f0e700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:01:45.326 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.299+0000 7f161e7fc700 1 -- 172.21.15.102:0/3549378622 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f160c001880 con 0x7f1620063f40 2021-05-25T12:01:45.327 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.299+0000 7f161e7fc700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:01:45.327 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.299+0000 7f161e7fc700 1 -- 172.21.15.102:0/3549378622 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f160c001c10 con 0x7f1620063f40 2021-05-25T12:01:45.327 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.299+0000 7f1624f0e700 10 monclient: get_monmap_and_config success 2021-05-25T12:01:45.327 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.299+0000 7f1624f0e700 4 set_mon_vals no callback set 2021-05-25T12:01:45.328 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.299+0000 7f161e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:01:45.328 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.299+0000 7f161e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:01:45.328 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.299+0000 7f161e7fc700 10 monclient: dump: 2021-05-25T12:01:45.328 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:01:45.328 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:45.329 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:45.329 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:45.329 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:01:45.329 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:45.329 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:01:45.329 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:01:45.330 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:01:45.330 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:45.330 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.300+0000 7f1624f0e700 10 monclient: shutdown 2021-05-25T12:01:45.330 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.300+0000 7f1624f0e700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:01:45.331 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.300+0000 7f1624f0e700 1 -- 172.21.15.102:0/3549378622 >> v1:172.21.15.102:6790/0 conn(0x7f1620063f40 legacy=0x7f1620128750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:45.331 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.300+0000 7f1624f0e700 1 -- 172.21.15.102:0/3549378622 shutdown_connections 2021-05-25T12:01:45.331 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.300+0000 7f1624f0e700 1 -- 172.21.15.102:0/3549378622 >> 172.21.15.102:0/3549378622 conn(0x7f162009ea40 msgr2=0x7f162009ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:01:45.331 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.300+0000 7f1624f0e700 1 -- 172.21.15.102:0/3549378622 shutdown_connections 2021-05-25T12:01:45.331 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.300+0000 7f1624f0e700 1 -- 172.21.15.102:0/3549378622 wait complete. 2021-05-25T12:01:45.332 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.300+0000 7f1624f0e700 10 monclient: build_initial_monmap 2021-05-25T12:01:45.332 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.300+0000 7f1624f0e700 10 monclient: monmap: 2021-05-25T12:01:45.332 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:01:45.332 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:01:45.332 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:01:45.333 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:01:45.333 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:01:45.333 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:01:45.333 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:01:45.333 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:01:45.334 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:01:45.334 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:45.334 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.300+0000 7f1624f0e700 1 librados: starting msgr at 2021-05-25T12:01:45.334 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.300+0000 7f1624f0e700 1 librados: starting objecter 2021-05-25T12:01:45.335 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.301+0000 7f1624f0e700 1 Processor -- start 2021-05-25T12:01:45.335 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.301+0000 7f1624f0e700 1 -- start start 2021-05-25T12:01:45.335 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.301+0000 7f1624f0e700 1 librados: setting wanted keys 2021-05-25T12:01:45.335 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.301+0000 7f1624f0e700 1 librados: calling monclient init 2021-05-25T12:01:45.335 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.301+0000 7f1624f0e700 10 monclient: init 2021-05-25T12:01:45.336 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.301+0000 7f1624f0e700 10 monclient: _reopen_session rank -1 2021-05-25T12:01:45.336 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.301+0000 7f1624f0e700 10 monclient: _add_conns ranks=[0,2,1] 2021-05-25T12:01:45.336 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.301+0000 7f1624f0e700 10 monclient(hunting): picked mon.noname-a con 0x7f1620063f40 addr v1:172.21.15.102:6789/0 2021-05-25T12:01:45.336 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.301+0000 7f1624f0e700 10 monclient(hunting): picked mon.noname-c con 0x7f1620098020 addr v1:172.21.15.102:6790/0 2021-05-25T12:01:45.336 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.301+0000 7f1624f0e700 10 monclient(hunting): picked mon.noname-b con 0x7f16200984a0 addr v1:172.21.15.181:6789/0 2021-05-25T12:01:45.337 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.301+0000 7f1624f0e700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f162009a430 con 0x7f1620063f40 2021-05-25T12:01:45.337 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.301+0000 7f1624f0e700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1620096940 con 0x7f16200984a0 2021-05-25T12:01:45.337 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.301+0000 7f1624f0e700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1620096be0 con 0x7f1620098020 2021-05-25T12:01:45.337 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.301+0000 7f1624f0e700 10 monclient(hunting): _renew_subs 2021-05-25T12:01:45.337 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.301+0000 7f1624f0e700 10 monclient(hunting): authenticate will time out at 905.898602s 2021-05-25T12:01:45.337 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.301+0000 7f161f7fe700 1 -- >> v1:172.21.15.102:6789/0 conn(0x7f1620063f40 legacy=0x7f1620093980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:01:45.338 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.301+0000 7f161effd700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f1620098020 0x7f162013fe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:54880/0 (socket says 172.21.15.102:54880) 2021-05-25T12:01:45.338 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.301+0000 7f161effd700 1 -- 172.21.15.102:0/3558575 learned_addr learned my addr 172.21.15.102:0/3558575 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:01:45.338 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.301+0000 7f160b7fe700 10 client.?.objecter ms_handle_connect 0x7f1620098020 2021-05-25T12:01:45.338 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.301+0000 7f160b7fe700 10 client.?.objecter resend_mon_ops 2021-05-25T12:01:45.339 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.301+0000 7f160b7fe700 10 client.?.objecter ms_handle_connect 0x7f16200984a0 2021-05-25T12:01:45.339 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.301+0000 7f160b7fe700 10 client.?.objecter resend_mon_ops 2021-05-25T12:01:45.339 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.301+0000 7f160b7fe700 1 -- 172.21.15.102:0/3558575 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2224946825 0 0) 0x7f1620096be0 con 0x7f1620098020 2021-05-25T12:01:45.339 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.301+0000 7f160b7fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:01:45.339 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.301+0000 7f160b7fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:01:45.340 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.301+0000 7f160b7fe700 10 monclient(hunting): my global_id is 17195 2021-05-25T12:01:45.340 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.301+0000 7f160b7fe700 1 -- 172.21.15.102:0/3558575 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1600004c10 con 0x7f1620098020 2021-05-25T12:01:45.340 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.302+0000 7f160b7fe700 1 -- 172.21.15.102:0/3558575 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1030448136 0 0) 0x7f1600004c10 con 0x7f1620098020 2021-05-25T12:01:45.340 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.302+0000 7f160b7fe700 1 -- 172.21.15.102:0/3558575 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1620096be0 con 0x7f1620098020 2021-05-25T12:01:45.341 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.302+0000 7f160b7fe700 1 -- 172.21.15.102:0/3558575 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f16100029b0 con 0x7f1620098020 2021-05-25T12:01:45.341 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.302+0000 7f160b7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:01:45.341 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.302+0000 7f160b7fe700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:01:45.341 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.302+0000 7f160b7fe700 10 monclient(hunting): dump: 2021-05-25T12:01:45.341 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:01:45.342 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:45.342 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:45.342 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:45.342 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:01:45.342 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:45.343 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:01:45.343 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:01:45.343 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:01:45.343 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:45.344 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.302+0000 7f160b7fe700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:01:45.344 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.302+0000 7f160b7fe700 1 -- 172.21.15.102:0/3558575 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1449536457 0 0) 0x7f1620096be0 con 0x7f1620098020 2021-05-25T12:01:45.344 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.302+0000 7f160b7fe700 1 -- 172.21.15.102:0/3558575 >> v1:172.21.15.181:6789/0 conn(0x7f16200984a0 legacy=0x7f16201435c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:45.344 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.302+0000 7f160b7fe700 1 -- 172.21.15.102:0/3558575 >> v1:172.21.15.102:6789/0 conn(0x7f1620063f40 legacy=0x7f1620093980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-05-25T12:01:45.344 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.302+0000 7f160b7fe700 10 monclient: _finish_hunting 0 2021-05-25T12:01:45.345 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.302+0000 7f160b7fe700 1 monclient: found mon.c 2021-05-25T12:01:45.345 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.302+0000 7f160b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:01:45.345 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.302+0000 7f160b7fe700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:01:45.345 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.302+0000 7f160b7fe700 1 -- 172.21.15.102:0/3558575 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1620096e80 con 0x7f1620098020 2021-05-25T12:01:45.345 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.302+0000 7f160b7fe700 10 monclient: _finish_auth 0 2021-05-25T12:01:45.346 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.302+0000 7f160b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:01:45.346 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.302+0000 7f1624f0e700 5 monclient: authenticate success, global_id 17195 2021-05-25T12:01:45.346 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.302+0000 7f1624f0e700 10 monclient: _renew_subs 2021-05-25T12:01:45.346 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.302+0000 7f1624f0e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:01:45.347 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.302+0000 7f1624f0e700 1 -- 172.21.15.102:0/3558575 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16200958b0 con 0x7f1620098020 2021-05-25T12:01:45.347 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.302+0000 7f1624f0e700 10 client.17195.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:01:45.347 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.302+0000 7f1624f0e700 10 monclient: _renew_subs 2021-05-25T12:01:45.347 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.302+0000 7f1624f0e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:01:45.347 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.302+0000 7f1624f0e700 1 -- 172.21.15.102:0/3558575 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f16200931a0 con 0x7f1620098020 2021-05-25T12:01:45.347 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.302+0000 7f1624f0e700 1 librados: init done 2021-05-25T12:01:45.348 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.302+0000 7f160b7fe700 1 -- 172.21.15.102:0/3558575 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1610002d30 con 0x7f1620098020 2021-05-25T12:01:45.348 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.302+0000 7f160b7fe700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:01:45.348 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.302+0000 7f160b7fe700 1 -- 172.21.15.102:0/3558575 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f1610001840 con 0x7f1620098020 2021-05-25T12:01:45.348 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.302+0000 7f160b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:01:45.349 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.302+0000 7f160b7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:01:45.349 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.302+0000 7f160b7fe700 10 monclient: dump: 2021-05-25T12:01:45.349 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:01:45.349 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:45.349 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:45.350 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:45.350 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:01:45.350 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:45.350 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:01:45.351 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:01:45.351 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:01:45.351 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:45.351 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.302+0000 7f161dffb700 4 set_mon_vals no callback set 2021-05-25T12:01:45.352 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.302+0000 7f160b7fe700 1 -- 172.21.15.102:0/3558575 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f16100019a0 con 0x7f1620098020 2021-05-25T12:01:45.352 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.302+0000 7f1624f0e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:01:45.352 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.302+0000 7f1624f0e700 1 -- 172.21.15.102:0/3558575 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f15f80065a0 con 0x7f15f8002840 2021-05-25T12:01:45.352 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.303+0000 7f160b7fe700 1 -- 172.21.15.102:0/3558575 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(42..42 src has 1..42) v4 ==== 8458+0+0 (unknown 4079512998 0 0) 0x7f161003e250 con 0x7f1620098020 2021-05-25T12:01:45.352 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.303+0000 7f160b7fe700 10 client.17195.objecter ms_dispatch 0x7f1620001550 osd_map(42..42 src has 1..42) v4 2021-05-25T12:01:45.353 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.303+0000 7f160b7fe700 3 client.17195.objecter handle_osd_map got epochs [42,42] > 0 2021-05-25T12:01:45.353 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.303+0000 7f160b7fe700 3 client.17195.objecter handle_osd_map decoding full epoch 42 2021-05-25T12:01:45.353 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.303+0000 7f160b7fe700 20 client.17195.objecter dump_active .. 0 homeless 2021-05-25T12:01:45.353 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.303+0000 7f160b7fe700 10 client.17195.objecter ms_handle_connect 0x7f15f8002840 2021-05-25T12:01:45.353 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.303+0000 7f160b7fe700 10 client.17195.objecter resend_mon_ops 2021-05-25T12:01:45.354 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.303+0000 7f161ffff700 10 monclient: get_auth_request con 0x7f1600033080 auth_method 0 2021-05-25T12:01:45.354 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.303+0000 7f160b7fe700 1 -- 172.21.15.102:0/3558575 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1825712568 0 0) 0x7f15f80065a0 con 0x7f15f8002840 2021-05-25T12:01:45.354 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.303+0000 7f160b7fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:01:45.354 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.303+0000 7f160b7fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:01:45.355 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.303+0000 7f160b7fe700 10 monclient(hunting): my global_id is 17198 2021-05-25T12:01:45.355 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.303+0000 7f160b7fe700 1 -- 172.21.15.102:0/3558575 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1620096be0 con 0x7f15f8002840 2021-05-25T12:01:45.355 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.303+0000 7f160b7fe700 1 -- 172.21.15.102:0/3558575 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3261705178 0 0) 0x7f1620096be0 con 0x7f15f8002840 2021-05-25T12:01:45.355 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.303+0000 7f160b7fe700 1 -- 172.21.15.102:0/3558575 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f15f80065a0 con 0x7f15f8002840 2021-05-25T12:01:45.355 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.303+0000 7f160b7fe700 1 -- 172.21.15.102:0/3558575 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f160c001b90 con 0x7f15f8002840 2021-05-25T12:01:45.356 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.303+0000 7f160b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:01:45.356 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.303+0000 7f160b7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:01:45.356 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.303+0000 7f160b7fe700 10 monclient: dump: 2021-05-25T12:01:45.356 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:01:45.356 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:45.357 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:45.357 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:45.357 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:01:45.357 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:45.357 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:01:45.357 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:01:45.358 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:01:45.358 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:45.358 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.303+0000 7f160b7fe700 10 client.17195.objecter ms_handle_connect 0x7f1600033080 2021-05-25T12:01:45.358 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.303+0000 7f160b7fe700 1 -- 172.21.15.102:0/3558575 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 822045356 0 0) 0x7f15f80065a0 con 0x7f15f8002840 2021-05-25T12:01:45.358 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.303+0000 7f160b7fe700 1 -- 172.21.15.102:0/3558575 --> v1:172.21.15.102:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f15f8009e40 con 0x7f15f8002840 2021-05-25T12:01:45.359 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.303+0000 7f160b7fe700 1 -- 172.21.15.102:0/3558575 <== mon.2 v1:172.21.15.102:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7f160c004530 con 0x7f15f8002840 2021-05-25T12:01:45.359 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.303+0000 7f160b7fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:01:45.359 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.303+0000 7f160b7fe700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:01:45.359 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.303+0000 7f160b7fe700 1 -- 172.21.15.102:0/3558575 >> v1:172.21.15.102:6790/0 conn(0x7f15f8002840 legacy=0x7f15f8004ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:45.359 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.303+0000 7f160b7fe700 1 -- 172.21.15.102:0/3558575 >> v1:172.21.15.102:6790/0 conn(0x7f15f8002840 legacy=0x7f15f8004ce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:01:45.360 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.311+0000 7f1624f0e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:01:45.360 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.311+0000 7f1624f0e700 1 -- 172.21.15.102:0/3558575 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f15f8007510 con 0x7f15f80068b0 2021-05-25T12:01:45.360 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.311+0000 7f160b7fe700 10 client.17195.objecter ms_handle_connect 0x7f15f80068b0 2021-05-25T12:01:45.360 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.311+0000 7f160b7fe700 10 client.17195.objecter resend_mon_ops 2021-05-25T12:01:45.360 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.311+0000 7f160b7fe700 1 -- 172.21.15.102:0/3558575 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3183587781 0 0) 0x7f15f8007510 con 0x7f15f80068b0 2021-05-25T12:01:45.361 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.311+0000 7f160b7fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:01:45.361 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.311+0000 7f160b7fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:01:45.361 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.311+0000 7f160b7fe700 10 monclient(hunting): my global_id is 17207 2021-05-25T12:01:45.361 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.311+0000 7f160b7fe700 1 -- 172.21.15.102:0/3558575 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15f80065a0 con 0x7f15f80068b0 2021-05-25T12:01:45.361 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.311+0000 7f160b7fe700 1 -- 172.21.15.102:0/3558575 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2049888007 0 0) 0x7f15f80065a0 con 0x7f15f80068b0 2021-05-25T12:01:45.362 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.311+0000 7f160b7fe700 1 -- 172.21.15.102:0/3558575 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f15f8007510 con 0x7f15f80068b0 2021-05-25T12:01:45.362 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.311+0000 7f160b7fe700 1 -- 172.21.15.102:0/3558575 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f160c004ad0 con 0x7f15f80068b0 2021-05-25T12:01:45.362 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.311+0000 7f160b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:01:45.362 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.311+0000 7f160b7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:01:45.362 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.311+0000 7f160b7fe700 10 monclient: dump: 2021-05-25T12:01:45.363 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:01:45.363 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:45.363 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:45.363 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:45.364 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:01:45.364 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:45.364 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:01:45.364 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:01:45.364 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:01:45.365 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:45.365 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.311+0000 7f160b7fe700 1 -- 172.21.15.102:0/3558575 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2606613669 0 0) 0x7f15f8007510 con 0x7f15f80068b0 2021-05-25T12:01:45.365 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.311+0000 7f160b7fe700 1 -- 172.21.15.102:0/3558575 --> v1:172.21.15.102:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f15f8008770 con 0x7f15f80068b0 2021-05-25T12:01:45.365 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.311+0000 7f160b7fe700 1 -- 172.21.15.102:0/3558575 <== mon.2 v1:172.21.15.102:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4102 (unknown 0 0 2202913268) 0x7f160c006010 con 0x7f15f80068b0 2021-05-25T12:01:45.365 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.311+0000 7f160b7fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:01:45.366 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.311+0000 7f160b7fe700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:01:45.366 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.311+0000 7f160b7fe700 1 -- 172.21.15.102:0/3558575 >> v1:172.21.15.102:6790/0 conn(0x7f15f80068b0 legacy=0x7f15f8006d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:45.366 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.311+0000 7f160b7fe700 1 -- 172.21.15.102:0/3558575 >> v1:172.21.15.102:6790/0 conn(0x7f15f80068b0 legacy=0x7f15f8006d40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:01:45.367 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:01:45.367 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:01:45.367 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:01:45.367 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "leader", 2021-05-25T12:01:45.367 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 24, 2021-05-25T12:01:45.368 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:01:45.368 INFO:teuthology.orchestra.run.smithi102.stdout: 1, 2021-05-25T12:01:45.368 INFO:teuthology.orchestra.run.smithi102.stdout: 2 2021-05-25T12:01:45.368 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:01:45.368 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 0, 2021-05-25T12:01:45.369 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:01:45.369 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:01:45.369 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:01:45.369 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:01:45.369 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:01:45.370 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:01:45.370 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:01:45.370 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:01:45.370 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:01:45.370 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:01:45.371 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:01:45.371 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:01:45.371 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:01:45.371 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:01:45.371 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:01:45.372 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:01:45.372 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:01:45.372 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:01:45.372 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:01:45.372 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:01:45.373 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:01:45.373 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:01:45.373 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:01:45.373 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:01:45.373 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:01:45.374 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:01:45.374 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:01:45.374 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:01:45.374 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:01:45.374 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:01:45.375 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:01:45.375 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:01:45.375 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:01:45.375 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:01:45.375 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:01:45.376 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:01:45.376 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:01:45.376 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:01:45.376 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:01:45.376 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:01:45.377 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:01:45.377 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:01:45.377 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:01:45.377 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:01:45.377 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:01:45.378 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:01:45.378 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:01:45.378 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:01:45.378 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:01:45.378 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:01:45.379 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:01:45.379 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:01:45.379 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:01:45.379 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:01:45.379 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:01:45.380 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:01:45.380 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:01:45.380 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:01:45.380 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:01:45.380 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:01:45.381 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:01:45.381 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:01:45.381 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:01:45.381 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:01:45.381 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:01:45.382 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:01:45.382 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:01:45.382 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:01:45.382 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:01:45.382 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:01:45.383 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:01:45.383 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:01:45.383 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:01:45.383 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:01:45.383 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:01:45.384 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:01:45.384 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:01:45.384 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:01:45.384 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:01:45.384 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:01:45.385 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:01:45.385 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:01:45.385 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:01:45.385 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:01:45.385 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:01:45.386 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:01:45.386 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:01:45.386 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:01:45.386 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:01:45.387 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:01:45.387 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:01:45.387 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:01:45.387 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:01:45.387 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:01:45.388 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:01:45.388 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:01:45.388 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:01:45.388 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:01:45.389 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:01:45.389 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:01:45.389 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:01:45.389 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:01:45.389 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:01:45.390 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:01:45.390 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:01:45.390 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:01:45.390 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:01:45.391 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:01:45.391 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:01:45.391 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:01:45.391 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:01:45.391 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:01:45.392 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:01:45.392 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:01:45.392 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:01:45.392 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:01:45.392 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:01:45.393 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:01:45.393 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": [ 2021-05-25T12:01:45.393 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:01:45.393 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:01:45.393 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:01:45.394 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 5 2021-05-25T12:01:45.394 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:01:45.394 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:01:45.394 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:01:45.394 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:01:45.395 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:01:45.395 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:01:45.395 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 30 2021-05-25T12:01:45.395 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:01:45.395 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:01:45.396 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": [ 2021-05-25T12:01:45.396 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:01:45.396 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:01:45.396 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:01:45.396 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:01:45.397 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:01:45.397 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:01:45.397 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:01:45.397 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:01:45.398 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:01:45.400 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.312+0000 7f1624f0e700 10 librados: watch_flush enter 2021-05-25T12:01:45.400 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.312+0000 7f1624f0e700 10 librados: watch_flush exit 2021-05-25T12:01:45.400 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.312+0000 7f1624f0e700 20 client.17195.objecter shutdown clearing up homeless session... 2021-05-25T12:01:45.400 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.312+0000 7f1624f0e700 10 client.17195.objecter successfully canceled tick 2021-05-25T12:01:45.401 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.312+0000 7f1624f0e700 1 -- 172.21.15.102:0/3558575 >> v1:172.21.15.181:6812/29582 conn(0x7f1600033080 legacy=0x7f1600035540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:45.401 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.312+0000 7f1624f0e700 10 monclient: shutdown 2021-05-25T12:01:45.401 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.312+0000 7f1624f0e700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:01:45.401 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.312+0000 7f1624f0e700 1 -- 172.21.15.102:0/3558575 >> v1:172.21.15.102:6790/0 conn(0x7f1620098020 legacy=0x7f162013fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:45.401 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.312+0000 7f161ffff700 1 -- 172.21.15.102:0/3558575 reap_dead start 2021-05-25T12:01:45.402 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.312+0000 7f161ffff700 1 -- 172.21.15.102:0/3558575 reap_dead start 2021-05-25T12:01:45.402 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.312+0000 7f1624f0e700 1 -- 172.21.15.102:0/3558575 shutdown_connections 2021-05-25T12:01:45.402 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.312+0000 7f1624f0e700 1 -- 172.21.15.102:0/3558575 >> 172.21.15.102:0/3558575 conn(0x7f162009ea40 msgr2=0x7f16201279b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:01:45.402 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.312+0000 7f1624f0e700 1 -- 172.21.15.102:0/3558575 shutdown_connections 2021-05-25T12:01:45.403 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.312+0000 7f1624f0e700 1 -- 172.21.15.102:0/3558575 wait complete. 2021-05-25T12:01:45.403 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.312+0000 7f1624f0e700 1 librados: shutdown 2021-05-25T12:01:45.403 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2021-05-25T12:01:45.516 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.515+0000 7f944214b700 10 monclient: get_monmap_and_config 2021-05-25T12:01:45.517 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.515+0000 7f944214b700 10 monclient: build_initial_monmap 2021-05-25T12:01:45.517 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.515+0000 7f944214b700 10 monclient: monmap: 2021-05-25T12:01:45.517 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:01:45.518 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:45.518 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:01:45.516479+0000 2021-05-25T12:01:45.518 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:01:45.516479+0000 2021-05-25T12:01:45.518 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:01:45.519 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:45.519 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:01:45.519 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:01:45.519 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:01:45.519 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:45.531 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.516+0000 7f944214b700 1 Processor -- start 2021-05-25T12:01:45.532 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.516+0000 7f944214b700 1 -- start start 2021-05-25T12:01:45.532 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.516+0000 7f944214b700 10 monclient: init 2021-05-25T12:01:45.532 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.516+0000 7f944214b700 10 monclient: _reopen_session rank -1 2021-05-25T12:01:45.532 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.516+0000 7f944214b700 10 monclient: _add_conns ranks=[1,2,0] 2021-05-25T12:01:45.533 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.516+0000 7f944214b700 10 monclient(hunting): picked mon.noname-c con 0x7f943c098930 addr v1:172.21.15.181:6789/0 2021-05-25T12:01:45.533 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.516+0000 7f944214b700 10 monclient(hunting): picked mon.noname-b con 0x7f943c12bca0 addr v1:172.21.15.102:6790/0 2021-05-25T12:01:45.533 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.516+0000 7f944214b700 10 monclient(hunting): picked mon.noname-a con 0x7f943c12c150 addr v1:172.21.15.102:6789/0 2021-05-25T12:01:45.533 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.516+0000 7f944214b700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f943c099b20 con 0x7f943c12c150 2021-05-25T12:01:45.534 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.516+0000 7f944214b700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f943c09a3a0 con 0x7f943c098930 2021-05-25T12:01:45.534 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.516+0000 7f944214b700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f943c09ac60 con 0x7f943c12bca0 2021-05-25T12:01:45.534 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.516+0000 7f944214b700 10 monclient(hunting): _renew_subs 2021-05-25T12:01:45.534 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.516+0000 7f944214b700 10 monclient(hunting): authenticate will time out at 906.113777s 2021-05-25T12:01:45.534 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.516+0000 7f944194a700 1 -- >> v1:172.21.15.102:6789/0 conn(0x7f943c12c150 legacy=0x7f943c133ff0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:01:45.535 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.516+0000 7f9440948700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f943c12bca0 0x7f943c1308c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:54892/0 (socket says 172.21.15.102:54892) 2021-05-25T12:01:45.535 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.516+0000 7f9440948700 1 -- 172.21.15.102:0/3774456789 learned_addr learned my addr 172.21.15.102:0/3774456789 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:01:45.535 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.517+0000 7f942bfff700 1 -- 172.21.15.102:0/3774456789 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 450252081 0 0) 0x7f943c09ac60 con 0x7f943c12bca0 2021-05-25T12:01:45.535 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.517+0000 7f942bfff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:01:45.536 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.517+0000 7f942bfff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:01:45.536 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.517+0000 7f942bfff700 10 monclient(hunting): my global_id is 17357 2021-05-25T12:01:45.536 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.517+0000 7f942bfff700 1 -- 172.21.15.102:0/3774456789 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9424003270 con 0x7f943c12bca0 2021-05-25T12:01:45.536 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.517+0000 7f942bfff700 1 -- 172.21.15.102:0/3774456789 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1157212658 0 0) 0x7f9424003270 con 0x7f943c12bca0 2021-05-25T12:01:45.536 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.517+0000 7f942bfff700 1 -- 172.21.15.102:0/3774456789 >> v1:172.21.15.181:6789/0 conn(0x7f943c098930 legacy=0x7f943c12b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:45.537 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.517+0000 7f942bfff700 1 -- 172.21.15.102:0/3774456789 >> v1:172.21.15.102:6789/0 conn(0x7f943c12c150 legacy=0x7f943c133ff0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-05-25T12:01:45.537 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.517+0000 7f942bfff700 10 monclient: _finish_hunting 0 2021-05-25T12:01:45.537 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.517+0000 7f942bfff700 1 monclient: found mon.noname-b 2021-05-25T12:01:45.537 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.517+0000 7f942bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:01:45.538 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.517+0000 7f942bfff700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T12:01:45.538 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.517+0000 7f942bfff700 1 -- 172.21.15.102:0/3774456789 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f943c09af00 con 0x7f943c12bca0 2021-05-25T12:01:45.538 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.517+0000 7f942bfff700 10 monclient: _finish_auth 0 2021-05-25T12:01:45.538 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.517+0000 7f942bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:01:45.538 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.517+0000 7f942bfff700 1 -- 172.21.15.102:0/3774456789 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f9430001610 con 0x7f943c12bca0 2021-05-25T12:01:45.539 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.517+0000 7f944214b700 5 monclient: authenticate success, global_id 17357 2021-05-25T12:01:45.539 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.517+0000 7f944214b700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:01:45.539 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.517+0000 7f942bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:01:45.539 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.517+0000 7f942bfff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:01:45.539 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.517+0000 7f942bfff700 10 monclient: dump: 2021-05-25T12:01:45.540 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:01:45.540 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:45.540 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:45.540 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:45.540 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:01:45.541 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:45.541 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:01:45.541 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:01:45.541 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:01:45.542 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:45.542 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.517+0000 7f944214b700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:01:45.542 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.517+0000 7f942bfff700 1 -- 172.21.15.102:0/3774456789 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f94300018c0 con 0x7f943c12bca0 2021-05-25T12:01:45.542 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.517+0000 7f942bfff700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:01:45.542 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.517+0000 7f942bfff700 1 -- 172.21.15.102:0/3774456789 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f9430001c90 con 0x7f943c12bca0 2021-05-25T12:01:45.543 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.517+0000 7f942bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:01:45.543 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.517+0000 7f942bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:01:45.543 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.517+0000 7f942bfff700 10 monclient: dump: 2021-05-25T12:01:45.543 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:01:45.543 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:45.544 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:45.544 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:45.544 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:01:45.544 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:45.544 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:01:45.545 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:01:45.545 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:01:45.545 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:45.545 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.517+0000 7f944214b700 10 monclient: get_monmap_and_config success 2021-05-25T12:01:45.546 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.517+0000 7f944214b700 4 set_mon_vals no callback set 2021-05-25T12:01:45.546 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.517+0000 7f944214b700 10 monclient: shutdown 2021-05-25T12:01:45.546 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.517+0000 7f944214b700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:01:45.546 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.517+0000 7f944214b700 1 -- 172.21.15.102:0/3774456789 >> v1:172.21.15.102:6790/0 conn(0x7f943c12bca0 legacy=0x7f943c1308c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:45.546 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.517+0000 7f944214b700 1 -- 172.21.15.102:0/3774456789 shutdown_connections 2021-05-25T12:01:45.547 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.517+0000 7f944214b700 1 -- 172.21.15.102:0/3774456789 >> 172.21.15.102:0/3774456789 conn(0x7f943c09e9d0 msgr2=0x7f943c09ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:01:45.547 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.517+0000 7f944214b700 1 -- 172.21.15.102:0/3774456789 shutdown_connections 2021-05-25T12:01:45.547 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.517+0000 7f944214b700 1 -- 172.21.15.102:0/3774456789 wait complete. 2021-05-25T12:01:45.547 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.518+0000 7f944214b700 10 monclient: build_initial_monmap 2021-05-25T12:01:45.547 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.518+0000 7f944214b700 10 monclient: monmap: 2021-05-25T12:01:45.548 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:01:45.548 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:01:45.548 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:01:45.548 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:01:45.549 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:01:45.549 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:01:45.549 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:01:45.549 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:01:45.549 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:01:45.550 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:45.550 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.518+0000 7f944214b700 1 librados: starting msgr at 2021-05-25T12:01:45.550 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.518+0000 7f944214b700 1 librados: starting objecter 2021-05-25T12:01:45.550 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.518+0000 7f944214b700 1 Processor -- start 2021-05-25T12:01:45.550 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.518+0000 7f944214b700 1 -- start start 2021-05-25T12:01:45.551 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.518+0000 7f944214b700 1 librados: setting wanted keys 2021-05-25T12:01:45.551 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.518+0000 7f944214b700 1 librados: calling monclient init 2021-05-25T12:01:45.551 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.518+0000 7f944214b700 10 monclient: init 2021-05-25T12:01:45.551 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.519+0000 7f944214b700 10 monclient: _reopen_session rank -1 2021-05-25T12:01:45.552 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.519+0000 7f944214b700 10 monclient: _add_conns ranks=[2,0,1] 2021-05-25T12:01:45.552 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.519+0000 7f944214b700 10 monclient(hunting): picked mon.noname-c con 0x7f943c098930 addr v1:172.21.15.102:6790/0 2021-05-25T12:01:45.552 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.519+0000 7f944214b700 10 monclient(hunting): picked mon.noname-a con 0x7f943c12bca0 addr v1:172.21.15.102:6789/0 2021-05-25T12:01:45.552 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.519+0000 7f944214b700 10 monclient(hunting): picked mon.noname-b con 0x7f943c12c150 addr v1:172.21.15.181:6789/0 2021-05-25T12:01:45.552 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.519+0000 7f944214b700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f943c09a190 con 0x7f943c12bca0 2021-05-25T12:01:45.553 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.519+0000 7f944214b700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f943c126360 con 0x7f943c12c150 2021-05-25T12:01:45.553 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.519+0000 7f944214b700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f943c126600 con 0x7f943c098930 2021-05-25T12:01:45.553 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.519+0000 7f944214b700 10 monclient(hunting): _renew_subs 2021-05-25T12:01:45.553 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.519+0000 7f944214b700 10 monclient(hunting): authenticate will time out at 906.116253s 2021-05-25T12:01:45.553 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.519+0000 7f9440948700 1 -- >> v1:172.21.15.102:6789/0 conn(0x7f943c12bca0 legacy=0x7f943c13ad00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:01:45.554 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.519+0000 7f9441149700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f943c098930 0x7f943c128ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:54896/0 (socket says 172.21.15.102:54896) 2021-05-25T12:01:45.554 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.519+0000 7f9441149700 1 -- 172.21.15.102:0/2892010427 learned_addr learned my addr 172.21.15.102:0/2892010427 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:01:45.554 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.519+0000 7f94297fa700 10 client.?.objecter ms_handle_connect 0x7f943c098930 2021-05-25T12:01:45.554 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.519+0000 7f94297fa700 10 client.?.objecter resend_mon_ops 2021-05-25T12:01:45.555 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.519+0000 7f94297fa700 10 client.?.objecter ms_handle_connect 0x7f943c12c150 2021-05-25T12:01:45.555 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.519+0000 7f94297fa700 10 client.?.objecter resend_mon_ops 2021-05-25T12:01:45.555 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.519+0000 7f94297fa700 1 -- 172.21.15.102:0/2892010427 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3484465020 0 0) 0x7f943c126600 con 0x7f943c098930 2021-05-25T12:01:45.555 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.519+0000 7f94297fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:01:45.555 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.519+0000 7f94297fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:01:45.556 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.519+0000 7f94297fa700 10 monclient(hunting): my global_id is 17360 2021-05-25T12:01:45.556 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.519+0000 7f94297fa700 1 -- 172.21.15.102:0/2892010427 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f941c004c10 con 0x7f943c098930 2021-05-25T12:01:45.556 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.519+0000 7f94297fa700 1 -- 172.21.15.102:0/2892010427 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3451830725 0 0) 0x7f943c126360 con 0x7f943c12c150 2021-05-25T12:01:45.556 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.519+0000 7f94297fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:01:45.556 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.519+0000 7f94297fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:01:45.557 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.519+0000 7f94297fa700 10 monclient(hunting): my global_id is 90106 2021-05-25T12:01:45.557 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.519+0000 7f94297fa700 1 -- 172.21.15.102:0/2892010427 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f943c126600 con 0x7f943c12c150 2021-05-25T12:01:45.557 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.519+0000 7f94297fa700 1 -- 172.21.15.102:0/2892010427 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3587286127 0 0) 0x7f941c004c10 con 0x7f943c098930 2021-05-25T12:01:45.557 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.519+0000 7f94297fa700 1 -- 172.21.15.102:0/2892010427 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f943c126360 con 0x7f943c098930 2021-05-25T12:01:45.557 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.519+0000 7f94297fa700 1 -- 172.21.15.102:0/2892010427 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f942c001980 con 0x7f943c098930 2021-05-25T12:01:45.558 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.519+0000 7f94297fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:01:45.558 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.519+0000 7f94297fa700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:01:45.558 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.519+0000 7f94297fa700 10 monclient(hunting): dump: 2021-05-25T12:01:45.558 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:01:45.559 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:45.559 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:45.559 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:45.559 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:01:45.559 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:45.560 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:01:45.560 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:01:45.560 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:01:45.560 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:45.560 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.519+0000 7f94297fa700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:01:45.561 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.519+0000 7f94297fa700 1 -- 172.21.15.102:0/2892010427 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3806801510 0 0) 0x7f943c126360 con 0x7f943c098930 2021-05-25T12:01:45.561 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.519+0000 7f94297fa700 1 -- 172.21.15.102:0/2892010427 >> v1:172.21.15.181:6789/0 conn(0x7f943c12c150 legacy=0x7f943c1430b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:45.561 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.519+0000 7f94297fa700 1 -- 172.21.15.102:0/2892010427 >> v1:172.21.15.102:6789/0 conn(0x7f943c12bca0 legacy=0x7f943c13ad00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-05-25T12:01:45.561 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.519+0000 7f94297fa700 10 monclient: _finish_hunting 0 2021-05-25T12:01:45.561 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.519+0000 7f94297fa700 1 monclient: found mon.c 2021-05-25T12:01:45.562 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.519+0000 7f94297fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:01:45.562 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.519+0000 7f94297fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:01:45.562 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.519+0000 7f94297fa700 1 -- 172.21.15.102:0/2892010427 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f943c124f70 con 0x7f943c098930 2021-05-25T12:01:45.562 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.519+0000 7f94297fa700 10 monclient: _finish_auth 0 2021-05-25T12:01:45.563 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.519+0000 7f94297fa700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:01:45.563 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.519+0000 7f944214b700 5 monclient: authenticate success, global_id 17360 2021-05-25T12:01:45.563 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.519+0000 7f944214b700 10 monclient: _renew_subs 2021-05-25T12:01:45.563 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.519+0000 7f944214b700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:01:45.563 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.519+0000 7f944214b700 1 -- 172.21.15.102:0/2892010427 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f943c1291d0 con 0x7f943c098930 2021-05-25T12:01:45.564 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.519+0000 7f944214b700 10 client.17360.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:01:45.564 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.519+0000 7f944214b700 10 monclient: _renew_subs 2021-05-25T12:01:45.564 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.519+0000 7f944214b700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:01:45.564 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.519+0000 7f944214b700 1 -- 172.21.15.102:0/2892010427 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f943c13b5d0 con 0x7f943c098930 2021-05-25T12:01:45.564 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.519+0000 7f944214b700 1 librados: init done 2021-05-25T12:01:45.565 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.520+0000 7f94297fa700 1 -- 172.21.15.102:0/2892010427 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f942c001cc0 con 0x7f943c098930 2021-05-25T12:01:45.565 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.520+0000 7f94297fa700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:01:45.565 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.520+0000 7f94297fa700 1 -- 172.21.15.102:0/2892010427 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f942c002410 con 0x7f943c098930 2021-05-25T12:01:45.565 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.520+0000 7f94297fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:01:45.566 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.520+0000 7f942b7fe700 4 set_mon_vals no callback set 2021-05-25T12:01:45.566 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.520+0000 7f94297fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:01:45.566 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.520+0000 7f94297fa700 10 monclient: dump: 2021-05-25T12:01:45.566 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:01:45.566 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:45.567 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:45.567 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:45.567 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:01:45.567 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:45.568 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:01:45.568 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:01:45.568 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:01:45.568 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:45.568 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.520+0000 7f944214b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:01:45.569 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.520+0000 7f944214b700 1 -- 172.21.15.102:0/2892010427 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f94140065a0 con 0x7f9414002840 2021-05-25T12:01:45.569 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.520+0000 7f94297fa700 1 -- 172.21.15.102:0/2892010427 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f942c002570 con 0x7f943c098930 2021-05-25T12:01:45.569 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.520+0000 7f94297fa700 1 -- 172.21.15.102:0/2892010427 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(42..42 src has 1..42) v4 ==== 8458+0+0 (unknown 4079512998 0 0) 0x7f942c03e090 con 0x7f943c098930 2021-05-25T12:01:45.569 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.520+0000 7f94297fa700 10 client.17360.objecter ms_dispatch 0x7f943c001550 osd_map(42..42 src has 1..42) v4 2021-05-25T12:01:45.569 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.520+0000 7f94297fa700 3 client.17360.objecter handle_osd_map got epochs [42,42] > 0 2021-05-25T12:01:45.570 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.520+0000 7f94297fa700 3 client.17360.objecter handle_osd_map decoding full epoch 42 2021-05-25T12:01:45.570 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.520+0000 7f94297fa700 20 client.17360.objecter dump_active .. 0 homeless 2021-05-25T12:01:45.570 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.520+0000 7f94297fa700 10 client.17360.objecter ms_handle_connect 0x7f9414002840 2021-05-25T12:01:45.570 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.520+0000 7f94297fa700 10 client.17360.objecter resend_mon_ops 2021-05-25T12:01:45.571 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.521+0000 7f94297fa700 1 -- 172.21.15.102:0/2892010427 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1354037245 0 0) 0x7f94140065a0 con 0x7f9414002840 2021-05-25T12:01:45.571 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.521+0000 7f94297fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:01:45.571 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.521+0000 7f94297fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:01:45.571 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.521+0000 7f94297fa700 10 monclient(hunting): my global_id is 90109 2021-05-25T12:01:45.571 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.521+0000 7f94297fa700 1 -- 172.21.15.102:0/2892010427 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f943c126360 con 0x7f9414002840 2021-05-25T12:01:45.572 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.521+0000 7f944194a700 10 monclient: get_auth_request con 0x7f941c033150 auth_method 0 2021-05-25T12:01:45.572 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.521+0000 7f94297fa700 1 -- 172.21.15.102:0/2892010427 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4077297246 0 0) 0x7f943c126360 con 0x7f9414002840 2021-05-25T12:01:45.572 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.521+0000 7f94297fa700 1 -- 172.21.15.102:0/2892010427 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f94140065a0 con 0x7f9414002840 2021-05-25T12:01:45.572 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.521+0000 7f94297fa700 1 -- 172.21.15.102:0/2892010427 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f9430001210 con 0x7f9414002840 2021-05-25T12:01:45.572 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.521+0000 7f94297fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:01:45.573 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.521+0000 7f94297fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T12:01:45.573 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.521+0000 7f94297fa700 10 monclient: dump: 2021-05-25T12:01:45.573 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:01:45.573 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:45.574 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:45.574 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:45.574 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:01:45.574 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:45.574 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:01:45.575 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:01:45.575 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:01:45.575 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:45.575 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.521+0000 7f94297fa700 1 -- 172.21.15.102:0/2892010427 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2304084972 0 0) 0x7f94140065a0 con 0x7f9414002840 2021-05-25T12:01:45.575 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.521+0000 7f94297fa700 1 -- 172.21.15.102:0/2892010427 --> v1:172.21.15.181:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9414009e40 con 0x7f9414002840 2021-05-25T12:01:45.576 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.521+0000 7f94297fa700 1 -- 172.21.15.102:0/2892010427 <== mon.1 v1:172.21.15.181:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7f9430004690 con 0x7f9414002840 2021-05-25T12:01:45.576 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.521+0000 7f94297fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:01:45.576 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.521+0000 7f94297fa700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:01:45.576 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.521+0000 7f94297fa700 1 -- 172.21.15.102:0/2892010427 >> v1:172.21.15.181:6789/0 conn(0x7f9414002840 legacy=0x7f9414004ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:45.576 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.522+0000 7f94297fa700 1 -- 172.21.15.102:0/2892010427 >> v1:172.21.15.181:6789/0 conn(0x7f9414002840 legacy=0x7f9414004ce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:01:45.577 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.522+0000 7f94297fa700 10 client.17360.objecter ms_handle_connect 0x7f941c033150 2021-05-25T12:01:45.577 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.528+0000 7f944214b700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:01:45.577 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.528+0000 7f944214b700 1 -- 172.21.15.102:0/2892010427 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f9414009ad0 con 0x7f9414008d90 2021-05-25T12:01:45.577 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.528+0000 7f94297fa700 10 client.17360.objecter ms_handle_connect 0x7f9414008d90 2021-05-25T12:01:45.578 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.528+0000 7f94297fa700 10 client.17360.objecter resend_mon_ops 2021-05-25T12:01:45.578 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.529+0000 7f94297fa700 1 -- 172.21.15.102:0/2892010427 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1063912426 0 0) 0x7f9414009ad0 con 0x7f9414008d90 2021-05-25T12:01:45.578 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.529+0000 7f94297fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:01:45.578 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.529+0000 7f94297fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:01:45.578 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.529+0000 7f94297fa700 10 monclient(hunting): my global_id is 90112 2021-05-25T12:01:45.579 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.529+0000 7f94297fa700 1 -- 172.21.15.102:0/2892010427 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94140065a0 con 0x7f9414008d90 2021-05-25T12:01:45.579 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.529+0000 7f94297fa700 1 -- 172.21.15.102:0/2892010427 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1004424013 0 0) 0x7f94140065a0 con 0x7f9414008d90 2021-05-25T12:01:45.579 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.529+0000 7f94297fa700 1 -- 172.21.15.102:0/2892010427 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9414009ad0 con 0x7f9414008d90 2021-05-25T12:01:45.579 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.529+0000 7f94297fa700 1 -- 172.21.15.102:0/2892010427 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f9430004c10 con 0x7f9414008d90 2021-05-25T12:01:45.579 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.529+0000 7f94297fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:01:45.580 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.529+0000 7f94297fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T12:01:45.580 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.529+0000 7f94297fa700 10 monclient: dump: 2021-05-25T12:01:45.580 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:01:45.580 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:01:45.581 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:45.581 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:01:45.581 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:01:45.581 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:01:45.581 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:01:45.582 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:01:45.582 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:01:45.582 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:01:45.582 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.529+0000 7f94297fa700 1 -- 172.21.15.102:0/2892010427 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 385195439 0 0) 0x7f9414009ad0 con 0x7f9414008d90 2021-05-25T12:01:45.582 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.529+0000 7f94297fa700 1 -- 172.21.15.102:0/2892010427 --> v1:172.21.15.181:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f9414008740 con 0x7f9414008d90 2021-05-25T12:01:45.583 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.529+0000 7f94297fa700 1 -- 172.21.15.102:0/2892010427 <== mon.1 v1:172.21.15.181:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3930 (unknown 0 0 3316107990) 0x7f9430005f70 con 0x7f9414008d90 2021-05-25T12:01:45.583 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.529+0000 7f94297fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:01:45.583 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.529+0000 7f94297fa700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:01:45.583 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.529+0000 7f94297fa700 1 -- 172.21.15.102:0/2892010427 >> v1:172.21.15.181:6789/0 conn(0x7f9414008d90 legacy=0x7f9414009220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:45.583 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.529+0000 7f94297fa700 1 -- 172.21.15.102:0/2892010427 >> v1:172.21.15.181:6789/0 conn(0x7f9414008d90 legacy=0x7f9414009220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:01:45.584 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:01:45.584 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:01:45.585 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:01:45.585 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "peon", 2021-05-25T12:01:45.585 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 24, 2021-05-25T12:01:45.585 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:01:45.585 INFO:teuthology.orchestra.run.smithi102.stdout: 1, 2021-05-25T12:01:45.586 INFO:teuthology.orchestra.run.smithi102.stdout: 2 2021-05-25T12:01:45.586 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:01:45.586 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 0, 2021-05-25T12:01:45.586 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:01:45.587 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:01:45.587 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:01:45.587 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:01:45.587 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:01:45.587 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:01:45.588 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:01:45.588 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:01:45.588 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:01:45.588 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:01:45.588 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:01:45.589 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:01:45.589 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:01:45.589 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:01:45.589 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:01:45.589 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:01:45.590 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:01:45.590 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:01:45.590 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:01:45.590 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:01:45.590 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:01:45.591 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:01:45.591 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:01:45.591 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:01:45.591 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:01:45.592 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:01:45.592 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:01:45.592 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:01:45.592 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:01:45.592 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:01:45.593 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:01:45.593 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:01:45.593 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:01:45.593 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:01:45.593 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:01:45.594 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:01:45.594 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:01:45.594 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:01:45.594 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:01:45.594 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:01:45.595 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:01:45.595 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:01:45.595 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:01:45.595 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:01:45.596 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:01:45.596 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:01:45.596 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:01:45.596 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:01:45.596 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:01:45.597 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:01:45.597 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:01:45.597 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:01:45.597 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:01:45.597 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:01:45.598 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:01:45.598 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:01:45.598 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:01:45.598 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:01:45.598 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:01:45.599 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:01:45.599 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:01:45.599 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:01:45.599 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:01:45.599 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:01:45.600 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:01:45.600 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:01:45.600 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:01:45.600 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:01:45.600 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:01:45.601 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:01:45.601 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:01:45.601 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:01:45.601 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:01:45.602 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:01:45.602 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:01:45.602 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:01:45.602 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:01:45.602 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:01:45.603 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:01:45.603 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:01:45.603 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:01:45.603 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:01:45.603 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:01:45.604 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:01:45.604 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:01:45.604 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:01:45.604 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:01:45.605 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:01:45.605 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:01:45.605 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:01:45.605 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:01:45.605 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:01:45.606 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:01:45.606 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:01:45.606 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:01:45.606 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:01:45.606 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:01:45.607 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:01:45.607 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:01:45.607 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:01:45.607 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:01:45.607 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:01:45.608 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:01:45.608 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:01:45.608 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:01:45.608 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:01:45.608 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:01:45.609 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:01:45.609 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:01:45.609 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:01:45.609 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:01:45.610 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:01:45.610 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:01:45.610 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:01:45.610 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:01:45.610 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:01:45.611 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:01:45.611 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:01:45.611 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": [ 2021-05-25T12:01:45.611 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:01:45.611 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:01:45.612 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:01:45.612 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:01:45.612 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:01:45.612 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:01:45.612 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:01:45.613 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:01:45.613 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:01:45.613 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:01:45.613 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 14 2021-05-25T12:01:45.613 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:01:45.614 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:01:45.614 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:01:45.614 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:01:45.614 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:01:45.617 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.530+0000 7f944214b700 10 librados: watch_flush enter 2021-05-25T12:01:45.617 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.530+0000 7f944214b700 10 librados: watch_flush exit 2021-05-25T12:01:45.617 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.530+0000 7f944214b700 20 client.17360.objecter shutdown clearing up homeless session... 2021-05-25T12:01:45.617 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.530+0000 7f944214b700 10 client.17360.objecter successfully canceled tick 2021-05-25T12:01:45.618 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.530+0000 7f944214b700 1 -- 172.21.15.102:0/2892010427 >> v1:172.21.15.181:6812/29582 conn(0x7f941c033150 legacy=0x7f941c035610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:45.618 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.530+0000 7f944214b700 10 monclient: shutdown 2021-05-25T12:01:45.618 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.530+0000 7f944214b700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:01:45.618 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.530+0000 7f944214b700 1 -- 172.21.15.102:0/2892010427 >> v1:172.21.15.102:6790/0 conn(0x7f943c098930 legacy=0x7f943c128ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:01:45.618 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.530+0000 7f944194a700 1 -- 172.21.15.102:0/2892010427 reap_dead start 2021-05-25T12:01:45.619 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.530+0000 7f944194a700 1 -- 172.21.15.102:0/2892010427 reap_dead start 2021-05-25T12:01:45.619 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.530+0000 7f944214b700 1 -- 172.21.15.102:0/2892010427 shutdown_connections 2021-05-25T12:01:45.619 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.530+0000 7f944214b700 1 -- 172.21.15.102:0/2892010427 >> 172.21.15.102:0/2892010427 conn(0x7f943c09e9d0 msgr2=0x7f943c12a850 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:01:45.619 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.530+0000 7f944214b700 1 -- 172.21.15.102:0/2892010427 shutdown_connections 2021-05-25T12:01:45.620 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.530+0000 7f944214b700 1 -- 172.21.15.102:0/2892010427 wait complete. 2021-05-25T12:01:45.620 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:01:45.530+0000 7f944214b700 1 librados: shutdown 2021-05-25T12:01:45.620 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2021-05-25T12:01:48.895 INFO:tasks.workunit.client.0.smithi181.stdout:tch_notify_pp: flushed 2021-05-25T12:01:48.896 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 (3007 ms) 2021-05-25T12:01:48.896 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 2021-05-25T12:01:48.896 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify_pp: trying... 2021-05-25T12:01:48.897 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify_pp: handle_notify cookie 94792742248176 notify_id 141733920773 notifier_gid 74773 2021-05-25T12:01:48.897 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify_pp: timed out 2021-05-25T12:01:48.897 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify_pp: flushing 2021-05-25T12:01:48.898 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify_pp: flushed 2021-05-25T12:01:48.898 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 (3007 ms) 2021-05-25T12:01:48.898 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 2021-05-25T12:01:48.898 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify_pp: List watches 2021-05-25T12:01:48.898 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify_pp: notify2 2021-05-25T12:01:48.899 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify_pp: handle_notify cookie 94792742248176 notify_id 154618822661 notifier_gid 74773 2021-05-25T12:01:48.899 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify_pp: notify2 done 2021-05-25T12:01:48.899 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify_pp: watch_check 2021-05-25T12:01:48.899 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify_pp: unwatch2 2021-05-25T12:01:48.899 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify_pp: flushing 2021-05-25T12:01:48.900 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify_pp: done 2021-05-25T12:01:48.900 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 (3049 ms) 2021-05-25T12:01:48.900 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 2021-05-25T12:01:48.900 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify_pp: List watches 2021-05-25T12:01:48.900 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify_pp: notify2 2021-05-25T12:01:48.901 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify_pp: handle_notify cookie 94792742241200 notify_id 167503724548 notifier_gid 74773 2021-05-25T12:01:48.901 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify_pp: notify2 done 2021-05-25T12:01:48.901 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify_pp: watch_check 2021-05-25T12:01:48.901 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify_pp: unwatch2 2021-05-25T12:01:48.901 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify_pp: flushing 2021-05-25T12:01:48.902 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify_pp: done 2021-05-25T12:01:48.902 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 (3012 ms) 2021-05-25T12:01:48.902 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP (15898 ms total) 2021-05-25T12:01:48.902 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify_pp: 2021-05-25T12:01:48.903 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify_pp: [----------] Global test environment tear-down 2021-05-25T12:01:48.903 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify_pp: [==========] 16 tests from 2 test suites ran. (69085 ms total) 2021-05-25T12:01:48.903 INFO:tasks.workunit.client.0.smithi181.stdout: api_watch_notify_pp: [ PASSED ] 16 tests. 2021-05-25T12:01:51.842 INFO:tasks.workunit.client.0.smithi181.stdout:ting): shutdown 2021-05-25T12:01:51.842 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.768+0000 7f4d7a69b540 20 monclient(hunting): shutdown discarding 1 pending message(s) 2021-05-25T12:01:51.842 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.768+0000 7f4d7a69b540 1 -- 172.21.15.181:0/149592805 >> v1:172.21.15.102:6790/0 conn(0x55ddff66abc0 legacy=0x55ddff721e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-05-25T12:01:51.843 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.768+0000 7f4d7a69b540 1 -- 172.21.15.181:0/149592805 >> v1:172.21.15.181:6789/0 conn(0x55ddff6fb9a0 legacy=0x55ddff66b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-05-25T12:01:51.843 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.768+0000 7f4d7a69b540 1 -- 172.21.15.181:0/149592805 >> v1:172.21.15.102:6789/0 conn(0x55ddff63d7b0 legacy=0x55ddff6f5580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-05-25T12:01:51.843 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.768+0000 7f4d7a69b540 10 monclient: shutdown 2021-05-25T12:01:51.843 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.768+0000 7f4d7a69b540 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:01:51.844 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.768+0000 7f4d7a69b540 1 -- 172.21.15.181:0/149592805 shutdown_connections 2021-05-25T12:01:51.844 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.768+0000 7f4d7a69b540 1 -- 172.21.15.181:0/149592805 >> 172.21.15.181:0/149592805 conn(0x55ddff66a380 msgr2=0x55ddff667190 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:01:51.844 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.768+0000 7f4d7a69b540 1 -- 172.21.15.181:0/149592805 shutdown_connections 2021-05-25T12:01:51.844 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.768+0000 7f4d7a69b540 1 -- 172.21.15.181:0/149592805 wait complete. 2021-05-25T12:01:51.844 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.768+0000 7f4d7a69b540 10 monclient: shutdown 2021-05-25T12:01:51.845 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.768+0000 7f4d7a69b540 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:01:51.845 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:00:39.768+0000 7f4d7a69b540 1 librados: shutdown 2021-05-25T12:01:51.845 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectFailure (35 ms) 2021-05-25T12:01:51.845 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: [----------] 1 test from LibRadosMiscConnectFailure (35 ms total) 2021-05-25T12:01:51.845 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:01:51.846 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: [----------] 1 test from LibRadosMiscPool 2021-05-25T12:01:51.846 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: [ RUN ] LibRadosMiscPool.PoolCreationRace 2021-05-25T12:01:51.846 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: started 0x55ddff61cca0 2021-05-25T12:01:51.846 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: started 0x55ddff5541c0 2021-05-25T12:01:51.847 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: started 2 aios 2021-05-25T12:01:51.847 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: waiting 0x55ddff61cca0 2021-05-25T12:01:51.847 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: waiting 0x55ddff5541c0 2021-05-25T12:01:51.847 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: done. 2021-05-25T12:01:51.847 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: [ OK ] LibRadosMiscPool.PoolCreationRace (20634 ms) 2021-05-25T12:01:51.848 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: [----------] 1 test from LibRadosMiscPool (20634 ms total) 2021-05-25T12:01:51.848 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:01:51.848 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: [----------] 8 tests from LibRadosMisc 2021-05-25T12:01:51.848 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: [ RUN ] LibRadosMisc.ClusterFSID 2021-05-25T12:01:51.848 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: [ OK ] LibRadosMisc.ClusterFSID (0 ms) 2021-05-25T12:01:51.849 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: [ RUN ] LibRadosMisc.Exec 2021-05-25T12:01:51.849 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: [ OK ] LibRadosMisc.Exec (58 ms) 2021-05-25T12:01:51.849 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: [ RUN ] LibRadosMisc.WriteSame 2021-05-25T12:01:51.849 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: [ OK ] LibRadosMisc.WriteSame (16 ms) 2021-05-25T12:01:51.850 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: [ RUN ] LibRadosMisc.CmpExt 2021-05-25T12:01:51.850 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: [ OK ] LibRadosMisc.CmpExt (3 ms) 2021-05-25T12:01:51.850 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: [ RUN ] LibRadosMisc.Applications 2021-05-25T12:01:51.851 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: [ OK ] LibRadosMisc.Applications (4597 ms) 2021-05-25T12:01:51.851 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatOSD 2021-05-25T12:01:51.851 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatOSD (0 ms) 2021-05-25T12:01:51.851 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatClient 2021-05-25T12:01:51.851 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatClient (0 ms) 2021-05-25T12:01:51.852 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: [ RUN ] LibRadosMisc.ShutdownRace 2021-05-25T12:01:51.852 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: [ OK ] LibRadosMisc.ShutdownRace (44271 ms) 2021-05-25T12:01:51.852 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: [----------] 8 tests from LibRadosMisc (48945 ms total) 2021-05-25T12:01:51.852 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: 2021-05-25T12:01:51.853 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: [----------] Global test environment tear-d api_pool: Running main() from gmock_main.cc 2021-05-25T12:01:51.853 INFO:tasks.workunit.client.0.smithi181.stdout: api_pool: [==========] Running 7 tests from 1 test suite. 2021-05-25T12:01:51.853 INFO:tasks.workunit.client.0.smithi181.stdout: api_pool: [----------] Global test environment set-up. 2021-05-25T12:01:51.854 INFO:tasks.workunit.client.0.smithi181.stdout: api_pool: [----------] 7 tests from LibRadosPools 2021-05-25T12:01:51.854 INFO:tasks.workunit.client.0.smithi181.stdout: api_pool: [ RUN ] LibRadosPools.PoolList 2021-05-25T12:01:51.854 INFO:tasks.workunit.client.0.smithi181.stdout: api_pool: [ OK ] LibRadosPools.PoolList (17628 ms) 2021-05-25T12:01:51.854 INFO:tasks.workunit.client.0.smithi181.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup 2021-05-25T12:01:51.855 INFO:tasks.workunit.client.0.smithi181.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup (3007 ms) 2021-05-25T12:01:51.855 INFO:tasks.workunit.client.0.smithi181.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup2 2021-05-25T12:01:51.855 INFO:tasks.workunit.client.0.smithi181.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup2 (2995 ms) 2021-05-25T12:01:51.855 INFO:tasks.workunit.client.0.smithi181.stdout: api_pool: [ RUN ] LibRadosPools.PoolDelete 2021-05-25T12:01:51.855 INFO:tasks.workunit.client.0.smithi181.stdout: api_pool: [ OK ] LibRadosPools.PoolDelete (4656 ms) 2021-05-25T12:01:51.856 INFO:tasks.workunit.client.0.smithi181.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateDelete 2021-05-25T12:01:51.856 INFO:tasks.workunit.client.0.smithi181.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateDelete (5045 ms) 2021-05-25T12:01:51.856 INFO:tasks.workunit.client.0.smithi181.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateWithCrushRule 2021-05-25T12:01:51.856 INFO:tasks.workunit.client.0.smithi181.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateWithCrushRule (5064 ms) 2021-05-25T12:01:51.857 INFO:tasks.workunit.client.0.smithi181.stdout: api_pool: [ RUN ] LibRadosPools.PoolGetBaseTier 2021-05-25T12:01:51.857 INFO:tasks.workunit.client.0.smithi181.stdout: api_pool: [ OK ] LibRadosPools.PoolGetBaseTier (33688 ms) 2021-05-25T12:01:51.857 INFO:tasks.workunit.client.0.smithi181.stdout: api_pool: [----------] 7 tests from LibRadosPools (72083 ms total) 2021-05-25T12:01:51.857 INFO:tasks.workunit.client.0.smithi181.stdout: api_pool: 2021-05-25T12:01:51.858 INFO:tasks.workunit.client.0.smithi181.stdout: api_pool: [----------] Global test environment tear-down 2021-05-25T12:01:51.858 INFO:tasks.workunit.client.0.smithi181.stdout: api_pool: [==========] 7 tests from 1 test suite ran. (72083 ms total) 2021-05-25T12:01:51.858 INFO:tasks.workunit.client.0.smithi181.stdout: api_pool: [ PASSED ] 7 tests. 2021-05-25T12:01:51.858 INFO:tasks.workunit.client.0.smithi181.stdout:own 2021-05-25T12:01:51.859 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: [==========] 11 tests from 4 test suites ran. (72106 ms total) 2021-05-25T12:01:51.859 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc: [ PASSED ] 11 tests. 2021-05-25T12:01:54.609 INFO:tasks.ceph.osd.4.smithi181.stderr:2021-05-25T12:01:54.606+0000 7f0006e6f700 -1 osd.4 pg_epoch: 51 pg[62.d( v 50'3 (0'0,50'3] local-lis/les=32/33 n=2 ec=32/32 lis/c=32/32 les/c/f=33/33/0 sis=32) [4,3] r=0 lpr=32 crt=50'3 lcod 49'1 mlcod 49'1 active+clean+snaptrim trimq=[13~1] ps=[b~1,d~2]] removing snap head 2021-05-25T12:01:55.857 INFO:tasks.ceph.osd.5.smithi181.stderr:2021-05-25T12:01:55.850+0000 7f04714c7700 -1 osd.5 pg_epoch: 53 pg[58.2( v 49'3 (0'0,49'3] local-lis/les=30/31 n=2 ec=30/30 lis/c=30/30 les/c/f=31/31/0 sis=30) [5,4] r=0 lpr=30 crt=49'3 lcod 48'1 mlcod 48'1 active+clean+snaptrim trimq=[2~1]] removing snap head 2021-05-25T12:01:55.858 INFO:tasks.ceph.osd.0.smithi102.stderr:2021-05-25T12:01:55.850+0000 7f38ddb4b700 -1 osd.0 pg_epoch: 53 pg[81.2( v 52'2 (0'0,52'2] local-lis/les=47/48 n=2 ec=47/47 lis/c=47/47 les/c/f=48/48/0 sis=47) [0,2] r=0 lpr=47 crt=52'2 lcod 52'1 mlcod 52'1 active+clean+snaptrim trimq=[2~1]] removing snap head 2021-05-25T12:02:05.621 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2021-05-25T12:02:05.621 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2021-05-25T12:02:05.621 INFO:tasks.ceph.mon.a:Restarting daemon 2021-05-25T12:02:05.622 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2021-05-25T12:02:05.625 INFO:tasks.ceph.mon.a:Started 2021-05-25T12:02:05.625 INFO:tasks.mon_thrash:Sending STOP to mon c 2021-05-25T12:02:05.626 INFO:tasks.ceph.mon.c:Sent signal 19 2021-05-25T12:02:05.626 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs to unfreeze mons 2021-05-25T12:02:25.627 INFO:tasks.mon_thrash:Sending CONT to mon c 2021-05-25T12:02:25.628 INFO:tasks.ceph.mon.c:Sent signal 18 2021-05-25T12:02:25.628 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2021-05-25T12:02:25.629 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2021-05-25T12:02:25.746 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.744+0000 7f76bfdf8700 10 monclient: get_monmap_and_config 2021-05-25T12:02:25.747 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.744+0000 7f76bfdf8700 10 monclient: build_initial_monmap 2021-05-25T12:02:25.747 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.744+0000 7f76bfdf8700 10 monclient: monmap: 2021-05-25T12:02:25.747 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:02:25.747 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:25.748 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:02:25.746105+0000 2021-05-25T12:02:25.748 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:02:25.746105+0000 2021-05-25T12:02:25.748 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:02:25.748 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:25.749 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:02:25.749 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:02:25.749 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:02:25.749 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:25.750 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.744+0000 7f76bfdf8700 1 Processor -- start 2021-05-25T12:02:25.750 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.744+0000 7f76bfdf8700 1 -- start start 2021-05-25T12:02:25.750 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.744+0000 7f76bfdf8700 10 monclient: init 2021-05-25T12:02:25.759 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.745+0000 7f76bfdf8700 10 monclient: _reopen_session rank -1 2021-05-25T12:02:25.760 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.745+0000 7f76bfdf8700 10 monclient: _add_conns ranks=[0,1,2] 2021-05-25T12:02:25.760 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.745+0000 7f76bfdf8700 10 monclient(hunting): picked mon.noname-a con 0x7f76b8090990 addr v1:172.21.15.102:6789/0 2021-05-25T12:02:25.760 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.745+0000 7f76bfdf8700 10 monclient(hunting): picked mon.noname-c con 0x7f76b8090d70 addr v1:172.21.15.181:6789/0 2021-05-25T12:02:25.760 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.745+0000 7f76bfdf8700 10 monclient(hunting): picked mon.noname-b con 0x7f76b812e330 addr v1:172.21.15.102:6790/0 2021-05-25T12:02:25.761 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.745+0000 7f76bfdf8700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76b8096fa0 con 0x7f76b8090990 2021-05-25T12:02:25.761 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.745+0000 7f76bfdf8700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76b80926a0 con 0x7f76b8090d70 2021-05-25T12:02:25.761 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.745+0000 7f76bfdf8700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76b8092940 con 0x7f76b812e330 2021-05-25T12:02:25.761 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.745+0000 7f76bfdf8700 10 monclient(hunting): _renew_subs 2021-05-25T12:02:25.762 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.745+0000 7f76bfdf8700 10 monclient(hunting): authenticate will time out at 946.343404s 2021-05-25T12:02:25.762 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.745+0000 7f76be395700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f76b812e330 0x7f76b8130720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:54980/0 (socket says 172.21.15.102:54980) 2021-05-25T12:02:25.762 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.745+0000 7f76be395700 1 -- 172.21.15.102:0/737553949 learned_addr learned my addr 172.21.15.102:0/737553949 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:02:25.762 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.745+0000 7f76bdb94700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f76b8090990 0x7f76b8093240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60102/0 (socket says 172.21.15.102:60102) 2021-05-25T12:02:25.762 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.745+0000 7f76bcb92700 1 -- 172.21.15.102:0/737553949 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2579634381 0 0) 0x7f76b8096fa0 con 0x7f76b8090990 2021-05-25T12:02:25.763 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.745+0000 7f76bcb92700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:02:25.763 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.745+0000 7f76bcb92700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:02:25.763 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.745+0000 7f76bcb92700 10 monclient(hunting): my global_id is 174114 2021-05-25T12:02:25.763 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.745+0000 7f76bcb92700 1 -- 172.21.15.102:0/737553949 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76a0003270 con 0x7f76b8090990 2021-05-25T12:02:25.764 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.745+0000 7f76bcb92700 1 -- 172.21.15.102:0/737553949 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3768416557 0 0) 0x7f76b80926a0 con 0x7f76b8090d70 2021-05-25T12:02:25.764 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.745+0000 7f76bcb92700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:02:25.764 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.745+0000 7f76bcb92700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:02:25.764 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.745+0000 7f76bcb92700 10 monclient(hunting): my global_id is 95065 2021-05-25T12:02:25.764 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.745+0000 7f76bcb92700 1 -- 172.21.15.102:0/737553949 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76b8096fa0 con 0x7f76b8090d70 2021-05-25T12:02:25.765 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.746+0000 7f76bcb92700 1 -- 172.21.15.102:0/737553949 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2272638807 0 0) 0x7f76a0003270 con 0x7f76b8090990 2021-05-25T12:02:25.765 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.746+0000 7f76bcb92700 1 -- 172.21.15.102:0/737553949 >> v1:172.21.15.102:6790/0 conn(0x7f76b812e330 legacy=0x7f76b8130720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:25.765 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.746+0000 7f76bcb92700 1 -- 172.21.15.102:0/737553949 >> v1:172.21.15.181:6789/0 conn(0x7f76b8090d70 legacy=0x7f76b8091200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:25.765 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.746+0000 7f76bcb92700 10 monclient: _finish_hunting 0 2021-05-25T12:02:25.765 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.746+0000 7f76bcb92700 1 monclient: found mon.noname-a 2021-05-25T12:02:25.766 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.746+0000 7f76bcb92700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:02:25.766 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.746+0000 7f76bcb92700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T12:02:25.766 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.746+0000 7f76bcb92700 1 -- 172.21.15.102:0/737553949 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76b8092be0 con 0x7f76b8090990 2021-05-25T12:02:25.766 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.746+0000 7f76bcb92700 10 monclient: _finish_auth 0 2021-05-25T12:02:25.767 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.746+0000 7f76bcb92700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:02:25.767 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.746+0000 7f76bfdf8700 5 monclient: authenticate success, global_id 174114 2021-05-25T12:02:25.767 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.746+0000 7f76bfdf8700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:02:25.767 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.746+0000 7f76bcb92700 1 -- 172.21.15.102:0/737553949 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f76a8001610 con 0x7f76b8090990 2021-05-25T12:02:25.768 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.746+0000 7f76bcb92700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:25.768 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.746+0000 7f76bcb92700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:02:25.768 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.746+0000 7f76bcb92700 10 monclient: dump: 2021-05-25T12:02:25.768 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:25.768 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:25.769 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:25.770 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:25.770 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:25.770 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:25.770 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:25.771 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:25.771 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:25.771 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:25.771 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.746+0000 7f76bfdf8700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:02:25.771 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.746+0000 7f76bcb92700 1 -- 172.21.15.102:0/737553949 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f76a8001880 con 0x7f76b8090990 2021-05-25T12:02:25.772 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.746+0000 7f76bcb92700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:02:25.772 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.746+0000 7f76bfdf8700 10 monclient: get_monmap_and_config success 2021-05-25T12:02:25.772 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.746+0000 7f76bcb92700 1 -- 172.21.15.102:0/737553949 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f76a8001c10 con 0x7f76b8090990 2021-05-25T12:02:25.772 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.746+0000 7f76bcb92700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:25.773 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.746+0000 7f76bfdf8700 4 set_mon_vals no callback set 2021-05-25T12:02:25.773 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.746+0000 7f76bcb92700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:02:25.773 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.746+0000 7f76bcb92700 10 monclient: dump: 2021-05-25T12:02:25.773 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:25.774 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:25.774 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:25.774 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:25.775 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:25.775 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:25.775 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:25.776 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:25.776 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:25.776 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:25.776 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.746+0000 7f76bfdf8700 10 monclient: shutdown 2021-05-25T12:02:25.777 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.746+0000 7f76bfdf8700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:02:25.777 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.746+0000 7f76bfdf8700 1 -- 172.21.15.102:0/737553949 >> v1:172.21.15.102:6789/0 conn(0x7f76b8090990 legacy=0x7f76b8093240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:25.777 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.746+0000 7f76bfdf8700 1 -- 172.21.15.102:0/737553949 shutdown_connections 2021-05-25T12:02:25.777 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.746+0000 7f76bfdf8700 1 -- 172.21.15.102:0/737553949 >> 172.21.15.102:0/737553949 conn(0x7f76b809b370 msgr2=0x7f76b8097430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:02:25.778 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.746+0000 7f76bfdf8700 1 -- 172.21.15.102:0/737553949 shutdown_connections 2021-05-25T12:02:25.778 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.746+0000 7f76bfdf8700 1 -- 172.21.15.102:0/737553949 wait complete. 2021-05-25T12:02:25.778 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.747+0000 7f76bfdf8700 10 monclient: build_initial_monmap 2021-05-25T12:02:25.778 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.747+0000 7f76bfdf8700 10 monclient: monmap: 2021-05-25T12:02:25.779 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:02:25.779 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:02:25.779 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:02:25.779 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:02:25.779 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:02:25.780 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:02:25.780 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:02:25.780 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:02:25.780 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:02:25.780 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:25.781 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.747+0000 7f76bfdf8700 1 librados: starting msgr at 2021-05-25T12:02:25.781 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.747+0000 7f76bfdf8700 1 librados: starting objecter 2021-05-25T12:02:25.781 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.747+0000 7f76bfdf8700 1 Processor -- start 2021-05-25T12:02:25.781 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.747+0000 7f76bfdf8700 1 -- start start 2021-05-25T12:02:25.782 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.747+0000 7f76bfdf8700 1 librados: setting wanted keys 2021-05-25T12:02:25.782 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.747+0000 7f76bfdf8700 1 librados: calling monclient init 2021-05-25T12:02:25.782 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.747+0000 7f76bfdf8700 10 monclient: init 2021-05-25T12:02:25.782 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.748+0000 7f76bfdf8700 10 monclient: _reopen_session rank -1 2021-05-25T12:02:25.783 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.748+0000 7f76bfdf8700 10 monclient: _add_conns ranks=[0,2,1] 2021-05-25T12:02:25.783 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.748+0000 7f76bfdf8700 10 monclient(hunting): picked mon.noname-a con 0x7f76b8090990 addr v1:172.21.15.102:6789/0 2021-05-25T12:02:25.783 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.748+0000 7f76bfdf8700 10 monclient(hunting): picked mon.noname-c con 0x7f76b8090d70 addr v1:172.21.15.102:6790/0 2021-05-25T12:02:25.783 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.748+0000 7f76bfdf8700 10 monclient(hunting): picked mon.noname-b con 0x7f76b812e330 addr v1:172.21.15.181:6789/0 2021-05-25T12:02:25.783 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.748+0000 7f76bfdf8700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76b8134070 con 0x7f76b8090990 2021-05-25T12:02:25.784 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.748+0000 7f76bfdf8700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76b8113670 con 0x7f76b812e330 2021-05-25T12:02:25.784 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.748+0000 7f76bfdf8700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76b8113910 con 0x7f76b8090d70 2021-05-25T12:02:25.784 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.748+0000 7f76bfdf8700 10 monclient(hunting): _renew_subs 2021-05-25T12:02:25.784 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.748+0000 7f76bfdf8700 10 monclient(hunting): authenticate will time out at 946.346158s 2021-05-25T12:02:25.784 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.748+0000 7f76bdb94700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f76b8090990 0x7f76b812bf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60106/0 (socket says 172.21.15.102:60106) 2021-05-25T12:02:25.785 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.748+0000 7f76bdb94700 1 -- 172.21.15.102:0/3108471665 learned_addr learned my addr 172.21.15.102:0/3108471665 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:02:25.785 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.748+0000 7f76a5ffb700 10 client.?.objecter ms_handle_connect 0x7f76b8090990 2021-05-25T12:02:25.785 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.748+0000 7f76a5ffb700 10 client.?.objecter resend_mon_ops 2021-05-25T12:02:25.785 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.748+0000 7f76a5ffb700 10 client.?.objecter ms_handle_connect 0x7f76b8090d70 2021-05-25T12:02:25.786 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.748+0000 7f76a5ffb700 10 client.?.objecter resend_mon_ops 2021-05-25T12:02:25.786 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.748+0000 7f76a5ffb700 1 -- 172.21.15.102:0/3108471665 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1999548716 0 0) 0x7f76b8134070 con 0x7f76b8090990 2021-05-25T12:02:25.786 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.748+0000 7f76a5ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:02:25.786 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.748+0000 7f76a5ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:02:25.786 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.748+0000 7f76a5ffb700 10 monclient(hunting): my global_id is 174117 2021-05-25T12:02:25.787 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.748+0000 7f76a5ffb700 1 -- 172.21.15.102:0/3108471665 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7698004c10 con 0x7f76b8090990 2021-05-25T12:02:25.787 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.748+0000 7f76a5ffb700 10 client.?.objecter ms_handle_connect 0x7f76b812e330 2021-05-25T12:02:25.787 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.748+0000 7f76a5ffb700 10 client.?.objecter resend_mon_ops 2021-05-25T12:02:25.787 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.748+0000 7f76a5ffb700 1 -- 172.21.15.102:0/3108471665 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2142254889 0 0) 0x7f7698004c10 con 0x7f76b8090990 2021-05-25T12:02:25.788 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.748+0000 7f76a5ffb700 1 -- 172.21.15.102:0/3108471665 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f76b8134070 con 0x7f76b8090990 2021-05-25T12:02:25.788 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.748+0000 7f76a5ffb700 1 -- 172.21.15.102:0/3108471665 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f76a8001690 con 0x7f76b8090990 2021-05-25T12:02:25.788 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.748+0000 7f76a5ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:25.788 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.748+0000 7f76a5ffb700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:02:25.788 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.748+0000 7f76a5ffb700 10 monclient(hunting): dump: 2021-05-25T12:02:25.789 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:25.789 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:25.789 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:25.789 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:25.790 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:25.790 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:25.790 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:25.790 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:25.791 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:25.791 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:25.791 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.748+0000 7f76a5ffb700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:02:25.791 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.748+0000 7f76a5ffb700 1 -- 172.21.15.102:0/3108471665 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 858020568 0 0) 0x7f76b8113670 con 0x7f76b812e330 2021-05-25T12:02:25.792 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.748+0000 7f76a5ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:02:25.792 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.748+0000 7f76a5ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:02:25.792 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.748+0000 7f76a5ffb700 10 monclient(hunting): my global_id is 95068 2021-05-25T12:02:25.792 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.748+0000 7f76a5ffb700 1 -- 172.21.15.102:0/3108471665 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7698004c10 con 0x7f76b812e330 2021-05-25T12:02:25.793 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.748+0000 7f76a5ffb700 1 -- 172.21.15.102:0/3108471665 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2193116128 0 0) 0x7f76b8134070 con 0x7f76b8090990 2021-05-25T12:02:25.793 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.748+0000 7f76a5ffb700 1 -- 172.21.15.102:0/3108471665 >> v1:172.21.15.102:6790/0 conn(0x7f76b8090d70 legacy=0x7f76b813dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:25.793 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.748+0000 7f76a5ffb700 1 -- 172.21.15.102:0/3108471665 >> v1:172.21.15.181:6789/0 conn(0x7f76b812e330 legacy=0x7f76b8141610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:25.794 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.748+0000 7f76a5ffb700 10 monclient: _finish_hunting 0 2021-05-25T12:02:25.794 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.748+0000 7f76a5ffb700 1 monclient: found mon.a 2021-05-25T12:02:25.794 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.748+0000 7f76a5ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:02:25.795 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.748+0000 7f76a5ffb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:02:25.795 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.748+0000 7f76a5ffb700 1 -- 172.21.15.102:0/3108471665 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76b8091fe0 con 0x7f76b8090990 2021-05-25T12:02:25.796 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.748+0000 7f76a5ffb700 10 monclient: _finish_auth 0 2021-05-25T12:02:25.796 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.748+0000 7f76a5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:02:25.797 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.748+0000 7f76bfdf8700 5 monclient: authenticate success, global_id 174117 2021-05-25T12:02:25.797 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.748+0000 7f76bfdf8700 10 monclient: _renew_subs 2021-05-25T12:02:25.797 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.748+0000 7f76bfdf8700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:02:25.798 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.748+0000 7f76bfdf8700 1 -- 172.21.15.102:0/3108471665 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76b8111480 con 0x7f76b8090990 2021-05-25T12:02:25.798 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.749+0000 7f76a5ffb700 1 -- 172.21.15.102:0/3108471665 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f76a8004a60 con 0x7f76b8090990 2021-05-25T12:02:25.799 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.749+0000 7f76a5ffb700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:02:25.799 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.749+0000 7f76a5ffb700 1 -- 172.21.15.102:0/3108471665 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f76a8004df0 con 0x7f76b8090990 2021-05-25T12:02:25.800 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.749+0000 7f76a5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:25.800 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.749+0000 7f76a5ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:02:25.800 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.749+0000 7f76a5ffb700 10 monclient: dump: 2021-05-25T12:02:25.801 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:25.801 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:25.801 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:25.802 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:25.802 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:25.803 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:25.803 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:25.803 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:25.804 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:25.804 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:25.804 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.749+0000 7f76bfdf8700 10 client.174117.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:02:25.805 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.749+0000 7f76bfdf8700 10 monclient: _renew_subs 2021-05-25T12:02:25.805 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.749+0000 7f76a7fff700 4 set_mon_vals no callback set 2021-05-25T12:02:25.805 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.749+0000 7f76bfdf8700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:02:25.805 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.749+0000 7f76bfdf8700 1 -- 172.21.15.102:0/3108471665 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f76b8133770 con 0x7f76b8090990 2021-05-25T12:02:25.806 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.749+0000 7f76bfdf8700 1 librados: init done 2021-05-25T12:02:25.806 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.749+0000 7f76a5ffb700 1 -- 172.21.15.102:0/3108471665 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f76a8004fd0 con 0x7f76b8090990 2021-05-25T12:02:25.806 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.749+0000 7f76bfdf8700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:02:25.806 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.749+0000 7f76bfdf8700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:02:25.807 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.749+0000 7f76bfdf8700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:02:25.807 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.749+0000 7f76bfdf8700 1 -- 172.21.15.102:0/3108471665 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76b8059730 con 0x7f76b8090990 2021-05-25T12:02:25.807 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.749+0000 7f76a5ffb700 1 -- 172.21.15.102:0/3108471665 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(68..68 src has 1..68) v4 ==== 8198+0+0 (unknown 272484497 0 0) 0x7f76a803e410 con 0x7f76b8090990 2021-05-25T12:02:25.807 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.749+0000 7f76a5ffb700 10 client.174117.objecter ms_dispatch 0x7f76b8001550 osd_map(68..68 src has 1..68) v4 2021-05-25T12:02:25.808 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.749+0000 7f76a5ffb700 3 client.174117.objecter handle_osd_map got epochs [68,68] > 0 2021-05-25T12:02:25.808 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.749+0000 7f76a5ffb700 3 client.174117.objecter handle_osd_map decoding full epoch 68 2021-05-25T12:02:25.808 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.750+0000 7f76a5ffb700 20 client.174117.objecter dump_active .. 0 homeless 2021-05-25T12:02:25.808 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.750+0000 7f76bd393700 10 monclient: get_auth_request con 0x7f76980331e0 auth_method 0 2021-05-25T12:02:25.809 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.750+0000 7f76a5ffb700 10 client.174117.objecter ms_handle_connect 0x7f76980331e0 2021-05-25T12:02:25.809 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.754+0000 7f76a5ffb700 1 -- 172.21.15.102:0/3108471665 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7f76a8013390 con 0x7f76b8090990 2021-05-25T12:02:25.809 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.754+0000 7f76a5ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:02:25.809 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.754+0000 7f76a5ffb700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:02:25.967 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.964+0000 7f76bfdf8700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2021-05-25T12:02:25.967 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.964+0000 7f76bfdf8700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2021-05-25T12:02:25.967 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.964+0000 7f76bfdf8700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:02:25.968 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.964+0000 7f76bfdf8700 1 -- 172.21.15.102:0/3108471665 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f76b8096500 con 0x7f76b8090990 2021-05-25T12:02:25.968 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.964+0000 7f76a5ffb700 1 -- 172.21.15.102:0/3108471665 <== mon.0 v1:172.21.15.102:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1336 (unknown 254746657 0 1604559632) 0x7f76a8013390 con 0x7f76b8090990 2021-05-25T12:02:25.968 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.964+0000 7f76a5ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2021-05-25T12:02:25.968 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.964+0000 7f76a5ffb700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:02:25.969 INFO:teuthology.orchestra.run.smithi102.stdout:{"election_epoch":26,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"b","quorum_age":6,"features":{"quorum_con":"4540138303579357183","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"20ee91b6-8935-4b98-b868-85c03789ccc4","modified":"2021-05-25T11:59:11.011522Z","created":"2021-05-25T11:59:11.011522Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6790","nonce":0}]},"addr":"172.21.15.102:6790/0","public_addr":"172.21.15.102:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2021-05-25T12:02:25.970 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.966+0000 7f76bfdf8700 10 librados: watch_flush enter 2021-05-25T12:02:25.970 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.966+0000 7f76bfdf8700 10 librados: watch_flush exit 2021-05-25T12:02:25.971 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.966+0000 7f76bfdf8700 20 client.174117.objecter shutdown clearing up homeless session... 2021-05-25T12:02:25.971 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.966+0000 7f76bfdf8700 10 client.174117.objecter successfully canceled tick 2021-05-25T12:02:25.971 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.966+0000 7f76bfdf8700 1 -- 172.21.15.102:0/3108471665 >> v1:172.21.15.181:6812/29582 conn(0x7f76980331e0 legacy=0x7f76980356a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:25.971 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.966+0000 7f76bfdf8700 10 monclient: shutdown 2021-05-25T12:02:25.972 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.966+0000 7f76bfdf8700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:02:25.972 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.966+0000 7f76bfdf8700 1 -- 172.21.15.102:0/3108471665 >> v1:172.21.15.102:6789/0 conn(0x7f76b8090990 legacy=0x7f76b812bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:25.972 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.966+0000 7f76bfdf8700 1 -- 172.21.15.102:0/3108471665 shutdown_connections 2021-05-25T12:02:25.972 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.966+0000 7f76bfdf8700 1 -- 172.21.15.102:0/3108471665 >> 172.21.15.102:0/3108471665 conn(0x7f76b809b370 msgr2=0x7f76b8093a60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:02:25.972 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.966+0000 7f76bfdf8700 1 -- 172.21.15.102:0/3108471665 shutdown_connections 2021-05-25T12:02:25.973 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.966+0000 7f76bfdf8700 1 -- 172.21.15.102:0/3108471665 wait complete. 2021-05-25T12:02:25.973 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:25.966+0000 7f76bfdf8700 1 librados: shutdown 2021-05-25T12:02:28.981 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2021-05-25T12:02:29.098 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.097+0000 7f98eb469700 10 monclient: get_monmap_and_config 2021-05-25T12:02:29.099 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.097+0000 7f98eb469700 10 monclient: build_initial_monmap 2021-05-25T12:02:29.099 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.097+0000 7f98eb469700 10 monclient: monmap: 2021-05-25T12:02:29.099 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:02:29.099 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:29.100 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:02:29.098323+0000 2021-05-25T12:02:29.100 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:02:29.098323+0000 2021-05-25T12:02:29.100 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:02:29.100 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:29.100 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:02:29.101 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:02:29.101 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:02:29.101 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:29.111 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.097+0000 7f98eb469700 1 Processor -- start 2021-05-25T12:02:29.111 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.097+0000 7f98eb469700 1 -- start start 2021-05-25T12:02:29.111 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.097+0000 7f98eb469700 10 monclient: init 2021-05-25T12:02:29.112 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.097+0000 7f98eb469700 10 monclient: _reopen_session rank -1 2021-05-25T12:02:29.112 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.097+0000 7f98eb469700 10 monclient: _add_conns ranks=[2,0,1] 2021-05-25T12:02:29.112 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.097+0000 7f98eb469700 10 monclient(hunting): picked mon.noname-b con 0x7f98e4090990 addr v1:172.21.15.102:6790/0 2021-05-25T12:02:29.112 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.097+0000 7f98eb469700 10 monclient(hunting): picked mon.noname-a con 0x7f98e4090d70 addr v1:172.21.15.102:6789/0 2021-05-25T12:02:29.112 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.097+0000 7f98eb469700 10 monclient(hunting): picked mon.noname-c con 0x7f98e412e330 addr v1:172.21.15.181:6789/0 2021-05-25T12:02:29.113 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.097+0000 7f98eb469700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98e4096fa0 con 0x7f98e4090d70 2021-05-25T12:02:29.113 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.097+0000 7f98eb469700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98e40926a0 con 0x7f98e412e330 2021-05-25T12:02:29.113 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.097+0000 7f98eb469700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98e4092940 con 0x7f98e4090990 2021-05-25T12:02:29.113 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.097+0000 7f98eb469700 10 monclient(hunting): _renew_subs 2021-05-25T12:02:29.114 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.097+0000 7f98eb469700 10 monclient(hunting): authenticate will time out at 949.695605s 2021-05-25T12:02:29.114 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.097+0000 7f98e9205700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f98e4090990 0x7f98e4093240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:54996/0 (socket says 172.21.15.102:54996) 2021-05-25T12:02:29.114 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.097+0000 7f98e9205700 1 -- 172.21.15.102:0/842314618 learned_addr learned my addr 172.21.15.102:0/842314618 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:02:29.114 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.098+0000 7f98d3fff700 1 -- 172.21.15.102:0/842314618 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3959737453 0 0) 0x7f98e4096fa0 con 0x7f98e4090d70 2021-05-25T12:02:29.114 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.098+0000 7f98d3fff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:02:29.115 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.098+0000 7f98d3fff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:02:29.115 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.098+0000 7f98d3fff700 10 monclient(hunting): my global_id is 174132 2021-05-25T12:02:29.115 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.098+0000 7f98d3fff700 1 -- 172.21.15.102:0/842314618 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98cc003270 con 0x7f98e4090d70 2021-05-25T12:02:29.115 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.098+0000 7f98d3fff700 1 -- 172.21.15.102:0/842314618 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3981047879 0 0) 0x7f98cc003270 con 0x7f98e4090d70 2021-05-25T12:02:29.116 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.098+0000 7f98d3fff700 1 -- 172.21.15.102:0/842314618 >> v1:172.21.15.102:6790/0 conn(0x7f98e4090990 legacy=0x7f98e4093240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:29.116 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.098+0000 7f98d3fff700 1 -- 172.21.15.102:0/842314618 >> v1:172.21.15.181:6789/0 conn(0x7f98e412e330 legacy=0x7f98e4130720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:29.116 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.098+0000 7f98d3fff700 10 monclient: _finish_hunting 0 2021-05-25T12:02:29.116 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.098+0000 7f98d3fff700 1 monclient: found mon.noname-a 2021-05-25T12:02:29.116 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.098+0000 7f98d3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:02:29.117 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.098+0000 7f98d3fff700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T12:02:29.117 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.098+0000 7f98d3fff700 1 -- 172.21.15.102:0/842314618 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98e4092be0 con 0x7f98e4090d70 2021-05-25T12:02:29.117 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.098+0000 7f98d3fff700 10 monclient: _finish_auth 0 2021-05-25T12:02:29.117 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.098+0000 7f98d3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:02:29.118 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.098+0000 7f98eb469700 5 monclient: authenticate success, global_id 174132 2021-05-25T12:02:29.118 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.098+0000 7f98eb469700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:02:29.118 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.098+0000 7f98d3fff700 1 -- 172.21.15.102:0/842314618 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f98d8001610 con 0x7f98e4090d70 2021-05-25T12:02:29.118 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.098+0000 7f98d3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:29.118 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.098+0000 7f98d3fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:02:29.119 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.098+0000 7f98d3fff700 10 monclient: dump: 2021-05-25T12:02:29.119 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:29.119 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:29.119 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:29.119 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:29.120 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:29.120 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:29.120 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:29.120 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:29.120 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:29.121 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:29.121 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.098+0000 7f98eb469700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:02:29.121 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.098+0000 7f98d3fff700 1 -- 172.21.15.102:0/842314618 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f98d80018c0 con 0x7f98e4090d70 2021-05-25T12:02:29.121 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.098+0000 7f98d3fff700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:02:29.121 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.098+0000 7f98eb469700 10 monclient: get_monmap_and_config success 2021-05-25T12:02:29.122 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.098+0000 7f98d3fff700 1 -- 172.21.15.102:0/842314618 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f98d8001c50 con 0x7f98e4090d70 2021-05-25T12:02:29.122 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.098+0000 7f98d3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:29.122 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.098+0000 7f98eb469700 4 set_mon_vals no callback set 2021-05-25T12:02:29.122 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.098+0000 7f98d3fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:02:29.123 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.098+0000 7f98d3fff700 10 monclient: dump: 2021-05-25T12:02:29.123 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:29.123 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:29.123 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:29.123 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:29.124 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:29.124 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:29.124 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:29.124 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:29.124 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:29.125 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:29.125 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.098+0000 7f98eb469700 10 monclient: shutdown 2021-05-25T12:02:29.125 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.098+0000 7f98eb469700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:02:29.125 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.098+0000 7f98eb469700 1 -- 172.21.15.102:0/842314618 >> v1:172.21.15.102:6789/0 conn(0x7f98e4090d70 legacy=0x7f98e4091200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:29.125 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.098+0000 7f98eb469700 1 -- 172.21.15.102:0/842314618 shutdown_connections 2021-05-25T12:02:29.126 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.098+0000 7f98eb469700 1 -- 172.21.15.102:0/842314618 >> 172.21.15.102:0/842314618 conn(0x7f98e409b370 msgr2=0x7f98e4097430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:02:29.126 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.099+0000 7f98eb469700 1 -- 172.21.15.102:0/842314618 shutdown_connections 2021-05-25T12:02:29.126 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.099+0000 7f98eb469700 1 -- 172.21.15.102:0/842314618 wait complete. 2021-05-25T12:02:29.126 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.099+0000 7f98eb469700 10 monclient: build_initial_monmap 2021-05-25T12:02:29.127 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.099+0000 7f98eb469700 10 monclient: monmap: 2021-05-25T12:02:29.127 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:02:29.127 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:02:29.127 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:02:29.127 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:02:29.128 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:02:29.128 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:02:29.128 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:02:29.128 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:02:29.128 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:02:29.128 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:29.129 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.099+0000 7f98eb469700 1 librados: starting msgr at 2021-05-25T12:02:29.129 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.099+0000 7f98eb469700 1 librados: starting objecter 2021-05-25T12:02:29.129 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.099+0000 7f98eb469700 1 Processor -- start 2021-05-25T12:02:29.129 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.099+0000 7f98eb469700 1 -- start start 2021-05-25T12:02:29.129 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.099+0000 7f98eb469700 1 librados: setting wanted keys 2021-05-25T12:02:29.130 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.099+0000 7f98eb469700 1 librados: calling monclient init 2021-05-25T12:02:29.130 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.099+0000 7f98eb469700 10 monclient: init 2021-05-25T12:02:29.130 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.100+0000 7f98eb469700 10 monclient: _reopen_session rank -1 2021-05-25T12:02:29.130 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.100+0000 7f98eb469700 10 monclient: _add_conns ranks=[1,2,0] 2021-05-25T12:02:29.130 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.100+0000 7f98eb469700 10 monclient(hunting): picked mon.noname-b con 0x7f98e4090990 addr v1:172.21.15.181:6789/0 2021-05-25T12:02:29.131 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.100+0000 7f98eb469700 10 monclient(hunting): picked mon.noname-c con 0x7f98e4090d70 addr v1:172.21.15.102:6790/0 2021-05-25T12:02:29.131 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.100+0000 7f98eb469700 10 monclient(hunting): picked mon.noname-a con 0x7f98e412e330 addr v1:172.21.15.102:6789/0 2021-05-25T12:02:29.131 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.100+0000 7f98eb469700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98e4091fe0 con 0x7f98e412e330 2021-05-25T12:02:29.131 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.100+0000 7f98eb469700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98e4068850 con 0x7f98e4090990 2021-05-25T12:02:29.132 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.100+0000 7f98eb469700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98e4068af0 con 0x7f98e4090d70 2021-05-25T12:02:29.132 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.100+0000 7f98eb469700 10 monclient(hunting): _renew_subs 2021-05-25T12:02:29.132 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.100+0000 7f98eb469700 10 monclient(hunting): authenticate will time out at 949.698429s 2021-05-25T12:02:29.132 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.100+0000 7f98e8a04700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f98e4090d70 0x7f98e4140160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:55002/0 (socket says 172.21.15.102:55002) 2021-05-25T12:02:29.132 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.100+0000 7f98e8a04700 1 -- 172.21.15.102:0/3795512235 learned_addr learned my addr 172.21.15.102:0/3795512235 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:02:29.133 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.100+0000 7f98e9a06700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f98e412e330 0x7f98e4143890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60126/0 (socket says 172.21.15.102:60126) 2021-05-25T12:02:29.133 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.100+0000 7f98d17fa700 10 client.?.objecter ms_handle_connect 0x7f98e4090d70 2021-05-25T12:02:29.133 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.100+0000 7f98d17fa700 10 client.?.objecter resend_mon_ops 2021-05-25T12:02:29.133 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.100+0000 7f98d17fa700 10 client.?.objecter ms_handle_connect 0x7f98e412e330 2021-05-25T12:02:29.133 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.100+0000 7f98d17fa700 10 client.?.objecter resend_mon_ops 2021-05-25T12:02:29.134 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.100+0000 7f98d17fa700 10 client.?.objecter ms_handle_connect 0x7f98e4090990 2021-05-25T12:02:29.134 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.100+0000 7f98d17fa700 10 client.?.objecter resend_mon_ops 2021-05-25T12:02:29.134 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.100+0000 7f98d17fa700 1 -- 172.21.15.102:0/3795512235 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1586828179 0 0) 0x7f98e4091fe0 con 0x7f98e412e330 2021-05-25T12:02:29.134 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.100+0000 7f98d17fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:02:29.134 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.100+0000 7f98d17fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:02:29.135 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.100+0000 7f98d17fa700 10 monclient(hunting): my global_id is 174135 2021-05-25T12:02:29.135 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.100+0000 7f98d17fa700 1 -- 172.21.15.102:0/3795512235 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98c4004c10 con 0x7f98e412e330 2021-05-25T12:02:29.135 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.100+0000 7f98d17fa700 1 -- 172.21.15.102:0/3795512235 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 381435661 0 0) 0x7f98e4068850 con 0x7f98e4090990 2021-05-25T12:02:29.135 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.100+0000 7f98d17fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:02:29.136 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.100+0000 7f98d17fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:02:29.136 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.100+0000 7f98d17fa700 10 monclient(hunting): my global_id is 95086 2021-05-25T12:02:29.136 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.100+0000 7f98d17fa700 1 -- 172.21.15.102:0/3795512235 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98e4091fe0 con 0x7f98e4090990 2021-05-25T12:02:29.136 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.100+0000 7f98d17fa700 1 -- 172.21.15.102:0/3795512235 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2079486252 0 0) 0x7f98c4004c10 con 0x7f98e412e330 2021-05-25T12:02:29.136 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.101+0000 7f98d17fa700 1 -- 172.21.15.102:0/3795512235 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f98e4068850 con 0x7f98e412e330 2021-05-25T12:02:29.137 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.101+0000 7f98d17fa700 1 -- 172.21.15.102:0/3795512235 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f98e0001b40 con 0x7f98e412e330 2021-05-25T12:02:29.137 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.101+0000 7f98d17fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:29.137 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.101+0000 7f98d17fa700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:02:29.137 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.101+0000 7f98d17fa700 10 monclient(hunting): dump: 2021-05-25T12:02:29.137 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:29.138 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:29.138 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:29.138 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:29.138 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:29.138 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:29.139 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:29.139 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:29.139 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:29.139 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:29.139 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.101+0000 7f98d17fa700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:02:29.140 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.101+0000 7f98d17fa700 1 -- 172.21.15.102:0/3795512235 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2085553672 0 0) 0x7f98e4091fe0 con 0x7f98e4090990 2021-05-25T12:02:29.140 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.101+0000 7f98d17fa700 1 -- 172.21.15.102:0/3795512235 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f98c4004c10 con 0x7f98e4090990 2021-05-25T12:02:29.140 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.101+0000 7f98d17fa700 1 -- 172.21.15.102:0/3795512235 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f98d40017f0 con 0x7f98e4090990 2021-05-25T12:02:29.140 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.101+0000 7f98d17fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:29.140 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.101+0000 7f98d17fa700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2021-05-25T12:02:29.141 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.101+0000 7f98d17fa700 10 monclient(hunting): dump: 2021-05-25T12:02:29.141 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:29.141 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:29.141 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:29.142 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:29.142 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:29.142 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:29.142 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:29.143 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:29.143 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:29.143 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:29.143 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.101+0000 7f98d17fa700 1 -- 172.21.15.102:0/3795512235 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4164136483 0 0) 0x7f98e4068850 con 0x7f98e412e330 2021-05-25T12:02:29.143 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.101+0000 7f98d17fa700 1 -- 172.21.15.102:0/3795512235 >> v1:172.21.15.102:6790/0 conn(0x7f98e4090d70 legacy=0x7f98e4140160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:29.144 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.101+0000 7f98d17fa700 1 -- 172.21.15.102:0/3795512235 >> v1:172.21.15.181:6789/0 conn(0x7f98e4090990 legacy=0x7f98e4126690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:29.144 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.101+0000 7f98d17fa700 10 monclient: _finish_hunting 0 2021-05-25T12:02:29.144 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.101+0000 7f98d17fa700 1 monclient: found mon.a 2021-05-25T12:02:29.144 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.101+0000 7f98d17fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:02:29.144 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.101+0000 7f98d17fa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:02:29.145 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.101+0000 7f98d17fa700 1 -- 172.21.15.102:0/3795512235 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98e4126da0 con 0x7f98e412e330 2021-05-25T12:02:29.145 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.101+0000 7f98d17fa700 10 monclient: _finish_auth 0 2021-05-25T12:02:29.145 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.101+0000 7f98d17fa700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:02:29.145 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.101+0000 7f98eb469700 5 monclient: authenticate success, global_id 174135 2021-05-25T12:02:29.146 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.101+0000 7f98eb469700 10 monclient: _renew_subs 2021-05-25T12:02:29.146 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.101+0000 7f98eb469700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:02:29.146 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.101+0000 7f98eb469700 1 -- 172.21.15.102:0/3795512235 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98e4067460 con 0x7f98e412e330 2021-05-25T12:02:29.146 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.101+0000 7f98eb469700 10 client.174135.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:02:29.146 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.101+0000 7f98eb469700 10 monclient: _renew_subs 2021-05-25T12:02:29.147 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.101+0000 7f98eb469700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:02:29.147 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.101+0000 7f98eb469700 1 -- 172.21.15.102:0/3795512235 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f98e413b6c0 con 0x7f98e412e330 2021-05-25T12:02:29.147 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.101+0000 7f98eb469700 1 librados: init done 2021-05-25T12:02:29.147 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.101+0000 7f98d17fa700 1 -- 172.21.15.102:0/3795512235 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f98e0002130 con 0x7f98e412e330 2021-05-25T12:02:29.147 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.101+0000 7f98d17fa700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:02:29.148 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.101+0000 7f98d17fa700 1 -- 172.21.15.102:0/3795512235 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f98e00024c0 con 0x7f98e412e330 2021-05-25T12:02:29.148 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.101+0000 7f98d17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:29.148 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.101+0000 7f98d17fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:02:29.148 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.101+0000 7f98d37fe700 4 set_mon_vals no callback set 2021-05-25T12:02:29.148 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.101+0000 7f98d17fa700 10 monclient: dump: 2021-05-25T12:02:29.149 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:29.149 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:29.149 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:29.149 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:29.149 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:29.150 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:29.150 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:29.150 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:29.150 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:29.150 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:29.151 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.101+0000 7f98d17fa700 1 -- 172.21.15.102:0/3795512235 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f98e0002620 con 0x7f98e412e330 2021-05-25T12:02:29.151 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.102+0000 7f98eb469700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:02:29.151 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.102+0000 7f98eb469700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:02:29.151 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.102+0000 7f98d17fa700 1 -- 172.21.15.102:0/3795512235 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(72..72 src has 1..72) v4 ==== 7857+0+0 (unknown 879564551 0 0) 0x7f98e003e010 con 0x7f98e412e330 2021-05-25T12:02:29.152 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.102+0000 7f98eb469700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:02:29.152 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.102+0000 7f98d17fa700 10 client.174135.objecter ms_dispatch 0x7f98e4001550 osd_map(72..72 src has 1..72) v4 2021-05-25T12:02:29.152 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.102+0000 7f98d17fa700 3 client.174135.objecter handle_osd_map got epochs [72,72] > 0 2021-05-25T12:02:29.152 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.102+0000 7f98d17fa700 3 client.174135.objecter handle_osd_map decoding full epoch 72 2021-05-25T12:02:29.152 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.102+0000 7f98eb469700 1 -- 172.21.15.102:0/3795512235 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98bc004db0 con 0x7f98e412e330 2021-05-25T12:02:29.153 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.102+0000 7f98d17fa700 20 client.174135.objecter dump_active .. 0 homeless 2021-05-25T12:02:29.153 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.102+0000 7f98e9205700 10 monclient: get_auth_request con 0x7f98c4033340 auth_method 0 2021-05-25T12:02:29.153 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.102+0000 7f98d17fa700 10 client.174135.objecter ms_handle_connect 0x7f98c4033340 2021-05-25T12:02:29.153 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.107+0000 7f98d17fa700 1 -- 172.21.15.102:0/3795512235 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7f98e0012e90 con 0x7f98e412e330 2021-05-25T12:02:29.153 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.107+0000 7f98d17fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:02:29.154 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.107+0000 7f98d17fa700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:02:29.320 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.317+0000 7f98eb469700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2021-05-25T12:02:29.320 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.317+0000 7f98eb469700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2021-05-25T12:02:29.320 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.317+0000 7f98eb469700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:02:29.320 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.317+0000 7f98eb469700 1 -- 172.21.15.102:0/3795512235 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f98bc005060 con 0x7f98e412e330 2021-05-25T12:02:29.321 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.317+0000 7f98d17fa700 1 -- 172.21.15.102:0/3795512235 <== mon.0 v1:172.21.15.102:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1336 (unknown 254746657 0 69982291) 0x7f98e0012a80 con 0x7f98e412e330 2021-05-25T12:02:29.321 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.317+0000 7f98d17fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2021-05-25T12:02:29.321 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.317+0000 7f98d17fa700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:02:29.322 INFO:teuthology.orchestra.run.smithi102.stdout:{"election_epoch":26,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"b","quorum_age":9,"features":{"quorum_con":"4540138303579357183","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"20ee91b6-8935-4b98-b868-85c03789ccc4","modified":"2021-05-25T11:59:11.011522Z","created":"2021-05-25T11:59:11.011522Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6790","nonce":0}]},"addr":"172.21.15.102:6790/0","public_addr":"172.21.15.102:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2021-05-25T12:02:29.323 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.319+0000 7f98eb469700 10 librados: watch_flush enter 2021-05-25T12:02:29.323 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.319+0000 7f98eb469700 10 librados: watch_flush exit 2021-05-25T12:02:29.324 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.319+0000 7f98eb469700 20 client.174135.objecter shutdown clearing up homeless session... 2021-05-25T12:02:29.324 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.319+0000 7f98eb469700 10 client.174135.objecter successfully canceled tick 2021-05-25T12:02:29.324 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.319+0000 7f98eb469700 1 -- 172.21.15.102:0/3795512235 >> v1:172.21.15.181:6812/29582 conn(0x7f98c4033340 legacy=0x7f98c4035800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:29.324 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.319+0000 7f98eb469700 10 monclient: shutdown 2021-05-25T12:02:29.324 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.319+0000 7f98eb469700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:02:29.325 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.319+0000 7f98eb469700 1 -- 172.21.15.102:0/3795512235 >> v1:172.21.15.102:6789/0 conn(0x7f98e412e330 legacy=0x7f98e4143890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:29.325 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.319+0000 7f98eb469700 1 -- 172.21.15.102:0/3795512235 shutdown_connections 2021-05-25T12:02:29.325 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.319+0000 7f98eb469700 1 -- 172.21.15.102:0/3795512235 >> 172.21.15.102:0/3795512235 conn(0x7f98e409b370 msgr2=0x7f98e41291b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:02:29.325 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.319+0000 7f98eb469700 1 -- 172.21.15.102:0/3795512235 shutdown_connections 2021-05-25T12:02:29.325 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.319+0000 7f98eb469700 1 -- 172.21.15.102:0/3795512235 wait complete. 2021-05-25T12:02:29.326 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:29.319+0000 7f98eb469700 1 librados: shutdown 2021-05-25T12:02:32.333 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2021-05-25T12:02:32.451 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.449+0000 7eff1c980700 10 monclient: get_monmap_and_config 2021-05-25T12:02:32.452 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.449+0000 7eff1c980700 10 monclient: build_initial_monmap 2021-05-25T12:02:32.452 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.449+0000 7eff1c980700 10 monclient: monmap: 2021-05-25T12:02:32.452 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:02:32.452 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:32.452 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:02:32.450691+0000 2021-05-25T12:02:32.453 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:02:32.450691+0000 2021-05-25T12:02:32.453 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:02:32.453 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:32.453 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:02:32.454 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:02:32.454 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:02:32.454 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:32.454 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.450+0000 7eff1c980700 1 Processor -- start 2021-05-25T12:02:32.454 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.450+0000 7eff1c980700 1 -- start start 2021-05-25T12:02:32.455 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.450+0000 7eff1c980700 10 monclient: init 2021-05-25T12:02:32.465 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.450+0000 7eff1c980700 10 monclient: _reopen_session rank -1 2021-05-25T12:02:32.465 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.450+0000 7eff1c980700 10 monclient: _add_conns ranks=[2,1,0] 2021-05-25T12:02:32.465 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.450+0000 7eff1c980700 10 monclient(hunting): picked mon.noname-b con 0x7eff18090990 addr v1:172.21.15.102:6790/0 2021-05-25T12:02:32.465 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.450+0000 7eff1c980700 10 monclient(hunting): picked mon.noname-c con 0x7eff18090d70 addr v1:172.21.15.181:6789/0 2021-05-25T12:02:32.466 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.450+0000 7eff1c980700 10 monclient(hunting): picked mon.noname-a con 0x7eff1812e330 addr v1:172.21.15.102:6789/0 2021-05-25T12:02:32.466 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.450+0000 7eff1c980700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff18096fa0 con 0x7eff1812e330 2021-05-25T12:02:32.466 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.450+0000 7eff1c980700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff180926a0 con 0x7eff18090d70 2021-05-25T12:02:32.466 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.450+0000 7eff1c980700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff18092940 con 0x7eff18090990 2021-05-25T12:02:32.466 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.450+0000 7eff1c980700 10 monclient(hunting): _renew_subs 2021-05-25T12:02:32.467 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.450+0000 7eff1c980700 10 monclient(hunting): authenticate will time out at 953.047976s 2021-05-25T12:02:32.467 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.451+0000 7eff16d9d700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7eff1812e330 0x7eff18130720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60144/0 (socket says 172.21.15.102:60144) 2021-05-25T12:02:32.467 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.451+0000 7eff16d9d700 1 -- 172.21.15.102:0/409232137 learned_addr learned my addr 172.21.15.102:0/409232137 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:02:32.467 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.451+0000 7eff1659c700 1 --1- 172.21.15.102:0/409232137 >> v1:172.21.15.102:6790/0 conn(0x7eff18090990 0x7eff18093240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:55018/0 (socket says 172.21.15.102:55018) 2021-05-25T12:02:32.468 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.451+0000 7eff1559a700 1 -- 172.21.15.102:0/409232137 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2164134018 0 0) 0x7eff18096fa0 con 0x7eff1812e330 2021-05-25T12:02:32.468 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.451+0000 7eff1559a700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:02:32.468 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.451+0000 7eff1559a700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:02:32.469 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.451+0000 7eff1559a700 10 monclient(hunting): my global_id is 174138 2021-05-25T12:02:32.469 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.451+0000 7eff1559a700 1 -- 172.21.15.102:0/409232137 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efefc003270 con 0x7eff1812e330 2021-05-25T12:02:32.469 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.451+0000 7eff1559a700 1 -- 172.21.15.102:0/409232137 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3146178479 0 0) 0x7efefc003270 con 0x7eff1812e330 2021-05-25T12:02:32.469 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.451+0000 7eff1559a700 1 -- 172.21.15.102:0/409232137 >> v1:172.21.15.102:6790/0 conn(0x7eff18090990 legacy=0x7eff18093240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:32.469 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.451+0000 7eff1559a700 1 -- 172.21.15.102:0/409232137 >> v1:172.21.15.181:6789/0 conn(0x7eff18090d70 legacy=0x7eff18091200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:32.470 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.451+0000 7eff1559a700 10 monclient: _finish_hunting 0 2021-05-25T12:02:32.470 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.451+0000 7eff1559a700 1 monclient: found mon.noname-a 2021-05-25T12:02:32.470 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.451+0000 7eff1559a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:02:32.470 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.451+0000 7eff1559a700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T12:02:32.471 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.451+0000 7eff1559a700 1 -- 172.21.15.102:0/409232137 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff18092be0 con 0x7eff1812e330 2021-05-25T12:02:32.471 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.451+0000 7eff1559a700 10 monclient: _finish_auth 0 2021-05-25T12:02:32.471 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.451+0000 7eff1559a700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:02:32.471 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.451+0000 7eff1559a700 1 -- 172.21.15.102:0/409232137 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7eff08001990 con 0x7eff1812e330 2021-05-25T12:02:32.471 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.451+0000 7eff1c980700 5 monclient: authenticate success, global_id 174138 2021-05-25T12:02:32.472 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.451+0000 7eff1c980700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:02:32.472 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.451+0000 7eff1559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:32.472 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.451+0000 7eff1559a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:02:32.472 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.451+0000 7eff1559a700 10 monclient: dump: 2021-05-25T12:02:32.473 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:32.473 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:32.473 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:32.473 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:32.473 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:32.474 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:32.474 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:32.474 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:32.474 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:32.475 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:32.475 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.451+0000 7eff1c980700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:02:32.475 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.451+0000 7eff1559a700 1 -- 172.21.15.102:0/409232137 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7eff08001c40 con 0x7eff1812e330 2021-05-25T12:02:32.475 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.451+0000 7eff1559a700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:02:32.475 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.451+0000 7eff1c980700 10 monclient: get_monmap_and_config success 2021-05-25T12:02:32.476 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.451+0000 7eff1559a700 1 -- 172.21.15.102:0/409232137 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7eff08001fd0 con 0x7eff1812e330 2021-05-25T12:02:32.476 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.451+0000 7eff1c980700 4 set_mon_vals no callback set 2021-05-25T12:02:32.476 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.451+0000 7eff1559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:32.476 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.451+0000 7eff1559a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:02:32.477 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.451+0000 7eff1559a700 10 monclient: dump: 2021-05-25T12:02:32.477 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:32.477 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:32.477 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:32.477 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:32.478 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:32.478 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:32.478 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:32.479 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:32.479 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:32.479 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:32.479 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.452+0000 7eff1c980700 10 monclient: shutdown 2021-05-25T12:02:32.479 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.452+0000 7eff1c980700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:02:32.480 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.452+0000 7eff1c980700 1 -- 172.21.15.102:0/409232137 >> v1:172.21.15.102:6789/0 conn(0x7eff1812e330 legacy=0x7eff18130720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:32.480 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.452+0000 7eff1c980700 1 -- 172.21.15.102:0/409232137 shutdown_connections 2021-05-25T12:02:32.480 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.452+0000 7eff1c980700 1 -- 172.21.15.102:0/409232137 >> 172.21.15.102:0/409232137 conn(0x7eff1809b370 msgr2=0x7eff18097430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:02:32.480 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.452+0000 7eff1c980700 1 -- 172.21.15.102:0/409232137 shutdown_connections 2021-05-25T12:02:32.481 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.452+0000 7eff1c980700 1 -- 172.21.15.102:0/409232137 wait complete. 2021-05-25T12:02:32.481 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.452+0000 7eff1c980700 10 monclient: build_initial_monmap 2021-05-25T12:02:32.481 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.452+0000 7eff1c980700 10 monclient: monmap: 2021-05-25T12:02:32.481 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:02:32.481 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:02:32.482 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:02:32.482 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:02:32.482 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:02:32.482 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:02:32.483 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:02:32.483 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:02:32.483 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:02:32.483 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:32.483 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.452+0000 7eff1c980700 1 librados: starting msgr at 2021-05-25T12:02:32.484 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.452+0000 7eff1c980700 1 librados: starting objecter 2021-05-25T12:02:32.484 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.453+0000 7eff1c980700 1 Processor -- start 2021-05-25T12:02:32.484 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.453+0000 7eff1c980700 1 -- start start 2021-05-25T12:02:32.485 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.453+0000 7eff1c980700 1 librados: setting wanted keys 2021-05-25T12:02:32.485 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.453+0000 7eff1c980700 1 librados: calling monclient init 2021-05-25T12:02:32.485 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.453+0000 7eff1c980700 10 monclient: init 2021-05-25T12:02:32.485 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.453+0000 7eff1c980700 10 monclient: _reopen_session rank -1 2021-05-25T12:02:32.485 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.453+0000 7eff1c980700 10 monclient: _add_conns ranks=[1,0,2] 2021-05-25T12:02:32.486 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.453+0000 7eff1c980700 10 monclient(hunting): picked mon.noname-b con 0x7eff18090990 addr v1:172.21.15.181:6789/0 2021-05-25T12:02:32.486 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.453+0000 7eff1c980700 10 monclient(hunting): picked mon.noname-a con 0x7eff18090d70 addr v1:172.21.15.102:6789/0 2021-05-25T12:02:32.486 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.453+0000 7eff1c980700 10 monclient(hunting): picked mon.noname-c con 0x7eff1812e330 addr v1:172.21.15.102:6790/0 2021-05-25T12:02:32.487 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.453+0000 7eff1c980700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff18132320 con 0x7eff18090d70 2021-05-25T12:02:32.487 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.453+0000 7eff1c980700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff181162d0 con 0x7eff18090990 2021-05-25T12:02:32.487 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.453+0000 7eff1c980700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff18116570 con 0x7eff1812e330 2021-05-25T12:02:32.487 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.453+0000 7eff1c980700 10 monclient(hunting): _renew_subs 2021-05-25T12:02:32.487 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.453+0000 7eff1c980700 10 monclient(hunting): authenticate will time out at 953.050487s 2021-05-25T12:02:32.488 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.453+0000 7eff15d9b700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7eff18090d70 0x7eff18140160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60148/0 (socket says 172.21.15.102:60148) 2021-05-25T12:02:32.488 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.453+0000 7eff15d9b700 1 -- 172.21.15.102:0/2570140753 learned_addr learned my addr 172.21.15.102:0/2570140753 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:02:32.488 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.453+0000 7eff067fc700 10 client.?.objecter ms_handle_connect 0x7eff18090d70 2021-05-25T12:02:32.488 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.453+0000 7eff067fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:02:32.488 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.453+0000 7eff067fc700 10 client.?.objecter ms_handle_connect 0x7eff1812e330 2021-05-25T12:02:32.489 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.453+0000 7eff067fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:02:32.489 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.453+0000 7eff067fc700 1 -- 172.21.15.102:0/2570140753 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2875078113 0 0) 0x7eff18132320 con 0x7eff18090d70 2021-05-25T12:02:32.489 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.453+0000 7eff067fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:02:32.489 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.453+0000 7eff067fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:02:32.489 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.453+0000 7eff067fc700 10 monclient(hunting): my global_id is 174141 2021-05-25T12:02:32.490 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.453+0000 7eff067fc700 1 -- 172.21.15.102:0/2570140753 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efef4004c10 con 0x7eff18090d70 2021-05-25T12:02:32.490 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.453+0000 7eff067fc700 10 client.?.objecter ms_handle_connect 0x7eff18090990 2021-05-25T12:02:32.490 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.453+0000 7eff067fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:02:32.490 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.453+0000 7eff067fc700 1 -- 172.21.15.102:0/2570140753 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2691256211 0 0) 0x7efef4004c10 con 0x7eff18090d70 2021-05-25T12:02:32.491 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.454+0000 7eff067fc700 1 -- 172.21.15.102:0/2570140753 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7eff18132320 con 0x7eff18090d70 2021-05-25T12:02:32.491 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.454+0000 7eff067fc700 1 -- 172.21.15.102:0/2570140753 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7eff0c001980 con 0x7eff18090d70 2021-05-25T12:02:32.491 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.454+0000 7eff067fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:32.491 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.454+0000 7eff067fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:02:32.491 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.454+0000 7eff067fc700 10 monclient(hunting): dump: 2021-05-25T12:02:32.492 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:32.492 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:32.492 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:32.492 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:32.492 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:32.493 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:32.493 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:32.493 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:32.493 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:32.493 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:32.494 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.454+0000 7eff067fc700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:02:32.494 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.454+0000 7eff067fc700 1 -- 172.21.15.102:0/2570140753 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1092827385 0 0) 0x7eff18132320 con 0x7eff18090d70 2021-05-25T12:02:32.495 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.454+0000 7eff067fc700 1 -- 172.21.15.102:0/2570140753 >> v1:172.21.15.102:6790/0 conn(0x7eff1812e330 legacy=0x7eff18143890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:32.495 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.454+0000 7eff067fc700 1 -- 172.21.15.102:0/2570140753 >> v1:172.21.15.181:6789/0 conn(0x7eff18090990 legacy=0x7eff1813b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:32.495 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.454+0000 7eff067fc700 10 monclient: _finish_hunting 0 2021-05-25T12:02:32.496 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.454+0000 7eff067fc700 1 monclient: found mon.a 2021-05-25T12:02:32.496 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.454+0000 7eff067fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:02:32.496 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.454+0000 7eff067fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:02:32.496 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.454+0000 7eff067fc700 1 -- 172.21.15.102:0/2570140753 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff18116810 con 0x7eff18090d70 2021-05-25T12:02:32.497 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.454+0000 7eff067fc700 10 monclient: _finish_auth 0 2021-05-25T12:02:32.497 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.454+0000 7eff067fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:02:32.497 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.454+0000 7eff1c980700 5 monclient: authenticate success, global_id 174141 2021-05-25T12:02:32.497 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.454+0000 7eff1c980700 10 monclient: _renew_subs 2021-05-25T12:02:32.497 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.454+0000 7eff1c980700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:02:32.498 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.454+0000 7eff1c980700 1 -- 172.21.15.102:0/2570140753 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff1813bc50 con 0x7eff18090d70 2021-05-25T12:02:32.498 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.454+0000 7eff1c980700 10 client.174141.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:02:32.498 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.454+0000 7eff1c980700 10 monclient: _renew_subs 2021-05-25T12:02:32.499 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.454+0000 7eff1c980700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:02:32.499 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.454+0000 7eff1c980700 1 -- 172.21.15.102:0/2570140753 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7eff181155f0 con 0x7eff18090d70 2021-05-25T12:02:32.499 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.454+0000 7eff1c980700 1 librados: init done 2021-05-25T12:02:32.499 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.454+0000 7eff067fc700 1 -- 172.21.15.102:0/2570140753 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7eff0c001d50 con 0x7eff18090d70 2021-05-25T12:02:32.499 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.454+0000 7eff067fc700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:02:32.500 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.454+0000 7eff067fc700 1 -- 172.21.15.102:0/2570140753 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7eff0c002410 con 0x7eff18090d70 2021-05-25T12:02:32.500 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.454+0000 7eff067fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:32.500 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.454+0000 7eff14d99700 4 set_mon_vals no callback set 2021-05-25T12:02:32.500 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.454+0000 7eff067fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:02:32.501 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.454+0000 7eff067fc700 10 monclient: dump: 2021-05-25T12:02:32.501 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:32.501 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:32.501 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:32.501 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:32.502 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:32.502 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:32.502 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:32.502 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:32.502 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:32.503 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:32.503 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.455+0000 7eff067fc700 1 -- 172.21.15.102:0/2570140753 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7eff0c002570 con 0x7eff18090d70 2021-05-25T12:02:32.503 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.455+0000 7eff1c980700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:02:32.503 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.455+0000 7eff1c980700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:02:32.504 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.455+0000 7eff1c980700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:02:32.504 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.455+0000 7eff1c980700 1 -- 172.21.15.102:0/2570140753 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efef8004db0 con 0x7eff18090d70 2021-05-25T12:02:32.504 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.455+0000 7eff067fc700 1 -- 172.21.15.102:0/2570140753 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(74..74 src has 1..74) v4 ==== 8260+0+0 (unknown 2607749494 0 0) 0x7eff0c03e1d0 con 0x7eff18090d70 2021-05-25T12:02:32.505 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.455+0000 7eff067fc700 10 client.174141.objecter ms_dispatch 0x7eff18001550 osd_map(74..74 src has 1..74) v4 2021-05-25T12:02:32.505 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.455+0000 7eff067fc700 3 client.174141.objecter handle_osd_map got epochs [74,74] > 0 2021-05-25T12:02:32.505 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.455+0000 7eff067fc700 3 client.174141.objecter handle_osd_map decoding full epoch 74 2021-05-25T12:02:32.505 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.455+0000 7eff067fc700 20 client.174141.objecter dump_active .. 0 homeless 2021-05-25T12:02:32.506 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.455+0000 7eff1659c700 10 monclient: get_auth_request con 0x7efef4033160 auth_method 0 2021-05-25T12:02:32.506 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.455+0000 7eff067fc700 10 client.174141.objecter ms_handle_connect 0x7efef4033160 2021-05-25T12:02:32.506 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.459+0000 7eff067fc700 1 -- 172.21.15.102:0/2570140753 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7eff0c012e20 con 0x7eff18090d70 2021-05-25T12:02:32.506 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.459+0000 7eff067fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:02:32.507 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.459+0000 7eff067fc700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:02:32.669 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.667+0000 7eff1c980700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2021-05-25T12:02:32.669 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.667+0000 7eff1c980700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2021-05-25T12:02:32.669 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.667+0000 7eff1c980700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:02:32.670 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.667+0000 7eff1c980700 1 -- 172.21.15.102:0/2570140753 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7efef8005060 con 0x7eff18090d70 2021-05-25T12:02:32.670 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.667+0000 7eff067fc700 1 -- 172.21.15.102:0/2570140753 <== mon.0 v1:172.21.15.102:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1337 (unknown 254746657 0 3207251635) 0x7eff0c012a10 con 0x7eff18090d70 2021-05-25T12:02:32.670 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.667+0000 7eff067fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2021-05-25T12:02:32.670 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.667+0000 7eff067fc700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:02:32.671 INFO:teuthology.orchestra.run.smithi102.stdout:{"election_epoch":26,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"b","quorum_age":12,"features":{"quorum_con":"4540138303579357183","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"20ee91b6-8935-4b98-b868-85c03789ccc4","modified":"2021-05-25T11:59:11.011522Z","created":"2021-05-25T11:59:11.011522Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6790","nonce":0}]},"addr":"172.21.15.102:6790/0","public_addr":"172.21.15.102:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2021-05-25T12:02:32.672 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.669+0000 7eff1c980700 10 librados: watch_flush enter 2021-05-25T12:02:32.673 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.669+0000 7eff1c980700 10 librados: watch_flush exit 2021-05-25T12:02:32.673 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.669+0000 7eff1c980700 20 client.174141.objecter shutdown clearing up homeless session... 2021-05-25T12:02:32.673 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.669+0000 7eff1c980700 10 client.174141.objecter successfully canceled tick 2021-05-25T12:02:32.674 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.669+0000 7eff1c980700 1 -- 172.21.15.102:0/2570140753 >> v1:172.21.15.181:6812/29582 conn(0x7efef4033160 legacy=0x7efef4035620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:32.674 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.669+0000 7eff1c980700 10 monclient: shutdown 2021-05-25T12:02:32.674 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.669+0000 7eff1c980700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:02:32.674 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.669+0000 7eff1c980700 1 -- 172.21.15.102:0/2570140753 >> v1:172.21.15.102:6789/0 conn(0x7eff18090d70 legacy=0x7eff18140160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:32.675 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.669+0000 7eff1c980700 1 -- 172.21.15.102:0/2570140753 shutdown_connections 2021-05-25T12:02:32.675 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.669+0000 7eff1c980700 1 -- 172.21.15.102:0/2570140753 >> 172.21.15.102:0/2570140753 conn(0x7eff1809b370 msgr2=0x7eff18068110 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:02:32.675 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.669+0000 7eff1c980700 1 -- 172.21.15.102:0/2570140753 shutdown_connections 2021-05-25T12:02:32.675 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.669+0000 7eff1c980700 1 -- 172.21.15.102:0/2570140753 wait complete. 2021-05-25T12:02:32.676 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:32.669+0000 7eff1c980700 1 librados: shutdown 2021-05-25T12:02:35.631 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:02:35.629+0000 7fd73b235700 -1 mon.c@2(electing) e1 get_health_metrics reporting 3 slow ops, oldest is pool_op(delete pool 92 tid 4 name delete v61) 2021-05-25T12:02:35.682 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2021-05-25T12:02:35.800 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.799+0000 7fe79ff40700 10 monclient: get_monmap_and_config 2021-05-25T12:02:35.801 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.799+0000 7fe79ff40700 10 monclient: build_initial_monmap 2021-05-25T12:02:35.801 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.799+0000 7fe79ff40700 10 monclient: monmap: 2021-05-25T12:02:35.802 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:02:35.802 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:35.802 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:02:35.800414+0000 2021-05-25T12:02:35.802 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:02:35.800414+0000 2021-05-25T12:02:35.803 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:02:35.803 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:35.803 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:02:35.803 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:02:35.803 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:02:35.804 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:35.813 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.799+0000 7fe79ff40700 1 Processor -- start 2021-05-25T12:02:35.814 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.800+0000 7fe79ff40700 1 -- start start 2021-05-25T12:02:35.814 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.800+0000 7fe79ff40700 10 monclient: init 2021-05-25T12:02:35.814 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.800+0000 7fe79ff40700 10 monclient: _reopen_session rank -1 2021-05-25T12:02:35.814 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.800+0000 7fe79ff40700 10 monclient: _add_conns ranks=[1,0,2] 2021-05-25T12:02:35.815 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.800+0000 7fe79ff40700 10 monclient(hunting): picked mon.noname-c con 0x7fe798090990 addr v1:172.21.15.181:6789/0 2021-05-25T12:02:35.815 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.800+0000 7fe79ff40700 10 monclient(hunting): picked mon.noname-a con 0x7fe798090d70 addr v1:172.21.15.102:6789/0 2021-05-25T12:02:35.815 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.800+0000 7fe79ff40700 10 monclient(hunting): picked mon.noname-b con 0x7fe79812e330 addr v1:172.21.15.102:6790/0 2021-05-25T12:02:35.815 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.800+0000 7fe79ff40700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe798096fa0 con 0x7fe798090d70 2021-05-25T12:02:35.816 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.800+0000 7fe79ff40700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7980926a0 con 0x7fe798090990 2021-05-25T12:02:35.816 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.800+0000 7fe79ff40700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe798092940 con 0x7fe79812e330 2021-05-25T12:02:35.816 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.800+0000 7fe79ff40700 10 monclient(hunting): _renew_subs 2021-05-25T12:02:35.816 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.800+0000 7fe79ff40700 10 monclient(hunting): authenticate will time out at 956.397715s 2021-05-25T12:02:35.817 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.800+0000 7fe79d4db700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fe798090d70 0x7fe798091200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60154/0 (socket says 172.21.15.102:60154) 2021-05-25T12:02:35.817 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.800+0000 7fe79d4db700 1 -- 172.21.15.102:0/1369317122 learned_addr learned my addr 172.21.15.102:0/1369317122 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:02:35.817 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.800+0000 7fe79e4dd700 1 --1- 172.21.15.102:0/1369317122 >> v1:172.21.15.102:6790/0 conn(0x7fe79812e330 0x7fe798130720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:55036/0 (socket says 172.21.15.102:55036) 2021-05-25T12:02:35.817 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.801+0000 7fe79ccda700 1 -- 172.21.15.102:0/1369317122 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3117581995 0 0) 0x7fe798096fa0 con 0x7fe798090d70 2021-05-25T12:02:35.817 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.801+0000 7fe79ccda700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:02:35.818 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.801+0000 7fe79ccda700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:02:35.818 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.801+0000 7fe79ccda700 10 monclient(hunting): my global_id is 174144 2021-05-25T12:02:35.818 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.801+0000 7fe79ccda700 1 -- 172.21.15.102:0/1369317122 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe780003270 con 0x7fe798090d70 2021-05-25T12:02:35.818 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.801+0000 7fe79ccda700 1 -- 172.21.15.102:0/1369317122 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3447001601 0 0) 0x7fe780003270 con 0x7fe798090d70 2021-05-25T12:02:35.819 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.801+0000 7fe79ccda700 1 -- 172.21.15.102:0/1369317122 >> v1:172.21.15.102:6790/0 conn(0x7fe79812e330 legacy=0x7fe798130720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:35.819 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.801+0000 7fe79ccda700 1 -- 172.21.15.102:0/1369317122 >> v1:172.21.15.181:6789/0 conn(0x7fe798090990 legacy=0x7fe798093240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:35.819 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.801+0000 7fe79ccda700 10 monclient: _finish_hunting 0 2021-05-25T12:02:35.819 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.801+0000 7fe79ccda700 1 monclient: found mon.noname-a 2021-05-25T12:02:35.819 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.801+0000 7fe79ccda700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:02:35.820 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.801+0000 7fe79ccda700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T12:02:35.820 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.801+0000 7fe79ccda700 1 -- 172.21.15.102:0/1369317122 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe798092be0 con 0x7fe798090d70 2021-05-25T12:02:35.820 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.801+0000 7fe79ccda700 10 monclient: _finish_auth 0 2021-05-25T12:02:35.820 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.801+0000 7fe79ccda700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:02:35.821 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.801+0000 7fe79ccda700 1 -- 172.21.15.102:0/1369317122 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fe78c001610 con 0x7fe798090d70 2021-05-25T12:02:35.821 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.801+0000 7fe79ff40700 5 monclient: authenticate success, global_id 174144 2021-05-25T12:02:35.821 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.801+0000 7fe79ff40700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:02:35.821 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.801+0000 7fe79ccda700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:35.821 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.801+0000 7fe79ccda700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:02:35.822 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.801+0000 7fe79ccda700 10 monclient: dump: 2021-05-25T12:02:35.822 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:35.822 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:35.823 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:35.823 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:35.823 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:35.823 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:35.824 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:35.824 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:35.824 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:35.824 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:35.824 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.801+0000 7fe79ff40700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:02:35.825 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.801+0000 7fe79ccda700 1 -- 172.21.15.102:0/1369317122 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe78c0018c0 con 0x7fe798090d70 2021-05-25T12:02:35.825 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.801+0000 7fe79ccda700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:02:35.825 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.801+0000 7fe79ff40700 10 monclient: get_monmap_and_config success 2021-05-25T12:02:35.825 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.801+0000 7fe79ccda700 1 -- 172.21.15.102:0/1369317122 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fe78c001c10 con 0x7fe798090d70 2021-05-25T12:02:35.825 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.801+0000 7fe79ff40700 4 set_mon_vals no callback set 2021-05-25T12:02:35.826 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.801+0000 7fe79ccda700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:35.826 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.801+0000 7fe79ccda700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:02:35.826 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.801+0000 7fe79ccda700 10 monclient: dump: 2021-05-25T12:02:35.826 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:35.827 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:35.827 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:35.827 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:35.827 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:35.828 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:35.828 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:35.828 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:35.828 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:35.828 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:35.829 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.801+0000 7fe79ff40700 10 monclient: shutdown 2021-05-25T12:02:35.829 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.801+0000 7fe79ff40700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:02:35.829 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.801+0000 7fe79ff40700 1 -- 172.21.15.102:0/1369317122 >> v1:172.21.15.102:6789/0 conn(0x7fe798090d70 legacy=0x7fe798091200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:35.829 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.801+0000 7fe79ff40700 1 -- 172.21.15.102:0/1369317122 shutdown_connections 2021-05-25T12:02:35.829 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.801+0000 7fe79ff40700 1 -- 172.21.15.102:0/1369317122 >> 172.21.15.102:0/1369317122 conn(0x7fe79809b370 msgr2=0x7fe798097430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:02:35.830 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.802+0000 7fe79ff40700 1 -- 172.21.15.102:0/1369317122 shutdown_connections 2021-05-25T12:02:35.830 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.802+0000 7fe79ff40700 1 -- 172.21.15.102:0/1369317122 wait complete. 2021-05-25T12:02:35.830 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.802+0000 7fe79ff40700 10 monclient: build_initial_monmap 2021-05-25T12:02:35.830 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.802+0000 7fe79ff40700 10 monclient: monmap: 2021-05-25T12:02:35.831 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:02:35.831 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:02:35.831 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:02:35.831 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:02:35.831 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:02:35.832 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:02:35.832 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:02:35.832 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:02:35.832 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:02:35.833 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:35.833 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.802+0000 7fe79ff40700 1 librados: starting msgr at 2021-05-25T12:02:35.833 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.802+0000 7fe79ff40700 1 librados: starting objecter 2021-05-25T12:02:35.833 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.802+0000 7fe79ff40700 1 Processor -- start 2021-05-25T12:02:35.833 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.802+0000 7fe79ff40700 1 -- start start 2021-05-25T12:02:35.834 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.802+0000 7fe79ff40700 1 librados: setting wanted keys 2021-05-25T12:02:35.834 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.802+0000 7fe79ff40700 1 librados: calling monclient init 2021-05-25T12:02:35.834 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.802+0000 7fe79ff40700 10 monclient: init 2021-05-25T12:02:35.834 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.803+0000 7fe79ff40700 10 monclient: _reopen_session rank -1 2021-05-25T12:02:35.835 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.803+0000 7fe79ff40700 10 monclient: _add_conns ranks=[2,1,0] 2021-05-25T12:02:35.835 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.803+0000 7fe79ff40700 10 monclient(hunting): picked mon.noname-c con 0x7fe798090990 addr v1:172.21.15.102:6790/0 2021-05-25T12:02:35.835 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.803+0000 7fe79ff40700 10 monclient(hunting): picked mon.noname-b con 0x7fe798090d70 addr v1:172.21.15.181:6789/0 2021-05-25T12:02:35.835 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.803+0000 7fe79ff40700 10 monclient(hunting): picked mon.noname-a con 0x7fe79812e330 addr v1:172.21.15.102:6789/0 2021-05-25T12:02:35.835 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.803+0000 7fe79ff40700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe79812a330 con 0x7fe79812e330 2021-05-25T12:02:35.836 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.803+0000 7fe79ff40700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7981172e0 con 0x7fe798090d70 2021-05-25T12:02:35.836 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.803+0000 7fe79ff40700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe798068bb0 con 0x7fe798090990 2021-05-25T12:02:35.836 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.803+0000 7fe79ff40700 10 monclient(hunting): _renew_subs 2021-05-25T12:02:35.836 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.803+0000 7fe79ff40700 10 monclient(hunting): authenticate will time out at 956.400401s 2021-05-25T12:02:35.836 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.803+0000 7fe79dcdc700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fe798090990 0x7fe798128870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:55038/0 (socket says 172.21.15.102:55038) 2021-05-25T12:02:35.837 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.803+0000 7fe79dcdc700 1 -- 172.21.15.102:0/3387679319 learned_addr learned my addr 172.21.15.102:0/3387679319 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:02:35.837 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.803+0000 7fe785ffb700 10 client.?.objecter ms_handle_connect 0x7fe798090990 2021-05-25T12:02:35.837 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.803+0000 7fe785ffb700 10 client.?.objecter resend_mon_ops 2021-05-25T12:02:35.837 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.803+0000 7fe785ffb700 10 client.?.objecter ms_handle_connect 0x7fe79812e330 2021-05-25T12:02:35.838 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.803+0000 7fe785ffb700 10 client.?.objecter resend_mon_ops 2021-05-25T12:02:35.838 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.803+0000 7fe785ffb700 10 client.?.objecter ms_handle_connect 0x7fe798090d70 2021-05-25T12:02:35.838 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.803+0000 7fe785ffb700 10 client.?.objecter resend_mon_ops 2021-05-25T12:02:35.838 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.803+0000 7fe785ffb700 1 -- 172.21.15.102:0/3387679319 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1899282254 0 0) 0x7fe79812a330 con 0x7fe79812e330 2021-05-25T12:02:35.838 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.803+0000 7fe785ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:02:35.839 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.803+0000 7fe785ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:02:35.839 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.803+0000 7fe785ffb700 10 monclient(hunting): my global_id is 174147 2021-05-25T12:02:35.839 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.803+0000 7fe785ffb700 1 -- 172.21.15.102:0/3387679319 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe778004c10 con 0x7fe79812e330 2021-05-25T12:02:35.839 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.803+0000 7fe785ffb700 1 -- 172.21.15.102:0/3387679319 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 351502081 0 0) 0x7fe778004c10 con 0x7fe79812e330 2021-05-25T12:02:35.839 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.803+0000 7fe785ffb700 1 -- 172.21.15.102:0/3387679319 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe79812a330 con 0x7fe79812e330 2021-05-25T12:02:35.840 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.803+0000 7fe785ffb700 1 -- 172.21.15.102:0/3387679319 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fe794001af0 con 0x7fe79812e330 2021-05-25T12:02:35.840 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.803+0000 7fe785ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:35.840 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.803+0000 7fe785ffb700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:02:35.840 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.803+0000 7fe785ffb700 10 monclient(hunting): dump: 2021-05-25T12:02:35.841 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:35.841 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:35.841 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:35.841 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:35.841 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:35.842 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:35.842 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:35.842 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:35.842 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:35.843 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:35.843 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.803+0000 7fe785ffb700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:02:35.843 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.804+0000 7fe785ffb700 1 -- 172.21.15.102:0/3387679319 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3902495128 0 0) 0x7fe79812a330 con 0x7fe79812e330 2021-05-25T12:02:35.843 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.804+0000 7fe785ffb700 1 -- 172.21.15.102:0/3387679319 >> v1:172.21.15.102:6790/0 conn(0x7fe798090990 legacy=0x7fe798128870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:35.843 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.804+0000 7fe785ffb700 1 -- 172.21.15.102:0/3387679319 >> v1:172.21.15.181:6789/0 conn(0x7fe798090d70 legacy=0x7fe7981401b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:35.844 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.804+0000 7fe785ffb700 10 monclient: _finish_hunting 0 2021-05-25T12:02:35.844 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.804+0000 7fe785ffb700 1 monclient: found mon.a 2021-05-25T12:02:35.844 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.804+0000 7fe785ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:02:35.844 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.804+0000 7fe785ffb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:02:35.845 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.804+0000 7fe785ffb700 1 -- 172.21.15.102:0/3387679319 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe798128fc0 con 0x7fe79812e330 2021-05-25T12:02:35.845 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.804+0000 7fe785ffb700 10 monclient: _finish_auth 0 2021-05-25T12:02:35.845 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.804+0000 7fe785ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:02:35.845 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.804+0000 7fe79ff40700 5 monclient: authenticate success, global_id 174147 2021-05-25T12:02:35.845 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.804+0000 7fe79ff40700 10 monclient: _renew_subs 2021-05-25T12:02:35.846 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.804+0000 7fe79ff40700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:02:35.846 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.804+0000 7fe79ff40700 1 -- 172.21.15.102:0/3387679319 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe798137b60 con 0x7fe79812e330 2021-05-25T12:02:35.846 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.804+0000 7fe785ffb700 1 -- 172.21.15.102:0/3387679319 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe794004890 con 0x7fe79812e330 2021-05-25T12:02:35.846 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.804+0000 7fe785ffb700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:02:35.847 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.804+0000 7fe785ffb700 1 -- 172.21.15.102:0/3387679319 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fe794004c20 con 0x7fe79812e330 2021-05-25T12:02:35.847 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.804+0000 7fe785ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:35.847 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.804+0000 7fe785ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:02:35.847 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.804+0000 7fe785ffb700 10 monclient: dump: 2021-05-25T12:02:35.847 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:35.848 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:35.848 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:35.848 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:35.848 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:35.848 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:35.849 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:35.849 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:35.849 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:35.849 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:35.849 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.804+0000 7fe79ff40700 10 client.174147.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:02:35.850 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.804+0000 7fe787fff700 4 set_mon_vals no callback set 2021-05-25T12:02:35.850 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.804+0000 7fe79ff40700 10 monclient: _renew_subs 2021-05-25T12:02:35.850 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.804+0000 7fe79ff40700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:02:35.850 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.804+0000 7fe79ff40700 1 -- 172.21.15.102:0/3387679319 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe798138470 con 0x7fe79812e330 2021-05-25T12:02:35.851 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.804+0000 7fe79ff40700 1 librados: init done 2021-05-25T12:02:35.851 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.804+0000 7fe785ffb700 1 -- 172.21.15.102:0/3387679319 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7fe794004e50 con 0x7fe79812e330 2021-05-25T12:02:35.851 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.805+0000 7fe785ffb700 1 -- 172.21.15.102:0/3387679319 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(74..74 src has 1..74) v4 ==== 8260+0+0 (unknown 2607749494 0 0) 0x7fe79403e400 con 0x7fe79812e330 2021-05-25T12:02:35.851 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.805+0000 7fe785ffb700 10 client.174147.objecter ms_dispatch 0x7fe798001550 osd_map(74..74 src has 1..74) v4 2021-05-25T12:02:35.851 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.805+0000 7fe785ffb700 3 client.174147.objecter handle_osd_map got epochs [74,74] > 0 2021-05-25T12:02:35.852 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.805+0000 7fe785ffb700 3 client.174147.objecter handle_osd_map decoding full epoch 74 2021-05-25T12:02:35.852 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.805+0000 7fe785ffb700 20 client.174147.objecter dump_active .. 0 homeless 2021-05-25T12:02:35.852 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.805+0000 7fe79ff40700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:02:35.852 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.805+0000 7fe79ff40700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:02:35.853 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.805+0000 7fe79ff40700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:02:35.853 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.805+0000 7fe79ff40700 1 -- 172.21.15.102:0/3387679319 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe79806bab0 con 0x7fe79812e330 2021-05-25T12:02:35.853 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.805+0000 7fe79dcdc700 10 monclient: get_auth_request con 0x7fe778033140 auth_method 0 2021-05-25T12:02:35.853 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.805+0000 7fe785ffb700 10 client.174147.objecter ms_handle_connect 0x7fe778033140 2021-05-25T12:02:35.853 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.809+0000 7fe785ffb700 1 -- 172.21.15.102:0/3387679319 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7fe7940130b0 con 0x7fe79812e330 2021-05-25T12:02:35.854 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.809+0000 7fe785ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:02:35.854 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:35.809+0000 7fe785ffb700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:02:36.019 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:36.017+0000 7fe79ff40700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2021-05-25T12:02:36.019 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:36.017+0000 7fe79ff40700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2021-05-25T12:02:36.019 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:36.017+0000 7fe79ff40700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:02:36.019 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:36.017+0000 7fe79ff40700 1 -- 172.21.15.102:0/3387679319 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fe798059730 con 0x7fe79812e330 2021-05-25T12:02:36.020 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:36.017+0000 7fe785ffb700 1 -- 172.21.15.102:0/3387679319 <== mon.0 v1:172.21.15.102:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1337 (unknown 254746657 0 3555053442) 0x7fe794012e80 con 0x7fe79812e330 2021-05-25T12:02:36.020 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:36.017+0000 7fe785ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2021-05-25T12:02:36.020 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:36.017+0000 7fe785ffb700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:02:36.021 INFO:teuthology.orchestra.run.smithi102.stdout:{"election_epoch":26,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"b","quorum_age":16,"features":{"quorum_con":"4540138303579357183","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"20ee91b6-8935-4b98-b868-85c03789ccc4","modified":"2021-05-25T11:59:11.011522Z","created":"2021-05-25T11:59:11.011522Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6790","nonce":0}]},"addr":"172.21.15.102:6790/0","public_addr":"172.21.15.102:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2021-05-25T12:02:36.022 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:36.018+0000 7fe79ff40700 10 librados: watch_flush enter 2021-05-25T12:02:36.023 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:36.019+0000 7fe79ff40700 10 librados: watch_flush exit 2021-05-25T12:02:36.024 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:36.019+0000 7fe79ff40700 20 client.174147.objecter shutdown clearing up homeless session... 2021-05-25T12:02:36.024 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:36.019+0000 7fe79ff40700 10 client.174147.objecter successfully canceled tick 2021-05-25T12:02:36.024 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:36.019+0000 7fe79ff40700 1 -- 172.21.15.102:0/3387679319 >> v1:172.21.15.181:6812/29582 conn(0x7fe778033140 legacy=0x7fe778035600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:36.024 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:36.019+0000 7fe79ff40700 10 monclient: shutdown 2021-05-25T12:02:36.025 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:36.019+0000 7fe79ff40700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:02:36.025 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:36.019+0000 7fe79ff40700 1 -- 172.21.15.102:0/3387679319 >> v1:172.21.15.102:6789/0 conn(0x7fe79812e330 legacy=0x7fe7981438e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:36.025 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:36.019+0000 7fe79ff40700 1 -- 172.21.15.102:0/3387679319 shutdown_connections 2021-05-25T12:02:36.025 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:36.019+0000 7fe79ff40700 1 -- 172.21.15.102:0/3387679319 >> 172.21.15.102:0/3387679319 conn(0x7fe79809b370 msgr2=0x7fe798099f60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:02:36.025 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:36.019+0000 7fe79ff40700 1 -- 172.21.15.102:0/3387679319 shutdown_connections 2021-05-25T12:02:36.026 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:36.019+0000 7fe79ff40700 1 -- 172.21.15.102:0/3387679319 wait complete. 2021-05-25T12:02:36.026 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:36.019+0000 7fe79ff40700 1 librados: shutdown 2021-05-25T12:02:39.032 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2021-05-25T12:02:39.170 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.159+0000 7efbf676a700 10 monclient: get_monmap_and_config 2021-05-25T12:02:39.170 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.159+0000 7efbf676a700 10 monclient: build_initial_monmap 2021-05-25T12:02:39.171 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.159+0000 7efbf676a700 10 monclient: monmap: 2021-05-25T12:02:39.171 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:02:39.171 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:39.171 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:02:39.160030+0000 2021-05-25T12:02:39.172 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:02:39.160030+0000 2021-05-25T12:02:39.172 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:02:39.172 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:39.172 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:02:39.173 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:02:39.173 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:02:39.173 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:39.173 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.159+0000 7efbf676a700 1 Processor -- start 2021-05-25T12:02:39.173 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.159+0000 7efbf676a700 1 -- start start 2021-05-25T12:02:39.174 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.159+0000 7efbf676a700 10 monclient: init 2021-05-25T12:02:39.174 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.160+0000 7efbf676a700 10 monclient: _reopen_session rank -1 2021-05-25T12:02:39.174 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.160+0000 7efbf676a700 10 monclient: _add_conns ranks=[2,0,1] 2021-05-25T12:02:39.174 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.160+0000 7efbf676a700 10 monclient(hunting): picked mon.noname-b con 0x7efbf0064740 addr v1:172.21.15.102:6790/0 2021-05-25T12:02:39.175 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.160+0000 7efbf676a700 10 monclient(hunting): picked mon.noname-a con 0x7efbf0096b60 addr v1:172.21.15.102:6789/0 2021-05-25T12:02:39.175 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.160+0000 7efbf676a700 10 monclient(hunting): picked mon.noname-c con 0x7efbf01327f0 addr v1:172.21.15.181:6789/0 2021-05-25T12:02:39.175 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.160+0000 7efbf676a700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efbf0098410 con 0x7efbf0096b60 2021-05-25T12:02:39.175 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.160+0000 7efbf676a700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efbf0063ac0 con 0x7efbf01327f0 2021-05-25T12:02:39.176 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.160+0000 7efbf676a700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efbf0063d60 con 0x7efbf0064740 2021-05-25T12:02:39.176 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.160+0000 7efbf676a700 10 monclient(hunting): _renew_subs 2021-05-25T12:02:39.176 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.160+0000 7efbf676a700 10 monclient(hunting): authenticate will time out at 959.757397s 2021-05-25T12:02:39.176 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.160+0000 7efbef7fe700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7efbf0096b60 0x7efbf0097d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60176/0 (socket says 172.21.15.102:60176) 2021-05-25T12:02:39.176 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.160+0000 7efbeffff700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7efbf0064740 0x7efbf00975f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:55052/0 (socket says 172.21.15.102:55052) 2021-05-25T12:02:39.177 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.160+0000 7efbef7fe700 1 -- 172.21.15.102:0/1720821257 learned_addr learned my addr 172.21.15.102:0/1720821257 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:02:39.177 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.160+0000 7efbeeffd700 1 -- 172.21.15.102:0/1720821257 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3025143526 0 0) 0x7efbf0063d60 con 0x7efbf0064740 2021-05-25T12:02:39.177 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.160+0000 7efbeeffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:02:39.177 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.160+0000 7efbeeffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:02:39.178 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.160+0000 7efbeeffd700 10 monclient(hunting): my global_id is 23027 2021-05-25T12:02:39.178 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.160+0000 7efbeeffd700 1 -- 172.21.15.102:0/1720821257 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efbd8003270 con 0x7efbf0064740 2021-05-25T12:02:39.178 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.160+0000 7efbeeffd700 1 -- 172.21.15.102:0/1720821257 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2318992356 0 0) 0x7efbf0063ac0 con 0x7efbf01327f0 2021-05-25T12:02:39.178 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.160+0000 7efbeeffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:02:39.179 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.160+0000 7efbeeffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:02:39.179 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.160+0000 7efbeeffd700 10 monclient(hunting): my global_id is 95101 2021-05-25T12:02:39.179 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.160+0000 7efbeeffd700 1 -- 172.21.15.102:0/1720821257 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efbf0063d60 con 0x7efbf01327f0 2021-05-25T12:02:39.179 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.161+0000 7efbeeffd700 1 -- 172.21.15.102:0/1720821257 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4003313801 0 0) 0x7efbd8003270 con 0x7efbf0064740 2021-05-25T12:02:39.180 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.161+0000 7efbeeffd700 1 -- 172.21.15.102:0/1720821257 >> v1:172.21.15.181:6789/0 conn(0x7efbf01327f0 legacy=0x7efbf0134be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:39.180 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.161+0000 7efbeeffd700 1 -- 172.21.15.102:0/1720821257 >> v1:172.21.15.102:6789/0 conn(0x7efbf0096b60 legacy=0x7efbf0097d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:39.180 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.161+0000 7efbeeffd700 10 monclient: _finish_hunting 0 2021-05-25T12:02:39.180 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.161+0000 7efbeeffd700 1 monclient: found mon.noname-b 2021-05-25T12:02:39.180 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.161+0000 7efbeeffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:02:39.181 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.161+0000 7efbeeffd700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T12:02:39.181 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.161+0000 7efbeeffd700 1 -- 172.21.15.102:0/1720821257 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbf0064000 con 0x7efbf0064740 2021-05-25T12:02:39.181 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.161+0000 7efbeeffd700 10 monclient: _finish_auth 0 2021-05-25T12:02:39.181 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.161+0000 7efbeeffd700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:02:39.181 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.161+0000 7efbeeffd700 1 -- 172.21.15.102:0/1720821257 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7efbd4001610 con 0x7efbf0064740 2021-05-25T12:02:39.182 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.161+0000 7efbf676a700 5 monclient: authenticate success, global_id 23027 2021-05-25T12:02:39.182 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.161+0000 7efbf676a700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:02:39.182 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.161+0000 7efbeeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:39.183 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.161+0000 7efbeeffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:02:39.183 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.161+0000 7efbeeffd700 10 monclient: dump: 2021-05-25T12:02:39.183 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:39.183 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:39.184 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:39.184 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:39.184 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:39.184 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:39.184 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:39.185 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:39.185 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:39.185 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:39.185 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.161+0000 7efbf676a700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:02:39.186 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.161+0000 7efbeeffd700 1 -- 172.21.15.102:0/1720821257 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efbd4001880 con 0x7efbf0064740 2021-05-25T12:02:39.186 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.161+0000 7efbeeffd700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:02:39.186 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.161+0000 7efbeeffd700 1 -- 172.21.15.102:0/1720821257 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7efbd4001c10 con 0x7efbf0064740 2021-05-25T12:02:39.186 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.161+0000 7efbf676a700 10 monclient: get_monmap_and_config success 2021-05-25T12:02:39.186 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.161+0000 7efbf676a700 4 set_mon_vals no callback set 2021-05-25T12:02:39.187 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.161+0000 7efbeeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:39.187 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.161+0000 7efbeeffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:02:39.187 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.161+0000 7efbeeffd700 10 monclient: dump: 2021-05-25T12:02:39.187 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:39.188 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:39.188 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:39.188 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:39.188 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:39.189 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:39.189 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:39.189 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:39.189 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:39.189 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:39.190 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.161+0000 7efbf676a700 10 monclient: shutdown 2021-05-25T12:02:39.190 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.161+0000 7efbf676a700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:02:39.190 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.161+0000 7efbf676a700 1 -- 172.21.15.102:0/1720821257 >> v1:172.21.15.102:6790/0 conn(0x7efbf0064740 legacy=0x7efbf00975f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:39.190 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.161+0000 7efbf676a700 1 -- 172.21.15.102:0/1720821257 shutdown_connections 2021-05-25T12:02:39.191 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.161+0000 7efbf676a700 1 -- 172.21.15.102:0/1720821257 >> 172.21.15.102:0/1720821257 conn(0x7efbf00625c0 msgr2=0x7efbf00996f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:02:39.191 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.162+0000 7efbf676a700 1 -- 172.21.15.102:0/1720821257 shutdown_connections 2021-05-25T12:02:39.191 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.162+0000 7efbf676a700 1 -- 172.21.15.102:0/1720821257 wait complete. 2021-05-25T12:02:39.191 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.162+0000 7efbf676a700 10 monclient: build_initial_monmap 2021-05-25T12:02:39.192 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.162+0000 7efbf676a700 10 monclient: monmap: 2021-05-25T12:02:39.192 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:02:39.192 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:02:39.192 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:02:39.192 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:02:39.193 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:02:39.193 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:02:39.193 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:02:39.193 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:02:39.194 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:02:39.194 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:39.194 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.162+0000 7efbf676a700 1 librados: starting msgr at 2021-05-25T12:02:39.194 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.162+0000 7efbf676a700 1 librados: starting objecter 2021-05-25T12:02:39.195 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.162+0000 7efbf676a700 1 Processor -- start 2021-05-25T12:02:39.195 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.163+0000 7efbf676a700 1 -- start start 2021-05-25T12:02:39.195 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.163+0000 7efbf676a700 1 librados: setting wanted keys 2021-05-25T12:02:39.195 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.163+0000 7efbf676a700 1 librados: calling monclient init 2021-05-25T12:02:39.196 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.163+0000 7efbf676a700 10 monclient: init 2021-05-25T12:02:39.196 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.163+0000 7efbf676a700 10 monclient: _reopen_session rank -1 2021-05-25T12:02:39.196 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.163+0000 7efbf676a700 10 monclient: _add_conns ranks=[1,2,0] 2021-05-25T12:02:39.196 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.163+0000 7efbf676a700 10 monclient(hunting): picked mon.noname-b con 0x7efbf0064740 addr v1:172.21.15.181:6789/0 2021-05-25T12:02:39.196 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.163+0000 7efbf676a700 10 monclient(hunting): picked mon.noname-c con 0x7efbf0096b60 addr v1:172.21.15.102:6790/0 2021-05-25T12:02:39.197 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.163+0000 7efbf676a700 10 monclient(hunting): picked mon.noname-a con 0x7efbf01327f0 addr v1:172.21.15.102:6789/0 2021-05-25T12:02:39.197 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.163+0000 7efbf676a700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efbf0118fb0 con 0x7efbf01327f0 2021-05-25T12:02:39.197 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.163+0000 7efbf676a700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efbf0119250 con 0x7efbf0064740 2021-05-25T12:02:39.197 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.163+0000 7efbf676a700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efbf01194f0 con 0x7efbf0096b60 2021-05-25T12:02:39.198 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.163+0000 7efbf676a700 10 monclient(hunting): _renew_subs 2021-05-25T12:02:39.198 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.163+0000 7efbf676a700 10 monclient(hunting): authenticate will time out at 959.760610s 2021-05-25T12:02:39.198 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.163+0000 7efbf4d07700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7efbf01327f0 0x7efbf01432f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60184/0 (socket says 172.21.15.102:60184) 2021-05-25T12:02:39.198 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.163+0000 7efbf4d07700 1 -- 172.21.15.102:0/1722099660 learned_addr learned my addr 172.21.15.102:0/1722099660 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:02:39.199 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.163+0000 7efbe3fff700 10 client.?.objecter ms_handle_connect 0x7efbf0096b60 2021-05-25T12:02:39.199 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.163+0000 7efbe3fff700 10 client.?.objecter resend_mon_ops 2021-05-25T12:02:39.199 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.163+0000 7efbe3fff700 10 client.?.objecter ms_handle_connect 0x7efbf0064740 2021-05-25T12:02:39.199 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.163+0000 7efbe3fff700 10 client.?.objecter resend_mon_ops 2021-05-25T12:02:39.199 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.163+0000 7efbe3fff700 10 client.?.objecter ms_handle_connect 0x7efbf01327f0 2021-05-25T12:02:39.200 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.163+0000 7efbe3fff700 10 client.?.objecter resend_mon_ops 2021-05-25T12:02:39.200 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.163+0000 7efbe3fff700 1 -- 172.21.15.102:0/1722099660 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 577319149 0 0) 0x7efbf01194f0 con 0x7efbf0096b60 2021-05-25T12:02:39.200 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.163+0000 7efbe3fff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:02:39.200 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.163+0000 7efbe3fff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:02:39.201 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.163+0000 7efbe3fff700 10 monclient(hunting): my global_id is 23030 2021-05-25T12:02:39.201 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.163+0000 7efbe3fff700 1 -- 172.21.15.102:0/1722099660 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efbcc004c10 con 0x7efbf0096b60 2021-05-25T12:02:39.201 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.163+0000 7efbe3fff700 1 -- 172.21.15.102:0/1722099660 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 460034425 0 0) 0x7efbf0119250 con 0x7efbf0064740 2021-05-25T12:02:39.201 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.163+0000 7efbe3fff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:02:39.201 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.163+0000 7efbe3fff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:02:39.202 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.163+0000 7efbe3fff700 10 monclient(hunting): my global_id is 95104 2021-05-25T12:02:39.202 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.163+0000 7efbe3fff700 1 -- 172.21.15.102:0/1722099660 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efbf01194f0 con 0x7efbf0064740 2021-05-25T12:02:39.202 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.164+0000 7efbe3fff700 1 -- 172.21.15.102:0/1722099660 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 325228338 0 0) 0x7efbf01194f0 con 0x7efbf0064740 2021-05-25T12:02:39.202 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.164+0000 7efbe3fff700 1 -- 172.21.15.102:0/1722099660 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efbf0119250 con 0x7efbf0064740 2021-05-25T12:02:39.203 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.164+0000 7efbe3fff700 1 -- 172.21.15.102:0/1722099660 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7efbd4001690 con 0x7efbf0064740 2021-05-25T12:02:39.203 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.164+0000 7efbe3fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:39.203 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.164+0000 7efbe3fff700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:02:39.203 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.164+0000 7efbe3fff700 10 monclient(hunting): dump: 2021-05-25T12:02:39.203 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:39.204 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:39.204 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:39.204 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:39.204 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:39.205 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:39.205 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:39.205 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:39.205 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:39.205 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:39.206 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.164+0000 7efbe3fff700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:02:39.206 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.164+0000 7efbe3fff700 1 -- 172.21.15.102:0/1722099660 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4276309634 0 0) 0x7efbcc004c10 con 0x7efbf0096b60 2021-05-25T12:02:39.206 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.164+0000 7efbe3fff700 1 -- 172.21.15.102:0/1722099660 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efbf01194f0 con 0x7efbf0096b60 2021-05-25T12:02:39.206 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.164+0000 7efbe3fff700 1 -- 172.21.15.102:0/1722099660 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7efbdc001770 con 0x7efbf0096b60 2021-05-25T12:02:39.207 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.164+0000 7efbe3fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:39.207 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.164+0000 7efbe3fff700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2021-05-25T12:02:39.207 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.164+0000 7efbe3fff700 10 monclient(hunting): dump: 2021-05-25T12:02:39.207 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:39.207 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:39.208 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:39.208 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:39.208 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:39.208 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:39.208 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:39.209 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:39.209 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:39.209 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:39.209 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.164+0000 7efbe3fff700 1 -- 172.21.15.102:0/1722099660 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3693559303 0 0) 0x7efbf0119250 con 0x7efbf0064740 2021-05-25T12:02:39.210 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.164+0000 7efbe3fff700 1 -- 172.21.15.102:0/1722099660 >> v1:172.21.15.102:6790/0 conn(0x7efbf0096b60 legacy=0x7efbf013b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:39.210 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.164+0000 7efbe3fff700 1 -- 172.21.15.102:0/1722099660 >> v1:172.21.15.102:6789/0 conn(0x7efbf01327f0 legacy=0x7efbf01432f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:39.210 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.164+0000 7efbe3fff700 10 monclient: _finish_hunting 0 2021-05-25T12:02:39.210 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.164+0000 7efbe3fff700 1 monclient: found mon.b 2021-05-25T12:02:39.210 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.164+0000 7efbe3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:02:39.211 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.164+0000 7efbe3fff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T12:02:39.211 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.164+0000 7efbe3fff700 1 -- 172.21.15.102:0/1722099660 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbf009d780 con 0x7efbf0064740 2021-05-25T12:02:39.211 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.164+0000 7efbe3fff700 10 monclient: _finish_auth 0 2021-05-25T12:02:39.211 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.164+0000 7efbe3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:02:39.212 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.164+0000 7efbf676a700 5 monclient: authenticate success, global_id 95104 2021-05-25T12:02:39.212 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.164+0000 7efbf676a700 10 monclient: _renew_subs 2021-05-25T12:02:39.212 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.164+0000 7efbf676a700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T12:02:39.212 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.164+0000 7efbf676a700 1 -- 172.21.15.102:0/1722099660 --> v1:172.21.15.181:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efbf0117fb0 con 0x7efbf0064740 2021-05-25T12:02:39.213 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.164+0000 7efbf676a700 10 client.95104.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:02:39.213 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.164+0000 7efbf676a700 10 monclient: _renew_subs 2021-05-25T12:02:39.213 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.164+0000 7efbf676a700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T12:02:39.213 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.164+0000 7efbf676a700 1 -- 172.21.15.102:0/1722099660 --> v1:172.21.15.181:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efbf012ef70 con 0x7efbf0064740 2021-05-25T12:02:39.213 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.164+0000 7efbf676a700 1 librados: init done 2021-05-25T12:02:39.214 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.164+0000 7efbe3fff700 1 -- 172.21.15.102:0/1722099660 <== mon.1 v1:172.21.15.181:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efbd4004920 con 0x7efbf0064740 2021-05-25T12:02:39.214 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.164+0000 7efbe3fff700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:02:39.214 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.164+0000 7efbe3fff700 1 -- 172.21.15.102:0/1722099660 <== mon.1 v1:172.21.15.181:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7efbd4004cb0 con 0x7efbf0064740 2021-05-25T12:02:39.214 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.164+0000 7efbe3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:39.214 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.164+0000 7efbe3fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T12:02:39.215 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.164+0000 7efbe3fff700 10 monclient: dump: 2021-05-25T12:02:39.215 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:39.215 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:39.215 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:39.216 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:39.216 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:39.216 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:39.216 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:39.216 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:39.217 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:39.217 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:39.217 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.164+0000 7efbee7fc700 4 set_mon_vals no callback set 2021-05-25T12:02:39.218 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.165+0000 7efbe3fff700 1 -- 172.21.15.102:0/1722099660 <== mon.1 v1:172.21.15.181:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7efbd4004e10 con 0x7efbf0064740 2021-05-25T12:02:39.218 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.165+0000 7efbe3fff700 1 -- 172.21.15.102:0/1722099660 <== mon.1 v1:172.21.15.181:6789/0 8 ==== osd_map(77..77 src has 1..77) v4 ==== 7909+0+0 (unknown 2542179268 0 0) 0x7efbd403e0d0 con 0x7efbf0064740 2021-05-25T12:02:39.218 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.165+0000 7efbe3fff700 10 client.95104.objecter ms_dispatch 0x7efbf0001550 osd_map(77..77 src has 1..77) v4 2021-05-25T12:02:39.218 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.165+0000 7efbe3fff700 3 client.95104.objecter handle_osd_map got epochs [77,77] > 0 2021-05-25T12:02:39.218 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.165+0000 7efbe3fff700 3 client.95104.objecter handle_osd_map decoding full epoch 77 2021-05-25T12:02:39.219 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.165+0000 7efbf676a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:02:39.219 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.165+0000 7efbf676a700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:02:39.219 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.165+0000 7efbf676a700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T12:02:39.219 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.165+0000 7efbf676a700 1 -- 172.21.15.102:0/1722099660 --> v1:172.21.15.181:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efbf0059730 con 0x7efbf0064740 2021-05-25T12:02:39.220 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.165+0000 7efbe3fff700 20 client.95104.objecter dump_active .. 0 homeless 2021-05-25T12:02:39.220 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.165+0000 7efbef7fe700 10 monclient: get_auth_request con 0x7efbcc0332d0 auth_method 0 2021-05-25T12:02:39.220 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.166+0000 7efbe3fff700 10 client.95104.objecter ms_handle_connect 0x7efbcc0332d0 2021-05-25T12:02:39.221 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.170+0000 7efbe3fff700 1 -- 172.21.15.102:0/1722099660 <== mon.1 v1:172.21.15.181:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7efbd4012f20 con 0x7efbf0064740 2021-05-25T12:02:39.221 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.170+0000 7efbe3fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:02:39.221 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.170+0000 7efbe3fff700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:02:39.404 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.403+0000 7efbf676a700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2021-05-25T12:02:39.405 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.403+0000 7efbf676a700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2021-05-25T12:02:39.405 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.403+0000 7efbf676a700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T12:02:39.405 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.403+0000 7efbf676a700 1 -- 172.21.15.102:0/1722099660 --> v1:172.21.15.181:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7efbf006a0f0 con 0x7efbf0064740 2021-05-25T12:02:39.405 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.403+0000 7efbe3fff700 1 -- 172.21.15.102:0/1722099660 <== mon.1 v1:172.21.15.181:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1336 (unknown 254746657 0 46650154) 0x7efbd4012cf0 con 0x7efbf0064740 2021-05-25T12:02:39.406 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.403+0000 7efbe3fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2021-05-25T12:02:39.406 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.403+0000 7efbe3fff700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:02:39.406 INFO:teuthology.orchestra.run.smithi102.stdout:{"election_epoch":28,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"c","quorum_age":2,"features":{"quorum_con":"4540138303579357183","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"20ee91b6-8935-4b98-b868-85c03789ccc4","modified":"2021-05-25T11:59:11.011522Z","created":"2021-05-25T11:59:11.011522Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6790","nonce":0}]},"addr":"172.21.15.102:6790/0","public_addr":"172.21.15.102:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2021-05-25T12:02:39.408 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.405+0000 7efbf676a700 10 librados: watch_flush enter 2021-05-25T12:02:39.408 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.405+0000 7efbf676a700 10 librados: watch_flush exit 2021-05-25T12:02:39.409 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.405+0000 7efbf676a700 20 client.95104.objecter shutdown clearing up homeless session... 2021-05-25T12:02:39.409 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.405+0000 7efbf676a700 10 client.95104.objecter successfully canceled tick 2021-05-25T12:02:39.409 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.405+0000 7efbf676a700 1 -- 172.21.15.102:0/1722099660 >> v1:172.21.15.181:6812/29582 conn(0x7efbcc0332d0 legacy=0x7efbcc035790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:39.409 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.405+0000 7efbf676a700 10 monclient: shutdown 2021-05-25T12:02:39.409 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.405+0000 7efbf676a700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:02:39.410 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.405+0000 7efbf676a700 1 -- 172.21.15.102:0/1722099660 >> v1:172.21.15.181:6789/0 conn(0x7efbf0064740 legacy=0x7efbf013afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:39.410 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.405+0000 7efbf676a700 1 -- 172.21.15.102:0/1722099660 shutdown_connections 2021-05-25T12:02:39.410 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.405+0000 7efbf676a700 1 -- 172.21.15.102:0/1722099660 >> 172.21.15.102:0/1722099660 conn(0x7efbf00625c0 msgr2=0x7efbf00930b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:02:39.410 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.405+0000 7efbf676a700 1 -- 172.21.15.102:0/1722099660 shutdown_connections 2021-05-25T12:02:39.411 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.405+0000 7efbf676a700 1 -- 172.21.15.102:0/1722099660 wait complete. 2021-05-25T12:02:39.411 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:39.405+0000 7efbf676a700 1 librados: shutdown 2021-05-25T12:02:40.646 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots: Running main() from gmock_main.cc 2021-05-25T12:02:40.647 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots: [==========] Running 12 tests from 4 test suites. 2021-05-25T12:02:40.647 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots: [----------] Global test environment set-up. 2021-05-25T12:02:40.647 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots 2021-05-25T12:02:40.647 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapList 2021-05-25T12:02:40.647 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapList (2062 ms) 2021-05-25T12:02:40.648 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapRemove 2021-05-25T12:02:40.648 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapRemove (1964 ms) 2021-05-25T12:02:40.648 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.Rollback 2021-05-25T12:02:40.648 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots: [ OK ] LibRadosSnapshots.Rollback (1977 ms) 2021-05-25T12:02:40.649 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapGetName 2021-05-25T12:02:40.649 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapGetName (2003 ms) 2021-05-25T12:02:40.649 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots (8006 ms total) 2021-05-25T12:02:40.649 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots: 2021-05-25T12:02:40.649 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManaged 2021-05-25T12:02:40.650 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Snap 2021-05-25T12:02:40.650 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Snap (4094 ms) 2021-05-25T12:02:40.650 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Rollback 2021-05-25T12:02:40.650 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Rollback (3950 ms) 2021-05-25T12:02:40.650 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManaged (8044 ms total) 2021-05-25T12:02:40.651 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots: 2021-05-25T12:02:40.651 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC 2021-05-25T12:02:40.651 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapList 2021-05-25T12:02:40.651 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapList (27741 ms) 2021-05-25T12:02:40.651 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapRemove 2021-05-25T12:02:40.652 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapRemove (2059 ms) 2021-05-25T12:02:40.652 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.Rollback 2021-05-25T12:02:40.652 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.Rollback (1913 ms) 2021-05-25T12:02:40.652 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapGetName 2021-05-25T12:02:40.652 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapGetName (2005 ms) 2021-05-25T12:02:40.653 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC (33718 ms total) 2021-05-25T12:02:40.653 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots: 2021-05-25T12:02:40.653 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC 2021-05-25T12:02:40.653 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Snap 2021-05-25T12:02:40.653 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Snap (4004 ms) 2021-05-25T12:02:40.654 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Rollback 2021-05-25T12:02:40.654 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Rollback (34876 ms) 2021-05-25T12:02:40.654 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC (38880 ms total) 2021-05-25T12:02:40.655 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots: 2021-05-25T12:02:40.655 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots: [----------] Global test environment tear-down 2021-05-25T12:02:40.655 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots: [==========] 12 tests from 4 test suites ran. (120883 ms total) 2021-05-25T12:02:40.655 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots: [ PASSED ] 12 tests. 2021-05-25T12:02:40.656 INFO:tasks.workunit.client.0.smithi181.stderr:+ for t in "${!pids[@]}" 2021-05-25T12:02:40.657 INFO:tasks.workunit.client.0.smithi181.stderr:+ pid=33323 2021-05-25T12:02:40.657 INFO:tasks.workunit.client.0.smithi181.stderr:+ wait 33323 2021-05-25T12:02:40.657 INFO:tasks.workunit.client.0.smithi181.stderr:+ for t in "${!pids[@]}" 2021-05-25T12:02:40.657 INFO:tasks.workunit.client.0.smithi181.stderr:+ pid=32869 2021-05-25T12:02:40.658 INFO:tasks.workunit.client.0.smithi181.stderr:+ wait 32869 2021-05-25T12:02:40.658 INFO:tasks.workunit.client.0.smithi181.stderr:+ for t in "${!pids[@]}" 2021-05-25T12:02:40.658 INFO:tasks.workunit.client.0.smithi181.stderr:+ pid=33194 2021-05-25T12:02:40.658 INFO:tasks.workunit.client.0.smithi181.stderr:+ wait 33194 2021-05-25T12:02:40.658 INFO:tasks.workunit.client.0.smithi181.stderr:+ for t in "${!pids[@]}" 2021-05-25T12:02:40.659 INFO:tasks.workunit.client.0.smithi181.stderr:+ pid=33067 2021-05-25T12:02:40.659 INFO:tasks.workunit.client.0.smithi181.stderr:+ wait 33067 2021-05-25T12:02:41.764 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: [==========] Running 31 tests from 7 test suites. 2021-05-25T12:02:41.764 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: [----------] Global test environment set-up. 2021-05-25T12:02:41.764 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion 2021-05-25T12:02:41.765 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: [ RUN ] LibRadosMiscVersion.VersionPP 2021-05-25T12:02:41.765 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: [ OK ] LibRadosMiscVersion.VersionPP (0 ms) 2021-05-25T12:02:41.765 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2021-05-25T12:02:41.765 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: 2021-05-25T12:02:41.765 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP 2021-05-25T12:02:41.766 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: seed 33001 2021-05-25T12:02:41.766 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WaitOSDMapPP 2021-05-25T12:02:41.766 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WaitOSDMapPP (0 ms) 2021-05-25T12:02:41.766 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNamePP 2021-05-25T12:02:41.767 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNamePP (9 ms) 2021-05-25T12:02:41.767 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongLocatorPP 2021-05-25T12:02:41.767 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongLocatorPP (12 ms) 2021-05-25T12:02:41.767 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNSpacePP 2021-05-25T12:02:41.767 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNSpacePP (10 ms) 2021-05-25T12:02:41.768 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongAttrNamePP 2021-05-25T12:02:41.768 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongAttrNamePP (12 ms) 2021-05-25T12:02:41.768 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.ExecPP 2021-05-25T12:02:41.768 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.ExecPP (2 ms) 2021-05-25T12:02:41.769 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BadFlagsPP 2021-05-25T12:02:41.769 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BadFlagsPP (4 ms) 2021-05-25T12:02:41.769 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate1PP 2021-05-25T12:02:41.769 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate1PP (5 ms) 2021-05-25T12:02:41.769 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate2PP 2021-05-25T12:02:41.770 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate2PP (3 ms) 2021-05-25T12:02:41.770 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigObjectPP 2021-05-25T12:02:41.770 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigObjectPP (18 ms) 2021-05-25T12:02:41.771 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AioOperatePP 2021-05-25T12:02:41.771 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AioOperatePP (4 ms) 2021-05-25T12:02:41.771 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertExistsPP 2021-05-25T12:02:41.771 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertExistsPP (15 ms) 2021-05-25T12:02:41.771 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertVersionPP 2021-05-25T12:02:41.772 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertVersionPP (23 ms) 2021-05-25T12:02:41.772 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigAttrPP 2021-05-25T12:02:41.772 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: osd_max_attr_size = 0 2021-05-25T12:02:41.772 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: osd_max_attr_size == 0; skipping test 2021-05-25T12:02:41.773 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigAttrPP (2403 ms) 2021-05-25T12:02:41.773 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyPP 2021-05-25T12:02:41.773 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyPP (1059 ms) 2021-05-25T12:02:41.773 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyScrubPP 2021-05-25T12:02:41.773 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: waiting for initial deep scrubs... 2021-05-25T12:02:41.774 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: done waiting, doing copies 2021-05-25T12:02:41.774 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: waiting for final deep scrubs... 2021-05-25T12:02:41.774 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: done waiting 2021-05-25T12:02:41.774 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyScrubPP (70734 ms) 2021-05-25T12:02:41.774 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WriteSamePP 2021-05-25T12:02:41.775 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WriteSamePP (7 ms) 2021-05-25T12:02:41.775 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CmpExtPP 2021-05-25T12:02:41.775 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CmpExtPP (3 ms) 2021-05-25T12:02:41.775 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Applications 2021-05-25T12:02:41.776 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Applications (30780 ms) 2021-05-25T12:02:41.776 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatOSD 2021-05-25T12:02:41.776 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatOSD (0 ms) 2021-05-25T12:02:41.776 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatClient 2021-05-25T12:02:41.776 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatClient (0 ms) 2021-05-25T12:02:41.777 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Conf 2021-05-25T12:02:41.777 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Conf (0 ms) 2021-05-25T12:02:42.420 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2021-05-25T12:02:42.530 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.529+0000 7ff57aa20700 10 monclient: get_monmap_and_config 2021-05-25T12:02:42.531 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.529+0000 7ff57aa20700 10 monclient: build_initial_monmap 2021-05-25T12:02:42.531 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.529+0000 7ff57aa20700 10 monclient: monmap: 2021-05-25T12:02:42.531 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:02:42.531 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:42.531 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:02:42.530107+0000 2021-05-25T12:02:42.532 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:02:42.530107+0000 2021-05-25T12:02:42.532 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:02:42.532 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:42.532 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:02:42.533 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:02:42.533 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:02:42.533 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:42.544 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.529+0000 7ff57aa20700 1 Processor -- start 2021-05-25T12:02:42.544 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.529+0000 7ff57aa20700 1 -- start start 2021-05-25T12:02:42.545 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.529+0000 7ff57aa20700 10 monclient: init 2021-05-25T12:02:42.545 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.530+0000 7ff57aa20700 10 monclient: _reopen_session rank -1 2021-05-25T12:02:42.545 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.530+0000 7ff57aa20700 10 monclient: _add_conns ranks=[2,1,0] 2021-05-25T12:02:42.545 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.530+0000 7ff57aa20700 10 monclient(hunting): picked mon.noname-b con 0x7ff574068d50 addr v1:172.21.15.102:6790/0 2021-05-25T12:02:42.545 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.530+0000 7ff57aa20700 10 monclient(hunting): picked mon.noname-c con 0x7ff574069130 addr v1:172.21.15.181:6789/0 2021-05-25T12:02:42.546 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.530+0000 7ff57aa20700 10 monclient(hunting): picked mon.noname-a con 0x7ff5740695e0 addr v1:172.21.15.102:6789/0 2021-05-25T12:02:42.546 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.530+0000 7ff57aa20700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff574094380 con 0x7ff5740695e0 2021-05-25T12:02:42.546 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.530+0000 7ff57aa20700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff574094620 con 0x7ff574069130 2021-05-25T12:02:42.546 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.530+0000 7ff57aa20700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5740948c0 con 0x7ff574068d50 2021-05-25T12:02:42.547 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.530+0000 7ff57aa20700 10 monclient(hunting): _renew_subs 2021-05-25T12:02:42.547 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.530+0000 7ff57aa20700 10 monclient(hunting): authenticate will time out at 963.127402s 2021-05-25T12:02:42.547 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.530+0000 7ff578fbd700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7ff5740695e0 0x7ff574132160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60198/0 (socket says 172.21.15.102:60198) 2021-05-25T12:02:42.547 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.530+0000 7ff578fbd700 1 -- 172.21.15.102:0/1010959454 learned_addr learned my addr 172.21.15.102:0/1010959454 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:02:42.547 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.530+0000 7ff573fff700 1 --1- 172.21.15.102:0/1010959454 >> v1:172.21.15.102:6790/0 conn(0x7ff574068d50 0x7ff574129c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:55072/0 (socket says 172.21.15.102:55072) 2021-05-25T12:02:42.548 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.530+0000 7ff572ffd700 1 -- 172.21.15.102:0/1010959454 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 726879746 0 0) 0x7ff5740948c0 con 0x7ff574068d50 2021-05-25T12:02:42.548 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.530+0000 7ff572ffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:02:42.548 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.530+0000 7ff572ffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:02:42.548 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.530+0000 7ff572ffd700 10 monclient(hunting): my global_id is 23051 2021-05-25T12:02:42.549 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.530+0000 7ff572ffd700 1 -- 172.21.15.102:0/1010959454 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff558003270 con 0x7ff574068d50 2021-05-25T12:02:42.549 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.530+0000 7ff572ffd700 1 -- 172.21.15.102:0/1010959454 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1167494080 0 0) 0x7ff574094380 con 0x7ff5740695e0 2021-05-25T12:02:42.549 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.530+0000 7ff572ffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:02:42.549 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.530+0000 7ff572ffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:02:42.549 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.530+0000 7ff572ffd700 10 monclient(hunting): my global_id is 174156 2021-05-25T12:02:42.550 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.530+0000 7ff572ffd700 1 -- 172.21.15.102:0/1010959454 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5740948c0 con 0x7ff5740695e0 2021-05-25T12:02:42.550 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.530+0000 7ff572ffd700 1 -- 172.21.15.102:0/1010959454 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 584496980 0 0) 0x7ff5740948c0 con 0x7ff5740695e0 2021-05-25T12:02:42.550 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.530+0000 7ff572ffd700 1 -- 172.21.15.102:0/1010959454 >> v1:172.21.15.102:6790/0 conn(0x7ff574068d50 legacy=0x7ff574129c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:42.551 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.530+0000 7ff572ffd700 1 -- 172.21.15.102:0/1010959454 >> v1:172.21.15.181:6789/0 conn(0x7ff574069130 legacy=0x7ff57412ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:42.551 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.530+0000 7ff572ffd700 10 monclient: _finish_hunting 0 2021-05-25T12:02:42.551 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.530+0000 7ff572ffd700 1 monclient: found mon.noname-a 2021-05-25T12:02:42.551 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.530+0000 7ff572ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:02:42.552 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.531+0000 7ff572ffd700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T12:02:42.552 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.531+0000 7ff572ffd700 1 -- 172.21.15.102:0/1010959454 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff574094c80 con 0x7ff5740695e0 2021-05-25T12:02:42.552 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.531+0000 7ff572ffd700 10 monclient: _finish_auth 0 2021-05-25T12:02:42.552 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.531+0000 7ff572ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:02:42.552 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.531+0000 7ff572ffd700 1 -- 172.21.15.102:0/1010959454 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7ff568001990 con 0x7ff5740695e0 2021-05-25T12:02:42.553 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.531+0000 7ff572ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:42.553 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.531+0000 7ff572ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:02:42.553 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.531+0000 7ff572ffd700 10 monclient: dump: 2021-05-25T12:02:42.553 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:42.553 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:42.554 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:42.554 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:42.554 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:42.554 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:42.555 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:42.555 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:42.555 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:42.555 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:42.555 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.531+0000 7ff57aa20700 5 monclient: authenticate success, global_id 174156 2021-05-25T12:02:42.556 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.531+0000 7ff57aa20700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:02:42.556 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.531+0000 7ff572ffd700 1 -- 172.21.15.102:0/1010959454 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff568001bb0 con 0x7ff5740695e0 2021-05-25T12:02:42.556 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.531+0000 7ff572ffd700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:02:42.556 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.531+0000 7ff572ffd700 1 -- 172.21.15.102:0/1010959454 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7ff568001f40 con 0x7ff5740695e0 2021-05-25T12:02:42.556 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.531+0000 7ff57aa20700 10 monclient: get_monmap_and_config success 2021-05-25T12:02:42.557 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.531+0000 7ff57aa20700 4 set_mon_vals no callback set 2021-05-25T12:02:42.557 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.531+0000 7ff572ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:42.557 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.531+0000 7ff572ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:02:42.557 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.531+0000 7ff572ffd700 10 monclient: dump: 2021-05-25T12:02:42.558 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:42.558 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:42.558 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:42.558 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:42.558 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:42.559 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:42.559 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:42.559 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:42.559 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:42.560 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:42.560 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.531+0000 7ff57aa20700 10 monclient: shutdown 2021-05-25T12:02:42.561 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.531+0000 7ff57aa20700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:02:42.561 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.531+0000 7ff57aa20700 1 -- 172.21.15.102:0/1010959454 >> v1:172.21.15.102:6789/0 conn(0x7ff5740695e0 legacy=0x7ff574132160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:42.562 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.531+0000 7ff57aa20700 1 -- 172.21.15.102:0/1010959454 shutdown_connections 2021-05-25T12:02:42.562 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.531+0000 7ff57aa20700 1 -- 172.21.15.102:0/1010959454 >> 172.21.15.102:0/1010959454 conn(0x7ff57409b370 msgr2=0x7ff574097430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:02:42.562 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.531+0000 7ff57aa20700 1 -- 172.21.15.102:0/1010959454 shutdown_connections 2021-05-25T12:02:42.562 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.531+0000 7ff57aa20700 1 -- 172.21.15.102:0/1010959454 wait complete. 2021-05-25T12:02:42.562 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.532+0000 7ff57aa20700 10 monclient: build_initial_monmap 2021-05-25T12:02:42.563 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.532+0000 7ff57aa20700 10 monclient: monmap: 2021-05-25T12:02:42.563 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:02:42.563 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:02:42.563 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:02:42.564 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:02:42.564 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:02:42.564 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:02:42.564 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:02:42.565 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:02:42.565 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:02:42.565 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:42.565 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.532+0000 7ff57aa20700 1 librados: starting msgr at 2021-05-25T12:02:42.565 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.532+0000 7ff57aa20700 1 librados: starting objecter 2021-05-25T12:02:42.566 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.532+0000 7ff57aa20700 1 Processor -- start 2021-05-25T12:02:42.566 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.532+0000 7ff57aa20700 1 -- start start 2021-05-25T12:02:42.566 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.532+0000 7ff57aa20700 1 librados: setting wanted keys 2021-05-25T12:02:42.566 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.532+0000 7ff57aa20700 1 librados: calling monclient init 2021-05-25T12:02:42.567 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.532+0000 7ff57aa20700 10 monclient: init 2021-05-25T12:02:42.567 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.533+0000 7ff57aa20700 10 monclient: _reopen_session rank -1 2021-05-25T12:02:42.567 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.533+0000 7ff57aa20700 10 monclient: _add_conns ranks=[0,2,1] 2021-05-25T12:02:42.567 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.533+0000 7ff57aa20700 10 monclient(hunting): picked mon.noname-a con 0x7ff574068d50 addr v1:172.21.15.102:6789/0 2021-05-25T12:02:42.567 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.533+0000 7ff57aa20700 10 monclient(hunting): picked mon.noname-c con 0x7ff574069130 addr v1:172.21.15.102:6790/0 2021-05-25T12:02:42.568 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.533+0000 7ff57aa20700 10 monclient(hunting): picked mon.noname-b con 0x7ff5740695e0 addr v1:172.21.15.181:6789/0 2021-05-25T12:02:42.568 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.533+0000 7ff57aa20700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff574139b00 con 0x7ff574068d50 2021-05-25T12:02:42.568 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.533+0000 7ff57aa20700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff574124ba0 con 0x7ff5740695e0 2021-05-25T12:02:42.568 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.533+0000 7ff57aa20700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff574091240 con 0x7ff574069130 2021-05-25T12:02:42.569 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.533+0000 7ff57aa20700 10 monclient(hunting): _renew_subs 2021-05-25T12:02:42.569 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.533+0000 7ff57aa20700 10 monclient(hunting): authenticate will time out at 963.130116s 2021-05-25T12:02:42.569 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.533+0000 7ff573fff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7ff574068d50 0x7ff574124490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60200/0 (socket says 172.21.15.102:60200) 2021-05-25T12:02:42.569 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.533+0000 7ff573fff700 1 -- 172.21.15.102:0/1404725600 learned_addr learned my addr 172.21.15.102:0/1404725600 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:02:42.569 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.533+0000 7ff55ffff700 10 client.?.objecter ms_handle_connect 0x7ff574069130 2021-05-25T12:02:42.570 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.533+0000 7ff55ffff700 10 client.?.objecter resend_mon_ops 2021-05-25T12:02:42.570 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.533+0000 7ff55ffff700 10 client.?.objecter ms_handle_connect 0x7ff574068d50 2021-05-25T12:02:42.570 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.533+0000 7ff55ffff700 10 client.?.objecter resend_mon_ops 2021-05-25T12:02:42.570 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.533+0000 7ff55ffff700 1 -- 172.21.15.102:0/1404725600 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 962346452 0 0) 0x7ff574091240 con 0x7ff574069130 2021-05-25T12:02:42.571 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.533+0000 7ff55ffff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:02:42.571 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.533+0000 7ff55ffff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:02:42.571 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.533+0000 7ff55ffff700 10 monclient(hunting): my global_id is 23054 2021-05-25T12:02:42.572 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.533+0000 7ff55ffff700 1 -- 172.21.15.102:0/1404725600 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff554004c10 con 0x7ff574069130 2021-05-25T12:02:42.572 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.533+0000 7ff55ffff700 10 client.?.objecter ms_handle_connect 0x7ff5740695e0 2021-05-25T12:02:42.572 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.533+0000 7ff55ffff700 10 client.?.objecter resend_mon_ops 2021-05-25T12:02:42.572 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.533+0000 7ff55ffff700 1 -- 172.21.15.102:0/1404725600 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2621646723 0 0) 0x7ff574139b00 con 0x7ff574068d50 2021-05-25T12:02:42.573 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.533+0000 7ff55ffff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:02:42.573 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.533+0000 7ff55ffff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:02:42.573 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.533+0000 7ff55ffff700 10 monclient(hunting): my global_id is 174159 2021-05-25T12:02:42.573 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.533+0000 7ff55ffff700 1 -- 172.21.15.102:0/1404725600 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff574091240 con 0x7ff574068d50 2021-05-25T12:02:42.573 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.533+0000 7ff55ffff700 1 -- 172.21.15.102:0/1404725600 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3275513769 0 0) 0x7ff574124ba0 con 0x7ff5740695e0 2021-05-25T12:02:42.574 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.533+0000 7ff55ffff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:02:42.574 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.533+0000 7ff55ffff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:02:42.574 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.533+0000 7ff55ffff700 10 monclient(hunting): my global_id is 95122 2021-05-25T12:02:42.574 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.533+0000 7ff55ffff700 1 -- 172.21.15.102:0/1404725600 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff574139b00 con 0x7ff5740695e0 2021-05-25T12:02:42.575 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.533+0000 7ff55ffff700 1 -- 172.21.15.102:0/1404725600 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3122827057 0 0) 0x7ff554004c10 con 0x7ff574069130 2021-05-25T12:02:42.575 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.533+0000 7ff55ffff700 1 -- 172.21.15.102:0/1404725600 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff574124ba0 con 0x7ff574069130 2021-05-25T12:02:42.575 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.533+0000 7ff55ffff700 1 -- 172.21.15.102:0/1404725600 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7ff560001880 con 0x7ff574069130 2021-05-25T12:02:42.575 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.533+0000 7ff55ffff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:42.575 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.533+0000 7ff55ffff700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:02:42.576 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.533+0000 7ff55ffff700 10 monclient(hunting): dump: 2021-05-25T12:02:42.576 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:42.576 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:42.576 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:42.577 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:42.577 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:42.577 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:42.577 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:42.577 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:42.578 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:42.578 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:42.578 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.533+0000 7ff55ffff700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:02:42.578 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.533+0000 7ff55ffff700 1 -- 172.21.15.102:0/1404725600 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3908049460 0 0) 0x7ff574091240 con 0x7ff574068d50 2021-05-25T12:02:42.578 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.533+0000 7ff55ffff700 1 -- 172.21.15.102:0/1404725600 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff554004c10 con 0x7ff574068d50 2021-05-25T12:02:42.579 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.533+0000 7ff55ffff700 1 -- 172.21.15.102:0/1404725600 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7ff5640018d0 con 0x7ff574068d50 2021-05-25T12:02:42.579 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.533+0000 7ff55ffff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:42.579 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.533+0000 7ff55ffff700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2021-05-25T12:02:42.579 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.533+0000 7ff55ffff700 10 monclient(hunting): dump: 2021-05-25T12:02:42.580 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:42.580 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:42.580 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:42.580 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:42.580 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:42.581 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:42.581 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:42.581 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:42.581 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:42.582 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:42.582 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.533+0000 7ff55ffff700 1 -- 172.21.15.102:0/1404725600 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1227998595 0 0) 0x7ff574139b00 con 0x7ff5740695e0 2021-05-25T12:02:42.582 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.533+0000 7ff55ffff700 1 -- 172.21.15.102:0/1404725600 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff574091240 con 0x7ff5740695e0 2021-05-25T12:02:42.582 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.533+0000 7ff55ffff700 1 -- 172.21.15.102:0/1404725600 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7ff568001950 con 0x7ff5740695e0 2021-05-25T12:02:42.583 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.533+0000 7ff55ffff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:42.583 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.533+0000 7ff55ffff700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2021-05-25T12:02:42.583 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.533+0000 7ff55ffff700 10 monclient(hunting): dump: 2021-05-25T12:02:42.583 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:42.584 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:42.584 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:42.584 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:42.584 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:42.584 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:42.585 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:42.585 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:42.585 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:42.585 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:42.586 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.533+0000 7ff55ffff700 1 -- 172.21.15.102:0/1404725600 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3435374243 0 0) 0x7ff574124ba0 con 0x7ff574069130 2021-05-25T12:02:42.586 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.533+0000 7ff55ffff700 1 -- 172.21.15.102:0/1404725600 >> v1:172.21.15.181:6789/0 conn(0x7ff5740695e0 legacy=0x7ff574140f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:42.586 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.533+0000 7ff55ffff700 1 -- 172.21.15.102:0/1404725600 >> v1:172.21.15.102:6789/0 conn(0x7ff574068d50 legacy=0x7ff574124490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:42.586 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.533+0000 7ff55ffff700 10 monclient: _finish_hunting 0 2021-05-25T12:02:42.586 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.533+0000 7ff55ffff700 1 monclient: found mon.c 2021-05-25T12:02:42.587 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.533+0000 7ff55ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:02:42.587 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.533+0000 7ff55ffff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:02:42.587 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.533+0000 7ff55ffff700 1 -- 172.21.15.102:0/1404725600 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff574123020 con 0x7ff574069130 2021-05-25T12:02:42.587 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.533+0000 7ff55ffff700 10 monclient: _finish_auth 0 2021-05-25T12:02:42.587 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.533+0000 7ff55ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:02:42.588 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.533+0000 7ff57aa20700 5 monclient: authenticate success, global_id 23054 2021-05-25T12:02:42.588 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.533+0000 7ff57aa20700 10 monclient: _renew_subs 2021-05-25T12:02:42.588 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.533+0000 7ff57aa20700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:02:42.588 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.533+0000 7ff57aa20700 1 -- 172.21.15.102:0/1404725600 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff574121a30 con 0x7ff574069130 2021-05-25T12:02:42.589 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.534+0000 7ff55ffff700 1 -- 172.21.15.102:0/1404725600 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff560001af0 con 0x7ff574069130 2021-05-25T12:02:42.589 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.534+0000 7ff55ffff700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:02:42.589 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.534+0000 7ff55ffff700 1 -- 172.21.15.102:0/1404725600 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7ff560001e80 con 0x7ff574069130 2021-05-25T12:02:42.589 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.534+0000 7ff55ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:42.589 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.534+0000 7ff55ffff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:02:42.590 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.534+0000 7ff55ffff700 10 monclient: dump: 2021-05-25T12:02:42.590 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:42.590 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:42.590 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:42.591 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:42.591 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:42.591 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:42.591 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:42.592 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:42.592 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:42.592 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:42.592 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.534+0000 7ff5727fc700 4 set_mon_vals no callback set 2021-05-25T12:02:42.593 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.534+0000 7ff57aa20700 10 client.23054.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:02:42.593 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.534+0000 7ff57aa20700 10 monclient: _renew_subs 2021-05-25T12:02:42.593 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.534+0000 7ff57aa20700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:02:42.593 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.534+0000 7ff57aa20700 1 -- 172.21.15.102:0/1404725600 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff574139500 con 0x7ff574069130 2021-05-25T12:02:42.593 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.534+0000 7ff57aa20700 1 librados: init done 2021-05-25T12:02:42.594 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.534+0000 7ff55ffff700 1 -- 172.21.15.102:0/1404725600 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7ff560002060 con 0x7ff574069130 2021-05-25T12:02:42.594 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.534+0000 7ff55ffff700 1 -- 172.21.15.102:0/1404725600 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(80..80 src has 1..80) v4 ==== 7087+0+0 (unknown 3831610251 0 0) 0x7ff56003b4a0 con 0x7ff574069130 2021-05-25T12:02:42.594 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.534+0000 7ff55ffff700 10 client.23054.objecter ms_dispatch 0x7ff574001550 osd_map(80..80 src has 1..80) v4 2021-05-25T12:02:42.594 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.534+0000 7ff55ffff700 3 client.23054.objecter handle_osd_map got epochs [80,80] > 0 2021-05-25T12:02:42.595 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.534+0000 7ff55ffff700 3 client.23054.objecter handle_osd_map decoding full epoch 80 2021-05-25T12:02:42.595 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.534+0000 7ff55ffff700 20 client.23054.objecter dump_active .. 0 homeless 2021-05-25T12:02:42.595 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.534+0000 7ff573fff700 10 monclient: get_auth_request con 0x7ff554033540 auth_method 0 2021-05-25T12:02:42.595 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.535+0000 7ff57aa20700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:02:42.595 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.535+0000 7ff57aa20700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:02:42.596 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.535+0000 7ff57aa20700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:02:42.596 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.535+0000 7ff57aa20700 1 -- 172.21.15.102:0/1404725600 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff574059730 con 0x7ff574069130 2021-05-25T12:02:42.596 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.535+0000 7ff55ffff700 10 client.23054.objecter ms_handle_connect 0x7ff554033540 2021-05-25T12:02:42.596 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.539+0000 7ff55ffff700 1 -- 172.21.15.102:0/1404725600 <== mon.2 v1:172.21.15.102:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7ff560010670 con 0x7ff574069130 2021-05-25T12:02:42.597 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.539+0000 7ff55ffff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:02:42.597 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.539+0000 7ff55ffff700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:02:42.748 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.747+0000 7ff57aa20700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2021-05-25T12:02:42.749 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.747+0000 7ff57aa20700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2021-05-25T12:02:42.749 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.747+0000 7ff57aa20700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:02:42.749 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.747+0000 7ff57aa20700 1 -- 172.21.15.102:0/1404725600 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7ff574097d40 con 0x7ff574069130 2021-05-25T12:02:42.749 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.747+0000 7ff55ffff700 1 -- 172.21.15.102:0/1404725600 <== mon.2 v1:172.21.15.102:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1342 (unknown 254746657 0 800495581) 0x7ff560010260 con 0x7ff574069130 2021-05-25T12:02:42.750 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.747+0000 7ff55ffff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2021-05-25T12:02:42.750 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.747+0000 7ff55ffff700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:02:42.751 INFO:teuthology.orchestra.run.smithi102.stdout:{"election_epoch":30,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":1,"features":{"quorum_con":"4540138303579357183","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"20ee91b6-8935-4b98-b868-85c03789ccc4","modified":"2021-05-25T11:59:11.011522Z","created":"2021-05-25T11:59:11.011522Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6790","nonce":0}]},"addr":"172.21.15.102:6790/0","public_addr":"172.21.15.102:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2021-05-25T12:02:42.752 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.748+0000 7ff57aa20700 10 librados: watch_flush enter 2021-05-25T12:02:42.752 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.749+0000 7ff57aa20700 10 librados: watch_flush exit 2021-05-25T12:02:42.753 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.749+0000 7ff57aa20700 20 client.23054.objecter shutdown clearing up homeless session... 2021-05-25T12:02:42.753 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.749+0000 7ff57aa20700 10 client.23054.objecter successfully canceled tick 2021-05-25T12:02:42.753 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.749+0000 7ff57aa20700 1 -- 172.21.15.102:0/1404725600 >> v1:172.21.15.181:6812/29582 conn(0x7ff554033540 legacy=0x7ff554035a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:42.753 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.749+0000 7ff57aa20700 10 monclient: shutdown 2021-05-25T12:02:42.754 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.749+0000 7ff57aa20700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:02:42.754 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.749+0000 7ff57aa20700 1 -- 172.21.15.102:0/1404725600 >> v1:172.21.15.102:6790/0 conn(0x7ff574069130 legacy=0x7ff574138c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:42.754 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.749+0000 7ff57aa20700 1 -- 172.21.15.102:0/1404725600 shutdown_connections 2021-05-25T12:02:42.754 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.749+0000 7ff57aa20700 1 -- 172.21.15.102:0/1404725600 >> 172.21.15.102:0/1404725600 conn(0x7ff57409b370 msgr2=0x7ff574125570 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:02:42.755 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.749+0000 7ff57aa20700 1 -- 172.21.15.102:0/1404725600 shutdown_connections 2021-05-25T12:02:42.755 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.749+0000 7ff57aa20700 1 -- 172.21.15.102:0/1404725600 wait complete. 2021-05-25T12:02:42.755 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.749+0000 7ff57aa20700 1 librados: shutdown 2021-05-25T12:02:42.761 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2021-05-25T12:02:42.762 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2021-05-25T12:02:42.883 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.882+0000 7f612f868700 10 monclient: get_monmap_and_config 2021-05-25T12:02:42.884 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.882+0000 7f612f868700 10 monclient: build_initial_monmap 2021-05-25T12:02:42.884 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.882+0000 7f612f868700 10 monclient: monmap: 2021-05-25T12:02:42.884 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:02:42.884 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:42.885 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:02:42.883507+0000 2021-05-25T12:02:42.885 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:02:42.883507+0000 2021-05-25T12:02:42.885 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:02:42.885 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:42.885 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:02:42.886 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:02:42.886 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:02:42.886 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:42.898 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.883+0000 7f612f868700 1 Processor -- start 2021-05-25T12:02:42.899 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.883+0000 7f612f868700 1 -- start start 2021-05-25T12:02:42.899 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.883+0000 7f612f868700 10 monclient: init 2021-05-25T12:02:42.899 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.883+0000 7f612f868700 10 monclient: _reopen_session rank -1 2021-05-25T12:02:42.899 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.883+0000 7f612f868700 10 monclient: _add_conns ranks=[1,0,2] 2021-05-25T12:02:42.899 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.883+0000 7f612f868700 10 monclient(hunting): picked mon.noname-c con 0x7f612809a640 addr v1:172.21.15.181:6789/0 2021-05-25T12:02:42.900 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.883+0000 7f612f868700 10 monclient(hunting): picked mon.noname-a con 0x7f6128063bb0 addr v1:172.21.15.102:6789/0 2021-05-25T12:02:42.900 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.883+0000 7f612f868700 10 monclient(hunting): picked mon.noname-b con 0x7f61281325c0 addr v1:172.21.15.102:6790/0 2021-05-25T12:02:42.900 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.883+0000 7f612f868700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6128096b20 con 0x7f6128063bb0 2021-05-25T12:02:42.900 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.883+0000 7f612f868700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f612809a210 con 0x7f612809a640 2021-05-25T12:02:42.901 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.883+0000 7f612f868700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f612809b460 con 0x7f61281325c0 2021-05-25T12:02:42.901 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.883+0000 7f612f868700 10 monclient(hunting): _renew_subs 2021-05-25T12:02:42.901 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.883+0000 7f612f868700 10 monclient(hunting): authenticate will time out at 963.480742s 2021-05-25T12:02:42.901 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.883+0000 7f612de05700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f61281325c0 0x7f61281349b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:55086/0 (socket says 172.21.15.102:55086) 2021-05-25T12:02:42.901 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.883+0000 7f612de05700 1 -- 172.21.15.102:0/38816146 learned_addr learned my addr 172.21.15.102:0/38816146 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:02:42.902 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.884+0000 7f6117fff700 1 -- 172.21.15.102:0/38816146 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 517409250 0 0) 0x7f612809b460 con 0x7f61281325c0 2021-05-25T12:02:42.902 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.884+0000 7f6117fff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:02:42.902 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.884+0000 7f6117fff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:02:42.902 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.884+0000 7f6117fff700 10 monclient(hunting): my global_id is 23066 2021-05-25T12:02:42.903 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.884+0000 7f6117fff700 1 -- 172.21.15.102:0/38816146 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6110003270 con 0x7f61281325c0 2021-05-25T12:02:42.903 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.884+0000 7f6117fff700 1 -- 172.21.15.102:0/38816146 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3735741792 0 0) 0x7f6128096b20 con 0x7f6128063bb0 2021-05-25T12:02:42.903 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.884+0000 7f6117fff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:02:42.903 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.884+0000 7f6117fff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:02:42.903 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.884+0000 7f6117fff700 10 monclient(hunting): my global_id is 174174 2021-05-25T12:02:42.904 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.884+0000 7f6117fff700 1 -- 172.21.15.102:0/38816146 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f612809b460 con 0x7f6128063bb0 2021-05-25T12:02:42.904 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.884+0000 7f6117fff700 1 -- 172.21.15.102:0/38816146 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3868588725 0 0) 0x7f612809a210 con 0x7f612809a640 2021-05-25T12:02:42.904 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.884+0000 7f6117fff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:02:42.904 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.884+0000 7f6117fff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:02:42.904 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.884+0000 7f6117fff700 10 monclient(hunting): my global_id is 95137 2021-05-25T12:02:42.905 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.884+0000 7f6117fff700 1 -- 172.21.15.102:0/38816146 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6128096b20 con 0x7f612809a640 2021-05-25T12:02:42.905 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.884+0000 7f6117fff700 1 -- 172.21.15.102:0/38816146 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 106732216 0 0) 0x7f612809b460 con 0x7f6128063bb0 2021-05-25T12:02:42.905 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.884+0000 7f6117fff700 1 -- 172.21.15.102:0/38816146 >> v1:172.21.15.102:6790/0 conn(0x7f61281325c0 legacy=0x7f61281349b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:42.905 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.884+0000 7f6117fff700 1 -- 172.21.15.102:0/38816146 >> v1:172.21.15.181:6789/0 conn(0x7f612809a640 legacy=0x7f61280634a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:42.906 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.884+0000 7f6117fff700 10 monclient: _finish_hunting 0 2021-05-25T12:02:42.906 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.884+0000 7f6117fff700 1 monclient: found mon.noname-a 2021-05-25T12:02:42.906 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.884+0000 7f6117fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:02:42.906 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.884+0000 7f6117fff700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T12:02:42.906 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.884+0000 7f6117fff700 1 -- 172.21.15.102:0/38816146 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61280964c0 con 0x7f6128063bb0 2021-05-25T12:02:42.907 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.884+0000 7f6117fff700 10 monclient: _finish_auth 0 2021-05-25T12:02:42.907 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.884+0000 7f6117fff700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:02:42.907 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.884+0000 7f6117fff700 1 -- 172.21.15.102:0/38816146 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f611c0015f0 con 0x7f6128063bb0 2021-05-25T12:02:42.907 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.884+0000 7f612f868700 5 monclient: authenticate success, global_id 174174 2021-05-25T12:02:42.907 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.884+0000 7f612f868700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:02:42.908 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.884+0000 7f6117fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:42.908 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.884+0000 7f6117fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:02:42.908 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.884+0000 7f6117fff700 10 monclient: dump: 2021-05-25T12:02:42.908 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:42.908 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:42.909 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:42.909 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:42.909 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:42.909 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:42.910 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:42.910 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:42.910 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:42.910 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:42.910 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.884+0000 7f612f868700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:02:42.911 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.884+0000 7f6117fff700 1 -- 172.21.15.102:0/38816146 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f611c001890 con 0x7f6128063bb0 2021-05-25T12:02:42.911 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.884+0000 7f6117fff700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:02:42.911 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.884+0000 7f6117fff700 1 -- 172.21.15.102:0/38816146 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f611c001c20 con 0x7f6128063bb0 2021-05-25T12:02:42.911 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.884+0000 7f612f868700 10 monclient: get_monmap_and_config success 2021-05-25T12:02:42.911 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.884+0000 7f612f868700 4 set_mon_vals no callback set 2021-05-25T12:02:42.912 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.884+0000 7f6117fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:42.912 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.884+0000 7f6117fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:02:42.912 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.884+0000 7f6117fff700 10 monclient: dump: 2021-05-25T12:02:42.912 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:42.913 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:42.913 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:42.913 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:42.913 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:42.913 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:42.914 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:42.914 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:42.914 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:42.914 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:42.914 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.885+0000 7f612f868700 10 monclient: shutdown 2021-05-25T12:02:42.915 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.885+0000 7f612f868700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:02:42.915 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.885+0000 7f612f868700 1 -- 172.21.15.102:0/38816146 >> v1:172.21.15.102:6789/0 conn(0x7f6128063bb0 legacy=0x7f6128064030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:42.915 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.885+0000 7f612f868700 1 -- 172.21.15.102:0/38816146 shutdown_connections 2021-05-25T12:02:42.915 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.885+0000 7f612f868700 1 -- 172.21.15.102:0/38816146 >> 172.21.15.102:0/38816146 conn(0x7f612809ef50 msgr2=0x7f612809f360 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:02:42.916 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.885+0000 7f612f868700 1 -- 172.21.15.102:0/38816146 shutdown_connections 2021-05-25T12:02:42.916 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.885+0000 7f612f868700 1 -- 172.21.15.102:0/38816146 wait complete. 2021-05-25T12:02:42.916 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.885+0000 7f612f868700 10 monclient: build_initial_monmap 2021-05-25T12:02:42.916 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.885+0000 7f612f868700 10 monclient: monmap: 2021-05-25T12:02:42.916 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:02:42.917 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:02:42.917 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:02:42.917 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:02:42.917 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:02:42.917 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:02:42.918 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:02:42.918 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:02:42.918 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:02:42.918 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:42.919 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.886+0000 7f612f868700 1 librados: starting msgr at 2021-05-25T12:02:42.919 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.886+0000 7f612f868700 1 librados: starting objecter 2021-05-25T12:02:42.919 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.886+0000 7f612f868700 1 Processor -- start 2021-05-25T12:02:42.919 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.886+0000 7f612f868700 1 -- start start 2021-05-25T12:02:42.919 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.886+0000 7f612f868700 1 librados: setting wanted keys 2021-05-25T12:02:42.920 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.886+0000 7f612f868700 1 librados: calling monclient init 2021-05-25T12:02:42.920 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.886+0000 7f612f868700 10 monclient: init 2021-05-25T12:02:42.920 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.886+0000 7f612f868700 10 monclient: _reopen_session rank -1 2021-05-25T12:02:42.920 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.886+0000 7f612f868700 10 monclient: _add_conns ranks=[1,2,0] 2021-05-25T12:02:42.920 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.886+0000 7f612f868700 10 monclient(hunting): picked mon.noname-b con 0x7f6128063bb0 addr v1:172.21.15.181:6789/0 2021-05-25T12:02:42.921 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.886+0000 7f612f868700 10 monclient(hunting): picked mon.noname-c con 0x7f612809a640 addr v1:172.21.15.102:6790/0 2021-05-25T12:02:42.921 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.886+0000 7f612f868700 10 monclient(hunting): picked mon.noname-a con 0x7f61281325c0 addr v1:172.21.15.102:6789/0 2021-05-25T12:02:42.921 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.886+0000 7f612f868700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f612811c330 con 0x7f61281325c0 2021-05-25T12:02:42.921 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.886+0000 7f612f868700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f612811c5d0 con 0x7f6128063bb0 2021-05-25T12:02:42.921 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.886+0000 7f612f868700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f612811c870 con 0x7f612809a640 2021-05-25T12:02:42.922 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.886+0000 7f612f868700 10 monclient(hunting): _renew_subs 2021-05-25T12:02:42.922 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.886+0000 7f612f868700 10 monclient(hunting): authenticate will time out at 963.483770s 2021-05-25T12:02:42.922 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.886+0000 7f612de05700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f61281325c0 0x7f6128143a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60216/0 (socket says 172.21.15.102:60216) 2021-05-25T12:02:42.922 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.886+0000 7f612de05700 1 -- 172.21.15.102:0/1464219347 learned_addr learned my addr 172.21.15.102:0/1464219347 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:02:42.923 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.886+0000 7f61157fa700 10 client.?.objecter ms_handle_connect 0x7f612809a640 2021-05-25T12:02:42.923 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.886+0000 7f61157fa700 10 client.?.objecter resend_mon_ops 2021-05-25T12:02:42.923 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.886+0000 7f61157fa700 10 client.?.objecter ms_handle_connect 0x7f61281325c0 2021-05-25T12:02:42.923 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.886+0000 7f61157fa700 10 client.?.objecter resend_mon_ops 2021-05-25T12:02:42.923 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.887+0000 7f61157fa700 1 -- 172.21.15.102:0/1464219347 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1124383675 0 0) 0x7f612811c870 con 0x7f612809a640 2021-05-25T12:02:42.924 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.887+0000 7f61157fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:02:42.924 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.887+0000 7f61157fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:02:42.924 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.887+0000 7f61157fa700 10 monclient(hunting): my global_id is 23069 2021-05-25T12:02:42.924 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.887+0000 7f61157fa700 1 -- 172.21.15.102:0/1464219347 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6108004c10 con 0x7f612809a640 2021-05-25T12:02:42.924 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.887+0000 7f61157fa700 1 -- 172.21.15.102:0/1464219347 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 247620953 0 0) 0x7f612811c330 con 0x7f61281325c0 2021-05-25T12:02:42.925 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.887+0000 7f61157fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:02:42.925 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.887+0000 7f61157fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:02:42.925 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.887+0000 7f61157fa700 10 monclient(hunting): my global_id is 174177 2021-05-25T12:02:42.925 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.887+0000 7f61157fa700 1 -- 172.21.15.102:0/1464219347 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f612811c870 con 0x7f61281325c0 2021-05-25T12:02:42.925 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.887+0000 7f61157fa700 10 client.?.objecter ms_handle_connect 0x7f6128063bb0 2021-05-25T12:02:42.926 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.887+0000 7f61157fa700 10 client.?.objecter resend_mon_ops 2021-05-25T12:02:42.926 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.887+0000 7f61157fa700 1 -- 172.21.15.102:0/1464219347 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3607423426 0 0) 0x7f612811c5d0 con 0x7f6128063bb0 2021-05-25T12:02:42.926 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.887+0000 7f61157fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:02:42.926 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.887+0000 7f61157fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:02:42.927 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.887+0000 7f61157fa700 10 monclient(hunting): my global_id is 95140 2021-05-25T12:02:42.927 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.887+0000 7f61157fa700 1 -- 172.21.15.102:0/1464219347 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f612811c330 con 0x7f6128063bb0 2021-05-25T12:02:42.927 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.887+0000 7f61157fa700 1 -- 172.21.15.102:0/1464219347 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3925530331 0 0) 0x7f6108004c10 con 0x7f612809a640 2021-05-25T12:02:42.927 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.887+0000 7f61157fa700 1 -- 172.21.15.102:0/1464219347 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f612811c5d0 con 0x7f612809a640 2021-05-25T12:02:42.927 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.887+0000 7f61157fa700 1 -- 172.21.15.102:0/1464219347 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2040215919 0 0) 0x7f612811c870 con 0x7f61281325c0 2021-05-25T12:02:42.928 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.887+0000 7f61157fa700 1 -- 172.21.15.102:0/1464219347 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6108004c10 con 0x7f61281325c0 2021-05-25T12:02:42.928 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.887+0000 7f61157fa700 1 -- 172.21.15.102:0/1464219347 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f611c001c60 con 0x7f612809a640 2021-05-25T12:02:42.928 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.887+0000 7f61157fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:42.928 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.887+0000 7f61157fa700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:02:42.928 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.887+0000 7f61157fa700 10 monclient(hunting): dump: 2021-05-25T12:02:42.929 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:42.929 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:42.929 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:42.929 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:42.929 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:42.930 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:42.930 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:42.930 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:42.930 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:42.930 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:42.931 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.887+0000 7f61157fa700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:02:42.931 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.887+0000 7f61157fa700 1 -- 172.21.15.102:0/1464219347 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f6124001540 con 0x7f61281325c0 2021-05-25T12:02:42.931 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.887+0000 7f61157fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:42.931 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.887+0000 7f61157fa700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2021-05-25T12:02:42.932 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.887+0000 7f61157fa700 10 monclient(hunting): dump: 2021-05-25T12:02:42.932 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:42.932 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:42.932 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:42.932 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:42.933 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:42.933 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:42.933 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:42.933 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:42.933 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:42.934 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:42.934 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.887+0000 7f61157fa700 1 -- 172.21.15.102:0/1464219347 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3255223209 0 0) 0x7f612811c330 con 0x7f6128063bb0 2021-05-25T12:02:42.934 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.887+0000 7f61157fa700 1 -- 172.21.15.102:0/1464219347 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f612811c870 con 0x7f6128063bb0 2021-05-25T12:02:42.934 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.887+0000 7f61157fa700 1 -- 172.21.15.102:0/1464219347 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2823102205 0 0) 0x7f612811c5d0 con 0x7f612809a640 2021-05-25T12:02:42.935 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.887+0000 7f61157fa700 1 -- 172.21.15.102:0/1464219347 >> v1:172.21.15.181:6789/0 conn(0x7f6128063bb0 legacy=0x7f6128118bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:42.935 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.887+0000 7f61157fa700 1 -- 172.21.15.102:0/1464219347 >> v1:172.21.15.102:6789/0 conn(0x7f61281325c0 legacy=0x7f6128143a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:42.935 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.887+0000 7f61157fa700 10 monclient: _finish_hunting 0 2021-05-25T12:02:42.935 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.887+0000 7f61157fa700 1 monclient: found mon.c 2021-05-25T12:02:42.935 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.887+0000 7f61157fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:02:42.936 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.887+0000 7f61157fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:02:42.936 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.887+0000 7f61157fa700 1 -- 172.21.15.102:0/1464219347 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f612811a700 con 0x7f612809a640 2021-05-25T12:02:42.936 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.887+0000 7f61157fa700 10 monclient: _finish_auth 0 2021-05-25T12:02:42.936 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.887+0000 7f61157fa700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:02:42.936 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.887+0000 7f612f868700 5 monclient: authenticate success, global_id 23069 2021-05-25T12:02:42.937 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.887+0000 7f612f868700 10 monclient: _renew_subs 2021-05-25T12:02:42.937 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.887+0000 7f612f868700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:02:42.937 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.887+0000 7f612f868700 1 -- 172.21.15.102:0/1464219347 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6128132240 con 0x7f612809a640 2021-05-25T12:02:42.937 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.887+0000 7f612f868700 10 client.23069.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:02:42.938 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.887+0000 7f612f868700 10 monclient: _renew_subs 2021-05-25T12:02:42.938 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.887+0000 7f612f868700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:02:42.938 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.887+0000 7f612f868700 1 -- 172.21.15.102:0/1464219347 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f612811cdf0 con 0x7f612809a640 2021-05-25T12:02:42.938 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.887+0000 7f612f868700 1 librados: init done 2021-05-25T12:02:42.938 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.887+0000 7f61157fa700 1 -- 172.21.15.102:0/1464219347 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f611c0021e0 con 0x7f612809a640 2021-05-25T12:02:42.939 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.887+0000 7f61157fa700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:02:42.939 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.887+0000 7f61157fa700 1 -- 172.21.15.102:0/1464219347 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f611c002570 con 0x7f612809a640 2021-05-25T12:02:42.939 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.887+0000 7f61157fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:42.939 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.887+0000 7f61177fe700 4 set_mon_vals no callback set 2021-05-25T12:02:42.939 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.887+0000 7f61157fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:02:42.940 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.887+0000 7f61157fa700 10 monclient: dump: 2021-05-25T12:02:42.940 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:42.940 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:42.940 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:42.940 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:42.941 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:42.941 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:42.941 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:42.941 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:42.942 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:42.942 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:42.942 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.888+0000 7f612f868700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:02:42.942 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.888+0000 7f61157fa700 1 -- 172.21.15.102:0/1464219347 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f611c002750 con 0x7f612809a640 2021-05-25T12:02:42.942 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.888+0000 7f612f868700 1 -- 172.21.15.102:0/1464219347 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f61000065a0 con 0x7f6100002840 2021-05-25T12:02:42.943 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.888+0000 7f61157fa700 1 -- 172.21.15.102:0/1464219347 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(81..81 src has 1..81) v4 ==== 6162+0+0 (unknown 275132006 0 0) 0x7f611c03b380 con 0x7f612809a640 2021-05-25T12:02:42.943 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.888+0000 7f61157fa700 10 client.23069.objecter ms_dispatch 0x7f6128001550 osd_map(81..81 src has 1..81) v4 2021-05-25T12:02:42.943 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.888+0000 7f61157fa700 3 client.23069.objecter handle_osd_map got epochs [81,81] > 0 2021-05-25T12:02:42.943 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.888+0000 7f61157fa700 3 client.23069.objecter handle_osd_map decoding full epoch 81 2021-05-25T12:02:42.943 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.888+0000 7f61157fa700 20 client.23069.objecter dump_active .. 0 homeless 2021-05-25T12:02:42.944 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.888+0000 7f61157fa700 10 client.23069.objecter ms_handle_connect 0x7f6100002840 2021-05-25T12:02:42.944 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.888+0000 7f61157fa700 10 client.23069.objecter resend_mon_ops 2021-05-25T12:02:42.944 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.888+0000 7f61157fa700 1 -- 172.21.15.102:0/1464219347 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3601476488 0 0) 0x7f61000065a0 con 0x7f6100002840 2021-05-25T12:02:42.944 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.888+0000 7f61157fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:02:42.945 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.888+0000 7f61157fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:02:42.945 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.888+0000 7f61157fa700 10 monclient(hunting): my global_id is 174180 2021-05-25T12:02:42.945 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.888+0000 7f61157fa700 1 -- 172.21.15.102:0/1464219347 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f612811c5d0 con 0x7f6100002840 2021-05-25T12:02:42.945 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.888+0000 7f612de05700 10 monclient: get_auth_request con 0x7f61080334d0 auth_method 0 2021-05-25T12:02:42.945 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.888+0000 7f61157fa700 1 -- 172.21.15.102:0/1464219347 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2233265158 0 0) 0x7f612811c5d0 con 0x7f6100002840 2021-05-25T12:02:42.946 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.888+0000 7f61157fa700 1 -- 172.21.15.102:0/1464219347 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f61000065a0 con 0x7f6100002840 2021-05-25T12:02:42.946 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.889+0000 7f61157fa700 1 -- 172.21.15.102:0/1464219347 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f6118001f00 con 0x7f6100002840 2021-05-25T12:02:42.946 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.889+0000 7f61157fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:42.946 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.889+0000 7f61157fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:02:42.947 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.889+0000 7f61157fa700 10 monclient: dump: 2021-05-25T12:02:42.947 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:42.947 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:42.947 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:42.947 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:42.948 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:42.948 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:42.948 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:42.948 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:42.948 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:42.949 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:42.949 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.889+0000 7f61157fa700 1 -- 172.21.15.102:0/1464219347 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1165988858 0 0) 0x7f61000065a0 con 0x7f6100002840 2021-05-25T12:02:42.949 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.889+0000 7f61157fa700 1 -- 172.21.15.102:0/1464219347 --> v1:172.21.15.102:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6100009e40 con 0x7f6100002840 2021-05-25T12:02:42.949 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.889+0000 7f61157fa700 10 client.23069.objecter ms_handle_connect 0x7f61080334d0 2021-05-25T12:02:42.949 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.889+0000 7f61157fa700 1 -- 172.21.15.102:0/1464219347 <== mon.0 v1:172.21.15.102:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7f6118004230 con 0x7f6100002840 2021-05-25T12:02:42.950 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.889+0000 7f61157fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:02:42.950 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.889+0000 7f61157fa700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:02:42.950 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.889+0000 7f61157fa700 1 -- 172.21.15.102:0/1464219347 >> v1:172.21.15.102:6789/0 conn(0x7f6100002840 legacy=0x7f6100004ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:42.950 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.889+0000 7f61157fa700 1 -- 172.21.15.102:0/1464219347 >> v1:172.21.15.102:6789/0 conn(0x7f6100002840 legacy=0x7f6100004ce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:02:42.950 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.895+0000 7f612f868700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:02:42.951 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.895+0000 7f612f868700 1 -- 172.21.15.102:0/1464219347 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f6100009ad0 con 0x7f6100008d90 2021-05-25T12:02:42.951 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.896+0000 7f61157fa700 10 client.23069.objecter ms_handle_connect 0x7f6100008d90 2021-05-25T12:02:42.951 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.896+0000 7f61157fa700 10 client.23069.objecter resend_mon_ops 2021-05-25T12:02:42.951 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.896+0000 7f61157fa700 1 -- 172.21.15.102:0/1464219347 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2438273404 0 0) 0x7f6100009ad0 con 0x7f6100008d90 2021-05-25T12:02:42.952 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.896+0000 7f61157fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:02:42.952 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.896+0000 7f61157fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:02:42.952 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.896+0000 7f61157fa700 10 monclient(hunting): my global_id is 174183 2021-05-25T12:02:42.952 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.896+0000 7f61157fa700 1 -- 172.21.15.102:0/1464219347 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61000065a0 con 0x7f6100008d90 2021-05-25T12:02:42.952 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.896+0000 7f61157fa700 1 -- 172.21.15.102:0/1464219347 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 323415353 0 0) 0x7f61000065a0 con 0x7f6100008d90 2021-05-25T12:02:42.953 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.896+0000 7f61157fa700 1 -- 172.21.15.102:0/1464219347 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6100009ad0 con 0x7f6100008d90 2021-05-25T12:02:42.953 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.896+0000 7f61157fa700 1 -- 172.21.15.102:0/1464219347 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f6118002670 con 0x7f6100008d90 2021-05-25T12:02:42.953 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.896+0000 7f61157fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:42.953 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.896+0000 7f61157fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:02:42.953 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.896+0000 7f61157fa700 10 monclient: dump: 2021-05-25T12:02:42.954 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:42.954 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:42.954 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:42.954 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:42.955 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:42.955 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:42.955 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:42.955 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:42.955 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:42.956 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:42.956 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.896+0000 7f61157fa700 1 -- 172.21.15.102:0/1464219347 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1051953828 0 0) 0x7f6100009ad0 con 0x7f6100008d90 2021-05-25T12:02:42.956 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.896+0000 7f61157fa700 1 -- 172.21.15.102:0/1464219347 --> v1:172.21.15.102:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f6100008740 con 0x7f6100008d90 2021-05-25T12:02:42.956 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.896+0000 7f61157fa700 1 -- 172.21.15.102:0/1464219347 <== mon.0 v1:172.21.15.102:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3940 (unknown 0 0 3617818820) 0x7f6118003ab0 con 0x7f6100008d90 2021-05-25T12:02:42.956 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.896+0000 7f61157fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:02:42.957 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.896+0000 7f61157fa700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:02:42.957 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.896+0000 7f61157fa700 1 -- 172.21.15.102:0/1464219347 >> v1:172.21.15.102:6789/0 conn(0x7f6100008d90 legacy=0x7f6100009220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:42.957 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.896+0000 7f61157fa700 1 -- 172.21.15.102:0/1464219347 >> v1:172.21.15.102:6789/0 conn(0x7f6100008d90 legacy=0x7f6100009220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:02:42.957 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:02:42.958 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:02:42.958 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:02:42.958 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "peon", 2021-05-25T12:02:42.958 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 30, 2021-05-25T12:02:42.959 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:02:42.959 INFO:teuthology.orchestra.run.smithi102.stdout: 0, 2021-05-25T12:02:42.959 INFO:teuthology.orchestra.run.smithi102.stdout: 1, 2021-05-25T12:02:42.959 INFO:teuthology.orchestra.run.smithi102.stdout: 2 2021-05-25T12:02:42.960 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:02:42.960 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 1, 2021-05-25T12:02:42.960 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:02:42.960 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:02:42.960 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:02:42.961 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:02:42.961 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:02:42.961 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:02:42.961 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:02:42.961 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:02:42.962 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:02:42.962 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:02:42.962 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:02:42.962 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:02:42.962 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:02:42.963 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:02:42.963 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:02:42.963 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:02:42.963 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:02:42.964 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:02:42.964 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:02:42.964 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:02:42.964 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:02:42.965 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:02:42.965 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:02:42.965 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:02:42.965 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:02:42.965 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:02:42.966 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:02:42.966 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:02:42.966 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:02:42.966 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:02:42.966 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:02:42.967 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:02:42.967 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:02:42.967 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:02:42.967 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:02:42.968 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:02:42.968 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:02:42.968 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:02:42.968 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:02:42.968 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:02:42.969 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:02:42.969 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:02:42.969 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:02:42.969 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:02:42.969 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:02:42.970 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:02:42.970 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:02:42.970 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:02:42.970 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:02:42.971 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:02:42.971 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:02:42.971 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:02:42.971 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:02:42.971 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:02:42.972 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:02:42.972 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:02:42.972 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:02:42.972 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:02:42.972 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:02:42.973 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:02:42.973 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:02:42.973 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:02:42.973 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:02:42.974 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:02:42.974 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:02:42.974 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:02:42.974 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:02:42.974 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:02:42.975 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:02:42.975 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:02:42.975 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:02:42.975 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:02:42.975 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:02:42.976 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:02:42.976 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:02:42.976 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:02:42.976 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:02:42.976 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:02:42.977 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:02:42.977 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:02:42.977 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:02:42.977 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:02:42.978 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:02:42.978 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:02:42.978 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:02:42.978 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:02:42.978 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:02:42.979 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:02:42.979 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:02:42.979 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:02:42.979 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:02:42.979 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:02:42.980 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:02:42.980 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:02:42.980 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:02:42.980 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:02:42.981 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:02:42.981 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:02:42.981 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:02:42.981 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:02:42.981 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:02:42.982 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:02:42.982 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:02:42.982 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:02:42.982 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:02:42.982 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:02:42.983 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:02:42.983 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:02:42.983 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:02:42.983 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:02:42.984 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:02:42.984 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:02:42.984 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:02:42.984 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:02:42.984 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:02:42.985 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:02:42.985 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:02:42.985 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:02:42.985 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": [ 2021-05-25T12:02:42.985 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:02:42.986 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:02:42.986 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:02:42.986 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 3 2021-05-25T12:02:42.986 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:02:42.986 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:02:42.987 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:02:42.987 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:02:42.987 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:02:42.987 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:02:42.987 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 2 2021-05-25T12:02:42.988 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:02:42.988 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:02:42.988 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:02:42.988 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:02:42.988 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:02:42.991 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.897+0000 7f612f868700 10 librados: watch_flush enter 2021-05-25T12:02:42.991 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.897+0000 7f612f868700 10 librados: watch_flush exit 2021-05-25T12:02:42.991 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.897+0000 7f612f868700 20 client.23069.objecter shutdown clearing up homeless session... 2021-05-25T12:02:42.991 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.897+0000 7f612f868700 10 client.23069.objecter successfully canceled tick 2021-05-25T12:02:42.992 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.897+0000 7f612f868700 1 -- 172.21.15.102:0/1464219347 >> v1:172.21.15.181:6812/29582 conn(0x7f61080334d0 legacy=0x7f6108035990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:42.992 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.897+0000 7f612f868700 10 monclient: shutdown 2021-05-25T12:02:42.992 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.897+0000 7f612f868700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:02:42.992 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.897+0000 7f612f868700 1 -- 172.21.15.102:0/1464219347 >> v1:172.21.15.102:6790/0 conn(0x7f612809a640 legacy=0x7f6128119300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:42.992 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.897+0000 7f612de05700 1 -- 172.21.15.102:0/1464219347 reap_dead start 2021-05-25T12:02:42.993 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.897+0000 7f612de05700 1 -- 172.21.15.102:0/1464219347 reap_dead start 2021-05-25T12:02:42.993 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.897+0000 7f612f868700 1 -- 172.21.15.102:0/1464219347 shutdown_connections 2021-05-25T12:02:42.993 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.897+0000 7f612f868700 1 -- 172.21.15.102:0/1464219347 >> 172.21.15.102:0/1464219347 conn(0x7f612809ef50 msgr2=0x7f61280945d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:02:42.993 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.897+0000 7f612f868700 1 -- 172.21.15.102:0/1464219347 shutdown_connections 2021-05-25T12:02:42.994 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.897+0000 7f612f868700 1 -- 172.21.15.102:0/1464219347 wait complete. 2021-05-25T12:02:42.994 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:42.897+0000 7f612f868700 1 librados: shutdown 2021-05-25T12:02:42.994 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2021-05-25T12:02:43.104 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.103+0000 7f6c73fff700 10 monclient: get_monmap_and_config 2021-05-25T12:02:43.105 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.103+0000 7f6c73fff700 10 monclient: build_initial_monmap 2021-05-25T12:02:43.105 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.103+0000 7f6c73fff700 10 monclient: monmap: 2021-05-25T12:02:43.105 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:02:43.105 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:43.106 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:02:43.104552+0000 2021-05-25T12:02:43.106 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:02:43.104552+0000 2021-05-25T12:02:43.106 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:02:43.106 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:43.106 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:02:43.107 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:02:43.107 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:02:43.107 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:43.120 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.104+0000 7f6c73fff700 1 Processor -- start 2021-05-25T12:02:43.120 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.104+0000 7f6c73fff700 1 -- start start 2021-05-25T12:02:43.120 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.104+0000 7f6c73fff700 10 monclient: init 2021-05-25T12:02:43.120 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.104+0000 7f6c73fff700 10 monclient: _reopen_session rank -1 2021-05-25T12:02:43.121 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.104+0000 7f6c73fff700 10 monclient: _add_conns ranks=[0,1,2] 2021-05-25T12:02:43.121 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.104+0000 7f6c73fff700 10 monclient(hunting): picked mon.noname-a con 0x7f6c74093130 addr v1:172.21.15.102:6789/0 2021-05-25T12:02:43.121 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.104+0000 7f6c73fff700 10 monclient(hunting): picked mon.noname-c con 0x7f6c7412a430 addr v1:172.21.15.181:6789/0 2021-05-25T12:02:43.121 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.104+0000 7f6c73fff700 10 monclient(hunting): picked mon.noname-b con 0x7f6c74068e70 addr v1:172.21.15.102:6790/0 2021-05-25T12:02:43.122 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.104+0000 7f6c73fff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c740945b0 con 0x7f6c74093130 2021-05-25T12:02:43.122 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.104+0000 7f6c73fff700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c74094850 con 0x7f6c7412a430 2021-05-25T12:02:43.122 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.104+0000 7f6c73fff700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c74093db0 con 0x7f6c74068e70 2021-05-25T12:02:43.122 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.104+0000 7f6c73fff700 10 monclient(hunting): _renew_subs 2021-05-25T12:02:43.122 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.104+0000 7f6c73fff700 10 monclient(hunting): authenticate will time out at 963.701689s 2021-05-25T12:02:43.123 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.104+0000 7f6c72ffd700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f6c74093130 0x7f6c74129d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60224/0 (socket says 172.21.15.102:60224) 2021-05-25T12:02:43.123 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.104+0000 7f6c72ffd700 1 -- 172.21.15.102:0/2409802533 learned_addr learned my addr 172.21.15.102:0/2409802533 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:02:43.123 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.104+0000 7f6c737fe700 1 --1- 172.21.15.102:0/2409802533 >> v1:172.21.15.102:6790/0 conn(0x7f6c74068e70 0x7f6c741322d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:55106/0 (socket says 172.21.15.102:55106) 2021-05-25T12:02:43.123 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.105+0000 7f6c71ffb700 1 -- 172.21.15.102:0/2409802533 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1538555931 0 0) 0x7f6c74093db0 con 0x7f6c74068e70 2021-05-25T12:02:43.124 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.105+0000 7f6c71ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:02:43.124 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.105+0000 7f6c71ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:02:43.124 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.105+0000 7f6c71ffb700 10 monclient(hunting): my global_id is 23072 2021-05-25T12:02:43.124 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.105+0000 7f6c71ffb700 1 -- 172.21.15.102:0/2409802533 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c60003270 con 0x7f6c74068e70 2021-05-25T12:02:43.124 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.105+0000 7f6c71ffb700 1 -- 172.21.15.102:0/2409802533 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3389652840 0 0) 0x7f6c740945b0 con 0x7f6c74093130 2021-05-25T12:02:43.125 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.105+0000 7f6c71ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:02:43.125 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.105+0000 7f6c71ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:02:43.125 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.105+0000 7f6c71ffb700 10 monclient(hunting): my global_id is 174186 2021-05-25T12:02:43.125 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.105+0000 7f6c71ffb700 1 -- 172.21.15.102:0/2409802533 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c74093db0 con 0x7f6c74093130 2021-05-25T12:02:43.125 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.105+0000 7f6c71ffb700 1 -- 172.21.15.102:0/2409802533 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3398908524 0 0) 0x7f6c60003270 con 0x7f6c74068e70 2021-05-25T12:02:43.126 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.105+0000 7f6c71ffb700 1 -- 172.21.15.102:0/2409802533 >> v1:172.21.15.181:6789/0 conn(0x7f6c7412a430 legacy=0x7f6c7412eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:43.126 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.105+0000 7f6c71ffb700 1 -- 172.21.15.102:0/2409802533 >> v1:172.21.15.102:6789/0 conn(0x7f6c74093130 legacy=0x7f6c74129d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:43.126 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.105+0000 7f6c71ffb700 10 monclient: _finish_hunting 0 2021-05-25T12:02:43.126 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.105+0000 7f6c71ffb700 1 monclient: found mon.noname-b 2021-05-25T12:02:43.127 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.105+0000 7f6c71ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:02:43.127 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.105+0000 7f6c71ffb700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T12:02:43.127 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.105+0000 7f6c71ffb700 1 -- 172.21.15.102:0/2409802533 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c74069300 con 0x7f6c74068e70 2021-05-25T12:02:43.127 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.105+0000 7f6c71ffb700 10 monclient: _finish_auth 0 2021-05-25T12:02:43.127 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.105+0000 7f6c71ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:02:43.128 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.105+0000 7f6c71ffb700 1 -- 172.21.15.102:0/2409802533 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f6c640019e0 con 0x7f6c74068e70 2021-05-25T12:02:43.128 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.105+0000 7f6c73fff700 5 monclient: authenticate success, global_id 23072 2021-05-25T12:02:43.128 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.105+0000 7f6c73fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:02:43.128 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.105+0000 7f6c71ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:43.129 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.105+0000 7f6c71ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:02:43.129 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.105+0000 7f6c71ffb700 10 monclient: dump: 2021-05-25T12:02:43.129 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:43.129 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:43.129 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:43.129 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:43.130 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:43.130 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:43.130 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:43.130 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:43.131 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:43.131 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:43.131 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.105+0000 7f6c73fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:02:43.131 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.105+0000 7f6c71ffb700 1 -- 172.21.15.102:0/2409802533 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6c64001c10 con 0x7f6c74068e70 2021-05-25T12:02:43.131 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.105+0000 7f6c71ffb700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:02:43.132 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.105+0000 7f6c71ffb700 1 -- 172.21.15.102:0/2409802533 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f6c64001fa0 con 0x7f6c74068e70 2021-05-25T12:02:43.132 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.105+0000 7f6c73fff700 10 monclient: get_monmap_and_config success 2021-05-25T12:02:43.132 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.105+0000 7f6c73fff700 4 set_mon_vals no callback set 2021-05-25T12:02:43.132 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.105+0000 7f6c71ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:43.132 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.105+0000 7f6c71ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:02:43.133 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.105+0000 7f6c71ffb700 10 monclient: dump: 2021-05-25T12:02:43.133 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:43.133 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:43.133 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:43.134 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:43.134 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:43.134 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:43.134 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:43.134 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:43.135 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:43.135 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:43.135 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.105+0000 7f6c73fff700 10 monclient: shutdown 2021-05-25T12:02:43.135 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.105+0000 7f6c73fff700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:02:43.135 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.105+0000 7f6c73fff700 1 -- 172.21.15.102:0/2409802533 >> v1:172.21.15.102:6790/0 conn(0x7f6c74068e70 legacy=0x7f6c741322d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:43.136 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.105+0000 7f6c73fff700 1 -- 172.21.15.102:0/2409802533 shutdown_connections 2021-05-25T12:02:43.136 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.105+0000 7f6c73fff700 1 -- 172.21.15.102:0/2409802533 >> 172.21.15.102:0/2409802533 conn(0x7f6c740974b0 msgr2=0x7f6c740978c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:02:43.136 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.105+0000 7f6c73fff700 1 -- 172.21.15.102:0/2409802533 shutdown_connections 2021-05-25T12:02:43.136 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.105+0000 7f6c73fff700 1 -- 172.21.15.102:0/2409802533 wait complete. 2021-05-25T12:02:43.137 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.106+0000 7f6c73fff700 10 monclient: build_initial_monmap 2021-05-25T12:02:43.137 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.106+0000 7f6c73fff700 10 monclient: monmap: 2021-05-25T12:02:43.137 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:02:43.137 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:02:43.137 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:02:43.138 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:02:43.138 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:02:43.138 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:02:43.138 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:02:43.139 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:02:43.139 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:02:43.139 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:43.139 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.106+0000 7f6c73fff700 1 librados: starting msgr at 2021-05-25T12:02:43.139 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.106+0000 7f6c73fff700 1 librados: starting objecter 2021-05-25T12:02:43.140 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.106+0000 7f6c73fff700 1 Processor -- start 2021-05-25T12:02:43.140 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.106+0000 7f6c73fff700 1 -- start start 2021-05-25T12:02:43.140 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.106+0000 7f6c73fff700 1 librados: setting wanted keys 2021-05-25T12:02:43.140 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.106+0000 7f6c73fff700 1 librados: calling monclient init 2021-05-25T12:02:43.140 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.106+0000 7f6c73fff700 10 monclient: init 2021-05-25T12:02:43.141 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.107+0000 7f6c73fff700 10 monclient: _reopen_session rank -1 2021-05-25T12:02:43.141 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.107+0000 7f6c73fff700 10 monclient: _add_conns ranks=[1,2,0] 2021-05-25T12:02:43.141 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.107+0000 7f6c73fff700 10 monclient(hunting): picked mon.noname-b con 0x7f6c74068e70 addr v1:172.21.15.181:6789/0 2021-05-25T12:02:43.141 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.107+0000 7f6c73fff700 10 monclient(hunting): picked mon.noname-c con 0x7f6c74093130 addr v1:172.21.15.102:6790/0 2021-05-25T12:02:43.142 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.107+0000 7f6c73fff700 10 monclient(hunting): picked mon.noname-a con 0x7f6c7412a430 addr v1:172.21.15.102:6789/0 2021-05-25T12:02:43.142 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.107+0000 7f6c73fff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c74137c60 con 0x7f6c7412a430 2021-05-25T12:02:43.142 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.107+0000 7f6c73fff700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c74067900 con 0x7f6c74068e70 2021-05-25T12:02:43.142 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.107+0000 7f6c73fff700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c74067ba0 con 0x7f6c74093130 2021-05-25T12:02:43.142 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.107+0000 7f6c73fff700 10 monclient(hunting): _renew_subs 2021-05-25T12:02:43.143 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.107+0000 7f6c73fff700 10 monclient(hunting): authenticate will time out at 963.704296s 2021-05-25T12:02:43.143 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.107+0000 7f6c737fe700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f6c7412a430 0x7f6c7413f3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60234/0 (socket says 172.21.15.102:60234) 2021-05-25T12:02:43.143 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.107+0000 7f6c727fc700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f6c74093130 0x7f6c74137550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:55110/0 (socket says 172.21.15.102:55110) 2021-05-25T12:02:43.143 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.107+0000 7f6c737fe700 1 -- 172.21.15.102:0/2377385149 learned_addr learned my addr 172.21.15.102:0/2377385149 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:02:43.144 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.107+0000 7f6c5affd700 10 client.?.objecter ms_handle_connect 0x7f6c7412a430 2021-05-25T12:02:43.144 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.107+0000 7f6c5affd700 10 client.?.objecter resend_mon_ops 2021-05-25T12:02:43.144 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.107+0000 7f6c5affd700 10 client.?.objecter ms_handle_connect 0x7f6c74093130 2021-05-25T12:02:43.144 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.107+0000 7f6c5affd700 10 client.?.objecter resend_mon_ops 2021-05-25T12:02:43.144 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.107+0000 7f6c5affd700 10 client.?.objecter ms_handle_connect 0x7f6c74068e70 2021-05-25T12:02:43.145 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.107+0000 7f6c5affd700 10 client.?.objecter resend_mon_ops 2021-05-25T12:02:43.145 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.107+0000 7f6c5affd700 1 -- 172.21.15.102:0/2377385149 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3025305375 0 0) 0x7f6c74137c60 con 0x7f6c7412a430 2021-05-25T12:02:43.145 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.107+0000 7f6c5affd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:02:43.145 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.107+0000 7f6c5affd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:02:43.145 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.107+0000 7f6c5affd700 10 monclient(hunting): my global_id is 174189 2021-05-25T12:02:43.146 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.107+0000 7f6c5affd700 1 -- 172.21.15.102:0/2377385149 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c54004c10 con 0x7f6c7412a430 2021-05-25T12:02:43.146 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.107+0000 7f6c5affd700 1 -- 172.21.15.102:0/2377385149 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2136287864 0 0) 0x7f6c74067ba0 con 0x7f6c74093130 2021-05-25T12:02:43.146 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.107+0000 7f6c5affd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:02:43.146 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.107+0000 7f6c5affd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:02:43.147 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.107+0000 7f6c5affd700 10 monclient(hunting): my global_id is 23075 2021-05-25T12:02:43.147 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.107+0000 7f6c5affd700 1 -- 172.21.15.102:0/2377385149 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c74137c60 con 0x7f6c74093130 2021-05-25T12:02:43.147 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.107+0000 7f6c5affd700 1 -- 172.21.15.102:0/2377385149 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 487015936 0 0) 0x7f6c74067900 con 0x7f6c74068e70 2021-05-25T12:02:43.147 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.107+0000 7f6c5affd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:02:43.147 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.107+0000 7f6c5affd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:02:43.148 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.107+0000 7f6c5affd700 10 monclient(hunting): my global_id is 95146 2021-05-25T12:02:43.148 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.107+0000 7f6c5affd700 1 -- 172.21.15.102:0/2377385149 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c74067ba0 con 0x7f6c74068e70 2021-05-25T12:02:43.148 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.107+0000 7f6c5affd700 1 -- 172.21.15.102:0/2377385149 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1329009752 0 0) 0x7f6c54004c10 con 0x7f6c7412a430 2021-05-25T12:02:43.148 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.107+0000 7f6c5affd700 1 -- 172.21.15.102:0/2377385149 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6c74067900 con 0x7f6c7412a430 2021-05-25T12:02:43.149 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.107+0000 7f6c5affd700 1 -- 172.21.15.102:0/2377385149 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f6c64001f20 con 0x7f6c7412a430 2021-05-25T12:02:43.149 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.107+0000 7f6c5affd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:43.149 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.107+0000 7f6c5affd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:02:43.149 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.107+0000 7f6c5affd700 10 monclient(hunting): dump: 2021-05-25T12:02:43.149 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:43.150 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:43.150 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:43.150 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:43.150 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:43.150 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:43.151 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:43.151 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:43.151 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:43.151 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:43.152 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.107+0000 7f6c5affd700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:02:43.152 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.107+0000 7f6c5affd700 1 -- 172.21.15.102:0/2377385149 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3531450355 0 0) 0x7f6c74137c60 con 0x7f6c74093130 2021-05-25T12:02:43.152 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.107+0000 7f6c5affd700 1 -- 172.21.15.102:0/2377385149 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6c54004c10 con 0x7f6c74093130 2021-05-25T12:02:43.152 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.107+0000 7f6c5affd700 1 -- 172.21.15.102:0/2377385149 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f6c5c001840 con 0x7f6c74093130 2021-05-25T12:02:43.152 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.107+0000 7f6c5affd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:43.153 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.107+0000 7f6c5affd700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2021-05-25T12:02:43.153 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.107+0000 7f6c5affd700 10 monclient(hunting): dump: 2021-05-25T12:02:43.153 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:43.153 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:43.153 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:43.154 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:43.154 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:43.154 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:43.154 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:43.155 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:43.155 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:43.155 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:43.155 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.107+0000 7f6c5affd700 1 -- 172.21.15.102:0/2377385149 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2839523994 0 0) 0x7f6c74067ba0 con 0x7f6c74068e70 2021-05-25T12:02:43.155 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.107+0000 7f6c5affd700 1 -- 172.21.15.102:0/2377385149 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6c74137c60 con 0x7f6c74068e70 2021-05-25T12:02:43.156 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.107+0000 7f6c5affd700 1 -- 172.21.15.102:0/2377385149 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f6c680018e0 con 0x7f6c74068e70 2021-05-25T12:02:43.156 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.108+0000 7f6c5affd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:43.156 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.108+0000 7f6c5affd700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2021-05-25T12:02:43.156 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.108+0000 7f6c5affd700 10 monclient(hunting): dump: 2021-05-25T12:02:43.156 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:43.157 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:43.157 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:43.157 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:43.157 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:43.158 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:43.158 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:43.158 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:43.158 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:43.158 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:43.159 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.108+0000 7f6c5affd700 1 -- 172.21.15.102:0/2377385149 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 267036834 0 0) 0x7f6c74067900 con 0x7f6c7412a430 2021-05-25T12:02:43.159 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.108+0000 7f6c5affd700 1 -- 172.21.15.102:0/2377385149 >> v1:172.21.15.102:6790/0 conn(0x7f6c74093130 legacy=0x7f6c74137550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:43.159 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.108+0000 7f6c5affd700 1 -- 172.21.15.102:0/2377385149 >> v1:172.21.15.181:6789/0 conn(0x7f6c74068e70 legacy=0x7f6c74136e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:43.159 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.108+0000 7f6c5affd700 10 monclient: _finish_hunting 0 2021-05-25T12:02:43.159 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.108+0000 7f6c5affd700 1 monclient: found mon.a 2021-05-25T12:02:43.160 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.108+0000 7f6c5affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:02:43.160 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.108+0000 7f6c5affd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:02:43.160 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.108+0000 7f6c5affd700 1 -- 172.21.15.102:0/2377385149 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c74067e40 con 0x7f6c7412a430 2021-05-25T12:02:43.160 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.108+0000 7f6c5affd700 10 monclient: _finish_auth 0 2021-05-25T12:02:43.161 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.108+0000 7f6c5affd700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:02:43.161 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.108+0000 7f6c73fff700 5 monclient: authenticate success, global_id 174189 2021-05-25T12:02:43.161 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.108+0000 7f6c73fff700 10 monclient: _renew_subs 2021-05-25T12:02:43.161 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.108+0000 7f6c73fff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:02:43.161 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.108+0000 7f6c73fff700 1 -- 172.21.15.102:0/2377385149 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c74065cd0 con 0x7f6c7412a430 2021-05-25T12:02:43.162 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.108+0000 7f6c73fff700 10 client.174189.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:02:43.162 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.108+0000 7f6c73fff700 10 monclient: _renew_subs 2021-05-25T12:02:43.162 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.108+0000 7f6c73fff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:02:43.162 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.108+0000 7f6c73fff700 1 -- 172.21.15.102:0/2377385149 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c74068660 con 0x7f6c7412a430 2021-05-25T12:02:43.162 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.108+0000 7f6c73fff700 1 librados: init done 2021-05-25T12:02:43.163 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.108+0000 7f6c5affd700 1 -- 172.21.15.102:0/2377385149 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6c640015a0 con 0x7f6c7412a430 2021-05-25T12:02:43.163 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.108+0000 7f6c5affd700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:02:43.163 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.108+0000 7f6c5affd700 1 -- 172.21.15.102:0/2377385149 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f6c64004e20 con 0x7f6c7412a430 2021-05-25T12:02:43.163 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.108+0000 7f6c5affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:43.163 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.108+0000 7f6c717fa700 4 set_mon_vals no callback set 2021-05-25T12:02:43.164 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.108+0000 7f6c5affd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:02:43.164 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.108+0000 7f6c5affd700 10 monclient: dump: 2021-05-25T12:02:43.164 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:43.164 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:43.164 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:43.165 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:43.165 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:43.165 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:43.165 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:43.166 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:43.166 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:43.166 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:43.166 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.108+0000 7f6c73fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:02:43.166 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.108+0000 7f6c5affd700 1 -- 172.21.15.102:0/2377385149 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f6c64005000 con 0x7f6c7412a430 2021-05-25T12:02:43.167 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.108+0000 7f6c73fff700 1 -- 172.21.15.102:0/2377385149 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f6c4c0065a0 con 0x7f6c4c002840 2021-05-25T12:02:43.167 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.108+0000 7f6c5affd700 10 client.174189.objecter ms_handle_connect 0x7f6c4c002840 2021-05-25T12:02:43.167 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.108+0000 7f6c5affd700 10 client.174189.objecter resend_mon_ops 2021-05-25T12:02:43.167 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.108+0000 7f6c5affd700 1 -- 172.21.15.102:0/2377385149 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(81..81 src has 1..81) v4 ==== 6162+0+0 (unknown 275132006 0 0) 0x7f6c6403dd60 con 0x7f6c7412a430 2021-05-25T12:02:43.168 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.108+0000 7f6c5affd700 10 client.174189.objecter ms_dispatch 0x7f6c74001550 osd_map(81..81 src has 1..81) v4 2021-05-25T12:02:43.168 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.108+0000 7f6c5affd700 3 client.174189.objecter handle_osd_map got epochs [81,81] > 0 2021-05-25T12:02:43.168 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.108+0000 7f6c5affd700 3 client.174189.objecter handle_osd_map decoding full epoch 81 2021-05-25T12:02:43.168 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.109+0000 7f6c5affd700 20 client.174189.objecter dump_active .. 0 homeless 2021-05-25T12:02:43.168 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.109+0000 7f6c5affd700 1 -- 172.21.15.102:0/2377385149 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3594594975 0 0) 0x7f6c4c0065a0 con 0x7f6c4c002840 2021-05-25T12:02:43.169 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.109+0000 7f6c5affd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:02:43.169 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.109+0000 7f6c5affd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:02:43.169 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.109+0000 7f6c5affd700 10 monclient(hunting): my global_id is 23078 2021-05-25T12:02:43.169 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.109+0000 7f6c5affd700 1 -- 172.21.15.102:0/2377385149 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c74067900 con 0x7f6c4c002840 2021-05-25T12:02:43.169 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.109+0000 7f6c727fc700 10 monclient: get_auth_request con 0x7f6c54033550 auth_method 0 2021-05-25T12:02:43.170 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.109+0000 7f6c5affd700 1 -- 172.21.15.102:0/2377385149 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4111478189 0 0) 0x7f6c74067900 con 0x7f6c4c002840 2021-05-25T12:02:43.170 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.109+0000 7f6c5affd700 1 -- 172.21.15.102:0/2377385149 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6c4c0065a0 con 0x7f6c4c002840 2021-05-25T12:02:43.170 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.109+0000 7f6c5affd700 1 -- 172.21.15.102:0/2377385149 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f6c680046d0 con 0x7f6c4c002840 2021-05-25T12:02:43.170 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.109+0000 7f6c5affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:43.171 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.109+0000 7f6c5affd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:02:43.171 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.109+0000 7f6c5affd700 10 monclient: dump: 2021-05-25T12:02:43.171 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:43.171 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:43.171 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:43.172 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:43.172 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:43.172 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:43.172 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:43.172 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:43.173 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:43.173 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:43.173 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.109+0000 7f6c5affd700 10 client.174189.objecter ms_handle_connect 0x7f6c54033550 2021-05-25T12:02:43.173 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.109+0000 7f6c5affd700 1 -- 172.21.15.102:0/2377385149 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3145228784 0 0) 0x7f6c4c0065a0 con 0x7f6c4c002840 2021-05-25T12:02:43.173 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.109+0000 7f6c5affd700 1 -- 172.21.15.102:0/2377385149 --> v1:172.21.15.102:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6c4c009e40 con 0x7f6c4c002840 2021-05-25T12:02:43.174 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.109+0000 7f6c5affd700 1 -- 172.21.15.102:0/2377385149 <== mon.2 v1:172.21.15.102:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7f6c68005620 con 0x7f6c4c002840 2021-05-25T12:02:43.174 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.109+0000 7f6c5affd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:02:43.174 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.109+0000 7f6c5affd700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:02:43.174 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.109+0000 7f6c5affd700 1 -- 172.21.15.102:0/2377385149 >> v1:172.21.15.102:6790/0 conn(0x7f6c4c002840 legacy=0x7f6c4c004ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:43.175 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.109+0000 7f6c5affd700 1 -- 172.21.15.102:0/2377385149 >> v1:172.21.15.102:6790/0 conn(0x7f6c4c002840 legacy=0x7f6c4c004ce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:02:43.175 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.116+0000 7f6c73fff700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:02:43.175 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.116+0000 7f6c73fff700 1 -- 172.21.15.102:0/2377385149 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f6c4c007540 con 0x7f6c4c0068e0 2021-05-25T12:02:43.175 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.116+0000 7f6c5affd700 10 client.174189.objecter ms_handle_connect 0x7f6c4c0068e0 2021-05-25T12:02:43.175 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.116+0000 7f6c5affd700 10 client.174189.objecter resend_mon_ops 2021-05-25T12:02:43.176 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.116+0000 7f6c5affd700 1 -- 172.21.15.102:0/2377385149 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 175656969 0 0) 0x7f6c4c007540 con 0x7f6c4c0068e0 2021-05-25T12:02:43.176 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.116+0000 7f6c5affd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:02:43.176 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.116+0000 7f6c5affd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:02:43.176 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.116+0000 7f6c5affd700 10 monclient(hunting): my global_id is 23081 2021-05-25T12:02:43.176 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.116+0000 7f6c5affd700 1 -- 172.21.15.102:0/2377385149 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c4c0065a0 con 0x7f6c4c0068e0 2021-05-25T12:02:43.177 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.116+0000 7f6c5affd700 1 -- 172.21.15.102:0/2377385149 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1370046716 0 0) 0x7f6c4c0065a0 con 0x7f6c4c0068e0 2021-05-25T12:02:43.177 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.116+0000 7f6c5affd700 1 -- 172.21.15.102:0/2377385149 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6c4c007540 con 0x7f6c4c0068e0 2021-05-25T12:02:43.177 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.116+0000 7f6c5affd700 1 -- 172.21.15.102:0/2377385149 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f6c68004e40 con 0x7f6c4c0068e0 2021-05-25T12:02:43.177 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.116+0000 7f6c5affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:43.177 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.116+0000 7f6c5affd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:02:43.178 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.116+0000 7f6c5affd700 10 monclient: dump: 2021-05-25T12:02:43.178 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:43.178 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:43.178 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:43.179 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:43.179 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:43.179 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:43.179 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:43.179 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:43.180 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:43.180 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:43.180 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.116+0000 7f6c5affd700 1 -- 172.21.15.102:0/2377385149 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3796456741 0 0) 0x7f6c4c007540 con 0x7f6c4c0068e0 2021-05-25T12:02:43.180 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.116+0000 7f6c5affd700 1 -- 172.21.15.102:0/2377385149 --> v1:172.21.15.102:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f6c4c008770 con 0x7f6c4c0068e0 2021-05-25T12:02:43.180 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.117+0000 7f6c5affd700 1 -- 172.21.15.102:0/2377385149 <== mon.2 v1:172.21.15.102:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3942 (unknown 0 0 582699588) 0x7f6c68005250 con 0x7f6c4c0068e0 2021-05-25T12:02:43.181 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.117+0000 7f6c5affd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:02:43.181 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.117+0000 7f6c5affd700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:02:43.181 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.117+0000 7f6c5affd700 1 -- 172.21.15.102:0/2377385149 >> v1:172.21.15.102:6790/0 conn(0x7f6c4c0068e0 legacy=0x7f6c4c006d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:43.181 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.117+0000 7f6c5affd700 1 -- 172.21.15.102:0/2377385149 >> v1:172.21.15.102:6790/0 conn(0x7f6c4c0068e0 legacy=0x7f6c4c006d70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:02:43.182 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:02:43.182 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:02:43.182 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:02:43.183 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "leader", 2021-05-25T12:02:43.183 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 30, 2021-05-25T12:02:43.183 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:02:43.183 INFO:teuthology.orchestra.run.smithi102.stdout: 0, 2021-05-25T12:02:43.183 INFO:teuthology.orchestra.run.smithi102.stdout: 1, 2021-05-25T12:02:43.184 INFO:teuthology.orchestra.run.smithi102.stdout: 2 2021-05-25T12:02:43.184 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:02:43.184 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 2, 2021-05-25T12:02:43.184 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:02:43.185 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:02:43.185 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:02:43.185 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:02:43.185 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:02:43.185 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:02:43.186 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:02:43.186 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:02:43.186 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:02:43.186 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:02:43.186 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:02:43.187 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:02:43.187 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:02:43.187 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:02:43.187 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:02:43.187 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:02:43.188 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:02:43.188 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:02:43.188 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:02:43.188 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:02:43.189 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:02:43.189 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:02:43.189 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:02:43.189 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:02:43.189 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:02:43.190 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:02:43.190 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:02:43.190 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:02:43.190 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:02:43.191 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:02:43.191 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:02:43.191 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:02:43.191 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:02:43.191 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:02:43.192 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:02:43.192 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:02:43.192 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:02:43.192 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:02:43.192 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:02:43.193 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:02:43.193 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:02:43.193 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:02:43.193 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:02:43.193 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:02:43.194 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:02:43.194 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:02:43.194 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:02:43.194 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:02:43.195 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:02:43.195 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:02:43.195 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:02:43.195 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:02:43.195 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:02:43.196 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:02:43.196 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:02:43.196 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:02:43.196 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:02:43.197 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:02:43.197 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:02:43.197 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:02:43.197 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:02:43.197 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:02:43.198 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:02:43.198 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:02:43.198 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:02:43.198 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:02:43.198 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:02:43.199 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:02:43.199 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:02:43.199 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:02:43.199 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:02:43.200 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:02:43.200 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:02:43.200 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:02:43.200 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:02:43.200 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:02:43.201 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:02:43.201 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:02:43.201 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:02:43.201 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:02:43.201 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:02:43.202 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:02:43.202 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:02:43.202 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:02:43.202 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:02:43.203 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:02:43.203 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:02:43.203 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:02:43.203 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:02:43.203 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:02:43.204 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:02:43.204 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:02:43.204 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:02:43.204 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:02:43.204 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:02:43.205 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:02:43.205 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:02:43.205 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:02:43.205 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:02:43.205 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:02:43.206 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:02:43.206 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:02:43.206 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:02:43.206 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:02:43.206 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:02:43.207 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:02:43.207 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:02:43.207 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:02:43.207 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:02:43.208 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:02:43.208 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:02:43.208 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:02:43.208 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:02:43.208 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:02:43.209 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:02:43.209 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:02:43.209 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:02:43.209 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:02:43.209 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:02:43.210 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:02:43.210 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:02:43.210 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:02:43.210 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 6 2021-05-25T12:02:43.210 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:02:43.211 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:02:43.211 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": [ 2021-05-25T12:02:43.211 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:02:43.211 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:02:43.211 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:02:43.212 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:02:43.212 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:02:43.212 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:02:43.212 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:02:43.212 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:02:43.213 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:02:43.215 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.117+0000 7f6c73fff700 10 librados: watch_flush enter 2021-05-25T12:02:43.215 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.117+0000 7f6c73fff700 10 librados: watch_flush exit 2021-05-25T12:02:43.216 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.117+0000 7f6c73fff700 20 client.174189.objecter shutdown clearing up homeless session... 2021-05-25T12:02:43.216 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.117+0000 7f6c73fff700 10 client.174189.objecter successfully canceled tick 2021-05-25T12:02:43.216 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.117+0000 7f6c73fff700 1 -- 172.21.15.102:0/2377385149 >> v1:172.21.15.181:6812/29582 conn(0x7f6c54033550 legacy=0x7f6c54035a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:43.216 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.117+0000 7f6c73fff700 10 monclient: shutdown 2021-05-25T12:02:43.216 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.117+0000 7f6c73fff700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:02:43.217 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.117+0000 7f6c73fff700 1 -- 172.21.15.102:0/2377385149 >> v1:172.21.15.102:6789/0 conn(0x7f6c7412a430 legacy=0x7f6c7413f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:43.217 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.117+0000 7f6c737fe700 1 -- 172.21.15.102:0/2377385149 reap_dead start 2021-05-25T12:02:43.217 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.117+0000 7f6c737fe700 1 -- 172.21.15.102:0/2377385149 reap_dead start 2021-05-25T12:02:43.217 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.117+0000 7f6c73fff700 1 -- 172.21.15.102:0/2377385149 shutdown_connections 2021-05-25T12:02:43.217 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.117+0000 7f6c73fff700 1 -- 172.21.15.102:0/2377385149 >> 172.21.15.102:0/2377385149 conn(0x7f6c740974b0 msgr2=0x7f6c74098fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:02:43.218 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.117+0000 7f6c73fff700 1 -- 172.21.15.102:0/2377385149 shutdown_connections 2021-05-25T12:02:43.218 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.117+0000 7f6c73fff700 1 -- 172.21.15.102:0/2377385149 wait complete. 2021-05-25T12:02:43.218 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.117+0000 7f6c73fff700 1 librados: shutdown 2021-05-25T12:02:43.219 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2021-05-25T12:02:43.328 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.326+0000 7f9adbf47700 10 monclient: get_monmap_and_config 2021-05-25T12:02:43.328 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.326+0000 7f9adbf47700 10 monclient: build_initial_monmap 2021-05-25T12:02:43.328 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.326+0000 7f9adbf47700 10 monclient: monmap: 2021-05-25T12:02:43.328 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:02:43.329 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:43.329 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:02:43.327747+0000 2021-05-25T12:02:43.329 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:02:43.327747+0000 2021-05-25T12:02:43.329 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:02:43.329 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:43.330 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:02:43.330 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:02:43.330 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:02:43.330 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:43.344 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.327+0000 7f9adbf47700 1 Processor -- start 2021-05-25T12:02:43.344 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.327+0000 7f9adbf47700 1 -- start start 2021-05-25T12:02:43.344 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.327+0000 7f9adbf47700 10 monclient: init 2021-05-25T12:02:43.344 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.327+0000 7f9adbf47700 10 monclient: _reopen_session rank -1 2021-05-25T12:02:43.345 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.327+0000 7f9adbf47700 10 monclient: _add_conns ranks=[2,0,1] 2021-05-25T12:02:43.345 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.328+0000 7f9adbf47700 10 monclient(hunting): picked mon.noname-b con 0x7f9ad409a5e0 addr v1:172.21.15.102:6790/0 2021-05-25T12:02:43.345 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.328+0000 7f9adbf47700 10 monclient(hunting): picked mon.noname-a con 0x7f9ad4063de0 addr v1:172.21.15.102:6789/0 2021-05-25T12:02:43.345 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.328+0000 7f9adbf47700 10 monclient(hunting): picked mon.noname-c con 0x7f9ad41325c0 addr v1:172.21.15.181:6789/0 2021-05-25T12:02:43.346 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.328+0000 7f9adbf47700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ad409a3a0 con 0x7f9ad4063de0 2021-05-25T12:02:43.346 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.328+0000 7f9adbf47700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ad409b400 con 0x7f9ad41325c0 2021-05-25T12:02:43.346 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.328+0000 7f9adbf47700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ad40963f0 con 0x7f9ad409a5e0 2021-05-25T12:02:43.346 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.328+0000 7f9adbf47700 10 monclient(hunting): _renew_subs 2021-05-25T12:02:43.346 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.328+0000 7f9adbf47700 10 monclient(hunting): authenticate will time out at 963.925097s 2021-05-25T12:02:43.347 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.328+0000 7f9ad94e2700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f9ad4063de0 0x7f9ad4064260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60244/0 (socket says 172.21.15.102:60244) 2021-05-25T12:02:43.347 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.328+0000 7f9ad9ce3700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f9ad409a5e0 0x7f9ad40636d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:55120/0 (socket says 172.21.15.102:55120) 2021-05-25T12:02:43.347 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.328+0000 7f9ad94e2700 1 -- 172.21.15.102:0/3056535088 learned_addr learned my addr 172.21.15.102:0/3056535088 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:02:43.347 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.328+0000 7f9ad8ce1700 1 -- 172.21.15.102:0/3056535088 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3246395699 0 0) 0x7f9ad409a3a0 con 0x7f9ad4063de0 2021-05-25T12:02:43.347 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.328+0000 7f9ad8ce1700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:02:43.348 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.328+0000 7f9ad8ce1700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:02:43.348 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.328+0000 7f9ad8ce1700 10 monclient(hunting): my global_id is 174192 2021-05-25T12:02:43.348 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.328+0000 7f9ad8ce1700 1 -- 172.21.15.102:0/3056535088 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9abc003270 con 0x7f9ad4063de0 2021-05-25T12:02:43.348 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.328+0000 7f9ad8ce1700 1 -- 172.21.15.102:0/3056535088 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2848302682 0 0) 0x7f9ad40963f0 con 0x7f9ad409a5e0 2021-05-25T12:02:43.349 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.328+0000 7f9ad8ce1700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:02:43.349 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.328+0000 7f9ad8ce1700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:02:43.349 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.328+0000 7f9ad8ce1700 10 monclient(hunting): my global_id is 23084 2021-05-25T12:02:43.349 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.328+0000 7f9ad8ce1700 1 -- 172.21.15.102:0/3056535088 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ad409a3a0 con 0x7f9ad409a5e0 2021-05-25T12:02:43.349 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.328+0000 7f9ad8ce1700 1 -- 172.21.15.102:0/3056535088 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4186658683 0 0) 0x7f9ad409b400 con 0x7f9ad41325c0 2021-05-25T12:02:43.350 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.328+0000 7f9ad8ce1700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:02:43.350 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.328+0000 7f9ad8ce1700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:02:43.350 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.328+0000 7f9ad8ce1700 10 monclient(hunting): my global_id is 95149 2021-05-25T12:02:43.350 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.328+0000 7f9ad8ce1700 1 -- 172.21.15.102:0/3056535088 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ad40963f0 con 0x7f9ad41325c0 2021-05-25T12:02:43.351 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.328+0000 7f9ad8ce1700 1 -- 172.21.15.102:0/3056535088 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2358642624 0 0) 0x7f9abc003270 con 0x7f9ad4063de0 2021-05-25T12:02:43.351 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.328+0000 7f9ad8ce1700 1 -- 172.21.15.102:0/3056535088 >> v1:172.21.15.102:6790/0 conn(0x7f9ad409a5e0 legacy=0x7f9ad40636d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:43.351 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.328+0000 7f9ad8ce1700 1 -- 172.21.15.102:0/3056535088 >> v1:172.21.15.181:6789/0 conn(0x7f9ad41325c0 legacy=0x7f9ad41349b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:43.351 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.328+0000 7f9ad8ce1700 10 monclient: _finish_hunting 0 2021-05-25T12:02:43.351 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.328+0000 7f9ad8ce1700 1 monclient: found mon.noname-a 2021-05-25T12:02:43.352 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.328+0000 7f9ad8ce1700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:02:43.352 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.328+0000 7f9ad8ce1700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T12:02:43.352 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.328+0000 7f9ad8ce1700 1 -- 172.21.15.102:0/3056535088 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ad4096690 con 0x7f9ad4063de0 2021-05-25T12:02:43.352 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.328+0000 7f9ad8ce1700 10 monclient: _finish_auth 0 2021-05-25T12:02:43.352 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.328+0000 7f9ad8ce1700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:02:43.353 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.328+0000 7f9ad8ce1700 1 -- 172.21.15.102:0/3056535088 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f9ac4001610 con 0x7f9ad4063de0 2021-05-25T12:02:43.353 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.328+0000 7f9ad8ce1700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:43.353 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.328+0000 7f9ad8ce1700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:02:43.353 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.328+0000 7f9ad8ce1700 10 monclient: dump: 2021-05-25T12:02:43.354 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:43.354 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:43.354 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:43.354 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:43.354 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:43.355 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:43.355 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:43.355 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:43.355 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:43.355 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:43.356 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.328+0000 7f9adbf47700 5 monclient: authenticate success, global_id 174192 2021-05-25T12:02:43.356 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.328+0000 7f9adbf47700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:02:43.356 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.328+0000 7f9ad8ce1700 1 -- 172.21.15.102:0/3056535088 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9ac4001880 con 0x7f9ad4063de0 2021-05-25T12:02:43.356 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.328+0000 7f9ad8ce1700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:02:43.357 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.328+0000 7f9adbf47700 10 monclient: get_monmap_and_config success 2021-05-25T12:02:43.357 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.328+0000 7f9ad8ce1700 1 -- 172.21.15.102:0/3056535088 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f9ac4001c10 con 0x7f9ad4063de0 2021-05-25T12:02:43.357 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.328+0000 7f9adbf47700 4 set_mon_vals no callback set 2021-05-25T12:02:43.357 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.328+0000 7f9ad8ce1700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:43.357 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.328+0000 7f9ad8ce1700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:02:43.358 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.328+0000 7f9ad8ce1700 10 monclient: dump: 2021-05-25T12:02:43.358 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:43.358 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:43.358 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:43.359 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:43.359 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:43.359 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:43.359 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:43.359 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:43.360 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:43.360 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:43.360 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.329+0000 7f9adbf47700 10 monclient: shutdown 2021-05-25T12:02:43.360 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.329+0000 7f9adbf47700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:02:43.361 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.329+0000 7f9adbf47700 1 -- 172.21.15.102:0/3056535088 >> v1:172.21.15.102:6789/0 conn(0x7f9ad4063de0 legacy=0x7f9ad4064260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:43.361 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.329+0000 7f9adbf47700 1 -- 172.21.15.102:0/3056535088 shutdown_connections 2021-05-25T12:02:43.361 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.329+0000 7f9adbf47700 1 -- 172.21.15.102:0/3056535088 >> 172.21.15.102:0/3056535088 conn(0x7f9ad409bc30 msgr2=0x7f9ad409c040 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:02:43.361 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.329+0000 7f9adbf47700 1 -- 172.21.15.102:0/3056535088 shutdown_connections 2021-05-25T12:02:43.362 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.329+0000 7f9adbf47700 1 -- 172.21.15.102:0/3056535088 wait complete. 2021-05-25T12:02:43.362 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.329+0000 7f9adbf47700 10 monclient: build_initial_monmap 2021-05-25T12:02:43.362 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.329+0000 7f9adbf47700 10 monclient: monmap: 2021-05-25T12:02:43.362 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:02:43.362 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:02:43.363 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:02:43.363 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:02:43.363 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:02:43.363 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:02:43.364 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:02:43.364 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:02:43.364 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:02:43.364 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:43.364 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.330+0000 7f9adbf47700 1 librados: starting msgr at 2021-05-25T12:02:43.365 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.330+0000 7f9adbf47700 1 librados: starting objecter 2021-05-25T12:02:43.365 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.330+0000 7f9adbf47700 1 Processor -- start 2021-05-25T12:02:43.365 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.330+0000 7f9adbf47700 1 -- start start 2021-05-25T12:02:43.365 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.330+0000 7f9adbf47700 1 librados: setting wanted keys 2021-05-25T12:02:43.365 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.330+0000 7f9adbf47700 1 librados: calling monclient init 2021-05-25T12:02:43.366 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.330+0000 7f9adbf47700 10 monclient: init 2021-05-25T12:02:43.366 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.330+0000 7f9adbf47700 10 monclient: _reopen_session rank -1 2021-05-25T12:02:43.366 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.330+0000 7f9adbf47700 10 monclient: _add_conns ranks=[2,1,0] 2021-05-25T12:02:43.366 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.330+0000 7f9adbf47700 10 monclient(hunting): picked mon.noname-c con 0x7f9ad4063de0 addr v1:172.21.15.102:6790/0 2021-05-25T12:02:43.367 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.330+0000 7f9adbf47700 10 monclient(hunting): picked mon.noname-b con 0x7f9ad409a5e0 addr v1:172.21.15.181:6789/0 2021-05-25T12:02:43.367 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.330+0000 7f9adbf47700 10 monclient(hunting): picked mon.noname-a con 0x7f9ad41325c0 addr v1:172.21.15.102:6789/0 2021-05-25T12:02:43.367 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.330+0000 7f9adbf47700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ad409ab10 con 0x7f9ad41325c0 2021-05-25T12:02:43.367 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.330+0000 7f9adbf47700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ad412e900 con 0x7f9ad409a5e0 2021-05-25T12:02:43.367 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.330+0000 7f9adbf47700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ad412b2d0 con 0x7f9ad4063de0 2021-05-25T12:02:43.368 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.330+0000 7f9adbf47700 10 monclient(hunting): _renew_subs 2021-05-25T12:02:43.368 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.330+0000 7f9adbf47700 10 monclient(hunting): authenticate will time out at 963.927691s 2021-05-25T12:02:43.368 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.330+0000 7f9ad9ce3700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f9ad4063de0 0x7f9ad413b540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:55126/0 (socket says 172.21.15.102:55126) 2021-05-25T12:02:43.368 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.330+0000 7f9ad9ce3700 1 -- 172.21.15.102:0/3558392646 learned_addr learned my addr 172.21.15.102:0/3558392646 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:02:43.368 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.330+0000 7f9ac1ffb700 10 client.?.objecter ms_handle_connect 0x7f9ad41325c0 2021-05-25T12:02:43.369 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.330+0000 7f9ac1ffb700 10 client.?.objecter resend_mon_ops 2021-05-25T12:02:43.369 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.330+0000 7f9ac1ffb700 10 client.?.objecter ms_handle_connect 0x7f9ad4063de0 2021-05-25T12:02:43.369 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.330+0000 7f9ac1ffb700 10 client.?.objecter resend_mon_ops 2021-05-25T12:02:43.369 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.330+0000 7f9ac1ffb700 10 client.?.objecter ms_handle_connect 0x7f9ad409a5e0 2021-05-25T12:02:43.370 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.330+0000 7f9ac1ffb700 10 client.?.objecter resend_mon_ops 2021-05-25T12:02:43.370 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.330+0000 7f9ac1ffb700 1 -- 172.21.15.102:0/3558392646 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 492957967 0 0) 0x7f9ad412b2d0 con 0x7f9ad4063de0 2021-05-25T12:02:43.370 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.330+0000 7f9ac1ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:02:43.370 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.330+0000 7f9ac1ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:02:43.370 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.330+0000 7f9ac1ffb700 10 monclient(hunting): my global_id is 23087 2021-05-25T12:02:43.371 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.330+0000 7f9ac1ffb700 1 -- 172.21.15.102:0/3558392646 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ab4004c10 con 0x7f9ad4063de0 2021-05-25T12:02:43.371 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.331+0000 7f9ac1ffb700 1 -- 172.21.15.102:0/3558392646 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2878276860 0 0) 0x7f9ad409ab10 con 0x7f9ad41325c0 2021-05-25T12:02:43.371 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.331+0000 7f9ac1ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:02:43.371 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.331+0000 7f9ac1ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:02:43.372 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.331+0000 7f9ac1ffb700 10 monclient(hunting): my global_id is 174195 2021-05-25T12:02:43.372 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.331+0000 7f9ac1ffb700 1 -- 172.21.15.102:0/3558392646 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ad412b2d0 con 0x7f9ad41325c0 2021-05-25T12:02:43.372 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.331+0000 7f9ac1ffb700 1 -- 172.21.15.102:0/3558392646 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 127180150 0 0) 0x7f9ad412e900 con 0x7f9ad409a5e0 2021-05-25T12:02:43.372 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.331+0000 7f9ac1ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:02:43.372 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.331+0000 7f9ac1ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:02:43.373 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.331+0000 7f9ac1ffb700 10 monclient(hunting): my global_id is 95152 2021-05-25T12:02:43.373 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.331+0000 7f9ac1ffb700 1 -- 172.21.15.102:0/3558392646 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ad409ab10 con 0x7f9ad409a5e0 2021-05-25T12:02:43.373 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.331+0000 7f9ac1ffb700 1 -- 172.21.15.102:0/3558392646 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1922341772 0 0) 0x7f9ab4004c10 con 0x7f9ad4063de0 2021-05-25T12:02:43.373 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.331+0000 7f9ac1ffb700 1 -- 172.21.15.102:0/3558392646 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9ad412e900 con 0x7f9ad4063de0 2021-05-25T12:02:43.373 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.331+0000 7f9ac1ffb700 1 -- 172.21.15.102:0/3558392646 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f9ac8001a10 con 0x7f9ad4063de0 2021-05-25T12:02:43.374 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.331+0000 7f9ac1ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:43.374 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.331+0000 7f9ac1ffb700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:02:43.374 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.331+0000 7f9ac1ffb700 10 monclient(hunting): dump: 2021-05-25T12:02:43.374 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:43.374 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:43.375 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:43.375 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:43.375 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:43.375 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:43.376 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:43.376 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:43.376 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:43.376 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:43.376 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.331+0000 7f9ac1ffb700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:02:43.377 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.331+0000 7f9ac1ffb700 1 -- 172.21.15.102:0/3558392646 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1920332086 0 0) 0x7f9ad412b2d0 con 0x7f9ad41325c0 2021-05-25T12:02:43.377 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.331+0000 7f9ac1ffb700 1 -- 172.21.15.102:0/3558392646 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9ab4004c10 con 0x7f9ad41325c0 2021-05-25T12:02:43.377 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.331+0000 7f9ac1ffb700 1 -- 172.21.15.102:0/3558392646 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f9ad0001bc0 con 0x7f9ad41325c0 2021-05-25T12:02:43.377 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.331+0000 7f9ac1ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:43.377 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.331+0000 7f9ac1ffb700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2021-05-25T12:02:43.378 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.331+0000 7f9ac1ffb700 10 monclient(hunting): dump: 2021-05-25T12:02:43.378 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:43.378 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:43.378 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:43.378 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:43.379 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:43.379 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:43.379 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:43.379 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:43.380 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:43.380 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:43.380 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.331+0000 7f9ac1ffb700 1 -- 172.21.15.102:0/3558392646 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4246163136 0 0) 0x7f9ad409ab10 con 0x7f9ad409a5e0 2021-05-25T12:02:43.380 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.331+0000 7f9ac1ffb700 1 -- 172.21.15.102:0/3558392646 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9ad412b2d0 con 0x7f9ad409a5e0 2021-05-25T12:02:43.380 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.331+0000 7f9ac1ffb700 1 -- 172.21.15.102:0/3558392646 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f9ac40015d0 con 0x7f9ad409a5e0 2021-05-25T12:02:43.381 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.331+0000 7f9ac1ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:43.381 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.331+0000 7f9ac1ffb700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2021-05-25T12:02:43.381 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.331+0000 7f9ac1ffb700 10 monclient(hunting): dump: 2021-05-25T12:02:43.381 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:43.381 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:43.382 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:43.382 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:43.382 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:43.382 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:43.383 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:43.383 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:43.383 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:43.383 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:43.383 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.331+0000 7f9ac1ffb700 1 -- 172.21.15.102:0/3558392646 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3459729575 0 0) 0x7f9ad412e900 con 0x7f9ad4063de0 2021-05-25T12:02:43.384 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.331+0000 7f9ac1ffb700 1 -- 172.21.15.102:0/3558392646 >> v1:172.21.15.181:6789/0 conn(0x7f9ad409a5e0 legacy=0x7f9ad412abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:43.384 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.331+0000 7f9ac1ffb700 1 -- 172.21.15.102:0/3558392646 >> v1:172.21.15.102:6789/0 conn(0x7f9ad41325c0 legacy=0x7f9ad4143430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:43.384 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.331+0000 7f9ac1ffb700 10 monclient: _finish_hunting 0 2021-05-25T12:02:43.384 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.331+0000 7f9ac1ffb700 1 monclient: found mon.c 2021-05-25T12:02:43.384 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.331+0000 7f9ac1ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:02:43.385 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.331+0000 7f9ac1ffb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:02:43.385 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.331+0000 7f9ac1ffb700 1 -- 172.21.15.102:0/3558392646 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ad4095650 con 0x7f9ad4063de0 2021-05-25T12:02:43.385 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.331+0000 7f9ac1ffb700 10 monclient: _finish_auth 0 2021-05-25T12:02:43.385 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.331+0000 7f9ac1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:02:43.386 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.331+0000 7f9adbf47700 5 monclient: authenticate success, global_id 23087 2021-05-25T12:02:43.386 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.331+0000 7f9adbf47700 10 monclient: _renew_subs 2021-05-25T12:02:43.386 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.331+0000 7f9adbf47700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:02:43.386 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.331+0000 7f9adbf47700 1 -- 172.21.15.102:0/3558392646 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ad40946c0 con 0x7f9ad4063de0 2021-05-25T12:02:43.386 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.331+0000 7f9ac1ffb700 1 -- 172.21.15.102:0/3558392646 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9ac8004530 con 0x7f9ad4063de0 2021-05-25T12:02:43.387 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.331+0000 7f9ac1ffb700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:02:43.387 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.331+0000 7f9ac1ffb700 1 -- 172.21.15.102:0/3558392646 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f9ac8004840 con 0x7f9ad4063de0 2021-05-25T12:02:43.387 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.331+0000 7f9ac1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:43.387 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.331+0000 7f9ac1ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:02:43.388 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.331+0000 7f9ac1ffb700 10 monclient: dump: 2021-05-25T12:02:43.388 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:43.388 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:43.388 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:43.388 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:43.389 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:43.389 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:43.389 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:43.389 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:43.389 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:43.390 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:43.390 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.331+0000 7f9adbf47700 10 client.23087.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:02:43.390 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.331+0000 7f9ac3fff700 4 set_mon_vals no callback set 2021-05-25T12:02:43.390 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.331+0000 7f9adbf47700 10 monclient: _renew_subs 2021-05-25T12:02:43.391 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.331+0000 7f9adbf47700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:02:43.391 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.331+0000 7f9adbf47700 1 -- 172.21.15.102:0/3558392646 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ad4093080 con 0x7f9ad4063de0 2021-05-25T12:02:43.391 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.331+0000 7f9adbf47700 1 librados: init done 2021-05-25T12:02:43.391 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.332+0000 7f9adbf47700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:02:43.391 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.332+0000 7f9ac1ffb700 1 -- 172.21.15.102:0/3558392646 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f9ac800dbe0 con 0x7f9ad4063de0 2021-05-25T12:02:43.392 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.332+0000 7f9adbf47700 1 -- 172.21.15.102:0/3558392646 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f9ad4063940 con 0x7f9ad412b780 2021-05-25T12:02:43.392 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.332+0000 7f9ac1ffb700 1 -- 172.21.15.102:0/3558392646 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(81..81 src has 1..81) v4 ==== 6162+0+0 (unknown 275132006 0 0) 0x7f9ac803d7c0 con 0x7f9ad4063de0 2021-05-25T12:02:43.392 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.332+0000 7f9ac1ffb700 10 client.23087.objecter ms_dispatch 0x7f9ad4001550 osd_map(81..81 src has 1..81) v4 2021-05-25T12:02:43.392 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.332+0000 7f9ac1ffb700 3 client.23087.objecter handle_osd_map got epochs [81,81] > 0 2021-05-25T12:02:43.392 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.332+0000 7f9ac1ffb700 3 client.23087.objecter handle_osd_map decoding full epoch 81 2021-05-25T12:02:43.393 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.332+0000 7f9ac1ffb700 20 client.23087.objecter dump_active .. 0 homeless 2021-05-25T12:02:43.393 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.332+0000 7f9ac1ffb700 10 client.23087.objecter ms_handle_connect 0x7f9ad412b780 2021-05-25T12:02:43.393 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.332+0000 7f9ac1ffb700 10 client.23087.objecter resend_mon_ops 2021-05-25T12:02:43.393 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.332+0000 7f9ada4e4700 10 monclient: get_auth_request con 0x7f9ab4033480 auth_method 0 2021-05-25T12:02:43.394 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.332+0000 7f9ac1ffb700 1 -- 172.21.15.102:0/3558392646 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 968916688 0 0) 0x7f9ad4063940 con 0x7f9ad412b780 2021-05-25T12:02:43.394 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.332+0000 7f9ac1ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:02:43.394 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.332+0000 7f9ac1ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:02:43.394 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.332+0000 7f9ac1ffb700 10 monclient(hunting): my global_id is 95155 2021-05-25T12:02:43.395 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.332+0000 7f9ac1ffb700 1 -- 172.21.15.102:0/3558392646 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ad412e900 con 0x7f9ad412b780 2021-05-25T12:02:43.395 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.332+0000 7f9ac1ffb700 10 client.23087.objecter ms_handle_connect 0x7f9ab4033480 2021-05-25T12:02:43.395 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.333+0000 7f9ac1ffb700 1 -- 172.21.15.102:0/3558392646 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3403688053 0 0) 0x7f9ad412e900 con 0x7f9ad412b780 2021-05-25T12:02:43.395 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.333+0000 7f9ac1ffb700 1 -- 172.21.15.102:0/3558392646 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9ad4063940 con 0x7f9ad412b780 2021-05-25T12:02:43.396 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.333+0000 7f9ac1ffb700 1 -- 172.21.15.102:0/3558392646 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f9ac4004940 con 0x7f9ad412b780 2021-05-25T12:02:43.396 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.333+0000 7f9ac1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:43.396 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.333+0000 7f9ac1ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T12:02:43.396 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.333+0000 7f9ac1ffb700 10 monclient: dump: 2021-05-25T12:02:43.396 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:43.397 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:43.397 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:43.397 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:43.397 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:43.398 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:43.399 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:43.399 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:43.399 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:43.399 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:43.399 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.333+0000 7f9ac1ffb700 1 -- 172.21.15.102:0/3558392646 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2795667403 0 0) 0x7f9ad4063940 con 0x7f9ad412b780 2021-05-25T12:02:43.400 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.333+0000 7f9ac1ffb700 1 -- 172.21.15.102:0/3558392646 --> v1:172.21.15.181:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9ad40948b0 con 0x7f9ad412b780 2021-05-25T12:02:43.400 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.333+0000 7f9ac1ffb700 1 -- 172.21.15.102:0/3558392646 <== mon.1 v1:172.21.15.181:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7f9ac4004cf0 con 0x7f9ad412b780 2021-05-25T12:02:43.400 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.333+0000 7f9ac1ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:02:43.400 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.333+0000 7f9ac1ffb700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:02:43.400 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.333+0000 7f9ac1ffb700 1 -- 172.21.15.102:0/3558392646 >> v1:172.21.15.181:6789/0 conn(0x7f9ad412b780 legacy=0x7f9ad412bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:43.401 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.333+0000 7f9ac1ffb700 1 -- 172.21.15.102:0/3558392646 >> v1:172.21.15.181:6789/0 conn(0x7f9ad412b780 legacy=0x7f9ad412bb60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:02:43.401 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.340+0000 7f9adbf47700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:02:43.401 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.340+0000 7f9adbf47700 1 -- 172.21.15.102:0/3558392646 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f9ad4059730 con 0x7f9ad4027ce0 2021-05-25T12:02:43.401 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.340+0000 7f9ac1ffb700 10 client.23087.objecter ms_handle_connect 0x7f9ad4027ce0 2021-05-25T12:02:43.402 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.340+0000 7f9ac1ffb700 10 client.23087.objecter resend_mon_ops 2021-05-25T12:02:43.402 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.340+0000 7f9ac1ffb700 1 -- 172.21.15.102:0/3558392646 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1471128776 0 0) 0x7f9ad4059730 con 0x7f9ad4027ce0 2021-05-25T12:02:43.402 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.340+0000 7f9ac1ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:02:43.402 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.340+0000 7f9ac1ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:02:43.402 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.340+0000 7f9ac1ffb700 10 monclient(hunting): my global_id is 95158 2021-05-25T12:02:43.403 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.340+0000 7f9ac1ffb700 1 -- 172.21.15.102:0/3558392646 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ad4063940 con 0x7f9ad4027ce0 2021-05-25T12:02:43.403 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.340+0000 7f9ac1ffb700 1 -- 172.21.15.102:0/3558392646 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4117768405 0 0) 0x7f9ad4063940 con 0x7f9ad4027ce0 2021-05-25T12:02:43.403 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.340+0000 7f9ac1ffb700 1 -- 172.21.15.102:0/3558392646 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9ad4059730 con 0x7f9ad4027ce0 2021-05-25T12:02:43.403 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.340+0000 7f9ac1ffb700 1 -- 172.21.15.102:0/3558392646 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f9ac40053a0 con 0x7f9ad4027ce0 2021-05-25T12:02:43.403 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.340+0000 7f9ac1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:43.404 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.340+0000 7f9ac1ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T12:02:43.404 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.340+0000 7f9ac1ffb700 10 monclient: dump: 2021-05-25T12:02:43.404 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:43.404 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:43.404 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:43.405 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:43.405 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:43.405 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:43.405 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:43.406 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:43.406 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:43.406 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:43.406 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.341+0000 7f9ac1ffb700 1 -- 172.21.15.102:0/3558392646 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3208278212 0 0) 0x7f9ad4059730 con 0x7f9ad4027ce0 2021-05-25T12:02:43.406 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.341+0000 7f9ac1ffb700 1 -- 172.21.15.102:0/3558392646 --> v1:172.21.15.181:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f9ad406c9a0 con 0x7f9ad4027ce0 2021-05-25T12:02:43.407 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.341+0000 7f9ac1ffb700 1 -- 172.21.15.102:0/3558392646 <== mon.1 v1:172.21.15.181:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3940 (unknown 0 0 2758042646) 0x7f9ac40066b0 con 0x7f9ad4027ce0 2021-05-25T12:02:43.407 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.341+0000 7f9ac1ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:02:43.407 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.341+0000 7f9ac1ffb700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:02:43.407 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.341+0000 7f9ac1ffb700 1 -- 172.21.15.102:0/3558392646 >> v1:172.21.15.181:6789/0 conn(0x7f9ad4027ce0 legacy=0x7f9ad4148b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:43.407 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.341+0000 7f9ac1ffb700 1 -- 172.21.15.102:0/3558392646 >> v1:172.21.15.181:6789/0 conn(0x7f9ad4027ce0 legacy=0x7f9ad4148b70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:02:43.408 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:02:43.408 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:02:43.408 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:02:43.409 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "peon", 2021-05-25T12:02:43.409 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 30, 2021-05-25T12:02:43.409 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:02:43.409 INFO:teuthology.orchestra.run.smithi102.stdout: 0, 2021-05-25T12:02:43.410 INFO:teuthology.orchestra.run.smithi102.stdout: 1, 2021-05-25T12:02:43.410 INFO:teuthology.orchestra.run.smithi102.stdout: 2 2021-05-25T12:02:43.410 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:02:43.410 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 2, 2021-05-25T12:02:43.410 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:02:43.411 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:02:43.411 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:02:43.411 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:02:43.411 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:02:43.411 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:02:43.412 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:02:43.412 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:02:43.412 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:02:43.412 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:02:43.412 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:02:43.413 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:02:43.413 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:02:43.413 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:02:43.413 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:02:43.413 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:02:43.414 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:02:43.414 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:02:43.414 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:02:43.415 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:02:43.415 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:02:43.415 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:02:43.415 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:02:43.415 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:02:43.416 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:02:43.416 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:02:43.416 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:02:43.416 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:02:43.416 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:02:43.417 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:02:43.417 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:02:43.417 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:02:43.417 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:02:43.417 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:02:43.418 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:02:43.418 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:02:43.418 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:02:43.418 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:02:43.418 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:02:43.419 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:02:43.419 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:02:43.419 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:02:43.419 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:02:43.419 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:02:43.420 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:02:43.420 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:02:43.420 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:02:43.420 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:02:43.420 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:02:43.421 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:02:43.421 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:02:43.421 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:02:43.421 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:02:43.421 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:02:43.422 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:02:43.422 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:02:43.422 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:02:43.422 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:02:43.422 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:02:43.423 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:02:43.423 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:02:43.423 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:02:43.423 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:02:43.423 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:02:43.424 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:02:43.424 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:02:43.424 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:02:43.424 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:02:43.424 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:02:43.425 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:02:43.425 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:02:43.425 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:02:43.425 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:02:43.426 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:02:43.426 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:02:43.426 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:02:43.426 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:02:43.426 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:02:43.427 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:02:43.427 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:02:43.427 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:02:43.427 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:02:43.427 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:02:43.428 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:02:43.428 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:02:43.428 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:02:43.428 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:02:43.428 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:02:43.429 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:02:43.429 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:02:43.429 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:02:43.429 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:02:43.429 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:02:43.430 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:02:43.430 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:02:43.430 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:02:43.430 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:02:43.430 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:02:43.431 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:02:43.431 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:02:43.431 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:02:43.431 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:02:43.431 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:02:43.432 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:02:43.432 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:02:43.432 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:02:43.432 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:02:43.432 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:02:43.433 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:02:43.433 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:02:43.433 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:02:43.433 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:02:43.433 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:02:43.434 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:02:43.434 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:02:43.434 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:02:43.434 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:02:43.435 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:02:43.435 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": [ 2021-05-25T12:02:43.435 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:02:43.435 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:02:43.436 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:02:43.436 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 3 2021-05-25T12:02:43.436 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:02:43.436 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:02:43.436 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:02:43.437 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:02:43.437 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:02:43.437 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:02:43.437 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 4 2021-05-25T12:02:43.437 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:02:43.438 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:02:43.438 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:02:43.438 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:02:43.438 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:02:43.440 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.341+0000 7f9adbf47700 10 librados: watch_flush enter 2021-05-25T12:02:43.441 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.342+0000 7f9adbf47700 10 librados: watch_flush exit 2021-05-25T12:02:43.441 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.342+0000 7f9adbf47700 20 client.23087.objecter shutdown clearing up homeless session... 2021-05-25T12:02:43.441 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.342+0000 7f9adbf47700 10 client.23087.objecter successfully canceled tick 2021-05-25T12:02:43.441 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.342+0000 7f9adbf47700 1 -- 172.21.15.102:0/3558392646 >> v1:172.21.15.181:6812/29582 conn(0x7f9ab4033480 legacy=0x7f9ab4035940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:43.442 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.342+0000 7f9adbf47700 10 monclient: shutdown 2021-05-25T12:02:43.442 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.342+0000 7f9adbf47700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:02:43.442 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.342+0000 7f9adbf47700 1 -- 172.21.15.102:0/3558392646 >> v1:172.21.15.102:6790/0 conn(0x7f9ad4063de0 legacy=0x7f9ad413b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:43.442 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.342+0000 7f9ada4e4700 1 -- 172.21.15.102:0/3558392646 reap_dead start 2021-05-25T12:02:43.443 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.342+0000 7f9ada4e4700 1 -- 172.21.15.102:0/3558392646 reap_dead start 2021-05-25T12:02:43.443 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.342+0000 7f9adbf47700 1 -- 172.21.15.102:0/3558392646 shutdown_connections 2021-05-25T12:02:43.443 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.342+0000 7f9adbf47700 1 -- 172.21.15.102:0/3558392646 >> 172.21.15.102:0/3558392646 conn(0x7f9ad409bc30 msgr2=0x7f9ad4131d50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:02:43.443 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.342+0000 7f9adbf47700 1 -- 172.21.15.102:0/3558392646 shutdown_connections 2021-05-25T12:02:43.443 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.342+0000 7f9adbf47700 1 -- 172.21.15.102:0/3558392646 wait complete. 2021-05-25T12:02:43.444 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.342+0000 7f9adbf47700 1 librados: shutdown 2021-05-25T12:02:43.444 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2021-05-25T12:02:43.445 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2021-05-25T12:02:43.554 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.552+0000 7fd6c101b700 10 monclient: get_monmap_and_config 2021-05-25T12:02:43.555 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.552+0000 7fd6c101b700 10 monclient: build_initial_monmap 2021-05-25T12:02:43.555 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.552+0000 7fd6c101b700 10 monclient: monmap: 2021-05-25T12:02:43.555 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:02:43.555 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:43.556 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:02:43.553778+0000 2021-05-25T12:02:43.556 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:02:43.553778+0000 2021-05-25T12:02:43.556 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:02:43.556 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:43.557 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:02:43.557 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:02:43.557 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:02:43.557 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:43.557 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.553+0000 7fd6c101b700 1 Processor -- start 2021-05-25T12:02:43.558 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.553+0000 7fd6c101b700 1 -- start start 2021-05-25T12:02:43.558 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.553+0000 7fd6c101b700 10 monclient: init 2021-05-25T12:02:43.568 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.553+0000 7fd6c101b700 10 monclient: _reopen_session rank -1 2021-05-25T12:02:43.568 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.553+0000 7fd6c101b700 10 monclient: _add_conns ranks=[0,1,2] 2021-05-25T12:02:43.569 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.553+0000 7fd6c101b700 10 monclient(hunting): picked mon.noname-a con 0x7fd6bc093d90 addr v1:172.21.15.102:6789/0 2021-05-25T12:02:43.569 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.553+0000 7fd6c101b700 10 monclient(hunting): picked mon.noname-c con 0x7fd6bc132900 addr v1:172.21.15.181:6789/0 2021-05-25T12:02:43.569 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.553+0000 7fd6c101b700 10 monclient(hunting): picked mon.noname-b con 0x7fd6bc136410 addr v1:172.21.15.102:6790/0 2021-05-25T12:02:43.569 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.553+0000 7fd6c101b700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6bc091ed0 con 0x7fd6bc093d90 2021-05-25T12:02:43.569 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.553+0000 7fd6c101b700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6bc092170 con 0x7fd6bc132900 2021-05-25T12:02:43.570 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.553+0000 7fd6c101b700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6bc092410 con 0x7fd6bc136410 2021-05-25T12:02:43.570 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.553+0000 7fd6c101b700 10 monclient(hunting): _renew_subs 2021-05-25T12:02:43.570 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.553+0000 7fd6c101b700 10 monclient(hunting): authenticate will time out at 964.150904s 2021-05-25T12:02:43.570 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.554+0000 7fd6bb59e700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fd6bc136410 0x7fd6bc138800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:55142/0 (socket says 172.21.15.102:55142) 2021-05-25T12:02:43.571 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.554+0000 7fd6bad9d700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fd6bc093d90 0x7fd6bc0691c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60260/0 (socket says 172.21.15.102:60260) 2021-05-25T12:02:43.571 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.554+0000 7fd6bb59e700 1 -- 172.21.15.102:0/4018048955 learned_addr learned my addr 172.21.15.102:0/4018048955 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:02:43.571 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.554+0000 7fd6b9d9b700 1 -- 172.21.15.102:0/4018048955 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2043372543 0 0) 0x7fd6bc091ed0 con 0x7fd6bc093d90 2021-05-25T12:02:43.571 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.554+0000 7fd6b9d9b700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:02:43.571 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.554+0000 7fd6b9d9b700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:02:43.572 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.554+0000 7fd6b9d9b700 10 monclient(hunting): my global_id is 174198 2021-05-25T12:02:43.572 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.554+0000 7fd6b9d9b700 1 -- 172.21.15.102:0/4018048955 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6a8003270 con 0x7fd6bc093d90 2021-05-25T12:02:43.572 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.554+0000 7fd6b9d9b700 1 -- 172.21.15.102:0/4018048955 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 34612049 0 0) 0x7fd6bc092410 con 0x7fd6bc136410 2021-05-25T12:02:43.572 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.554+0000 7fd6b9d9b700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:02:43.572 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.554+0000 7fd6b9d9b700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:02:43.573 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.554+0000 7fd6b9d9b700 10 monclient(hunting): my global_id is 23090 2021-05-25T12:02:43.573 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.554+0000 7fd6b9d9b700 1 -- 172.21.15.102:0/4018048955 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6bc091ed0 con 0x7fd6bc136410 2021-05-25T12:02:43.573 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.554+0000 7fd6b9d9b700 1 -- 172.21.15.102:0/4018048955 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4115634008 0 0) 0x7fd6a8003270 con 0x7fd6bc093d90 2021-05-25T12:02:43.573 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.554+0000 7fd6b9d9b700 1 -- 172.21.15.102:0/4018048955 >> v1:172.21.15.102:6790/0 conn(0x7fd6bc136410 legacy=0x7fd6bc138800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:43.574 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.554+0000 7fd6b9d9b700 1 -- 172.21.15.102:0/4018048955 >> v1:172.21.15.181:6789/0 conn(0x7fd6bc132900 legacy=0x7fd6bc134cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:43.574 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.554+0000 7fd6b9d9b700 10 monclient: _finish_hunting 0 2021-05-25T12:02:43.574 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.554+0000 7fd6b9d9b700 1 monclient: found mon.noname-a 2021-05-25T12:02:43.574 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.554+0000 7fd6b9d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:02:43.574 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.554+0000 7fd6b9d9b700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T12:02:43.575 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.554+0000 7fd6b9d9b700 1 -- 172.21.15.102:0/4018048955 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6bc094170 con 0x7fd6bc093d90 2021-05-25T12:02:43.575 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.554+0000 7fd6b9d9b700 10 monclient: _finish_auth 0 2021-05-25T12:02:43.575 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.554+0000 7fd6b9d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:02:43.575 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.554+0000 7fd6b9d9b700 1 -- 172.21.15.102:0/4018048955 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fd6b0001610 con 0x7fd6bc093d90 2021-05-25T12:02:43.575 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.554+0000 7fd6c101b700 5 monclient: authenticate success, global_id 174198 2021-05-25T12:02:43.576 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.554+0000 7fd6c101b700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:02:43.576 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.554+0000 7fd6b9d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:43.576 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.554+0000 7fd6b9d9b700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:02:43.576 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.554+0000 7fd6b9d9b700 10 monclient: dump: 2021-05-25T12:02:43.577 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:43.577 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:43.577 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:43.577 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:43.577 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:43.578 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:43.578 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:43.578 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:43.578 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:43.578 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:43.579 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.554+0000 7fd6c101b700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:02:43.579 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.554+0000 7fd6b9d9b700 1 -- 172.21.15.102:0/4018048955 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd6b0001830 con 0x7fd6bc093d90 2021-05-25T12:02:43.579 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.554+0000 7fd6b9d9b700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:02:43.579 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.554+0000 7fd6c101b700 10 monclient: get_monmap_and_config success 2021-05-25T12:02:43.579 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.554+0000 7fd6b9d9b700 1 -- 172.21.15.102:0/4018048955 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fd6b0001bc0 con 0x7fd6bc093d90 2021-05-25T12:02:43.580 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.554+0000 7fd6c101b700 4 set_mon_vals no callback set 2021-05-25T12:02:43.580 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.554+0000 7fd6b9d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:43.580 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.554+0000 7fd6b9d9b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:02:43.580 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.554+0000 7fd6b9d9b700 10 monclient: dump: 2021-05-25T12:02:43.581 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:43.581 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:43.581 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:43.581 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:43.581 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:43.582 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:43.582 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:43.582 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:43.582 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:43.582 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:43.583 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.555+0000 7fd6c101b700 10 monclient: shutdown 2021-05-25T12:02:43.583 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.555+0000 7fd6c101b700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:02:43.583 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.555+0000 7fd6c101b700 1 -- 172.21.15.102:0/4018048955 >> v1:172.21.15.102:6789/0 conn(0x7fd6bc093d90 legacy=0x7fd6bc0691c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:43.583 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.555+0000 7fd6c101b700 1 -- 172.21.15.102:0/4018048955 shutdown_connections 2021-05-25T12:02:43.584 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.555+0000 7fd6c101b700 1 -- 172.21.15.102:0/4018048955 >> 172.21.15.102:0/4018048955 conn(0x7fd6bc09b370 msgr2=0x7fd6bc097430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:02:43.584 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.555+0000 7fd6c101b700 1 -- 172.21.15.102:0/4018048955 shutdown_connections 2021-05-25T12:02:43.584 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.555+0000 7fd6c101b700 1 -- 172.21.15.102:0/4018048955 wait complete. 2021-05-25T12:02:43.584 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.555+0000 7fd6c101b700 10 monclient: build_initial_monmap 2021-05-25T12:02:43.584 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.555+0000 7fd6c101b700 10 monclient: monmap: 2021-05-25T12:02:43.585 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:02:43.585 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:02:43.585 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:02:43.585 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:02:43.585 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:02:43.586 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:02:43.586 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:02:43.586 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:02:43.586 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:02:43.587 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:43.587 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.556+0000 7fd6c101b700 1 librados: starting msgr at 2021-05-25T12:02:43.587 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.556+0000 7fd6c101b700 1 librados: starting objecter 2021-05-25T12:02:43.587 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.556+0000 7fd6c101b700 1 Processor -- start 2021-05-25T12:02:43.588 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.556+0000 7fd6c101b700 1 -- start start 2021-05-25T12:02:43.588 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.556+0000 7fd6c101b700 1 librados: setting wanted keys 2021-05-25T12:02:43.588 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.556+0000 7fd6c101b700 1 librados: calling monclient init 2021-05-25T12:02:43.588 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.556+0000 7fd6c101b700 10 monclient: init 2021-05-25T12:02:43.588 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.556+0000 7fd6c101b700 10 monclient: _reopen_session rank -1 2021-05-25T12:02:43.589 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.556+0000 7fd6c101b700 10 monclient: _add_conns ranks=[2,0,1] 2021-05-25T12:02:43.589 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.556+0000 7fd6c101b700 10 monclient(hunting): picked mon.noname-c con 0x7fd6bc093d90 addr v1:172.21.15.102:6790/0 2021-05-25T12:02:43.589 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.556+0000 7fd6c101b700 10 monclient(hunting): picked mon.noname-a con 0x7fd6bc132900 addr v1:172.21.15.102:6789/0 2021-05-25T12:02:43.589 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.556+0000 7fd6c101b700 10 monclient(hunting): picked mon.noname-b con 0x7fd6bc136410 addr v1:172.21.15.181:6789/0 2021-05-25T12:02:43.589 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.556+0000 7fd6c101b700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6bc13a4b0 con 0x7fd6bc132900 2021-05-25T12:02:43.590 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.556+0000 7fd6c101b700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6bc0694c0 con 0x7fd6bc136410 2021-05-25T12:02:43.590 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.556+0000 7fd6c101b700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6bc094420 con 0x7fd6bc093d90 2021-05-25T12:02:43.590 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.556+0000 7fd6c101b700 10 monclient(hunting): _renew_subs 2021-05-25T12:02:43.590 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.556+0000 7fd6c101b700 10 monclient(hunting): authenticate will time out at 964.153627s 2021-05-25T12:02:43.591 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.556+0000 7fd6bad9d700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fd6bc093d90 0x7fd6bc13f700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:55144/0 (socket says 172.21.15.102:55144) 2021-05-25T12:02:43.591 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.556+0000 7fd6bad9d700 1 -- 172.21.15.102:0/415213794 learned_addr learned my addr 172.21.15.102:0/415213794 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:02:43.591 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.556+0000 7fd6ba59c700 1 --1- 172.21.15.102:0/415213794 >> v1:172.21.15.102:6789/0 conn(0x7fd6bc132900 0x7fd6bc144320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60268/0 (socket says 172.21.15.102:60268) 2021-05-25T12:02:43.591 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.556+0000 7fd6a2ffd700 10 client.?.objecter ms_handle_connect 0x7fd6bc132900 2021-05-25T12:02:43.591 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.556+0000 7fd6a2ffd700 10 client.?.objecter resend_mon_ops 2021-05-25T12:02:43.592 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.556+0000 7fd6a2ffd700 10 client.?.objecter ms_handle_connect 0x7fd6bc093d90 2021-05-25T12:02:43.592 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.556+0000 7fd6a2ffd700 10 client.?.objecter resend_mon_ops 2021-05-25T12:02:43.592 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.556+0000 7fd6a2ffd700 1 -- 172.21.15.102:0/415213794 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3494876663 0 0) 0x7fd6bc13a4b0 con 0x7fd6bc132900 2021-05-25T12:02:43.592 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.556+0000 7fd6a2ffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:02:43.593 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.556+0000 7fd6a2ffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:02:43.593 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.556+0000 7fd6a2ffd700 10 monclient(hunting): my global_id is 174201 2021-05-25T12:02:43.593 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.556+0000 7fd6a2ffd700 1 -- 172.21.15.102:0/415213794 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd69c004c10 con 0x7fd6bc132900 2021-05-25T12:02:43.593 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.556+0000 7fd6a2ffd700 1 -- 172.21.15.102:0/415213794 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 443471229 0 0) 0x7fd6bc094420 con 0x7fd6bc093d90 2021-05-25T12:02:43.593 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.556+0000 7fd6a2ffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:02:43.594 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.556+0000 7fd6a2ffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:02:43.594 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.556+0000 7fd6a2ffd700 10 monclient(hunting): my global_id is 23093 2021-05-25T12:02:43.594 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.556+0000 7fd6a2ffd700 1 -- 172.21.15.102:0/415213794 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6bc13a4b0 con 0x7fd6bc093d90 2021-05-25T12:02:43.594 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.557+0000 7fd6a2ffd700 10 client.?.objecter ms_handle_connect 0x7fd6bc136410 2021-05-25T12:02:43.595 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.557+0000 7fd6a2ffd700 10 client.?.objecter resend_mon_ops 2021-05-25T12:02:43.595 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.557+0000 7fd6a2ffd700 1 -- 172.21.15.102:0/415213794 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1587197337 0 0) 0x7fd69c004c10 con 0x7fd6bc132900 2021-05-25T12:02:43.595 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.557+0000 7fd6a2ffd700 1 -- 172.21.15.102:0/415213794 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd6bc094420 con 0x7fd6bc132900 2021-05-25T12:02:43.595 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.557+0000 7fd6a2ffd700 1 -- 172.21.15.102:0/415213794 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fd6a4001900 con 0x7fd6bc132900 2021-05-25T12:02:43.595 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.557+0000 7fd6a2ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:43.596 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.557+0000 7fd6a2ffd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:02:43.596 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.557+0000 7fd6a2ffd700 10 monclient(hunting): dump: 2021-05-25T12:02:43.596 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:43.596 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:43.596 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:43.597 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:43.597 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:43.597 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:43.597 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:43.598 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:43.598 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:43.598 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:43.598 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.557+0000 7fd6a2ffd700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:02:43.599 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.557+0000 7fd6a2ffd700 1 -- 172.21.15.102:0/415213794 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2135462120 0 0) 0x7fd6bc13a4b0 con 0x7fd6bc093d90 2021-05-25T12:02:43.599 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.557+0000 7fd6a2ffd700 1 -- 172.21.15.102:0/415213794 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd69c004c10 con 0x7fd6bc093d90 2021-05-25T12:02:43.599 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.557+0000 7fd6a2ffd700 1 -- 172.21.15.102:0/415213794 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fd6b0001c00 con 0x7fd6bc093d90 2021-05-25T12:02:43.599 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.557+0000 7fd6a2ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:43.599 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.557+0000 7fd6a2ffd700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2021-05-25T12:02:43.600 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.557+0000 7fd6a2ffd700 10 monclient(hunting): dump: 2021-05-25T12:02:43.600 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:43.600 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:43.600 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:43.600 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:43.601 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:43.601 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:43.601 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:43.601 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:43.601 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:43.602 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:43.602 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.557+0000 7fd6a2ffd700 1 -- 172.21.15.102:0/415213794 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3282017357 0 0) 0x7fd6bc0694c0 con 0x7fd6bc136410 2021-05-25T12:02:43.602 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.557+0000 7fd6a2ffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:02:43.602 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.557+0000 7fd6a2ffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:02:43.603 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.557+0000 7fd6a2ffd700 10 monclient(hunting): my global_id is 95164 2021-05-25T12:02:43.603 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.557+0000 7fd6a2ffd700 1 -- 172.21.15.102:0/415213794 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6bc13a4b0 con 0x7fd6bc136410 2021-05-25T12:02:43.603 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.557+0000 7fd6a2ffd700 1 -- 172.21.15.102:0/415213794 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2408263257 0 0) 0x7fd6bc094420 con 0x7fd6bc132900 2021-05-25T12:02:43.603 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.557+0000 7fd6a2ffd700 1 -- 172.21.15.102:0/415213794 >> v1:172.21.15.102:6790/0 conn(0x7fd6bc093d90 legacy=0x7fd6bc13f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:43.603 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.557+0000 7fd6a2ffd700 1 -- 172.21.15.102:0/415213794 >> v1:172.21.15.181:6789/0 conn(0x7fd6bc136410 legacy=0x7fd6bc147a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:43.604 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.557+0000 7fd6a2ffd700 10 monclient: _finish_hunting 0 2021-05-25T12:02:43.604 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.557+0000 7fd6a2ffd700 1 monclient: found mon.a 2021-05-25T12:02:43.604 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.557+0000 7fd6a2ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:02:43.604 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.557+0000 7fd6a2ffd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:02:43.605 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.557+0000 7fd6a2ffd700 1 -- 172.21.15.102:0/415213794 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6bc0964d0 con 0x7fd6bc132900 2021-05-25T12:02:43.605 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.557+0000 7fd6a2ffd700 10 monclient: _finish_auth 0 2021-05-25T12:02:43.605 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.557+0000 7fd6a2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:02:43.605 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.557+0000 7fd6c101b700 5 monclient: authenticate success, global_id 174201 2021-05-25T12:02:43.605 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.557+0000 7fd6c101b700 10 monclient: _renew_subs 2021-05-25T12:02:43.606 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.557+0000 7fd6c101b700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:02:43.606 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.557+0000 7fd6c101b700 1 -- 172.21.15.102:0/415213794 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6bc13fe10 con 0x7fd6bc132900 2021-05-25T12:02:43.606 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.557+0000 7fd6c101b700 10 client.174201.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:02:43.606 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.557+0000 7fd6c101b700 10 monclient: _renew_subs 2021-05-25T12:02:43.607 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.557+0000 7fd6c101b700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:02:43.607 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.557+0000 7fd6c101b700 1 -- 172.21.15.102:0/415213794 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6bc067e30 con 0x7fd6bc132900 2021-05-25T12:02:43.607 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.557+0000 7fd6c101b700 1 librados: init done 2021-05-25T12:02:43.607 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.557+0000 7fd6a2ffd700 1 -- 172.21.15.102:0/415213794 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd6a4001c70 con 0x7fd6bc132900 2021-05-25T12:02:43.607 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.557+0000 7fd6a2ffd700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:02:43.608 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.557+0000 7fd6a2ffd700 1 -- 172.21.15.102:0/415213794 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fd6a4002450 con 0x7fd6bc132900 2021-05-25T12:02:43.608 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.557+0000 7fd6a2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:43.608 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.557+0000 7fd6b959a700 4 set_mon_vals no callback set 2021-05-25T12:02:43.608 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.557+0000 7fd6a2ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:02:43.608 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.557+0000 7fd6a2ffd700 10 monclient: dump: 2021-05-25T12:02:43.609 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:43.609 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:43.609 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:43.609 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:43.610 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:43.610 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:43.610 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:43.610 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:43.610 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:43.611 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:43.611 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.558+0000 7fd6a2ffd700 1 -- 172.21.15.102:0/415213794 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7fd6a4002630 con 0x7fd6bc132900 2021-05-25T12:02:43.611 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.558+0000 7fd6c101b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:02:43.611 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.558+0000 7fd6c101b700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:02:43.611 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.558+0000 7fd6c101b700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:02:43.612 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.558+0000 7fd6c101b700 1 -- 172.21.15.102:0/415213794 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd694004db0 con 0x7fd6bc132900 2021-05-25T12:02:43.612 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.558+0000 7fd6a2ffd700 1 -- 172.21.15.102:0/415213794 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(81..81 src has 1..81) v4 ==== 6162+0+0 (unknown 275132006 0 0) 0x7fd6a403d7f0 con 0x7fd6bc132900 2021-05-25T12:02:43.612 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.558+0000 7fd6a2ffd700 10 client.174201.objecter ms_dispatch 0x7fd6bc001550 osd_map(81..81 src has 1..81) v4 2021-05-25T12:02:43.612 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.558+0000 7fd6a2ffd700 3 client.174201.objecter handle_osd_map got epochs [81,81] > 0 2021-05-25T12:02:43.612 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.558+0000 7fd6a2ffd700 3 client.174201.objecter handle_osd_map decoding full epoch 81 2021-05-25T12:02:43.613 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.558+0000 7fd6a2ffd700 20 client.174201.objecter dump_active .. 0 homeless 2021-05-25T12:02:43.613 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.558+0000 7fd6bad9d700 10 monclient: get_auth_request con 0x7fd69c0333e0 auth_method 0 2021-05-25T12:02:43.613 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.558+0000 7fd6a2ffd700 10 client.174201.objecter ms_handle_connect 0x7fd69c0333e0 2021-05-25T12:02:43.613 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.562+0000 7fd6a2ffd700 1 -- 172.21.15.102:0/415213794 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7fd6a4012b30 con 0x7fd6bc132900 2021-05-25T12:02:43.614 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.562+0000 7fd6a2ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:02:43.614 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.562+0000 7fd6a2ffd700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:02:43.780 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.779+0000 7fd6c101b700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2021-05-25T12:02:43.781 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.779+0000 7fd6c101b700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2021-05-25T12:02:43.781 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.779+0000 7fd6c101b700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:02:43.781 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.779+0000 7fd6c101b700 1 -- 172.21.15.102:0/415213794 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7fd694005870 con 0x7fd6bc132900 2021-05-25T12:02:43.798 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.796+0000 7fd6a2ffd700 1 -- 172.21.15.102:0/415213794 <== mon.0 v1:172.21.15.102:6789/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7fd6a4012950 con 0x7fd6bc132900 2021-05-25T12:02:43.798 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.796+0000 7fd6a2ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2021-05-25T12:02:43.798 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.796+0000 7fd6a2ffd700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:02:43.801 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.798+0000 7fd6c101b700 10 librados: watch_flush enter 2021-05-25T12:02:43.801 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.798+0000 7fd6c101b700 10 librados: watch_flush exit 2021-05-25T12:02:43.801 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.798+0000 7fd6c101b700 20 client.174201.objecter shutdown clearing up homeless session... 2021-05-25T12:02:43.802 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.798+0000 7fd6c101b700 10 client.174201.objecter successfully canceled tick 2021-05-25T12:02:43.802 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.798+0000 7fd6c101b700 1 -- 172.21.15.102:0/415213794 >> v1:172.21.15.181:6812/29582 conn(0x7fd69c0333e0 legacy=0x7fd69c0358a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:43.802 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.798+0000 7fd6c101b700 10 monclient: shutdown 2021-05-25T12:02:43.802 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.798+0000 7fd6c101b700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:02:43.803 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.798+0000 7fd6c101b700 1 -- 172.21.15.102:0/415213794 >> v1:172.21.15.102:6789/0 conn(0x7fd6bc132900 legacy=0x7fd6bc144320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:43.803 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.798+0000 7fd6c101b700 1 -- 172.21.15.102:0/415213794 shutdown_connections 2021-05-25T12:02:43.803 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.798+0000 7fd6c101b700 1 -- 172.21.15.102:0/415213794 >> 172.21.15.102:0/415213794 conn(0x7fd6bc09b370 msgr2=0x7fd6bc138d10 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:02:43.803 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.798+0000 7fd6c101b700 1 -- 172.21.15.102:0/415213794 shutdown_connections 2021-05-25T12:02:43.804 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.798+0000 7fd6c101b700 1 -- 172.21.15.102:0/415213794 wait complete. 2021-05-25T12:02:43.804 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:43.798+0000 7fd6c101b700 1 librados: shutdown 2021-05-25T12:02:43.805 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:02:43.799+0000 7fd738a30700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:02:43.805 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:02:43.799+0000 7fd738a30700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=25,config=1,health=12,kv=5,logm=57} crc {auth=3829669040,config=1228700967,health=3826145356,kv=2716774850,logm=2383311228}) 2021-05-25T12:02:43.806 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:02:43.799+0000 7fd738a30700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=24,config=2,health=11,kv=6,logm=57} crc {auth=2451630166,config=3140514009,health=65985201,kv=1184736763,logm=1853365132}) 2021-05-25T12:02:43.806 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:02:43.799+0000 7fd738a30700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:02:43.806 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:02:43.799+0000 7fd738a30700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=25,config=1,health=12,kv=5,logm=57} crc {auth=3829669040,config=1228700967,health=3826145356,kv=2716774850,logm=2383311228}) 2021-05-25T12:02:43.806 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:02:43.799+0000 7fd738a30700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=25,config=2,health=11,kv=6,logm=56} crc {auth=444469487,config=3140514009,health=2945087396,kv=1184736763,logm=2780684189}) 2021-05-25T12:02:43.807 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:02:43.802+0000 7fd738a30700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:02:43.807 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:02:43.802+0000 7fd738a30700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1073998796}) 2021-05-25T12:02:43.807 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:02:43.802+0000 7fd738a30700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3601542485}) 2021-05-25T12:02:43.807 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:02:43.802+0000 7fd738a30700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:02:43.808 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:02:43.802+0000 7fd738a30700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1073998796}) 2021-05-25T12:02:43.808 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:02:43.802+0000 7fd738a30700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2998181480}) 2021-05-25T12:02:43.808 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:02:43.804+0000 7fd738a30700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:02:43.808 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:02:43.804+0000 7fd738a30700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=30,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=4,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=19,osdmap=16} crc {logm=797878744,mdsmap=1531151339,mgr=1723869108,mgr_command_descs=3824783157,mgr_metadata=1267308364,mgrstat=959900461,mon_config_key=4118394475,monmap=3809614708,osd_metadata=725478305,osd_pg_creating=234821021,osd_snap=3072014463,osdmap=1117649566}) 2021-05-25T12:02:43.808 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:02:43.804+0000 7fd738a30700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=32,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=4,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=19,osdmap=14} crc {logm=287241412,mdsmap=1531151339,mgr=2996579194,mgr_command_descs=3824783157,mgr_metadata=1267308364,mgrstat=959900461,mon_config_key=4118394475,monmap=287184522,osd_metadata=2149013858,osd_pg_creating=234821021,osd_snap=3072014463,osdmap=2574761753}) 2021-05-25T12:02:43.809 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:02:43.804+0000 7fd738a30700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:02:43.809 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:02:43.804+0000 7fd738a30700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=30,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=4,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=19,osdmap=16} crc {logm=797878744,mdsmap=1531151339,mgr=1723869108,mgr_command_descs=3824783157,mgr_metadata=1267308364,mgrstat=959900461,mon_config_key=4118394475,monmap=3809614708,osd_metadata=725478305,osd_pg_creating=234821021,osd_snap=3072014463,osdmap=1117649566}) 2021-05-25T12:02:43.809 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:02:43.804+0000 7fd738a30700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=31,mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=4,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=16,osdmap=19} crc {logm=3412134790,mdsmap=3140514009,mgr=2996579194,mgr_command_descs=3824783157,mgr_metadata=1267308364,mgrstat=959900461,mon_config_key=4118394475,monmap=1140339822,osd_metadata=725478305,osd_pg_creating=234821021,osd_snap=2496675215,osdmap=577782753}) 2021-05-25T12:02:43.811 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:02:43.805+0000 7fd738a30700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:02:43.811 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:02:43.805+0000 7fd738a30700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3548084064}) 2021-05-25T12:02:43.811 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:02:43.805+0000 7fd738a30700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3772438645}) 2021-05-25T12:02:43.811 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:02:43.805+0000 7fd738a30700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:02:43.812 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:02:43.805+0000 7fd738a30700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3548084064}) 2021-05-25T12:02:43.812 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:02:43.805+0000 7fd738a30700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2688671406}) 2021-05-25T12:02:43.812 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:02:43.809+0000 7fd738a30700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:02:43.812 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:02:43.809+0000 7fd738a30700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=16} crc {osdmap=3328498707}) 2021-05-25T12:02:43.813 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:02:43.809+0000 7fd738a30700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=15} crc {osdmap=4028413364}) 2021-05-25T12:02:43.814 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2021-05-25T12:02:44.815 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2021-05-25T12:02:44.977 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2021-05-25T12:02:45.094 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.092+0000 7f8b6ce9c700 10 monclient: get_monmap_and_config 2021-05-25T12:02:45.094 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.092+0000 7f8b6ce9c700 10 monclient: build_initial_monmap 2021-05-25T12:02:45.094 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.092+0000 7f8b6ce9c700 10 monclient: monmap: 2021-05-25T12:02:45.095 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:02:45.095 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:45.095 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:02:45.093251+0000 2021-05-25T12:02:45.095 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:02:45.093251+0000 2021-05-25T12:02:45.095 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:02:45.096 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:45.096 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:02:45.096 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:02:45.096 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:02:45.096 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:45.097 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.092+0000 7f8b6ce9c700 1 Processor -- start 2021-05-25T12:02:45.097 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.092+0000 7f8b6ce9c700 1 -- start start 2021-05-25T12:02:45.097 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.092+0000 7f8b6ce9c700 10 monclient: init 2021-05-25T12:02:45.108 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.093+0000 7f8b6ce9c700 10 monclient: _reopen_session rank -1 2021-05-25T12:02:45.108 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.093+0000 7f8b6ce9c700 10 monclient: _add_conns ranks=[2,1,0] 2021-05-25T12:02:45.108 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.093+0000 7f8b6ce9c700 10 monclient(hunting): picked mon.noname-b con 0x7f8b68093fc0 addr v1:172.21.15.102:6790/0 2021-05-25T12:02:45.108 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.093+0000 7f8b6ce9c700 10 monclient(hunting): picked mon.noname-c con 0x7f8b680692d0 addr v1:172.21.15.181:6789/0 2021-05-25T12:02:45.109 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.093+0000 7f8b6ce9c700 10 monclient(hunting): picked mon.noname-a con 0x7f8b6812f130 addr v1:172.21.15.102:6789/0 2021-05-25T12:02:45.109 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.093+0000 7f8b6ce9c700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b68097160 con 0x7f8b6812f130 2021-05-25T12:02:45.109 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.093+0000 7f8b6ce9c700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b68092fe0 con 0x7f8b680692d0 2021-05-25T12:02:45.109 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.093+0000 7f8b6ce9c700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b68093280 con 0x7f8b68093fc0 2021-05-25T12:02:45.109 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.093+0000 7f8b6ce9c700 10 monclient(hunting): _renew_subs 2021-05-25T12:02:45.110 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.093+0000 7f8b6ce9c700 10 monclient(hunting): authenticate will time out at 965.690528s 2021-05-25T12:02:45.110 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.093+0000 7f8b6659c700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f8b68093fc0 0x7f8b68096340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:55156/0 (socket says 172.21.15.102:55156) 2021-05-25T12:02:45.110 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.093+0000 7f8b6659c700 1 -- 172.21.15.102:0/837552807 learned_addr learned my addr 172.21.15.102:0/837552807 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:02:45.110 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.093+0000 7f8b6559a700 1 -- 172.21.15.102:0/837552807 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4278303837 0 0) 0x7f8b68097160 con 0x7f8b6812f130 2021-05-25T12:02:45.111 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.093+0000 7f8b6559a700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:02:45.111 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.093+0000 7f8b6559a700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:02:45.111 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.093+0000 7f8b6559a700 10 monclient(hunting): my global_id is 174210 2021-05-25T12:02:45.111 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.093+0000 7f8b6559a700 1 -- 172.21.15.102:0/837552807 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b4c003270 con 0x7f8b6812f130 2021-05-25T12:02:45.111 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.093+0000 7f8b6559a700 1 -- 172.21.15.102:0/837552807 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3316941697 0 0) 0x7f8b68093280 con 0x7f8b68093fc0 2021-05-25T12:02:45.112 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.093+0000 7f8b6559a700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:02:45.112 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.093+0000 7f8b6559a700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:02:45.112 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.093+0000 7f8b6559a700 10 monclient(hunting): my global_id is 23096 2021-05-25T12:02:45.112 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.093+0000 7f8b6559a700 1 -- 172.21.15.102:0/837552807 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b68097160 con 0x7f8b68093fc0 2021-05-25T12:02:45.113 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.094+0000 7f8b6559a700 1 -- 172.21.15.102:0/837552807 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3857011116 0 0) 0x7f8b4c003270 con 0x7f8b6812f130 2021-05-25T12:02:45.113 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.094+0000 7f8b6559a700 1 -- 172.21.15.102:0/837552807 >> v1:172.21.15.102:6790/0 conn(0x7f8b68093fc0 legacy=0x7f8b68096340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:45.113 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.094+0000 7f8b6559a700 1 -- 172.21.15.102:0/837552807 >> v1:172.21.15.181:6789/0 conn(0x7f8b680692d0 legacy=0x7f8b68096a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:45.113 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.094+0000 7f8b6559a700 10 monclient: _finish_hunting 0 2021-05-25T12:02:45.114 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.094+0000 7f8b6559a700 1 monclient: found mon.noname-a 2021-05-25T12:02:45.114 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.094+0000 7f8b6559a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:02:45.114 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.094+0000 7f8b6559a700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T12:02:45.114 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.094+0000 7f8b6559a700 1 -- 172.21.15.102:0/837552807 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b68093520 con 0x7f8b6812f130 2021-05-25T12:02:45.114 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.094+0000 7f8b6559a700 10 monclient: _finish_auth 0 2021-05-25T12:02:45.115 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.094+0000 7f8b6559a700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:02:45.115 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.094+0000 7f8b6ce9c700 5 monclient: authenticate success, global_id 174210 2021-05-25T12:02:45.115 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.094+0000 7f8b6559a700 1 -- 172.21.15.102:0/837552807 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f8b58001990 con 0x7f8b6812f130 2021-05-25T12:02:45.115 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.094+0000 7f8b6ce9c700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:02:45.115 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.094+0000 7f8b6559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:45.116 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.094+0000 7f8b6559a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:02:45.116 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.094+0000 7f8b6559a700 10 monclient: dump: 2021-05-25T12:02:45.116 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:45.116 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:45.116 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:45.117 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:45.117 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:45.117 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:45.117 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:45.118 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:45.118 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:45.118 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:45.118 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.094+0000 7f8b6ce9c700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:02:45.118 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.094+0000 7f8b6559a700 1 -- 172.21.15.102:0/837552807 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8b58001c30 con 0x7f8b6812f130 2021-05-25T12:02:45.119 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.094+0000 7f8b6559a700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:02:45.119 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.094+0000 7f8b6559a700 1 -- 172.21.15.102:0/837552807 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f8b58001fc0 con 0x7f8b6812f130 2021-05-25T12:02:45.119 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.094+0000 7f8b6ce9c700 10 monclient: get_monmap_and_config success 2021-05-25T12:02:45.119 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.094+0000 7f8b6ce9c700 4 set_mon_vals no callback set 2021-05-25T12:02:45.119 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.094+0000 7f8b6559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:45.120 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.094+0000 7f8b6559a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:02:45.120 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.094+0000 7f8b6559a700 10 monclient: dump: 2021-05-25T12:02:45.120 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:45.120 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:45.120 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:45.121 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:45.121 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:45.121 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:45.121 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:45.121 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:45.122 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:45.122 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:45.122 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.094+0000 7f8b6ce9c700 10 monclient: shutdown 2021-05-25T12:02:45.122 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.094+0000 7f8b6ce9c700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:02:45.123 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.094+0000 7f8b6ce9c700 1 -- 172.21.15.102:0/837552807 >> v1:172.21.15.102:6789/0 conn(0x7f8b6812f130 legacy=0x7f8b68131520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:45.123 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.094+0000 7f8b6ce9c700 1 -- 172.21.15.102:0/837552807 shutdown_connections 2021-05-25T12:02:45.123 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.094+0000 7f8b6ce9c700 1 -- 172.21.15.102:0/837552807 >> 172.21.15.102:0/837552807 conn(0x7f8b6809b370 msgr2=0x7f8b68097430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:02:45.123 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.094+0000 7f8b6ce9c700 1 -- 172.21.15.102:0/837552807 shutdown_connections 2021-05-25T12:02:45.123 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.094+0000 7f8b6ce9c700 1 -- 172.21.15.102:0/837552807 wait complete. 2021-05-25T12:02:45.124 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.095+0000 7f8b6ce9c700 10 monclient: build_initial_monmap 2021-05-25T12:02:45.124 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.095+0000 7f8b6ce9c700 10 monclient: monmap: 2021-05-25T12:02:45.124 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:02:45.124 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:02:45.124 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:02:45.125 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:02:45.125 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:02:45.125 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:02:45.125 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:02:45.125 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:02:45.125 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:02:45.126 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:45.126 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.095+0000 7f8b6ce9c700 1 librados: starting msgr at 2021-05-25T12:02:45.126 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.095+0000 7f8b6ce9c700 1 librados: starting objecter 2021-05-25T12:02:45.126 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.095+0000 7f8b6ce9c700 1 Processor -- start 2021-05-25T12:02:45.127 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.095+0000 7f8b6ce9c700 1 -- start start 2021-05-25T12:02:45.127 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.095+0000 7f8b6ce9c700 1 librados: setting wanted keys 2021-05-25T12:02:45.127 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.095+0000 7f8b6ce9c700 1 librados: calling monclient init 2021-05-25T12:02:45.127 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.095+0000 7f8b6ce9c700 10 monclient: init 2021-05-25T12:02:45.127 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.095+0000 7f8b6ce9c700 10 monclient: _reopen_session rank -1 2021-05-25T12:02:45.128 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.095+0000 7f8b6ce9c700 10 monclient: _add_conns ranks=[0,2,1] 2021-05-25T12:02:45.128 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.095+0000 7f8b6ce9c700 10 monclient(hunting): picked mon.noname-a con 0x7f8b680692d0 addr v1:172.21.15.102:6789/0 2021-05-25T12:02:45.128 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.095+0000 7f8b6ce9c700 10 monclient(hunting): picked mon.noname-c con 0x7f8b68093fc0 addr v1:172.21.15.102:6790/0 2021-05-25T12:02:45.128 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.095+0000 7f8b6ce9c700 10 monclient(hunting): picked mon.noname-b con 0x7f8b6812f130 addr v1:172.21.15.181:6789/0 2021-05-25T12:02:45.128 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.095+0000 7f8b6ce9c700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b6812a2a0 con 0x7f8b680692d0 2021-05-25T12:02:45.129 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.096+0000 7f8b6ce9c700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b68116de0 con 0x7f8b6812f130 2021-05-25T12:02:45.129 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.096+0000 7f8b6ce9c700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b68117080 con 0x7f8b68093fc0 2021-05-25T12:02:45.129 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.096+0000 7f8b6ce9c700 10 monclient(hunting): _renew_subs 2021-05-25T12:02:45.129 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.096+0000 7f8b6ce9c700 10 monclient(hunting): authenticate will time out at 965.693027s 2021-05-25T12:02:45.129 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.096+0000 7f8b6659c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f8b680692d0 0x7f8b680674f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60284/0 (socket says 172.21.15.102:60284) 2021-05-25T12:02:45.130 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.096+0000 7f8b6659c700 1 -- 172.21.15.102:0/2790337663 learned_addr learned my addr 172.21.15.102:0/2790337663 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:02:45.130 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.096+0000 7f8b65d9b700 1 --1- 172.21.15.102:0/2790337663 >> v1:172.21.15.102:6790/0 conn(0x7f8b68093fc0 0x7f8b68067c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:55164/0 (socket says 172.21.15.102:55164) 2021-05-25T12:02:45.130 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.096+0000 7f8b567fc700 10 client.?.objecter ms_handle_connect 0x7f8b680692d0 2021-05-25T12:02:45.130 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.096+0000 7f8b567fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:02:45.131 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.096+0000 7f8b567fc700 10 client.?.objecter ms_handle_connect 0x7f8b68093fc0 2021-05-25T12:02:45.131 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.096+0000 7f8b567fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:02:45.131 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.096+0000 7f8b567fc700 10 client.?.objecter ms_handle_connect 0x7f8b6812f130 2021-05-25T12:02:45.131 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.096+0000 7f8b567fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:02:45.131 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.096+0000 7f8b567fc700 1 -- 172.21.15.102:0/2790337663 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1112270403 0 0) 0x7f8b6812a2a0 con 0x7f8b680692d0 2021-05-25T12:02:45.132 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.096+0000 7f8b567fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:02:45.132 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.096+0000 7f8b567fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:02:45.132 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.096+0000 7f8b567fc700 10 monclient(hunting): my global_id is 174213 2021-05-25T12:02:45.132 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.096+0000 7f8b567fc700 1 -- 172.21.15.102:0/2790337663 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b44004c10 con 0x7f8b680692d0 2021-05-25T12:02:45.133 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.096+0000 7f8b567fc700 1 -- 172.21.15.102:0/2790337663 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4023660302 0 0) 0x7f8b68117080 con 0x7f8b68093fc0 2021-05-25T12:02:45.133 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.096+0000 7f8b567fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:02:45.133 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.096+0000 7f8b567fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:02:45.133 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.096+0000 7f8b567fc700 10 monclient(hunting): my global_id is 23099 2021-05-25T12:02:45.133 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.096+0000 7f8b567fc700 1 -- 172.21.15.102:0/2790337663 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b6812a2a0 con 0x7f8b68093fc0 2021-05-25T12:02:45.134 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.096+0000 7f8b567fc700 1 -- 172.21.15.102:0/2790337663 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2418778794 0 0) 0x7f8b68116de0 con 0x7f8b6812f130 2021-05-25T12:02:45.134 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.096+0000 7f8b567fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:02:45.134 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.096+0000 7f8b567fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:02:45.134 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.096+0000 7f8b567fc700 10 monclient(hunting): my global_id is 95176 2021-05-25T12:02:45.135 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.096+0000 7f8b567fc700 1 -- 172.21.15.102:0/2790337663 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b68117080 con 0x7f8b6812f130 2021-05-25T12:02:45.135 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.096+0000 7f8b567fc700 1 -- 172.21.15.102:0/2790337663 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2789819205 0 0) 0x7f8b44004c10 con 0x7f8b680692d0 2021-05-25T12:02:45.135 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.096+0000 7f8b567fc700 1 -- 172.21.15.102:0/2790337663 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8b68116de0 con 0x7f8b680692d0 2021-05-25T12:02:45.135 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.096+0000 7f8b567fc700 1 -- 172.21.15.102:0/2790337663 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f8b5c001970 con 0x7f8b680692d0 2021-05-25T12:02:45.135 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.096+0000 7f8b567fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:45.136 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.096+0000 7f8b567fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:02:45.136 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.096+0000 7f8b567fc700 10 monclient(hunting): dump: 2021-05-25T12:02:45.136 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:45.136 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:45.136 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:45.137 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:45.137 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:45.137 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:45.137 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:45.138 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:45.138 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:45.138 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:45.138 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.096+0000 7f8b567fc700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:02:45.138 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.096+0000 7f8b567fc700 1 -- 172.21.15.102:0/2790337663 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1406481604 0 0) 0x7f8b6812a2a0 con 0x7f8b68093fc0 2021-05-25T12:02:45.139 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.096+0000 7f8b567fc700 1 -- 172.21.15.102:0/2790337663 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8b44004c10 con 0x7f8b68093fc0 2021-05-25T12:02:45.139 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.096+0000 7f8b567fc700 1 -- 172.21.15.102:0/2790337663 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f8b50001900 con 0x7f8b68093fc0 2021-05-25T12:02:45.139 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.096+0000 7f8b567fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:45.139 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.096+0000 7f8b567fc700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2021-05-25T12:02:45.139 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.096+0000 7f8b567fc700 10 monclient(hunting): dump: 2021-05-25T12:02:45.140 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:45.140 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:45.140 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:45.140 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:45.140 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:45.141 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:45.141 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:45.141 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:45.141 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:45.142 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:45.142 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.096+0000 7f8b567fc700 1 -- 172.21.15.102:0/2790337663 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2042974642 0 0) 0x7f8b68117080 con 0x7f8b6812f130 2021-05-25T12:02:45.142 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.096+0000 7f8b567fc700 1 -- 172.21.15.102:0/2790337663 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8b6812a2a0 con 0x7f8b6812f130 2021-05-25T12:02:45.142 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.096+0000 7f8b567fc700 1 -- 172.21.15.102:0/2790337663 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4220953992 0 0) 0x7f8b68116de0 con 0x7f8b680692d0 2021-05-25T12:02:45.142 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.096+0000 7f8b567fc700 1 -- 172.21.15.102:0/2790337663 >> v1:172.21.15.102:6790/0 conn(0x7f8b68093fc0 legacy=0x7f8b68067c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:45.143 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.096+0000 7f8b567fc700 1 -- 172.21.15.102:0/2790337663 >> v1:172.21.15.181:6789/0 conn(0x7f8b6812f130 legacy=0x7f8b68143620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:45.143 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.096+0000 7f8b567fc700 10 monclient: _finish_hunting 0 2021-05-25T12:02:45.143 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.096+0000 7f8b567fc700 1 monclient: found mon.a 2021-05-25T12:02:45.143 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.096+0000 7f8b567fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:02:45.143 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.096+0000 7f8b567fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:02:45.144 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.096+0000 7f8b567fc700 1 -- 172.21.15.102:0/2790337663 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b68117320 con 0x7f8b680692d0 2021-05-25T12:02:45.144 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.096+0000 7f8b567fc700 10 monclient: _finish_auth 0 2021-05-25T12:02:45.144 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.096+0000 7f8b567fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:02:45.144 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.096+0000 7f8b6ce9c700 5 monclient: authenticate success, global_id 174213 2021-05-25T12:02:45.145 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.096+0000 7f8b6ce9c700 10 monclient: _renew_subs 2021-05-25T12:02:45.145 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.096+0000 7f8b6ce9c700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:02:45.145 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.096+0000 7f8b6ce9c700 1 -- 172.21.15.102:0/2790337663 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b68129ed0 con 0x7f8b680692d0 2021-05-25T12:02:45.145 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.096+0000 7f8b6ce9c700 10 client.174213.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:02:45.145 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.096+0000 7f8b6ce9c700 10 monclient: _renew_subs 2021-05-25T12:02:45.146 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.096+0000 7f8b6ce9c700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:02:45.146 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.096+0000 7f8b6ce9c700 1 -- 172.21.15.102:0/2790337663 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b681160f0 con 0x7f8b680692d0 2021-05-25T12:02:45.146 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.096+0000 7f8b6ce9c700 1 librados: init done 2021-05-25T12:02:45.146 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.096+0000 7f8b567fc700 1 -- 172.21.15.102:0/2790337663 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8b5c004530 con 0x7f8b680692d0 2021-05-25T12:02:45.146 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.096+0000 7f8b567fc700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:02:45.147 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.096+0000 7f8b567fc700 1 -- 172.21.15.102:0/2790337663 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f8b5c0048c0 con 0x7f8b680692d0 2021-05-25T12:02:45.147 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.096+0000 7f8b567fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:45.147 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.096+0000 7f8b567fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:02:45.147 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.096+0000 7f8b64d99700 4 set_mon_vals no callback set 2021-05-25T12:02:45.147 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.096+0000 7f8b567fc700 10 monclient: dump: 2021-05-25T12:02:45.148 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:45.148 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:45.148 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:45.148 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:45.148 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:45.149 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:45.149 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:45.149 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:45.149 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:45.149 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:45.150 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.097+0000 7f8b567fc700 1 -- 172.21.15.102:0/2790337663 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f8b5c00dc60 con 0x7f8b680692d0 2021-05-25T12:02:45.150 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.097+0000 7f8b6ce9c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:02:45.150 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.097+0000 7f8b567fc700 1 -- 172.21.15.102:0/2790337663 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(83..83 src has 1..83) v4 ==== 7059+0+0 (unknown 555442269 0 0) 0x7f8b5c03dbf0 con 0x7f8b680692d0 2021-05-25T12:02:45.150 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.097+0000 7f8b6ce9c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:02:45.150 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.097+0000 7f8b567fc700 10 client.174213.objecter ms_dispatch 0x7f8b68001550 osd_map(83..83 src has 1..83) v4 2021-05-25T12:02:45.151 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.097+0000 7f8b567fc700 3 client.174213.objecter handle_osd_map got epochs [83,83] > 0 2021-05-25T12:02:45.151 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.097+0000 7f8b567fc700 3 client.174213.objecter handle_osd_map decoding full epoch 83 2021-05-25T12:02:45.151 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.097+0000 7f8b6ce9c700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:02:45.151 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.097+0000 7f8b6ce9c700 1 -- 172.21.15.102:0/2790337663 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b48004db0 con 0x7f8b680692d0 2021-05-25T12:02:45.152 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.097+0000 7f8b567fc700 20 client.174213.objecter dump_active .. 0 homeless 2021-05-25T12:02:45.152 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.097+0000 7f8b65d9b700 10 monclient: get_auth_request con 0x7f8b440334d0 auth_method 0 2021-05-25T12:02:45.152 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.098+0000 7f8b567fc700 10 client.174213.objecter ms_handle_connect 0x7f8b440334d0 2021-05-25T12:02:45.152 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.102+0000 7f8b567fc700 1 -- 172.21.15.102:0/2790337663 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7f8b5c012d90 con 0x7f8b680692d0 2021-05-25T12:02:45.152 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.102+0000 7f8b567fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:02:45.153 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.102+0000 7f8b567fc700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:02:45.310 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.308+0000 7f8b6ce9c700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2021-05-25T12:02:45.310 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.308+0000 7f8b6ce9c700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2021-05-25T12:02:45.310 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.308+0000 7f8b6ce9c700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:02:45.311 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.308+0000 7f8b6ce9c700 1 -- 172.21.15.102:0/2790337663 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f8b48005060 con 0x7f8b680692d0 2021-05-25T12:02:45.311 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.308+0000 7f8b567fc700 1 -- 172.21.15.102:0/2790337663 <== mon.0 v1:172.21.15.102:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1342 (unknown 254746657 0 422658332) 0x7f8b5c012980 con 0x7f8b680692d0 2021-05-25T12:02:45.311 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.308+0000 7f8b567fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2021-05-25T12:02:45.311 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.308+0000 7f8b567fc700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:02:45.312 INFO:teuthology.orchestra.run.smithi102.stdout:{"election_epoch":30,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":4,"features":{"quorum_con":"4540138303579357183","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"20ee91b6-8935-4b98-b868-85c03789ccc4","modified":"2021-05-25T11:59:11.011522Z","created":"2021-05-25T11:59:11.011522Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6790","nonce":0}]},"addr":"172.21.15.102:6790/0","public_addr":"172.21.15.102:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2021-05-25T12:02:45.314 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.310+0000 7f8b6ce9c700 10 librados: watch_flush enter 2021-05-25T12:02:45.314 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.310+0000 7f8b6ce9c700 10 librados: watch_flush exit 2021-05-25T12:02:45.314 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.310+0000 7f8b6ce9c700 20 client.174213.objecter shutdown clearing up homeless session... 2021-05-25T12:02:45.314 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.310+0000 7f8b6ce9c700 10 client.174213.objecter successfully canceled tick 2021-05-25T12:02:45.315 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.310+0000 7f8b6ce9c700 1 -- 172.21.15.102:0/2790337663 >> v1:172.21.15.181:6812/29582 conn(0x7f8b440334d0 legacy=0x7f8b44035990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:45.315 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.310+0000 7f8b6ce9c700 10 monclient: shutdown 2021-05-25T12:02:45.315 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.310+0000 7f8b6ce9c700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:02:45.315 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.310+0000 7f8b6ce9c700 1 -- 172.21.15.102:0/2790337663 >> v1:172.21.15.102:6789/0 conn(0x7f8b680692d0 legacy=0x7f8b680674f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:45.315 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.310+0000 7f8b6ce9c700 1 -- 172.21.15.102:0/2790337663 shutdown_connections 2021-05-25T12:02:45.316 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.310+0000 7f8b6ce9c700 1 -- 172.21.15.102:0/2790337663 >> 172.21.15.102:0/2790337663 conn(0x7f8b6809b370 msgr2=0x7f8b680990e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:02:45.316 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.310+0000 7f8b6ce9c700 1 -- 172.21.15.102:0/2790337663 shutdown_connections 2021-05-25T12:02:45.316 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.310+0000 7f8b6ce9c700 1 -- 172.21.15.102:0/2790337663 wait complete. 2021-05-25T12:02:45.316 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.310+0000 7f8b6ce9c700 1 librados: shutdown 2021-05-25T12:02:45.322 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2021-05-25T12:02:45.322 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2021-05-25T12:02:45.323 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2021-05-25T12:02:45.432 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.431+0000 7ff3a511c700 10 monclient: get_monmap_and_config 2021-05-25T12:02:45.433 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.431+0000 7ff3a511c700 10 monclient: build_initial_monmap 2021-05-25T12:02:45.433 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.431+0000 7ff3a511c700 10 monclient: monmap: 2021-05-25T12:02:45.433 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:02:45.433 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:45.434 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:02:45.432256+0000 2021-05-25T12:02:45.434 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:02:45.432256+0000 2021-05-25T12:02:45.434 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:02:45.434 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:45.434 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:02:45.435 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:02:45.435 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:02:45.435 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:45.448 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.432+0000 7ff3a511c700 1 Processor -- start 2021-05-25T12:02:45.449 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.432+0000 7ff3a511c700 1 -- start start 2021-05-25T12:02:45.449 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.432+0000 7ff3a511c700 10 monclient: init 2021-05-25T12:02:45.449 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.432+0000 7ff3a511c700 10 monclient: _reopen_session rank -1 2021-05-25T12:02:45.449 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.432+0000 7ff3a511c700 10 monclient: _add_conns ranks=[0,1,2] 2021-05-25T12:02:45.449 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.432+0000 7ff3a511c700 10 monclient(hunting): picked mon.noname-a con 0x7ff3a00931e0 addr v1:172.21.15.102:6789/0 2021-05-25T12:02:45.450 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.432+0000 7ff3a511c700 10 monclient(hunting): picked mon.noname-c con 0x7ff3a0126f80 addr v1:172.21.15.181:6789/0 2021-05-25T12:02:45.450 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.432+0000 7ff3a511c700 10 monclient(hunting): picked mon.noname-b con 0x7ff3a0068e70 addr v1:172.21.15.102:6790/0 2021-05-25T12:02:45.450 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.432+0000 7ff3a511c700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3a00945d0 con 0x7ff3a00931e0 2021-05-25T12:02:45.450 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.432+0000 7ff3a511c700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3a0094870 con 0x7ff3a0126f80 2021-05-25T12:02:45.450 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.432+0000 7ff3a511c700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3a0093dd0 con 0x7ff3a0068e70 2021-05-25T12:02:45.451 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.432+0000 7ff3a511c700 10 monclient(hunting): _renew_subs 2021-05-25T12:02:45.451 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.432+0000 7ff3a511c700 10 monclient(hunting): authenticate will time out at 966.029567s 2021-05-25T12:02:45.451 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.432+0000 7ff3a491b700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7ff3a0068e70 0x7ff3a0132300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:55170/0 (socket says 172.21.15.102:55170) 2021-05-25T12:02:45.451 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.432+0000 7ff3a491b700 1 -- 172.21.15.102:0/364968854 learned_addr learned my addr 172.21.15.102:0/364968854 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:02:45.452 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.432+0000 7ff39effd700 1 -- 172.21.15.102:0/364968854 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2152154607 0 0) 0x7ff3a0093dd0 con 0x7ff3a0068e70 2021-05-25T12:02:45.452 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.432+0000 7ff39effd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:02:45.452 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.432+0000 7ff39effd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:02:45.452 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.432+0000 7ff39effd700 10 monclient(hunting): my global_id is 23102 2021-05-25T12:02:45.453 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.432+0000 7ff39effd700 1 -- 172.21.15.102:0/364968854 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff388003270 con 0x7ff3a0068e70 2021-05-25T12:02:45.453 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.432+0000 7ff39effd700 1 -- 172.21.15.102:0/364968854 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2050523156 0 0) 0x7ff3a00945d0 con 0x7ff3a00931e0 2021-05-25T12:02:45.453 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.432+0000 7ff39effd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:02:45.453 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.432+0000 7ff39effd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:02:45.453 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.432+0000 7ff39effd700 10 monclient(hunting): my global_id is 174216 2021-05-25T12:02:45.454 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.432+0000 7ff39effd700 1 -- 172.21.15.102:0/364968854 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3a0093dd0 con 0x7ff3a00931e0 2021-05-25T12:02:45.454 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.433+0000 7ff39effd700 1 -- 172.21.15.102:0/364968854 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1498139124 0 0) 0x7ff388003270 con 0x7ff3a0068e70 2021-05-25T12:02:45.454 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.433+0000 7ff39effd700 1 -- 172.21.15.102:0/364968854 >> v1:172.21.15.181:6789/0 conn(0x7ff3a0126f80 legacy=0x7ff3a012ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:45.454 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.433+0000 7ff39effd700 1 -- 172.21.15.102:0/364968854 >> v1:172.21.15.102:6789/0 conn(0x7ff3a00931e0 legacy=0x7ff3a0126870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:45.455 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.433+0000 7ff39effd700 10 monclient: _finish_hunting 0 2021-05-25T12:02:45.455 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.433+0000 7ff39effd700 1 monclient: found mon.noname-b 2021-05-25T12:02:45.455 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.433+0000 7ff39effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:02:45.455 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.433+0000 7ff39effd700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T12:02:45.455 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.433+0000 7ff39effd700 1 -- 172.21.15.102:0/364968854 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3a00692e0 con 0x7ff3a0068e70 2021-05-25T12:02:45.456 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.433+0000 7ff39effd700 10 monclient: _finish_auth 0 2021-05-25T12:02:45.456 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.433+0000 7ff39effd700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:02:45.456 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.433+0000 7ff3a511c700 5 monclient: authenticate success, global_id 23102 2021-05-25T12:02:45.456 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.433+0000 7ff39effd700 1 -- 172.21.15.102:0/364968854 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7ff3940019e0 con 0x7ff3a0068e70 2021-05-25T12:02:45.457 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.433+0000 7ff3a511c700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:02:45.457 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.433+0000 7ff39effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:45.457 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.433+0000 7ff39effd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:02:45.457 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.433+0000 7ff39effd700 10 monclient: dump: 2021-05-25T12:02:45.457 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:45.458 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:45.458 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:45.458 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:45.458 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:45.459 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:45.459 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:45.459 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:45.459 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:45.460 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:45.460 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.433+0000 7ff3a511c700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:02:45.460 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.433+0000 7ff39effd700 1 -- 172.21.15.102:0/364968854 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff394001b90 con 0x7ff3a0068e70 2021-05-25T12:02:45.460 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.433+0000 7ff39effd700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:02:45.460 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.433+0000 7ff3a511c700 10 monclient: get_monmap_and_config success 2021-05-25T12:02:45.461 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.433+0000 7ff39effd700 1 -- 172.21.15.102:0/364968854 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7ff394001f20 con 0x7ff3a0068e70 2021-05-25T12:02:45.461 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.433+0000 7ff39effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:45.461 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.433+0000 7ff3a511c700 4 set_mon_vals no callback set 2021-05-25T12:02:45.461 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.433+0000 7ff39effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:02:45.461 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.433+0000 7ff39effd700 10 monclient: dump: 2021-05-25T12:02:45.462 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:45.462 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:45.462 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:45.462 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:45.462 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:45.463 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:45.463 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:45.463 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:45.463 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:45.464 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:45.464 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.433+0000 7ff3a511c700 10 monclient: shutdown 2021-05-25T12:02:45.464 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.433+0000 7ff3a511c700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:02:45.464 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.433+0000 7ff3a511c700 1 -- 172.21.15.102:0/364968854 >> v1:172.21.15.102:6790/0 conn(0x7ff3a0068e70 legacy=0x7ff3a0132300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:45.464 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.433+0000 7ff3a511c700 1 -- 172.21.15.102:0/364968854 shutdown_connections 2021-05-25T12:02:45.465 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.433+0000 7ff3a511c700 1 -- 172.21.15.102:0/364968854 >> 172.21.15.102:0/364968854 conn(0x7ff3a00974b0 msgr2=0x7ff3a00978c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:02:45.465 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.433+0000 7ff3a511c700 1 -- 172.21.15.102:0/364968854 shutdown_connections 2021-05-25T12:02:45.465 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.433+0000 7ff3a511c700 1 -- 172.21.15.102:0/364968854 wait complete. 2021-05-25T12:02:45.465 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.434+0000 7ff3a511c700 10 monclient: build_initial_monmap 2021-05-25T12:02:45.466 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.434+0000 7ff3a511c700 10 monclient: monmap: 2021-05-25T12:02:45.466 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:02:45.466 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:02:45.466 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:02:45.466 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:02:45.467 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:02:45.467 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:02:45.467 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:02:45.467 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:02:45.468 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:02:45.468 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:45.468 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.434+0000 7ff3a511c700 1 librados: starting msgr at 2021-05-25T12:02:45.468 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.434+0000 7ff3a511c700 1 librados: starting objecter 2021-05-25T12:02:45.468 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.434+0000 7ff3a511c700 1 Processor -- start 2021-05-25T12:02:45.469 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.434+0000 7ff3a511c700 1 -- start start 2021-05-25T12:02:45.469 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.434+0000 7ff3a511c700 1 librados: setting wanted keys 2021-05-25T12:02:45.469 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.434+0000 7ff3a511c700 1 librados: calling monclient init 2021-05-25T12:02:45.469 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.434+0000 7ff3a511c700 10 monclient: init 2021-05-25T12:02:45.469 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.435+0000 7ff3a511c700 10 monclient: _reopen_session rank -1 2021-05-25T12:02:45.470 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.435+0000 7ff3a511c700 10 monclient: _add_conns ranks=[2,1,0] 2021-05-25T12:02:45.470 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.435+0000 7ff3a511c700 10 monclient(hunting): picked mon.noname-c con 0x7ff3a0068e70 addr v1:172.21.15.102:6790/0 2021-05-25T12:02:45.470 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.435+0000 7ff3a511c700 10 monclient(hunting): picked mon.noname-b con 0x7ff3a00931e0 addr v1:172.21.15.181:6789/0 2021-05-25T12:02:45.470 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.435+0000 7ff3a511c700 10 monclient(hunting): picked mon.noname-a con 0x7ff3a0126f80 addr v1:172.21.15.102:6789/0 2021-05-25T12:02:45.471 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.435+0000 7ff3a511c700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3a00666d0 con 0x7ff3a0126f80 2021-05-25T12:02:45.471 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.435+0000 7ff3a511c700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3a0066970 con 0x7ff3a00931e0 2021-05-25T12:02:45.471 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.435+0000 7ff3a511c700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3a0066c10 con 0x7ff3a0068e70 2021-05-25T12:02:45.471 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.435+0000 7ff3a511c700 10 monclient(hunting): _renew_subs 2021-05-25T12:02:45.471 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.435+0000 7ff3a511c700 10 monclient(hunting): authenticate will time out at 966.032315s 2021-05-25T12:02:45.472 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.435+0000 7ff3a491b700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7ff3a0126f80 0x7ff3a0143370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60300/0 (socket says 172.21.15.102:60300) 2021-05-25T12:02:45.472 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.435+0000 7ff39ffff700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7ff3a0068e70 0x7ff3a00680c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:55174/0 (socket says 172.21.15.102:55174) 2021-05-25T12:02:45.472 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.435+0000 7ff3a491b700 1 -- 172.21.15.102:0/1636847955 learned_addr learned my addr 172.21.15.102:0/1636847955 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:02:45.472 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.435+0000 7ff393fff700 10 client.?.objecter ms_handle_connect 0x7ff3a0126f80 2021-05-25T12:02:45.472 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.435+0000 7ff393fff700 10 client.?.objecter resend_mon_ops 2021-05-25T12:02:45.473 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.435+0000 7ff393fff700 10 client.?.objecter ms_handle_connect 0x7ff3a0068e70 2021-05-25T12:02:45.473 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.435+0000 7ff393fff700 10 client.?.objecter resend_mon_ops 2021-05-25T12:02:45.473 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.435+0000 7ff393fff700 10 client.?.objecter ms_handle_connect 0x7ff3a00931e0 2021-05-25T12:02:45.473 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.435+0000 7ff393fff700 10 client.?.objecter resend_mon_ops 2021-05-25T12:02:45.474 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.435+0000 7ff393fff700 1 -- 172.21.15.102:0/1636847955 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4247146803 0 0) 0x7ff3a00666d0 con 0x7ff3a0126f80 2021-05-25T12:02:45.474 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.435+0000 7ff393fff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:02:45.474 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.435+0000 7ff393fff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:02:45.474 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.435+0000 7ff393fff700 10 monclient(hunting): my global_id is 174219 2021-05-25T12:02:45.475 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.435+0000 7ff393fff700 1 -- 172.21.15.102:0/1636847955 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff37c004c10 con 0x7ff3a0126f80 2021-05-25T12:02:45.475 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.435+0000 7ff393fff700 1 -- 172.21.15.102:0/1636847955 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2569975031 0 0) 0x7ff3a0066c10 con 0x7ff3a0068e70 2021-05-25T12:02:45.475 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.435+0000 7ff393fff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:02:45.475 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.435+0000 7ff393fff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:02:45.476 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.435+0000 7ff393fff700 10 monclient(hunting): my global_id is 23105 2021-05-25T12:02:45.476 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.435+0000 7ff393fff700 1 -- 172.21.15.102:0/1636847955 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3a00666d0 con 0x7ff3a0068e70 2021-05-25T12:02:45.476 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.435+0000 7ff393fff700 1 -- 172.21.15.102:0/1636847955 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4102862275 0 0) 0x7ff3a0066970 con 0x7ff3a00931e0 2021-05-25T12:02:45.476 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.435+0000 7ff393fff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:02:45.477 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.435+0000 7ff393fff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:02:45.477 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.435+0000 7ff393fff700 10 monclient(hunting): my global_id is 95182 2021-05-25T12:02:45.477 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.435+0000 7ff393fff700 1 -- 172.21.15.102:0/1636847955 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3a0066c10 con 0x7ff3a00931e0 2021-05-25T12:02:45.477 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.435+0000 7ff393fff700 1 -- 172.21.15.102:0/1636847955 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1691642533 0 0) 0x7ff37c004c10 con 0x7ff3a0126f80 2021-05-25T12:02:45.477 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.435+0000 7ff393fff700 1 -- 172.21.15.102:0/1636847955 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff3a0066970 con 0x7ff3a0126f80 2021-05-25T12:02:45.478 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.435+0000 7ff393fff700 1 -- 172.21.15.102:0/1636847955 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7ff3940019a0 con 0x7ff3a0126f80 2021-05-25T12:02:45.478 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.435+0000 7ff393fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:45.478 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.435+0000 7ff393fff700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:02:45.478 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.435+0000 7ff393fff700 10 monclient(hunting): dump: 2021-05-25T12:02:45.479 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:45.479 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:45.479 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:45.479 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:45.479 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:45.480 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:45.480 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:45.480 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:45.480 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:45.481 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:45.481 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.435+0000 7ff393fff700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:02:45.481 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.435+0000 7ff393fff700 1 -- 172.21.15.102:0/1636847955 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2201439871 0 0) 0x7ff3a00666d0 con 0x7ff3a0068e70 2021-05-25T12:02:45.481 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.435+0000 7ff393fff700 1 -- 172.21.15.102:0/1636847955 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff37c004c10 con 0x7ff3a0068e70 2021-05-25T12:02:45.481 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.435+0000 7ff393fff700 1 -- 172.21.15.102:0/1636847955 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7ff384001a80 con 0x7ff3a0068e70 2021-05-25T12:02:45.482 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.435+0000 7ff393fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:45.482 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.435+0000 7ff393fff700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2021-05-25T12:02:45.482 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.435+0000 7ff393fff700 10 monclient(hunting): dump: 2021-05-25T12:02:45.482 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:45.483 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:45.483 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:45.483 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:45.483 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:45.484 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:45.484 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:45.484 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:45.484 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:45.484 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:45.485 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.436+0000 7ff393fff700 1 -- 172.21.15.102:0/1636847955 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3831630857 0 0) 0x7ff3a0066c10 con 0x7ff3a00931e0 2021-05-25T12:02:45.485 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.436+0000 7ff393fff700 1 -- 172.21.15.102:0/1636847955 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff3a00666d0 con 0x7ff3a00931e0 2021-05-25T12:02:45.485 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.436+0000 7ff393fff700 1 -- 172.21.15.102:0/1636847955 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7ff38c0018c0 con 0x7ff3a00931e0 2021-05-25T12:02:45.485 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.436+0000 7ff393fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:45.485 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.436+0000 7ff393fff700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2021-05-25T12:02:45.486 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.436+0000 7ff393fff700 10 monclient(hunting): dump: 2021-05-25T12:02:45.486 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:45.486 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:45.486 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:45.486 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:45.487 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:45.487 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:45.487 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:45.487 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:45.488 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:45.488 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:45.488 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.436+0000 7ff393fff700 1 -- 172.21.15.102:0/1636847955 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1176706083 0 0) 0x7ff37c004c10 con 0x7ff3a0068e70 2021-05-25T12:02:45.488 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.436+0000 7ff393fff700 1 -- 172.21.15.102:0/1636847955 >> v1:172.21.15.181:6789/0 conn(0x7ff3a00931e0 legacy=0x7ff3a013b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:45.488 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.436+0000 7ff393fff700 1 -- 172.21.15.102:0/1636847955 >> v1:172.21.15.102:6789/0 conn(0x7ff3a0126f80 legacy=0x7ff3a0143370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:45.489 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.436+0000 7ff393fff700 10 monclient: _finish_hunting 0 2021-05-25T12:02:45.489 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.436+0000 7ff393fff700 1 monclient: found mon.c 2021-05-25T12:02:45.489 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.436+0000 7ff393fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:02:45.489 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.436+0000 7ff393fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:02:45.489 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.436+0000 7ff393fff700 1 -- 172.21.15.102:0/1636847955 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3a0092660 con 0x7ff3a0068e70 2021-05-25T12:02:45.490 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.436+0000 7ff393fff700 10 monclient: _finish_auth 0 2021-05-25T12:02:45.490 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.436+0000 7ff393fff700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:02:45.490 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.436+0000 7ff3a511c700 5 monclient: authenticate success, global_id 23105 2021-05-25T12:02:45.490 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.436+0000 7ff3a511c700 10 monclient: _renew_subs 2021-05-25T12:02:45.491 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.436+0000 7ff3a511c700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:02:45.491 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.436+0000 7ff3a511c700 1 -- 172.21.15.102:0/1636847955 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3a00687d0 con 0x7ff3a0068e70 2021-05-25T12:02:45.491 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.436+0000 7ff3a511c700 10 client.23105.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:02:45.491 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.436+0000 7ff3a511c700 10 monclient: _renew_subs 2021-05-25T12:02:45.491 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.436+0000 7ff3a511c700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:02:45.492 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.436+0000 7ff3a511c700 1 -- 172.21.15.102:0/1636847955 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3a013b8d0 con 0x7ff3a0068e70 2021-05-25T12:02:45.492 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.436+0000 7ff3a511c700 1 librados: init done 2021-05-25T12:02:45.492 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.436+0000 7ff393fff700 1 -- 172.21.15.102:0/1636847955 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff384001640 con 0x7ff3a0068e70 2021-05-25T12:02:45.492 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.436+0000 7ff393fff700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:02:45.492 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.436+0000 7ff393fff700 1 -- 172.21.15.102:0/1636847955 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7ff384002170 con 0x7ff3a0068e70 2021-05-25T12:02:45.493 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.436+0000 7ff393fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:45.493 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.436+0000 7ff393fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:02:45.493 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.436+0000 7ff39e7fc700 4 set_mon_vals no callback set 2021-05-25T12:02:45.493 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.436+0000 7ff393fff700 10 monclient: dump: 2021-05-25T12:02:45.493 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:45.494 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:45.494 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:45.494 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:45.494 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:45.495 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:45.495 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:45.495 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:45.495 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:45.495 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:45.496 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.436+0000 7ff3a511c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:02:45.496 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.436+0000 7ff3a511c700 1 -- 172.21.15.102:0/1636847955 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff3800065a0 con 0x7ff380002840 2021-05-25T12:02:45.496 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.436+0000 7ff393fff700 10 client.23105.objecter ms_handle_connect 0x7ff380002840 2021-05-25T12:02:45.496 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.437+0000 7ff393fff700 10 client.23105.objecter resend_mon_ops 2021-05-25T12:02:45.496 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.437+0000 7ff393fff700 1 -- 172.21.15.102:0/1636847955 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7ff384002350 con 0x7ff3a0068e70 2021-05-25T12:02:45.497 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.437+0000 7ff393fff700 1 -- 172.21.15.102:0/1636847955 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(83..83 src has 1..83) v4 ==== 7059+0+0 (unknown 555442269 0 0) 0x7ff38403d900 con 0x7ff3a0068e70 2021-05-25T12:02:45.497 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.437+0000 7ff393fff700 10 client.23105.objecter ms_dispatch 0x7ff3a0001550 osd_map(83..83 src has 1..83) v4 2021-05-25T12:02:45.497 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.437+0000 7ff393fff700 3 client.23105.objecter handle_osd_map got epochs [83,83] > 0 2021-05-25T12:02:45.497 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.437+0000 7ff393fff700 3 client.23105.objecter handle_osd_map decoding full epoch 83 2021-05-25T12:02:45.497 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.437+0000 7ff393fff700 20 client.23105.objecter dump_active .. 0 homeless 2021-05-25T12:02:45.498 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.437+0000 7ff393fff700 1 -- 172.21.15.102:0/1636847955 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3700258941 0 0) 0x7ff3800065a0 con 0x7ff380002840 2021-05-25T12:02:45.498 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.437+0000 7ff393fff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:02:45.498 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.437+0000 7ff393fff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:02:45.498 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.437+0000 7ff393fff700 10 monclient(hunting): my global_id is 174222 2021-05-25T12:02:45.498 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.437+0000 7ff393fff700 1 -- 172.21.15.102:0/1636847955 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff37c004c10 con 0x7ff380002840 2021-05-25T12:02:45.499 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.437+0000 7ff3a491b700 10 monclient: get_auth_request con 0x7ff37c033520 auth_method 0 2021-05-25T12:02:45.499 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.437+0000 7ff393fff700 1 -- 172.21.15.102:0/1636847955 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2587042461 0 0) 0x7ff37c004c10 con 0x7ff380002840 2021-05-25T12:02:45.499 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.437+0000 7ff393fff700 1 -- 172.21.15.102:0/1636847955 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff3800065a0 con 0x7ff380002840 2021-05-25T12:02:45.499 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.437+0000 7ff393fff700 1 -- 172.21.15.102:0/1636847955 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7ff38c001c40 con 0x7ff380002840 2021-05-25T12:02:45.499 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.437+0000 7ff393fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:45.500 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.437+0000 7ff393fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:02:45.500 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.437+0000 7ff393fff700 10 monclient: dump: 2021-05-25T12:02:45.500 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:45.500 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:45.500 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:45.501 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:45.501 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:45.501 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:45.501 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:45.501 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:45.502 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:45.502 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:45.502 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.437+0000 7ff393fff700 1 -- 172.21.15.102:0/1636847955 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 457895217 0 0) 0x7ff3800065a0 con 0x7ff380002840 2021-05-25T12:02:45.502 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.437+0000 7ff393fff700 1 -- 172.21.15.102:0/1636847955 --> v1:172.21.15.102:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff380009e40 con 0x7ff380002840 2021-05-25T12:02:45.502 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.437+0000 7ff393fff700 10 client.23105.objecter ms_handle_connect 0x7ff37c033520 2021-05-25T12:02:45.503 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.438+0000 7ff393fff700 1 -- 172.21.15.102:0/1636847955 <== mon.0 v1:172.21.15.102:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7ff38c001f40 con 0x7ff380002840 2021-05-25T12:02:45.503 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.438+0000 7ff393fff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:02:45.503 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.438+0000 7ff393fff700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:02:45.503 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.438+0000 7ff393fff700 1 -- 172.21.15.102:0/1636847955 >> v1:172.21.15.102:6789/0 conn(0x7ff380002840 legacy=0x7ff380004ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:45.504 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.438+0000 7ff393fff700 1 -- 172.21.15.102:0/1636847955 >> v1:172.21.15.102:6789/0 conn(0x7ff380002840 legacy=0x7ff380004ce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:02:45.504 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.444+0000 7ff3a511c700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:02:45.504 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.444+0000 7ff3a511c700 1 -- 172.21.15.102:0/1636847955 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff380009b00 con 0x7ff380008dc0 2021-05-25T12:02:45.504 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.444+0000 7ff393fff700 10 client.23105.objecter ms_handle_connect 0x7ff380008dc0 2021-05-25T12:02:45.504 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.444+0000 7ff393fff700 10 client.23105.objecter resend_mon_ops 2021-05-25T12:02:45.505 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.445+0000 7ff393fff700 1 -- 172.21.15.102:0/1636847955 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2979777940 0 0) 0x7ff380009b00 con 0x7ff380008dc0 2021-05-25T12:02:45.505 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.445+0000 7ff393fff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:02:45.505 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.445+0000 7ff393fff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:02:45.505 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.445+0000 7ff393fff700 10 monclient(hunting): my global_id is 174225 2021-05-25T12:02:45.505 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.445+0000 7ff393fff700 1 -- 172.21.15.102:0/1636847955 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3800065a0 con 0x7ff380008dc0 2021-05-25T12:02:45.506 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.445+0000 7ff393fff700 1 -- 172.21.15.102:0/1636847955 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3658638918 0 0) 0x7ff3800065a0 con 0x7ff380008dc0 2021-05-25T12:02:45.506 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.445+0000 7ff393fff700 1 -- 172.21.15.102:0/1636847955 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff380009b00 con 0x7ff380008dc0 2021-05-25T12:02:45.506 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.445+0000 7ff393fff700 1 -- 172.21.15.102:0/1636847955 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7ff38c002660 con 0x7ff380008dc0 2021-05-25T12:02:45.506 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.445+0000 7ff393fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:45.506 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.445+0000 7ff393fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:02:45.507 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.445+0000 7ff393fff700 10 monclient: dump: 2021-05-25T12:02:45.507 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:45.507 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:45.507 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:45.507 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:45.508 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:45.508 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:45.508 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:45.508 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:45.508 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:45.509 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:45.509 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.445+0000 7ff393fff700 1 -- 172.21.15.102:0/1636847955 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3243700506 0 0) 0x7ff380009b00 con 0x7ff380008dc0 2021-05-25T12:02:45.509 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.445+0000 7ff393fff700 1 -- 172.21.15.102:0/1636847955 --> v1:172.21.15.102:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ff380008740 con 0x7ff380008dc0 2021-05-25T12:02:45.509 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.445+0000 7ff393fff700 1 -- 172.21.15.102:0/1636847955 <== mon.0 v1:172.21.15.102:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3940 (unknown 0 0 787376944) 0x7ff38c0029e0 con 0x7ff380008dc0 2021-05-25T12:02:45.509 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.445+0000 7ff393fff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:02:45.510 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.445+0000 7ff393fff700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:02:45.510 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.445+0000 7ff393fff700 1 -- 172.21.15.102:0/1636847955 >> v1:172.21.15.102:6789/0 conn(0x7ff380008dc0 legacy=0x7ff380009250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:45.510 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.445+0000 7ff393fff700 1 -- 172.21.15.102:0/1636847955 >> v1:172.21.15.102:6789/0 conn(0x7ff380008dc0 legacy=0x7ff380009250 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:02:45.511 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:02:45.511 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:02:45.511 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:02:45.511 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "peon", 2021-05-25T12:02:45.512 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 30, 2021-05-25T12:02:45.512 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:02:45.512 INFO:teuthology.orchestra.run.smithi102.stdout: 0, 2021-05-25T12:02:45.512 INFO:teuthology.orchestra.run.smithi102.stdout: 1, 2021-05-25T12:02:45.512 INFO:teuthology.orchestra.run.smithi102.stdout: 2 2021-05-25T12:02:45.513 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:02:45.513 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 4, 2021-05-25T12:02:45.513 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:02:45.513 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:02:45.514 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:02:45.514 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:02:45.514 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:02:45.514 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:02:45.514 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:02:45.515 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:02:45.515 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:02:45.515 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:02:45.515 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:02:45.515 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:02:45.516 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:02:45.516 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:02:45.516 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:02:45.516 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:02:45.516 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:02:45.517 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:02:45.517 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:02:45.517 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:02:45.517 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:02:45.517 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:02:45.517 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:02:45.518 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:02:45.518 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:02:45.518 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:02:45.518 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:02:45.519 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:02:45.519 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:02:45.519 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:02:45.519 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:02:45.519 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:02:45.520 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:02:45.520 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:02:45.520 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:02:45.520 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:02:45.520 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:02:45.521 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:02:45.521 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:02:45.521 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:02:45.521 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:02:45.521 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:02:45.522 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:02:45.522 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:02:45.522 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:02:45.522 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:02:45.522 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:02:45.523 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:02:45.523 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:02:45.523 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:02:45.523 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:02:45.524 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:02:45.524 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:02:45.524 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:02:45.524 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:02:45.524 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:02:45.525 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:02:45.525 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:02:45.525 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:02:45.525 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:02:45.525 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:02:45.526 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:02:45.526 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:02:45.526 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:02:45.526 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:02:45.526 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:02:45.527 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:02:45.527 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:02:45.527 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:02:45.527 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:02:45.527 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:02:45.528 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:02:45.528 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:02:45.528 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:02:45.528 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:02:45.528 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:02:45.529 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:02:45.529 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:02:45.529 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:02:45.529 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:02:45.529 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:02:45.530 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:02:45.530 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:02:45.530 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:02:45.530 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:02:45.530 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:02:45.531 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:02:45.531 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:02:45.531 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:02:45.531 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:02:45.531 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:02:45.532 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:02:45.532 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:02:45.532 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:02:45.532 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:02:45.532 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:02:45.533 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:02:45.533 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:02:45.533 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:02:45.533 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:02:45.533 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:02:45.534 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:02:45.534 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:02:45.534 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:02:45.534 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:02:45.534 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:02:45.535 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:02:45.535 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:02:45.535 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:02:45.535 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:02:45.536 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:02:45.536 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:02:45.536 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:02:45.536 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:02:45.536 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:02:45.537 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:02:45.537 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:02:45.537 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:02:45.537 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": [ 2021-05-25T12:02:45.537 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:02:45.538 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:02:45.538 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:02:45.538 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 3 2021-05-25T12:02:45.539 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:02:45.539 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:02:45.539 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:02:45.539 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:02:45.539 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:02:45.540 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:02:45.540 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:02:45.540 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:02:45.540 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:02:45.540 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:02:45.541 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:02:45.541 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:02:45.543 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.445+0000 7ff3a511c700 10 librados: watch_flush enter 2021-05-25T12:02:45.543 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.445+0000 7ff3a511c700 10 librados: watch_flush exit 2021-05-25T12:02:45.544 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.445+0000 7ff3a511c700 20 client.23105.objecter shutdown clearing up homeless session... 2021-05-25T12:02:45.544 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.445+0000 7ff3a511c700 10 client.23105.objecter successfully canceled tick 2021-05-25T12:02:45.544 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.445+0000 7ff3a511c700 1 -- 172.21.15.102:0/1636847955 >> v1:172.21.15.181:6812/29582 conn(0x7ff37c033520 legacy=0x7ff37c0359e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:45.544 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.445+0000 7ff3a511c700 10 monclient: shutdown 2021-05-25T12:02:45.545 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.445+0000 7ff3a511c700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:02:45.545 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.445+0000 7ff3a511c700 1 -- 172.21.15.102:0/1636847955 >> v1:172.21.15.102:6790/0 conn(0x7ff3a0068e70 legacy=0x7ff3a00680c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:45.545 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.445+0000 7ff3a491b700 1 -- 172.21.15.102:0/1636847955 reap_dead start 2021-05-25T12:02:45.545 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.445+0000 7ff3a491b700 1 -- 172.21.15.102:0/1636847955 reap_dead start 2021-05-25T12:02:45.545 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.445+0000 7ff3a511c700 1 -- 172.21.15.102:0/1636847955 shutdown_connections 2021-05-25T12:02:45.546 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.445+0000 7ff3a511c700 1 -- 172.21.15.102:0/1636847955 >> 172.21.15.102:0/1636847955 conn(0x7ff3a00974b0 msgr2=0x7ff3a012cbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:02:45.546 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.446+0000 7ff3a511c700 1 -- 172.21.15.102:0/1636847955 shutdown_connections 2021-05-25T12:02:45.546 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.446+0000 7ff3a511c700 1 -- 172.21.15.102:0/1636847955 wait complete. 2021-05-25T12:02:45.546 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.446+0000 7ff3a511c700 1 librados: shutdown 2021-05-25T12:02:45.547 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2021-05-25T12:02:45.659 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.657+0000 7f789797f700 10 monclient: get_monmap_and_config 2021-05-25T12:02:45.659 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.657+0000 7f789797f700 10 monclient: build_initial_monmap 2021-05-25T12:02:45.659 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.657+0000 7f789797f700 10 monclient: monmap: 2021-05-25T12:02:45.660 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:02:45.660 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:45.660 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:02:45.658499+0000 2021-05-25T12:02:45.660 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:02:45.658499+0000 2021-05-25T12:02:45.660 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:02:45.661 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:45.661 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:02:45.661 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:02:45.661 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:02:45.662 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:45.662 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.658+0000 7f789797f700 1 Processor -- start 2021-05-25T12:02:45.662 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.658+0000 7f789797f700 1 -- start start 2021-05-25T12:02:45.662 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.658+0000 7f789797f700 10 monclient: init 2021-05-25T12:02:45.674 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.658+0000 7f789797f700 10 monclient: _reopen_session rank -1 2021-05-25T12:02:45.675 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.658+0000 7f789797f700 10 monclient: _add_conns ranks=[0,2,1] 2021-05-25T12:02:45.675 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.658+0000 7f789797f700 10 monclient(hunting): picked mon.noname-a con 0x7f7890092550 addr v1:172.21.15.102:6789/0 2021-05-25T12:02:45.675 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.658+0000 7f789797f700 10 monclient(hunting): picked mon.noname-b con 0x7f789012b100 addr v1:172.21.15.102:6790/0 2021-05-25T12:02:45.675 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.658+0000 7f789797f700 10 monclient(hunting): picked mon.noname-c con 0x7f789012ec10 addr v1:172.21.15.181:6789/0 2021-05-25T12:02:45.676 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.658+0000 7f789797f700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7890092930 con 0x7f7890092550 2021-05-25T12:02:45.676 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.658+0000 7f789797f700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78900931b0 con 0x7f789012ec10 2021-05-25T12:02:45.676 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.658+0000 7f789797f700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7890093450 con 0x7f789012b100 2021-05-25T12:02:45.676 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.658+0000 7f789797f700 10 monclient(hunting): _renew_subs 2021-05-25T12:02:45.676 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.658+0000 7f789797f700 10 monclient(hunting): authenticate will time out at 966.255817s 2021-05-25T12:02:45.677 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.658+0000 7f789571b700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f7890092550 0x7f7890093d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60308/0 (socket says 172.21.15.102:60308) 2021-05-25T12:02:45.677 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.658+0000 7f7894f1a700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f789012b100 0x7f789012d4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:55188/0 (socket says 172.21.15.102:55188) 2021-05-25T12:02:45.677 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.658+0000 7f789571b700 1 -- 172.21.15.102:0/2226210332 learned_addr learned my addr 172.21.15.102:0/2226210332 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:02:45.677 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.659+0000 7f787ffff700 1 -- 172.21.15.102:0/2226210332 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2468253981 0 0) 0x7f7890092930 con 0x7f7890092550 2021-05-25T12:02:45.678 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.659+0000 7f787ffff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:02:45.678 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.659+0000 7f787ffff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:02:45.678 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.659+0000 7f787ffff700 10 monclient(hunting): my global_id is 174228 2021-05-25T12:02:45.678 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.659+0000 7f787ffff700 1 -- 172.21.15.102:0/2226210332 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7878003270 con 0x7f7890092550 2021-05-25T12:02:45.679 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.659+0000 7f787ffff700 1 -- 172.21.15.102:0/2226210332 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 701989199 0 0) 0x7f7890093450 con 0x7f789012b100 2021-05-25T12:02:45.679 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.659+0000 7f787ffff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:02:45.679 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.659+0000 7f787ffff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:02:45.679 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.659+0000 7f787ffff700 10 monclient(hunting): my global_id is 23108 2021-05-25T12:02:45.679 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.659+0000 7f787ffff700 1 -- 172.21.15.102:0/2226210332 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7890092930 con 0x7f789012b100 2021-05-25T12:02:45.680 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.659+0000 7f787ffff700 1 -- 172.21.15.102:0/2226210332 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4241491849 0 0) 0x7f7878003270 con 0x7f7890092550 2021-05-25T12:02:45.680 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.659+0000 7f787ffff700 1 -- 172.21.15.102:0/2226210332 >> v1:172.21.15.102:6790/0 conn(0x7f789012b100 legacy=0x7f789012d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:45.680 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.659+0000 7f787ffff700 1 -- 172.21.15.102:0/2226210332 >> v1:172.21.15.181:6789/0 conn(0x7f789012ec10 legacy=0x7f7890131000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:45.680 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.659+0000 7f787ffff700 10 monclient: _finish_hunting 0 2021-05-25T12:02:45.681 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.659+0000 7f787ffff700 1 monclient: found mon.noname-a 2021-05-25T12:02:45.681 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.659+0000 7f787ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:02:45.681 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.659+0000 7f787ffff700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T12:02:45.681 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.659+0000 7f787ffff700 1 -- 172.21.15.102:0/2226210332 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78900936f0 con 0x7f7890092550 2021-05-25T12:02:45.681 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.659+0000 7f787ffff700 10 monclient: _finish_auth 0 2021-05-25T12:02:45.682 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.659+0000 7f787ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:02:45.682 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.659+0000 7f787ffff700 1 -- 172.21.15.102:0/2226210332 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f7880001610 con 0x7f7890092550 2021-05-25T12:02:45.682 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.659+0000 7f789797f700 5 monclient: authenticate success, global_id 174228 2021-05-25T12:02:45.682 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.659+0000 7f787ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:45.682 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.659+0000 7f787ffff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:02:45.683 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.659+0000 7f787ffff700 10 monclient: dump: 2021-05-25T12:02:45.683 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:45.683 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:45.683 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:45.684 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:45.684 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:45.684 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:45.684 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:45.684 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:45.685 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:45.685 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:45.685 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.659+0000 7f789797f700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:02:45.685 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.659+0000 7f787ffff700 1 -- 172.21.15.102:0/2226210332 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7880001840 con 0x7f7890092550 2021-05-25T12:02:45.685 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.659+0000 7f787ffff700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:02:45.686 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.659+0000 7f789797f700 10 monclient: get_monmap_and_config success 2021-05-25T12:02:45.686 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.659+0000 7f787ffff700 1 -- 172.21.15.102:0/2226210332 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f7880001bd0 con 0x7f7890092550 2021-05-25T12:02:45.686 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.659+0000 7f789797f700 4 set_mon_vals no callback set 2021-05-25T12:02:45.686 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.659+0000 7f787ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:45.687 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.659+0000 7f787ffff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:02:45.687 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.659+0000 7f787ffff700 10 monclient: dump: 2021-05-25T12:02:45.687 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:45.687 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:45.687 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:45.688 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:45.688 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:45.688 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:45.688 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:45.688 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:45.689 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:45.689 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:45.689 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.659+0000 7f789797f700 10 monclient: shutdown 2021-05-25T12:02:45.689 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.659+0000 7f789797f700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:02:45.689 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.659+0000 7f789797f700 1 -- 172.21.15.102:0/2226210332 >> v1:172.21.15.102:6789/0 conn(0x7f7890092550 legacy=0x7f7890093d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:45.690 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.659+0000 7f789797f700 1 -- 172.21.15.102:0/2226210332 shutdown_connections 2021-05-25T12:02:45.690 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.659+0000 7f789797f700 1 -- 172.21.15.102:0/2226210332 >> 172.21.15.102:0/2226210332 conn(0x7f789009b370 msgr2=0x7f7890097430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:02:45.690 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.660+0000 7f789797f700 1 -- 172.21.15.102:0/2226210332 shutdown_connections 2021-05-25T12:02:45.690 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.660+0000 7f789797f700 1 -- 172.21.15.102:0/2226210332 wait complete. 2021-05-25T12:02:45.690 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.660+0000 7f789797f700 10 monclient: build_initial_monmap 2021-05-25T12:02:45.691 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.660+0000 7f789797f700 10 monclient: monmap: 2021-05-25T12:02:45.691 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:02:45.691 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:02:45.691 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:02:45.692 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:02:45.692 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:02:45.692 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:02:45.692 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:02:45.692 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:02:45.693 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:02:45.693 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:45.693 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.660+0000 7f789797f700 1 librados: starting msgr at 2021-05-25T12:02:45.693 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.660+0000 7f789797f700 1 librados: starting objecter 2021-05-25T12:02:45.693 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.660+0000 7f789797f700 1 Processor -- start 2021-05-25T12:02:45.694 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.660+0000 7f789797f700 1 -- start start 2021-05-25T12:02:45.694 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.660+0000 7f789797f700 1 librados: setting wanted keys 2021-05-25T12:02:45.694 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.660+0000 7f789797f700 1 librados: calling monclient init 2021-05-25T12:02:45.694 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.660+0000 7f789797f700 10 monclient: init 2021-05-25T12:02:45.695 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.661+0000 7f789797f700 10 monclient: _reopen_session rank -1 2021-05-25T12:02:45.695 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.661+0000 7f789797f700 10 monclient: _add_conns ranks=[2,0,1] 2021-05-25T12:02:45.695 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.661+0000 7f789797f700 10 monclient(hunting): picked mon.noname-c con 0x7f7890092550 addr v1:172.21.15.102:6790/0 2021-05-25T12:02:45.695 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.661+0000 7f789797f700 10 monclient(hunting): picked mon.noname-a con 0x7f789012b100 addr v1:172.21.15.102:6789/0 2021-05-25T12:02:45.695 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.661+0000 7f789797f700 10 monclient(hunting): picked mon.noname-b con 0x7f789012ec10 addr v1:172.21.15.181:6789/0 2021-05-25T12:02:45.696 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.661+0000 7f789797f700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78901140f0 con 0x7f789012b100 2021-05-25T12:02:45.696 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.661+0000 7f789797f700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7890114390 con 0x7f789012ec10 2021-05-25T12:02:45.696 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.661+0000 7f789797f700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7890114630 con 0x7f7890092550 2021-05-25T12:02:45.696 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.661+0000 7f789797f700 10 monclient(hunting): _renew_subs 2021-05-25T12:02:45.696 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.661+0000 7f789797f700 10 monclient(hunting): authenticate will time out at 966.258464s 2021-05-25T12:02:45.697 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.661+0000 7f7894f1a700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f789012b100 0x7f7890111680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60314/0 (socket says 172.21.15.102:60314) 2021-05-25T12:02:45.697 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.661+0000 7f7894f1a700 1 -- 172.21.15.102:0/322972292 learned_addr learned my addr 172.21.15.102:0/322972292 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:02:45.697 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.661+0000 7f787d7fa700 10 client.?.objecter ms_handle_connect 0x7f789012b100 2021-05-25T12:02:45.697 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.661+0000 7f787d7fa700 10 client.?.objecter resend_mon_ops 2021-05-25T12:02:45.698 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.661+0000 7f787d7fa700 10 client.?.objecter ms_handle_connect 0x7f7890092550 2021-05-25T12:02:45.698 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.661+0000 7f787d7fa700 10 client.?.objecter resend_mon_ops 2021-05-25T12:02:45.698 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.661+0000 7f787d7fa700 10 client.?.objecter ms_handle_connect 0x7f789012ec10 2021-05-25T12:02:45.698 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.661+0000 7f787d7fa700 10 client.?.objecter resend_mon_ops 2021-05-25T12:02:45.699 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.661+0000 7f787d7fa700 1 -- 172.21.15.102:0/322972292 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1599163755 0 0) 0x7f78901140f0 con 0x7f789012b100 2021-05-25T12:02:45.699 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.661+0000 7f787d7fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:02:45.699 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.661+0000 7f787d7fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:02:45.699 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.661+0000 7f787d7fa700 10 monclient(hunting): my global_id is 174231 2021-05-25T12:02:45.699 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.661+0000 7f787d7fa700 1 -- 172.21.15.102:0/322972292 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7870004c10 con 0x7f789012b100 2021-05-25T12:02:45.700 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.661+0000 7f787d7fa700 1 -- 172.21.15.102:0/322972292 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3011533063 0 0) 0x7f7890114630 con 0x7f7890092550 2021-05-25T12:02:45.700 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.661+0000 7f787d7fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:02:45.700 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.661+0000 7f787d7fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:02:45.700 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.661+0000 7f787d7fa700 10 monclient(hunting): my global_id is 23111 2021-05-25T12:02:45.701 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.661+0000 7f787d7fa700 1 -- 172.21.15.102:0/322972292 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78901140f0 con 0x7f7890092550 2021-05-25T12:02:45.701 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.661+0000 7f787d7fa700 1 -- 172.21.15.102:0/322972292 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1114879995 0 0) 0x7f7890114390 con 0x7f789012ec10 2021-05-25T12:02:45.701 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.661+0000 7f787d7fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:02:45.701 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.661+0000 7f787d7fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:02:45.702 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.661+0000 7f787d7fa700 10 monclient(hunting): my global_id is 95188 2021-05-25T12:02:45.702 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.661+0000 7f787d7fa700 1 -- 172.21.15.102:0/322972292 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7890114630 con 0x7f789012ec10 2021-05-25T12:02:45.702 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.661+0000 7f787d7fa700 1 -- 172.21.15.102:0/322972292 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1311947543 0 0) 0x7f7870004c10 con 0x7f789012b100 2021-05-25T12:02:45.702 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.662+0000 7f787d7fa700 1 -- 172.21.15.102:0/322972292 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7890114390 con 0x7f789012b100 2021-05-25T12:02:45.702 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.662+0000 7f787d7fa700 1 -- 172.21.15.102:0/322972292 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f7884001920 con 0x7f789012b100 2021-05-25T12:02:45.703 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.662+0000 7f787d7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:45.703 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.662+0000 7f787d7fa700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:02:45.703 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.662+0000 7f787d7fa700 10 monclient(hunting): dump: 2021-05-25T12:02:45.703 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:45.703 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:45.704 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:45.704 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:45.704 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:45.704 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:45.704 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:45.705 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:45.705 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:45.705 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:45.705 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.662+0000 7f787d7fa700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:02:45.706 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.662+0000 7f787d7fa700 1 -- 172.21.15.102:0/322972292 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1032741452 0 0) 0x7f78901140f0 con 0x7f7890092550 2021-05-25T12:02:45.706 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.662+0000 7f787d7fa700 1 -- 172.21.15.102:0/322972292 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7870004c10 con 0x7f7890092550 2021-05-25T12:02:45.706 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.662+0000 7f787d7fa700 1 -- 172.21.15.102:0/322972292 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f7880001b50 con 0x7f7890092550 2021-05-25T12:02:45.706 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.662+0000 7f787d7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:45.706 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.662+0000 7f787d7fa700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2021-05-25T12:02:45.707 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.662+0000 7f787d7fa700 10 monclient(hunting): dump: 2021-05-25T12:02:45.707 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:45.707 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:45.707 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:45.707 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:45.708 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:45.708 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:45.708 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:45.708 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:45.708 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:45.709 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:45.709 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.662+0000 7f787d7fa700 1 -- 172.21.15.102:0/322972292 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2757045164 0 0) 0x7f7890114390 con 0x7f789012b100 2021-05-25T12:02:45.709 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.662+0000 7f787d7fa700 1 -- 172.21.15.102:0/322972292 >> v1:172.21.15.102:6790/0 conn(0x7f7890092550 legacy=0x7f7890110f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:45.709 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.662+0000 7f787d7fa700 1 -- 172.21.15.102:0/322972292 >> v1:172.21.15.181:6789/0 conn(0x7f789012ec10 legacy=0x7f789013f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:45.709 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.662+0000 7f787d7fa700 10 monclient: _finish_hunting 0 2021-05-25T12:02:45.710 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.662+0000 7f787d7fa700 1 monclient: found mon.a 2021-05-25T12:02:45.710 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.662+0000 7f787d7fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:02:45.710 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.662+0000 7f787d7fa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:02:45.710 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.662+0000 7f787d7fa700 1 -- 172.21.15.102:0/322972292 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78901124c0 con 0x7f789012b100 2021-05-25T12:02:45.711 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.662+0000 7f787d7fa700 10 monclient: _finish_auth 0 2021-05-25T12:02:45.711 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.662+0000 7f787d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:02:45.711 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.662+0000 7f789797f700 5 monclient: authenticate success, global_id 174231 2021-05-25T12:02:45.711 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.662+0000 7f789797f700 10 monclient: _renew_subs 2021-05-25T12:02:45.712 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.662+0000 7f789797f700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:02:45.712 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.662+0000 7f789797f700 1 -- 172.21.15.102:0/322972292 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7890113160 con 0x7f789012b100 2021-05-25T12:02:45.712 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.662+0000 7f789797f700 10 client.174231.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:02:45.712 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.662+0000 7f789797f700 10 monclient: _renew_subs 2021-05-25T12:02:45.712 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.662+0000 7f789797f700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:02:45.713 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.662+0000 7f789797f700 1 -- 172.21.15.102:0/322972292 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7890114bb0 con 0x7f789012b100 2021-05-25T12:02:45.713 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.662+0000 7f789797f700 1 librados: init done 2021-05-25T12:02:45.713 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.662+0000 7f787d7fa700 1 -- 172.21.15.102:0/322972292 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7884001c80 con 0x7f789012b100 2021-05-25T12:02:45.713 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.662+0000 7f787d7fa700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:02:45.714 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.662+0000 7f787d7fa700 1 -- 172.21.15.102:0/322972292 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f7884004a40 con 0x7f789012b100 2021-05-25T12:02:45.714 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.662+0000 7f787d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:45.714 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.662+0000 7f787d7fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:02:45.714 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.662+0000 7f787f7fe700 4 set_mon_vals no callback set 2021-05-25T12:02:45.714 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.662+0000 7f787d7fa700 10 monclient: dump: 2021-05-25T12:02:45.715 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:45.715 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:45.715 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:45.715 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:45.715 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:45.716 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:45.716 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:45.716 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:45.716 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:45.716 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:45.717 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.662+0000 7f789797f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:02:45.717 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.663+0000 7f787d7fa700 1 -- 172.21.15.102:0/322972292 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f788400dd60 con 0x7f789012b100 2021-05-25T12:02:45.717 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.663+0000 7f789797f700 1 -- 172.21.15.102:0/322972292 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f78680065a0 con 0x7f7868002840 2021-05-25T12:02:45.717 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.663+0000 7f787d7fa700 10 client.174231.objecter ms_handle_connect 0x7f7868002840 2021-05-25T12:02:45.717 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.663+0000 7f787d7fa700 10 client.174231.objecter resend_mon_ops 2021-05-25T12:02:45.718 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.663+0000 7f787d7fa700 1 -- 172.21.15.102:0/322972292 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(83..83 src has 1..83) v4 ==== 7059+0+0 (unknown 555442269 0 0) 0x7f788403dc70 con 0x7f789012b100 2021-05-25T12:02:45.718 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.663+0000 7f787d7fa700 10 client.174231.objecter ms_dispatch 0x7f7890001550 osd_map(83..83 src has 1..83) v4 2021-05-25T12:02:45.718 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.663+0000 7f787d7fa700 3 client.174231.objecter handle_osd_map got epochs [83,83] > 0 2021-05-25T12:02:45.718 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.663+0000 7f787d7fa700 3 client.174231.objecter handle_osd_map decoding full epoch 83 2021-05-25T12:02:45.719 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.663+0000 7f787d7fa700 20 client.174231.objecter dump_active .. 0 homeless 2021-05-25T12:02:45.719 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.663+0000 7f787d7fa700 1 -- 172.21.15.102:0/322972292 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 173751070 0 0) 0x7f78680065a0 con 0x7f7868002840 2021-05-25T12:02:45.719 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.663+0000 7f787d7fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:02:45.719 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.663+0000 7f787d7fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:02:45.719 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.663+0000 7f787d7fa700 10 monclient(hunting): my global_id is 23114 2021-05-25T12:02:45.720 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.663+0000 7f787d7fa700 1 -- 172.21.15.102:0/322972292 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7890114390 con 0x7f7868002840 2021-05-25T12:02:45.720 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.663+0000 7f7895f1c700 10 monclient: get_auth_request con 0x7f7870033370 auth_method 0 2021-05-25T12:02:45.720 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.663+0000 7f787d7fa700 1 -- 172.21.15.102:0/322972292 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3383990570 0 0) 0x7f7890114390 con 0x7f7868002840 2021-05-25T12:02:45.720 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.663+0000 7f787d7fa700 1 -- 172.21.15.102:0/322972292 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f78680065a0 con 0x7f7868002840 2021-05-25T12:02:45.720 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.663+0000 7f787d7fa700 1 -- 172.21.15.102:0/322972292 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f7880004710 con 0x7f7868002840 2021-05-25T12:02:45.721 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.663+0000 7f787d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:45.721 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.663+0000 7f787d7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:02:45.721 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.663+0000 7f787d7fa700 10 monclient: dump: 2021-05-25T12:02:45.721 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:45.721 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:45.722 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:45.722 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:45.722 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:45.722 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:45.722 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:45.723 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:45.723 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:45.723 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:45.723 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.663+0000 7f787d7fa700 1 -- 172.21.15.102:0/322972292 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 65430486 0 0) 0x7f78680065a0 con 0x7f7868002840 2021-05-25T12:02:45.724 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.663+0000 7f787d7fa700 1 -- 172.21.15.102:0/322972292 --> v1:172.21.15.102:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7868009e40 con 0x7f7868002840 2021-05-25T12:02:45.724 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.663+0000 7f787d7fa700 10 client.174231.objecter ms_handle_connect 0x7f7870033370 2021-05-25T12:02:45.724 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.664+0000 7f787d7fa700 1 -- 172.21.15.102:0/322972292 <== mon.2 v1:172.21.15.102:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7f7880006b00 con 0x7f7868002840 2021-05-25T12:02:45.724 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.664+0000 7f787d7fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:02:45.724 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.664+0000 7f787d7fa700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:02:45.725 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.664+0000 7f787d7fa700 1 -- 172.21.15.102:0/322972292 >> v1:172.21.15.102:6790/0 conn(0x7f7868002840 legacy=0x7f7868004ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:45.725 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.664+0000 7f787d7fa700 1 -- 172.21.15.102:0/322972292 >> v1:172.21.15.102:6790/0 conn(0x7f7868002840 legacy=0x7f7868004ce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:02:45.725 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.670+0000 7f789797f700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:02:45.725 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.670+0000 7f789797f700 1 -- 172.21.15.102:0/322972292 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f7868007540 con 0x7f78680068e0 2021-05-25T12:02:45.725 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.670+0000 7f787d7fa700 10 client.174231.objecter ms_handle_connect 0x7f78680068e0 2021-05-25T12:02:45.726 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.670+0000 7f787d7fa700 10 client.174231.objecter resend_mon_ops 2021-05-25T12:02:45.726 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.670+0000 7f787d7fa700 1 -- 172.21.15.102:0/322972292 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3101571861 0 0) 0x7f7868007540 con 0x7f78680068e0 2021-05-25T12:02:45.726 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.670+0000 7f787d7fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:02:45.726 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.670+0000 7f787d7fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:02:45.727 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.670+0000 7f787d7fa700 10 monclient(hunting): my global_id is 23117 2021-05-25T12:02:45.727 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.670+0000 7f787d7fa700 1 -- 172.21.15.102:0/322972292 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78680065a0 con 0x7f78680068e0 2021-05-25T12:02:45.727 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.671+0000 7f787d7fa700 1 -- 172.21.15.102:0/322972292 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2807239924 0 0) 0x7f78680065a0 con 0x7f78680068e0 2021-05-25T12:02:45.727 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.671+0000 7f787d7fa700 1 -- 172.21.15.102:0/322972292 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7868007540 con 0x7f78680068e0 2021-05-25T12:02:45.727 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.671+0000 7f787d7fa700 1 -- 172.21.15.102:0/322972292 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f7880005040 con 0x7f78680068e0 2021-05-25T12:02:45.728 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.671+0000 7f787d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:45.728 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.671+0000 7f787d7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:02:45.728 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.671+0000 7f787d7fa700 10 monclient: dump: 2021-05-25T12:02:45.728 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:45.728 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:45.729 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:45.729 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:45.729 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:45.729 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:45.730 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:45.730 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:45.730 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:45.730 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:45.730 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.671+0000 7f787d7fa700 1 -- 172.21.15.102:0/322972292 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1336890785 0 0) 0x7f7868007540 con 0x7f78680068e0 2021-05-25T12:02:45.731 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.671+0000 7f787d7fa700 1 -- 172.21.15.102:0/322972292 --> v1:172.21.15.102:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f7868008770 con 0x7f78680068e0 2021-05-25T12:02:45.731 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.671+0000 7f787d7fa700 1 -- 172.21.15.102:0/322972292 <== mon.2 v1:172.21.15.102:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3942 (unknown 0 0 412226933) 0x7f78800063d0 con 0x7f78680068e0 2021-05-25T12:02:45.731 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.671+0000 7f787d7fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:02:45.731 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.671+0000 7f787d7fa700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:02:45.731 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.671+0000 7f787d7fa700 1 -- 172.21.15.102:0/322972292 >> v1:172.21.15.102:6790/0 conn(0x7f78680068e0 legacy=0x7f7868006d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:45.732 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.671+0000 7f787d7fa700 1 -- 172.21.15.102:0/322972292 >> v1:172.21.15.102:6790/0 conn(0x7f78680068e0 legacy=0x7f7868006d70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:02:45.732 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:02:45.732 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:02:45.733 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:02:45.733 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "leader", 2021-05-25T12:02:45.733 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 30, 2021-05-25T12:02:45.733 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:02:45.733 INFO:teuthology.orchestra.run.smithi102.stdout: 0, 2021-05-25T12:02:45.734 INFO:teuthology.orchestra.run.smithi102.stdout: 1, 2021-05-25T12:02:45.734 INFO:teuthology.orchestra.run.smithi102.stdout: 2 2021-05-25T12:02:45.734 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:02:45.734 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 4, 2021-05-25T12:02:45.735 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:02:45.735 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:02:45.735 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:02:45.735 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:02:45.735 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:02:45.736 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:02:45.736 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:02:45.736 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:02:45.736 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:02:45.736 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:02:45.737 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:02:45.737 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:02:45.737 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:02:45.737 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:02:45.737 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:02:45.738 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:02:45.738 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:02:45.738 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:02:45.738 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:02:45.739 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:02:45.739 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:02:45.739 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:02:45.739 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:02:45.739 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:02:45.740 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:02:45.740 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:02:45.740 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:02:45.740 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:02:45.740 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:02:45.741 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:02:45.741 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:02:45.741 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:02:45.741 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:02:45.741 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:02:45.742 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:02:45.742 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:02:45.742 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:02:45.742 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:02:45.742 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:02:45.743 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:02:45.743 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:02:45.743 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:02:45.743 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:02:45.744 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:02:45.744 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:02:45.744 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:02:45.744 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:02:45.745 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:02:45.745 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:02:45.745 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:02:45.745 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:02:45.745 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:02:45.746 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:02:45.746 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:02:45.746 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:02:45.746 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:02:45.747 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:02:45.747 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:02:45.747 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:02:45.747 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:02:45.747 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:02:45.748 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:02:45.748 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:02:45.748 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:02:45.748 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:02:45.748 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:02:45.749 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:02:45.749 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:02:45.749 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:02:45.749 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:02:45.749 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:02:45.750 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:02:45.750 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:02:45.750 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:02:45.750 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:02:45.751 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:02:45.751 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:02:45.751 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:02:45.751 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:02:45.751 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:02:45.752 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:02:45.752 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:02:45.752 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:02:45.752 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:02:45.753 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:02:45.753 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:02:45.753 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:02:45.753 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:02:45.753 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:02:45.754 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:02:45.754 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:02:45.754 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:02:45.755 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:02:45.755 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:02:45.755 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:02:45.756 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:02:45.756 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:02:45.756 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:02:45.756 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:02:45.756 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:02:45.757 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:02:45.757 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:02:45.757 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:02:45.757 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:02:45.758 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:02:45.758 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:02:45.758 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:02:45.758 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:02:45.758 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:02:45.759 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:02:45.759 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:02:45.759 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:02:45.759 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:02:45.760 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:02:45.760 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:02:45.760 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:02:45.760 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:02:45.760 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:02:45.761 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:02:45.761 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:02:45.761 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:02:45.761 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:02:45.761 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 6 2021-05-25T12:02:45.762 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:02:45.762 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:02:45.762 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": [ 2021-05-25T12:02:45.762 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:02:45.762 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:02:45.763 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:02:45.763 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:02:45.763 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:02:45.763 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:02:45.763 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:02:45.764 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:02:45.764 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:02:45.767 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.671+0000 7f789797f700 10 librados: watch_flush enter 2021-05-25T12:02:45.767 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.671+0000 7f789797f700 10 librados: watch_flush exit 2021-05-25T12:02:45.767 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.671+0000 7f789797f700 20 client.174231.objecter shutdown clearing up homeless session... 2021-05-25T12:02:45.768 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.671+0000 7f789797f700 10 client.174231.objecter successfully canceled tick 2021-05-25T12:02:45.768 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.671+0000 7f789797f700 1 -- 172.21.15.102:0/322972292 >> v1:172.21.15.181:6812/29582 conn(0x7f7870033370 legacy=0x7f7870035830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:45.768 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.671+0000 7f789797f700 10 monclient: shutdown 2021-05-25T12:02:45.768 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.671+0000 7f789797f700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:02:45.768 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.671+0000 7f789797f700 1 -- 172.21.15.102:0/322972292 >> v1:172.21.15.102:6789/0 conn(0x7f789012b100 legacy=0x7f7890111680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:45.769 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.671+0000 7f7895f1c700 1 -- 172.21.15.102:0/322972292 reap_dead start 2021-05-25T12:02:45.769 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.671+0000 7f7895f1c700 1 -- 172.21.15.102:0/322972292 reap_dead start 2021-05-25T12:02:45.769 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.671+0000 7f789797f700 1 -- 172.21.15.102:0/322972292 shutdown_connections 2021-05-25T12:02:45.769 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.671+0000 7f789797f700 1 -- 172.21.15.102:0/322972292 >> 172.21.15.102:0/322972292 conn(0x7f789009b370 msgr2=0x7f7890095ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:02:45.770 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.672+0000 7f789797f700 1 -- 172.21.15.102:0/322972292 shutdown_connections 2021-05-25T12:02:45.770 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.672+0000 7f789797f700 1 -- 172.21.15.102:0/322972292 wait complete. 2021-05-25T12:02:45.770 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.672+0000 7f789797f700 1 librados: shutdown 2021-05-25T12:02:45.770 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2021-05-25T12:02:45.899 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.898+0000 7fb3e2fc6700 10 monclient: get_monmap_and_config 2021-05-25T12:02:45.899 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.898+0000 7fb3e2fc6700 10 monclient: build_initial_monmap 2021-05-25T12:02:45.900 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.898+0000 7fb3e2fc6700 10 monclient: monmap: 2021-05-25T12:02:45.900 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:02:45.900 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:45.900 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:02:45.898960+0000 2021-05-25T12:02:45.900 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:02:45.898960+0000 2021-05-25T12:02:45.901 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:02:45.901 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:45.901 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:02:45.901 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:02:45.902 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:02:45.902 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:45.914 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.898+0000 7fb3e2fc6700 1 Processor -- start 2021-05-25T12:02:45.915 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.898+0000 7fb3e2fc6700 1 -- start start 2021-05-25T12:02:45.915 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.898+0000 7fb3e2fc6700 10 monclient: init 2021-05-25T12:02:45.915 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.899+0000 7fb3e2fc6700 10 monclient: _reopen_session rank -1 2021-05-25T12:02:45.915 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.899+0000 7fb3e2fc6700 10 monclient: _add_conns ranks=[1,0,2] 2021-05-25T12:02:45.916 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.899+0000 7fb3e2fc6700 10 monclient(hunting): picked mon.noname-c con 0x7fb3dc090990 addr v1:172.21.15.181:6789/0 2021-05-25T12:02:45.916 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.899+0000 7fb3e2fc6700 10 monclient(hunting): picked mon.noname-a con 0x7fb3dc090d70 addr v1:172.21.15.102:6789/0 2021-05-25T12:02:45.916 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.899+0000 7fb3e2fc6700 10 monclient(hunting): picked mon.noname-b con 0x7fb3dc12e330 addr v1:172.21.15.102:6790/0 2021-05-25T12:02:45.916 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.899+0000 7fb3e2fc6700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3dc096fa0 con 0x7fb3dc090d70 2021-05-25T12:02:45.916 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.899+0000 7fb3e2fc6700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3dc0926a0 con 0x7fb3dc090990 2021-05-25T12:02:45.917 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.899+0000 7fb3e2fc6700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3dc092940 con 0x7fb3dc12e330 2021-05-25T12:02:45.917 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.899+0000 7fb3e2fc6700 10 monclient(hunting): _renew_subs 2021-05-25T12:02:45.917 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.899+0000 7fb3e2fc6700 10 monclient(hunting): authenticate will time out at 966.496160s 2021-05-25T12:02:45.917 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.899+0000 7fb3d3fff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fb3dc090d70 0x7fb3dc091200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60330/0 (socket says 172.21.15.102:60330) 2021-05-25T12:02:45.918 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.899+0000 7fb3d3fff700 1 -- 172.21.15.102:0/2427262021 learned_addr learned my addr 172.21.15.102:0/2427262021 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:02:45.918 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.899+0000 7fb3e1563700 1 --1- 172.21.15.102:0/2427262021 >> v1:172.21.15.102:6790/0 conn(0x7fb3dc12e330 0x7fb3dc130720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:55206/0 (socket says 172.21.15.102:55206) 2021-05-25T12:02:45.918 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.899+0000 7fb3d37fe700 1 -- 172.21.15.102:0/2427262021 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1921776946 0 0) 0x7fb3dc096fa0 con 0x7fb3dc090d70 2021-05-25T12:02:45.918 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.899+0000 7fb3d37fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:02:45.918 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.899+0000 7fb3d37fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:02:45.919 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.899+0000 7fb3d37fe700 10 monclient(hunting): my global_id is 174243 2021-05-25T12:02:45.919 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.899+0000 7fb3d37fe700 1 -- 172.21.15.102:0/2427262021 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3c8003270 con 0x7fb3dc090d70 2021-05-25T12:02:45.919 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.899+0000 7fb3d37fe700 1 -- 172.21.15.102:0/2427262021 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1638573654 0 0) 0x7fb3dc092940 con 0x7fb3dc12e330 2021-05-25T12:02:45.919 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.899+0000 7fb3d37fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:02:45.920 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.899+0000 7fb3d37fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:02:45.920 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.899+0000 7fb3d37fe700 10 monclient(hunting): my global_id is 23135 2021-05-25T12:02:45.920 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.899+0000 7fb3d37fe700 1 -- 172.21.15.102:0/2427262021 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3dc096fa0 con 0x7fb3dc12e330 2021-05-25T12:02:45.920 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.899+0000 7fb3d37fe700 1 -- 172.21.15.102:0/2427262021 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 650423183 0 0) 0x7fb3dc096fa0 con 0x7fb3dc12e330 2021-05-25T12:02:45.920 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.899+0000 7fb3d37fe700 1 -- 172.21.15.102:0/2427262021 >> v1:172.21.15.181:6789/0 conn(0x7fb3dc090990 legacy=0x7fb3dc093240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:45.921 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.899+0000 7fb3d37fe700 1 -- 172.21.15.102:0/2427262021 >> v1:172.21.15.102:6789/0 conn(0x7fb3dc090d70 legacy=0x7fb3dc091200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:45.921 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.899+0000 7fb3d37fe700 10 monclient: _finish_hunting 0 2021-05-25T12:02:45.921 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.899+0000 7fb3d37fe700 1 monclient: found mon.noname-b 2021-05-25T12:02:45.921 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.899+0000 7fb3d37fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:02:45.921 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.899+0000 7fb3d37fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T12:02:45.922 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.899+0000 7fb3d37fe700 1 -- 172.21.15.102:0/2427262021 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3dc092be0 con 0x7fb3dc12e330 2021-05-25T12:02:45.922 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.899+0000 7fb3d37fe700 10 monclient: _finish_auth 0 2021-05-25T12:02:45.922 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.899+0000 7fb3d37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:02:45.923 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.899+0000 7fb3e2fc6700 5 monclient: authenticate success, global_id 23135 2021-05-25T12:02:45.923 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.899+0000 7fb3d37fe700 1 -- 172.21.15.102:0/2427262021 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fb3d8001990 con 0x7fb3dc12e330 2021-05-25T12:02:45.923 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.899+0000 7fb3d37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:45.923 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.899+0000 7fb3d37fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:02:45.923 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.899+0000 7fb3d37fe700 10 monclient: dump: 2021-05-25T12:02:45.924 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:45.924 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:45.924 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:45.924 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:45.924 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:45.925 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:45.925 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:45.925 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:45.925 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:45.925 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:45.926 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.899+0000 7fb3e2fc6700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:02:45.926 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.899+0000 7fb3d37fe700 1 -- 172.21.15.102:0/2427262021 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb3d8001bc0 con 0x7fb3dc12e330 2021-05-25T12:02:45.926 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.899+0000 7fb3d37fe700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:02:45.926 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.899+0000 7fb3e2fc6700 10 monclient: get_monmap_and_config success 2021-05-25T12:02:45.926 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.899+0000 7fb3d37fe700 1 -- 172.21.15.102:0/2427262021 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fb3d8001f50 con 0x7fb3dc12e330 2021-05-25T12:02:45.927 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.899+0000 7fb3d37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:45.927 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.899+0000 7fb3e2fc6700 4 set_mon_vals no callback set 2021-05-25T12:02:45.927 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.899+0000 7fb3d37fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:02:45.927 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.899+0000 7fb3d37fe700 10 monclient: dump: 2021-05-25T12:02:45.928 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:45.928 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:45.928 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:45.928 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:45.928 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:45.929 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:45.929 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:45.929 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:45.929 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:45.929 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:45.930 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.900+0000 7fb3e2fc6700 10 monclient: shutdown 2021-05-25T12:02:45.930 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.900+0000 7fb3e2fc6700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:02:45.930 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.900+0000 7fb3e2fc6700 1 -- 172.21.15.102:0/2427262021 >> v1:172.21.15.102:6790/0 conn(0x7fb3dc12e330 legacy=0x7fb3dc130720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:45.930 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.900+0000 7fb3e2fc6700 1 -- 172.21.15.102:0/2427262021 shutdown_connections 2021-05-25T12:02:45.930 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.900+0000 7fb3e2fc6700 1 -- 172.21.15.102:0/2427262021 >> 172.21.15.102:0/2427262021 conn(0x7fb3dc09b370 msgr2=0x7fb3dc097430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:02:45.931 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.900+0000 7fb3e2fc6700 1 -- 172.21.15.102:0/2427262021 shutdown_connections 2021-05-25T12:02:45.931 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.900+0000 7fb3e2fc6700 1 -- 172.21.15.102:0/2427262021 wait complete. 2021-05-25T12:02:45.931 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.900+0000 7fb3e2fc6700 10 monclient: build_initial_monmap 2021-05-25T12:02:45.931 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.900+0000 7fb3e2fc6700 10 monclient: monmap: 2021-05-25T12:02:45.931 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:02:45.932 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:02:45.932 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:02:45.932 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:02:45.932 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:02:45.932 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:02:45.933 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:02:45.933 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:02:45.933 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:02:45.934 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:45.934 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.901+0000 7fb3e2fc6700 1 librados: starting msgr at 2021-05-25T12:02:45.934 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.901+0000 7fb3e2fc6700 1 librados: starting objecter 2021-05-25T12:02:45.934 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.901+0000 7fb3e2fc6700 1 Processor -- start 2021-05-25T12:02:45.934 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.901+0000 7fb3e2fc6700 1 -- start start 2021-05-25T12:02:45.935 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.901+0000 7fb3e2fc6700 1 librados: setting wanted keys 2021-05-25T12:02:45.935 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.901+0000 7fb3e2fc6700 1 librados: calling monclient init 2021-05-25T12:02:45.935 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.901+0000 7fb3e2fc6700 10 monclient: init 2021-05-25T12:02:45.935 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.901+0000 7fb3e2fc6700 10 monclient: _reopen_session rank -1 2021-05-25T12:02:45.935 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.901+0000 7fb3e2fc6700 10 monclient: _add_conns ranks=[1,2,0] 2021-05-25T12:02:45.936 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.901+0000 7fb3e2fc6700 10 monclient(hunting): picked mon.noname-b con 0x7fb3dc090990 addr v1:172.21.15.181:6789/0 2021-05-25T12:02:45.936 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.901+0000 7fb3e2fc6700 10 monclient(hunting): picked mon.noname-c con 0x7fb3dc090d70 addr v1:172.21.15.102:6790/0 2021-05-25T12:02:45.936 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.901+0000 7fb3e2fc6700 10 monclient(hunting): picked mon.noname-a con 0x7fb3dc12e330 addr v1:172.21.15.102:6789/0 2021-05-25T12:02:45.936 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.901+0000 7fb3e2fc6700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3dc098890 con 0x7fb3dc12e330 2021-05-25T12:02:45.936 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.901+0000 7fb3e2fc6700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3dc1135b0 con 0x7fb3dc090990 2021-05-25T12:02:45.937 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.901+0000 7fb3e2fc6700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3dc113850 con 0x7fb3dc090d70 2021-05-25T12:02:45.937 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.901+0000 7fb3e2fc6700 10 monclient(hunting): _renew_subs 2021-05-25T12:02:45.937 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.901+0000 7fb3e2fc6700 10 monclient(hunting): authenticate will time out at 966.498871s 2021-05-25T12:02:45.937 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.901+0000 7fb3e1563700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fb3dc12e330 0x7fb3dc140f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60336/0 (socket says 172.21.15.102:60336) 2021-05-25T12:02:45.937 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.901+0000 7fb3d3fff700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fb3dc090d70 0x7fb3dc12d050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:55212/0 (socket says 172.21.15.102:55212) 2021-05-25T12:02:45.938 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.901+0000 7fb3e1563700 1 -- 172.21.15.102:0/349302262 learned_addr learned my addr 172.21.15.102:0/349302262 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:02:45.938 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.902+0000 7fb3d0ff9700 10 client.?.objecter ms_handle_connect 0x7fb3dc12e330 2021-05-25T12:02:45.938 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.902+0000 7fb3d0ff9700 10 client.?.objecter resend_mon_ops 2021-05-25T12:02:45.938 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.902+0000 7fb3d0ff9700 10 client.?.objecter ms_handle_connect 0x7fb3dc090d70 2021-05-25T12:02:45.939 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.902+0000 7fb3d0ff9700 10 client.?.objecter resend_mon_ops 2021-05-25T12:02:45.939 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.902+0000 7fb3d0ff9700 10 client.?.objecter ms_handle_connect 0x7fb3dc090990 2021-05-25T12:02:45.939 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.902+0000 7fb3d0ff9700 10 client.?.objecter resend_mon_ops 2021-05-25T12:02:45.939 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.902+0000 7fb3d0ff9700 1 -- 172.21.15.102:0/349302262 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 941935156 0 0) 0x7fb3dc098890 con 0x7fb3dc12e330 2021-05-25T12:02:45.939 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.902+0000 7fb3d0ff9700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:02:45.940 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.902+0000 7fb3d0ff9700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:02:45.940 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.902+0000 7fb3d0ff9700 10 monclient(hunting): my global_id is 174246 2021-05-25T12:02:45.940 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.902+0000 7fb3d0ff9700 1 -- 172.21.15.102:0/349302262 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3bc004c10 con 0x7fb3dc12e330 2021-05-25T12:02:45.940 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.902+0000 7fb3d0ff9700 1 -- 172.21.15.102:0/349302262 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1584087668 0 0) 0x7fb3dc113850 con 0x7fb3dc090d70 2021-05-25T12:02:45.940 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.902+0000 7fb3d0ff9700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:02:45.941 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.902+0000 7fb3d0ff9700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:02:45.941 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.902+0000 7fb3d0ff9700 10 monclient(hunting): my global_id is 23138 2021-05-25T12:02:45.941 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.902+0000 7fb3d0ff9700 1 -- 172.21.15.102:0/349302262 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3dc098890 con 0x7fb3dc090d70 2021-05-25T12:02:45.941 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.902+0000 7fb3d0ff9700 1 -- 172.21.15.102:0/349302262 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3649684602 0 0) 0x7fb3dc1135b0 con 0x7fb3dc090990 2021-05-25T12:02:45.941 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.902+0000 7fb3d0ff9700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:02:45.942 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.902+0000 7fb3d0ff9700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:02:45.942 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.902+0000 7fb3d0ff9700 10 monclient(hunting): my global_id is 95200 2021-05-25T12:02:45.942 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.902+0000 7fb3d0ff9700 1 -- 172.21.15.102:0/349302262 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3dc113850 con 0x7fb3dc090990 2021-05-25T12:02:45.942 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.902+0000 7fb3d0ff9700 1 -- 172.21.15.102:0/349302262 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3948855606 0 0) 0x7fb3bc004c10 con 0x7fb3dc12e330 2021-05-25T12:02:45.943 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.902+0000 7fb3d0ff9700 1 -- 172.21.15.102:0/349302262 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb3dc1135b0 con 0x7fb3dc12e330 2021-05-25T12:02:45.943 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.902+0000 7fb3d0ff9700 1 -- 172.21.15.102:0/349302262 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fb3d8001f90 con 0x7fb3dc12e330 2021-05-25T12:02:45.943 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.902+0000 7fb3d0ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:45.943 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.902+0000 7fb3d0ff9700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:02:45.943 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.902+0000 7fb3d0ff9700 10 monclient(hunting): dump: 2021-05-25T12:02:45.944 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:45.944 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:45.944 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:45.944 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:45.944 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:45.945 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:45.945 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:45.945 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:45.945 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:45.945 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:45.946 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.902+0000 7fb3d0ff9700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:02:45.946 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.902+0000 7fb3d0ff9700 1 -- 172.21.15.102:0/349302262 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3593368874 0 0) 0x7fb3dc113850 con 0x7fb3dc090990 2021-05-25T12:02:45.946 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.902+0000 7fb3d0ff9700 1 -- 172.21.15.102:0/349302262 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb3bc004c10 con 0x7fb3dc090990 2021-05-25T12:02:45.946 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.902+0000 7fb3d0ff9700 1 -- 172.21.15.102:0/349302262 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fb3cc001840 con 0x7fb3dc090990 2021-05-25T12:02:45.947 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.902+0000 7fb3d0ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:45.947 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.902+0000 7fb3d0ff9700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2021-05-25T12:02:45.947 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.902+0000 7fb3d0ff9700 10 monclient(hunting): dump: 2021-05-25T12:02:45.947 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:45.947 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:45.948 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:45.948 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:45.948 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:45.948 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:45.948 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:45.949 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:45.949 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:45.949 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:45.949 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.902+0000 7fb3d0ff9700 1 -- 172.21.15.102:0/349302262 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2672756985 0 0) 0x7fb3dc098890 con 0x7fb3dc090d70 2021-05-25T12:02:45.949 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.902+0000 7fb3d0ff9700 1 -- 172.21.15.102:0/349302262 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb3dc113850 con 0x7fb3dc090d70 2021-05-25T12:02:45.950 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.902+0000 7fb3d0ff9700 1 -- 172.21.15.102:0/349302262 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fb3c4001850 con 0x7fb3dc090d70 2021-05-25T12:02:45.950 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.902+0000 7fb3d0ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:45.950 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.902+0000 7fb3d0ff9700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2021-05-25T12:02:45.950 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.902+0000 7fb3d0ff9700 10 monclient(hunting): dump: 2021-05-25T12:02:45.951 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:45.951 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:45.951 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:45.951 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:45.951 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:45.952 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:45.952 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:45.952 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:45.952 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:45.952 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:45.953 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.902+0000 7fb3d0ff9700 1 -- 172.21.15.102:0/349302262 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 817519634 0 0) 0x7fb3bc004c10 con 0x7fb3dc090990 2021-05-25T12:02:45.953 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.902+0000 7fb3d0ff9700 1 -- 172.21.15.102:0/349302262 >> v1:172.21.15.102:6790/0 conn(0x7fb3dc090d70 legacy=0x7fb3dc12d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:45.953 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.902+0000 7fb3d0ff9700 1 -- 172.21.15.102:0/349302262 >> v1:172.21.15.102:6789/0 conn(0x7fb3dc12e330 legacy=0x7fb3dc140f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:45.953 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.902+0000 7fb3d0ff9700 10 monclient: _finish_hunting 0 2021-05-25T12:02:45.953 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.902+0000 7fb3d0ff9700 1 monclient: found mon.b 2021-05-25T12:02:45.954 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.902+0000 7fb3d0ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:02:45.954 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.902+0000 7fb3d0ff9700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T12:02:45.954 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.902+0000 7fb3d0ff9700 1 -- 172.21.15.102:0/349302262 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3dc12d760 con 0x7fb3dc090990 2021-05-25T12:02:45.954 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.902+0000 7fb3d0ff9700 10 monclient: _finish_auth 0 2021-05-25T12:02:45.954 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.902+0000 7fb3d0ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:02:45.955 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.902+0000 7fb3e2fc6700 5 monclient: authenticate success, global_id 95200 2021-05-25T12:02:45.955 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.902+0000 7fb3e2fc6700 10 monclient: _renew_subs 2021-05-25T12:02:45.955 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.902+0000 7fb3e2fc6700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T12:02:45.955 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.902+0000 7fb3e2fc6700 1 -- 172.21.15.102:0/349302262 --> v1:172.21.15.181:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3dc112dd0 con 0x7fb3dc090990 2021-05-25T12:02:45.955 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.902+0000 7fb3e2fc6700 10 client.95200.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:02:45.956 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.902+0000 7fb3e2fc6700 10 monclient: _renew_subs 2021-05-25T12:02:45.956 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.902+0000 7fb3e2fc6700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T12:02:45.956 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.902+0000 7fb3e2fc6700 1 -- 172.21.15.102:0/349302262 --> v1:172.21.15.181:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3dc12c140 con 0x7fb3dc090990 2021-05-25T12:02:45.956 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.902+0000 7fb3e2fc6700 1 librados: init done 2021-05-25T12:02:45.956 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.902+0000 7fb3d0ff9700 1 -- 172.21.15.102:0/349302262 <== mon.1 v1:172.21.15.181:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb3cc0019f0 con 0x7fb3dc090990 2021-05-25T12:02:45.957 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.902+0000 7fb3d0ff9700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:02:45.957 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.902+0000 7fb3d0ff9700 1 -- 172.21.15.102:0/349302262 <== mon.1 v1:172.21.15.181:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fb3cc001d80 con 0x7fb3dc090990 2021-05-25T12:02:45.957 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.902+0000 7fb3d0ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:45.957 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.902+0000 7fb3d0ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T12:02:45.958 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.902+0000 7fb3d2ffd700 4 set_mon_vals no callback set 2021-05-25T12:02:45.958 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.902+0000 7fb3d0ff9700 10 monclient: dump: 2021-05-25T12:02:45.958 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:45.958 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:45.958 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:45.959 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:45.959 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:45.959 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:45.959 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:45.959 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:45.960 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:45.960 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:45.960 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.903+0000 7fb3e2fc6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:02:45.960 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.903+0000 7fb3e2fc6700 1 -- 172.21.15.102:0/349302262 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fb3b40065a0 con 0x7fb3b4002840 2021-05-25T12:02:45.960 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.903+0000 7fb3d0ff9700 1 -- 172.21.15.102:0/349302262 <== mon.1 v1:172.21.15.181:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7fb3cc001f60 con 0x7fb3dc090990 2021-05-25T12:02:45.961 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.903+0000 7fb3d0ff9700 1 -- 172.21.15.102:0/349302262 <== mon.1 v1:172.21.15.181:6789/0 8 ==== osd_map(84..84 src has 1..84) v4 ==== 6567+0+0 (unknown 932624450 0 0) 0x7fb3cc03b3c0 con 0x7fb3dc090990 2021-05-25T12:02:45.961 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.903+0000 7fb3d0ff9700 10 client.95200.objecter ms_dispatch 0x7fb3dc001550 osd_map(84..84 src has 1..84) v4 2021-05-25T12:02:45.961 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.903+0000 7fb3d0ff9700 3 client.95200.objecter handle_osd_map got epochs [84,84] > 0 2021-05-25T12:02:45.961 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.903+0000 7fb3d0ff9700 3 client.95200.objecter handle_osd_map decoding full epoch 84 2021-05-25T12:02:45.962 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.903+0000 7fb3d0ff9700 20 client.95200.objecter dump_active .. 0 homeless 2021-05-25T12:02:45.962 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.903+0000 7fb3d0ff9700 10 client.95200.objecter ms_handle_connect 0x7fb3b4002840 2021-05-25T12:02:45.962 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.903+0000 7fb3d0ff9700 10 client.95200.objecter resend_mon_ops 2021-05-25T12:02:45.962 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.903+0000 7fb3e1563700 10 monclient: get_auth_request con 0x7fb3bc033520 auth_method 0 2021-05-25T12:02:45.962 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.903+0000 7fb3d0ff9700 1 -- 172.21.15.102:0/349302262 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 996736803 0 0) 0x7fb3b40065a0 con 0x7fb3b4002840 2021-05-25T12:02:45.963 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.903+0000 7fb3d0ff9700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:02:45.963 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.903+0000 7fb3d0ff9700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:02:45.963 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.903+0000 7fb3d0ff9700 10 monclient(hunting): my global_id is 95203 2021-05-25T12:02:45.963 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.903+0000 7fb3d0ff9700 1 -- 172.21.15.102:0/349302262 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3bc004c10 con 0x7fb3b4002840 2021-05-25T12:02:45.964 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.904+0000 7fb3d0ff9700 10 client.95200.objecter ms_handle_connect 0x7fb3bc033520 2021-05-25T12:02:45.964 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.904+0000 7fb3d0ff9700 1 -- 172.21.15.102:0/349302262 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4214579352 0 0) 0x7fb3bc004c10 con 0x7fb3b4002840 2021-05-25T12:02:45.964 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.904+0000 7fb3d0ff9700 1 -- 172.21.15.102:0/349302262 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb3b40065a0 con 0x7fb3b4002840 2021-05-25T12:02:45.964 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.904+0000 7fb3d0ff9700 1 -- 172.21.15.102:0/349302262 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fb3c4004530 con 0x7fb3b4002840 2021-05-25T12:02:45.964 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.904+0000 7fb3d0ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:45.965 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.904+0000 7fb3d0ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T12:02:45.965 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.904+0000 7fb3d0ff9700 10 monclient: dump: 2021-05-25T12:02:45.965 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:45.965 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:45.966 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:45.966 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:45.966 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:45.966 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:45.966 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:45.967 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:45.967 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:45.967 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:45.967 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.904+0000 7fb3d0ff9700 1 -- 172.21.15.102:0/349302262 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2939979555 0 0) 0x7fb3b40065a0 con 0x7fb3b4002840 2021-05-25T12:02:45.967 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.904+0000 7fb3d0ff9700 1 -- 172.21.15.102:0/349302262 --> v1:172.21.15.181:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb3b4009e40 con 0x7fb3b4002840 2021-05-25T12:02:45.968 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.904+0000 7fb3d0ff9700 1 -- 172.21.15.102:0/349302262 <== mon.1 v1:172.21.15.181:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7fb3c40067d0 con 0x7fb3b4002840 2021-05-25T12:02:45.968 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.904+0000 7fb3d0ff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:02:45.968 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.904+0000 7fb3d0ff9700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:02:45.968 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.904+0000 7fb3d0ff9700 1 -- 172.21.15.102:0/349302262 >> v1:172.21.15.181:6789/0 conn(0x7fb3b4002840 legacy=0x7fb3b4004ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:45.969 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.904+0000 7fb3d0ff9700 1 -- 172.21.15.102:0/349302262 >> v1:172.21.15.181:6789/0 conn(0x7fb3b4002840 legacy=0x7fb3b4004ce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:02:45.969 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.911+0000 7fb3e2fc6700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:02:45.969 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.911+0000 7fb3e2fc6700 1 -- 172.21.15.102:0/349302262 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fb3b4009b00 con 0x7fb3b4008dc0 2021-05-25T12:02:45.969 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.911+0000 7fb3d0ff9700 10 client.95200.objecter ms_handle_connect 0x7fb3b4008dc0 2021-05-25T12:02:45.969 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.911+0000 7fb3d0ff9700 10 client.95200.objecter resend_mon_ops 2021-05-25T12:02:45.970 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.911+0000 7fb3d0ff9700 1 -- 172.21.15.102:0/349302262 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4054434352 0 0) 0x7fb3b4009b00 con 0x7fb3b4008dc0 2021-05-25T12:02:45.970 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.911+0000 7fb3d0ff9700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:02:45.970 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.911+0000 7fb3d0ff9700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:02:45.970 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.911+0000 7fb3d0ff9700 10 monclient(hunting): my global_id is 95206 2021-05-25T12:02:45.970 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.911+0000 7fb3d0ff9700 1 -- 172.21.15.102:0/349302262 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3b40065a0 con 0x7fb3b4008dc0 2021-05-25T12:02:45.971 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.911+0000 7fb3d0ff9700 1 -- 172.21.15.102:0/349302262 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2200747652 0 0) 0x7fb3b40065a0 con 0x7fb3b4008dc0 2021-05-25T12:02:45.971 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.911+0000 7fb3d0ff9700 1 -- 172.21.15.102:0/349302262 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb3b4009b00 con 0x7fb3b4008dc0 2021-05-25T12:02:45.971 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.911+0000 7fb3d0ff9700 1 -- 172.21.15.102:0/349302262 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fb3c4004c50 con 0x7fb3b4008dc0 2021-05-25T12:02:45.971 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.911+0000 7fb3d0ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:02:45.971 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.911+0000 7fb3d0ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T12:02:45.972 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.911+0000 7fb3d0ff9700 10 monclient: dump: 2021-05-25T12:02:45.972 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:02:45.972 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:02:45.972 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:45.972 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:02:45.973 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:02:45.973 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:02:45.973 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:02:45.973 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:02:45.973 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:02:45.974 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:02:45.974 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.911+0000 7fb3d0ff9700 1 -- 172.21.15.102:0/349302262 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1157207068 0 0) 0x7fb3b4009b00 con 0x7fb3b4008dc0 2021-05-25T12:02:45.974 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.911+0000 7fb3d0ff9700 1 -- 172.21.15.102:0/349302262 --> v1:172.21.15.181:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fb3b4008740 con 0x7fb3b4008dc0 2021-05-25T12:02:45.974 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.912+0000 7fb3d0ff9700 1 -- 172.21.15.102:0/349302262 <== mon.1 v1:172.21.15.181:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3940 (unknown 0 0 2655861543) 0x7fb3c4006a60 con 0x7fb3b4008dc0 2021-05-25T12:02:45.975 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.912+0000 7fb3d0ff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:02:45.975 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.912+0000 7fb3d0ff9700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:02:45.975 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.912+0000 7fb3d0ff9700 1 -- 172.21.15.102:0/349302262 >> v1:172.21.15.181:6789/0 conn(0x7fb3b4008dc0 legacy=0x7fb3b4009250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:45.975 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.912+0000 7fb3d0ff9700 1 -- 172.21.15.102:0/349302262 >> v1:172.21.15.181:6789/0 conn(0x7fb3b4008dc0 legacy=0x7fb3b4009250 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:02:45.976 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:02:45.976 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:02:45.976 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:02:45.977 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "peon", 2021-05-25T12:02:45.977 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 30, 2021-05-25T12:02:45.977 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:02:45.977 INFO:teuthology.orchestra.run.smithi102.stdout: 0, 2021-05-25T12:02:45.977 INFO:teuthology.orchestra.run.smithi102.stdout: 1, 2021-05-25T12:02:45.978 INFO:teuthology.orchestra.run.smithi102.stdout: 2 2021-05-25T12:02:45.978 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:02:45.978 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 4, 2021-05-25T12:02:45.978 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:02:45.978 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:02:45.979 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:02:45.979 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:02:45.979 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:02:45.979 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:02:45.980 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:02:45.980 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:02:45.980 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:02:45.980 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:02:45.980 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:02:45.981 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:02:45.981 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:02:45.981 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:02:45.981 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:02:45.981 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:02:45.982 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:02:45.982 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:02:45.982 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:02:45.982 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:02:45.982 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:02:45.983 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:02:45.983 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:02:45.983 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:02:45.983 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:02:45.984 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:02:45.984 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:02:45.984 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:02:45.984 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:02:45.984 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:02:45.985 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:02:45.985 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:02:45.985 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:02:45.985 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:02:45.985 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:02:45.986 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:02:45.986 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:02:45.986 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:02:45.986 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:02:45.986 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:02:45.987 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:02:45.987 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:02:45.987 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:02:45.987 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:02:45.987 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:02:45.988 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:02:45.988 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:02:45.988 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:02:45.988 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:02:45.988 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:02:45.989 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:02:45.989 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:02:45.989 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:02:45.989 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:02:45.989 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:02:45.990 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:02:45.990 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:02:45.990 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:02:45.990 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:02:45.990 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:02:45.991 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:02:45.991 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:02:45.991 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:02:45.991 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:02:45.992 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:02:45.992 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:02:45.992 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:02:45.992 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:02:45.992 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:02:45.993 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:02:45.993 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:02:45.993 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:02:45.993 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:02:45.993 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:02:45.994 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:02:45.994 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:02:45.994 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:02:45.994 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:02:45.994 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:02:45.995 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:02:45.995 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:02:45.995 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:02:45.995 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:02:45.996 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:02:45.996 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:02:45.996 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:02:45.996 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:02:45.996 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:02:45.996 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:02:45.997 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:02:45.997 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:02:45.997 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:02:45.997 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:02:45.998 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:02:45.998 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:02:45.998 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:02:45.998 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:02:45.998 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:02:45.999 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:02:45.999 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:02:45.999 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:02:45.999 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:02:45.999 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:02:46.000 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:02:46.000 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:02:46.000 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:02:46.000 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:02:46.000 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:02:46.001 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:02:46.001 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:02:46.001 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:02:46.001 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:02:46.001 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:02:46.002 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:02:46.002 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:02:46.002 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:02:46.002 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:02:46.003 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:02:46.003 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": [ 2021-05-25T12:02:46.003 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:02:46.003 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:02:46.003 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:02:46.004 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 3 2021-05-25T12:02:46.004 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:02:46.004 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:02:46.004 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:02:46.004 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:02:46.005 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:02:46.005 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:02:46.005 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 4 2021-05-25T12:02:46.005 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:02:46.005 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:02:46.006 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:02:46.006 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:02:46.006 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:02:46.008 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.912+0000 7fb3e2fc6700 10 librados: watch_flush enter 2021-05-25T12:02:46.009 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.912+0000 7fb3e2fc6700 10 librados: watch_flush exit 2021-05-25T12:02:46.009 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.912+0000 7fb3e2fc6700 20 client.95200.objecter shutdown clearing up homeless session... 2021-05-25T12:02:46.009 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.912+0000 7fb3e2fc6700 10 client.95200.objecter successfully canceled tick 2021-05-25T12:02:46.009 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.912+0000 7fb3e2fc6700 1 -- 172.21.15.102:0/349302262 >> v1:172.21.15.181:6812/29582 conn(0x7fb3bc033520 legacy=0x7fb3bc0359e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:46.010 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.912+0000 7fb3e2fc6700 10 monclient: shutdown 2021-05-25T12:02:46.010 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.912+0000 7fb3e2fc6700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:02:46.010 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.912+0000 7fb3e2fc6700 1 -- 172.21.15.102:0/349302262 >> v1:172.21.15.181:6789/0 conn(0x7fb3dc090990 legacy=0x7fb3dc12c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:02:46.010 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.912+0000 7fb3e1563700 1 -- 172.21.15.102:0/349302262 reap_dead start 2021-05-25T12:02:46.010 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.912+0000 7fb3e1563700 1 -- 172.21.15.102:0/349302262 reap_dead start 2021-05-25T12:02:46.011 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.912+0000 7fb3e2fc6700 1 -- 172.21.15.102:0/349302262 shutdown_connections 2021-05-25T12:02:46.011 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.912+0000 7fb3e2fc6700 1 -- 172.21.15.102:0/349302262 >> 172.21.15.102:0/349302262 conn(0x7fb3dc09b370 msgr2=0x7fb3dc093a60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:02:46.011 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.912+0000 7fb3e2fc6700 1 -- 172.21.15.102:0/349302262 shutdown_connections 2021-05-25T12:02:46.011 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.912+0000 7fb3e2fc6700 1 -- 172.21.15.102:0/349302262 wait complete. 2021-05-25T12:02:46.011 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:02:45.912+0000 7fb3e2fc6700 1 librados: shutdown 2021-05-25T12:02:46.012 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2021-05-25T12:02:46.012 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['b'] 2021-05-25T12:02:46.013 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2021-05-25T12:02:46.013 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2021-05-25T12:02:46.013 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2021-05-25T12:02:46.014 INFO:teuthology.orchestra.run:waiting for 300 2021-05-25T12:02:46.024 INFO:tasks.ceph.mon.c:Stopped 2021-05-25T12:02:46.024 INFO:tasks.mon_thrash:Sending STOP to mon b 2021-05-25T12:02:46.025 INFO:tasks.ceph.mon.b:Sent signal 19 2021-05-25T12:02:46.025 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs to unfreeze mons 2021-05-25T12:03:06.026 INFO:tasks.mon_thrash:Sending CONT to mon b 2021-05-25T12:03:06.027 INFO:tasks.ceph.mon.b:Sent signal 18 2021-05-25T12:03:06.027 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2021-05-25T12:03:06.028 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2021-05-25T12:03:06.151 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:06.150+0000 7fbdcdb04700 10 monclient: get_monmap_and_config 2021-05-25T12:03:06.152 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:06.150+0000 7fbdcdb04700 10 monclient: build_initial_monmap 2021-05-25T12:03:06.152 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:06.150+0000 7fbdcdb04700 10 monclient: monmap: 2021-05-25T12:03:06.152 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:03:06.152 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:03:06.153 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:03:06.151276+0000 2021-05-25T12:03:06.153 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:03:06.151276+0000 2021-05-25T12:03:06.153 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:03:06.153 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:03:06.154 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:03:06.154 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:03:06.154 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:03:06.154 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:03:06.156 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:06.151+0000 7fbdcdb04700 1 Processor -- start 2021-05-25T12:03:06.156 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:06.151+0000 7fbdcdb04700 1 -- start start 2021-05-25T12:03:06.156 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:06.151+0000 7fbdcdb04700 10 monclient: init 2021-05-25T12:03:06.157 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:06.151+0000 7fbdcdb04700 10 monclient: _reopen_session rank -1 2021-05-25T12:03:06.157 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:06.151+0000 7fbdcdb04700 10 monclient: _add_conns ranks=[1,0,2] 2021-05-25T12:03:06.157 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:06.151+0000 7fbdcdb04700 10 monclient(hunting): picked mon.noname-c con 0x7fbdc8093e10 addr v1:172.21.15.181:6789/0 2021-05-25T12:03:06.157 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:06.151+0000 7fbdcdb04700 10 monclient(hunting): picked mon.noname-a con 0x7fbdc8091b00 addr v1:172.21.15.102:6789/0 2021-05-25T12:03:06.158 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:06.151+0000 7fbdcdb04700 10 monclient(hunting): picked mon.noname-b con 0x7fbdc8094900 addr v1:172.21.15.102:6790/0 2021-05-25T12:03:06.158 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:06.151+0000 7fbdcdb04700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdc8092a90 con 0x7fbdc8091b00 2021-05-25T12:03:06.158 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:06.151+0000 7fbdcdb04700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdc8092d30 con 0x7fbdc8093e10 2021-05-25T12:03:06.158 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:06.151+0000 7fbdcdb04700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdc8092fd0 con 0x7fbdc8094900 2021-05-25T12:03:06.158 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:06.151+0000 7fbdcdb04700 10 monclient(hunting): _renew_subs 2021-05-25T12:03:06.159 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:06.151+0000 7fbdcdb04700 10 monclient(hunting): authenticate will time out at 986.748650s 2021-05-25T12:03:06.159 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:06.151+0000 7fbdc7fff700 1 -- >> v1:172.21.15.102:6790/0 conn(0x7fbdc8094900 legacy=0x7fbdc8130130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6790/0 2021-05-25T12:03:06.159 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:06.151+0000 7fbdc6ffd700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fbdc8091b00 0x7fbdc80941f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60388/0 (socket says 172.21.15.102:60388) 2021-05-25T12:03:06.159 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:06.151+0000 7fbdc6ffd700 1 -- 172.21.15.102:0/1001455757 learned_addr learned my addr 172.21.15.102:0/1001455757 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:03:06.353 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:06.352+0000 7fbdc7fff700 1 -- 172.21.15.102:0/1001455757 >> v1:172.21.15.102:6790/0 conn(0x7fbdc8094900 legacy=0x7fbdc8130130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6790/0 2021-05-25T12:03:06.754 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:06.753+0000 7fbdc7fff700 1 -- 172.21.15.102:0/1001455757 >> v1:172.21.15.102:6790/0 conn(0x7fbdc8094900 legacy=0x7fbdc8130130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6790/0 2021-05-25T12:03:07.556 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:07.554+0000 7fbdc7fff700 1 -- 172.21.15.102:0/1001455757 >> v1:172.21.15.102:6790/0 conn(0x7fbdc8094900 legacy=0x7fbdc8130130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6790/0 2021-05-25T12:03:09.153 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:09.151+0000 7fbdc57fa700 10 monclient(hunting): tick 2021-05-25T12:03:09.154 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:09.151+0000 7fbdc57fa700 1 monclient(hunting): continuing hunt 2021-05-25T12:03:09.154 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:09.151+0000 7fbdc57fa700 10 monclient(hunting): _reopen_session rank -1 2021-05-25T12:03:09.154 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:09.151+0000 7fbdc57fa700 1 -- 172.21.15.102:0/1001455757 >> v1:172.21.15.102:6790/0 conn(0x7fbdc8094900 legacy=0x7fbdc8130130 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-05-25T12:03:09.154 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:09.151+0000 7fbdc57fa700 1 -- 172.21.15.102:0/1001455757 >> v1:172.21.15.181:6789/0 conn(0x7fbdc8093e10 legacy=0x7fbdc8093550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:03:09.155 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:09.151+0000 7fbdc57fa700 1 -- 172.21.15.102:0/1001455757 >> v1:172.21.15.102:6789/0 conn(0x7fbdc8091b00 legacy=0x7fbdc80941f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:03:09.155 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:09.151+0000 7fbdc57fa700 10 monclient: _add_conns ranks=[2,1,0] 2021-05-25T12:03:09.155 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:09.151+0000 7fbdc57fa700 10 monclient(hunting): picked mon.noname-b con 0x7fbdc8094900 addr v1:172.21.15.102:6790/0 2021-05-25T12:03:09.155 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:09.151+0000 7fbdc57fa700 10 monclient(hunting): picked mon.noname-c con 0x7fbdc8093e10 addr v1:172.21.15.181:6789/0 2021-05-25T12:03:09.155 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:09.151+0000 7fbdc57fa700 10 monclient(hunting): picked mon.noname-a con 0x7fbdc8091b00 addr v1:172.21.15.102:6789/0 2021-05-25T12:03:09.156 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:09.151+0000 7fbdc57fa700 1 -- 172.21.15.102:0/1001455757 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdc8134880 con 0x7fbdc8091b00 2021-05-25T12:03:09.156 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:09.151+0000 7fbdc57fa700 1 -- 172.21.15.102:0/1001455757 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdc8092fd0 con 0x7fbdc8093e10 2021-05-25T12:03:09.156 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:09.151+0000 7fbdc57fa700 1 -- 172.21.15.102:0/1001455757 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdb000fb80 con 0x7fbdc8094900 2021-05-25T12:03:09.156 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:09.151+0000 7fbdc57fa700 10 monclient(hunting): _renew_subs 2021-05-25T12:03:09.156 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:09.151+0000 7fbdc77fe700 1 -- 172.21.15.102:0/1001455757 >> v1:172.21.15.102:6790/0 conn(0x7fbdc8094900 legacy=0x7fbdb0005430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6790/0 2021-05-25T12:03:09.353 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:09.352+0000 7fbdc77fe700 1 -- 172.21.15.102:0/1001455757 >> v1:172.21.15.102:6790/0 conn(0x7fbdc8094900 legacy=0x7fbdb0005430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6790/0 2021-05-25T12:03:09.754 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:09.753+0000 7fbdc77fe700 1 -- 172.21.15.102:0/1001455757 >> v1:172.21.15.102:6790/0 conn(0x7fbdc8094900 legacy=0x7fbdb0005430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6790/0 2021-05-25T12:03:10.555 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:10.554+0000 7fbdc77fe700 1 -- 172.21.15.102:0/1001455757 >> v1:172.21.15.102:6790/0 conn(0x7fbdc8094900 legacy=0x7fbdb0005430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6790/0 2021-05-25T12:03:10.711 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:10.710+0000 7fbdc7fff700 1 -- 172.21.15.102:0/1001455757 >> v1:172.21.15.102:6789/0 conn(0x7fbdc8091b00 legacy=0x7fbdb000c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2021-05-25T12:03:10.711 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:10.710+0000 7fbdc7fff700 1 -- 172.21.15.102:0/1001455757 >> v1:172.21.15.102:6789/0 conn(0x7fbdc8091b00 legacy=0x7fbdb000c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-05-25T12:03:10.712 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:10.710+0000 7fbdc7fff700 1 --1- 172.21.15.102:0/1001455757 >> v1:172.21.15.102:6789/0 conn(0x7fbdc8091b00 0x7fbdb000c390 :-1 s=OPENED pgs=271 cs=1 l=1).handle_message read tag failed 2021-05-25T12:03:10.712 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:10.710+0000 7fbdc7fff700 1 --1- 172.21.15.102:0/1001455757 >> v1:172.21.15.102:6789/0 conn(0x7fbdc8091b00 0x7fbdb000c390 :-1 s=OPENED pgs=271 cs=1 l=1).fault on lossy channel, failing 2021-05-25T12:03:10.712 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:10.710+0000 7fbdc67fc700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.102:6789/0 2021-05-25T12:03:11.047 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.045+0000 7fbdc67fc700 1 -- 172.21.15.102:0/1001455757 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2840059668 0 0) 0x7fbdc8092fd0 con 0x7fbdc8093e10 2021-05-25T12:03:11.048 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.045+0000 7fbdc67fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:03:11.048 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.045+0000 7fbdc67fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:03:11.048 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.045+0000 7fbdc67fc700 10 monclient(hunting): my global_id is 95209 2021-05-25T12:03:11.049 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.045+0000 7fbdc67fc700 1 -- 172.21.15.102:0/1001455757 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdac003270 con 0x7fbdc8093e10 2021-05-25T12:03:11.063 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.048+0000 7fbdc67fc700 1 -- 172.21.15.102:0/1001455757 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 694711214 0 0) 0x7fbdac003270 con 0x7fbdc8093e10 2021-05-25T12:03:11.063 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.048+0000 7fbdc67fc700 1 -- 172.21.15.102:0/1001455757 >> v1:172.21.15.102:6790/0 conn(0x7fbdc8094900 legacy=0x7fbdb0005430 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-05-25T12:03:11.064 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.048+0000 7fbdc67fc700 1 -- 172.21.15.102:0/1001455757 >> v1:172.21.15.102:6789/0 conn(0x7fbdc8091b00 legacy=0x7fbdb000c390 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:03:11.064 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.048+0000 7fbdc67fc700 10 monclient: _finish_hunting 0 2021-05-25T12:03:11.065 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.048+0000 7fbdc67fc700 1 monclient: found mon.noname-c 2021-05-25T12:03:11.065 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.048+0000 7fbdc67fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:03:11.066 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.048+0000 7fbdc67fc700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.181:6789/0 2021-05-25T12:03:11.067 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.048+0000 7fbdc67fc700 1 -- 172.21.15.102:0/1001455757 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdb0010df0 con 0x7fbdc8093e10 2021-05-25T12:03:11.067 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.048+0000 7fbdc67fc700 10 monclient: _finish_auth 0 2021-05-25T12:03:11.067 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.048+0000 7fbdc67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:03:11.067 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.048+0000 7fbdc67fc700 1 -- 172.21.15.102:0/1001455757 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fbdb4001590 con 0x7fbdc8093e10 2021-05-25T12:03:11.068 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.048+0000 7fbdcdb04700 5 monclient: authenticate success, global_id 95209 2021-05-25T12:03:11.068 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.048+0000 7fbdcdb04700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:03:11.068 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.048+0000 7fbdc67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:03:11.068 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.048+0000 7fbdc67fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:03:11.068 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.048+0000 7fbdc67fc700 10 monclient: dump: 2021-05-25T12:03:11.069 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:03:11.069 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:03:11.069 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:03:11.069 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:03:11.069 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:03:11.070 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:03:11.070 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:03:11.070 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:03:11.070 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:03:11.071 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:03:11.071 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.048+0000 7fbdcdb04700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:03:11.071 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.050+0000 7fbdc67fc700 1 -- 172.21.15.102:0/1001455757 <== mon.1 v1:172.21.15.181:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbdb40017b0 con 0x7fbdc8093e10 2021-05-25T12:03:11.071 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.050+0000 7fbdc67fc700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:03:11.071 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.050+0000 7fbdcdb04700 10 monclient: get_monmap_and_config success 2021-05-25T12:03:11.072 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.050+0000 7fbdc67fc700 1 -- 172.21.15.102:0/1001455757 <== mon.1 v1:172.21.15.181:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fbdb4001bc0 con 0x7fbdc8093e10 2021-05-25T12:03:11.072 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.050+0000 7fbdc67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:03:11.072 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.050+0000 7fbdcdb04700 4 set_mon_vals no callback set 2021-05-25T12:03:11.072 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.050+0000 7fbdc67fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T12:03:11.072 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.050+0000 7fbdc67fc700 10 monclient: dump: 2021-05-25T12:03:11.073 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:03:11.073 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:03:11.073 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:03:11.073 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:03:11.073 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:03:11.074 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:03:11.074 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:03:11.074 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:03:11.074 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:03:11.075 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:03:11.075 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.051+0000 7fbdcdb04700 10 monclient: shutdown 2021-05-25T12:03:11.075 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.051+0000 7fbdcdb04700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:03:11.075 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.051+0000 7fbdcdb04700 1 -- 172.21.15.102:0/1001455757 >> v1:172.21.15.181:6789/0 conn(0x7fbdc8093e10 legacy=0x7fbdb0008be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:03:11.075 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.051+0000 7fbdcdb04700 1 -- 172.21.15.102:0/1001455757 shutdown_connections 2021-05-25T12:03:11.076 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.051+0000 7fbdcdb04700 1 -- 172.21.15.102:0/1001455757 >> 172.21.15.102:0/1001455757 conn(0x7fbdc809a910 msgr2=0x7fbdc809ad20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:03:11.076 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.051+0000 7fbdcdb04700 1 -- 172.21.15.102:0/1001455757 shutdown_connections 2021-05-25T12:03:11.076 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.051+0000 7fbdcdb04700 1 -- 172.21.15.102:0/1001455757 wait complete. 2021-05-25T12:03:11.076 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.052+0000 7fbdcdb04700 10 monclient: build_initial_monmap 2021-05-25T12:03:11.076 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.052+0000 7fbdcdb04700 10 monclient: monmap: 2021-05-25T12:03:11.077 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:03:11.077 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:03:11.077 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:03:11.077 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:03:11.077 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:03:11.078 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:03:11.078 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:03:11.078 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:03:11.078 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:03:11.078 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:03:11.079 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.052+0000 7fbdcdb04700 1 librados: starting msgr at 2021-05-25T12:03:11.079 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.053+0000 7fbdcdb04700 1 librados: starting objecter 2021-05-25T12:03:11.079 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.053+0000 7fbdcdb04700 1 Processor -- start 2021-05-25T12:03:11.079 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.053+0000 7fbdcdb04700 1 -- start start 2021-05-25T12:03:11.079 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.053+0000 7fbdcdb04700 1 librados: setting wanted keys 2021-05-25T12:03:11.080 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.053+0000 7fbdcdb04700 1 librados: calling monclient init 2021-05-25T12:03:11.080 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.053+0000 7fbdcdb04700 10 monclient: init 2021-05-25T12:03:11.080 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.053+0000 7fbdcdb04700 10 monclient: _reopen_session rank -1 2021-05-25T12:03:11.080 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.053+0000 7fbdcdb04700 10 monclient: _add_conns ranks=[0,2,1] 2021-05-25T12:03:11.080 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.053+0000 7fbdcdb04700 10 monclient(hunting): picked mon.noname-a con 0x7fbdc8091b00 addr v1:172.21.15.102:6789/0 2021-05-25T12:03:11.081 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.054+0000 7fbdcdb04700 10 monclient(hunting): picked mon.noname-c con 0x7fbdc8094900 addr v1:172.21.15.102:6790/0 2021-05-25T12:03:11.081 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.054+0000 7fbdcdb04700 10 monclient(hunting): picked mon.noname-b con 0x7fbdc81340a0 addr v1:172.21.15.181:6789/0 2021-05-25T12:03:11.081 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.054+0000 7fbdcdb04700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdc8139ad0 con 0x7fbdc8091b00 2021-05-25T12:03:11.081 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.054+0000 7fbdcdb04700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdc81314d0 con 0x7fbdc81340a0 2021-05-25T12:03:11.081 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.054+0000 7fbdcdb04700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdc8092220 con 0x7fbdc8094900 2021-05-25T12:03:11.082 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.054+0000 7fbdcdb04700 10 monclient(hunting): _renew_subs 2021-05-25T12:03:11.082 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.054+0000 7fbdcdb04700 10 monclient(hunting): authenticate will time out at 991.651064s 2021-05-25T12:03:11.082 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.054+0000 7fbdc6ffd700 1 -- >> v1:172.21.15.102:6790/0 conn(0x7fbdc8094900 legacy=0x7fbdc8133990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.102:6790/0 2021-05-25T12:03:11.082 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.054+0000 7fbdc77fe700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fbdc8091b00 0x7fbdc8138c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60442/0 (socket says 172.21.15.102:60442) 2021-05-25T12:03:11.082 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.054+0000 7fbdc77fe700 1 -- 172.21.15.102:0/2689717934 learned_addr learned my addr 172.21.15.102:0/2689717934 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:03:11.083 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.054+0000 7fbdab7fe700 10 client.?.objecter ms_handle_connect 0x7fbdc8091b00 2021-05-25T12:03:11.083 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.054+0000 7fbdab7fe700 10 client.?.objecter resend_mon_ops 2021-05-25T12:03:11.083 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.054+0000 7fbdab7fe700 1 -- 172.21.15.102:0/2689717934 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3186959841 0 0) 0x7fbdc8139ad0 con 0x7fbdc8091b00 2021-05-25T12:03:11.083 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.054+0000 7fbdab7fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:03:11.083 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.054+0000 7fbdab7fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:03:11.084 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.054+0000 7fbdab7fe700 10 monclient(hunting): my global_id is 174249 2021-05-25T12:03:11.084 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.054+0000 7fbdab7fe700 1 -- 172.21.15.102:0/2689717934 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdb00065c0 con 0x7fbdc8091b00 2021-05-25T12:03:11.084 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.054+0000 7fbdab7fe700 1 -- 172.21.15.102:0/2689717934 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 430828737 0 0) 0x7fbdb00065c0 con 0x7fbdc8091b00 2021-05-25T12:03:11.084 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.054+0000 7fbdab7fe700 1 -- 172.21.15.102:0/2689717934 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbdc8139ad0 con 0x7fbdc8091b00 2021-05-25T12:03:11.085 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.054+0000 7fbdab7fe700 1 -- 172.21.15.102:0/2689717934 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fbdb8002960 con 0x7fbdc8091b00 2021-05-25T12:03:11.085 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.054+0000 7fbdab7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:03:11.085 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.054+0000 7fbdab7fe700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:03:11.085 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.054+0000 7fbdab7fe700 10 monclient(hunting): dump: 2021-05-25T12:03:11.085 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:03:11.086 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:03:11.086 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:03:11.086 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:03:11.086 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:03:11.086 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:03:11.087 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:03:11.087 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:03:11.087 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:03:11.087 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:03:11.087 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.054+0000 7fbdab7fe700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:03:11.088 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.055+0000 7fbdab7fe700 1 -- 172.21.15.102:0/2689717934 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 116353534 0 0) 0x7fbdc8139ad0 con 0x7fbdc8091b00 2021-05-25T12:03:11.088 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.055+0000 7fbdab7fe700 1 -- 172.21.15.102:0/2689717934 >> v1:172.21.15.102:6790/0 conn(0x7fbdc8094900 legacy=0x7fbdc8133990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-05-25T12:03:11.088 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.055+0000 7fbdab7fe700 1 -- 172.21.15.102:0/2689717934 >> v1:172.21.15.181:6789/0 conn(0x7fbdc81340a0 legacy=0x7fbdc8140f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:03:11.088 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.055+0000 7fbdab7fe700 10 monclient: _finish_hunting 0 2021-05-25T12:03:11.089 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.055+0000 7fbdab7fe700 1 monclient: found mon.a 2021-05-25T12:03:11.089 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.055+0000 7fbdab7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:03:11.089 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.055+0000 7fbdab7fe700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:03:11.089 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.055+0000 7fbdab7fe700 1 -- 172.21.15.102:0/2689717934 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdc8068250 con 0x7fbdc8091b00 2021-05-25T12:03:11.089 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.055+0000 7fbdab7fe700 10 monclient: _finish_auth 0 2021-05-25T12:03:11.090 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.055+0000 7fbdab7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:03:11.090 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.055+0000 7fbdcdb04700 5 monclient: authenticate success, global_id 174249 2021-05-25T12:03:11.090 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.055+0000 7fbdcdb04700 10 monclient: _renew_subs 2021-05-25T12:03:11.090 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.055+0000 7fbdcdb04700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:03:11.090 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.055+0000 7fbdcdb04700 1 -- 172.21.15.102:0/2689717934 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdc812ff10 con 0x7fbdc8091b00 2021-05-25T12:03:11.091 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.055+0000 7fbdcdb04700 10 client.174249.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:03:11.091 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.055+0000 7fbdcdb04700 10 monclient: _renew_subs 2021-05-25T12:03:11.091 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.055+0000 7fbdcdb04700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:03:11.091 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.055+0000 7fbdcdb04700 1 -- 172.21.15.102:0/2689717934 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdc8142670 con 0x7fbdc8091b00 2021-05-25T12:03:11.091 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.055+0000 7fbdcdb04700 1 librados: init done 2021-05-25T12:03:11.092 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.055+0000 7fbdab7fe700 1 -- 172.21.15.102:0/2689717934 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbdb8001690 con 0x7fbdc8091b00 2021-05-25T12:03:11.092 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.055+0000 7fbdab7fe700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:03:11.092 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.055+0000 7fbdab7fe700 1 -- 172.21.15.102:0/2689717934 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fbdb8002db0 con 0x7fbdc8091b00 2021-05-25T12:03:11.092 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.055+0000 7fbdab7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:03:11.092 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.055+0000 7fbdc5ffb700 4 set_mon_vals no callback set 2021-05-25T12:03:11.093 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.055+0000 7fbdab7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:03:11.093 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.055+0000 7fbdab7fe700 10 monclient: dump: 2021-05-25T12:03:11.093 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:03:11.093 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:03:11.093 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:03:11.094 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:03:11.094 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:03:11.094 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:03:11.094 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:03:11.095 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:03:11.095 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:03:11.095 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:03:11.095 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.055+0000 7fbdab7fe700 1 -- 172.21.15.102:0/2689717934 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7fbdb8001aa0 con 0x7fbdc8091b00 2021-05-25T12:03:11.095 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.056+0000 7fbdcdb04700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:03:11.096 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.056+0000 7fbdcdb04700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:03:11.096 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.056+0000 7fbdcdb04700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:03:11.096 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.056+0000 7fbdab7fe700 1 -- 172.21.15.102:0/2689717934 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(84..84 src has 1..84) v4 ==== 6567+0+0 (unknown 932624450 0 0) 0x7fbdb803db90 con 0x7fbdc8091b00 2021-05-25T12:03:11.096 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.056+0000 7fbdab7fe700 10 client.174249.objecter ms_dispatch 0x7fbdc8001550 osd_map(84..84 src has 1..84) v4 2021-05-25T12:03:11.096 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.056+0000 7fbdcdb04700 1 -- 172.21.15.102:0/2689717934 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbda4004db0 con 0x7fbdc8091b00 2021-05-25T12:03:11.097 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.056+0000 7fbdab7fe700 3 client.174249.objecter handle_osd_map got epochs [84,84] > 0 2021-05-25T12:03:11.097 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.056+0000 7fbdab7fe700 3 client.174249.objecter handle_osd_map decoding full epoch 84 2021-05-25T12:03:11.097 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.056+0000 7fbdab7fe700 20 client.174249.objecter dump_active .. 0 homeless 2021-05-25T12:03:11.097 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.056+0000 7fbdc6ffd700 10 monclient: get_auth_request con 0x7fbdb00334e0 auth_method 0 2021-05-25T12:03:11.097 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.056+0000 7fbdab7fe700 10 client.174249.objecter ms_handle_connect 0x7fbdb00334e0 2021-05-25T12:03:11.098 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.060+0000 7fbdab7fe700 1 -- 172.21.15.102:0/2689717934 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7fbdb8012ec0 con 0x7fbdc8091b00 2021-05-25T12:03:11.098 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.060+0000 7fbdab7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:03:11.098 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.060+0000 7fbdab7fe700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:03:11.303 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.301+0000 7fbdcdb04700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2021-05-25T12:03:11.303 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.301+0000 7fbdcdb04700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2021-05-25T12:03:11.304 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.301+0000 7fbdcdb04700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:03:11.304 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.301+0000 7fbdcdb04700 1 -- 172.21.15.102:0/2689717934 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fbda4005060 con 0x7fbdc8091b00 2021-05-25T12:03:11.304 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.302+0000 7fbdab7fe700 1 -- 172.21.15.102:0/2689717934 <== mon.0 v1:172.21.15.102:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1336 (unknown 254746657 0 3626982539) 0x7fbdb8012ab0 con 0x7fbdc8091b00 2021-05-25T12:03:11.304 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.302+0000 7fbdab7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2021-05-25T12:03:11.305 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.302+0000 7fbdab7fe700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:03:11.305 INFO:teuthology.orchestra.run.smithi102.stdout:{"election_epoch":32,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138303579357183","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"20ee91b6-8935-4b98-b868-85c03789ccc4","modified":"2021-05-25T11:59:11.011522Z","created":"2021-05-25T11:59:11.011522Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6790","nonce":0}]},"addr":"172.21.15.102:6790/0","public_addr":"172.21.15.102:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2021-05-25T12:03:11.306 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.303+0000 7fbdcdb04700 10 librados: watch_flush enter 2021-05-25T12:03:11.307 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.303+0000 7fbdcdb04700 10 librados: watch_flush exit 2021-05-25T12:03:11.307 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.303+0000 7fbdcdb04700 20 client.174249.objecter shutdown clearing up homeless session... 2021-05-25T12:03:11.307 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.303+0000 7fbdcdb04700 10 client.174249.objecter successfully canceled tick 2021-05-25T12:03:11.307 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.303+0000 7fbdcdb04700 1 -- 172.21.15.102:0/2689717934 >> v1:172.21.15.181:6812/29582 conn(0x7fbdb00334e0 legacy=0x7fbdb00359a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:03:11.307 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.303+0000 7fbdcdb04700 10 monclient: shutdown 2021-05-25T12:03:11.308 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.303+0000 7fbdcdb04700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:03:11.308 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.303+0000 7fbdcdb04700 1 -- 172.21.15.102:0/2689717934 >> v1:172.21.15.102:6789/0 conn(0x7fbdc8091b00 legacy=0x7fbdc8138c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:03:11.308 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.303+0000 7fbdcdb04700 1 -- 172.21.15.102:0/2689717934 shutdown_connections 2021-05-25T12:03:11.308 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.303+0000 7fbdcdb04700 1 -- 172.21.15.102:0/2689717934 >> 172.21.15.102:0/2689717934 conn(0x7fbdc809a910 msgr2=0x7fbdc8095260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:03:11.309 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.303+0000 7fbdcdb04700 1 -- 172.21.15.102:0/2689717934 shutdown_connections 2021-05-25T12:03:11.309 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.304+0000 7fbdcdb04700 1 -- 172.21.15.102:0/2689717934 wait complete. 2021-05-25T12:03:11.309 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.304+0000 7fbdcdb04700 1 librados: shutdown 2021-05-25T12:03:11.316 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2021-05-25T12:03:11.317 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2021-05-25T12:03:11.436 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.434+0000 7f36d32e5700 10 monclient: get_monmap_and_config 2021-05-25T12:03:11.436 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.434+0000 7f36d32e5700 10 monclient: build_initial_monmap 2021-05-25T12:03:11.436 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.434+0000 7f36d32e5700 10 monclient: monmap: 2021-05-25T12:03:11.437 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:03:11.437 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:03:11.437 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:03:11.435868+0000 2021-05-25T12:03:11.437 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:03:11.435868+0000 2021-05-25T12:03:11.438 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:03:11.438 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:03:11.438 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:03:11.438 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:03:11.439 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:03:11.439 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:03:11.453 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.435+0000 7f36d32e5700 1 Processor -- start 2021-05-25T12:03:11.454 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.435+0000 7f36d32e5700 1 -- start start 2021-05-25T12:03:11.454 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.435+0000 7f36d32e5700 10 monclient: init 2021-05-25T12:03:11.454 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.436+0000 7f36d32e5700 10 monclient: _reopen_session rank -1 2021-05-25T12:03:11.454 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.436+0000 7f36d32e5700 10 monclient: _add_conns ranks=[0,1,2] 2021-05-25T12:03:11.455 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.436+0000 7f36d32e5700 10 monclient(hunting): picked mon.noname-a con 0x7f36cc069190 addr v1:172.21.15.102:6789/0 2021-05-25T12:03:11.455 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.436+0000 7f36d32e5700 10 monclient(hunting): picked mon.noname-c con 0x7f36cc069c80 addr v1:172.21.15.181:6789/0 2021-05-25T12:03:11.455 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.436+0000 7f36d32e5700 10 monclient(hunting): picked mon.noname-b con 0x7f36cc096ab0 addr v1:172.21.15.102:6790/0 2021-05-25T12:03:11.455 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.436+0000 7f36d32e5700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36cc0959b0 con 0x7f36cc069190 2021-05-25T12:03:11.455 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.436+0000 7f36d32e5700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36cc0914e0 con 0x7f36cc069c80 2021-05-25T12:03:11.456 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.436+0000 7f36d32e5700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36cc095c10 con 0x7f36cc096ab0 2021-05-25T12:03:11.456 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.436+0000 7f36d32e5700 10 monclient(hunting): _renew_subs 2021-05-25T12:03:11.456 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.436+0000 7f36d32e5700 10 monclient(hunting): authenticate will time out at 992.033257s 2021-05-25T12:03:11.456 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.436+0000 7f36d1882700 1 -- >> v1:172.21.15.102:6790/0 conn(0x7f36cc096ab0 legacy=0x7f36cc134560 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6790/0 2021-05-25T12:03:11.457 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.436+0000 7f36d1081700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f36cc069190 0x7f36cc069570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60454/0 (socket says 172.21.15.102:60454) 2021-05-25T12:03:11.457 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.436+0000 7f36d1081700 1 -- 172.21.15.102:0/1301748579 learned_addr learned my addr 172.21.15.102:0/1301748579 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:03:11.457 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.436+0000 7f36c3fff700 1 -- 172.21.15.102:0/1301748579 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3958099829 0 0) 0x7f36cc0959b0 con 0x7f36cc069190 2021-05-25T12:03:11.457 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.436+0000 7f36c3fff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:03:11.457 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.436+0000 7f36c3fff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:03:11.458 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.436+0000 7f36c3fff700 10 monclient(hunting): my global_id is 174252 2021-05-25T12:03:11.458 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.436+0000 7f36c3fff700 1 -- 172.21.15.102:0/1301748579 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36b8003270 con 0x7f36cc069190 2021-05-25T12:03:11.458 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.436+0000 7f36c3fff700 1 -- 172.21.15.102:0/1301748579 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2977605666 0 0) 0x7f36cc0914e0 con 0x7f36cc069c80 2021-05-25T12:03:11.458 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.436+0000 7f36c3fff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:03:11.459 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.436+0000 7f36c3fff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:03:11.459 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.436+0000 7f36c3fff700 10 monclient(hunting): my global_id is 95212 2021-05-25T12:03:11.459 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.436+0000 7f36c3fff700 1 -- 172.21.15.102:0/1301748579 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36cc0959b0 con 0x7f36cc069c80 2021-05-25T12:03:11.459 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.436+0000 7f36c3fff700 1 -- 172.21.15.102:0/1301748579 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 127639200 0 0) 0x7f36b8003270 con 0x7f36cc069190 2021-05-25T12:03:11.460 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.436+0000 7f36c3fff700 1 -- 172.21.15.102:0/1301748579 >> v1:172.21.15.102:6790/0 conn(0x7f36cc096ab0 legacy=0x7f36cc134560 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-05-25T12:03:11.460 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.436+0000 7f36c3fff700 1 -- 172.21.15.102:0/1301748579 >> v1:172.21.15.181:6789/0 conn(0x7f36cc069c80 legacy=0x7f36cc130e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:03:11.460 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.436+0000 7f36c3fff700 10 monclient: _finish_hunting 0 2021-05-25T12:03:11.460 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.436+0000 7f36c3fff700 1 monclient: found mon.noname-a 2021-05-25T12:03:11.460 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.436+0000 7f36c3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:03:11.461 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.436+0000 7f36c3fff700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T12:03:11.461 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.436+0000 7f36c3fff700 1 -- 172.21.15.102:0/1301748579 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36cc095eb0 con 0x7f36cc069190 2021-05-25T12:03:11.461 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.436+0000 7f36c3fff700 10 monclient: _finish_auth 0 2021-05-25T12:03:11.461 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.436+0000 7f36c3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:03:11.462 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.436+0000 7f36c3fff700 1 -- 172.21.15.102:0/1301748579 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f36c8001990 con 0x7f36cc069190 2021-05-25T12:03:11.462 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.436+0000 7f36d32e5700 5 monclient: authenticate success, global_id 174252 2021-05-25T12:03:11.462 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.436+0000 7f36d32e5700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:03:11.462 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.436+0000 7f36c3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:03:11.463 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.436+0000 7f36c3fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:03:11.463 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.436+0000 7f36c3fff700 10 monclient: dump: 2021-05-25T12:03:11.463 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:03:11.463 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:03:11.463 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:03:11.464 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:03:11.464 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:03:11.464 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:03:11.464 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:03:11.465 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:03:11.465 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:03:11.465 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:03:11.465 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.436+0000 7f36d32e5700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:03:11.466 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.436+0000 7f36c3fff700 1 -- 172.21.15.102:0/1301748579 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f36c8001c70 con 0x7f36cc069190 2021-05-25T12:03:11.466 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.436+0000 7f36c3fff700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:03:11.466 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.436+0000 7f36c3fff700 1 -- 172.21.15.102:0/1301748579 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f36c8002000 con 0x7f36cc069190 2021-05-25T12:03:11.466 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.436+0000 7f36d32e5700 10 monclient: get_monmap_and_config success 2021-05-25T12:03:11.467 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.436+0000 7f36d32e5700 4 set_mon_vals no callback set 2021-05-25T12:03:11.467 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.436+0000 7f36c3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:03:11.467 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.436+0000 7f36c3fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:03:11.467 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.436+0000 7f36c3fff700 10 monclient: dump: 2021-05-25T12:03:11.467 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:03:11.468 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:03:11.468 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:03:11.468 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:03:11.468 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:03:11.469 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:03:11.469 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:03:11.469 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:03:11.469 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:03:11.469 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:03:11.470 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.437+0000 7f36d32e5700 10 monclient: shutdown 2021-05-25T12:03:11.470 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.437+0000 7f36d32e5700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:03:11.470 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.437+0000 7f36d32e5700 1 -- 172.21.15.102:0/1301748579 >> v1:172.21.15.102:6789/0 conn(0x7f36cc069190 legacy=0x7f36cc069570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:03:11.470 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.437+0000 7f36d32e5700 1 -- 172.21.15.102:0/1301748579 shutdown_connections 2021-05-25T12:03:11.471 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.437+0000 7f36d32e5700 1 -- 172.21.15.102:0/1301748579 >> 172.21.15.102:0/1301748579 conn(0x7f36cc09af90 msgr2=0x7f36cc09b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:03:11.471 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.437+0000 7f36d32e5700 1 -- 172.21.15.102:0/1301748579 shutdown_connections 2021-05-25T12:03:11.471 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.437+0000 7f36d32e5700 1 -- 172.21.15.102:0/1301748579 wait complete. 2021-05-25T12:03:11.471 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.437+0000 7f36d32e5700 10 monclient: build_initial_monmap 2021-05-25T12:03:11.472 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.437+0000 7f36d32e5700 10 monclient: monmap: 2021-05-25T12:03:11.472 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:03:11.472 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:03:11.472 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:03:11.473 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:03:11.473 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:03:11.473 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:03:11.473 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:03:11.474 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:03:11.474 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:03:11.474 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:03:11.474 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.438+0000 7f36d32e5700 1 librados: starting msgr at 2021-05-25T12:03:11.475 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.438+0000 7f36d32e5700 1 librados: starting objecter 2021-05-25T12:03:11.475 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.438+0000 7f36d32e5700 1 Processor -- start 2021-05-25T12:03:11.475 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.438+0000 7f36d32e5700 1 -- start start 2021-05-25T12:03:11.475 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.438+0000 7f36d32e5700 1 librados: setting wanted keys 2021-05-25T12:03:11.476 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.438+0000 7f36d32e5700 1 librados: calling monclient init 2021-05-25T12:03:11.476 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.438+0000 7f36d32e5700 10 monclient: init 2021-05-25T12:03:11.476 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.438+0000 7f36d32e5700 10 monclient: _reopen_session rank -1 2021-05-25T12:03:11.476 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.438+0000 7f36d32e5700 10 monclient: _add_conns ranks=[1,0,2] 2021-05-25T12:03:11.476 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.438+0000 7f36d32e5700 10 monclient(hunting): picked mon.noname-b con 0x7f36cc069190 addr v1:172.21.15.181:6789/0 2021-05-25T12:03:11.477 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.438+0000 7f36d32e5700 10 monclient(hunting): picked mon.noname-a con 0x7f36cc069c80 addr v1:172.21.15.102:6789/0 2021-05-25T12:03:11.477 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.438+0000 7f36d32e5700 10 monclient(hunting): picked mon.noname-c con 0x7f36cc096ab0 addr v1:172.21.15.102:6790/0 2021-05-25T12:03:11.477 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.438+0000 7f36d32e5700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36cc12e9c0 con 0x7f36cc069c80 2021-05-25T12:03:11.477 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.438+0000 7f36d32e5700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36cc11acd0 con 0x7f36cc069190 2021-05-25T12:03:11.478 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.438+0000 7f36d32e5700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36cc11af70 con 0x7f36cc096ab0 2021-05-25T12:03:11.478 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.438+0000 7f36d32e5700 10 monclient(hunting): _renew_subs 2021-05-25T12:03:11.478 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.438+0000 7f36d32e5700 10 monclient(hunting): authenticate will time out at 992.035895s 2021-05-25T12:03:11.478 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.438+0000 7f36d1882700 1 -- >> v1:172.21.15.102:6790/0 conn(0x7f36cc096ab0 legacy=0x7f36cc143810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.102:6790/0 2021-05-25T12:03:11.478 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.438+0000 7f36d0880700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f36cc069c80 0x7f36cc1400e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60462/0 (socket says 172.21.15.102:60462) 2021-05-25T12:03:11.479 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.439+0000 7f36d0880700 1 -- 172.21.15.102:0/3499007804 learned_addr learned my addr 172.21.15.102:0/3499007804 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:03:11.479 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.439+0000 7f36c17fa700 10 client.?.objecter ms_handle_connect 0x7f36cc069c80 2021-05-25T12:03:11.479 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.439+0000 7f36c17fa700 10 client.?.objecter resend_mon_ops 2021-05-25T12:03:11.479 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.439+0000 7f36c17fa700 10 client.?.objecter ms_handle_connect 0x7f36cc069190 2021-05-25T12:03:11.480 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.439+0000 7f36c17fa700 10 client.?.objecter resend_mon_ops 2021-05-25T12:03:11.480 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.439+0000 7f36c17fa700 1 -- 172.21.15.102:0/3499007804 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2290378022 0 0) 0x7f36cc12e9c0 con 0x7f36cc069c80 2021-05-25T12:03:11.480 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.439+0000 7f36c17fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:03:11.480 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.439+0000 7f36c17fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:03:11.481 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.439+0000 7f36c17fa700 10 monclient(hunting): my global_id is 174255 2021-05-25T12:03:11.481 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.439+0000 7f36c17fa700 1 -- 172.21.15.102:0/3499007804 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36ac004c10 con 0x7f36cc069c80 2021-05-25T12:03:11.481 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.439+0000 7f36c17fa700 1 -- 172.21.15.102:0/3499007804 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2834134581 0 0) 0x7f36cc11acd0 con 0x7f36cc069190 2021-05-25T12:03:11.481 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.439+0000 7f36c17fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:03:11.481 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.439+0000 7f36c17fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:03:11.482 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.439+0000 7f36c17fa700 10 monclient(hunting): my global_id is 95215 2021-05-25T12:03:11.482 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.439+0000 7f36c17fa700 1 -- 172.21.15.102:0/3499007804 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36cc12e9c0 con 0x7f36cc069190 2021-05-25T12:03:11.482 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.439+0000 7f36c17fa700 1 -- 172.21.15.102:0/3499007804 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4134460686 0 0) 0x7f36ac004c10 con 0x7f36cc069c80 2021-05-25T12:03:11.482 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.439+0000 7f36c17fa700 1 -- 172.21.15.102:0/3499007804 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f36cc11acd0 con 0x7f36cc069c80 2021-05-25T12:03:11.483 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.439+0000 7f36c17fa700 1 -- 172.21.15.102:0/3499007804 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f36b4002950 con 0x7f36cc069c80 2021-05-25T12:03:11.483 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.439+0000 7f36c17fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:03:11.483 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.439+0000 7f36c17fa700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:03:11.483 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.439+0000 7f36c17fa700 10 monclient(hunting): dump: 2021-05-25T12:03:11.483 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:03:11.484 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:03:11.484 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:03:11.484 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:03:11.484 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:03:11.485 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:03:11.485 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:03:11.485 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:03:11.485 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:03:11.485 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:03:11.486 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.439+0000 7f36c17fa700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:03:11.486 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.439+0000 7f36c17fa700 1 -- 172.21.15.102:0/3499007804 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3489695907 0 0) 0x7f36cc12e9c0 con 0x7f36cc069190 2021-05-25T12:03:11.486 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.439+0000 7f36c17fa700 1 -- 172.21.15.102:0/3499007804 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f36ac004c10 con 0x7f36cc069190 2021-05-25T12:03:11.486 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.439+0000 7f36c17fa700 1 -- 172.21.15.102:0/3499007804 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f36c8001f80 con 0x7f36cc069190 2021-05-25T12:03:11.487 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.439+0000 7f36c17fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:03:11.487 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.439+0000 7f36c17fa700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2021-05-25T12:03:11.487 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.439+0000 7f36c17fa700 10 monclient(hunting): dump: 2021-05-25T12:03:11.487 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:03:11.487 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:03:11.488 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:03:11.488 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:03:11.488 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:03:11.488 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:03:11.489 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:03:11.489 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:03:11.489 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:03:11.489 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:03:11.489 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.439+0000 7f36c17fa700 1 -- 172.21.15.102:0/3499007804 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3782542399 0 0) 0x7f36cc11acd0 con 0x7f36cc069c80 2021-05-25T12:03:11.490 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.439+0000 7f36c17fa700 1 -- 172.21.15.102:0/3499007804 >> v1:172.21.15.102:6790/0 conn(0x7f36cc096ab0 legacy=0x7f36cc143810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-05-25T12:03:11.490 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.439+0000 7f36c17fa700 1 -- 172.21.15.102:0/3499007804 >> v1:172.21.15.181:6789/0 conn(0x7f36cc069190 legacy=0x7f36cc12d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:03:11.490 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.439+0000 7f36c17fa700 10 monclient: _finish_hunting 0 2021-05-25T12:03:11.490 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.439+0000 7f36c17fa700 1 monclient: found mon.a 2021-05-25T12:03:11.491 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.439+0000 7f36c17fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:03:11.491 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.439+0000 7f36c17fa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:03:11.491 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.439+0000 7f36c17fa700 1 -- 172.21.15.102:0/3499007804 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36cc13bc70 con 0x7f36cc069c80 2021-05-25T12:03:11.491 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.439+0000 7f36c17fa700 10 monclient: _finish_auth 0 2021-05-25T12:03:11.492 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.439+0000 7f36c17fa700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:03:11.492 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.439+0000 7f36d32e5700 5 monclient: authenticate success, global_id 174255 2021-05-25T12:03:11.492 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.439+0000 7f36d32e5700 10 monclient: _renew_subs 2021-05-25T12:03:11.492 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.439+0000 7f36d32e5700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:03:11.492 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.439+0000 7f36d32e5700 1 -- 172.21.15.102:0/3499007804 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36cc118d90 con 0x7f36cc069c80 2021-05-25T12:03:11.493 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.439+0000 7f36d32e5700 10 client.174255.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:03:11.493 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.439+0000 7f36d32e5700 10 monclient: _renew_subs 2021-05-25T12:03:11.493 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.439+0000 7f36d32e5700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:03:11.493 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.439+0000 7f36d32e5700 1 -- 172.21.15.102:0/3499007804 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f36cc11a390 con 0x7f36cc069c80 2021-05-25T12:03:11.494 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.439+0000 7f36d32e5700 1 librados: init done 2021-05-25T12:03:11.494 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.439+0000 7f36c17fa700 1 -- 172.21.15.102:0/3499007804 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f36b4002ce0 con 0x7f36cc069c80 2021-05-25T12:03:11.494 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.439+0000 7f36c17fa700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:03:11.494 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.439+0000 7f36c17fa700 1 -- 172.21.15.102:0/3499007804 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f36b4001800 con 0x7f36cc069c80 2021-05-25T12:03:11.494 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.439+0000 7f36c17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:03:11.495 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.439+0000 7f36c17fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:03:11.495 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.439+0000 7f36c17fa700 10 monclient: dump: 2021-05-25T12:03:11.495 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:03:11.495 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:03:11.496 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:03:11.496 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:03:11.496 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:03:11.496 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:03:11.496 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:03:11.497 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:03:11.497 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:03:11.497 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:03:11.497 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.439+0000 7f36c37fe700 4 set_mon_vals no callback set 2021-05-25T12:03:11.498 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.440+0000 7f36c17fa700 1 -- 172.21.15.102:0/3499007804 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f36b40019e0 con 0x7f36cc069c80 2021-05-25T12:03:11.498 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.440+0000 7f36d32e5700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:03:11.498 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.440+0000 7f36d32e5700 1 -- 172.21.15.102:0/3499007804 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f36a40065a0 con 0x7f36a4002840 2021-05-25T12:03:11.498 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.440+0000 7f36c17fa700 10 client.174255.objecter ms_handle_connect 0x7f36a4002840 2021-05-25T12:03:11.499 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.440+0000 7f36c17fa700 10 client.174255.objecter resend_mon_ops 2021-05-25T12:03:11.499 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.440+0000 7f36c17fa700 1 -- 172.21.15.102:0/3499007804 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(85..85 src has 1..85) v4 ==== 7834+0+0 (unknown 3013755738 0 0) 0x7f36b403df80 con 0x7f36cc069c80 2021-05-25T12:03:11.499 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.440+0000 7f36c17fa700 10 client.174255.objecter ms_dispatch 0x7f36cc001550 osd_map(85..85 src has 1..85) v4 2021-05-25T12:03:11.499 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.440+0000 7f36c17fa700 3 client.174255.objecter handle_osd_map got epochs [85,85] > 0 2021-05-25T12:03:11.499 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.440+0000 7f36c17fa700 3 client.174255.objecter handle_osd_map decoding full epoch 85 2021-05-25T12:03:11.500 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.440+0000 7f36c17fa700 20 client.174255.objecter dump_active .. 0 homeless 2021-05-25T12:03:11.500 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.440+0000 7f36c17fa700 1 -- 172.21.15.102:0/3499007804 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1765085325 0 0) 0x7f36a40065a0 con 0x7f36a4002840 2021-05-25T12:03:11.500 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.440+0000 7f36c17fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:03:11.500 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.440+0000 7f36c17fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:03:11.500 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.440+0000 7f36c17fa700 10 monclient(hunting): my global_id is 174258 2021-05-25T12:03:11.501 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.440+0000 7f36c17fa700 1 -- 172.21.15.102:0/3499007804 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36cc11acd0 con 0x7f36a4002840 2021-05-25T12:03:11.501 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.440+0000 7f36d1882700 10 monclient: get_auth_request con 0x7f36ac0332a0 auth_method 0 2021-05-25T12:03:11.501 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.441+0000 7f36c17fa700 1 -- 172.21.15.102:0/3499007804 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 767239110 0 0) 0x7f36cc11acd0 con 0x7f36a4002840 2021-05-25T12:03:11.501 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.441+0000 7f36c17fa700 1 -- 172.21.15.102:0/3499007804 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f36a40065a0 con 0x7f36a4002840 2021-05-25T12:03:11.502 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.441+0000 7f36c17fa700 1 -- 172.21.15.102:0/3499007804 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f36c80015b0 con 0x7f36a4002840 2021-05-25T12:03:11.502 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.441+0000 7f36c17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:03:11.502 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.441+0000 7f36c17fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:03:11.502 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.441+0000 7f36c17fa700 10 monclient: dump: 2021-05-25T12:03:11.503 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:03:11.503 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:03:11.503 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:03:11.503 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:03:11.503 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:03:11.504 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:03:11.504 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:03:11.504 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:03:11.504 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:03:11.505 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:03:11.505 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.441+0000 7f36c17fa700 10 client.174255.objecter ms_handle_connect 0x7f36ac0332a0 2021-05-25T12:03:11.505 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.441+0000 7f36c17fa700 1 -- 172.21.15.102:0/3499007804 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3401612324 0 0) 0x7f36a40065a0 con 0x7f36a4002840 2021-05-25T12:03:11.505 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.441+0000 7f36c17fa700 1 -- 172.21.15.102:0/3499007804 --> v1:172.21.15.102:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f36a4009e40 con 0x7f36a4002840 2021-05-25T12:03:11.506 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.441+0000 7f36c17fa700 1 -- 172.21.15.102:0/3499007804 <== mon.0 v1:172.21.15.102:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7f36c8005010 con 0x7f36a4002840 2021-05-25T12:03:11.506 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.441+0000 7f36c17fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:03:11.506 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.441+0000 7f36c17fa700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:03:11.506 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.441+0000 7f36c17fa700 1 -- 172.21.15.102:0/3499007804 >> v1:172.21.15.102:6789/0 conn(0x7f36a4002840 legacy=0x7f36a4004ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:03:11.507 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.441+0000 7f36c17fa700 1 -- 172.21.15.102:0/3499007804 >> v1:172.21.15.102:6789/0 conn(0x7f36a4002840 legacy=0x7f36a4004ce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:03:11.507 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.448+0000 7f36d32e5700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:03:11.507 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.448+0000 7f36d32e5700 1 -- 172.21.15.102:0/3499007804 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f36a4009b00 con 0x7f36a4008dc0 2021-05-25T12:03:11.507 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.448+0000 7f36c17fa700 10 client.174255.objecter ms_handle_connect 0x7f36a4008dc0 2021-05-25T12:03:11.507 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.448+0000 7f36c17fa700 10 client.174255.objecter resend_mon_ops 2021-05-25T12:03:11.508 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.448+0000 7f36c17fa700 1 -- 172.21.15.102:0/3499007804 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 550143527 0 0) 0x7f36a4009b00 con 0x7f36a4008dc0 2021-05-25T12:03:11.508 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.448+0000 7f36c17fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:03:11.508 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.448+0000 7f36c17fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:03:11.508 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.448+0000 7f36c17fa700 10 monclient(hunting): my global_id is 174261 2021-05-25T12:03:11.508 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.448+0000 7f36c17fa700 1 -- 172.21.15.102:0/3499007804 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36a40065a0 con 0x7f36a4008dc0 2021-05-25T12:03:11.509 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.448+0000 7f36c17fa700 1 -- 172.21.15.102:0/3499007804 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3395131345 0 0) 0x7f36a40065a0 con 0x7f36a4008dc0 2021-05-25T12:03:11.509 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.448+0000 7f36c17fa700 1 -- 172.21.15.102:0/3499007804 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f36a4009b00 con 0x7f36a4008dc0 2021-05-25T12:03:11.509 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.448+0000 7f36c17fa700 1 -- 172.21.15.102:0/3499007804 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f36c8005730 con 0x7f36a4008dc0 2021-05-25T12:03:11.509 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.448+0000 7f36c17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:03:11.510 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.448+0000 7f36c17fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:03:11.510 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.448+0000 7f36c17fa700 10 monclient: dump: 2021-05-25T12:03:11.510 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:03:11.510 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:03:11.511 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:03:11.511 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:03:11.511 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:03:11.511 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:03:11.512 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:03:11.512 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:03:11.512 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:03:11.512 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:03:11.513 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.448+0000 7f36c17fa700 1 -- 172.21.15.102:0/3499007804 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3768864016 0 0) 0x7f36a4009b00 con 0x7f36a4008dc0 2021-05-25T12:03:11.513 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.448+0000 7f36c17fa700 1 -- 172.21.15.102:0/3499007804 --> v1:172.21.15.102:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f36a4008740 con 0x7f36a4008dc0 2021-05-25T12:03:11.513 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.449+0000 7f36c17fa700 1 -- 172.21.15.102:0/3499007804 <== mon.0 v1:172.21.15.102:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4099 (unknown 0 0 3123222087) 0x7f36c8007a20 con 0x7f36a4008dc0 2021-05-25T12:03:11.513 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.449+0000 7f36c17fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:03:11.514 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.449+0000 7f36c17fa700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:03:11.514 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.449+0000 7f36c17fa700 1 -- 172.21.15.102:0/3499007804 >> v1:172.21.15.102:6789/0 conn(0x7f36a4008dc0 legacy=0x7f36a4009250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:03:11.514 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.449+0000 7f36c17fa700 1 -- 172.21.15.102:0/3499007804 >> v1:172.21.15.102:6789/0 conn(0x7f36a4008dc0 legacy=0x7f36a4009250 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:03:11.514 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:03:11.515 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:03:11.515 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:03:11.515 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "peon", 2021-05-25T12:03:11.515 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 32, 2021-05-25T12:03:11.516 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:03:11.516 INFO:teuthology.orchestra.run.smithi102.stdout: 0, 2021-05-25T12:03:11.516 INFO:teuthology.orchestra.run.smithi102.stdout: 1 2021-05-25T12:03:11.516 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:03:11.517 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 0, 2021-05-25T12:03:11.517 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:03:11.517 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:03:11.517 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:03:11.517 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:03:11.518 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:03:11.518 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:03:11.518 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:03:11.518 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:03:11.519 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:03:11.519 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:03:11.519 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:03:11.519 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:03:11.520 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:03:11.520 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:03:11.520 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:03:11.520 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:03:11.520 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:03:11.521 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:03:11.521 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:03:11.521 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:03:11.521 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:03:11.522 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:03:11.522 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:03:11.522 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:03:11.522 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:03:11.523 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:03:11.523 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:03:11.523 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:03:11.523 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:03:11.524 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:03:11.524 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:03:11.524 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:03:11.524 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:03:11.525 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:03:11.525 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:03:11.525 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:03:11.525 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:03:11.526 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:03:11.526 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:03:11.526 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:03:11.526 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:03:11.527 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:03:11.527 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:03:11.527 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:03:11.527 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:03:11.527 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:03:11.528 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:03:11.528 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:03:11.528 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:03:11.528 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:03:11.529 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:03:11.529 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:03:11.529 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:03:11.529 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:03:11.529 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:03:11.530 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:03:11.530 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:03:11.530 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:03:11.530 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:03:11.531 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:03:11.531 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:03:11.531 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:03:11.531 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:03:11.531 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:03:11.532 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:03:11.532 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:03:11.532 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:03:11.532 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:03:11.533 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:03:11.533 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:03:11.533 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:03:11.533 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:03:11.534 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:03:11.534 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:03:11.534 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:03:11.534 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:03:11.534 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:03:11.535 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:03:11.535 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:03:11.535 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:03:11.535 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:03:11.536 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:03:11.536 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:03:11.536 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:03:11.536 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:03:11.536 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:03:11.537 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:03:11.537 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:03:11.537 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:03:11.537 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:03:11.538 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:03:11.538 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:03:11.538 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:03:11.538 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:03:11.538 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:03:11.539 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:03:11.539 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:03:11.539 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:03:11.539 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:03:11.540 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:03:11.540 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:03:11.540 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:03:11.540 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:03:11.541 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:03:11.541 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:03:11.541 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:03:11.541 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:03:11.542 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:03:11.542 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:03:11.542 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:03:11.542 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:03:11.543 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:03:11.543 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:03:11.543 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:03:11.543 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:03:11.544 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:03:11.544 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:03:11.544 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:03:11.544 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": [ 2021-05-25T12:03:11.545 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:03:11.545 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:03:11.545 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:03:11.545 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 3 2021-05-25T12:03:11.545 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:03:11.546 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:03:11.546 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:03:11.546 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:03:11.546 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:03:11.546 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:03:11.547 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 8 2021-05-25T12:03:11.547 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:03:11.547 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:03:11.547 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": [ 2021-05-25T12:03:11.548 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:03:11.548 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:03:11.548 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:03:11.548 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:03:11.548 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:03:11.549 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:03:11.549 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:03:11.549 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:03:11.549 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:03:11.552 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.449+0000 7f36d32e5700 10 librados: watch_flush enter 2021-05-25T12:03:11.552 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.449+0000 7f36d32e5700 10 librados: watch_flush exit 2021-05-25T12:03:11.552 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.449+0000 7f36d32e5700 20 client.174255.objecter shutdown clearing up homeless session... 2021-05-25T12:03:11.552 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.449+0000 7f36d32e5700 10 client.174255.objecter successfully canceled tick 2021-05-25T12:03:11.553 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.449+0000 7f36d32e5700 1 -- 172.21.15.102:0/3499007804 >> v1:172.21.15.181:6812/29582 conn(0x7f36ac0332a0 legacy=0x7f36ac035760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:03:11.553 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.449+0000 7f36d32e5700 10 monclient: shutdown 2021-05-25T12:03:11.553 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.449+0000 7f36d32e5700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:03:11.553 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.449+0000 7f36d32e5700 1 -- 172.21.15.102:0/3499007804 >> v1:172.21.15.102:6789/0 conn(0x7f36cc069c80 legacy=0x7f36cc1400e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:03:11.554 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.449+0000 7f36d1882700 1 -- 172.21.15.102:0/3499007804 reap_dead start 2021-05-25T12:03:11.554 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.449+0000 7f36d1882700 1 -- 172.21.15.102:0/3499007804 reap_dead start 2021-05-25T12:03:11.554 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.449+0000 7f36d32e5700 1 -- 172.21.15.102:0/3499007804 shutdown_connections 2021-05-25T12:03:11.554 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.449+0000 7f36d32e5700 1 -- 172.21.15.102:0/3499007804 >> 172.21.15.102:0/3499007804 conn(0x7f36cc09af90 msgr2=0x7f36cc12ebd0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:03:11.555 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.449+0000 7f36d32e5700 1 -- 172.21.15.102:0/3499007804 shutdown_connections 2021-05-25T12:03:11.555 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.449+0000 7f36d32e5700 1 -- 172.21.15.102:0/3499007804 wait complete. 2021-05-25T12:03:11.555 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.449+0000 7f36d32e5700 1 librados: shutdown 2021-05-25T12:03:11.556 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2021-05-25T12:03:11.666 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.664+0000 7f187551a700 10 monclient: get_monmap_and_config 2021-05-25T12:03:11.667 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.664+0000 7f187551a700 10 monclient: build_initial_monmap 2021-05-25T12:03:11.667 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.665+0000 7f187551a700 10 monclient: monmap: 2021-05-25T12:03:11.667 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:03:11.667 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:03:11.667 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:03:11.665883+0000 2021-05-25T12:03:11.668 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:03:11.665883+0000 2021-05-25T12:03:11.668 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:03:11.668 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:03:11.668 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:03:11.669 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:03:11.669 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:03:11.669 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:03:11.669 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.665+0000 7f187551a700 1 Processor -- start 2021-05-25T12:03:11.669 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.665+0000 7f187551a700 1 -- start start 2021-05-25T12:03:11.670 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.665+0000 7f187551a700 10 monclient: init 2021-05-25T12:03:11.684 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.665+0000 7f187551a700 10 monclient: _reopen_session rank -1 2021-05-25T12:03:11.685 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.665+0000 7f187551a700 10 monclient: _add_conns ranks=[1,0,2] 2021-05-25T12:03:11.685 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.665+0000 7f187551a700 10 monclient(hunting): picked mon.noname-c con 0x7f1870093d90 addr v1:172.21.15.181:6789/0 2021-05-25T12:03:11.685 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.665+0000 7f187551a700 10 monclient(hunting): picked mon.noname-a con 0x7f1870132900 addr v1:172.21.15.102:6789/0 2021-05-25T12:03:11.685 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.665+0000 7f187551a700 10 monclient(hunting): picked mon.noname-b con 0x7f1870136410 addr v1:172.21.15.102:6790/0 2021-05-25T12:03:11.686 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.665+0000 7f187551a700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1870091ed0 con 0x7f1870132900 2021-05-25T12:03:11.686 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.665+0000 7f187551a700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1870092170 con 0x7f1870093d90 2021-05-25T12:03:11.686 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.665+0000 7f187551a700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1870092410 con 0x7f1870136410 2021-05-25T12:03:11.686 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.665+0000 7f187551a700 10 monclient(hunting): _renew_subs 2021-05-25T12:03:11.687 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.665+0000 7f187551a700 10 monclient(hunting): authenticate will time out at 992.262993s 2021-05-25T12:03:11.687 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.666+0000 7f186f7fe700 1 -- >> v1:172.21.15.102:6790/0 conn(0x7f1870136410 legacy=0x7f1870138800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6790/0 2021-05-25T12:03:11.687 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.666+0000 7f186e7fc700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f1870132900 0x7f1870134cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60474/0 (socket says 172.21.15.102:60474) 2021-05-25T12:03:11.687 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.666+0000 7f186e7fc700 1 -- 172.21.15.102:0/36297592 learned_addr learned my addr 172.21.15.102:0/36297592 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:03:11.687 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.666+0000 7f186dffb700 1 -- 172.21.15.102:0/36297592 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2644244085 0 0) 0x7f1870091ed0 con 0x7f1870132900 2021-05-25T12:03:11.688 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.666+0000 7f186dffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:03:11.688 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.666+0000 7f186dffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:03:11.688 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.666+0000 7f186dffb700 10 monclient(hunting): my global_id is 174264 2021-05-25T12:03:11.688 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.666+0000 7f186dffb700 1 -- 172.21.15.102:0/36297592 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f185c003270 con 0x7f1870132900 2021-05-25T12:03:11.689 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.666+0000 7f186dffb700 1 -- 172.21.15.102:0/36297592 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2835440605 0 0) 0x7f185c003270 con 0x7f1870132900 2021-05-25T12:03:11.689 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.666+0000 7f186dffb700 1 -- 172.21.15.102:0/36297592 >> v1:172.21.15.102:6790/0 conn(0x7f1870136410 legacy=0x7f1870138800 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-05-25T12:03:11.689 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.666+0000 7f186dffb700 1 -- 172.21.15.102:0/36297592 >> v1:172.21.15.181:6789/0 conn(0x7f1870093d90 legacy=0x7f18700691c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:03:11.689 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.666+0000 7f186dffb700 10 monclient: _finish_hunting 0 2021-05-25T12:03:11.689 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.666+0000 7f186dffb700 1 monclient: found mon.noname-a 2021-05-25T12:03:11.690 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.666+0000 7f186dffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:03:11.690 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.666+0000 7f186dffb700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T12:03:11.690 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.666+0000 7f186dffb700 1 -- 172.21.15.102:0/36297592 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1870094170 con 0x7f1870132900 2021-05-25T12:03:11.690 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.666+0000 7f186dffb700 10 monclient: _finish_auth 0 2021-05-25T12:03:11.691 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.666+0000 7f186dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:03:11.691 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.666+0000 7f186dffb700 1 -- 172.21.15.102:0/36297592 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f1858001610 con 0x7f1870132900 2021-05-25T12:03:11.691 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.666+0000 7f187551a700 5 monclient: authenticate success, global_id 174264 2021-05-25T12:03:11.691 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.666+0000 7f187551a700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:03:11.692 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.666+0000 7f186dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:03:11.692 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.666+0000 7f186dffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:03:11.692 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.666+0000 7f186dffb700 10 monclient: dump: 2021-05-25T12:03:11.692 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:03:11.692 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:03:11.693 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:03:11.693 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:03:11.693 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:03:11.693 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:03:11.694 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:03:11.694 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:03:11.694 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:03:11.694 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:03:11.694 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.666+0000 7f187551a700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:03:11.695 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.666+0000 7f186dffb700 1 -- 172.21.15.102:0/36297592 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1858001930 con 0x7f1870132900 2021-05-25T12:03:11.695 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.666+0000 7f186dffb700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:03:11.695 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.666+0000 7f187551a700 10 monclient: get_monmap_and_config success 2021-05-25T12:03:11.695 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.666+0000 7f186dffb700 1 -- 172.21.15.102:0/36297592 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f1858001cc0 con 0x7f1870132900 2021-05-25T12:03:11.696 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.666+0000 7f186dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:03:11.696 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.666+0000 7f187551a700 4 set_mon_vals no callback set 2021-05-25T12:03:11.696 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.666+0000 7f186dffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:03:11.696 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.666+0000 7f186dffb700 10 monclient: dump: 2021-05-25T12:03:11.696 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:03:11.697 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:03:11.697 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:03:11.697 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:03:11.697 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:03:11.698 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:03:11.698 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:03:11.698 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:03:11.698 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:03:11.698 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:03:11.699 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.667+0000 7f187551a700 10 monclient: shutdown 2021-05-25T12:03:11.699 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.667+0000 7f187551a700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:03:11.699 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.667+0000 7f187551a700 1 -- 172.21.15.102:0/36297592 >> v1:172.21.15.102:6789/0 conn(0x7f1870132900 legacy=0x7f1870134cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:03:11.699 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.667+0000 7f187551a700 1 -- 172.21.15.102:0/36297592 shutdown_connections 2021-05-25T12:03:11.700 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.667+0000 7f187551a700 1 -- 172.21.15.102:0/36297592 >> 172.21.15.102:0/36297592 conn(0x7f187009b370 msgr2=0x7f1870097430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:03:11.700 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.667+0000 7f187551a700 1 -- 172.21.15.102:0/36297592 shutdown_connections 2021-05-25T12:03:11.700 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.667+0000 7f187551a700 1 -- 172.21.15.102:0/36297592 wait complete. 2021-05-25T12:03:11.700 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.667+0000 7f187551a700 10 monclient: build_initial_monmap 2021-05-25T12:03:11.700 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.667+0000 7f187551a700 10 monclient: monmap: 2021-05-25T12:03:11.701 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:03:11.701 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:03:11.701 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:03:11.701 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:03:11.702 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:03:11.702 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:03:11.702 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:03:11.702 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:03:11.703 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:03:11.703 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:03:11.703 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.667+0000 7f187551a700 1 librados: starting msgr at 2021-05-25T12:03:11.703 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.667+0000 7f187551a700 1 librados: starting objecter 2021-05-25T12:03:11.703 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.668+0000 7f187551a700 1 Processor -- start 2021-05-25T12:03:11.704 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.668+0000 7f187551a700 1 -- start start 2021-05-25T12:03:11.704 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.668+0000 7f187551a700 1 librados: setting wanted keys 2021-05-25T12:03:11.704 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.668+0000 7f187551a700 1 librados: calling monclient init 2021-05-25T12:03:11.704 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.668+0000 7f187551a700 10 monclient: init 2021-05-25T12:03:11.705 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.668+0000 7f187551a700 10 monclient: _reopen_session rank -1 2021-05-25T12:03:11.705 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.668+0000 7f187551a700 10 monclient: _add_conns ranks=[2,1,0] 2021-05-25T12:03:11.705 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.668+0000 7f187551a700 10 monclient(hunting): picked mon.noname-c con 0x7f1870093d90 addr v1:172.21.15.102:6790/0 2021-05-25T12:03:11.705 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.668+0000 7f187551a700 10 monclient(hunting): picked mon.noname-b con 0x7f1870132900 addr v1:172.21.15.181:6789/0 2021-05-25T12:03:11.705 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.668+0000 7f187551a700 10 monclient(hunting): picked mon.noname-a con 0x7f1870136410 addr v1:172.21.15.102:6789/0 2021-05-25T12:03:11.706 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.668+0000 7f187551a700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1870066c40 con 0x7f1870136410 2021-05-25T12:03:11.706 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.668+0000 7f187551a700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18701401d0 con 0x7f1870132900 2021-05-25T12:03:11.706 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.668+0000 7f187551a700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1870068790 con 0x7f1870093d90 2021-05-25T12:03:11.706 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.668+0000 7f187551a700 10 monclient(hunting): _renew_subs 2021-05-25T12:03:11.707 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.668+0000 7f187551a700 10 monclient(hunting): authenticate will time out at 992.265557s 2021-05-25T12:03:11.707 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.668+0000 7f186effd700 1 -- >> v1:172.21.15.102:6790/0 conn(0x7f1870093d90 legacy=0x7f187013f3b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.102:6790/0 2021-05-25T12:03:11.707 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.668+0000 7f186f7fe700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f1870136410 0x7f1870147760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60482/0 (socket says 172.21.15.102:60482) 2021-05-25T12:03:11.707 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.668+0000 7f186f7fe700 1 -- 172.21.15.102:0/256816174 learned_addr learned my addr 172.21.15.102:0/256816174 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:03:11.708 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.668+0000 7f1856ffd700 10 client.?.objecter ms_handle_connect 0x7f1870136410 2021-05-25T12:03:11.708 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.668+0000 7f1856ffd700 10 client.?.objecter resend_mon_ops 2021-05-25T12:03:11.708 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.668+0000 7f1856ffd700 1 -- 172.21.15.102:0/256816174 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1497111252 0 0) 0x7f1870066c40 con 0x7f1870136410 2021-05-25T12:03:11.708 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.668+0000 7f1856ffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:03:11.709 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.668+0000 7f1856ffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:03:11.709 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.668+0000 7f1856ffd700 10 monclient(hunting): my global_id is 174267 2021-05-25T12:03:11.709 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.668+0000 7f1856ffd700 1 -- 172.21.15.102:0/256816174 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1850004c10 con 0x7f1870136410 2021-05-25T12:03:11.709 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.668+0000 7f1856ffd700 10 client.?.objecter ms_handle_connect 0x7f1870132900 2021-05-25T12:03:11.709 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.668+0000 7f1856ffd700 10 client.?.objecter resend_mon_ops 2021-05-25T12:03:11.710 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.669+0000 7f1856ffd700 1 -- 172.21.15.102:0/256816174 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3620219482 0 0) 0x7f18701401d0 con 0x7f1870132900 2021-05-25T12:03:11.710 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.669+0000 7f1856ffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:03:11.710 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.669+0000 7f1856ffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:03:11.710 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.669+0000 7f1856ffd700 10 monclient(hunting): my global_id is 95221 2021-05-25T12:03:11.711 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.669+0000 7f1856ffd700 1 -- 172.21.15.102:0/256816174 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1870066c40 con 0x7f1870132900 2021-05-25T12:03:11.711 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.669+0000 7f1856ffd700 1 -- 172.21.15.102:0/256816174 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 779203933 0 0) 0x7f1850004c10 con 0x7f1870136410 2021-05-25T12:03:11.711 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.669+0000 7f1856ffd700 1 -- 172.21.15.102:0/256816174 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f18701401d0 con 0x7f1870136410 2021-05-25T12:03:11.711 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.669+0000 7f1856ffd700 1 -- 172.21.15.102:0/256816174 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f1860004110 con 0x7f1870136410 2021-05-25T12:03:11.711 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.669+0000 7f1856ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:03:11.712 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.669+0000 7f1856ffd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:03:11.712 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.669+0000 7f1856ffd700 10 monclient(hunting): dump: 2021-05-25T12:03:11.712 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:03:11.712 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:03:11.713 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:03:11.713 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:03:11.713 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:03:11.713 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:03:11.714 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:03:11.714 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:03:11.714 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:03:11.714 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:03:11.714 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.669+0000 7f1856ffd700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:03:11.715 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.669+0000 7f1856ffd700 1 -- 172.21.15.102:0/256816174 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 652767719 0 0) 0x7f1870066c40 con 0x7f1870132900 2021-05-25T12:03:11.715 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.669+0000 7f1856ffd700 1 -- 172.21.15.102:0/256816174 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1850004c10 con 0x7f1870132900 2021-05-25T12:03:11.715 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.669+0000 7f1856ffd700 1 -- 172.21.15.102:0/256816174 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f1858001c40 con 0x7f1870132900 2021-05-25T12:03:11.715 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.669+0000 7f1856ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:03:11.716 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.669+0000 7f1856ffd700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2021-05-25T12:03:11.716 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.669+0000 7f1856ffd700 10 monclient(hunting): dump: 2021-05-25T12:03:11.716 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:03:11.716 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:03:11.717 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:03:11.717 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:03:11.717 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:03:11.717 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:03:11.717 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:03:11.718 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:03:11.718 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:03:11.718 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:03:11.718 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.669+0000 7f1856ffd700 1 -- 172.21.15.102:0/256816174 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1822705695 0 0) 0x7f18701401d0 con 0x7f1870136410 2021-05-25T12:03:11.719 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.669+0000 7f1856ffd700 1 -- 172.21.15.102:0/256816174 >> v1:172.21.15.102:6790/0 conn(0x7f1870093d90 legacy=0x7f187013f3b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-05-25T12:03:11.719 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.669+0000 7f1856ffd700 1 -- 172.21.15.102:0/256816174 >> v1:172.21.15.181:6789/0 conn(0x7f1870132900 legacy=0x7f187013fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:03:11.719 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.669+0000 7f1856ffd700 10 monclient: _finish_hunting 0 2021-05-25T12:03:11.719 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.669+0000 7f1856ffd700 1 monclient: found mon.a 2021-05-25T12:03:11.720 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.669+0000 7f1856ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:03:11.720 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.669+0000 7f1856ffd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:03:11.720 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.669+0000 7f1856ffd700 1 -- 172.21.15.102:0/256816174 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1870068a30 con 0x7f1870136410 2021-05-25T12:03:11.720 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.669+0000 7f1856ffd700 10 monclient: _finish_auth 0 2021-05-25T12:03:11.720 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.669+0000 7f1856ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:03:11.721 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.669+0000 7f187551a700 5 monclient: authenticate success, global_id 174267 2021-05-25T12:03:11.721 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.669+0000 7f187551a700 10 monclient: _renew_subs 2021-05-25T12:03:11.721 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.669+0000 7f187551a700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:03:11.721 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.669+0000 7f187551a700 1 -- 172.21.15.102:0/256816174 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1870094420 con 0x7f1870136410 2021-05-25T12:03:11.722 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.669+0000 7f187551a700 10 client.174267.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:03:11.722 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.669+0000 7f187551a700 10 monclient: _renew_subs 2021-05-25T12:03:11.722 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.669+0000 7f187551a700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:03:11.722 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.669+0000 7f187551a700 1 -- 172.21.15.102:0/256816174 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1870067640 con 0x7f1870136410 2021-05-25T12:03:11.723 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.669+0000 7f187551a700 1 librados: init done 2021-05-25T12:03:11.723 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.669+0000 7f1856ffd700 1 -- 172.21.15.102:0/256816174 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1860004380 con 0x7f1870136410 2021-05-25T12:03:11.723 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.669+0000 7f1856ffd700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:03:11.723 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.669+0000 7f1856ffd700 1 -- 172.21.15.102:0/256816174 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f1860004710 con 0x7f1870136410 2021-05-25T12:03:11.724 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.669+0000 7f1856ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:03:11.724 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.669+0000 7f1856ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:03:11.724 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.669+0000 7f186d7fa700 4 set_mon_vals no callback set 2021-05-25T12:03:11.724 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.669+0000 7f1856ffd700 10 monclient: dump: 2021-05-25T12:03:11.724 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:03:11.725 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:03:11.725 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:03:11.725 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:03:11.725 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:03:11.726 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:03:11.726 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:03:11.726 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:03:11.726 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:03:11.727 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:03:11.727 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.669+0000 7f1856ffd700 1 -- 172.21.15.102:0/256816174 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f18600048f0 con 0x7f1870136410 2021-05-25T12:03:11.727 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.670+0000 7f1856ffd700 1 -- 172.21.15.102:0/256816174 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(85..85 src has 1..85) v4 ==== 7834+0+0 (unknown 3013755738 0 0) 0x7f186003e130 con 0x7f1870136410 2021-05-25T12:03:11.727 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.670+0000 7f1856ffd700 10 client.174267.objecter ms_dispatch 0x7f1870001550 osd_map(85..85 src has 1..85) v4 2021-05-25T12:03:11.728 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.670+0000 7f1856ffd700 3 client.174267.objecter handle_osd_map got epochs [85,85] > 0 2021-05-25T12:03:11.728 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.670+0000 7f1856ffd700 3 client.174267.objecter handle_osd_map decoding full epoch 85 2021-05-25T12:03:11.728 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.670+0000 7f187551a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:03:11.728 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.670+0000 7f187551a700 1 -- 172.21.15.102:0/256816174 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f18480065a0 con 0x7f1848002840 2021-05-25T12:03:11.728 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.670+0000 7f1856ffd700 20 client.174267.objecter dump_active .. 0 homeless 2021-05-25T12:03:11.729 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.670+0000 7f186effd700 10 monclient: get_auth_request con 0x7f18500332a0 auth_method 0 2021-05-25T12:03:11.729 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.670+0000 7f1856ffd700 10 client.174267.objecter ms_handle_connect 0x7f1848002840 2021-05-25T12:03:11.729 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.670+0000 7f1856ffd700 10 client.174267.objecter resend_mon_ops 2021-05-25T12:03:11.729 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.670+0000 7f1856ffd700 10 client.174267.objecter ms_handle_connect 0x7f18500332a0 2021-05-25T12:03:11.730 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.670+0000 7f1856ffd700 1 -- 172.21.15.102:0/256816174 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1796167500 0 0) 0x7f18480065a0 con 0x7f1848002840 2021-05-25T12:03:11.730 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.670+0000 7f1856ffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:03:11.730 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.670+0000 7f1856ffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:03:11.730 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.670+0000 7f1856ffd700 10 monclient(hunting): my global_id is 95224 2021-05-25T12:03:11.731 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.670+0000 7f1856ffd700 1 -- 172.21.15.102:0/256816174 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18701401d0 con 0x7f1848002840 2021-05-25T12:03:11.731 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.670+0000 7f1856ffd700 1 -- 172.21.15.102:0/256816174 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3515631237 0 0) 0x7f18701401d0 con 0x7f1848002840 2021-05-25T12:03:11.731 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.670+0000 7f1856ffd700 1 -- 172.21.15.102:0/256816174 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f18480065a0 con 0x7f1848002840 2021-05-25T12:03:11.731 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.670+0000 7f1856ffd700 1 -- 172.21.15.102:0/256816174 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f1858004a70 con 0x7f1848002840 2021-05-25T12:03:11.731 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.670+0000 7f1856ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:03:11.732 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.670+0000 7f1856ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T12:03:11.732 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.670+0000 7f1856ffd700 10 monclient: dump: 2021-05-25T12:03:11.732 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:03:11.732 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:03:11.733 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:03:11.733 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:03:11.733 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:03:11.733 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:03:11.733 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:03:11.734 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:03:11.734 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:03:11.734 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:03:11.734 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.670+0000 7f1856ffd700 1 -- 172.21.15.102:0/256816174 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3839735399 0 0) 0x7f18480065a0 con 0x7f1848002840 2021-05-25T12:03:11.735 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.670+0000 7f1856ffd700 1 -- 172.21.15.102:0/256816174 --> v1:172.21.15.181:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1848009e40 con 0x7f1848002840 2021-05-25T12:03:11.735 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.671+0000 7f1856ffd700 1 -- 172.21.15.102:0/256816174 <== mon.1 v1:172.21.15.181:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7f1858004dc0 con 0x7f1848002840 2021-05-25T12:03:11.735 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.671+0000 7f1856ffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:03:11.735 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.671+0000 7f1856ffd700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:03:11.736 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.671+0000 7f1856ffd700 1 -- 172.21.15.102:0/256816174 >> v1:172.21.15.181:6789/0 conn(0x7f1848002840 legacy=0x7f1848004ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:03:11.736 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.671+0000 7f1856ffd700 1 -- 172.21.15.102:0/256816174 >> v1:172.21.15.181:6789/0 conn(0x7f1848002840 legacy=0x7f1848004ce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:03:11.736 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.677+0000 7f187551a700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:03:11.736 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.677+0000 7f187551a700 1 -- 172.21.15.102:0/256816174 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1848007540 con 0x7f18480068e0 2021-05-25T12:03:11.737 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.678+0000 7f1856ffd700 10 client.174267.objecter ms_handle_connect 0x7f18480068e0 2021-05-25T12:03:11.737 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.678+0000 7f1856ffd700 10 client.174267.objecter resend_mon_ops 2021-05-25T12:03:11.737 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.678+0000 7f186e7fc700 1 -- 172.21.15.102:0/256816174 >> v1:172.21.15.181:6789/0 conn(0x7f18480068e0 legacy=0x7f1848006d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-05-25T12:03:11.737 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.678+0000 7f186e7fc700 1 -- 172.21.15.102:0/256816174 >> v1:172.21.15.181:6789/0 conn(0x7f18480068e0 legacy=0x7f1848006d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-05-25T12:03:11.737 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.678+0000 7f186e7fc700 1 --1- 172.21.15.102:0/256816174 >> v1:172.21.15.181:6789/0 conn(0x7f18480068e0 0x7f1848006d70 :-1 s=OPENED pgs=8867 cs=1 l=1).handle_message read tag failed 2021-05-25T12:03:11.738 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.678+0000 7f186e7fc700 1 --1- 172.21.15.102:0/256816174 >> v1:172.21.15.181:6789/0 conn(0x7f18480068e0 0x7f1848006d70 :-1 s=OPENED pgs=8867 cs=1 l=1).fault on lossy channel, failing 2021-05-25T12:03:11.738 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.678+0000 7f1856ffd700 1 -- 172.21.15.102:0/256816174 >> v1:172.21.15.181:6789/0 conn(0x7f18480068e0 legacy=0x7f1848006d70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:03:11.738 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.678+0000 7f1856ffd700 1 -- 172.21.15.102:0/256816174 >> v1:172.21.15.181:6789/0 conn(0x7f18480068e0 legacy=0x7f1848006d70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:03:11.738 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.678+0000 7f1856ffd700 1 -- 172.21.15.102:0/256816174 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f18480065a0 con 0x7f1850041e30 2021-05-25T12:03:11.739 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.678+0000 7f1856ffd700 10 client.174267.objecter ms_handle_connect 0x7f1850041e30 2021-05-25T12:03:11.739 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.678+0000 7f1856ffd700 10 client.174267.objecter resend_mon_ops 2021-05-25T12:03:11.739 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.678+0000 7f1856ffd700 1 -- 172.21.15.102:0/256816174 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 768863589 0 0) 0x7f18480065a0 con 0x7f1850041e30 2021-05-25T12:03:11.739 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.678+0000 7f1856ffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:03:11.739 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.678+0000 7f1856ffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:03:11.740 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.678+0000 7f1856ffd700 10 monclient(hunting): my global_id is 95227 2021-05-25T12:03:11.740 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.678+0000 7f1856ffd700 1 -- 172.21.15.102:0/256816174 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18701401d0 con 0x7f1850041e30 2021-05-25T12:03:11.740 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.678+0000 7f1856ffd700 1 -- 172.21.15.102:0/256816174 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1042437457 0 0) 0x7f18701401d0 con 0x7f1850041e30 2021-05-25T12:03:11.740 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.678+0000 7f1856ffd700 1 -- 172.21.15.102:0/256816174 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f18480065a0 con 0x7f1850041e30 2021-05-25T12:03:11.741 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.678+0000 7f1856ffd700 1 -- 172.21.15.102:0/256816174 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f1858005560 con 0x7f1850041e30 2021-05-25T12:03:11.741 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.678+0000 7f1856ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:03:11.741 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.678+0000 7f1856ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T12:03:11.741 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.678+0000 7f1856ffd700 10 monclient: dump: 2021-05-25T12:03:11.742 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:03:11.742 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:03:11.742 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:03:11.742 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:03:11.742 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:03:11.743 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:03:11.743 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:03:11.743 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:03:11.743 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:03:11.744 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:03:11.744 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.678+0000 7f1856ffd700 1 -- 172.21.15.102:0/256816174 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2127894915 0 0) 0x7f18480065a0 con 0x7f1850041e30 2021-05-25T12:03:11.744 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.678+0000 7f1856ffd700 1 -- 172.21.15.102:0/256816174 --> v1:172.21.15.181:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f1858004dc0 con 0x7f1850041e30 2021-05-25T12:03:11.744 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.679+0000 7f1856ffd700 1 -- 172.21.15.102:0/256816174 <== mon.1 v1:172.21.15.181:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3931 (unknown 0 0 3445730903) 0x7f1858005840 con 0x7f1850041e30 2021-05-25T12:03:11.745 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.679+0000 7f1856ffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:03:11.745 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.679+0000 7f1856ffd700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:03:11.745 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.679+0000 7f1856ffd700 1 -- 172.21.15.102:0/256816174 >> v1:172.21.15.181:6789/0 conn(0x7f1850041e30 legacy=0x7f1850040890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:03:11.745 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.679+0000 7f1856ffd700 1 -- 172.21.15.102:0/256816174 >> v1:172.21.15.181:6789/0 conn(0x7f1850041e30 legacy=0x7f1850040890 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:03:11.745 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.679+0000 7f186f7fe700 1 -- 172.21.15.102:0/256816174 reap_dead start 2021-05-25T12:03:11.746 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:03:11.746 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:03:11.747 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:03:11.747 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "leader", 2021-05-25T12:03:11.747 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 32, 2021-05-25T12:03:11.747 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:03:11.747 INFO:teuthology.orchestra.run.smithi102.stdout: 0, 2021-05-25T12:03:11.748 INFO:teuthology.orchestra.run.smithi102.stdout: 1 2021-05-25T12:03:11.748 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:03:11.748 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 0, 2021-05-25T12:03:11.748 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:03:11.749 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:03:11.749 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:03:11.749 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:03:11.749 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:03:11.749 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:03:11.750 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:03:11.750 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:03:11.750 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:03:11.750 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:03:11.751 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:03:11.751 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:03:11.751 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:03:11.751 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:03:11.752 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:03:11.752 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:03:11.752 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:03:11.752 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:03:11.752 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:03:11.753 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:03:11.753 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:03:11.753 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:03:11.753 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:03:11.753 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:03:11.754 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:03:11.754 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:03:11.754 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:03:11.754 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:03:11.755 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:03:11.755 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:03:11.755 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:03:11.755 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:03:11.755 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:03:11.756 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:03:11.756 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:03:11.756 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:03:11.756 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:03:11.757 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:03:11.757 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:03:11.757 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:03:11.757 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:03:11.757 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:03:11.758 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:03:11.758 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:03:11.758 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:03:11.758 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:03:11.759 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:03:11.759 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:03:11.759 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:03:11.759 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:03:11.759 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:03:11.760 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:03:11.760 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:03:11.760 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:03:11.760 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:03:11.761 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:03:11.761 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:03:11.761 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:03:11.761 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:03:11.761 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:03:11.762 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:03:11.762 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:03:11.762 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:03:11.762 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:03:11.763 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:03:11.763 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:03:11.763 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:03:11.763 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:03:11.763 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:03:11.764 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:03:11.764 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:03:11.764 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:03:11.764 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:03:11.765 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:03:11.765 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:03:11.765 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:03:11.765 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:03:11.765 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:03:11.766 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:03:11.766 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:03:11.766 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:03:11.766 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:03:11.767 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:03:11.767 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:03:11.767 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:03:11.767 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:03:11.767 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:03:11.768 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:03:11.768 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:03:11.768 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:03:11.768 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:03:11.769 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:03:11.769 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:03:11.769 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:03:11.769 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:03:11.770 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:03:11.770 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:03:11.770 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:03:11.770 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:03:11.770 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:03:11.771 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:03:11.771 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:03:11.771 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:03:11.771 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:03:11.772 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:03:11.772 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:03:11.772 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:03:11.772 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:03:11.772 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:03:11.773 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:03:11.773 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:03:11.773 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:03:11.773 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:03:11.774 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:03:11.774 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:03:11.774 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:03:11.774 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:03:11.774 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:03:11.775 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": [ 2021-05-25T12:03:11.775 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:03:11.775 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:03:11.775 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:03:11.776 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 3 2021-05-25T12:03:11.776 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:03:11.776 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:03:11.776 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:03:11.776 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:03:11.777 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:03:11.777 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:03:11.777 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 4 2021-05-25T12:03:11.777 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:03:11.778 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:03:11.778 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:03:11.778 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:03:11.778 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:03:11.780 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.679+0000 7f187551a700 10 librados: watch_flush enter 2021-05-25T12:03:11.781 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.679+0000 7f187551a700 10 librados: watch_flush exit 2021-05-25T12:03:11.781 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.679+0000 7f187551a700 20 client.174267.objecter shutdown clearing up homeless session... 2021-05-25T12:03:11.781 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.679+0000 7f187551a700 10 client.174267.objecter successfully canceled tick 2021-05-25T12:03:11.781 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.679+0000 7f187551a700 1 -- 172.21.15.102:0/256816174 >> v1:172.21.15.181:6812/29582 conn(0x7f18500332a0 legacy=0x7f1850035760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:03:11.782 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.679+0000 7f187551a700 10 monclient: shutdown 2021-05-25T12:03:11.782 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.679+0000 7f187551a700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:03:11.782 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.679+0000 7f187551a700 1 -- 172.21.15.102:0/256816174 >> v1:172.21.15.102:6789/0 conn(0x7f1870136410 legacy=0x7f1870147760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:03:11.782 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.679+0000 7f187551a700 1 -- 172.21.15.102:0/256816174 shutdown_connections 2021-05-25T12:03:11.783 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.679+0000 7f187551a700 1 -- 172.21.15.102:0/256816174 >> 172.21.15.102:0/256816174 conn(0x7f187009b370 msgr2=0x7f1870139110 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:03:11.783 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.679+0000 7f187551a700 1 -- 172.21.15.102:0/256816174 shutdown_connections 2021-05-25T12:03:11.783 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.679+0000 7f187551a700 1 -- 172.21.15.102:0/256816174 wait complete. 2021-05-25T12:03:11.783 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:11.679+0000 7f187551a700 1 librados: shutdown 2021-05-25T12:03:11.784 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2021-05-25T12:03:14.341 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: [----------] 22 tests from LibRadosMisc_list: : cursor()=8:62a1935d:::14:head expected=8:62a1935d:::14:head 2021-05-25T12:03:14.341 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: > 8:62a1935d:::14:head -> 14 2021-05-25T12:03:14.342 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: : entry=14 expected=14 2021-05-25T12:03:14.342 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: : seek to 8:e960b815:::9:head 2021-05-25T12:03:14.342 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: : cursor()=8:e960b815:::9:head expected=8:e960b815:::9:head 2021-05-25T12:03:14.342 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: > 8:e960b815:::9:head -> 9 2021-05-25T12:03:14.343 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: : entry=9 expected=9 2021-05-25T12:03:14.343 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: : seek to 8:de5d7c5f:::12:head 2021-05-25T12:03:14.343 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: : cursor()=8:de5d7c5f:::12:head expected=8:de5d7c5f:::12:head 2021-05-25T12:03:14.343 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: > 8:de5d7c5f:::12:head -> 12 2021-05-25T12:03:14.344 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: : entry=12 expected=12 2021-05-25T12:03:14.344 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: : seek to 8:d83876eb:::4:head 2021-05-25T12:03:14.344 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: : cursor()=8:d83876eb:::4:head expected=8:d83876eb:::4:head 2021-05-25T12:03:14.344 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: > 8:d83876eb:::4:head -> 4 2021-05-25T12:03:14.344 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: : entry=4 expected=4 2021-05-25T12:03:14.345 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: : seek to 8:cfc208b3:::3:head 2021-05-25T12:03:14.345 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: : cursor()=8:cfc208b3:::3:head expected=8:cfc208b3:::3:head 2021-05-25T12:03:14.345 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: > 8:cfc208b3:::3:head -> 3 2021-05-25T12:03:14.345 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: : entry=3 expected=3 2021-05-25T12:03:14.345 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: : seek to 8:c4fdafeb:::6:head 2021-05-25T12:03:14.346 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: : cursor()=8:c4fdafeb:::6:head expected=8:c4fdafeb:::6:head 2021-05-25T12:03:14.346 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: > 8:c4fdafeb:::6:head -> 6 2021-05-25T12:03:14.346 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: : entry=6 expected=6 2021-05-25T12:03:14.346 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: : seek to 8:bd63b0f1:::8:head 2021-05-25T12:03:14.347 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: : cursor()=8:bd63b0f1:::8:head expected=8:bd63b0f1:::8:head 2021-05-25T12:03:14.347 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: > 8:bd63b0f1:::8:head -> 8 2021-05-25T12:03:14.347 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: : entry=8 expected=8 2021-05-25T12:03:14.347 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: : seek to 8:52ea6a34:::10:head 2021-05-25T12:03:14.347 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: : cursor()=8:52ea6a34:::10:head expected=8:52ea6a34:::10:head 2021-05-25T12:03:14.348 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: > 8:52ea6a34:::10:head -> 10 2021-05-25T12:03:14.348 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: : entry=10 expected=10 2021-05-25T12:03:14.348 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: [ OK ] LibRadosList.ListObjectsCursor (207 ms) 2021-05-25T12:03:14.348 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjects 2021-05-25T12:03:14.349 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: [ OK ] LibRadosList.EnumerateObjects (62976 ms) 2021-05-25T12:03:14.349 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjectsSplit 2021-05-25T12:03:14.349 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: split 0/5 -> MIN 8:33333333::::head 2021-05-25T12:03:14.349 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: split 1/5 -> 8:33333333::::head 8:66666666::::head 2021-05-25T12:03:14.349 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: split 2/5 -> 8:66666666::::head 8:99999999::::head 2021-05-25T12:03:14.350 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: split 3/5 -> 8:99999999::::head 8:cccccccc::::head 2021-05-25T12:03:14.350 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: split 4/5 -> 8:cccccccc::::head MAX 2021-05-25T12:03:14.350 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: [ OK ] LibRadosList.EnumerateObjectsSplit (44931 ms) 2021-05-25T12:03:14.350 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: [----------] 7 tests from LibRadosList (108481 ms total) 2021-05-25T12:03:14.351 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: 2021-05-25T12:03:14.351 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: [----------] 3 tests from LibRadosListEC 2021-05-25T12:03:14.351 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: [ RUN ] LibRadosListEC.ListObjects 2021-05-25T12:03:14.352 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: [ OK ] LibRadosListEC.ListObjects (1051 ms) 2021-05-25T12:03:14.352 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsNS 2021-05-25T12:03:14.352 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: myset foo1,foo2,foo3 2021-05-25T12:03:14.352 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: foo1 2021-05-25T12:03:14.352 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: foo2 2021-05-25T12:03:14.353 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: foo3 2021-05-25T12:03:14.353 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: myset foo1,foo4,foo5 2021-05-25T12:03:14.353 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: foo4 2021-05-25T12:03:14.353 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: foo5 2021-05-25T12:03:14.354 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: foo1 2021-05-25T12:03:14.354 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: myset foo6,foo7 2021-05-25T12:03:14.354 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: foo7 2021-05-25T12:03:14.354 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: foo6 2021-05-25T12:03:14.355 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2021-05-25T12:03:14.355 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: ns1:foo4 2021-05-25T12:03:14.355 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: ns1:foo5 2021-05-25T12:03:14.355 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: ns2:foo7 2021-05-25T12:03:14.356 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: ns2:foo6 2021-05-25T12:03:14.356 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: ns1:foo1 2021-05-25T12:03:14.356 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: :foo1 2021-05-25T12:03:14.357 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: :foo2 2021-05-25T12:03:14.357 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: :foo3 2021-05-25T12:03:14.357 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsNS (95 ms) 2021-05-25T12:03:14.357 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsStart 2021-05-25T12:03:14.357 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: 1 0 2021-05-25T12:03:14.358 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: 10 0 2021-05-25T12:03:14.358 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: 13 0 2021-05-25T12:03:20.161 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: 7 0 2021-05-25T12:03:20.162 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: 14 0 2021-05-25T12:03:20.162 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: 0 0 2021-05-25T12:03:20.162 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: 15 0 2021-05-25T12:03:20.163 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: 11 0 2021-05-25T12:03:20.163 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: 5 0 2021-05-25T12:03:20.163 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: 8 0 2021-05-25T12:03:20.163 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: 6 0 2021-05-25T12:03:20.164 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: 3 0 2021-05-25T12:03:20.164 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: 4 0 2021-05-25T12:03:20.164 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: 12 0 2021-05-25T12:03:20.164 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: 9 0 2021-05-25T12:03:20.164 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: 2 0 2021-05-25T12:03:20.165 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2021-05-25T12:03:20.165 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsStart (125 ms) 2021-05-25T12:03:20.165 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: [----------] 3 tests from LibRadosListEC (1271 ms total) 2021-05-25T12:03:20.165 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: 2021-05-25T12:03:20.165 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: [----------] 1 test from LibRadosListNP 2021-05-25T12:03:20.166 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: [ RUN ] LibRadosListNP.ListObjectsError 2021-05-25T12:03:20.166 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: [ OK ] LibRadosListNP.ListObjectsError (3006 ms) 2021-05-25T12:03:20.166 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: [----------] 1 test from LibRadosListNP (3006 ms total) 2021-05-25T12:03:20.166 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: 2021-05-25T12:03:20.167 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: [----------] Global test environment tear-down 2021-05-25T12:03:20.167 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: [==========] 11 tests from 3 test suites ran. (160444 ms total) 2021-05-25T12:03:20.167 INFO:tasks.workunit.client.0.smithi181.stdout: api_list: [ PASSED ] 11 tests. 2021-05-25T12:03:23.066 INFO:tasks.ceph.osd.4.smithi181.stderr:2021-05-25T12:03:23.062+0000 7f0006e6f700 -1 osd.4 pg_epoch: 97 pg[109.5s0( v 96'3 (0'0,96'3] local-lis/les=86/87 n=2 ec=86/86 lis/c=86/86 les/c/f=87/87/0 sis=86) [4,0,1]p4(0) r=0 lpr=86 crt=96'3 lcod 95'1 mlcod 95'1 active+clean+snaptrim trimq=[b~1] ps=[2~1,6~2]] removing snap head 2021-05-25T12:03:27.120 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots_pp: Running main() from gmock_main.cc 2021-05-25T12:03:27.120 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots_pp: [==========] Running 18 tests from 4 test suites. 2021-05-25T12:03:27.121 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots_pp: [----------] Global test environment set-up. 2021-05-25T12:03:27.121 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP 2021-05-25T12:03:27.121 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots_pp: seed 33083 2021-05-25T12:03:27.121 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapListPP 2021-05-25T12:03:27.121 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapListPP (2062 ms) 2021-05-25T12:03:27.122 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapRemovePP 2021-05-25T12:03:27.122 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapRemovePP (1965 ms) 2021-05-25T12:03:27.122 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.RollbackPP 2021-05-25T12:03:27.122 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.RollbackPP (1985 ms) 2021-05-25T12:03:27.123 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapGetNamePP 2021-05-25T12:03:27.123 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapGetNamePP (2001 ms) 2021-05-25T12:03:27.123 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapCreateRemovePP 2021-05-25T12:03:27.123 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapCreateRemovePP (3665 ms) 2021-05-25T12:03:27.123 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP (11678 ms total) 2021-05-25T12:03:27.124 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots_pp: 2021-05-25T12:03:27.124 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots_pp: [----------] 6 tests from LibRadosSnapshotsSelfManagedPP 2021-05-25T12:03:27.124 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapPP 2021-05-25T12:03:27.124 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapPP (4013 ms) 2021-05-25T12:03:27.125 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.RollbackPP 2021-05-25T12:03:27.125 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.RollbackPP (4047 ms) 2021-05-25T12:03:27.125 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP 2021-05-25T12:03:27.125 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP (31715 ms) 2021-05-25T12:03:27.125 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.Bug11677 2021-05-25T12:03:27.126 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.Bug11677 (3991 ms) 2021-05-25T12:03:27.126 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.OrderSnap 2021-05-25T12:03:27.126 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.OrderSnap (2043 ms) 2021-05-25T12:03:27.126 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap 2021-05-25T12:03:27.127 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots_pp: deleting snap 25 in pool test-rados-api-smithi181-33083-7 2021-05-25T12:03:27.127 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots_pp: waiting for snaps to purge 2021-05-25T12:03:27.127 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap (17960 ms) 2021-05-25T12:03:27.127 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots_pp: [----------] 6 tests from LibRadosSnapshotsSelfManagedPP (63770 ms total) 2021-05-25T12:03:27.127 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots_pp: 2021-05-25T12:03:27.128 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP 2021-05-25T12:03:27.128 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapListPP 2021-05-25T12:03:27.128 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapListPP (3002 ms) 2021-05-25T12:03:27.128 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapRemovePP 2021-05-25T12:03:27.129 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapRemovePP (2006 ms) 2021-05-25T12:03:27.129 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.RollbackPP 2021-05-25T12:03:27.129 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.RollbackPP (2009 ms) 2021-05-25T12:03:27.129 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapGetNamePP 2021-05-25T12:03:27.130 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapGetNamePP (11837 ms) 2021-05-25T12:03:27.130 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP (18854 ms total) 2021-05-25T12:03:27.130 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots_pp: 2021-05-25T12:03:27.130 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP 2021-05-25T12:03:27.130 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.SnapPP 2021-05-25T12:03:27.131 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.SnapPP (3984 ms) 2021-05-25T12:03:27.131 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.RollbackPP 2021-05-25T12:03:27.131 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.RollbackPP (4008 ms) 2021-05-25T12:03:27.131 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.Bug11677 2021-05-25T12:03:27.132 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.Bug11677 (3905 ms) 2021-05-25T12:03:27.132 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP (11897 ms total) 2021-05-25T12:03:27.132 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots_pp: 2021-05-25T12:03:27.132 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots_pp: [----------] Global test environment tear-down 2021-05-25T12:03:27.132 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots_pp: [==========] 18 tests from 4 test suites ran. (167352 ms total) 2021-05-25T12:03:27.133 INFO:tasks.workunit.client.0.smithi181.stdout: api_snapshots_pp: [ PASSED ] 18 tests. 2021-05-25T12:03:27.133 INFO:tasks.workunit.client.0.smithi181.stderr:+ for t in "${!pids[@]}" 2021-05-25T12:03:27.134 INFO:tasks.workunit.client.0.smithi181.stderr:+ pid=32971 2021-05-25T12:03:27.134 INFO:tasks.workunit.client.0.smithi181.stderr:+ wait 32971 2021-05-25T12:03:27.134 INFO:tasks.workunit.client.0.smithi181.stderr:+ for t in "${!pids[@]}" 2021-05-25T12:03:27.134 INFO:tasks.workunit.client.0.smithi181.stderr:+ pid=32921 2021-05-25T12:03:27.134 INFO:tasks.workunit.client.0.smithi181.stderr:+ wait 32921 2021-05-25T12:03:27.135 INFO:tasks.workunit.client.0.smithi181.stderr:+ for t in "${!pids[@]}" 2021-05-25T12:03:27.135 INFO:tasks.workunit.client.0.smithi181.stderr:+ pid=32877 2021-05-25T12:03:27.135 INFO:tasks.workunit.client.0.smithi181.stderr:+ wait 32877 2021-05-25T12:03:27.135 INFO:tasks.workunit.client.0.smithi181.stderr:+ for t in "${!pids[@]}" 2021-05-25T12:03:27.136 INFO:tasks.workunit.client.0.smithi181.stderr:+ pid=33175 2021-05-25T12:03:27.136 INFO:tasks.workunit.client.0.smithi181.stderr:+ wait 33175 2021-05-25T12:03:27.136 INFO:tasks.workunit.client.0.smithi181.stderr:+ for t in "${!pids[@]}" 2021-05-25T12:03:27.136 INFO:tasks.workunit.client.0.smithi181.stderr:+ pid=33141 2021-05-25T12:03:27.136 INFO:tasks.workunit.client.0.smithi181.stderr:+ wait 33141 2021-05-25T12:03:27.137 INFO:tasks.workunit.client.0.smithi181.stderr:+ for t in "${!pids[@]}" 2021-05-25T12:03:27.137 INFO:tasks.workunit.client.0.smithi181.stderr:+ pid=33339 2021-05-25T12:03:27.137 INFO:tasks.workunit.client.0.smithi181.stderr:+ wait 33339 2021-05-25T12:03:27.137 INFO:tasks.workunit.client.0.smithi181.stderr:+ for t in "${!pids[@]}" 2021-05-25T12:03:27.138 INFO:tasks.workunit.client.0.smithi181.stderr:+ pid=32845 2021-05-25T12:03:27.138 INFO:tasks.workunit.client.0.smithi181.stderr:+ wait 32845 2021-05-25T12:03:31.784 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2021-05-25T12:03:31.785 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2021-05-25T12:03:31.785 INFO:tasks.ceph.mon.c:Restarting daemon 2021-05-25T12:03:31.786 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2021-05-25T12:03:31.789 INFO:tasks.ceph.mon.c:Started 2021-05-25T12:03:31.790 INFO:tasks.mon_thrash:Sending STOP to mon b 2021-05-25T12:03:31.790 INFO:tasks.ceph.mon.b:Sent signal 19 2021-05-25T12:03:31.791 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs to unfreeze mons 2021-05-25T12:03:51.792 INFO:tasks.mon_thrash:Sending CONT to mon b 2021-05-25T12:03:51.793 INFO:tasks.ceph.mon.b:Sent signal 18 2021-05-25T12:03:51.794 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2021-05-25T12:03:51.794 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2021-05-25T12:03:51.903 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.902+0000 7f967a9eb700 10 monclient: get_monmap_and_config 2021-05-25T12:03:51.904 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.902+0000 7f967a9eb700 10 monclient: build_initial_monmap 2021-05-25T12:03:51.904 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.902+0000 7f967a9eb700 10 monclient: monmap: 2021-05-25T12:03:51.904 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:03:51.905 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:03:51.905 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:03:51.903523+0000 2021-05-25T12:03:51.905 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:03:51.903523+0000 2021-05-25T12:03:51.905 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:03:51.906 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:03:51.906 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:03:51.906 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:03:51.906 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:03:51.906 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:03:51.916 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.903+0000 7f967a9eb700 1 Processor -- start 2021-05-25T12:03:51.916 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.903+0000 7f967a9eb700 1 -- start start 2021-05-25T12:03:51.917 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.903+0000 7f967a9eb700 10 monclient: init 2021-05-25T12:03:51.917 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.903+0000 7f967a9eb700 10 monclient: _reopen_session rank -1 2021-05-25T12:03:51.917 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.903+0000 7f967a9eb700 10 monclient: _add_conns ranks=[1,2,0] 2021-05-25T12:03:51.917 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.903+0000 7f967a9eb700 10 monclient(hunting): picked mon.noname-c con 0x7f9674098930 addr v1:172.21.15.181:6789/0 2021-05-25T12:03:51.918 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.903+0000 7f967a9eb700 10 monclient(hunting): picked mon.noname-b con 0x7f967412bca0 addr v1:172.21.15.102:6790/0 2021-05-25T12:03:51.918 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.903+0000 7f967a9eb700 10 monclient(hunting): picked mon.noname-a con 0x7f967412c150 addr v1:172.21.15.102:6789/0 2021-05-25T12:03:51.918 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.903+0000 7f967a9eb700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9674099b20 con 0x7f967412c150 2021-05-25T12:03:51.919 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.903+0000 7f967a9eb700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f967409a3a0 con 0x7f9674098930 2021-05-25T12:03:51.919 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.903+0000 7f967a9eb700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f967409ac60 con 0x7f967412bca0 2021-05-25T12:03:51.919 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.903+0000 7f967a9eb700 10 monclient(hunting): _renew_subs 2021-05-25T12:03:51.919 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.903+0000 7f967a9eb700 10 monclient(hunting): authenticate will time out at 1032.500960s 2021-05-25T12:03:51.920 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.904+0000 7f967a1ea700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f967412c150 0x7f9674133ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60552/0 (socket says 172.21.15.102:60552) 2021-05-25T12:03:51.920 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.904+0000 7f96791e8700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f967412bca0 0x7f96741308c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:55428/0 (socket says 172.21.15.102:55428) 2021-05-25T12:03:51.920 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.904+0000 7f967a1ea700 1 -- 172.21.15.102:0/2878001560 learned_addr learned my addr 172.21.15.102:0/2878001560 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:03:51.920 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.904+0000 7f96789e7700 1 -- 172.21.15.102:0/2878001560 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3583649566 0 0) 0x7f9674099b20 con 0x7f967412c150 2021-05-25T12:03:51.921 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.904+0000 7f96789e7700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:03:51.921 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.904+0000 7f96789e7700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:03:51.921 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.904+0000 7f96789e7700 10 monclient(hunting): my global_id is 174366 2021-05-25T12:03:51.921 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.904+0000 7f96789e7700 1 -- 172.21.15.102:0/2878001560 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f965c003270 con 0x7f967412c150 2021-05-25T12:03:51.921 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.904+0000 7f96789e7700 1 -- 172.21.15.102:0/2878001560 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 279278525 0 0) 0x7f967409ac60 con 0x7f967412bca0 2021-05-25T12:03:51.922 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.904+0000 7f96789e7700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:03:51.922 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.904+0000 7f96789e7700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:03:51.922 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.904+0000 7f96789e7700 10 monclient(hunting): my global_id is 184100 2021-05-25T12:03:51.922 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.904+0000 7f96789e7700 1 -- 172.21.15.102:0/2878001560 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9674099b20 con 0x7f967412bca0 2021-05-25T12:03:51.923 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.904+0000 7f96789e7700 1 -- 172.21.15.102:0/2878001560 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1047982219 0 0) 0x7f9674099b20 con 0x7f967412bca0 2021-05-25T12:03:51.923 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.904+0000 7f96789e7700 1 -- 172.21.15.102:0/2878001560 >> v1:172.21.15.181:6789/0 conn(0x7f9674098930 legacy=0x7f967412b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:03:51.923 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.904+0000 7f96789e7700 1 -- 172.21.15.102:0/2878001560 >> v1:172.21.15.102:6789/0 conn(0x7f967412c150 legacy=0x7f9674133ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:03:51.923 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.904+0000 7f96789e7700 10 monclient: _finish_hunting 0 2021-05-25T12:03:51.924 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.904+0000 7f96789e7700 1 monclient: found mon.noname-b 2021-05-25T12:03:51.924 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.904+0000 7f96789e7700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:03:51.924 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.904+0000 7f96789e7700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T12:03:51.924 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.904+0000 7f96789e7700 1 -- 172.21.15.102:0/2878001560 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f967409af00 con 0x7f967412bca0 2021-05-25T12:03:51.925 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.904+0000 7f96789e7700 10 monclient: _finish_auth 0 2021-05-25T12:03:51.925 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.904+0000 7f96789e7700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:03:51.925 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.904+0000 7f967a9eb700 5 monclient: authenticate success, global_id 184100 2021-05-25T12:03:51.925 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.904+0000 7f96789e7700 1 -- 172.21.15.102:0/2878001560 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f96640015f0 con 0x7f967412bca0 2021-05-25T12:03:51.926 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.904+0000 7f967a9eb700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:03:51.926 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.904+0000 7f96789e7700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:03:51.926 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.904+0000 7f96789e7700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:03:51.926 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.904+0000 7f96789e7700 10 monclient: dump: 2021-05-25T12:03:51.926 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:03:51.927 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:03:51.927 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:03:51.927 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:03:51.927 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:03:51.927 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:03:51.928 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:03:51.928 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:03:51.928 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:03:51.928 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:03:51.929 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.904+0000 7f967a9eb700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:03:51.929 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.904+0000 7f96789e7700 1 -- 172.21.15.102:0/2878001560 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9664001820 con 0x7f967412bca0 2021-05-25T12:03:51.929 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.904+0000 7f96789e7700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:03:51.929 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.904+0000 7f96789e7700 1 -- 172.21.15.102:0/2878001560 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f9664001bf0 con 0x7f967412bca0 2021-05-25T12:03:51.929 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.904+0000 7f967a9eb700 10 monclient: get_monmap_and_config success 2021-05-25T12:03:51.930 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.904+0000 7f967a9eb700 4 set_mon_vals no callback set 2021-05-25T12:03:51.930 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.904+0000 7f96789e7700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:03:51.930 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.904+0000 7f96789e7700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:03:51.930 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.904+0000 7f96789e7700 10 monclient: dump: 2021-05-25T12:03:51.931 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:03:51.931 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:03:51.931 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:03:51.931 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:03:51.931 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:03:51.932 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:03:51.932 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:03:51.932 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:03:51.932 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:03:51.932 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:03:51.933 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.905+0000 7f967a9eb700 10 monclient: shutdown 2021-05-25T12:03:51.933 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.905+0000 7f967a9eb700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:03:51.933 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.905+0000 7f967a9eb700 1 -- 172.21.15.102:0/2878001560 >> v1:172.21.15.102:6790/0 conn(0x7f967412bca0 legacy=0x7f96741308c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:03:51.933 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.905+0000 7f967a9eb700 1 -- 172.21.15.102:0/2878001560 shutdown_connections 2021-05-25T12:03:51.933 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.905+0000 7f967a9eb700 1 -- 172.21.15.102:0/2878001560 >> 172.21.15.102:0/2878001560 conn(0x7f967409e9d0 msgr2=0x7f967409ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:03:51.934 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.905+0000 7f967a9eb700 1 -- 172.21.15.102:0/2878001560 shutdown_connections 2021-05-25T12:03:51.934 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.905+0000 7f967a9eb700 1 -- 172.21.15.102:0/2878001560 wait complete. 2021-05-25T12:03:51.934 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.905+0000 7f967a9eb700 10 monclient: build_initial_monmap 2021-05-25T12:03:51.934 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.905+0000 7f967a9eb700 10 monclient: monmap: 2021-05-25T12:03:51.934 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:03:51.935 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:03:51.935 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:03:51.935 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:03:51.935 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:03:51.935 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:03:51.936 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:03:51.936 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:03:51.936 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:03:51.936 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:03:51.937 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.905+0000 7f967a9eb700 1 librados: starting msgr at 2021-05-25T12:03:51.937 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.905+0000 7f967a9eb700 1 librados: starting objecter 2021-05-25T12:03:51.937 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.906+0000 7f967a9eb700 1 Processor -- start 2021-05-25T12:03:51.937 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.906+0000 7f967a9eb700 1 -- start start 2021-05-25T12:03:51.937 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.906+0000 7f967a9eb700 1 librados: setting wanted keys 2021-05-25T12:03:51.938 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.906+0000 7f967a9eb700 1 librados: calling monclient init 2021-05-25T12:03:51.938 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.906+0000 7f967a9eb700 10 monclient: init 2021-05-25T12:03:51.938 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.906+0000 7f967a9eb700 10 monclient: _reopen_session rank -1 2021-05-25T12:03:51.938 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.906+0000 7f967a9eb700 10 monclient: _add_conns ranks=[2,1,0] 2021-05-25T12:03:51.939 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.906+0000 7f967a9eb700 10 monclient(hunting): picked mon.noname-c con 0x7f9674098930 addr v1:172.21.15.102:6790/0 2021-05-25T12:03:51.939 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.906+0000 7f967a9eb700 10 monclient(hunting): picked mon.noname-b con 0x7f967412bca0 addr v1:172.21.15.181:6789/0 2021-05-25T12:03:51.939 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.906+0000 7f967a9eb700 10 monclient(hunting): picked mon.noname-a con 0x7f967412c150 addr v1:172.21.15.102:6789/0 2021-05-25T12:03:51.939 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.906+0000 7f967a9eb700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96740949e0 con 0x7f967412c150 2021-05-25T12:03:51.939 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.906+0000 7f967a9eb700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9674094c80 con 0x7f967412bca0 2021-05-25T12:03:51.940 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.906+0000 7f967a9eb700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9674094170 con 0x7f9674098930 2021-05-25T12:03:51.940 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.906+0000 7f967a9eb700 10 monclient(hunting): _renew_subs 2021-05-25T12:03:51.940 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.906+0000 7f967a9eb700 10 monclient(hunting): authenticate will time out at 1032.503491s 2021-05-25T12:03:51.940 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.906+0000 7f96799e9700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f9674098930 0x7f9674097140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:55432/0 (socket says 172.21.15.102:55432) 2021-05-25T12:03:51.940 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.906+0000 7f96799e9700 1 -- 172.21.15.102:0/2029561255 learned_addr learned my addr 172.21.15.102:0/2029561255 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:03:51.941 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.906+0000 7f9661ffb700 10 client.?.objecter ms_handle_connect 0x7f9674098930 2021-05-25T12:03:51.941 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.906+0000 7f9661ffb700 10 client.?.objecter resend_mon_ops 2021-05-25T12:03:51.941 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.906+0000 7f9661ffb700 10 client.?.objecter ms_handle_connect 0x7f967412c150 2021-05-25T12:03:51.941 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.906+0000 7f9661ffb700 10 client.?.objecter resend_mon_ops 2021-05-25T12:03:51.942 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.906+0000 7f9661ffb700 1 -- 172.21.15.102:0/2029561255 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1383386714 0 0) 0x7f9674094170 con 0x7f9674098930 2021-05-25T12:03:51.942 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.906+0000 7f9661ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:03:51.942 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.906+0000 7f9661ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:03:51.942 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.906+0000 7f9661ffb700 10 monclient(hunting): my global_id is 184103 2021-05-25T12:03:51.942 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.906+0000 7f9661ffb700 1 -- 172.21.15.102:0/2029561255 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9654004c10 con 0x7f9674098930 2021-05-25T12:03:51.943 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.906+0000 7f9661ffb700 1 -- 172.21.15.102:0/2029561255 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1151261107 0 0) 0x7f96740949e0 con 0x7f967412c150 2021-05-25T12:03:51.943 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.906+0000 7f9661ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:03:51.943 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.906+0000 7f9661ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:03:51.943 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.906+0000 7f9661ffb700 10 monclient(hunting): my global_id is 174369 2021-05-25T12:03:51.944 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.906+0000 7f9661ffb700 1 -- 172.21.15.102:0/2029561255 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9674094170 con 0x7f967412c150 2021-05-25T12:03:51.944 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.906+0000 7f9661ffb700 10 client.?.objecter ms_handle_connect 0x7f967412bca0 2021-05-25T12:03:51.944 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.906+0000 7f9661ffb700 10 client.?.objecter resend_mon_ops 2021-05-25T12:03:51.944 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.907+0000 7f9661ffb700 1 -- 172.21.15.102:0/2029561255 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1484510818 0 0) 0x7f9654004c10 con 0x7f9674098930 2021-05-25T12:03:51.945 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.907+0000 7f9661ffb700 1 -- 172.21.15.102:0/2029561255 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f96740949e0 con 0x7f9674098930 2021-05-25T12:03:51.945 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.907+0000 7f9661ffb700 1 -- 172.21.15.102:0/2029561255 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f96680018e0 con 0x7f9674098930 2021-05-25T12:03:51.945 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.907+0000 7f9661ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:03:51.945 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.907+0000 7f9661ffb700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:03:51.946 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.907+0000 7f9661ffb700 10 monclient(hunting): dump: 2021-05-25T12:03:51.946 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:03:51.946 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:03:51.946 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:03:51.946 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:03:51.947 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:03:51.947 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:03:51.947 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:03:51.947 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:03:51.947 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:03:51.948 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:03:51.948 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.907+0000 7f9661ffb700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:03:51.948 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.907+0000 7f9661ffb700 1 -- 172.21.15.102:0/2029561255 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2042228720 0 0) 0x7f9674094170 con 0x7f967412c150 2021-05-25T12:03:51.948 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.907+0000 7f9661ffb700 1 -- 172.21.15.102:0/2029561255 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9654004c10 con 0x7f967412c150 2021-05-25T12:03:51.949 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.907+0000 7f9661ffb700 1 -- 172.21.15.102:0/2029561255 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f9670001df0 con 0x7f967412c150 2021-05-25T12:03:51.949 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.907+0000 7f9661ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:03:51.949 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.907+0000 7f9661ffb700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2021-05-25T12:03:51.949 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.907+0000 7f9661ffb700 10 monclient(hunting): dump: 2021-05-25T12:03:51.949 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:03:51.950 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:03:51.950 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:03:51.950 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:03:51.950 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:03:51.950 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:03:51.951 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:03:51.951 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:03:51.951 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:03:51.951 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:03:51.951 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.907+0000 7f9661ffb700 1 -- 172.21.15.102:0/2029561255 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 643669503 0 0) 0x7f96740949e0 con 0x7f9674098930 2021-05-25T12:03:51.952 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.907+0000 7f9661ffb700 1 -- 172.21.15.102:0/2029561255 >> v1:172.21.15.181:6789/0 conn(0x7f967412bca0 legacy=0x7f967413a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:03:51.952 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.907+0000 7f9661ffb700 1 -- 172.21.15.102:0/2029561255 >> v1:172.21.15.102:6789/0 conn(0x7f967412c150 legacy=0x7f9674142d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:03:51.952 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.907+0000 7f9661ffb700 10 monclient: _finish_hunting 0 2021-05-25T12:03:51.952 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.907+0000 7f9661ffb700 1 monclient: found mon.c 2021-05-25T12:03:51.952 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.907+0000 7f9661ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:03:51.953 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.907+0000 7f9661ffb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:03:51.953 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.907+0000 7f9661ffb700 1 -- 172.21.15.102:0/2029561255 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9674097850 con 0x7f9674098930 2021-05-25T12:03:51.953 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.907+0000 7f9661ffb700 10 monclient: _finish_auth 0 2021-05-25T12:03:51.953 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.907+0000 7f9661ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:03:51.953 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.907+0000 7f967a9eb700 5 monclient: authenticate success, global_id 184103 2021-05-25T12:03:51.954 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.907+0000 7f967a9eb700 10 monclient: _renew_subs 2021-05-25T12:03:51.954 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.907+0000 7f967a9eb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:03:51.954 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.907+0000 7f967a9eb700 1 -- 172.21.15.102:0/2029561255 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9674097a40 con 0x7f9674098930 2021-05-25T12:03:51.954 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.907+0000 7f9661ffb700 1 -- 172.21.15.102:0/2029561255 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9668001f80 con 0x7f9674098930 2021-05-25T12:03:51.955 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.907+0000 7f9661ffb700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:03:51.955 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.907+0000 7f9661ffb700 1 -- 172.21.15.102:0/2029561255 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f9668002310 con 0x7f9674098930 2021-05-25T12:03:51.955 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.907+0000 7f9661ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:03:51.955 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.907+0000 7f9661ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:03:51.955 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.907+0000 7f9661ffb700 10 monclient: dump: 2021-05-25T12:03:51.956 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:03:51.956 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:03:51.956 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:03:51.956 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:03:51.956 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:03:51.957 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:03:51.957 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:03:51.957 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:03:51.957 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:03:51.957 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:03:51.958 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.907+0000 7f9663fff700 4 set_mon_vals no callback set 2021-05-25T12:03:51.958 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.907+0000 7f967a9eb700 10 client.184103.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:03:51.958 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.907+0000 7f967a9eb700 10 monclient: _renew_subs 2021-05-25T12:03:51.958 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.907+0000 7f967a9eb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:03:51.958 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.907+0000 7f967a9eb700 1 -- 172.21.15.102:0/2029561255 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f967413b240 con 0x7f9674098930 2021-05-25T12:03:51.959 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.907+0000 7f967a9eb700 1 librados: init done 2021-05-25T12:03:51.959 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.908+0000 7f9661ffb700 1 -- 172.21.15.102:0/2029561255 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f96680024f0 con 0x7f9674098930 2021-05-25T12:03:51.959 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.908+0000 7f967a9eb700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:03:51.959 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.908+0000 7f967a9eb700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:03:51.959 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.908+0000 7f967a9eb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:03:51.960 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.908+0000 7f967a9eb700 1 -- 172.21.15.102:0/2029561255 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9674059730 con 0x7f9674098930 2021-05-25T12:03:51.960 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.908+0000 7f9661ffb700 1 -- 172.21.15.102:0/2029561255 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(112..112 src has 1..112) v4 ==== 5539+0+0 (unknown 3347426993 0 0) 0x7f966803d650 con 0x7f9674098930 2021-05-25T12:03:51.960 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.908+0000 7f9661ffb700 10 client.184103.objecter ms_dispatch 0x7f9674001550 osd_map(112..112 src has 1..112) v4 2021-05-25T12:03:51.960 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.908+0000 7f9661ffb700 3 client.184103.objecter handle_osd_map got epochs [112,112] > 0 2021-05-25T12:03:51.961 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.908+0000 7f9661ffb700 3 client.184103.objecter handle_osd_map decoding full epoch 112 2021-05-25T12:03:51.961 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.908+0000 7f9661ffb700 20 client.184103.objecter dump_active .. 0 homeless 2021-05-25T12:03:51.961 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.908+0000 7f96791e8700 10 monclient: get_auth_request con 0x7f96540332a0 auth_method 0 2021-05-25T12:03:51.961 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.908+0000 7f9661ffb700 10 client.184103.objecter ms_handle_connect 0x7f96540332a0 2021-05-25T12:03:51.961 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.912+0000 7f9661ffb700 1 -- 172.21.15.102:0/2029561255 <== mon.2 v1:172.21.15.102:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7f9668012de0 con 0x7f9674098930 2021-05-25T12:03:51.962 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.912+0000 7f9661ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:03:51.962 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:51.912+0000 7f9661ffb700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:03:52.121 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:52.119+0000 7f967a9eb700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2021-05-25T12:03:52.121 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:52.119+0000 7f967a9eb700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2021-05-25T12:03:52.121 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:52.119+0000 7f967a9eb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:03:52.121 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:52.119+0000 7f967a9eb700 1 -- 172.21.15.102:0/2029561255 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f967405d560 con 0x7f9674098930 2021-05-25T12:03:52.122 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:52.120+0000 7f9661ffb700 1 -- 172.21.15.102:0/2029561255 <== mon.2 v1:172.21.15.102:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1336 (unknown 254746657 0 1749335151) 0x7f9668012bb0 con 0x7f9674098930 2021-05-25T12:03:52.122 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:52.120+0000 7f9661ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2021-05-25T12:03:52.122 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:52.120+0000 7f9661ffb700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:03:52.123 INFO:teuthology.orchestra.run.smithi102.stdout:{"election_epoch":34,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138303579357183","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"20ee91b6-8935-4b98-b868-85c03789ccc4","modified":"2021-05-25T11:59:11.011522Z","created":"2021-05-25T11:59:11.011522Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6790","nonce":0}]},"addr":"172.21.15.102:6790/0","public_addr":"172.21.15.102:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2021-05-25T12:03:52.124 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:52.121+0000 7f967a9eb700 10 librados: watch_flush enter 2021-05-25T12:03:52.124 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:52.121+0000 7f967a9eb700 10 librados: watch_flush exit 2021-05-25T12:03:52.125 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:52.121+0000 7f967a9eb700 20 client.184103.objecter shutdown clearing up homeless session... 2021-05-25T12:03:52.125 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:52.121+0000 7f967a9eb700 10 client.184103.objecter successfully canceled tick 2021-05-25T12:03:52.125 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:52.121+0000 7f967a9eb700 1 -- 172.21.15.102:0/2029561255 >> v1:172.21.15.181:6812/29582 conn(0x7f96540332a0 legacy=0x7f9654035760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:03:52.125 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:52.121+0000 7f967a9eb700 10 monclient: shutdown 2021-05-25T12:03:52.125 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:52.121+0000 7f967a9eb700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:03:52.126 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:52.121+0000 7f967a9eb700 1 -- 172.21.15.102:0/2029561255 >> v1:172.21.15.102:6790/0 conn(0x7f9674098930 legacy=0x7f9674097140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:03:52.126 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:52.121+0000 7f967a9eb700 1 -- 172.21.15.102:0/2029561255 shutdown_connections 2021-05-25T12:03:52.126 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:52.121+0000 7f967a9eb700 1 -- 172.21.15.102:0/2029561255 >> 172.21.15.102:0/2029561255 conn(0x7f967409e9d0 msgr2=0x7f967409c800 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:03:52.126 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:52.121+0000 7f967a9eb700 1 -- 172.21.15.102:0/2029561255 shutdown_connections 2021-05-25T12:03:52.127 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:52.121+0000 7f967a9eb700 1 -- 172.21.15.102:0/2029561255 wait complete. 2021-05-25T12:03:52.127 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:52.121+0000 7f967a9eb700 1 librados: shutdown 2021-05-25T12:03:55.134 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2021-05-25T12:03:55.258 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.256+0000 7fefcb44e700 10 monclient: get_monmap_and_config 2021-05-25T12:03:55.258 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.256+0000 7fefcb44e700 10 monclient: build_initial_monmap 2021-05-25T12:03:55.258 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.256+0000 7fefcb44e700 10 monclient: monmap: 2021-05-25T12:03:55.259 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:03:55.259 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:03:55.259 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:03:55.257301+0000 2021-05-25T12:03:55.259 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:03:55.257301+0000 2021-05-25T12:03:55.259 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:03:55.260 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:03:55.260 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:03:55.260 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:03:55.260 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:03:55.260 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:03:55.261 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.256+0000 7fefcb44e700 1 Processor -- start 2021-05-25T12:03:55.261 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.257+0000 7fefcb44e700 1 -- start start 2021-05-25T12:03:55.261 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.257+0000 7fefcb44e700 10 monclient: init 2021-05-25T12:03:55.270 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.257+0000 7fefcb44e700 10 monclient: _reopen_session rank -1 2021-05-25T12:03:55.270 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.257+0000 7fefcb44e700 10 monclient: _add_conns ranks=[0,1,2] 2021-05-25T12:03:55.271 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.257+0000 7fefcb44e700 10 monclient(hunting): picked mon.noname-a con 0x7fefc4068d50 addr v1:172.21.15.102:6789/0 2021-05-25T12:03:55.271 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.257+0000 7fefcb44e700 10 monclient(hunting): picked mon.noname-c con 0x7fefc4069130 addr v1:172.21.15.181:6789/0 2021-05-25T12:03:55.271 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.257+0000 7fefcb44e700 10 monclient(hunting): picked mon.noname-b con 0x7fefc40695e0 addr v1:172.21.15.102:6790/0 2021-05-25T12:03:55.271 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.257+0000 7fefcb44e700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefc4094380 con 0x7fefc4068d50 2021-05-25T12:03:55.272 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.257+0000 7fefcb44e700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefc4094620 con 0x7fefc4069130 2021-05-25T12:03:55.272 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.257+0000 7fefcb44e700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefc40948c0 con 0x7fefc40695e0 2021-05-25T12:03:55.272 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.257+0000 7fefcb44e700 10 monclient(hunting): _renew_subs 2021-05-25T12:03:55.272 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.257+0000 7fefcb44e700 10 monclient(hunting): authenticate will time out at 1035.854772s 2021-05-25T12:03:55.272 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.257+0000 7fefc91ea700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fefc4068d50 0x7fefc4129c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60562/0 (socket says 172.21.15.102:60562) 2021-05-25T12:03:55.273 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.257+0000 7fefc99eb700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fefc40695e0 0x7fefc4132160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:55444/0 (socket says 172.21.15.102:55444) 2021-05-25T12:03:55.273 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.257+0000 7fefc91ea700 1 -- 172.21.15.102:0/4201616764 learned_addr learned my addr 172.21.15.102:0/4201616764 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:03:55.273 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.258+0000 7fefb3fff700 1 -- 172.21.15.102:0/4201616764 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 85761074 0 0) 0x7fefc4094380 con 0x7fefc4068d50 2021-05-25T12:03:55.273 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.258+0000 7fefb3fff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:03:55.273 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.258+0000 7fefb3fff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:03:55.274 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.258+0000 7fefb3fff700 10 monclient(hunting): my global_id is 174372 2021-05-25T12:03:55.274 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.258+0000 7fefb3fff700 1 -- 172.21.15.102:0/4201616764 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefac003270 con 0x7fefc4068d50 2021-05-25T12:03:55.274 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.258+0000 7fefb3fff700 1 -- 172.21.15.102:0/4201616764 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3276738963 0 0) 0x7fefc40948c0 con 0x7fefc40695e0 2021-05-25T12:03:55.274 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.258+0000 7fefb3fff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:03:55.275 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.258+0000 7fefb3fff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:03:55.275 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.258+0000 7fefb3fff700 10 monclient(hunting): my global_id is 184106 2021-05-25T12:03:55.275 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.258+0000 7fefb3fff700 1 -- 172.21.15.102:0/4201616764 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefc4094380 con 0x7fefc40695e0 2021-05-25T12:03:55.275 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.258+0000 7fefb3fff700 1 -- 172.21.15.102:0/4201616764 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2841412429 0 0) 0x7fefac003270 con 0x7fefc4068d50 2021-05-25T12:03:55.275 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.258+0000 7fefb3fff700 1 -- 172.21.15.102:0/4201616764 >> v1:172.21.15.102:6790/0 conn(0x7fefc40695e0 legacy=0x7fefc4132160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:03:55.276 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.258+0000 7fefb3fff700 1 -- 172.21.15.102:0/4201616764 >> v1:172.21.15.181:6789/0 conn(0x7fefc4069130 legacy=0x7fefc412ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:03:55.276 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.258+0000 7fefb3fff700 10 monclient: _finish_hunting 0 2021-05-25T12:03:55.276 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.258+0000 7fefb3fff700 1 monclient: found mon.noname-a 2021-05-25T12:03:55.276 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.258+0000 7fefb3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:03:55.276 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.258+0000 7fefb3fff700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T12:03:55.277 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.258+0000 7fefb3fff700 1 -- 172.21.15.102:0/4201616764 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefc4094c80 con 0x7fefc4068d50 2021-05-25T12:03:55.277 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.258+0000 7fefb3fff700 10 monclient: _finish_auth 0 2021-05-25T12:03:55.277 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.258+0000 7fefb3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:03:55.277 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.258+0000 7fefb3fff700 1 -- 172.21.15.102:0/4201616764 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fefb4001610 con 0x7fefc4068d50 2021-05-25T12:03:55.278 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.258+0000 7fefcb44e700 5 monclient: authenticate success, global_id 174372 2021-05-25T12:03:55.278 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.258+0000 7fefcb44e700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:03:55.278 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.258+0000 7fefb3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:03:55.278 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.258+0000 7fefb3fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:03:55.278 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.258+0000 7fefb3fff700 10 monclient: dump: 2021-05-25T12:03:55.279 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:03:55.279 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:03:55.279 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:03:55.279 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:03:55.279 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:03:55.280 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:03:55.280 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:03:55.280 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:03:55.280 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:03:55.280 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:03:55.281 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.258+0000 7fefcb44e700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:03:55.281 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.258+0000 7fefb3fff700 1 -- 172.21.15.102:0/4201616764 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fefb4001870 con 0x7fefc4068d50 2021-05-25T12:03:55.281 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.258+0000 7fefb3fff700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:03:55.281 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.258+0000 7fefb3fff700 1 -- 172.21.15.102:0/4201616764 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fefb4001c00 con 0x7fefc4068d50 2021-05-25T12:03:55.282 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.258+0000 7fefcb44e700 10 monclient: get_monmap_and_config success 2021-05-25T12:03:55.282 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.258+0000 7fefcb44e700 4 set_mon_vals no callback set 2021-05-25T12:03:55.282 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.258+0000 7fefb3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:03:55.282 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.258+0000 7fefb3fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:03:55.282 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.258+0000 7fefb3fff700 10 monclient: dump: 2021-05-25T12:03:55.283 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:03:55.283 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:03:55.283 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:03:55.283 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:03:55.283 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:03:55.284 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:03:55.284 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:03:55.284 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:03:55.284 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:03:55.284 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:03:55.285 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.259+0000 7fefcb44e700 10 monclient: shutdown 2021-05-25T12:03:55.285 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.259+0000 7fefcb44e700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:03:55.285 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.259+0000 7fefcb44e700 1 -- 172.21.15.102:0/4201616764 >> v1:172.21.15.102:6789/0 conn(0x7fefc4068d50 legacy=0x7fefc4129c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:03:55.285 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.259+0000 7fefcb44e700 1 -- 172.21.15.102:0/4201616764 shutdown_connections 2021-05-25T12:03:55.285 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.259+0000 7fefcb44e700 1 -- 172.21.15.102:0/4201616764 >> 172.21.15.102:0/4201616764 conn(0x7fefc409b370 msgr2=0x7fefc4097430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:03:55.286 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.259+0000 7fefcb44e700 1 -- 172.21.15.102:0/4201616764 shutdown_connections 2021-05-25T12:03:55.286 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.259+0000 7fefcb44e700 1 -- 172.21.15.102:0/4201616764 wait complete. 2021-05-25T12:03:55.286 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.259+0000 7fefcb44e700 10 monclient: build_initial_monmap 2021-05-25T12:03:55.286 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.259+0000 7fefcb44e700 10 monclient: monmap: 2021-05-25T12:03:55.287 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:03:55.287 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:03:55.287 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:03:55.287 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:03:55.287 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:03:55.288 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:03:55.288 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:03:55.288 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:03:55.288 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:03:55.288 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:03:55.289 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.259+0000 7fefcb44e700 1 librados: starting msgr at 2021-05-25T12:03:55.289 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.259+0000 7fefcb44e700 1 librados: starting objecter 2021-05-25T12:03:55.289 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.260+0000 7fefcb44e700 1 Processor -- start 2021-05-25T12:03:55.289 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.260+0000 7fefcb44e700 1 -- start start 2021-05-25T12:03:55.290 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.260+0000 7fefcb44e700 1 librados: setting wanted keys 2021-05-25T12:03:55.290 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.260+0000 7fefcb44e700 1 librados: calling monclient init 2021-05-25T12:03:55.290 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.260+0000 7fefcb44e700 10 monclient: init 2021-05-25T12:03:55.290 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.260+0000 7fefcb44e700 10 monclient: _reopen_session rank -1 2021-05-25T12:03:55.290 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.260+0000 7fefcb44e700 10 monclient: _add_conns ranks=[1,0,2] 2021-05-25T12:03:55.291 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.260+0000 7fefcb44e700 10 monclient(hunting): picked mon.noname-b con 0x7fefc4068d50 addr v1:172.21.15.181:6789/0 2021-05-25T12:03:55.291 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.260+0000 7fefcb44e700 10 monclient(hunting): picked mon.noname-a con 0x7fefc4069130 addr v1:172.21.15.102:6789/0 2021-05-25T12:03:55.291 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.260+0000 7fefcb44e700 10 monclient(hunting): picked mon.noname-c con 0x7fefc40695e0 addr v1:172.21.15.102:6790/0 2021-05-25T12:03:55.291 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.260+0000 7fefcb44e700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefc4093ac0 con 0x7fefc4069130 2021-05-25T12:03:55.291 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.260+0000 7fefcb44e700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefc40921f0 con 0x7fefc4068d50 2021-05-25T12:03:55.292 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.260+0000 7fefcb44e700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefc40932e0 con 0x7fefc40695e0 2021-05-25T12:03:55.292 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.260+0000 7fefcb44e700 10 monclient(hunting): _renew_subs 2021-05-25T12:03:55.292 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.260+0000 7fefcb44e700 10 monclient(hunting): authenticate will time out at 1035.857734s 2021-05-25T12:03:55.292 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.260+0000 7fefc99eb700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fefc40695e0 0x7fefc41430f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:55448/0 (socket says 172.21.15.102:55448) 2021-05-25T12:03:55.293 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.260+0000 7fefc99eb700 1 -- 172.21.15.102:0/3793259714 learned_addr learned my addr 172.21.15.102:0/3793259714 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:03:55.293 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.260+0000 7fefb17fa700 10 client.?.objecter ms_handle_connect 0x7fefc4069130 2021-05-25T12:03:55.293 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.260+0000 7fefb17fa700 10 client.?.objecter resend_mon_ops 2021-05-25T12:03:55.293 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.260+0000 7fefb17fa700 10 client.?.objecter ms_handle_connect 0x7fefc40695e0 2021-05-25T12:03:55.293 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.260+0000 7fefb17fa700 10 client.?.objecter resend_mon_ops 2021-05-25T12:03:55.294 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.261+0000 7fefb17fa700 1 -- 172.21.15.102:0/3793259714 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1240290968 0 0) 0x7fefc4093ac0 con 0x7fefc4069130 2021-05-25T12:03:55.294 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.261+0000 7fefb17fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:03:55.294 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.261+0000 7fefb17fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:03:55.294 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.261+0000 7fefb17fa700 10 monclient(hunting): my global_id is 174375 2021-05-25T12:03:55.294 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.261+0000 7fefb17fa700 1 -- 172.21.15.102:0/3793259714 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefa4004c10 con 0x7fefc4069130 2021-05-25T12:03:55.295 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.261+0000 7fefb17fa700 1 -- 172.21.15.102:0/3793259714 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2266222222 0 0) 0x7fefc40932e0 con 0x7fefc40695e0 2021-05-25T12:03:55.295 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.261+0000 7fefb17fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:03:55.295 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.261+0000 7fefb17fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:03:55.295 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.261+0000 7fefb17fa700 10 monclient(hunting): my global_id is 184109 2021-05-25T12:03:55.296 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.261+0000 7fefb17fa700 1 -- 172.21.15.102:0/3793259714 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefc4093ac0 con 0x7fefc40695e0 2021-05-25T12:03:55.296 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.261+0000 7fefb17fa700 10 client.?.objecter ms_handle_connect 0x7fefc4068d50 2021-05-25T12:03:55.296 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.261+0000 7fefb17fa700 10 client.?.objecter resend_mon_ops 2021-05-25T12:03:55.296 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.261+0000 7fefb17fa700 1 -- 172.21.15.102:0/3793259714 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1755345108 0 0) 0x7fefa4004c10 con 0x7fefc4069130 2021-05-25T12:03:55.296 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.261+0000 7fefb17fa700 1 -- 172.21.15.102:0/3793259714 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fefc40932e0 con 0x7fefc4069130 2021-05-25T12:03:55.297 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.261+0000 7fefb17fa700 1 -- 172.21.15.102:0/3793259714 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fefb8001880 con 0x7fefc4069130 2021-05-25T12:03:55.297 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.261+0000 7fefb17fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:03:55.297 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.261+0000 7fefb17fa700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:03:55.297 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.261+0000 7fefb17fa700 10 monclient(hunting): dump: 2021-05-25T12:03:55.297 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:03:55.298 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:03:55.298 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:03:55.298 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:03:55.298 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:03:55.298 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:03:55.299 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:03:55.299 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:03:55.299 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:03:55.299 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:03:55.299 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.261+0000 7fefb17fa700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:03:55.300 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.261+0000 7fefb17fa700 1 -- 172.21.15.102:0/3793259714 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2315157374 0 0) 0x7fefc4093ac0 con 0x7fefc40695e0 2021-05-25T12:03:55.300 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.261+0000 7fefb17fa700 1 -- 172.21.15.102:0/3793259714 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fefa4004c10 con 0x7fefc40695e0 2021-05-25T12:03:55.300 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.261+0000 7fefb17fa700 1 -- 172.21.15.102:0/3793259714 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fefc0001c30 con 0x7fefc40695e0 2021-05-25T12:03:55.300 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.261+0000 7fefb17fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:03:55.301 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.261+0000 7fefb17fa700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2021-05-25T12:03:55.301 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.261+0000 7fefb17fa700 10 monclient(hunting): dump: 2021-05-25T12:03:55.301 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:03:55.301 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:03:55.301 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:03:55.302 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:03:55.302 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:03:55.302 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:03:55.302 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:03:55.302 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:03:55.303 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:03:55.303 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:03:55.303 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.261+0000 7fefb17fa700 1 -- 172.21.15.102:0/3793259714 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4277618397 0 0) 0x7fefc40932e0 con 0x7fefc4069130 2021-05-25T12:03:55.303 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.261+0000 7fefb17fa700 1 -- 172.21.15.102:0/3793259714 >> v1:172.21.15.102:6790/0 conn(0x7fefc40695e0 legacy=0x7fefc41430f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:03:55.303 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.261+0000 7fefb17fa700 1 -- 172.21.15.102:0/3793259714 >> v1:172.21.15.181:6789/0 conn(0x7fefc4068d50 legacy=0x7fefc4132820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:03:55.304 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.261+0000 7fefb17fa700 10 monclient: _finish_hunting 0 2021-05-25T12:03:55.304 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.261+0000 7fefb17fa700 1 monclient: found mon.a 2021-05-25T12:03:55.304 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.261+0000 7fefb17fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:03:55.304 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.261+0000 7fefb17fa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:03:55.304 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.261+0000 7fefb17fa700 1 -- 172.21.15.102:0/3793259714 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefc4093580 con 0x7fefc4069130 2021-05-25T12:03:55.305 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.261+0000 7fefb17fa700 10 monclient: _finish_auth 0 2021-05-25T12:03:55.305 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.261+0000 7fefb17fa700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:03:55.305 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.261+0000 7fefcb44e700 5 monclient: authenticate success, global_id 174375 2021-05-25T12:03:55.305 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.261+0000 7fefcb44e700 10 monclient: _renew_subs 2021-05-25T12:03:55.305 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.261+0000 7fefcb44e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:03:55.306 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.261+0000 7fefcb44e700 1 -- 172.21.15.102:0/3793259714 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefc4132f30 con 0x7fefc4069130 2021-05-25T12:03:55.306 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.261+0000 7fefcb44e700 10 client.174375.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:03:55.306 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.261+0000 7fefcb44e700 10 monclient: _renew_subs 2021-05-25T12:03:55.306 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.261+0000 7fefcb44e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:03:55.307 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.261+0000 7fefcb44e700 1 -- 172.21.15.102:0/3793259714 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fefc413b6d0 con 0x7fefc4069130 2021-05-25T12:03:55.307 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.261+0000 7fefcb44e700 1 librados: init done 2021-05-25T12:03:55.307 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.261+0000 7fefb17fa700 1 -- 172.21.15.102:0/3793259714 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fefb8001ab0 con 0x7fefc4069130 2021-05-25T12:03:55.307 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.261+0000 7fefb17fa700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:03:55.307 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.261+0000 7fefb17fa700 1 -- 172.21.15.102:0/3793259714 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fefb8001e40 con 0x7fefc4069130 2021-05-25T12:03:55.308 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.261+0000 7fefb17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:03:55.308 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.261+0000 7fefb17fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:03:55.308 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.261+0000 7fefb37fe700 4 set_mon_vals no callback set 2021-05-25T12:03:55.308 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.261+0000 7fefb17fa700 10 monclient: dump: 2021-05-25T12:03:55.308 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:03:55.309 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:03:55.309 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:03:55.309 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:03:55.309 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:03:55.309 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:03:55.310 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:03:55.310 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:03:55.310 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:03:55.310 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:03:55.310 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.262+0000 7fefb17fa700 1 -- 172.21.15.102:0/3793259714 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7fefb8001fa0 con 0x7fefc4069130 2021-05-25T12:03:55.311 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.262+0000 7fefcb44e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:03:55.311 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.262+0000 7fefcb44e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:03:55.311 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.262+0000 7fefcb44e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:03:55.311 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.262+0000 7fefb17fa700 1 -- 172.21.15.102:0/3793259714 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(115..115 src has 1..115) v4 ==== 5539+0+0 (unknown 2268674648 0 0) 0x7fefb803aec0 con 0x7fefc4069130 2021-05-25T12:03:55.312 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.262+0000 7fefb17fa700 10 client.174375.objecter ms_dispatch 0x7fefc4001550 osd_map(115..115 src has 1..115) v4 2021-05-25T12:03:55.312 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.262+0000 7fefb17fa700 3 client.174375.objecter handle_osd_map got epochs [115,115] > 0 2021-05-25T12:03:55.312 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.262+0000 7fefb17fa700 3 client.174375.objecter handle_osd_map decoding full epoch 115 2021-05-25T12:03:55.312 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.262+0000 7fefcb44e700 1 -- 172.21.15.102:0/3793259714 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef9c004db0 con 0x7fefc4069130 2021-05-25T12:03:55.312 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.262+0000 7fefb17fa700 20 client.174375.objecter dump_active .. 0 homeless 2021-05-25T12:03:55.313 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.262+0000 7fefc91ea700 10 monclient: get_auth_request con 0x7fefa40334e0 auth_method 0 2021-05-25T12:03:55.313 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.263+0000 7fefb17fa700 10 client.174375.objecter ms_handle_connect 0x7fefa40334e0 2021-05-25T12:03:55.313 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.266+0000 7fefb17fa700 1 -- 172.21.15.102:0/3793259714 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7fefb8002700 con 0x7fefc4069130 2021-05-25T12:03:55.313 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.266+0000 7fefb17fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:03:55.313 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.266+0000 7fefb17fa700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:03:55.474 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.472+0000 7fefcb44e700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2021-05-25T12:03:55.474 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.472+0000 7fefcb44e700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2021-05-25T12:03:55.475 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.472+0000 7fefcb44e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:03:55.475 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.472+0000 7fefcb44e700 1 -- 172.21.15.102:0/3793259714 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fef9c005060 con 0x7fefc4069130 2021-05-25T12:03:55.475 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.473+0000 7fefb17fa700 1 -- 172.21.15.102:0/3793259714 <== mon.0 v1:172.21.15.102:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1336 (unknown 254746657 0 869058348) 0x7fefb80103f0 con 0x7fefc4069130 2021-05-25T12:03:55.475 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.473+0000 7fefb17fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2021-05-25T12:03:55.476 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.473+0000 7fefb17fa700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:03:55.476 INFO:teuthology.orchestra.run.smithi102.stdout:{"election_epoch":34,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540138303579357183","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"20ee91b6-8935-4b98-b868-85c03789ccc4","modified":"2021-05-25T11:59:11.011522Z","created":"2021-05-25T11:59:11.011522Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6790","nonce":0}]},"addr":"172.21.15.102:6790/0","public_addr":"172.21.15.102:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2021-05-25T12:03:55.478 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.474+0000 7fefcb44e700 10 librados: watch_flush enter 2021-05-25T12:03:55.478 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.474+0000 7fefcb44e700 10 librados: watch_flush exit 2021-05-25T12:03:55.478 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.474+0000 7fefcb44e700 20 client.174375.objecter shutdown clearing up homeless session... 2021-05-25T12:03:55.478 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.474+0000 7fefcb44e700 10 client.174375.objecter successfully canceled tick 2021-05-25T12:03:55.479 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.474+0000 7fefcb44e700 1 -- 172.21.15.102:0/3793259714 >> v1:172.21.15.181:6812/29582 conn(0x7fefa40334e0 legacy=0x7fefa40359a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:03:55.479 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.474+0000 7fefcb44e700 10 monclient: shutdown 2021-05-25T12:03:55.479 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.474+0000 7fefcb44e700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:03:55.479 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.474+0000 7fefcb44e700 1 -- 172.21.15.102:0/3793259714 >> v1:172.21.15.102:6789/0 conn(0x7fefc4069130 legacy=0x7fefc413ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:03:55.479 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.474+0000 7fefcb44e700 1 -- 172.21.15.102:0/3793259714 shutdown_connections 2021-05-25T12:03:55.480 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.474+0000 7fefcb44e700 1 -- 172.21.15.102:0/3793259714 >> 172.21.15.102:0/3793259714 conn(0x7fefc409b370 msgr2=0x7fefc41254d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:03:55.480 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.474+0000 7fefcb44e700 1 -- 172.21.15.102:0/3793259714 shutdown_connections 2021-05-25T12:03:55.480 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.474+0000 7fefcb44e700 1 -- 172.21.15.102:0/3793259714 wait complete. 2021-05-25T12:03:55.480 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:55.474+0000 7fefcb44e700 1 librados: shutdown 2021-05-25T12:03:58.487 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2021-05-25T12:03:58.604 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.602+0000 7f0470431700 10 monclient: get_monmap_and_config 2021-05-25T12:03:58.604 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.602+0000 7f0470431700 10 monclient: build_initial_monmap 2021-05-25T12:03:58.604 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.602+0000 7f0470431700 10 monclient: monmap: 2021-05-25T12:03:58.605 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:03:58.605 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:03:58.605 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:03:58.603848+0000 2021-05-25T12:03:58.605 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:03:58.603848+0000 2021-05-25T12:03:58.606 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:03:58.606 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:03:58.606 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:03:58.606 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:03:58.607 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:03:58.607 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:03:58.616 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.603+0000 7f0470431700 1 Processor -- start 2021-05-25T12:03:58.616 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.603+0000 7f0470431700 1 -- start start 2021-05-25T12:03:58.616 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.603+0000 7f0470431700 10 monclient: init 2021-05-25T12:03:58.617 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.604+0000 7f0470431700 10 monclient: _reopen_session rank -1 2021-05-25T12:03:58.617 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.604+0000 7f0470431700 10 monclient: _add_conns ranks=[1,2,0] 2021-05-25T12:03:58.617 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.604+0000 7f0470431700 10 monclient(hunting): picked mon.noname-c con 0x7f0468090990 addr v1:172.21.15.181:6789/0 2021-05-25T12:03:58.617 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.604+0000 7f0470431700 10 monclient(hunting): picked mon.noname-b con 0x7f0468090d70 addr v1:172.21.15.102:6790/0 2021-05-25T12:03:58.617 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.604+0000 7f0470431700 10 monclient(hunting): picked mon.noname-a con 0x7f046812e330 addr v1:172.21.15.102:6789/0 2021-05-25T12:03:58.618 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.604+0000 7f0470431700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0468096fa0 con 0x7f046812e330 2021-05-25T12:03:58.618 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.604+0000 7f0470431700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04680926a0 con 0x7f0468090990 2021-05-25T12:03:58.618 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.604+0000 7f0470431700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0468092940 con 0x7f0468090d70 2021-05-25T12:03:58.619 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.604+0000 7f0470431700 10 monclient(hunting): _renew_subs 2021-05-25T12:03:58.619 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.604+0000 7f0470431700 10 monclient(hunting): authenticate will time out at 1039.201220s 2021-05-25T12:03:58.619 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.604+0000 7f046e9ce700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f046812e330 0x7f0468130720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60580/0 (socket says 172.21.15.102:60580) 2021-05-25T12:03:58.619 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.604+0000 7f046d9cc700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f0468090d70 0x7f0468091200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:55456/0 (socket says 172.21.15.102:55456) 2021-05-25T12:03:58.620 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.604+0000 7f046e9ce700 1 -- 172.21.15.102:0/1324442840 learned_addr learned my addr 172.21.15.102:0/1324442840 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:03:58.620 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.604+0000 7f046d1cb700 1 -- 172.21.15.102:0/1324442840 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3935660178 0 0) 0x7f0468092940 con 0x7f0468090d70 2021-05-25T12:03:58.620 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.604+0000 7f046d1cb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:03:58.620 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.604+0000 7f046d1cb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:03:58.620 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.604+0000 7f046d1cb700 10 monclient(hunting): my global_id is 184112 2021-05-25T12:03:58.621 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.604+0000 7f046d1cb700 1 -- 172.21.15.102:0/1324442840 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0450003270 con 0x7f0468090d70 2021-05-25T12:03:58.621 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.604+0000 7f046d1cb700 1 -- 172.21.15.102:0/1324442840 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1512504286 0 0) 0x7f0450003270 con 0x7f0468090d70 2021-05-25T12:03:58.621 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.604+0000 7f046d1cb700 1 -- 172.21.15.102:0/1324442840 >> v1:172.21.15.181:6789/0 conn(0x7f0468090990 legacy=0x7f0468093240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:03:58.622 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.604+0000 7f046d1cb700 1 -- 172.21.15.102:0/1324442840 >> v1:172.21.15.102:6789/0 conn(0x7f046812e330 legacy=0x7f0468130720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:03:58.622 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.604+0000 7f046d1cb700 10 monclient: _finish_hunting 0 2021-05-25T12:03:58.622 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.604+0000 7f046d1cb700 1 monclient: found mon.noname-b 2021-05-25T12:03:58.622 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.604+0000 7f046d1cb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:03:58.623 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.604+0000 7f046d1cb700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T12:03:58.623 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.604+0000 7f046d1cb700 1 -- 172.21.15.102:0/1324442840 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0468092be0 con 0x7f0468090d70 2021-05-25T12:03:58.623 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.604+0000 7f046d1cb700 10 monclient: _finish_auth 0 2021-05-25T12:03:58.623 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.604+0000 7f046d1cb700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:03:58.623 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.604+0000 7f046d1cb700 1 -- 172.21.15.102:0/1324442840 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f0458001540 con 0x7f0468090d70 2021-05-25T12:03:58.624 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.604+0000 7f046d1cb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:03:58.624 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.604+0000 7f046d1cb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:03:58.624 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.604+0000 7f046d1cb700 10 monclient: dump: 2021-05-25T12:03:58.624 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:03:58.625 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:03:58.625 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:03:58.625 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:03:58.625 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:03:58.625 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:03:58.626 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:03:58.626 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:03:58.626 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:03:58.626 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:03:58.626 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.604+0000 7f0470431700 5 monclient: authenticate success, global_id 184112 2021-05-25T12:03:58.627 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.604+0000 7f0470431700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:03:58.627 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.604+0000 7f046d1cb700 1 -- 172.21.15.102:0/1324442840 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f04580017f0 con 0x7f0468090d70 2021-05-25T12:03:58.627 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.604+0000 7f046d1cb700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:03:58.628 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.604+0000 7f0470431700 10 monclient: get_monmap_and_config success 2021-05-25T12:03:58.628 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.604+0000 7f046d1cb700 1 -- 172.21.15.102:0/1324442840 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f0458001b80 con 0x7f0468090d70 2021-05-25T12:03:58.628 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.604+0000 7f046d1cb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:03:58.628 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.604+0000 7f0470431700 4 set_mon_vals no callback set 2021-05-25T12:03:58.628 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.604+0000 7f046d1cb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:03:58.629 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.604+0000 7f046d1cb700 10 monclient: dump: 2021-05-25T12:03:58.629 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:03:58.629 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:03:58.629 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:03:58.629 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:03:58.630 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:03:58.630 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:03:58.630 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:03:58.630 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:03:58.631 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:03:58.631 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:03:58.631 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.605+0000 7f0470431700 10 monclient: shutdown 2021-05-25T12:03:58.631 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.605+0000 7f0470431700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:03:58.632 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.605+0000 7f0470431700 1 -- 172.21.15.102:0/1324442840 >> v1:172.21.15.102:6790/0 conn(0x7f0468090d70 legacy=0x7f0468091200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:03:58.632 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.605+0000 7f0470431700 1 -- 172.21.15.102:0/1324442840 shutdown_connections 2021-05-25T12:03:58.632 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.605+0000 7f0470431700 1 -- 172.21.15.102:0/1324442840 >> 172.21.15.102:0/1324442840 conn(0x7f046809b370 msgr2=0x7f0468097430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:03:58.632 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.605+0000 7f0470431700 1 -- 172.21.15.102:0/1324442840 shutdown_connections 2021-05-25T12:03:58.633 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.605+0000 7f0470431700 1 -- 172.21.15.102:0/1324442840 wait complete. 2021-05-25T12:03:58.633 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.605+0000 7f0470431700 10 monclient: build_initial_monmap 2021-05-25T12:03:58.633 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.605+0000 7f0470431700 10 monclient: monmap: 2021-05-25T12:03:58.633 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:03:58.634 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:03:58.634 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:03:58.634 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:03:58.634 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:03:58.634 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:03:58.635 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:03:58.635 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:03:58.635 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:03:58.635 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:03:58.636 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.606+0000 7f0470431700 1 librados: starting msgr at 2021-05-25T12:03:58.636 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.606+0000 7f0470431700 1 librados: starting objecter 2021-05-25T12:03:58.636 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.606+0000 7f0470431700 1 Processor -- start 2021-05-25T12:03:58.636 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.606+0000 7f0470431700 1 -- start start 2021-05-25T12:03:58.636 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.606+0000 7f0470431700 1 librados: setting wanted keys 2021-05-25T12:03:58.637 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.606+0000 7f0470431700 1 librados: calling monclient init 2021-05-25T12:03:58.637 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.606+0000 7f0470431700 10 monclient: init 2021-05-25T12:03:58.637 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.606+0000 7f0470431700 10 monclient: _reopen_session rank -1 2021-05-25T12:03:58.637 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.606+0000 7f0470431700 10 monclient: _add_conns ranks=[2,1,0] 2021-05-25T12:03:58.637 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.606+0000 7f0470431700 10 monclient(hunting): picked mon.noname-c con 0x7f0468090990 addr v1:172.21.15.102:6790/0 2021-05-25T12:03:58.638 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.606+0000 7f0470431700 10 monclient(hunting): picked mon.noname-b con 0x7f0468090d70 addr v1:172.21.15.181:6789/0 2021-05-25T12:03:58.638 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.606+0000 7f0470431700 10 monclient(hunting): picked mon.noname-a con 0x7f046812e330 addr v1:172.21.15.102:6789/0 2021-05-25T12:03:58.638 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.606+0000 7f0470431700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0468091fe0 con 0x7f046812e330 2021-05-25T12:03:58.638 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.606+0000 7f0470431700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0468068230 con 0x7f0468090d70 2021-05-25T12:03:58.639 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.606+0000 7f0470431700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f046812a390 con 0x7f0468090990 2021-05-25T12:03:58.639 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.606+0000 7f0470431700 10 monclient(hunting): _renew_subs 2021-05-25T12:03:58.639 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.606+0000 7f0470431700 10 monclient(hunting): authenticate will time out at 1039.203816s 2021-05-25T12:03:58.639 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.606+0000 7f046e9ce700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f046812e330 0x7f0468143070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60586/0 (socket says 172.21.15.102:60586) 2021-05-25T12:03:58.639 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.606+0000 7f046e9ce700 1 -- 172.21.15.102:0/2820661598 learned_addr learned my addr 172.21.15.102:0/2820661598 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:03:58.640 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.606+0000 7f04567fc700 10 client.?.objecter ms_handle_connect 0x7f0468090990 2021-05-25T12:03:58.640 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.607+0000 7f04567fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:03:58.640 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.607+0000 7f04567fc700 10 client.?.objecter ms_handle_connect 0x7f046812e330 2021-05-25T12:03:58.640 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.607+0000 7f04567fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:03:58.641 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.607+0000 7f04567fc700 10 client.?.objecter ms_handle_connect 0x7f0468090d70 2021-05-25T12:03:58.641 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.607+0000 7f04567fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:03:58.641 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.607+0000 7f04567fc700 1 -- 172.21.15.102:0/2820661598 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3516326055 0 0) 0x7f046812a390 con 0x7f0468090990 2021-05-25T12:03:58.642 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.607+0000 7f04567fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:03:58.642 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.607+0000 7f04567fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:03:58.642 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.607+0000 7f04567fc700 10 monclient(hunting): my global_id is 184115 2021-05-25T12:03:58.643 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.607+0000 7f04567fc700 1 -- 172.21.15.102:0/2820661598 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0448004c10 con 0x7f0468090990 2021-05-25T12:03:58.643 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.607+0000 7f04567fc700 1 -- 172.21.15.102:0/2820661598 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3553954896 0 0) 0x7f0448004c10 con 0x7f0468090990 2021-05-25T12:03:58.643 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.607+0000 7f04567fc700 1 -- 172.21.15.102:0/2820661598 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f046812a390 con 0x7f0468090990 2021-05-25T12:03:58.643 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.607+0000 7f04567fc700 1 -- 172.21.15.102:0/2820661598 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f045c001880 con 0x7f0468090990 2021-05-25T12:03:58.644 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.607+0000 7f04567fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:03:58.644 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.607+0000 7f04567fc700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:03:58.644 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.607+0000 7f04567fc700 10 monclient(hunting): dump: 2021-05-25T12:03:58.644 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:03:58.645 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:03:58.645 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:03:58.645 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:03:58.645 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:03:58.645 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:03:58.646 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:03:58.646 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:03:58.646 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:03:58.646 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:03:58.647 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.607+0000 7f04567fc700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:03:58.647 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.607+0000 7f04567fc700 1 -- 172.21.15.102:0/2820661598 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 203081675 0 0) 0x7f046812a390 con 0x7f0468090990 2021-05-25T12:03:58.647 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.607+0000 7f04567fc700 1 -- 172.21.15.102:0/2820661598 >> v1:172.21.15.181:6789/0 conn(0x7f0468090d70 legacy=0x7f0468126620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:03:58.648 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.607+0000 7f04567fc700 1 -- 172.21.15.102:0/2820661598 >> v1:172.21.15.102:6789/0 conn(0x7f046812e330 legacy=0x7f0468143070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:03:58.648 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.607+0000 7f04567fc700 10 monclient: _finish_hunting 0 2021-05-25T12:03:58.649 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.607+0000 7f04567fc700 1 monclient: found mon.c 2021-05-25T12:03:58.649 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.607+0000 7f04567fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:03:58.649 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.607+0000 7f04567fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:03:58.650 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.607+0000 7f04567fc700 1 -- 172.21.15.102:0/2820661598 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0468068cd0 con 0x7f0468090990 2021-05-25T12:03:58.650 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.607+0000 7f04567fc700 10 monclient: _finish_auth 0 2021-05-25T12:03:58.651 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.607+0000 7f04567fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:03:58.651 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.607+0000 7f0470431700 5 monclient: authenticate success, global_id 184115 2021-05-25T12:03:58.651 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.607+0000 7f0470431700 10 monclient: _renew_subs 2021-05-25T12:03:58.652 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.607+0000 7f0470431700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:03:58.652 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.607+0000 7f0470431700 1 -- 172.21.15.102:0/2820661598 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f046813b3d0 con 0x7f0468090990 2021-05-25T12:03:58.652 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.607+0000 7f0470431700 10 client.184115.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:03:58.653 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.607+0000 7f0470431700 10 monclient: _renew_subs 2021-05-25T12:03:58.653 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.607+0000 7f0470431700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:03:58.653 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.607+0000 7f0470431700 1 -- 172.21.15.102:0/2820661598 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0468126d30 con 0x7f0468090990 2021-05-25T12:03:58.654 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.607+0000 7f0470431700 1 librados: init done 2021-05-25T12:03:58.654 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.607+0000 7f04567fc700 1 -- 172.21.15.102:0/2820661598 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f045c001bf0 con 0x7f0468090990 2021-05-25T12:03:58.654 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.607+0000 7f04567fc700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:03:58.655 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.607+0000 7f04567fc700 1 -- 172.21.15.102:0/2820661598 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f045c001f80 con 0x7f0468090990 2021-05-25T12:03:58.655 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.607+0000 7f04567fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:03:58.655 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.607+0000 7f046c9ca700 4 set_mon_vals no callback set 2021-05-25T12:03:58.655 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.607+0000 7f04567fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:03:58.655 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.607+0000 7f04567fc700 10 monclient: dump: 2021-05-25T12:03:58.656 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:03:58.656 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:03:58.656 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:03:58.656 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:03:58.656 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:03:58.657 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:03:58.657 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:03:58.657 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:03:58.658 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:03:58.658 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:03:58.658 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.608+0000 7f04567fc700 1 -- 172.21.15.102:0/2820661598 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f045c002160 con 0x7f0468090990 2021-05-25T12:03:58.659 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.608+0000 7f0470431700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:03:58.659 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.608+0000 7f04567fc700 1 -- 172.21.15.102:0/2820661598 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(118..118 src has 1..118) v4 ==== 5518+0+0 (unknown 2737973433 0 0) 0x7f045c03aff0 con 0x7f0468090990 2021-05-25T12:03:58.660 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.608+0000 7f0470431700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:03:58.660 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.608+0000 7f04567fc700 10 client.184115.objecter ms_dispatch 0x7f0468001550 osd_map(118..118 src has 1..118) v4 2021-05-25T12:03:58.660 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.608+0000 7f0470431700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:03:58.660 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.608+0000 7f04567fc700 3 client.184115.objecter handle_osd_map got epochs [118,118] > 0 2021-05-25T12:03:58.661 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.608+0000 7f04567fc700 3 client.184115.objecter handle_osd_map decoding full epoch 118 2021-05-25T12:03:58.661 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.608+0000 7f0470431700 1 -- 172.21.15.102:0/2820661598 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f044c004db0 con 0x7f0468090990 2021-05-25T12:03:58.661 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.608+0000 7f04567fc700 20 client.184115.objecter dump_active .. 0 homeless 2021-05-25T12:03:58.661 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.608+0000 7f046d9cc700 10 monclient: get_auth_request con 0x7f0448033140 auth_method 0 2021-05-25T12:03:58.662 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.609+0000 7f04567fc700 10 client.184115.objecter ms_handle_connect 0x7f0448033140 2021-05-25T12:03:58.662 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.612+0000 7f04567fc700 1 -- 172.21.15.102:0/2820661598 <== mon.2 v1:172.21.15.102:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7f045c010730 con 0x7f0468090990 2021-05-25T12:03:58.662 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.612+0000 7f04567fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:03:58.662 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.612+0000 7f04567fc700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:03:58.823 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.821+0000 7f0470431700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2021-05-25T12:03:58.823 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.821+0000 7f0470431700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2021-05-25T12:03:58.824 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.821+0000 7f0470431700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:03:58.824 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.821+0000 7f0470431700 1 -- 172.21.15.102:0/2820661598 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f044c0055e0 con 0x7f0468090990 2021-05-25T12:03:58.824 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.822+0000 7f04567fc700 1 -- 172.21.15.102:0/2820661598 <== mon.2 v1:172.21.15.102:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1337 (unknown 254746657 0 2425768325) 0x7f045c010500 con 0x7f0468090990 2021-05-25T12:03:58.824 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.822+0000 7f04567fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2021-05-25T12:03:58.825 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.822+0000 7f04567fc700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:03:58.825 INFO:teuthology.orchestra.run.smithi102.stdout:{"election_epoch":34,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540138303579357183","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"20ee91b6-8935-4b98-b868-85c03789ccc4","modified":"2021-05-25T11:59:11.011522Z","created":"2021-05-25T11:59:11.011522Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6790","nonce":0}]},"addr":"172.21.15.102:6790/0","public_addr":"172.21.15.102:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2021-05-25T12:03:58.827 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.823+0000 7f0470431700 10 librados: watch_flush enter 2021-05-25T12:03:58.827 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.823+0000 7f0470431700 10 librados: watch_flush exit 2021-05-25T12:03:58.827 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.823+0000 7f0470431700 20 client.184115.objecter shutdown clearing up homeless session... 2021-05-25T12:03:58.827 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.823+0000 7f0470431700 10 client.184115.objecter successfully canceled tick 2021-05-25T12:03:58.828 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.823+0000 7f0470431700 1 -- 172.21.15.102:0/2820661598 >> v1:172.21.15.181:6812/29582 conn(0x7f0448033140 legacy=0x7f0448035600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:03:58.828 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.823+0000 7f0470431700 10 monclient: shutdown 2021-05-25T12:03:58.828 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.823+0000 7f0470431700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:03:58.828 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.823+0000 7f0470431700 1 -- 172.21.15.102:0/2820661598 >> v1:172.21.15.102:6790/0 conn(0x7f0468090990 legacy=0x7f046813acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:03:58.828 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.823+0000 7f0470431700 1 -- 172.21.15.102:0/2820661598 shutdown_connections 2021-05-25T12:03:58.829 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.823+0000 7f0470431700 1 -- 172.21.15.102:0/2820661598 >> 172.21.15.102:0/2820661598 conn(0x7f046809b370 msgr2=0x7f0468129330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:03:58.829 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.823+0000 7f0470431700 1 -- 172.21.15.102:0/2820661598 shutdown_connections 2021-05-25T12:03:58.829 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.823+0000 7f0470431700 1 -- 172.21.15.102:0/2820661598 wait complete. 2021-05-25T12:03:58.829 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:03:58.824+0000 7f0470431700 1 librados: shutdown 2021-05-25T12:04:01.796 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:04:01.794+0000 7fe1ac373700 -1 mon.b@1(electing) e1 get_health_metrics reporting 3 slow ops, oldest is mon_command({"prefix": "osd crush rule rm", "name":"test-rados-api-smithi181-33001-36"} v 0) 2021-05-25T12:04:01.837 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2021-05-25T12:04:01.958 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.955+0000 7f79dc592700 10 monclient: get_monmap_and_config 2021-05-25T12:04:01.959 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.956+0000 7f79dc592700 10 monclient: build_initial_monmap 2021-05-25T12:04:01.959 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.956+0000 7f79dc592700 10 monclient: monmap: 2021-05-25T12:04:01.959 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:04:01.959 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:01.960 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:04:01.957904+0000 2021-05-25T12:04:01.960 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:04:01.957904+0000 2021-05-25T12:04:01.960 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:04:01.961 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:01.961 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:04:01.961 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:04:01.961 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:04:01.962 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:01.962 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.956+0000 7f79dc592700 1 Processor -- start 2021-05-25T12:04:01.962 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.956+0000 7f79dc592700 1 -- start start 2021-05-25T12:04:01.962 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.956+0000 7f79dc592700 10 monclient: init 2021-05-25T12:04:01.973 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.957+0000 7f79dc592700 10 monclient: _reopen_session rank -1 2021-05-25T12:04:01.973 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.957+0000 7f79dc592700 10 monclient: _add_conns ranks=[1,0,2] 2021-05-25T12:04:01.974 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.957+0000 7f79dc592700 10 monclient(hunting): picked mon.noname-c con 0x7f79d4090da0 addr v1:172.21.15.181:6789/0 2021-05-25T12:04:01.974 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.957+0000 7f79dc592700 10 monclient(hunting): picked mon.noname-a con 0x7f79d40949c0 addr v1:172.21.15.102:6789/0 2021-05-25T12:04:01.974 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.957+0000 7f79dc592700 10 monclient(hunting): picked mon.noname-b con 0x7f79d4094e40 addr v1:172.21.15.102:6790/0 2021-05-25T12:04:01.974 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.957+0000 7f79dc592700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79d409be80 con 0x7f79d40949c0 2021-05-25T12:04:01.975 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.957+0000 7f79dc592700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79d41254b0 con 0x7f79d4090da0 2021-05-25T12:04:01.975 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.957+0000 7f79dc592700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79d4097290 con 0x7f79d4094e40 2021-05-25T12:04:01.975 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.957+0000 7f79dc592700 10 monclient(hunting): _renew_subs 2021-05-25T12:04:01.975 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.957+0000 7f79dc592700 10 monclient(hunting): authenticate will time out at 1042.555258s 2021-05-25T12:04:01.976 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.957+0000 7f79d9b2d700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f79d40949c0 0x7f79d412cfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60592/0 (socket says 172.21.15.102:60592) 2021-05-25T12:04:01.976 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.957+0000 7f79d9b2d700 1 -- 172.21.15.102:0/3369808882 learned_addr learned my addr 172.21.15.102:0/3369808882 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:04:01.976 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.957+0000 7f79d932c700 1 -- 172.21.15.102:0/3369808882 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3911852809 0 0) 0x7f79d4097290 con 0x7f79d4094e40 2021-05-25T12:04:01.976 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.957+0000 7f79d932c700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:04:01.976 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.957+0000 7f79d932c700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:04:01.977 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.957+0000 7f79d932c700 10 monclient(hunting): my global_id is 184118 2021-05-25T12:04:01.977 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.957+0000 7f79d932c700 1 -- 172.21.15.102:0/3369808882 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79c0003270 con 0x7f79d4094e40 2021-05-25T12:04:01.977 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.957+0000 7f79d932c700 1 -- 172.21.15.102:0/3369808882 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1053492967 0 0) 0x7f79c0003270 con 0x7f79d4094e40 2021-05-25T12:04:01.977 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.957+0000 7f79d932c700 1 -- 172.21.15.102:0/3369808882 >> v1:172.21.15.181:6789/0 conn(0x7f79d4090da0 legacy=0x7f79d4124da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:01.977 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.957+0000 7f79d932c700 1 -- 172.21.15.102:0/3369808882 >> v1:172.21.15.102:6789/0 conn(0x7f79d40949c0 legacy=0x7f79d412cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:01.978 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.957+0000 7f79d932c700 10 monclient: _finish_hunting 0 2021-05-25T12:04:01.978 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.957+0000 7f79d932c700 1 monclient: found mon.noname-b 2021-05-25T12:04:01.978 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.957+0000 7f79d932c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:04:01.978 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.957+0000 7f79d932c700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T12:04:01.979 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.957+0000 7f79d932c700 1 -- 172.21.15.102:0/3369808882 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79d4097530 con 0x7f79d4094e40 2021-05-25T12:04:01.979 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.957+0000 7f79d932c700 10 monclient: _finish_auth 0 2021-05-25T12:04:01.979 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.957+0000 7f79d932c700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:04:01.979 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.957+0000 7f79dc592700 5 monclient: authenticate success, global_id 184118 2021-05-25T12:04:01.979 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.957+0000 7f79d932c700 1 -- 172.21.15.102:0/3369808882 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f79d0001990 con 0x7f79d4094e40 2021-05-25T12:04:01.980 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.957+0000 7f79dc592700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:04:01.980 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.957+0000 7f79d932c700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:01.980 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.957+0000 7f79d932c700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:04:01.980 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.957+0000 7f79d932c700 10 monclient: dump: 2021-05-25T12:04:01.981 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:01.981 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:01.981 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:01.981 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:01.982 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:01.982 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:01.982 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:01.982 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:01.983 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:01.983 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:01.983 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.957+0000 7f79dc592700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:04:01.983 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.958+0000 7f79d932c700 1 -- 172.21.15.102:0/3369808882 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f79d0001cb0 con 0x7f79d4094e40 2021-05-25T12:04:01.983 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.958+0000 7f79d932c700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:04:01.984 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.958+0000 7f79d932c700 1 -- 172.21.15.102:0/3369808882 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f79d0002040 con 0x7f79d4094e40 2021-05-25T12:04:01.984 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.958+0000 7f79dc592700 10 monclient: get_monmap_and_config success 2021-05-25T12:04:01.984 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.958+0000 7f79dc592700 4 set_mon_vals no callback set 2021-05-25T12:04:01.984 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.958+0000 7f79d932c700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:01.984 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.958+0000 7f79d932c700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:04:01.985 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.958+0000 7f79d932c700 10 monclient: dump: 2021-05-25T12:04:01.985 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:01.985 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:01.985 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:01.986 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:01.986 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:01.986 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:01.986 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:01.986 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:01.987 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:01.987 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:01.987 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.958+0000 7f79dc592700 10 monclient: shutdown 2021-05-25T12:04:01.988 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.958+0000 7f79dc592700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:04:01.988 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.958+0000 7f79dc592700 1 -- 172.21.15.102:0/3369808882 >> v1:172.21.15.102:6790/0 conn(0x7f79d4094e40 legacy=0x7f79d4130710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:01.988 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.958+0000 7f79dc592700 1 -- 172.21.15.102:0/3369808882 shutdown_connections 2021-05-25T12:04:01.988 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.958+0000 7f79dc592700 1 -- 172.21.15.102:0/3369808882 >> 172.21.15.102:0/3369808882 conn(0x7f79d409af90 msgr2=0x7f79d409b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:04:01.989 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.958+0000 7f79dc592700 1 -- 172.21.15.102:0/3369808882 shutdown_connections 2021-05-25T12:04:01.989 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.958+0000 7f79dc592700 1 -- 172.21.15.102:0/3369808882 wait complete. 2021-05-25T12:04:01.989 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.959+0000 7f79dc592700 10 monclient: build_initial_monmap 2021-05-25T12:04:01.989 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.959+0000 7f79dc592700 10 monclient: monmap: 2021-05-25T12:04:01.990 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:04:01.990 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:04:01.990 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:04:01.990 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:04:01.990 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:04:01.991 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:04:01.991 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:04:01.991 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:04:01.992 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:04:01.992 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:01.992 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.959+0000 7f79dc592700 1 librados: starting msgr at 2021-05-25T12:04:01.992 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.959+0000 7f79dc592700 1 librados: starting objecter 2021-05-25T12:04:01.992 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.959+0000 7f79dc592700 1 Processor -- start 2021-05-25T12:04:01.993 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.959+0000 7f79dc592700 1 -- start start 2021-05-25T12:04:01.993 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.959+0000 7f79dc592700 1 librados: setting wanted keys 2021-05-25T12:04:01.993 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.959+0000 7f79dc592700 1 librados: calling monclient init 2021-05-25T12:04:01.993 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.959+0000 7f79dc592700 10 monclient: init 2021-05-25T12:04:01.994 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.959+0000 7f79dc592700 10 monclient: _reopen_session rank -1 2021-05-25T12:04:01.994 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.959+0000 7f79dc592700 10 monclient: _add_conns ranks=[1,0,2] 2021-05-25T12:04:01.994 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.959+0000 7f79dc592700 10 monclient(hunting): picked mon.noname-b con 0x7f79d4090da0 addr v1:172.21.15.181:6789/0 2021-05-25T12:04:01.994 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.959+0000 7f79dc592700 10 monclient(hunting): picked mon.noname-a con 0x7f79d40949c0 addr v1:172.21.15.102:6789/0 2021-05-25T12:04:01.995 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.959+0000 7f79dc592700 10 monclient(hunting): picked mon.noname-c con 0x7f79d4094e40 addr v1:172.21.15.102:6790/0 2021-05-25T12:04:01.995 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.959+0000 7f79dc592700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79d4116ce0 con 0x7f79d40949c0 2021-05-25T12:04:01.995 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.959+0000 7f79dc592700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79d4116f80 con 0x7f79d4090da0 2021-05-25T12:04:01.995 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.959+0000 7f79dc592700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79d41150b0 con 0x7f79d4094e40 2021-05-25T12:04:01.995 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.959+0000 7f79dc592700 10 monclient(hunting): _renew_subs 2021-05-25T12:04:01.996 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.959+0000 7f79dc592700 10 monclient(hunting): authenticate will time out at 1042.557920s 2021-05-25T12:04:01.996 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.960+0000 7f79d9b2d700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f79d40949c0 0x7f79d4137290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60598/0 (socket says 172.21.15.102:60598) 2021-05-25T12:04:01.996 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.960+0000 7f79d9b2d700 1 -- 172.21.15.102:0/2044470665 learned_addr learned my addr 172.21.15.102:0/2044470665 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:04:01.996 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.960+0000 7f79ca7fc700 10 client.?.objecter ms_handle_connect 0x7f79d40949c0 2021-05-25T12:04:01.996 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.960+0000 7f79ca7fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:04:01.997 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.960+0000 7f79ca7fc700 10 client.?.objecter ms_handle_connect 0x7f79d4094e40 2021-05-25T12:04:01.997 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.960+0000 7f79ca7fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:04:01.997 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.960+0000 7f79ca7fc700 10 client.?.objecter ms_handle_connect 0x7f79d4090da0 2021-05-25T12:04:01.997 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.960+0000 7f79ca7fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:04:01.998 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.960+0000 7f79ca7fc700 1 -- 172.21.15.102:0/2044470665 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3905418668 0 0) 0x7f79d41150b0 con 0x7f79d4094e40 2021-05-25T12:04:01.998 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.960+0000 7f79ca7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:04:01.998 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.960+0000 7f79ca7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:04:01.998 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.960+0000 7f79ca7fc700 10 monclient(hunting): my global_id is 184121 2021-05-25T12:04:01.998 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.960+0000 7f79ca7fc700 1 -- 172.21.15.102:0/2044470665 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79b4004c10 con 0x7f79d4094e40 2021-05-25T12:04:01.999 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.960+0000 7f79ca7fc700 1 -- 172.21.15.102:0/2044470665 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1328355738 0 0) 0x7f79b4004c10 con 0x7f79d4094e40 2021-05-25T12:04:01.999 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.960+0000 7f79ca7fc700 1 -- 172.21.15.102:0/2044470665 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f79d41150b0 con 0x7f79d4094e40 2021-05-25T12:04:01.999 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.960+0000 7f79ca7fc700 1 -- 172.21.15.102:0/2044470665 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f79d0001fc0 con 0x7f79d4094e40 2021-05-25T12:04:01.999 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.960+0000 7f79ca7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:01.999 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.960+0000 7f79ca7fc700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:04:02.000 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.960+0000 7f79ca7fc700 10 monclient(hunting): dump: 2021-05-25T12:04:02.000 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:02.000 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:02.000 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:02.000 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:02.001 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:02.001 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:02.001 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:02.001 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:02.001 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:02.002 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:02.002 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.960+0000 7f79ca7fc700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:04:02.002 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.960+0000 7f79ca7fc700 1 -- 172.21.15.102:0/2044470665 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1271237507 0 0) 0x7f79d41150b0 con 0x7f79d4094e40 2021-05-25T12:04:02.002 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.960+0000 7f79ca7fc700 1 -- 172.21.15.102:0/2044470665 >> v1:172.21.15.181:6789/0 conn(0x7f79d4090da0 legacy=0x7f79d4136b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:02.002 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.960+0000 7f79ca7fc700 1 -- 172.21.15.102:0/2044470665 >> v1:172.21.15.102:6789/0 conn(0x7f79d40949c0 legacy=0x7f79d4137290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:02.003 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.960+0000 7f79ca7fc700 10 monclient: _finish_hunting 0 2021-05-25T12:04:02.003 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.960+0000 7f79ca7fc700 1 monclient: found mon.c 2021-05-25T12:04:02.003 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.960+0000 7f79ca7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:04:02.003 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.960+0000 7f79ca7fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:04:02.004 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.960+0000 7f79ca7fc700 1 -- 172.21.15.102:0/2044470665 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79d41144f0 con 0x7f79d4094e40 2021-05-25T12:04:02.004 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.960+0000 7f79ca7fc700 10 monclient: _finish_auth 0 2021-05-25T12:04:02.004 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.960+0000 7f79ca7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:04:02.004 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.960+0000 7f79dc592700 5 monclient: authenticate success, global_id 184121 2021-05-25T12:04:02.004 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.960+0000 7f79dc592700 10 monclient: _renew_subs 2021-05-25T12:04:02.004 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.960+0000 7f79dc592700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:04:02.005 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.960+0000 7f79dc592700 1 -- 172.21.15.102:0/2044470665 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79d4114740 con 0x7f79d4094e40 2021-05-25T12:04:02.005 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.960+0000 7f79dc592700 10 client.184121.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:04:02.005 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.960+0000 7f79dc592700 10 monclient: _renew_subs 2021-05-25T12:04:02.005 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.960+0000 7f79dc592700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:04:02.005 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.960+0000 7f79dc592700 1 -- 172.21.15.102:0/2044470665 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f79d4117ca0 con 0x7f79d4094e40 2021-05-25T12:04:02.006 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.960+0000 7f79dc592700 1 librados: init done 2021-05-25T12:04:02.006 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.960+0000 7f79ca7fc700 1 -- 172.21.15.102:0/2044470665 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f79d00026e0 con 0x7f79d4094e40 2021-05-25T12:04:02.006 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.960+0000 7f79ca7fc700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:04:02.006 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.960+0000 7f79ca7fc700 1 -- 172.21.15.102:0/2044470665 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f79d00029f0 con 0x7f79d4094e40 2021-05-25T12:04:02.006 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.960+0000 7f79ca7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:02.007 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.960+0000 7f79ca7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:04:02.007 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.960+0000 7f79d8b2b700 4 set_mon_vals no callback set 2021-05-25T12:04:02.007 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.960+0000 7f79ca7fc700 10 monclient: dump: 2021-05-25T12:04:02.007 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:02.007 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:02.008 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:02.008 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:02.008 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:02.008 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:02.008 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:02.009 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:02.009 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:02.009 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:02.009 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.961+0000 7f79ca7fc700 1 -- 172.21.15.102:0/2044470665 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f79d0002c10 con 0x7f79d4094e40 2021-05-25T12:04:02.009 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.961+0000 7f79ca7fc700 1 -- 172.21.15.102:0/2044470665 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(118..118 src has 1..118) v4 ==== 5518+0+0 (unknown 2737973433 0 0) 0x7f79d003b4e0 con 0x7f79d4094e40 2021-05-25T12:04:02.010 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.961+0000 7f79ca7fc700 10 client.184121.objecter ms_dispatch 0x7f79d4001550 osd_map(118..118 src has 1..118) v4 2021-05-25T12:04:02.010 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.961+0000 7f79ca7fc700 3 client.184121.objecter handle_osd_map got epochs [118,118] > 0 2021-05-25T12:04:02.010 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.961+0000 7f79ca7fc700 3 client.184121.objecter handle_osd_map decoding full epoch 118 2021-05-25T12:04:02.010 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.961+0000 7f79ca7fc700 20 client.184121.objecter dump_active .. 0 homeless 2021-05-25T12:04:02.010 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.961+0000 7f79dc592700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:04:02.011 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.961+0000 7f79dc592700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:04:02.011 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.961+0000 7f79dc592700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:04:02.011 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.961+0000 7f79dc592700 1 -- 172.21.15.102:0/2044470665 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79b8004db0 con 0x7f79d4094e40 2021-05-25T12:04:02.011 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.961+0000 7f79da32e700 10 monclient: get_auth_request con 0x7f79b4033090 auth_method 0 2021-05-25T12:04:02.011 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.962+0000 7f79ca7fc700 10 client.184121.objecter ms_handle_connect 0x7f79b4033090 2021-05-25T12:04:02.012 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.965+0000 7f79ca7fc700 1 -- 172.21.15.102:0/2044470665 <== mon.2 v1:172.21.15.102:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7f79d0010da0 con 0x7f79d4094e40 2021-05-25T12:04:02.012 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.965+0000 7f79ca7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:04:02.012 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:01.965+0000 7f79ca7fc700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:04:02.175 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:02.172+0000 7f79dc592700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2021-05-25T12:04:02.175 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:02.172+0000 7f79dc592700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2021-05-25T12:04:02.175 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:02.172+0000 7f79dc592700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:04:02.176 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:02.172+0000 7f79dc592700 1 -- 172.21.15.102:0/2044470665 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f79b80055e0 con 0x7f79d4094e40 2021-05-25T12:04:02.176 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:02.172+0000 7f79ca7fc700 1 -- 172.21.15.102:0/2044470665 <== mon.2 v1:172.21.15.102:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1337 (unknown 254746657 0 4233707700) 0x7f79d0010990 con 0x7f79d4094e40 2021-05-25T12:04:02.176 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:02.172+0000 7f79ca7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2021-05-25T12:04:02.176 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:02.172+0000 7f79ca7fc700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:04:02.177 INFO:teuthology.orchestra.run.smithi102.stdout:{"election_epoch":34,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540138303579357183","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"20ee91b6-8935-4b98-b868-85c03789ccc4","modified":"2021-05-25T11:59:11.011522Z","created":"2021-05-25T11:59:11.011522Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6790","nonce":0}]},"addr":"172.21.15.102:6790/0","public_addr":"172.21.15.102:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2021-05-25T12:04:02.178 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:02.173+0000 7f79dc592700 10 librados: watch_flush enter 2021-05-25T12:04:02.178 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:02.173+0000 7f79dc592700 10 librados: watch_flush exit 2021-05-25T12:04:02.179 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:02.173+0000 7f79dc592700 20 client.184121.objecter shutdown clearing up homeless session... 2021-05-25T12:04:02.179 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:02.173+0000 7f79dc592700 10 client.184121.objecter successfully canceled tick 2021-05-25T12:04:02.179 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:02.174+0000 7f79dc592700 1 -- 172.21.15.102:0/2044470665 >> v1:172.21.15.181:6812/29582 conn(0x7f79b4033090 legacy=0x7f79b4035550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:02.179 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:02.174+0000 7f79dc592700 10 monclient: shutdown 2021-05-25T12:04:02.180 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:02.174+0000 7f79dc592700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:04:02.180 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:02.174+0000 7f79dc592700 1 -- 172.21.15.102:0/2044470665 >> v1:172.21.15.102:6790/0 conn(0x7f79d4094e40 legacy=0x7f79d413ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:02.180 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:02.174+0000 7f79dc592700 1 -- 172.21.15.102:0/2044470665 shutdown_connections 2021-05-25T12:04:02.180 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:02.174+0000 7f79dc592700 1 -- 172.21.15.102:0/2044470665 >> 172.21.15.102:0/2044470665 conn(0x7f79d409af90 msgr2=0x7f79d4099dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:04:02.181 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:02.174+0000 7f79dc592700 1 -- 172.21.15.102:0/2044470665 shutdown_connections 2021-05-25T12:04:02.181 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:02.174+0000 7f79dc592700 1 -- 172.21.15.102:0/2044470665 wait complete. 2021-05-25T12:04:02.181 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:02.174+0000 7f79dc592700 1 librados: shutdown 2021-05-25T12:04:02.877 INFO:tasks.workunit.client.0.smithi181.stdout:PP (105103 ms total) 2021-05-25T12:04:02.878 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: 2021-05-25T12:04:02.878 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP 2021-05-25T12:04:02.878 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: [ RUN ] LibRadosTwoPoolsECPP.CopyFrom 2021-05-25T12:04:02.879 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: [ OK ] LibRadosTwoPoolsECPP.CopyFrom (209 ms) 2021-05-25T12:04:02.879 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP (209 ms total) 2021-05-25T12:04:02.879 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: 2021-05-25T12:04:02.879 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)0, Checksummer::xxhash32, ceph_le > 2021-05-25T12:04:02.879 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Subset 2021-05-25T12:04:02.880 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Subset (41 ms) 2021-05-25T12:04:02.880 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Chunked 2021-05-25T12:04:02.880 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Chunked (7 ms) 2021-05-25T12:04:02.880 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0 (48 ms total) 2021-05-25T12:04:02.880 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: 2021-05-25T12:04:02.881 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)1, Checksummer::xxhash64, ceph_le > 2021-05-25T12:04:02.881 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Subset 2021-05-25T12:04:02.881 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Subset (29 ms) 2021-05-25T12:04:02.881 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Chunked 2021-05-25T12:04:02.882 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Chunked (8 ms) 2021-05-25T12:04:02.882 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1 (37 ms total) 2021-05-25T12:04:02.882 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: 2021-05-25T12:04:02.883 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)2, Checksummer::crc32c, ceph_le > 2021-05-25T12:04:02.883 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Subset 2021-05-25T12:04:02.883 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Subset (43 ms) 2021-05-25T12:04:02.883 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Chunked 2021-05-25T12:04:02.883 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Chunked (4 ms) 2021-05-25T12:04:02.884 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2 (47 ms total) 2021-05-25T12:04:02.884 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: 2021-05-25T12:04:02.884 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP 2021-05-25T12:04:02.884 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: [ RUN ] LibRadosMiscECPP.CompareExtentRange 2021-05-25T12:04:02.884 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: [ OK ] LibRadosMiscECPP.CompareExtentRange (1032 ms) 2021-05-25T12:04:02.885 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP (1032 ms total) 2021-05-25T12:04:02.885 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: 2021-05-25T12:04:02.885 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: [----------] Global test environment tear-down 2021-05-25T12:04:02.885 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: [==========] 31 tests from 7 test suites ran. (203110 ms total) 2021-05-25T12:04:02.885 INFO:tasks.workunit.client.0.smithi181.stdout: api_misc_pp: [ PASSED ] 31 tests. 2021-05-25T12:04:05.188 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2021-05-25T12:04:05.308 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.306+0000 7fe03b39d700 10 monclient: get_monmap_and_config 2021-05-25T12:04:05.309 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.306+0000 7fe03b39d700 10 monclient: build_initial_monmap 2021-05-25T12:04:05.309 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.306+0000 7fe03b39d700 10 monclient: monmap: 2021-05-25T12:04:05.309 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:04:05.310 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:05.310 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:04:05.308062+0000 2021-05-25T12:04:05.310 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:04:05.308062+0000 2021-05-25T12:04:05.310 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:04:05.311 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:05.311 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:04:05.311 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:04:05.311 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:04:05.311 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:05.312 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.306+0000 7fe03b39d700 1 Processor -- start 2021-05-25T12:04:05.312 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.306+0000 7fe03b39d700 1 -- start start 2021-05-25T12:04:05.312 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.306+0000 7fe03b39d700 10 monclient: init 2021-05-25T12:04:05.322 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.307+0000 7fe03b39d700 10 monclient: _reopen_session rank -1 2021-05-25T12:04:05.322 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.307+0000 7fe03b39d700 10 monclient: _add_conns ranks=[0,2,1] 2021-05-25T12:04:05.323 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.307+0000 7fe03b39d700 10 monclient(hunting): picked mon.noname-a con 0x7fe034090990 addr v1:172.21.15.102:6789/0 2021-05-25T12:04:05.323 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.307+0000 7fe03b39d700 10 monclient(hunting): picked mon.noname-b con 0x7fe034090d70 addr v1:172.21.15.102:6790/0 2021-05-25T12:04:05.323 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.307+0000 7fe03b39d700 10 monclient(hunting): picked mon.noname-c con 0x7fe03412e330 addr v1:172.21.15.181:6789/0 2021-05-25T12:04:05.323 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.307+0000 7fe03b39d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe034096fa0 con 0x7fe034090990 2021-05-25T12:04:05.324 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.307+0000 7fe03b39d700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0340926a0 con 0x7fe03412e330 2021-05-25T12:04:05.324 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.307+0000 7fe03b39d700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe034092940 con 0x7fe034090d70 2021-05-25T12:04:05.324 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.307+0000 7fe03b39d700 10 monclient(hunting): _renew_subs 2021-05-25T12:04:05.324 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.307+0000 7fe03b39d700 10 monclient(hunting): authenticate will time out at 1045.905350s 2021-05-25T12:04:05.325 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.307+0000 7fe039139700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fe034090990 0x7fe034093240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60608/0 (socket says 172.21.15.102:60608) 2021-05-25T12:04:05.325 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.307+0000 7fe038938700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fe034090d70 0x7fe034091200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:55484/0 (socket says 172.21.15.102:55484) 2021-05-25T12:04:05.325 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.307+0000 7fe039139700 1 -- 172.21.15.102:0/520783374 learned_addr learned my addr 172.21.15.102:0/520783374 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:04:05.325 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.307+0000 7fe023fff700 1 -- 172.21.15.102:0/520783374 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4112562284 0 0) 0x7fe034096fa0 con 0x7fe034090990 2021-05-25T12:04:05.325 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.307+0000 7fe023fff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:04:05.326 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.307+0000 7fe023fff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:04:05.326 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.307+0000 7fe023fff700 10 monclient(hunting): my global_id is 174390 2021-05-25T12:04:05.326 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.307+0000 7fe023fff700 1 -- 172.21.15.102:0/520783374 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe01c003270 con 0x7fe034090990 2021-05-25T12:04:05.326 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.307+0000 7fe023fff700 1 -- 172.21.15.102:0/520783374 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 283662251 0 0) 0x7fe0340926a0 con 0x7fe03412e330 2021-05-25T12:04:05.327 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.307+0000 7fe023fff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:04:05.327 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.307+0000 7fe023fff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:04:05.327 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.307+0000 7fe023fff700 10 monclient(hunting): my global_id is 95311 2021-05-25T12:04:05.327 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.307+0000 7fe023fff700 1 -- 172.21.15.102:0/520783374 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe034096fa0 con 0x7fe03412e330 2021-05-25T12:04:05.327 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.308+0000 7fe023fff700 1 -- 172.21.15.102:0/520783374 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1960963231 0 0) 0x7fe01c003270 con 0x7fe034090990 2021-05-25T12:04:05.328 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.308+0000 7fe023fff700 1 -- 172.21.15.102:0/520783374 >> v1:172.21.15.102:6790/0 conn(0x7fe034090d70 legacy=0x7fe034091200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:05.328 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.308+0000 7fe023fff700 1 -- 172.21.15.102:0/520783374 >> v1:172.21.15.181:6789/0 conn(0x7fe03412e330 legacy=0x7fe034130720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:05.328 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.308+0000 7fe023fff700 10 monclient: _finish_hunting 0 2021-05-25T12:04:05.328 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.308+0000 7fe023fff700 1 monclient: found mon.noname-a 2021-05-25T12:04:05.328 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.308+0000 7fe023fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:04:05.329 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.308+0000 7fe023fff700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T12:04:05.329 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.308+0000 7fe023fff700 1 -- 172.21.15.102:0/520783374 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe034092be0 con 0x7fe034090990 2021-05-25T12:04:05.329 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.308+0000 7fe023fff700 10 monclient: _finish_auth 0 2021-05-25T12:04:05.329 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.308+0000 7fe023fff700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:04:05.329 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.308+0000 7fe023fff700 1 -- 172.21.15.102:0/520783374 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fe024001610 con 0x7fe034090990 2021-05-25T12:04:05.330 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.308+0000 7fe023fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:05.330 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.308+0000 7fe023fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:04:05.330 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.308+0000 7fe023fff700 10 monclient: dump: 2021-05-25T12:04:05.330 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:05.331 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:05.331 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:05.331 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:05.331 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:05.331 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:05.332 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:05.332 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:05.332 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:05.332 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:05.332 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.308+0000 7fe03b39d700 5 monclient: authenticate success, global_id 174390 2021-05-25T12:04:05.333 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.308+0000 7fe03b39d700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:04:05.333 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.308+0000 7fe023fff700 1 -- 172.21.15.102:0/520783374 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe024001880 con 0x7fe034090990 2021-05-25T12:04:05.333 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.308+0000 7fe023fff700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:04:05.333 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.308+0000 7fe023fff700 1 -- 172.21.15.102:0/520783374 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fe024001c10 con 0x7fe034090990 2021-05-25T12:04:05.333 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.308+0000 7fe03b39d700 10 monclient: get_monmap_and_config success 2021-05-25T12:04:05.334 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.308+0000 7fe03b39d700 4 set_mon_vals no callback set 2021-05-25T12:04:05.334 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.308+0000 7fe023fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:05.334 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.308+0000 7fe023fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:04:05.334 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.308+0000 7fe023fff700 10 monclient: dump: 2021-05-25T12:04:05.335 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:05.335 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:05.335 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:05.335 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:05.335 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:05.336 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:05.336 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:05.336 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:05.336 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:05.336 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:05.337 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.308+0000 7fe03b39d700 10 monclient: shutdown 2021-05-25T12:04:05.337 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.308+0000 7fe03b39d700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:04:05.337 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.308+0000 7fe03b39d700 1 -- 172.21.15.102:0/520783374 >> v1:172.21.15.102:6789/0 conn(0x7fe034090990 legacy=0x7fe034093240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:05.337 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.308+0000 7fe03b39d700 1 -- 172.21.15.102:0/520783374 shutdown_connections 2021-05-25T12:04:05.337 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.308+0000 7fe03b39d700 1 -- 172.21.15.102:0/520783374 >> 172.21.15.102:0/520783374 conn(0x7fe03409b370 msgr2=0x7fe034097430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:04:05.338 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.308+0000 7fe03b39d700 1 -- 172.21.15.102:0/520783374 shutdown_connections 2021-05-25T12:04:05.338 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.308+0000 7fe03b39d700 1 -- 172.21.15.102:0/520783374 wait complete. 2021-05-25T12:04:05.338 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.309+0000 7fe03b39d700 10 monclient: build_initial_monmap 2021-05-25T12:04:05.338 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.309+0000 7fe03b39d700 10 monclient: monmap: 2021-05-25T12:04:05.338 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:04:05.339 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:04:05.339 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:04:05.339 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:04:05.339 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:04:05.339 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:04:05.340 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:04:05.340 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:04:05.340 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:04:05.340 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:05.340 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.309+0000 7fe03b39d700 1 librados: starting msgr at 2021-05-25T12:04:05.341 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.309+0000 7fe03b39d700 1 librados: starting objecter 2021-05-25T12:04:05.341 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.309+0000 7fe03b39d700 1 Processor -- start 2021-05-25T12:04:05.341 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.309+0000 7fe03b39d700 1 -- start start 2021-05-25T12:04:05.341 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.309+0000 7fe03b39d700 1 librados: setting wanted keys 2021-05-25T12:04:05.342 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.309+0000 7fe03b39d700 1 librados: calling monclient init 2021-05-25T12:04:05.342 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.309+0000 7fe03b39d700 10 monclient: init 2021-05-25T12:04:05.342 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.310+0000 7fe03b39d700 10 monclient: _reopen_session rank -1 2021-05-25T12:04:05.342 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.310+0000 7fe03b39d700 10 monclient: _add_conns ranks=[2,0,1] 2021-05-25T12:04:05.343 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.310+0000 7fe03b39d700 10 monclient(hunting): picked mon.noname-c con 0x7fe034090990 addr v1:172.21.15.102:6790/0 2021-05-25T12:04:05.343 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.310+0000 7fe03b39d700 10 monclient(hunting): picked mon.noname-a con 0x7fe034090d70 addr v1:172.21.15.102:6789/0 2021-05-25T12:04:05.343 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.310+0000 7fe03b39d700 10 monclient(hunting): picked mon.noname-b con 0x7fe03412e330 addr v1:172.21.15.181:6789/0 2021-05-25T12:04:05.343 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.310+0000 7fe03b39d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe034113e10 con 0x7fe034090d70 2021-05-25T12:04:05.344 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.310+0000 7fe03b39d700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe034139a10 con 0x7fe03412e330 2021-05-25T12:04:05.344 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.310+0000 7fe03b39d700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe034115c00 con 0x7fe034090990 2021-05-25T12:04:05.344 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.310+0000 7fe03b39d700 10 monclient(hunting): _renew_subs 2021-05-25T12:04:05.344 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.310+0000 7fe03b39d700 10 monclient(hunting): authenticate will time out at 1045.908139s 2021-05-25T12:04:05.344 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.310+0000 7fe038938700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fe034090d70 0x7fe034139300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60614/0 (socket says 172.21.15.102:60614) 2021-05-25T12:04:05.345 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.310+0000 7fe039139700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fe034090990 0x7fe034138bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:55490/0 (socket says 172.21.15.102:55490) 2021-05-25T12:04:05.345 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.310+0000 7fe038938700 1 -- 172.21.15.102:0/2423427020 learned_addr learned my addr 172.21.15.102:0/2423427020 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:04:05.345 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.310+0000 7fe0217fa700 10 client.?.objecter ms_handle_connect 0x7fe034090990 2021-05-25T12:04:05.345 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.310+0000 7fe0217fa700 10 client.?.objecter resend_mon_ops 2021-05-25T12:04:05.345 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.310+0000 7fe0217fa700 10 client.?.objecter ms_handle_connect 0x7fe034090d70 2021-05-25T12:04:05.346 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.310+0000 7fe0217fa700 10 client.?.objecter resend_mon_ops 2021-05-25T12:04:05.346 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.310+0000 7fe0217fa700 10 client.?.objecter ms_handle_connect 0x7fe03412e330 2021-05-25T12:04:05.346 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.310+0000 7fe0217fa700 10 client.?.objecter resend_mon_ops 2021-05-25T12:04:05.346 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.310+0000 7fe0217fa700 1 -- 172.21.15.102:0/2423427020 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3906275558 0 0) 0x7fe034113e10 con 0x7fe034090d70 2021-05-25T12:04:05.346 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.310+0000 7fe0217fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:04:05.347 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.310+0000 7fe0217fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:04:05.347 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.310+0000 7fe0217fa700 10 monclient(hunting): my global_id is 174393 2021-05-25T12:04:05.347 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.310+0000 7fe0217fa700 1 -- 172.21.15.102:0/2423427020 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe014004c10 con 0x7fe034090d70 2021-05-25T12:04:05.347 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.310+0000 7fe0217fa700 1 -- 172.21.15.102:0/2423427020 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 943605066 0 0) 0x7fe034139a10 con 0x7fe03412e330 2021-05-25T12:04:05.348 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.310+0000 7fe0217fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:04:05.348 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.310+0000 7fe0217fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:04:05.348 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.310+0000 7fe0217fa700 10 monclient(hunting): my global_id is 95314 2021-05-25T12:04:05.348 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.310+0000 7fe0217fa700 1 -- 172.21.15.102:0/2423427020 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe034113e10 con 0x7fe03412e330 2021-05-25T12:04:05.348 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.310+0000 7fe0217fa700 1 -- 172.21.15.102:0/2423427020 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 559160556 0 0) 0x7fe014004c10 con 0x7fe034090d70 2021-05-25T12:04:05.349 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.310+0000 7fe0217fa700 1 -- 172.21.15.102:0/2423427020 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe034139a10 con 0x7fe034090d70 2021-05-25T12:04:05.349 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.310+0000 7fe0217fa700 1 -- 172.21.15.102:0/2423427020 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fe028001870 con 0x7fe034090d70 2021-05-25T12:04:05.349 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.310+0000 7fe0217fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:05.349 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.310+0000 7fe0217fa700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:04:05.350 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.310+0000 7fe0217fa700 10 monclient(hunting): dump: 2021-05-25T12:04:05.350 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:05.350 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:05.350 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:05.351 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:05.351 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:05.351 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:05.351 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:05.352 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:05.352 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:05.352 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:05.352 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.310+0000 7fe0217fa700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:04:05.352 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.310+0000 7fe0217fa700 1 -- 172.21.15.102:0/2423427020 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2657409940 0 0) 0x7fe034139a10 con 0x7fe034090d70 2021-05-25T12:04:05.353 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.310+0000 7fe0217fa700 1 -- 172.21.15.102:0/2423427020 >> v1:172.21.15.102:6790/0 conn(0x7fe034090990 legacy=0x7fe034138bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:05.353 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.310+0000 7fe0217fa700 1 -- 172.21.15.102:0/2423427020 >> v1:172.21.15.181:6789/0 conn(0x7fe03412e330 legacy=0x7fe034140f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:05.353 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.310+0000 7fe0217fa700 10 monclient: _finish_hunting 0 2021-05-25T12:04:05.353 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.310+0000 7fe0217fa700 1 monclient: found mon.a 2021-05-25T12:04:05.354 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.310+0000 7fe0217fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:04:05.354 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.310+0000 7fe0217fa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:04:05.354 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.310+0000 7fe0217fa700 1 -- 172.21.15.102:0/2423427020 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe034115ea0 con 0x7fe034090d70 2021-05-25T12:04:05.354 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.310+0000 7fe0217fa700 10 monclient: _finish_auth 0 2021-05-25T12:04:05.355 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.310+0000 7fe0217fa700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:04:05.355 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.310+0000 7fe03b39d700 5 monclient: authenticate success, global_id 174393 2021-05-25T12:04:05.355 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.310+0000 7fe03b39d700 10 monclient: _renew_subs 2021-05-25T12:04:05.355 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.310+0000 7fe03b39d700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:04:05.355 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.310+0000 7fe03b39d700 1 -- 172.21.15.102:0/2423427020 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0341149d0 con 0x7fe034090d70 2021-05-25T12:04:05.356 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.310+0000 7fe03b39d700 10 client.174393.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:04:05.356 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.310+0000 7fe03b39d700 10 monclient: _renew_subs 2021-05-25T12:04:05.356 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.310+0000 7fe03b39d700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:04:05.356 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.310+0000 7fe03b39d700 1 -- 172.21.15.102:0/2423427020 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe034129980 con 0x7fe034090d70 2021-05-25T12:04:05.356 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.310+0000 7fe03b39d700 1 librados: init done 2021-05-25T12:04:05.357 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.311+0000 7fe0217fa700 1 -- 172.21.15.102:0/2423427020 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe0280045f0 con 0x7fe034090d70 2021-05-25T12:04:05.357 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.311+0000 7fe0217fa700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:04:05.357 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.311+0000 7fe0217fa700 1 -- 172.21.15.102:0/2423427020 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fe028004980 con 0x7fe034090d70 2021-05-25T12:04:05.357 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.311+0000 7fe0217fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:05.357 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.311+0000 7fe0217fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:04:05.358 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.311+0000 7fe0237fe700 4 set_mon_vals no callback set 2021-05-25T12:04:05.358 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.311+0000 7fe0217fa700 10 monclient: dump: 2021-05-25T12:04:05.358 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:05.358 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:05.359 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:05.359 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:05.359 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:05.359 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:05.359 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:05.360 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:05.360 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:05.360 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:05.360 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.311+0000 7fe0217fa700 1 -- 172.21.15.102:0/2423427020 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7fe028004b60 con 0x7fe034090d70 2021-05-25T12:04:05.360 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.311+0000 7fe03b39d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:04:05.361 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.311+0000 7fe03b39d700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:04:05.361 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.311+0000 7fe03b39d700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:04:05.361 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.311+0000 7fe03b39d700 1 -- 172.21.15.102:0/2423427020 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe00c004db0 con 0x7fe034090d70 2021-05-25T12:04:05.361 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.311+0000 7fe0217fa700 1 -- 172.21.15.102:0/2423427020 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(121..121 src has 1..121) v4 ==== 5720+0+0 (unknown 2398408772 0 0) 0x7fe02803d6c0 con 0x7fe034090d70 2021-05-25T12:04:05.361 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.311+0000 7fe0217fa700 10 client.174393.objecter ms_dispatch 0x7fe034001550 osd_map(121..121 src has 1..121) v4 2021-05-25T12:04:05.362 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.311+0000 7fe0217fa700 3 client.174393.objecter handle_osd_map got epochs [121,121] > 0 2021-05-25T12:04:05.362 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.311+0000 7fe0217fa700 3 client.174393.objecter handle_osd_map decoding full epoch 121 2021-05-25T12:04:05.362 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.311+0000 7fe0217fa700 20 client.174393.objecter dump_active .. 0 homeless 2021-05-25T12:04:05.362 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.312+0000 7fe039139700 10 monclient: get_auth_request con 0x7fe014032f20 auth_method 0 2021-05-25T12:04:05.363 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.312+0000 7fe0217fa700 10 client.174393.objecter ms_handle_connect 0x7fe014032f20 2021-05-25T12:04:05.363 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.317+0000 7fe0217fa700 1 -- 172.21.15.102:0/2423427020 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7fe028012d60 con 0x7fe034090d70 2021-05-25T12:04:05.363 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.317+0000 7fe0217fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:04:05.363 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.317+0000 7fe0217fa700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:04:05.525 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.522+0000 7fe03b39d700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2021-05-25T12:04:05.526 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.522+0000 7fe03b39d700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2021-05-25T12:04:05.526 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.523+0000 7fe03b39d700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:04:05.527 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.523+0000 7fe03b39d700 1 -- 172.21.15.102:0/2423427020 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fe00c005060 con 0x7fe034090d70 2021-05-25T12:04:05.527 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.523+0000 7fe0217fa700 1 -- 172.21.15.102:0/2423427020 <== mon.0 v1:172.21.15.102:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1336 (unknown 254746657 0 3667312208) 0x7fe028012b30 con 0x7fe034090d70 2021-05-25T12:04:05.528 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.523+0000 7fe0217fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2021-05-25T12:04:05.528 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.523+0000 7fe0217fa700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:04:05.528 INFO:teuthology.orchestra.run.smithi102.stdout:{"election_epoch":36,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"b","quorum_age":2,"features":{"quorum_con":"4540138303579357183","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"20ee91b6-8935-4b98-b868-85c03789ccc4","modified":"2021-05-25T11:59:11.011522Z","created":"2021-05-25T11:59:11.011522Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6790","nonce":0}]},"addr":"172.21.15.102:6790/0","public_addr":"172.21.15.102:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2021-05-25T12:04:05.531 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.524+0000 7fe03b39d700 10 librados: watch_flush enter 2021-05-25T12:04:05.531 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.524+0000 7fe03b39d700 10 librados: watch_flush exit 2021-05-25T12:04:05.531 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.524+0000 7fe03b39d700 20 client.174393.objecter shutdown clearing up homeless session... 2021-05-25T12:04:05.531 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.524+0000 7fe03b39d700 10 client.174393.objecter successfully canceled tick 2021-05-25T12:04:05.531 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.524+0000 7fe03b39d700 1 -- 172.21.15.102:0/2423427020 >> v1:172.21.15.181:6812/29582 conn(0x7fe014032f20 legacy=0x7fe0140353e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:05.532 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.524+0000 7fe03b39d700 10 monclient: shutdown 2021-05-25T12:04:05.532 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.524+0000 7fe03b39d700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:04:05.532 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.524+0000 7fe03b39d700 1 -- 172.21.15.102:0/2423427020 >> v1:172.21.15.102:6789/0 conn(0x7fe034090d70 legacy=0x7fe034139300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:05.533 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.524+0000 7fe03b39d700 1 -- 172.21.15.102:0/2423427020 shutdown_connections 2021-05-25T12:04:05.533 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.524+0000 7fe03b39d700 1 -- 172.21.15.102:0/2423427020 >> 172.21.15.102:0/2423427020 conn(0x7fe03409b370 msgr2=0x7fe034068490 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:04:05.533 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.525+0000 7fe03b39d700 1 -- 172.21.15.102:0/2423427020 shutdown_connections 2021-05-25T12:04:05.534 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.525+0000 7fe03b39d700 1 -- 172.21.15.102:0/2423427020 wait complete. 2021-05-25T12:04:05.534 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:05.525+0000 7fe03b39d700 1 librados: shutdown 2021-05-25T12:04:08.539 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2021-05-25T12:04:08.657 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.656+0000 7f666056e700 10 monclient: get_monmap_and_config 2021-05-25T12:04:08.658 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.656+0000 7f666056e700 10 monclient: build_initial_monmap 2021-05-25T12:04:08.658 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.656+0000 7f666056e700 10 monclient: monmap: 2021-05-25T12:04:08.658 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:04:08.659 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:08.659 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:04:08.657196+0000 2021-05-25T12:04:08.659 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:04:08.657196+0000 2021-05-25T12:04:08.659 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:04:08.659 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:08.660 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:04:08.660 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:04:08.660 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:04:08.660 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:08.661 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.656+0000 7f666056e700 1 Processor -- start 2021-05-25T12:04:08.661 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.656+0000 7f666056e700 1 -- start start 2021-05-25T12:04:08.661 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.656+0000 7f666056e700 10 monclient: init 2021-05-25T12:04:08.671 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.657+0000 7f666056e700 10 monclient: _reopen_session rank -1 2021-05-25T12:04:08.671 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.657+0000 7f666056e700 10 monclient: _add_conns ranks=[1,2,0] 2021-05-25T12:04:08.671 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.657+0000 7f666056e700 10 monclient(hunting): picked mon.noname-c con 0x7f6658063840 addr v1:172.21.15.181:6789/0 2021-05-25T12:04:08.672 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.657+0000 7f666056e700 10 monclient(hunting): picked mon.noname-b con 0x7f6658094da0 addr v1:172.21.15.102:6790/0 2021-05-25T12:04:08.672 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.657+0000 7f666056e700 10 monclient(hunting): picked mon.noname-a con 0x7f66581323c0 addr v1:172.21.15.102:6789/0 2021-05-25T12:04:08.672 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.657+0000 7f666056e700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6658063fb0 con 0x7f66581323c0 2021-05-25T12:04:08.672 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.657+0000 7f666056e700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66580982f0 con 0x7f6658063840 2021-05-25T12:04:08.673 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.657+0000 7f666056e700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6658064510 con 0x7f6658094da0 2021-05-25T12:04:08.673 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.657+0000 7f666056e700 10 monclient(hunting): _renew_subs 2021-05-25T12:04:08.673 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.657+0000 7f666056e700 10 monclient(hunting): authenticate will time out at 1049.254485s 2021-05-25T12:04:08.673 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.657+0000 7f665eb0b700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f66581323c0 0x7f66581347b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60626/0 (socket says 172.21.15.102:60626) 2021-05-25T12:04:08.673 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.657+0000 7f665eb0b700 1 -- 172.21.15.102:0/1001611528 learned_addr learned my addr 172.21.15.102:0/1001611528 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:04:08.674 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.657+0000 7f665db09700 1 --1- 172.21.15.102:0/1001611528 >> v1:172.21.15.102:6790/0 conn(0x7f6658094da0 0x7f6658097be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:55502/0 (socket says 172.21.15.102:55502) 2021-05-25T12:04:08.674 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.657+0000 7f665d308700 1 -- 172.21.15.102:0/1001611528 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 331891122 0 0) 0x7f6658064510 con 0x7f6658094da0 2021-05-25T12:04:08.674 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.657+0000 7f665d308700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:04:08.674 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.657+0000 7f665d308700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:04:08.675 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.657+0000 7f665d308700 10 monclient(hunting): my global_id is 184130 2021-05-25T12:04:08.675 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.657+0000 7f665d308700 1 -- 172.21.15.102:0/1001611528 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6640003270 con 0x7f6658094da0 2021-05-25T12:04:08.675 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.657+0000 7f665d308700 1 -- 172.21.15.102:0/1001611528 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1478681219 0 0) 0x7f6658063fb0 con 0x7f66581323c0 2021-05-25T12:04:08.675 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.657+0000 7f665d308700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:04:08.675 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.657+0000 7f665d308700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:04:08.676 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.657+0000 7f665d308700 10 monclient(hunting): my global_id is 174402 2021-05-25T12:04:08.676 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.657+0000 7f665d308700 1 -- 172.21.15.102:0/1001611528 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6658064510 con 0x7f66581323c0 2021-05-25T12:04:08.676 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.658+0000 7f665d308700 1 -- 172.21.15.102:0/1001611528 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4250866008 0 0) 0x7f6640003270 con 0x7f6658094da0 2021-05-25T12:04:08.676 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.658+0000 7f665d308700 1 -- 172.21.15.102:0/1001611528 >> v1:172.21.15.181:6789/0 conn(0x7f6658063840 legacy=0x7f66580974d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:08.677 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.658+0000 7f665d308700 1 -- 172.21.15.102:0/1001611528 >> v1:172.21.15.102:6789/0 conn(0x7f66581323c0 legacy=0x7f66581347b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:08.677 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.658+0000 7f665d308700 10 monclient: _finish_hunting 0 2021-05-25T12:04:08.677 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.658+0000 7f665d308700 1 monclient: found mon.noname-b 2021-05-25T12:04:08.677 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.658+0000 7f665d308700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:04:08.677 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.658+0000 7f665d308700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T12:04:08.678 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.658+0000 7f665d308700 1 -- 172.21.15.102:0/1001611528 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66580647b0 con 0x7f6658094da0 2021-05-25T12:04:08.678 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.658+0000 7f665d308700 10 monclient: _finish_auth 0 2021-05-25T12:04:08.678 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.658+0000 7f665d308700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:04:08.678 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.658+0000 7f665d308700 1 -- 172.21.15.102:0/1001611528 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f6648001610 con 0x7f6658094da0 2021-05-25T12:04:08.678 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.658+0000 7f666056e700 5 monclient: authenticate success, global_id 184130 2021-05-25T12:04:08.679 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.658+0000 7f666056e700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:04:08.679 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.658+0000 7f665d308700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:08.679 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.658+0000 7f665d308700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:04:08.679 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.658+0000 7f665d308700 10 monclient: dump: 2021-05-25T12:04:08.680 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:08.680 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:08.680 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:08.680 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:08.680 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:08.681 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:08.681 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:08.681 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:08.681 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:08.681 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:08.682 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.658+0000 7f666056e700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:04:08.682 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.658+0000 7f665d308700 1 -- 172.21.15.102:0/1001611528 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6648001840 con 0x7f6658094da0 2021-05-25T12:04:08.682 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.658+0000 7f665d308700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:04:08.682 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.658+0000 7f665d308700 1 -- 172.21.15.102:0/1001611528 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f6648001bd0 con 0x7f6658094da0 2021-05-25T12:04:08.683 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.658+0000 7f666056e700 10 monclient: get_monmap_and_config success 2021-05-25T12:04:08.683 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.658+0000 7f666056e700 4 set_mon_vals no callback set 2021-05-25T12:04:08.683 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.658+0000 7f665d308700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:08.683 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.658+0000 7f665d308700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:04:08.683 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.658+0000 7f665d308700 10 monclient: dump: 2021-05-25T12:04:08.684 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:08.684 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:08.684 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:08.684 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:08.684 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:08.685 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:08.685 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:08.685 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:08.685 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:08.686 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:08.686 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.658+0000 7f666056e700 10 monclient: shutdown 2021-05-25T12:04:08.686 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.658+0000 7f666056e700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:04:08.686 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.658+0000 7f666056e700 1 -- 172.21.15.102:0/1001611528 >> v1:172.21.15.102:6790/0 conn(0x7f6658094da0 legacy=0x7f6658097be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:08.686 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.658+0000 7f666056e700 1 -- 172.21.15.102:0/1001611528 shutdown_connections 2021-05-25T12:04:08.687 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.658+0000 7f666056e700 1 -- 172.21.15.102:0/1001611528 >> 172.21.15.102:0/1001611528 conn(0x7f665809cb30 msgr2=0x7f665809cf40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:04:08.687 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.658+0000 7f666056e700 1 -- 172.21.15.102:0/1001611528 shutdown_connections 2021-05-25T12:04:08.687 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.658+0000 7f666056e700 1 -- 172.21.15.102:0/1001611528 wait complete. 2021-05-25T12:04:08.687 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.659+0000 7f666056e700 10 monclient: build_initial_monmap 2021-05-25T12:04:08.687 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.659+0000 7f666056e700 10 monclient: monmap: 2021-05-25T12:04:08.688 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:04:08.688 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:04:08.688 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:04:08.688 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:04:08.689 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:04:08.689 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:04:08.689 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:04:08.689 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:04:08.689 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:04:08.690 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:08.690 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.659+0000 7f666056e700 1 librados: starting msgr at 2021-05-25T12:04:08.690 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.659+0000 7f666056e700 1 librados: starting objecter 2021-05-25T12:04:08.690 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.659+0000 7f666056e700 1 Processor -- start 2021-05-25T12:04:08.690 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.659+0000 7f666056e700 1 -- start start 2021-05-25T12:04:08.691 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.659+0000 7f666056e700 1 librados: setting wanted keys 2021-05-25T12:04:08.691 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.659+0000 7f666056e700 1 librados: calling monclient init 2021-05-25T12:04:08.691 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.659+0000 7f666056e700 10 monclient: init 2021-05-25T12:04:08.691 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.659+0000 7f666056e700 10 monclient: _reopen_session rank -1 2021-05-25T12:04:08.692 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.659+0000 7f666056e700 10 monclient: _add_conns ranks=[1,0,2] 2021-05-25T12:04:08.692 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.659+0000 7f666056e700 10 monclient(hunting): picked mon.noname-b con 0x7f6658063840 addr v1:172.21.15.181:6789/0 2021-05-25T12:04:08.692 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.660+0000 7f666056e700 10 monclient(hunting): picked mon.noname-a con 0x7f6658094da0 addr v1:172.21.15.102:6789/0 2021-05-25T12:04:08.692 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.660+0000 7f666056e700 10 monclient(hunting): picked mon.noname-c con 0x7f66581323c0 addr v1:172.21.15.102:6790/0 2021-05-25T12:04:08.692 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.660+0000 7f666056e700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6658119b40 con 0x7f6658094da0 2021-05-25T12:04:08.693 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.660+0000 7f666056e700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f665811c1e0 con 0x7f6658063840 2021-05-25T12:04:08.693 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.660+0000 7f666056e700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f665811c480 con 0x7f66581323c0 2021-05-25T12:04:08.693 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.660+0000 7f666056e700 10 monclient(hunting): _renew_subs 2021-05-25T12:04:08.693 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.660+0000 7f666056e700 10 monclient(hunting): authenticate will time out at 1049.257059s 2021-05-25T12:04:08.694 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.660+0000 7f665db09700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f6658094da0 0x7f6658140270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60630/0 (socket says 172.21.15.102:60630) 2021-05-25T12:04:08.694 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.660+0000 7f665eb0b700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f66581323c0 0x7f66581439a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:55510/0 (socket says 172.21.15.102:55510) 2021-05-25T12:04:08.694 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.660+0000 7f665db09700 1 -- 172.21.15.102:0/2184078925 learned_addr learned my addr 172.21.15.102:0/2184078925 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:04:08.694 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.660+0000 7f66467fc700 10 client.?.objecter ms_handle_connect 0x7f66581323c0 2021-05-25T12:04:08.694 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.660+0000 7f66467fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:04:08.695 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.660+0000 7f66467fc700 10 client.?.objecter ms_handle_connect 0x7f6658094da0 2021-05-25T12:04:08.695 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.660+0000 7f66467fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:04:08.695 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.660+0000 7f66467fc700 10 client.?.objecter ms_handle_connect 0x7f6658063840 2021-05-25T12:04:08.695 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.660+0000 7f66467fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:04:08.695 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.660+0000 7f66467fc700 1 -- 172.21.15.102:0/2184078925 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3354022664 0 0) 0x7f665811c480 con 0x7f66581323c0 2021-05-25T12:04:08.696 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.660+0000 7f66467fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:04:08.696 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.660+0000 7f66467fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:04:08.696 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.660+0000 7f66467fc700 10 monclient(hunting): my global_id is 184133 2021-05-25T12:04:08.696 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.660+0000 7f66467fc700 1 -- 172.21.15.102:0/2184078925 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6638004c10 con 0x7f66581323c0 2021-05-25T12:04:08.697 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.660+0000 7f66467fc700 1 -- 172.21.15.102:0/2184078925 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 757364217 0 0) 0x7f6658119b40 con 0x7f6658094da0 2021-05-25T12:04:08.697 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.660+0000 7f66467fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:04:08.697 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.660+0000 7f66467fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:04:08.697 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.660+0000 7f66467fc700 10 monclient(hunting): my global_id is 174405 2021-05-25T12:04:08.697 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.660+0000 7f66467fc700 1 -- 172.21.15.102:0/2184078925 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f665811c480 con 0x7f6658094da0 2021-05-25T12:04:08.698 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.660+0000 7f66467fc700 1 -- 172.21.15.102:0/2184078925 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 614669264 0 0) 0x7f665811c1e0 con 0x7f6658063840 2021-05-25T12:04:08.698 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.660+0000 7f66467fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:04:08.698 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.660+0000 7f66467fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:04:08.698 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.660+0000 7f66467fc700 10 monclient(hunting): my global_id is 95326 2021-05-25T12:04:08.698 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.660+0000 7f66467fc700 1 -- 172.21.15.102:0/2184078925 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6658119b40 con 0x7f6658063840 2021-05-25T12:04:08.699 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.660+0000 7f66467fc700 1 -- 172.21.15.102:0/2184078925 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1617795705 0 0) 0x7f6638004c10 con 0x7f66581323c0 2021-05-25T12:04:08.699 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.660+0000 7f66467fc700 1 -- 172.21.15.102:0/2184078925 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f665811c1e0 con 0x7f66581323c0 2021-05-25T12:04:08.699 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.660+0000 7f66467fc700 1 -- 172.21.15.102:0/2184078925 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f6654001ca0 con 0x7f66581323c0 2021-05-25T12:04:08.699 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.660+0000 7f66467fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:08.700 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.660+0000 7f66467fc700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:04:08.700 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.660+0000 7f66467fc700 10 monclient(hunting): dump: 2021-05-25T12:04:08.700 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:08.700 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:08.700 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:08.701 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:08.701 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:08.701 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:08.701 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:08.701 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:08.702 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:08.702 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:08.702 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.660+0000 7f66467fc700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:04:08.702 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.660+0000 7f66467fc700 1 -- 172.21.15.102:0/2184078925 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1893179518 0 0) 0x7f665811c480 con 0x7f6658094da0 2021-05-25T12:04:08.702 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.660+0000 7f66467fc700 1 -- 172.21.15.102:0/2184078925 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6638004c10 con 0x7f6658094da0 2021-05-25T12:04:08.703 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.660+0000 7f66467fc700 1 -- 172.21.15.102:0/2184078925 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f6648001b50 con 0x7f6658094da0 2021-05-25T12:04:08.703 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.660+0000 7f66467fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:08.703 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.660+0000 7f66467fc700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2021-05-25T12:04:08.703 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.660+0000 7f66467fc700 10 monclient(hunting): dump: 2021-05-25T12:04:08.704 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:08.704 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:08.704 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:08.704 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:08.704 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:08.705 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:08.705 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:08.705 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:08.705 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:08.705 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:08.706 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.660+0000 7f66467fc700 1 -- 172.21.15.102:0/2184078925 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 857938102 0 0) 0x7f6658119b40 con 0x7f6658063840 2021-05-25T12:04:08.706 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.660+0000 7f66467fc700 1 -- 172.21.15.102:0/2184078925 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f665811c480 con 0x7f6658063840 2021-05-25T12:04:08.706 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.660+0000 7f66467fc700 1 -- 172.21.15.102:0/2184078925 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f664c001780 con 0x7f6658063840 2021-05-25T12:04:08.706 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.660+0000 7f66467fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:08.707 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.660+0000 7f66467fc700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2021-05-25T12:04:08.707 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.660+0000 7f66467fc700 10 monclient(hunting): dump: 2021-05-25T12:04:08.707 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:08.707 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:08.707 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:08.708 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:08.708 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:08.708 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:08.708 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:08.708 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:08.709 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:08.709 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:08.709 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.660+0000 7f66467fc700 1 -- 172.21.15.102:0/2184078925 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2419909032 0 0) 0x7f665811c1e0 con 0x7f66581323c0 2021-05-25T12:04:08.709 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.660+0000 7f66467fc700 1 -- 172.21.15.102:0/2184078925 >> v1:172.21.15.181:6789/0 conn(0x7f6658063840 legacy=0x7f6658131a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:08.709 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.660+0000 7f66467fc700 1 -- 172.21.15.102:0/2184078925 >> v1:172.21.15.102:6789/0 conn(0x7f6658094da0 legacy=0x7f6658140270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:08.710 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.660+0000 7f66467fc700 10 monclient: _finish_hunting 0 2021-05-25T12:04:08.710 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.660+0000 7f66467fc700 1 monclient: found mon.c 2021-05-25T12:04:08.710 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.660+0000 7f66467fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:04:08.710 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.660+0000 7f66467fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:04:08.711 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.660+0000 7f66467fc700 1 -- 172.21.15.102:0/2184078925 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6658132180 con 0x7f66581323c0 2021-05-25T12:04:08.711 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.660+0000 7f66467fc700 10 monclient: _finish_auth 0 2021-05-25T12:04:08.711 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.660+0000 7f66467fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:04:08.711 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.660+0000 7f666056e700 5 monclient: authenticate success, global_id 184133 2021-05-25T12:04:08.711 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.660+0000 7f666056e700 10 monclient: _renew_subs 2021-05-25T12:04:08.712 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.660+0000 7f666056e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:04:08.712 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.660+0000 7f666056e700 1 -- 172.21.15.102:0/2184078925 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f665811a310 con 0x7f66581323c0 2021-05-25T12:04:08.712 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.660+0000 7f666056e700 10 client.184133.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:04:08.712 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.660+0000 7f666056e700 10 monclient: _renew_subs 2021-05-25T12:04:08.712 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.660+0000 7f666056e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:04:08.713 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.660+0000 7f666056e700 1 -- 172.21.15.102:0/2184078925 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6658137bc0 con 0x7f66581323c0 2021-05-25T12:04:08.713 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.660+0000 7f666056e700 1 librados: init done 2021-05-25T12:04:08.713 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.660+0000 7f66467fc700 1 -- 172.21.15.102:0/2184078925 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f66540048b0 con 0x7f66581323c0 2021-05-25T12:04:08.713 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.661+0000 7f66467fc700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:04:08.713 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.661+0000 7f66467fc700 1 -- 172.21.15.102:0/2184078925 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f6654004bc0 con 0x7f66581323c0 2021-05-25T12:04:08.714 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.661+0000 7f66467fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:08.714 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.661+0000 7f665cb07700 4 set_mon_vals no callback set 2021-05-25T12:04:08.714 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.661+0000 7f66467fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:04:08.714 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.661+0000 7f66467fc700 10 monclient: dump: 2021-05-25T12:04:08.715 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:08.715 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:08.715 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:08.715 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:08.715 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:08.716 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:08.716 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:08.716 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:08.716 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:08.716 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:08.717 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.661+0000 7f66467fc700 1 -- 172.21.15.102:0/2184078925 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f665400dee0 con 0x7f66581323c0 2021-05-25T12:04:08.717 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.661+0000 7f666056e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:04:08.717 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.661+0000 7f666056e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:04:08.717 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.661+0000 7f666056e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:04:08.717 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.661+0000 7f666056e700 1 -- 172.21.15.102:0/2184078925 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f663c004db0 con 0x7f66581323c0 2021-05-25T12:04:08.718 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.661+0000 7f66467fc700 1 -- 172.21.15.102:0/2184078925 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(124..124 src has 1..124) v4 ==== 5720+0+0 (unknown 4058276862 0 0) 0x7f665403d900 con 0x7f66581323c0 2021-05-25T12:04:08.718 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.661+0000 7f66467fc700 10 client.184133.objecter ms_dispatch 0x7f6658001550 osd_map(124..124 src has 1..124) v4 2021-05-25T12:04:08.718 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.661+0000 7f66467fc700 3 client.184133.objecter handle_osd_map got epochs [124,124] > 0 2021-05-25T12:04:08.718 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.661+0000 7f66467fc700 3 client.184133.objecter handle_osd_map decoding full epoch 124 2021-05-25T12:04:08.719 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.662+0000 7f66467fc700 20 client.184133.objecter dump_active .. 0 homeless 2021-05-25T12:04:08.719 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.662+0000 7f665e30a700 10 monclient: get_auth_request con 0x7f6638033550 auth_method 0 2021-05-25T12:04:08.719 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.662+0000 7f66467fc700 10 client.184133.objecter ms_handle_connect 0x7f6638033550 2021-05-25T12:04:08.719 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.665+0000 7f66467fc700 1 -- 172.21.15.102:0/2184078925 <== mon.2 v1:172.21.15.102:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7f6654012e00 con 0x7f66581323c0 2021-05-25T12:04:08.719 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.665+0000 7f66467fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:04:08.720 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.665+0000 7f66467fc700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:04:08.877 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.875+0000 7f666056e700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2021-05-25T12:04:08.877 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.875+0000 7f666056e700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2021-05-25T12:04:08.877 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.875+0000 7f666056e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:04:08.877 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.875+0000 7f666056e700 1 -- 172.21.15.102:0/2184078925 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f663c005060 con 0x7f66581323c0 2021-05-25T12:04:08.878 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.875+0000 7f66467fc700 1 -- 172.21.15.102:0/2184078925 <== mon.2 v1:172.21.15.102:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1342 (unknown 254746657 0 2467925024) 0x7f6654012c20 con 0x7f66581323c0 2021-05-25T12:04:08.878 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.875+0000 7f66467fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2021-05-25T12:04:08.878 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.875+0000 7f66467fc700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:04:08.879 INFO:teuthology.orchestra.run.smithi102.stdout:{"election_epoch":38,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"b","quorum_age":1,"features":{"quorum_con":"4540138303579357183","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"20ee91b6-8935-4b98-b868-85c03789ccc4","modified":"2021-05-25T11:59:11.011522Z","created":"2021-05-25T11:59:11.011522Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6790","nonce":0}]},"addr":"172.21.15.102:6790/0","public_addr":"172.21.15.102:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2021-05-25T12:04:08.880 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.877+0000 7f666056e700 10 librados: watch_flush enter 2021-05-25T12:04:08.880 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.877+0000 7f666056e700 10 librados: watch_flush exit 2021-05-25T12:04:08.881 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.877+0000 7f666056e700 20 client.184133.objecter shutdown clearing up homeless session... 2021-05-25T12:04:08.881 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.877+0000 7f666056e700 10 client.184133.objecter successfully canceled tick 2021-05-25T12:04:08.881 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.877+0000 7f666056e700 1 -- 172.21.15.102:0/2184078925 >> v1:172.21.15.181:6812/29582 conn(0x7f6638033550 legacy=0x7f6638035a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:08.881 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.877+0000 7f666056e700 10 monclient: shutdown 2021-05-25T12:04:08.881 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.877+0000 7f666056e700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:04:08.882 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.877+0000 7f666056e700 1 -- 172.21.15.102:0/2184078925 >> v1:172.21.15.102:6790/0 conn(0x7f66581323c0 legacy=0x7f66581439a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:08.882 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.877+0000 7f666056e700 1 -- 172.21.15.102:0/2184078925 shutdown_connections 2021-05-25T12:04:08.882 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.877+0000 7f666056e700 1 -- 172.21.15.102:0/2184078925 >> 172.21.15.102:0/2184078925 conn(0x7f665809cb30 msgr2=0x7f6658067bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:04:08.882 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.877+0000 7f666056e700 1 -- 172.21.15.102:0/2184078925 shutdown_connections 2021-05-25T12:04:08.882 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.877+0000 7f666056e700 1 -- 172.21.15.102:0/2184078925 wait complete. 2021-05-25T12:04:08.883 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:08.877+0000 7f666056e700 1 librados: shutdown 2021-05-25T12:04:08.889 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2021-05-25T12:04:08.890 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2021-05-25T12:04:09.005 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.003+0000 7fae03a68700 10 monclient: get_monmap_and_config 2021-05-25T12:04:09.005 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.003+0000 7fae03a68700 10 monclient: build_initial_monmap 2021-05-25T12:04:09.006 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.003+0000 7fae03a68700 10 monclient: monmap: 2021-05-25T12:04:09.006 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:04:09.006 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:09.006 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:04:09.004751+0000 2021-05-25T12:04:09.007 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:04:09.004751+0000 2021-05-25T12:04:09.007 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:04:09.007 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:09.007 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:04:09.007 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:04:09.008 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:04:09.008 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:09.020 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.004+0000 7fae03a68700 1 Processor -- start 2021-05-25T12:04:09.020 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.004+0000 7fae03a68700 1 -- start start 2021-05-25T12:04:09.020 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.004+0000 7fae03a68700 10 monclient: init 2021-05-25T12:04:09.021 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.004+0000 7fae03a68700 10 monclient: _reopen_session rank -1 2021-05-25T12:04:09.021 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.004+0000 7fae03a68700 10 monclient: _add_conns ranks=[2,0,1] 2021-05-25T12:04:09.021 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.004+0000 7fae03a68700 10 monclient(hunting): picked mon.noname-b con 0x7fadfc0966d0 addr v1:172.21.15.102:6790/0 2021-05-25T12:04:09.021 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.004+0000 7fae03a68700 10 monclient(hunting): picked mon.noname-a con 0x7fadfc096ab0 addr v1:172.21.15.102:6789/0 2021-05-25T12:04:09.022 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.004+0000 7fae03a68700 10 monclient(hunting): picked mon.noname-c con 0x7fadfc132010 addr v1:172.21.15.181:6789/0 2021-05-25T12:04:09.022 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.005+0000 7fae03a68700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadfc064610 con 0x7fadfc096ab0 2021-05-25T12:04:09.022 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.005+0000 7fae03a68700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadfc064840 con 0x7fadfc132010 2021-05-25T12:04:09.022 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.005+0000 7fae03a68700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadfc064ae0 con 0x7fadfc0966d0 2021-05-25T12:04:09.022 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.005+0000 7fae03a68700 10 monclient(hunting): _renew_subs 2021-05-25T12:04:09.023 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.005+0000 7fae03a68700 10 monclient(hunting): authenticate will time out at 1049.602045s 2021-05-25T12:04:09.023 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.005+0000 7fae02005700 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7fadfc132010 0x7fadfc134400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.102:46290/0 (socket says 172.21.15.102:46290) 2021-05-25T12:04:09.023 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.005+0000 7fae02005700 1 -- 172.21.15.102:0/671816422 learned_addr learned my addr 172.21.15.102:0/671816422 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:04:09.023 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.005+0000 7fae01804700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fadfc0966d0 0x7fadfc0697d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:55514/0 (socket says 172.21.15.102:55514) 2021-05-25T12:04:09.024 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.005+0000 7fae01003700 1 --1- 172.21.15.102:0/671816422 >> v1:172.21.15.102:6789/0 conn(0x7fadfc096ab0 0x7fadfc1308f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60638/0 (socket says 172.21.15.102:60638) 2021-05-25T12:04:09.024 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.005+0000 7fae00802700 1 -- 172.21.15.102:0/671816422 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3380668108 0 0) 0x7fadfc064ae0 con 0x7fadfc0966d0 2021-05-25T12:04:09.024 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.005+0000 7fae00802700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:04:09.024 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.005+0000 7fae00802700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:04:09.024 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.005+0000 7fae00802700 10 monclient(hunting): my global_id is 184136 2021-05-25T12:04:09.025 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.005+0000 7fae00802700 1 -- 172.21.15.102:0/671816422 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fade8003270 con 0x7fadfc0966d0 2021-05-25T12:04:09.025 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.005+0000 7fae00802700 1 -- 172.21.15.102:0/671816422 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 645362928 0 0) 0x7fadfc064840 con 0x7fadfc132010 2021-05-25T12:04:09.025 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.005+0000 7fae00802700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:04:09.025 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.005+0000 7fae00802700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:04:09.025 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.005+0000 7fae00802700 10 monclient(hunting): my global_id is 95329 2021-05-25T12:04:09.026 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.005+0000 7fae00802700 1 -- 172.21.15.102:0/671816422 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadfc064ae0 con 0x7fadfc132010 2021-05-25T12:04:09.026 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.005+0000 7fae00802700 1 -- 172.21.15.102:0/671816422 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1966074557 0 0) 0x7fadfc064610 con 0x7fadfc096ab0 2021-05-25T12:04:09.026 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.005+0000 7fae00802700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:04:09.026 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.005+0000 7fae00802700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:04:09.027 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.005+0000 7fae00802700 10 monclient(hunting): my global_id is 174408 2021-05-25T12:04:09.027 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.005+0000 7fae00802700 1 -- 172.21.15.102:0/671816422 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadfc064840 con 0x7fadfc096ab0 2021-05-25T12:04:09.027 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.005+0000 7fae00802700 1 -- 172.21.15.102:0/671816422 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2989662245 0 0) 0x7fade8003270 con 0x7fadfc0966d0 2021-05-25T12:04:09.027 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.005+0000 7fae00802700 1 -- 172.21.15.102:0/671816422 >> v1:172.21.15.181:6789/0 conn(0x7fadfc132010 legacy=0x7fadfc134400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:09.027 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.005+0000 7fae00802700 1 -- 172.21.15.102:0/671816422 >> v1:172.21.15.102:6789/0 conn(0x7fadfc096ab0 legacy=0x7fadfc1308f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:09.028 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.005+0000 7fae00802700 10 monclient: _finish_hunting 0 2021-05-25T12:04:09.028 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.005+0000 7fae00802700 1 monclient: found mon.noname-b 2021-05-25T12:04:09.028 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.005+0000 7fae00802700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:04:09.028 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.005+0000 7fae00802700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T12:04:09.028 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.005+0000 7fae00802700 1 -- 172.21.15.102:0/671816422 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadfc069ee0 con 0x7fadfc0966d0 2021-05-25T12:04:09.029 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.005+0000 7fae00802700 10 monclient: _finish_auth 0 2021-05-25T12:04:09.029 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.005+0000 7fae00802700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:04:09.029 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.005+0000 7fae00802700 1 -- 172.21.15.102:0/671816422 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fade4001610 con 0x7fadfc0966d0 2021-05-25T12:04:09.029 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.005+0000 7fae03a68700 5 monclient: authenticate success, global_id 184136 2021-05-25T12:04:09.029 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.005+0000 7fae03a68700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:04:09.030 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.005+0000 7fae00802700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:09.030 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.005+0000 7fae00802700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:04:09.030 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.005+0000 7fae00802700 10 monclient: dump: 2021-05-25T12:04:09.030 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:09.031 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:09.031 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:09.031 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:09.031 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:09.031 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:09.032 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:09.032 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:09.032 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:09.032 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:09.032 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.005+0000 7fae03a68700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:04:09.033 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.005+0000 7fae00802700 1 -- 172.21.15.102:0/671816422 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fade4001870 con 0x7fadfc0966d0 2021-05-25T12:04:09.033 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.005+0000 7fae00802700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:04:09.033 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.005+0000 7fae03a68700 10 monclient: get_monmap_and_config success 2021-05-25T12:04:09.033 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.005+0000 7fae00802700 1 -- 172.21.15.102:0/671816422 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fade4001c00 con 0x7fadfc0966d0 2021-05-25T12:04:09.033 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.005+0000 7fae00802700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:09.034 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.005+0000 7fae03a68700 4 set_mon_vals no callback set 2021-05-25T12:04:09.034 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.005+0000 7fae00802700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:04:09.034 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.005+0000 7fae00802700 10 monclient: dump: 2021-05-25T12:04:09.034 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:09.035 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:09.035 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:09.035 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:09.035 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:09.035 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:09.036 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:09.036 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:09.036 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:09.036 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:09.036 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.006+0000 7fae03a68700 10 monclient: shutdown 2021-05-25T12:04:09.037 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.006+0000 7fae03a68700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:04:09.037 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.006+0000 7fae03a68700 1 -- 172.21.15.102:0/671816422 >> v1:172.21.15.102:6790/0 conn(0x7fadfc0966d0 legacy=0x7fadfc0697d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:09.037 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.006+0000 7fae03a68700 1 -- 172.21.15.102:0/671816422 shutdown_connections 2021-05-25T12:04:09.037 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.006+0000 7fae03a68700 1 -- 172.21.15.102:0/671816422 >> 172.21.15.102:0/671816422 conn(0x7fadfc09cd10 msgr2=0x7fadfc09d120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:04:09.037 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.006+0000 7fae03a68700 1 -- 172.21.15.102:0/671816422 shutdown_connections 2021-05-25T12:04:09.038 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.006+0000 7fae03a68700 1 -- 172.21.15.102:0/671816422 wait complete. 2021-05-25T12:04:09.038 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.006+0000 7fae03a68700 10 monclient: build_initial_monmap 2021-05-25T12:04:09.038 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.007+0000 7fae03a68700 10 monclient: monmap: 2021-05-25T12:04:09.038 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:04:09.039 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:04:09.039 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:04:09.039 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:04:09.039 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:04:09.039 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:04:09.040 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:04:09.040 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:04:09.040 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:04:09.040 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:09.040 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.007+0000 7fae03a68700 1 librados: starting msgr at 2021-05-25T12:04:09.041 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.007+0000 7fae03a68700 1 librados: starting objecter 2021-05-25T12:04:09.041 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.007+0000 7fae03a68700 1 Processor -- start 2021-05-25T12:04:09.041 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.007+0000 7fae03a68700 1 -- start start 2021-05-25T12:04:09.041 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.007+0000 7fae03a68700 1 librados: setting wanted keys 2021-05-25T12:04:09.041 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.007+0000 7fae03a68700 1 librados: calling monclient init 2021-05-25T12:04:09.042 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.007+0000 7fae03a68700 10 monclient: init 2021-05-25T12:04:09.042 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.007+0000 7fae03a68700 10 monclient: _reopen_session rank -1 2021-05-25T12:04:09.042 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.007+0000 7fae03a68700 10 monclient: _add_conns ranks=[0,1,2] 2021-05-25T12:04:09.042 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.007+0000 7fae03a68700 10 monclient(hunting): picked mon.noname-a con 0x7fadfc0966d0 addr v1:172.21.15.102:6789/0 2021-05-25T12:04:09.043 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.007+0000 7fae03a68700 10 monclient(hunting): picked mon.noname-b con 0x7fadfc096ab0 addr v1:172.21.15.181:6789/0 2021-05-25T12:04:09.043 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.007+0000 7fae03a68700 10 monclient(hunting): picked mon.noname-c con 0x7fadfc132010 addr v1:172.21.15.102:6790/0 2021-05-25T12:04:09.043 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.007+0000 7fae03a68700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadfc11bae0 con 0x7fadfc0966d0 2021-05-25T12:04:09.043 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.007+0000 7fae03a68700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadfc11bd80 con 0x7fadfc096ab0 2021-05-25T12:04:09.043 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.007+0000 7fae03a68700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadfc11c020 con 0x7fadfc132010 2021-05-25T12:04:09.044 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.007+0000 7fae03a68700 10 monclient(hunting): _renew_subs 2021-05-25T12:04:09.044 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.007+0000 7fae03a68700 10 monclient(hunting): authenticate will time out at 1049.604757s 2021-05-25T12:04:09.044 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.007+0000 7fae01804700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fadfc0966d0 0x7fadfc131610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60642/0 (socket says 172.21.15.102:60642) 2021-05-25T12:04:09.044 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.007+0000 7fae01804700 1 -- 172.21.15.102:0/3055332148 learned_addr learned my addr 172.21.15.102:0/3055332148 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:04:09.044 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.007+0000 7fae02005700 1 --1- 172.21.15.102:0/3055332148 >> v1:172.21.15.102:6790/0 conn(0x7fadfc132010 0x7fadfc143540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:55524/0 (socket says 172.21.15.102:55524) 2021-05-25T12:04:09.045 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.007+0000 7fadf1ffb700 10 client.?.objecter ms_handle_connect 0x7fadfc132010 2021-05-25T12:04:09.045 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.007+0000 7fadf1ffb700 10 client.?.objecter resend_mon_ops 2021-05-25T12:04:09.045 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.007+0000 7fadf1ffb700 10 client.?.objecter ms_handle_connect 0x7fadfc0966d0 2021-05-25T12:04:09.045 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.007+0000 7fadf1ffb700 10 client.?.objecter resend_mon_ops 2021-05-25T12:04:09.045 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.008+0000 7fadf1ffb700 1 -- 172.21.15.102:0/3055332148 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4224567503 0 0) 0x7fadfc11c020 con 0x7fadfc132010 2021-05-25T12:04:09.046 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.008+0000 7fadf1ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:04:09.046 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.008+0000 7fadf1ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:04:09.046 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.008+0000 7fadf1ffb700 10 monclient(hunting): my global_id is 184139 2021-05-25T12:04:09.046 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.008+0000 7fadf1ffb700 1 -- 172.21.15.102:0/3055332148 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faddc004c10 con 0x7fadfc132010 2021-05-25T12:04:09.046 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.008+0000 7fadf1ffb700 1 -- 172.21.15.102:0/3055332148 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3801620351 0 0) 0x7fadfc11bae0 con 0x7fadfc0966d0 2021-05-25T12:04:09.047 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.008+0000 7fadf1ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:04:09.047 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.008+0000 7fadf1ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:04:09.047 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.008+0000 7fadf1ffb700 10 monclient(hunting): my global_id is 174411 2021-05-25T12:04:09.047 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.008+0000 7fadf1ffb700 1 -- 172.21.15.102:0/3055332148 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadfc11c020 con 0x7fadfc0966d0 2021-05-25T12:04:09.048 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.008+0000 7fadf1ffb700 10 client.?.objecter ms_handle_connect 0x7fadfc096ab0 2021-05-25T12:04:09.048 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.008+0000 7fadf1ffb700 10 client.?.objecter resend_mon_ops 2021-05-25T12:04:09.048 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.008+0000 7fadf1ffb700 1 -- 172.21.15.102:0/3055332148 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1397566700 0 0) 0x7fadfc11bd80 con 0x7fadfc096ab0 2021-05-25T12:04:09.048 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.008+0000 7fadf1ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:04:09.048 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.008+0000 7fadf1ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:04:09.049 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.008+0000 7fadf1ffb700 10 monclient(hunting): my global_id is 95332 2021-05-25T12:04:09.049 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.008+0000 7fadf1ffb700 1 -- 172.21.15.102:0/3055332148 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadfc11bae0 con 0x7fadfc096ab0 2021-05-25T12:04:09.049 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.008+0000 7fadf1ffb700 1 -- 172.21.15.102:0/3055332148 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1441617887 0 0) 0x7faddc004c10 con 0x7fadfc132010 2021-05-25T12:04:09.049 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.008+0000 7fadf1ffb700 1 -- 172.21.15.102:0/3055332148 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fadfc11bd80 con 0x7fadfc132010 2021-05-25T12:04:09.049 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.008+0000 7fadf1ffb700 1 -- 172.21.15.102:0/3055332148 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 870843885 0 0) 0x7fadfc11c020 con 0x7fadfc0966d0 2021-05-25T12:04:09.050 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.008+0000 7fadf1ffb700 1 -- 172.21.15.102:0/3055332148 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faddc004c10 con 0x7fadfc0966d0 2021-05-25T12:04:09.050 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.008+0000 7fadf1ffb700 1 -- 172.21.15.102:0/3055332148 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fadf8001bb0 con 0x7fadfc132010 2021-05-25T12:04:09.050 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.008+0000 7fadf1ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:09.050 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.008+0000 7fadf1ffb700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:04:09.050 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.008+0000 7fadf1ffb700 10 monclient(hunting): dump: 2021-05-25T12:04:09.051 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:09.051 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:09.051 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:09.051 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:09.052 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:09.052 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:09.052 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:09.052 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:09.052 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:09.053 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:09.053 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.008+0000 7fadf1ffb700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:04:09.053 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.008+0000 7fadf1ffb700 1 -- 172.21.15.102:0/3055332148 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fade4001c00 con 0x7fadfc0966d0 2021-05-25T12:04:09.053 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.008+0000 7fadf1ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:09.053 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.008+0000 7fadf1ffb700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2021-05-25T12:04:09.054 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.008+0000 7fadf1ffb700 10 monclient(hunting): dump: 2021-05-25T12:04:09.054 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:09.054 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:09.054 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:09.054 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:09.055 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:09.055 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:09.055 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:09.055 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:09.055 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:09.056 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:09.056 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.008+0000 7fadf1ffb700 1 -- 172.21.15.102:0/3055332148 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 304962563 0 0) 0x7fadfc11bd80 con 0x7fadfc132010 2021-05-25T12:04:09.056 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.008+0000 7fadf1ffb700 1 -- 172.21.15.102:0/3055332148 >> v1:172.21.15.181:6789/0 conn(0x7fadfc096ab0 legacy=0x7fadfc13fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:09.056 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.008+0000 7fadf1ffb700 1 -- 172.21.15.102:0/3055332148 >> v1:172.21.15.102:6789/0 conn(0x7fadfc0966d0 legacy=0x7fadfc131610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:09.057 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.008+0000 7fadf1ffb700 10 monclient: _finish_hunting 0 2021-05-25T12:04:09.057 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.008+0000 7fadf1ffb700 1 monclient: found mon.c 2021-05-25T12:04:09.057 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.008+0000 7fadf1ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:04:09.057 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.008+0000 7fadf1ffb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:04:09.057 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.008+0000 7fadf1ffb700 1 -- 172.21.15.102:0/3055332148 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadfc119eb0 con 0x7fadfc132010 2021-05-25T12:04:09.058 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.008+0000 7fadf1ffb700 10 monclient: _finish_auth 0 2021-05-25T12:04:09.058 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.008+0000 7fadf1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:04:09.058 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.008+0000 7fae03a68700 5 monclient: authenticate success, global_id 184139 2021-05-25T12:04:09.058 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.008+0000 7fae03a68700 10 monclient: _renew_subs 2021-05-25T12:04:09.058 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.008+0000 7fae03a68700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:04:09.059 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.008+0000 7fae03a68700 1 -- 172.21.15.102:0/3055332148 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadfc11ab60 con 0x7fadfc132010 2021-05-25T12:04:09.059 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.008+0000 7fadf1ffb700 1 -- 172.21.15.102:0/3055332148 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fadf80048b0 con 0x7fadfc132010 2021-05-25T12:04:09.059 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.008+0000 7fadf1ffb700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:04:09.059 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.008+0000 7fadf1ffb700 1 -- 172.21.15.102:0/3055332148 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fadf8002120 con 0x7fadfc132010 2021-05-25T12:04:09.059 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.008+0000 7fadf1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:09.060 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.008+0000 7fadf1ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:04:09.060 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.008+0000 7fadf1ffb700 10 monclient: dump: 2021-05-25T12:04:09.060 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:09.060 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:09.061 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:09.061 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:09.061 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:09.061 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:09.061 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:09.062 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:09.062 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:09.062 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:09.062 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.008+0000 7fae03a68700 10 client.184139.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:04:09.062 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.008+0000 7fae03a68700 10 monclient: _renew_subs 2021-05-25T12:04:09.063 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.008+0000 7fae03a68700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:04:09.063 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.008+0000 7fadf3fff700 4 set_mon_vals no callback set 2021-05-25T12:04:09.063 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.008+0000 7fae03a68700 1 -- 172.21.15.102:0/3055332148 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fadfc137740 con 0x7fadfc132010 2021-05-25T12:04:09.063 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.008+0000 7fae03a68700 1 librados: init done 2021-05-25T12:04:09.064 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.009+0000 7fadf1ffb700 1 -- 172.21.15.102:0/3055332148 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7fadf800dd80 con 0x7fadfc132010 2021-05-25T12:04:09.064 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.009+0000 7fae03a68700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:04:09.064 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.009+0000 7fae03a68700 1 -- 172.21.15.102:0/3055332148 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fadfc131db0 con 0x7fadfc119350 2021-05-25T12:04:09.064 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.009+0000 7fadf1ffb700 1 -- 172.21.15.102:0/3055332148 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(125..125 src has 1..125) v4 ==== 5802+0+0 (unknown 2341125478 0 0) 0x7fadf803d820 con 0x7fadfc132010 2021-05-25T12:04:09.064 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.009+0000 7fadf1ffb700 10 client.184139.objecter ms_dispatch 0x7fadfc001550 osd_map(125..125 src has 1..125) v4 2021-05-25T12:04:09.065 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.009+0000 7fadf1ffb700 3 client.184139.objecter handle_osd_map got epochs [125,125] > 0 2021-05-25T12:04:09.065 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.009+0000 7fadf1ffb700 3 client.184139.objecter handle_osd_map decoding full epoch 125 2021-05-25T12:04:09.065 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.009+0000 7fadf1ffb700 20 client.184139.objecter dump_active .. 0 homeless 2021-05-25T12:04:09.065 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.009+0000 7fadf1ffb700 10 client.184139.objecter ms_handle_connect 0x7fadfc119350 2021-05-25T12:04:09.065 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.009+0000 7fadf1ffb700 10 client.184139.objecter resend_mon_ops 2021-05-25T12:04:09.066 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.009+0000 7fae01804700 10 monclient: get_auth_request con 0x7faddc033250 auth_method 0 2021-05-25T12:04:09.066 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.009+0000 7fadf1ffb700 1 -- 172.21.15.102:0/3055332148 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 498540382 0 0) 0x7fadfc131db0 con 0x7fadfc119350 2021-05-25T12:04:09.066 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.009+0000 7fadf1ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:04:09.066 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.009+0000 7fadf1ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:04:09.066 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.009+0000 7fadf1ffb700 10 monclient(hunting): my global_id is 174414 2021-05-25T12:04:09.067 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.009+0000 7fadf1ffb700 1 -- 172.21.15.102:0/3055332148 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadfc11bd80 con 0x7fadfc119350 2021-05-25T12:04:09.067 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.010+0000 7fadf1ffb700 1 -- 172.21.15.102:0/3055332148 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 871416634 0 0) 0x7fadfc11bd80 con 0x7fadfc119350 2021-05-25T12:04:09.067 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.010+0000 7fadf1ffb700 1 -- 172.21.15.102:0/3055332148 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fadfc131db0 con 0x7fadfc119350 2021-05-25T12:04:09.067 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.010+0000 7fadf1ffb700 10 client.184139.objecter ms_handle_connect 0x7faddc033250 2021-05-25T12:04:09.068 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.010+0000 7fadf1ffb700 1 -- 172.21.15.102:0/3055332148 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fadec004530 con 0x7fadfc119350 2021-05-25T12:04:09.068 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.010+0000 7fadf1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:09.068 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.010+0000 7fadf1ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:04:09.068 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.010+0000 7fadf1ffb700 10 monclient: dump: 2021-05-25T12:04:09.068 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:09.069 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:09.069 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:09.069 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:09.069 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:09.069 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:09.070 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:09.070 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:09.070 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:09.070 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:09.070 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.010+0000 7fadf1ffb700 1 -- 172.21.15.102:0/3055332148 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3460084740 0 0) 0x7fadfc131db0 con 0x7fadfc119350 2021-05-25T12:04:09.071 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.010+0000 7fadf1ffb700 1 -- 172.21.15.102:0/3055332148 --> v1:172.21.15.102:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fadfc138150 con 0x7fadfc119350 2021-05-25T12:04:09.071 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.010+0000 7fadf1ffb700 1 -- 172.21.15.102:0/3055332148 <== mon.0 v1:172.21.15.102:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7fadec0068a0 con 0x7fadfc119350 2021-05-25T12:04:09.071 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.010+0000 7fadf1ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:04:09.071 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.010+0000 7fadf1ffb700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:04:09.071 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.010+0000 7fadf1ffb700 1 -- 172.21.15.102:0/3055332148 >> v1:172.21.15.102:6789/0 conn(0x7fadfc119350 legacy=0x7fadfc066130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:09.072 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.010+0000 7fadf1ffb700 1 -- 172.21.15.102:0/3055332148 >> v1:172.21.15.102:6789/0 conn(0x7fadfc119350 legacy=0x7fadfc066130 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:04:09.072 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.016+0000 7fae03a68700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:04:09.072 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.016+0000 7fae03a68700 1 -- 172.21.15.102:0/3055332148 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fadfc119730 con 0x7fadfc027ce0 2021-05-25T12:04:09.072 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.017+0000 7fadf1ffb700 10 client.184139.objecter ms_handle_connect 0x7fadfc027ce0 2021-05-25T12:04:09.073 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.017+0000 7fadf1ffb700 10 client.184139.objecter resend_mon_ops 2021-05-25T12:04:09.073 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.017+0000 7fadf1ffb700 1 -- 172.21.15.102:0/3055332148 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3407204433 0 0) 0x7fadfc119730 con 0x7fadfc027ce0 2021-05-25T12:04:09.073 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.017+0000 7fadf1ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:04:09.073 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.017+0000 7fadf1ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:04:09.073 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.017+0000 7fadf1ffb700 10 monclient(hunting): my global_id is 174417 2021-05-25T12:04:09.074 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.017+0000 7fadf1ffb700 1 -- 172.21.15.102:0/3055332148 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadfc131db0 con 0x7fadfc027ce0 2021-05-25T12:04:09.074 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.017+0000 7fadf1ffb700 1 -- 172.21.15.102:0/3055332148 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3993580368 0 0) 0x7fadfc131db0 con 0x7fadfc027ce0 2021-05-25T12:04:09.074 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.017+0000 7fadf1ffb700 1 -- 172.21.15.102:0/3055332148 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fadfc119730 con 0x7fadfc027ce0 2021-05-25T12:04:09.074 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.017+0000 7fadf1ffb700 1 -- 172.21.15.102:0/3055332148 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fadec004dc0 con 0x7fadfc027ce0 2021-05-25T12:04:09.074 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.017+0000 7fadf1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:09.075 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.017+0000 7fadf1ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:04:09.075 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.017+0000 7fadf1ffb700 10 monclient: dump: 2021-05-25T12:04:09.075 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:09.075 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:09.075 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:09.076 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:09.076 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:09.076 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:09.076 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:09.077 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:09.077 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:09.077 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:09.077 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.017+0000 7fadf1ffb700 1 -- 172.21.15.102:0/3055332148 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3380895382 0 0) 0x7fadfc119730 con 0x7fadfc027ce0 2021-05-25T12:04:09.077 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.017+0000 7fadf1ffb700 1 -- 172.21.15.102:0/3055332148 --> v1:172.21.15.102:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fadfc059730 con 0x7fadfc027ce0 2021-05-25T12:04:09.078 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.017+0000 7fadf1ffb700 1 -- 172.21.15.102:0/3055332148 <== mon.0 v1:172.21.15.102:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4110 (unknown 0 0 2525850441) 0x7fadec006ab0 con 0x7fadfc027ce0 2021-05-25T12:04:09.078 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.017+0000 7fadf1ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:04:09.078 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.017+0000 7fadf1ffb700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:04:09.078 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.017+0000 7fadf1ffb700 1 -- 172.21.15.102:0/3055332148 >> v1:172.21.15.102:6789/0 conn(0x7fadfc027ce0 legacy=0x7fadfc066840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:09.078 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.017+0000 7fadf1ffb700 1 -- 172.21.15.102:0/3055332148 >> v1:172.21.15.102:6789/0 conn(0x7fadfc027ce0 legacy=0x7fadfc066840 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:04:09.079 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:04:09.079 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:04:09.079 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:04:09.080 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "peon", 2021-05-25T12:04:09.080 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 38, 2021-05-25T12:04:09.080 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:04:09.080 INFO:teuthology.orchestra.run.smithi102.stdout: 0, 2021-05-25T12:04:09.080 INFO:teuthology.orchestra.run.smithi102.stdout: 1, 2021-05-25T12:04:09.081 INFO:teuthology.orchestra.run.smithi102.stdout: 2 2021-05-25T12:04:09.081 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:04:09.081 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 1, 2021-05-25T12:04:09.081 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:04:09.081 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:04:09.082 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:04:09.082 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:04:09.082 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:04:09.082 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:04:09.083 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:04:09.083 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:04:09.083 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:04:09.083 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:04:09.083 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:04:09.084 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:04:09.084 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:04:09.084 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:04:09.084 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:04:09.084 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:04:09.085 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:04:09.085 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:04:09.085 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:04:09.085 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:04:09.085 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:04:09.086 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:04:09.086 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:04:09.086 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:04:09.086 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:04:09.087 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:04:09.087 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:04:09.087 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:04:09.087 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:04:09.087 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:04:09.088 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:04:09.088 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:04:09.088 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:04:09.088 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:04:09.088 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:04:09.089 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:04:09.089 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:04:09.089 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:04:09.089 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:04:09.089 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:04:09.090 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:04:09.090 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:04:09.090 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:04:09.090 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:04:09.091 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:04:09.091 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:04:09.091 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:04:09.091 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:04:09.091 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:04:09.092 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:04:09.092 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:04:09.092 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:04:09.092 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:04:09.092 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:04:09.093 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:04:09.093 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:04:09.093 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:04:09.093 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:04:09.093 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:04:09.094 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:04:09.094 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:04:09.094 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:04:09.094 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:04:09.094 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:04:09.095 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:04:09.095 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:04:09.095 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:04:09.095 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:04:09.095 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:04:09.096 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:04:09.096 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:04:09.096 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:04:09.096 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:04:09.096 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:04:09.097 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:04:09.097 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:04:09.097 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:04:09.097 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:04:09.097 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:04:09.098 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:04:09.098 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:04:09.098 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:04:09.098 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:04:09.099 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:04:09.099 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:04:09.099 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:04:09.099 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:04:09.100 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:04:09.100 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:04:09.100 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:04:09.100 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:04:09.100 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:04:09.101 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:04:09.101 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:04:09.101 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:04:09.101 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:04:09.101 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:04:09.102 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:04:09.102 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:04:09.102 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:04:09.102 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:04:09.103 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:04:09.103 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:04:09.103 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:04:09.103 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:04:09.103 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:04:09.104 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:04:09.104 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:04:09.104 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:04:09.104 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:04:09.104 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:04:09.105 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:04:09.105 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:04:09.105 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:04:09.105 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:04:09.105 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:04:09.106 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:04:09.106 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:04:09.106 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": [ 2021-05-25T12:04:09.106 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:04:09.106 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:04:09.107 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:04:09.107 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 3 2021-05-25T12:04:09.107 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:04:09.107 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:04:09.107 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:04:09.108 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:04:09.108 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:04:09.108 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:04:09.108 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 4 2021-05-25T12:04:09.108 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:04:09.109 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:04:09.109 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": [ 2021-05-25T12:04:09.109 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:04:09.109 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:04:09.109 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:04:09.110 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:04:09.110 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:04:09.110 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:04:09.110 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:04:09.111 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:04:09.111 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:04:09.113 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.018+0000 7fae03a68700 10 librados: watch_flush enter 2021-05-25T12:04:09.113 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.018+0000 7fae03a68700 10 librados: watch_flush exit 2021-05-25T12:04:09.114 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.018+0000 7fae03a68700 20 client.184139.objecter shutdown clearing up homeless session... 2021-05-25T12:04:09.114 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.018+0000 7fae03a68700 10 client.184139.objecter successfully canceled tick 2021-05-25T12:04:09.114 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.018+0000 7fae03a68700 1 -- 172.21.15.102:0/3055332148 >> v1:172.21.15.181:6812/29582 conn(0x7faddc033250 legacy=0x7faddc035710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:09.114 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.018+0000 7fae03a68700 10 monclient: shutdown 2021-05-25T12:04:09.115 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.018+0000 7fae03a68700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:04:09.115 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.018+0000 7fae03a68700 1 -- 172.21.15.102:0/3055332148 >> v1:172.21.15.102:6790/0 conn(0x7fadfc132010 legacy=0x7fadfc143540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:09.115 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.018+0000 7fae02005700 1 -- 172.21.15.102:0/3055332148 reap_dead start 2021-05-25T12:04:09.115 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.018+0000 7fae02005700 1 -- 172.21.15.102:0/3055332148 reap_dead start 2021-05-25T12:04:09.115 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.018+0000 7fae03a68700 1 -- 172.21.15.102:0/3055332148 shutdown_connections 2021-05-25T12:04:09.116 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.018+0000 7fae03a68700 1 -- 172.21.15.102:0/3055332148 >> 172.21.15.102:0/3055332148 conn(0x7fadfc09cd10 msgr2=0x7fadfc0929a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:04:09.116 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.018+0000 7fae03a68700 1 -- 172.21.15.102:0/3055332148 shutdown_connections 2021-05-25T12:04:09.116 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.018+0000 7fae03a68700 1 -- 172.21.15.102:0/3055332148 wait complete. 2021-05-25T12:04:09.116 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.018+0000 7fae03a68700 1 librados: shutdown 2021-05-25T12:04:09.117 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2021-05-25T12:04:09.227 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.225+0000 7f031751a700 10 monclient: get_monmap_and_config 2021-05-25T12:04:09.227 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.225+0000 7f031751a700 10 monclient: build_initial_monmap 2021-05-25T12:04:09.227 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.225+0000 7f031751a700 10 monclient: monmap: 2021-05-25T12:04:09.228 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:04:09.228 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:09.228 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:04:09.226194+0000 2021-05-25T12:04:09.228 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:04:09.226194+0000 2021-05-25T12:04:09.228 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:04:09.229 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:09.229 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:04:09.229 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:04:09.229 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:04:09.230 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:09.230 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.225+0000 7f031751a700 1 Processor -- start 2021-05-25T12:04:09.230 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.225+0000 7f031751a700 1 -- start start 2021-05-25T12:04:09.231 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.225+0000 7f031751a700 10 monclient: init 2021-05-25T12:04:09.244 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.226+0000 7f031751a700 10 monclient: _reopen_session rank -1 2021-05-25T12:04:09.245 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.226+0000 7f031751a700 10 monclient: _add_conns ranks=[1,2,0] 2021-05-25T12:04:09.245 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.226+0000 7f031751a700 10 monclient(hunting): picked mon.noname-c con 0x7f0310090990 addr v1:172.21.15.181:6789/0 2021-05-25T12:04:09.245 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.226+0000 7f031751a700 10 monclient(hunting): picked mon.noname-b con 0x7f0310090d70 addr v1:172.21.15.102:6790/0 2021-05-25T12:04:09.245 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.226+0000 7f031751a700 10 monclient(hunting): picked mon.noname-a con 0x7f031012e330 addr v1:172.21.15.102:6789/0 2021-05-25T12:04:09.246 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.226+0000 7f031751a700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0310096fa0 con 0x7f031012e330 2021-05-25T12:04:09.246 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.226+0000 7f031751a700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03100926a0 con 0x7f0310090990 2021-05-25T12:04:09.246 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.226+0000 7f031751a700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0310092940 con 0x7f0310090d70 2021-05-25T12:04:09.247 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.226+0000 7f031751a700 10 monclient(hunting): _renew_subs 2021-05-25T12:04:09.247 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.226+0000 7f031751a700 10 monclient(hunting): authenticate will time out at 1049.823438s 2021-05-25T12:04:09.247 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.226+0000 7f0315ab7700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f031012e330 0x7f0310130720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60656/0 (socket says 172.21.15.102:60656) 2021-05-25T12:04:09.247 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.226+0000 7f0315ab7700 1 -- 172.21.15.102:0/411723976 learned_addr learned my addr 172.21.15.102:0/411723976 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:04:09.248 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.226+0000 7f0314ab5700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f0310090d70 0x7f0310091200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:55532/0 (socket says 172.21.15.102:55532) 2021-05-25T12:04:09.248 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.226+0000 7f02fffff700 1 -- 172.21.15.102:0/411723976 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2235628289 0 0) 0x7f0310096fa0 con 0x7f031012e330 2021-05-25T12:04:09.248 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.226+0000 7f02fffff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:04:09.248 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.226+0000 7f02fffff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:04:09.249 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.226+0000 7f02fffff700 10 monclient(hunting): my global_id is 174420 2021-05-25T12:04:09.249 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.226+0000 7f02fffff700 1 -- 172.21.15.102:0/411723976 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02f8003270 con 0x7f031012e330 2021-05-25T12:04:09.249 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.226+0000 7f02fffff700 1 -- 172.21.15.102:0/411723976 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1840561002 0 0) 0x7f0310092940 con 0x7f0310090d70 2021-05-25T12:04:09.249 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.226+0000 7f02fffff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:04:09.249 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.226+0000 7f02fffff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:04:09.250 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.226+0000 7f02fffff700 10 monclient(hunting): my global_id is 184142 2021-05-25T12:04:09.250 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.226+0000 7f02fffff700 1 -- 172.21.15.102:0/411723976 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0310096fa0 con 0x7f0310090d70 2021-05-25T12:04:09.250 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.227+0000 7f02fffff700 1 -- 172.21.15.102:0/411723976 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2379205831 0 0) 0x7f02f8003270 con 0x7f031012e330 2021-05-25T12:04:09.250 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.227+0000 7f02fffff700 1 -- 172.21.15.102:0/411723976 >> v1:172.21.15.102:6790/0 conn(0x7f0310090d70 legacy=0x7f0310091200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:09.251 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.227+0000 7f02fffff700 1 -- 172.21.15.102:0/411723976 >> v1:172.21.15.181:6789/0 conn(0x7f0310090990 legacy=0x7f0310093240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:09.251 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.227+0000 7f02fffff700 10 monclient: _finish_hunting 0 2021-05-25T12:04:09.251 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.227+0000 7f02fffff700 1 monclient: found mon.noname-a 2021-05-25T12:04:09.251 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.227+0000 7f02fffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:04:09.252 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.227+0000 7f02fffff700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T12:04:09.252 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.227+0000 7f02fffff700 1 -- 172.21.15.102:0/411723976 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0310092be0 con 0x7f031012e330 2021-05-25T12:04:09.252 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.227+0000 7f02fffff700 10 monclient: _finish_auth 0 2021-05-25T12:04:09.252 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.227+0000 7f02fffff700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:04:09.253 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.227+0000 7f02fffff700 1 -- 172.21.15.102:0/411723976 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f030c001990 con 0x7f031012e330 2021-05-25T12:04:09.253 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.227+0000 7f031751a700 5 monclient: authenticate success, global_id 174420 2021-05-25T12:04:09.253 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.227+0000 7f031751a700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:04:09.253 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.227+0000 7f02fffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:09.254 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.227+0000 7f02fffff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:04:09.254 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.227+0000 7f02fffff700 10 monclient: dump: 2021-05-25T12:04:09.254 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:09.254 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:09.255 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:09.255 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:09.255 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:09.255 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:09.255 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:09.256 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:09.256 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:09.256 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:09.257 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.227+0000 7f031751a700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:04:09.257 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.227+0000 7f02fffff700 1 -- 172.21.15.102:0/411723976 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f030c001b40 con 0x7f031012e330 2021-05-25T12:04:09.257 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.227+0000 7f02fffff700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:04:09.257 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.227+0000 7f031751a700 10 monclient: get_monmap_and_config success 2021-05-25T12:04:09.258 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.227+0000 7f02fffff700 1 -- 172.21.15.102:0/411723976 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f030c001ed0 con 0x7f031012e330 2021-05-25T12:04:09.258 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.227+0000 7f031751a700 4 set_mon_vals no callback set 2021-05-25T12:04:09.258 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.227+0000 7f02fffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:09.258 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.227+0000 7f02fffff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:04:09.259 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.227+0000 7f02fffff700 10 monclient: dump: 2021-05-25T12:04:09.259 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:09.259 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:09.259 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:09.259 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:09.260 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:09.260 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:09.260 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:09.260 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:09.261 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:09.261 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:09.261 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.227+0000 7f031751a700 10 monclient: shutdown 2021-05-25T12:04:09.261 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.227+0000 7f031751a700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:04:09.262 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.227+0000 7f031751a700 1 -- 172.21.15.102:0/411723976 >> v1:172.21.15.102:6789/0 conn(0x7f031012e330 legacy=0x7f0310130720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:09.262 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.227+0000 7f031751a700 1 -- 172.21.15.102:0/411723976 shutdown_connections 2021-05-25T12:04:09.262 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.227+0000 7f031751a700 1 -- 172.21.15.102:0/411723976 >> 172.21.15.102:0/411723976 conn(0x7f031009b370 msgr2=0x7f0310097430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:04:09.262 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.227+0000 7f031751a700 1 -- 172.21.15.102:0/411723976 shutdown_connections 2021-05-25T12:04:09.262 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.227+0000 7f031751a700 1 -- 172.21.15.102:0/411723976 wait complete. 2021-05-25T12:04:09.263 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.228+0000 7f031751a700 10 monclient: build_initial_monmap 2021-05-25T12:04:09.263 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.228+0000 7f031751a700 10 monclient: monmap: 2021-05-25T12:04:09.263 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:04:09.263 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:04:09.264 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:04:09.264 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:04:09.264 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:04:09.264 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:04:09.264 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:04:09.265 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:04:09.265 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:04:09.265 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:09.265 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.228+0000 7f031751a700 1 librados: starting msgr at 2021-05-25T12:04:09.266 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.228+0000 7f031751a700 1 librados: starting objecter 2021-05-25T12:04:09.266 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.228+0000 7f031751a700 1 Processor -- start 2021-05-25T12:04:09.266 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.228+0000 7f031751a700 1 -- start start 2021-05-25T12:04:09.266 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.228+0000 7f031751a700 1 librados: setting wanted keys 2021-05-25T12:04:09.267 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.228+0000 7f031751a700 1 librados: calling monclient init 2021-05-25T12:04:09.267 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.228+0000 7f031751a700 10 monclient: init 2021-05-25T12:04:09.268 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.229+0000 7f031751a700 10 monclient: _reopen_session rank -1 2021-05-25T12:04:09.268 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.229+0000 7f031751a700 10 monclient: _add_conns ranks=[2,1,0] 2021-05-25T12:04:09.268 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.229+0000 7f031751a700 10 monclient(hunting): picked mon.noname-c con 0x7f0310090990 addr v1:172.21.15.102:6790/0 2021-05-25T12:04:09.268 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.229+0000 7f031751a700 10 monclient(hunting): picked mon.noname-b con 0x7f0310090d70 addr v1:172.21.15.181:6789/0 2021-05-25T12:04:09.269 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.229+0000 7f031751a700 10 monclient(hunting): picked mon.noname-a con 0x7f031012e330 addr v1:172.21.15.102:6789/0 2021-05-25T12:04:09.269 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.229+0000 7f031751a700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0310091fe0 con 0x7f031012e330 2021-05-25T12:04:09.269 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.229+0000 7f031751a700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03100686a0 con 0x7f0310090d70 2021-05-25T12:04:09.269 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.229+0000 7f031751a700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0310068940 con 0x7f0310090990 2021-05-25T12:04:09.270 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.229+0000 7f031751a700 10 monclient(hunting): _renew_subs 2021-05-25T12:04:09.270 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.229+0000 7f031751a700 10 monclient(hunting): authenticate will time out at 1049.826137s 2021-05-25T12:04:09.270 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.229+0000 7f03152b6700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f0310090990 0x7f0310064c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:55538/0 (socket says 172.21.15.102:55538) 2021-05-25T12:04:09.270 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.229+0000 7f0315ab7700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f031012e330 0x7f0310065a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60664/0 (socket says 172.21.15.102:60664) 2021-05-25T12:04:09.270 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.229+0000 7f03152b6700 1 -- 172.21.15.102:0/15608331 learned_addr learned my addr 172.21.15.102:0/15608331 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:04:09.271 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.229+0000 7f02fd7fa700 10 client.?.objecter ms_handle_connect 0x7f0310090990 2021-05-25T12:04:09.271 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.229+0000 7f02fd7fa700 10 client.?.objecter resend_mon_ops 2021-05-25T12:04:09.271 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.229+0000 7f02fd7fa700 10 client.?.objecter ms_handle_connect 0x7f031012e330 2021-05-25T12:04:09.271 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.229+0000 7f02fd7fa700 10 client.?.objecter resend_mon_ops 2021-05-25T12:04:09.272 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.229+0000 7f02fd7fa700 1 -- 172.21.15.102:0/15608331 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 346547179 0 0) 0x7f0310068940 con 0x7f0310090990 2021-05-25T12:04:09.272 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.229+0000 7f02fd7fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:04:09.272 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.229+0000 7f02fd7fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:04:09.272 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.229+0000 7f02fd7fa700 10 monclient(hunting): my global_id is 184145 2021-05-25T12:04:09.272 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.229+0000 7f02fd7fa700 1 -- 172.21.15.102:0/15608331 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02f0004c10 con 0x7f0310090990 2021-05-25T12:04:09.273 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.229+0000 7f02fd7fa700 1 -- 172.21.15.102:0/15608331 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2052062095 0 0) 0x7f0310091fe0 con 0x7f031012e330 2021-05-25T12:04:09.273 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.229+0000 7f02fd7fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:04:09.273 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.229+0000 7f02fd7fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:04:09.274 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.229+0000 7f02fd7fa700 10 monclient(hunting): my global_id is 174423 2021-05-25T12:04:09.274 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.229+0000 7f02fd7fa700 1 -- 172.21.15.102:0/15608331 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0310068940 con 0x7f031012e330 2021-05-25T12:04:09.275 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.229+0000 7f02fd7fa700 10 client.?.objecter ms_handle_connect 0x7f0310090d70 2021-05-25T12:04:09.275 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.229+0000 7f02fd7fa700 10 client.?.objecter resend_mon_ops 2021-05-25T12:04:09.275 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.229+0000 7f02fd7fa700 1 -- 172.21.15.102:0/15608331 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1424599760 0 0) 0x7f03100686a0 con 0x7f0310090d70 2021-05-25T12:04:09.275 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.229+0000 7f02fd7fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:04:09.276 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.229+0000 7f02fd7fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:04:09.276 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.229+0000 7f02fd7fa700 10 monclient(hunting): my global_id is 95338 2021-05-25T12:04:09.276 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.229+0000 7f02fd7fa700 1 -- 172.21.15.102:0/15608331 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0310091fe0 con 0x7f0310090d70 2021-05-25T12:04:09.276 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.229+0000 7f02fd7fa700 1 -- 172.21.15.102:0/15608331 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2529180546 0 0) 0x7f02f0004c10 con 0x7f0310090990 2021-05-25T12:04:09.276 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.229+0000 7f02fd7fa700 1 -- 172.21.15.102:0/15608331 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f03100686a0 con 0x7f0310090990 2021-05-25T12:04:09.277 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.229+0000 7f02fd7fa700 1 -- 172.21.15.102:0/15608331 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3861803973 0 0) 0x7f0310068940 con 0x7f031012e330 2021-05-25T12:04:09.277 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.229+0000 7f02fd7fa700 1 -- 172.21.15.102:0/15608331 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f02f0004c10 con 0x7f031012e330 2021-05-25T12:04:09.277 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.229+0000 7f02fd7fa700 1 -- 172.21.15.102:0/15608331 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f03000019a0 con 0x7f0310090990 2021-05-25T12:04:09.277 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.229+0000 7f02fd7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:09.277 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.229+0000 7f02fd7fa700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:04:09.278 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.229+0000 7f02fd7fa700 10 monclient(hunting): dump: 2021-05-25T12:04:09.278 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:09.278 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:09.278 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:09.279 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:09.279 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:09.279 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:09.279 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:09.279 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:09.280 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:09.280 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:09.280 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.229+0000 7f02fd7fa700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:04:09.280 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.229+0000 7f02fd7fa700 1 -- 172.21.15.102:0/15608331 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f030c001e50 con 0x7f031012e330 2021-05-25T12:04:09.281 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.229+0000 7f02fd7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:09.281 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.229+0000 7f02fd7fa700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2021-05-25T12:04:09.281 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.229+0000 7f02fd7fa700 10 monclient(hunting): dump: 2021-05-25T12:04:09.281 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:09.281 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:09.282 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:09.282 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:09.282 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:09.282 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:09.282 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:09.283 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:09.283 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:09.283 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:09.283 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.229+0000 7f02fd7fa700 1 -- 172.21.15.102:0/15608331 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 966960198 0 0) 0x7f03100686a0 con 0x7f0310090990 2021-05-25T12:04:09.283 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.229+0000 7f02fd7fa700 1 -- 172.21.15.102:0/15608331 >> v1:172.21.15.181:6789/0 conn(0x7f0310090d70 legacy=0x7f0310065330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:09.284 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.229+0000 7f02fd7fa700 1 -- 172.21.15.102:0/15608331 >> v1:172.21.15.102:6789/0 conn(0x7f031012e330 legacy=0x7f0310065a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:09.284 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.229+0000 7f02fd7fa700 10 monclient: _finish_hunting 0 2021-05-25T12:04:09.284 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.229+0000 7f02fd7fa700 1 monclient: found mon.c 2021-05-25T12:04:09.284 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.229+0000 7f02fd7fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:04:09.285 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.229+0000 7f02fd7fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:04:09.285 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.229+0000 7f02fd7fa700 1 -- 172.21.15.102:0/15608331 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03100672b0 con 0x7f0310090990 2021-05-25T12:04:09.285 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.229+0000 7f02fd7fa700 10 monclient: _finish_auth 0 2021-05-25T12:04:09.285 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.229+0000 7f02fd7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:04:09.285 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.229+0000 7f031751a700 5 monclient: authenticate success, global_id 184145 2021-05-25T12:04:09.286 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.229+0000 7f031751a700 10 monclient: _renew_subs 2021-05-25T12:04:09.286 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.229+0000 7f031751a700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:04:09.286 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.229+0000 7f031751a700 1 -- 172.21.15.102:0/15608331 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0310066830 con 0x7f0310090990 2021-05-25T12:04:09.286 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.229+0000 7f031751a700 10 client.184145.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:04:09.287 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.229+0000 7f031751a700 10 monclient: _renew_subs 2021-05-25T12:04:09.287 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.229+0000 7f031751a700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:04:09.287 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.229+0000 7f031751a700 1 -- 172.21.15.102:0/15608331 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0310114ed0 con 0x7f0310090990 2021-05-25T12:04:09.287 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.229+0000 7f031751a700 1 librados: init done 2021-05-25T12:04:09.288 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.230+0000 7f02fd7fa700 1 -- 172.21.15.102:0/15608331 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0300001ce0 con 0x7f0310090990 2021-05-25T12:04:09.288 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.230+0000 7f02fd7fa700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:04:09.289 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.230+0000 7f02fd7fa700 1 -- 172.21.15.102:0/15608331 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f03000023d0 con 0x7f0310090990 2021-05-25T12:04:09.289 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.230+0000 7f02fd7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:09.289 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.230+0000 7f02fd7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:04:09.289 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.230+0000 7f02ff7fe700 4 set_mon_vals no callback set 2021-05-25T12:04:09.290 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.230+0000 7f02fd7fa700 10 monclient: dump: 2021-05-25T12:04:09.290 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:09.290 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:09.290 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:09.291 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:09.291 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:09.291 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:09.291 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:09.291 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:09.295 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:09.296 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:09.296 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.230+0000 7f02fd7fa700 1 -- 172.21.15.102:0/15608331 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f03000025b0 con 0x7f0310090990 2021-05-25T12:04:09.297 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.230+0000 7f031751a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:04:09.297 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.230+0000 7f031751a700 1 -- 172.21.15.102:0/15608331 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f02e80065a0 con 0x7f02e8002840 2021-05-25T12:04:09.298 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.230+0000 7f02fd7fa700 1 -- 172.21.15.102:0/15608331 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(125..125 src has 1..125) v4 ==== 5802+0+0 (unknown 2341125478 0 0) 0x7f030003d6e0 con 0x7f0310090990 2021-05-25T12:04:09.298 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.230+0000 7f02fd7fa700 10 client.184145.objecter ms_dispatch 0x7f0310001550 osd_map(125..125 src has 1..125) v4 2021-05-25T12:04:09.298 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.230+0000 7f02fd7fa700 3 client.184145.objecter handle_osd_map got epochs [125,125] > 0 2021-05-25T12:04:09.299 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.230+0000 7f02fd7fa700 3 client.184145.objecter handle_osd_map decoding full epoch 125 2021-05-25T12:04:09.299 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.230+0000 7f02fd7fa700 20 client.184145.objecter dump_active .. 0 homeless 2021-05-25T12:04:09.300 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.230+0000 7f02fd7fa700 10 client.184145.objecter ms_handle_connect 0x7f02e8002840 2021-05-25T12:04:09.300 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.230+0000 7f02fd7fa700 10 client.184145.objecter resend_mon_ops 2021-05-25T12:04:09.301 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.230+0000 7f02fd7fa700 1 -- 172.21.15.102:0/15608331 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 712222262 0 0) 0x7f02e80065a0 con 0x7f02e8002840 2021-05-25T12:04:09.301 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.230+0000 7f02fd7fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:04:09.301 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.230+0000 7f02fd7fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:04:09.302 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.230+0000 7f02fd7fa700 10 monclient(hunting): my global_id is 184148 2021-05-25T12:04:09.302 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.230+0000 7f02fd7fa700 1 -- 172.21.15.102:0/15608331 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03100686a0 con 0x7f02e8002840 2021-05-25T12:04:09.303 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.230+0000 7f0315ab7700 10 monclient: get_auth_request con 0x7f02f0033190 auth_method 0 2021-05-25T12:04:09.303 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.231+0000 7f02fd7fa700 1 -- 172.21.15.102:0/15608331 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 735353710 0 0) 0x7f03100686a0 con 0x7f02e8002840 2021-05-25T12:04:09.304 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.231+0000 7f02fd7fa700 1 -- 172.21.15.102:0/15608331 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f02e80065a0 con 0x7f02e8002840 2021-05-25T12:04:09.304 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.231+0000 7f02fd7fa700 1 -- 172.21.15.102:0/15608331 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f03040046d0 con 0x7f02e8002840 2021-05-25T12:04:09.304 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.231+0000 7f02fd7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:09.305 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.231+0000 7f02fd7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:04:09.305 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.231+0000 7f02fd7fa700 10 monclient: dump: 2021-05-25T12:04:09.306 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:09.306 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:09.306 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:09.307 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:09.307 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:09.308 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:09.308 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:09.308 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:09.309 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:09.309 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:09.309 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.231+0000 7f02fd7fa700 10 client.184145.objecter ms_handle_connect 0x7f02f0033190 2021-05-25T12:04:09.310 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.231+0000 7f02fd7fa700 1 -- 172.21.15.102:0/15608331 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3367133436 0 0) 0x7f02e80065a0 con 0x7f02e8002840 2021-05-25T12:04:09.310 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.231+0000 7f02fd7fa700 1 -- 172.21.15.102:0/15608331 --> v1:172.21.15.102:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f02e8009e40 con 0x7f02e8002840 2021-05-25T12:04:09.311 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.231+0000 7f02fd7fa700 1 -- 172.21.15.102:0/15608331 <== mon.2 v1:172.21.15.102:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7f03040069a0 con 0x7f02e8002840 2021-05-25T12:04:09.311 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.231+0000 7f02fd7fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:04:09.311 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.231+0000 7f02fd7fa700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:04:09.312 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.231+0000 7f02fd7fa700 1 -- 172.21.15.102:0/15608331 >> v1:172.21.15.102:6790/0 conn(0x7f02e8002840 legacy=0x7f02e8004ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:09.312 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.231+0000 7f02fd7fa700 1 -- 172.21.15.102:0/15608331 >> v1:172.21.15.102:6790/0 conn(0x7f02e8002840 legacy=0x7f02e8004ce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:04:09.312 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.237+0000 7f031751a700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:04:09.313 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.238+0000 7f031751a700 1 -- 172.21.15.102:0/15608331 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f02e8007510 con 0x7f02e80068b0 2021-05-25T12:04:09.313 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.238+0000 7f02fd7fa700 10 client.184145.objecter ms_handle_connect 0x7f02e80068b0 2021-05-25T12:04:09.313 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.238+0000 7f02fd7fa700 10 client.184145.objecter resend_mon_ops 2021-05-25T12:04:09.314 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.238+0000 7f02fd7fa700 1 -- 172.21.15.102:0/15608331 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 226590941 0 0) 0x7f02e8007510 con 0x7f02e80068b0 2021-05-25T12:04:09.314 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.238+0000 7f02fd7fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:04:09.314 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.238+0000 7f02fd7fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:04:09.314 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.238+0000 7f02fd7fa700 10 monclient(hunting): my global_id is 184151 2021-05-25T12:04:09.315 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.238+0000 7f02fd7fa700 1 -- 172.21.15.102:0/15608331 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02e80065a0 con 0x7f02e80068b0 2021-05-25T12:04:09.315 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.238+0000 7f02fd7fa700 1 -- 172.21.15.102:0/15608331 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2831026341 0 0) 0x7f02e80065a0 con 0x7f02e80068b0 2021-05-25T12:04:09.315 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.238+0000 7f02fd7fa700 1 -- 172.21.15.102:0/15608331 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f02e8007510 con 0x7f02e80068b0 2021-05-25T12:04:09.315 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.238+0000 7f02fd7fa700 1 -- 172.21.15.102:0/15608331 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f0304004dc0 con 0x7f02e80068b0 2021-05-25T12:04:09.316 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.238+0000 7f02fd7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:09.316 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.238+0000 7f02fd7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:04:09.316 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.238+0000 7f02fd7fa700 10 monclient: dump: 2021-05-25T12:04:09.316 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:09.316 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:09.317 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:09.317 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:09.317 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:09.317 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:09.317 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:09.318 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:09.318 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:09.318 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:09.318 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.238+0000 7f02fd7fa700 1 -- 172.21.15.102:0/15608331 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 59936301 0 0) 0x7f02e8007510 con 0x7f02e80068b0 2021-05-25T12:04:09.319 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.238+0000 7f02fd7fa700 1 -- 172.21.15.102:0/15608331 --> v1:172.21.15.102:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f02e8008770 con 0x7f02e80068b0 2021-05-25T12:04:09.319 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.238+0000 7f02fd7fa700 1 -- 172.21.15.102:0/15608331 <== mon.2 v1:172.21.15.102:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3940 (unknown 0 0 204984563) 0x7f03040050e0 con 0x7f02e80068b0 2021-05-25T12:04:09.319 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.238+0000 7f02fd7fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:04:09.319 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.238+0000 7f02fd7fa700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:04:09.319 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.238+0000 7f02fd7fa700 1 -- 172.21.15.102:0/15608331 >> v1:172.21.15.102:6790/0 conn(0x7f02e80068b0 legacy=0x7f02e8006d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:09.320 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.238+0000 7f02fd7fa700 1 -- 172.21.15.102:0/15608331 >> v1:172.21.15.102:6790/0 conn(0x7f02e80068b0 legacy=0x7f02e8006d40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:04:09.320 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:04:09.320 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:04:09.321 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:04:09.321 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "peon", 2021-05-25T12:04:09.321 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 38, 2021-05-25T12:04:09.321 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:04:09.322 INFO:teuthology.orchestra.run.smithi102.stdout: 0, 2021-05-25T12:04:09.322 INFO:teuthology.orchestra.run.smithi102.stdout: 1, 2021-05-25T12:04:09.322 INFO:teuthology.orchestra.run.smithi102.stdout: 2 2021-05-25T12:04:09.322 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:04:09.323 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 1, 2021-05-25T12:04:09.323 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:04:09.323 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:04:09.323 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:04:09.323 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:04:09.324 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:04:09.324 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:04:09.324 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:04:09.324 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:04:09.324 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:04:09.325 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:04:09.325 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:04:09.325 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:04:09.325 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:04:09.325 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:04:09.326 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:04:09.326 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:04:09.326 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:04:09.326 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:04:09.327 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:04:09.328 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:04:09.328 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:04:09.328 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:04:09.331 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:04:09.331 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:04:09.331 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:04:09.332 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:04:09.333 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:04:09.333 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:04:09.333 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:04:09.333 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:04:09.334 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:04:09.334 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:04:09.334 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:04:09.334 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:04:09.335 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:04:09.335 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:04:09.335 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:04:09.335 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:04:09.335 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:04:09.336 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:04:09.336 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:04:09.336 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:04:09.336 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:04:09.337 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:04:09.337 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:04:09.337 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:04:09.338 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:04:09.338 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:04:09.339 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:04:09.339 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:04:09.340 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:04:09.340 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:04:09.340 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:04:09.341 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:04:09.341 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:04:09.342 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:04:09.342 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:04:09.342 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:04:09.343 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:04:09.343 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:04:09.344 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:04:09.344 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:04:09.344 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:04:09.345 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:04:09.345 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:04:09.346 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:04:09.346 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:04:09.347 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:04:09.347 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:04:09.347 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:04:09.348 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:04:09.348 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:04:09.349 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:04:09.349 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:04:09.349 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:04:09.350 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:04:09.350 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:04:09.351 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:04:09.351 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:04:09.351 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:04:09.352 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:04:09.352 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:04:09.353 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:04:09.353 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:04:09.353 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:04:09.354 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:04:09.354 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:04:09.354 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:04:09.355 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:04:09.355 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:04:09.355 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:04:09.355 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:04:09.356 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:04:09.356 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:04:09.356 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:04:09.357 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:04:09.357 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:04:09.357 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:04:09.357 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:04:09.358 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:04:09.358 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:04:09.358 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:04:09.359 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:04:09.359 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:04:09.359 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:04:09.360 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:04:09.360 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:04:09.360 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:04:09.361 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:04:09.361 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:04:09.361 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:04:09.361 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:04:09.362 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:04:09.362 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:04:09.362 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:04:09.363 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:04:09.363 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:04:09.363 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:04:09.364 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": [ 2021-05-25T12:04:09.364 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:04:09.365 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:04:09.365 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:04:09.365 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 3 2021-05-25T12:04:09.366 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:04:09.366 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:04:09.367 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:04:09.367 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:04:09.368 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:04:09.368 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:04:09.368 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 2 2021-05-25T12:04:09.369 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:04:09.369 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:04:09.370 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:04:09.370 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:04:09.371 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:04:09.374 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.239+0000 7f031751a700 10 librados: watch_flush enter 2021-05-25T12:04:09.375 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.239+0000 7f031751a700 10 librados: watch_flush exit 2021-05-25T12:04:09.375 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.239+0000 7f031751a700 20 client.184145.objecter shutdown clearing up homeless session... 2021-05-25T12:04:09.376 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.239+0000 7f031751a700 10 client.184145.objecter successfully canceled tick 2021-05-25T12:04:09.376 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.239+0000 7f031751a700 1 -- 172.21.15.102:0/15608331 >> v1:172.21.15.181:6812/29582 conn(0x7f02f0033190 legacy=0x7f02f0035650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:09.376 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.239+0000 7f031751a700 10 monclient: shutdown 2021-05-25T12:04:09.377 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.239+0000 7f031751a700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:04:09.377 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.239+0000 7f031751a700 1 -- 172.21.15.102:0/15608331 >> v1:172.21.15.102:6790/0 conn(0x7f0310090990 legacy=0x7f0310064c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:09.378 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.239+0000 7f0315ab7700 1 -- 172.21.15.102:0/15608331 reap_dead start 2021-05-25T12:04:09.378 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.239+0000 7f0315ab7700 1 -- 172.21.15.102:0/15608331 reap_dead start 2021-05-25T12:04:09.379 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.239+0000 7f031751a700 1 -- 172.21.15.102:0/15608331 shutdown_connections 2021-05-25T12:04:09.379 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.239+0000 7f031751a700 1 -- 172.21.15.102:0/15608331 >> 172.21.15.102:0/15608331 conn(0x7f031009b370 msgr2=0x7f0310129420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:04:09.379 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.239+0000 7f031751a700 1 -- 172.21.15.102:0/15608331 shutdown_connections 2021-05-25T12:04:09.380 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.239+0000 7f031751a700 1 -- 172.21.15.102:0/15608331 wait complete. 2021-05-25T12:04:09.380 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.239+0000 7f031751a700 1 librados: shutdown 2021-05-25T12:04:09.381 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2021-05-25T12:04:09.492 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.490+0000 7f0115ef9700 10 monclient: get_monmap_and_config 2021-05-25T12:04:09.492 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.490+0000 7f0115ef9700 10 monclient: build_initial_monmap 2021-05-25T12:04:09.493 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.490+0000 7f0115ef9700 10 monclient: monmap: 2021-05-25T12:04:09.493 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:04:09.494 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:09.494 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:04:09.491555+0000 2021-05-25T12:04:09.494 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:04:09.491555+0000 2021-05-25T12:04:09.495 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:04:09.495 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:09.496 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:04:09.496 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:04:09.496 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:04:09.497 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:09.516 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.491+0000 7f0115ef9700 1 Processor -- start 2021-05-25T12:04:09.517 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.491+0000 7f0115ef9700 1 -- start start 2021-05-25T12:04:09.517 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.491+0000 7f0115ef9700 10 monclient: init 2021-05-25T12:04:09.517 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.491+0000 7f0115ef9700 10 monclient: _reopen_session rank -1 2021-05-25T12:04:09.517 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.491+0000 7f0115ef9700 10 monclient: _add_conns ranks=[2,1,0] 2021-05-25T12:04:09.518 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.491+0000 7f0115ef9700 10 monclient(hunting): picked mon.noname-b con 0x7f0110068f80 addr v1:172.21.15.102:6790/0 2021-05-25T12:04:09.518 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.491+0000 7f0115ef9700 10 monclient(hunting): picked mon.noname-c con 0x7f0110126790 addr v1:172.21.15.181:6789/0 2021-05-25T12:04:09.518 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.491+0000 7f0115ef9700 10 monclient(hunting): picked mon.noname-a con 0x7f0110126c10 addr v1:172.21.15.102:6789/0 2021-05-25T12:04:09.518 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.491+0000 7f0115ef9700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0110094310 con 0x7f0110126c10 2021-05-25T12:04:09.519 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.491+0000 7f0115ef9700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01100945b0 con 0x7f0110126790 2021-05-25T12:04:09.519 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.491+0000 7f0115ef9700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0110094850 con 0x7f0110068f80 2021-05-25T12:04:09.519 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.491+0000 7f0115ef9700 10 monclient(hunting): _renew_subs 2021-05-25T12:04:09.519 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.491+0000 7f0115ef9700 10 monclient(hunting): authenticate will time out at 1050.088996s 2021-05-25T12:04:09.520 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.492+0000 7f010f7fe700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f0110068f80 0x7f0110069380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:55550/0 (socket says 172.21.15.102:55550) 2021-05-25T12:04:09.520 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.492+0000 7f010f7fe700 1 -- 172.21.15.102:0/2240354768 learned_addr learned my addr 172.21.15.102:0/2240354768 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:04:09.521 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.492+0000 7f010e7fc700 1 -- 172.21.15.102:0/2240354768 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3185437274 0 0) 0x7f0110094310 con 0x7f0110126c10 2021-05-25T12:04:09.521 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.492+0000 7f010e7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:04:09.522 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.492+0000 7f010e7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:04:09.522 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.492+0000 7f010e7fc700 10 monclient(hunting): my global_id is 174426 2021-05-25T12:04:09.522 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.492+0000 7f010e7fc700 1 -- 172.21.15.102:0/2240354768 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00f4003270 con 0x7f0110126c10 2021-05-25T12:04:09.523 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.492+0000 7f010e7fc700 1 -- 172.21.15.102:0/2240354768 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3042035116 0 0) 0x7f0110094850 con 0x7f0110068f80 2021-05-25T12:04:09.523 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.492+0000 7f010e7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:04:09.523 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.492+0000 7f010e7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:04:09.524 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.492+0000 7f010e7fc700 10 monclient(hunting): my global_id is 184154 2021-05-25T12:04:09.524 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.492+0000 7f010e7fc700 1 -- 172.21.15.102:0/2240354768 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0110094310 con 0x7f0110068f80 2021-05-25T12:04:09.524 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.492+0000 7f010e7fc700 1 -- 172.21.15.102:0/2240354768 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3918233100 0 0) 0x7f00f4003270 con 0x7f0110126c10 2021-05-25T12:04:09.525 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.492+0000 7f010e7fc700 1 -- 172.21.15.102:0/2240354768 >> v1:172.21.15.102:6790/0 conn(0x7f0110068f80 legacy=0x7f0110069380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:09.525 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.492+0000 7f010e7fc700 1 -- 172.21.15.102:0/2240354768 >> v1:172.21.15.181:6789/0 conn(0x7f0110126790 legacy=0x7f011012ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:09.525 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.492+0000 7f010e7fc700 10 monclient: _finish_hunting 0 2021-05-25T12:04:09.525 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.492+0000 7f010e7fc700 1 monclient: found mon.noname-a 2021-05-25T12:04:09.526 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.492+0000 7f010e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:04:09.526 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.492+0000 7f010e7fc700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T12:04:09.526 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.492+0000 7f010e7fc700 1 -- 172.21.15.102:0/2240354768 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0110094c10 con 0x7f0110126c10 2021-05-25T12:04:09.526 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.492+0000 7f010e7fc700 10 monclient: _finish_auth 0 2021-05-25T12:04:09.527 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.492+0000 7f010e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:04:09.527 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.492+0000 7f0115ef9700 5 monclient: authenticate success, global_id 174426 2021-05-25T12:04:09.527 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.492+0000 7f010e7fc700 1 -- 172.21.15.102:0/2240354768 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f0104001990 con 0x7f0110126c10 2021-05-25T12:04:09.527 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.492+0000 7f0115ef9700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:04:09.528 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.492+0000 7f010e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:09.528 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.492+0000 7f010e7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:04:09.528 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.492+0000 7f010e7fc700 10 monclient: dump: 2021-05-25T12:04:09.528 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:09.529 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:09.529 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:09.529 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:09.529 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:09.529 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:09.530 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:09.530 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:09.530 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:09.531 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:09.531 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.492+0000 7f0115ef9700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:04:09.531 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.492+0000 7f010e7fc700 1 -- 172.21.15.102:0/2240354768 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0104001c30 con 0x7f0110126c10 2021-05-25T12:04:09.531 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.492+0000 7f010e7fc700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:04:09.531 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.492+0000 7f0115ef9700 10 monclient: get_monmap_and_config success 2021-05-25T12:04:09.532 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.492+0000 7f010e7fc700 1 -- 172.21.15.102:0/2240354768 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f0104001fc0 con 0x7f0110126c10 2021-05-25T12:04:09.532 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.492+0000 7f0115ef9700 4 set_mon_vals no callback set 2021-05-25T12:04:09.532 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.492+0000 7f010e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:09.532 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.492+0000 7f010e7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:04:09.533 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.492+0000 7f010e7fc700 10 monclient: dump: 2021-05-25T12:04:09.533 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:09.533 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:09.533 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:09.533 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:09.534 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:09.534 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:09.534 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:09.534 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:09.535 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:09.535 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:09.535 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.493+0000 7f0115ef9700 10 monclient: shutdown 2021-05-25T12:04:09.535 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.493+0000 7f0115ef9700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:04:09.535 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.493+0000 7f0115ef9700 1 -- 172.21.15.102:0/2240354768 >> v1:172.21.15.102:6789/0 conn(0x7f0110126c10 legacy=0x7f01101321a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:09.536 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.493+0000 7f0115ef9700 1 -- 172.21.15.102:0/2240354768 shutdown_connections 2021-05-25T12:04:09.536 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.493+0000 7f0115ef9700 1 -- 172.21.15.102:0/2240354768 >> 172.21.15.102:0/2240354768 conn(0x7f011009b370 msgr2=0x7f0110097430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:04:09.536 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.493+0000 7f0115ef9700 1 -- 172.21.15.102:0/2240354768 shutdown_connections 2021-05-25T12:04:09.536 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.493+0000 7f0115ef9700 1 -- 172.21.15.102:0/2240354768 wait complete. 2021-05-25T12:04:09.536 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.493+0000 7f0115ef9700 10 monclient: build_initial_monmap 2021-05-25T12:04:09.537 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.493+0000 7f0115ef9700 10 monclient: monmap: 2021-05-25T12:04:09.537 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:04:09.537 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:04:09.537 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:04:09.537 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:04:09.538 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:04:09.538 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:04:09.538 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:04:09.538 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:04:09.539 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:04:09.539 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:09.539 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.493+0000 7f0115ef9700 1 librados: starting msgr at 2021-05-25T12:04:09.539 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.493+0000 7f0115ef9700 1 librados: starting objecter 2021-05-25T12:04:09.539 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.494+0000 7f0115ef9700 1 Processor -- start 2021-05-25T12:04:09.540 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.494+0000 7f0115ef9700 1 -- start start 2021-05-25T12:04:09.540 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.494+0000 7f0115ef9700 1 librados: setting wanted keys 2021-05-25T12:04:09.540 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.494+0000 7f0115ef9700 1 librados: calling monclient init 2021-05-25T12:04:09.540 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.494+0000 7f0115ef9700 10 monclient: init 2021-05-25T12:04:09.540 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.494+0000 7f0115ef9700 10 monclient: _reopen_session rank -1 2021-05-25T12:04:09.541 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.494+0000 7f0115ef9700 10 monclient: _add_conns ranks=[0,1,2] 2021-05-25T12:04:09.541 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.494+0000 7f0115ef9700 10 monclient(hunting): picked mon.noname-a con 0x7f0110068f80 addr v1:172.21.15.102:6789/0 2021-05-25T12:04:09.541 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.494+0000 7f0115ef9700 10 monclient(hunting): picked mon.noname-b con 0x7f0110126790 addr v1:172.21.15.181:6789/0 2021-05-25T12:04:09.541 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.494+0000 7f0115ef9700 10 monclient(hunting): picked mon.noname-c con 0x7f0110126c10 addr v1:172.21.15.102:6790/0 2021-05-25T12:04:09.542 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.494+0000 7f0115ef9700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0110123840 con 0x7f0110068f80 2021-05-25T12:04:09.542 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.494+0000 7f0115ef9700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0110123ae0 con 0x7f0110126790 2021-05-25T12:04:09.542 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.494+0000 7f0115ef9700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0110123d80 con 0x7f0110126c10 2021-05-25T12:04:09.542 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.494+0000 7f0115ef9700 10 monclient(hunting): _renew_subs 2021-05-25T12:04:09.542 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.494+0000 7f0115ef9700 10 monclient(hunting): authenticate will time out at 1050.091676s 2021-05-25T12:04:09.543 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.494+0000 7f010f7fe700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f0110068f80 0x7f01101245a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60678/0 (socket says 172.21.15.102:60678) 2021-05-25T12:04:09.543 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.494+0000 7f010f7fe700 1 -- 172.21.15.102:0/3853891921 learned_addr learned my addr 172.21.15.102:0/3853891921 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:04:09.543 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.494+0000 7f00fb7fe700 10 client.?.objecter ms_handle_connect 0x7f0110068f80 2021-05-25T12:04:09.543 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.494+0000 7f00fb7fe700 10 client.?.objecter resend_mon_ops 2021-05-25T12:04:09.543 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.494+0000 7f00fb7fe700 10 client.?.objecter ms_handle_connect 0x7f0110126c10 2021-05-25T12:04:09.544 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.494+0000 7f00fb7fe700 10 client.?.objecter resend_mon_ops 2021-05-25T12:04:09.544 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.494+0000 7f00fb7fe700 10 client.?.objecter ms_handle_connect 0x7f0110126790 2021-05-25T12:04:09.544 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.494+0000 7f00fb7fe700 10 client.?.objecter resend_mon_ops 2021-05-25T12:04:09.544 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.494+0000 7f00fb7fe700 1 -- 172.21.15.102:0/3853891921 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 44361906 0 0) 0x7f0110123d80 con 0x7f0110126c10 2021-05-25T12:04:09.544 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.494+0000 7f00fb7fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:04:09.545 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.494+0000 7f00fb7fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:04:09.545 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.494+0000 7f00fb7fe700 10 monclient(hunting): my global_id is 184157 2021-05-25T12:04:09.545 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.494+0000 7f00fb7fe700 1 -- 172.21.15.102:0/3853891921 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00f0004c10 con 0x7f0110126c10 2021-05-25T12:04:09.545 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.494+0000 7f00fb7fe700 1 -- 172.21.15.102:0/3853891921 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1549496358 0 0) 0x7f0110123840 con 0x7f0110068f80 2021-05-25T12:04:09.546 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.494+0000 7f00fb7fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:04:09.546 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.494+0000 7f00fb7fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:04:09.546 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.494+0000 7f00fb7fe700 10 monclient(hunting): my global_id is 174429 2021-05-25T12:04:09.546 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.494+0000 7f00fb7fe700 1 -- 172.21.15.102:0/3853891921 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0110123d80 con 0x7f0110068f80 2021-05-25T12:04:09.546 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.495+0000 7f00fb7fe700 1 -- 172.21.15.102:0/3853891921 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1975951345 0 0) 0x7f0110123ae0 con 0x7f0110126790 2021-05-25T12:04:09.547 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.495+0000 7f00fb7fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:04:09.547 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.495+0000 7f00fb7fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:04:09.547 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.495+0000 7f00fb7fe700 10 monclient(hunting): my global_id is 95344 2021-05-25T12:04:09.547 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.495+0000 7f00fb7fe700 1 -- 172.21.15.102:0/3853891921 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0110123840 con 0x7f0110126790 2021-05-25T12:04:09.547 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.495+0000 7f00fb7fe700 1 -- 172.21.15.102:0/3853891921 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3862563598 0 0) 0x7f00f0004c10 con 0x7f0110126c10 2021-05-25T12:04:09.548 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.495+0000 7f00fb7fe700 1 -- 172.21.15.102:0/3853891921 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0110123ae0 con 0x7f0110126c10 2021-05-25T12:04:09.548 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.495+0000 7f00fb7fe700 1 -- 172.21.15.102:0/3853891921 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f0104001da0 con 0x7f0110126c10 2021-05-25T12:04:09.548 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.495+0000 7f00fb7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:09.548 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.495+0000 7f00fb7fe700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:04:09.548 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.495+0000 7f00fb7fe700 10 monclient(hunting): dump: 2021-05-25T12:04:09.549 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:09.549 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:09.549 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:09.549 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:09.550 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:09.550 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:09.550 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:09.550 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:09.550 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:09.551 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:09.551 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.495+0000 7f00fb7fe700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:04:09.551 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.495+0000 7f00fb7fe700 1 -- 172.21.15.102:0/3853891921 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3708182750 0 0) 0x7f0110123d80 con 0x7f0110068f80 2021-05-25T12:04:09.551 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.495+0000 7f00fb7fe700 1 -- 172.21.15.102:0/3853891921 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f00f0004c10 con 0x7f0110068f80 2021-05-25T12:04:09.551 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.495+0000 7f00fb7fe700 1 -- 172.21.15.102:0/3853891921 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f00fc0019f0 con 0x7f0110068f80 2021-05-25T12:04:09.552 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.495+0000 7f00fb7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:09.553 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.495+0000 7f00fb7fe700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2021-05-25T12:04:09.553 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.495+0000 7f00fb7fe700 10 monclient(hunting): dump: 2021-05-25T12:04:09.553 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:09.553 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:09.553 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:09.554 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:09.554 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:09.554 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:09.554 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:09.555 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:09.555 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:09.555 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:09.555 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.495+0000 7f00fb7fe700 1 -- 172.21.15.102:0/3853891921 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1758810266 0 0) 0x7f0110123840 con 0x7f0110126790 2021-05-25T12:04:09.556 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.495+0000 7f00fb7fe700 1 -- 172.21.15.102:0/3853891921 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0110123d80 con 0x7f0110126790 2021-05-25T12:04:09.556 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.495+0000 7f00fb7fe700 1 -- 172.21.15.102:0/3853891921 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f01000018c0 con 0x7f0110126790 2021-05-25T12:04:09.556 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.495+0000 7f00fb7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:09.556 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.495+0000 7f00fb7fe700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2021-05-25T12:04:09.556 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.495+0000 7f00fb7fe700 10 monclient(hunting): dump: 2021-05-25T12:04:09.557 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:09.557 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:09.557 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:09.557 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:09.557 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:09.558 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:09.558 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:09.558 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:09.558 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:09.559 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:09.559 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.495+0000 7f00fb7fe700 1 -- 172.21.15.102:0/3853891921 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 743173247 0 0) 0x7f0110123ae0 con 0x7f0110126c10 2021-05-25T12:04:09.559 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.495+0000 7f00fb7fe700 1 -- 172.21.15.102:0/3853891921 >> v1:172.21.15.181:6789/0 conn(0x7f0110126790 legacy=0x7f0110132d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:09.559 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.495+0000 7f00fb7fe700 1 -- 172.21.15.102:0/3853891921 >> v1:172.21.15.102:6789/0 conn(0x7f0110068f80 legacy=0x7f01101245a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:09.559 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.495+0000 7f00fb7fe700 10 monclient: _finish_hunting 0 2021-05-25T12:04:09.560 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.495+0000 7f00fb7fe700 1 monclient: found mon.c 2021-05-25T12:04:09.560 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.495+0000 7f00fb7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:04:09.560 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.495+0000 7f00fb7fe700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:04:09.560 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.495+0000 7f00fb7fe700 1 -- 172.21.15.102:0/3853891921 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01100932b0 con 0x7f0110126c10 2021-05-25T12:04:09.560 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.495+0000 7f00fb7fe700 10 monclient: _finish_auth 0 2021-05-25T12:04:09.561 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.495+0000 7f00fb7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:04:09.561 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.495+0000 7f0115ef9700 5 monclient: authenticate success, global_id 184157 2021-05-25T12:04:09.561 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.495+0000 7f0115ef9700 10 monclient: _renew_subs 2021-05-25T12:04:09.561 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.495+0000 7f0115ef9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:04:09.562 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.495+0000 7f0115ef9700 1 -- 172.21.15.102:0/3853891921 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0110124090 con 0x7f0110126c10 2021-05-25T12:04:09.562 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.495+0000 7f0115ef9700 10 client.184157.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:04:09.562 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.495+0000 7f0115ef9700 10 monclient: _renew_subs 2021-05-25T12:04:09.562 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.495+0000 7f0115ef9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:04:09.562 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.495+0000 7f0115ef9700 1 -- 172.21.15.102:0/3853891921 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0110144470 con 0x7f0110126c10 2021-05-25T12:04:09.563 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.495+0000 7f0115ef9700 1 librados: init done 2021-05-25T12:04:09.563 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.495+0000 7f00fb7fe700 1 -- 172.21.15.102:0/3853891921 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0104002580 con 0x7f0110126c10 2021-05-25T12:04:09.563 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.495+0000 7f00fb7fe700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:04:09.563 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.495+0000 7f00fb7fe700 1 -- 172.21.15.102:0/3853891921 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f0104002890 con 0x7f0110126c10 2021-05-25T12:04:09.564 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.495+0000 7f00fb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:09.564 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.495+0000 7f00fb7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:04:09.564 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.495+0000 7f010dffb700 4 set_mon_vals no callback set 2021-05-25T12:04:09.564 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.495+0000 7f00fb7fe700 10 monclient: dump: 2021-05-25T12:04:09.564 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:09.565 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:09.565 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:09.565 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:09.565 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:09.565 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:09.566 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:09.566 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:09.566 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:09.566 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:09.566 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.496+0000 7f0115ef9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:04:09.567 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.496+0000 7f0115ef9700 1 -- 172.21.15.102:0/3853891921 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f00e80065a0 con 0x7f00e8002840 2021-05-25T12:04:09.567 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.496+0000 7f00fb7fe700 1 -- 172.21.15.102:0/3853891921 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f010400bbb0 con 0x7f0110126c10 2021-05-25T12:04:09.567 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.496+0000 7f00fb7fe700 1 -- 172.21.15.102:0/3853891921 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(125..125 src has 1..125) v4 ==== 5802+0+0 (unknown 2341125478 0 0) 0x7f010403b440 con 0x7f0110126c10 2021-05-25T12:04:09.567 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.496+0000 7f00fb7fe700 10 client.184157.objecter ms_dispatch 0x7f0110001550 osd_map(125..125 src has 1..125) v4 2021-05-25T12:04:09.568 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.496+0000 7f00fb7fe700 3 client.184157.objecter handle_osd_map got epochs [125,125] > 0 2021-05-25T12:04:09.568 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.496+0000 7f00fb7fe700 3 client.184157.objecter handle_osd_map decoding full epoch 125 2021-05-25T12:04:09.568 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.496+0000 7f00fb7fe700 20 client.184157.objecter dump_active .. 0 homeless 2021-05-25T12:04:09.568 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.496+0000 7f00fb7fe700 10 client.184157.objecter ms_handle_connect 0x7f00e8002840 2021-05-25T12:04:09.568 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.496+0000 7f00fb7fe700 10 client.184157.objecter resend_mon_ops 2021-05-25T12:04:09.569 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.496+0000 7f010effd700 10 monclient: get_auth_request con 0x7f00f0033500 auth_method 0 2021-05-25T12:04:09.569 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.496+0000 7f00fb7fe700 1 -- 172.21.15.102:0/3853891921 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2304457321 0 0) 0x7f00e80065a0 con 0x7f00e8002840 2021-05-25T12:04:09.569 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.496+0000 7f00fb7fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:04:09.569 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.496+0000 7f00fb7fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:04:09.570 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.496+0000 7f00fb7fe700 10 monclient(hunting): my global_id is 95347 2021-05-25T12:04:09.570 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.496+0000 7f00fb7fe700 1 -- 172.21.15.102:0/3853891921 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0110123ae0 con 0x7f00e8002840 2021-05-25T12:04:09.570 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.496+0000 7f00fb7fe700 10 client.184157.objecter ms_handle_connect 0x7f00f0033500 2021-05-25T12:04:09.570 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.496+0000 7f00fb7fe700 1 -- 172.21.15.102:0/3853891921 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2063128916 0 0) 0x7f0110123ae0 con 0x7f00e8002840 2021-05-25T12:04:09.570 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.496+0000 7f00fb7fe700 1 -- 172.21.15.102:0/3853891921 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f00e80065a0 con 0x7f00e8002840 2021-05-25T12:04:09.571 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.496+0000 7f00fb7fe700 1 -- 172.21.15.102:0/3853891921 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f00fc004980 con 0x7f00e8002840 2021-05-25T12:04:09.571 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.496+0000 7f00fb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:09.571 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.496+0000 7f00fb7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T12:04:09.571 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.496+0000 7f00fb7fe700 10 monclient: dump: 2021-05-25T12:04:09.572 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:09.572 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:09.572 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:09.572 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:09.572 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:09.573 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:09.573 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:09.573 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:09.573 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:09.573 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:09.574 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.497+0000 7f00fb7fe700 1 -- 172.21.15.102:0/3853891921 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1361185620 0 0) 0x7f00e80065a0 con 0x7f00e8002840 2021-05-25T12:04:09.574 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.497+0000 7f00fb7fe700 1 -- 172.21.15.102:0/3853891921 --> v1:172.21.15.181:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f00e8009e40 con 0x7f00e8002840 2021-05-25T12:04:09.574 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.497+0000 7f00fb7fe700 1 -- 172.21.15.102:0/3853891921 <== mon.1 v1:172.21.15.181:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7f00fc004d60 con 0x7f00e8002840 2021-05-25T12:04:09.574 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.497+0000 7f00fb7fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:04:09.574 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.497+0000 7f00fb7fe700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:04:09.575 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.497+0000 7f00fb7fe700 1 -- 172.21.15.102:0/3853891921 >> v1:172.21.15.181:6789/0 conn(0x7f00e8002840 legacy=0x7f00e8004ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:09.575 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.497+0000 7f00fb7fe700 1 -- 172.21.15.102:0/3853891921 >> v1:172.21.15.181:6789/0 conn(0x7f00e8002840 legacy=0x7f00e8004ce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:04:09.575 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.503+0000 7f0115ef9700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:04:09.575 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.503+0000 7f0115ef9700 1 -- 172.21.15.102:0/3853891921 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f00e8007510 con 0x7f00e80068b0 2021-05-25T12:04:09.576 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.504+0000 7f00fb7fe700 10 client.184157.objecter ms_handle_connect 0x7f00e80068b0 2021-05-25T12:04:09.576 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.504+0000 7f00fb7fe700 10 client.184157.objecter resend_mon_ops 2021-05-25T12:04:09.576 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.504+0000 7f00fb7fe700 1 -- 172.21.15.102:0/3853891921 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 757489272 0 0) 0x7f00e8007510 con 0x7f00e80068b0 2021-05-25T12:04:09.576 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.504+0000 7f00fb7fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:04:09.576 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.504+0000 7f00fb7fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:04:09.577 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.504+0000 7f00fb7fe700 10 monclient(hunting): my global_id is 95350 2021-05-25T12:04:09.577 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.504+0000 7f00fb7fe700 1 -- 172.21.15.102:0/3853891921 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00e80065a0 con 0x7f00e80068b0 2021-05-25T12:04:09.577 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.504+0000 7f00fb7fe700 1 -- 172.21.15.102:0/3853891921 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4265614932 0 0) 0x7f00e80065a0 con 0x7f00e80068b0 2021-05-25T12:04:09.577 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.504+0000 7f00fb7fe700 1 -- 172.21.15.102:0/3853891921 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f00e8007510 con 0x7f00e80068b0 2021-05-25T12:04:09.577 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.504+0000 7f00fb7fe700 1 -- 172.21.15.102:0/3853891921 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f00fc0053f0 con 0x7f00e80068b0 2021-05-25T12:04:09.578 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.504+0000 7f00fb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:09.578 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.504+0000 7f00fb7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T12:04:09.578 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.504+0000 7f00fb7fe700 10 monclient: dump: 2021-05-25T12:04:09.578 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:09.579 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:09.579 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:09.579 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:09.579 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:09.579 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:09.580 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:09.580 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:09.580 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:09.580 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:09.580 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.504+0000 7f00fb7fe700 1 -- 172.21.15.102:0/3853891921 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 71249677 0 0) 0x7f00e8007510 con 0x7f00e80068b0 2021-05-25T12:04:09.581 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.504+0000 7f00fb7fe700 1 -- 172.21.15.102:0/3853891921 --> v1:172.21.15.181:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f00e8008770 con 0x7f00e80068b0 2021-05-25T12:04:09.581 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.505+0000 7f00fb7fe700 1 -- 172.21.15.102:0/3853891921 <== mon.1 v1:172.21.15.181:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3772 (unknown 0 0 4007877631) 0x7f00fc006690 con 0x7f00e80068b0 2021-05-25T12:04:09.581 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.505+0000 7f00fb7fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:04:09.581 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.505+0000 7f00fb7fe700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:04:09.581 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.505+0000 7f00fb7fe700 1 -- 172.21.15.102:0/3853891921 >> v1:172.21.15.181:6789/0 conn(0x7f00e80068b0 legacy=0x7f00e8006d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:09.582 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.505+0000 7f00fb7fe700 1 -- 172.21.15.102:0/3853891921 >> v1:172.21.15.181:6789/0 conn(0x7f00e80068b0 legacy=0x7f00e8006d40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:04:09.582 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:04:09.582 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:04:09.583 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:04:09.583 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "leader", 2021-05-25T12:04:09.583 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 38, 2021-05-25T12:04:09.583 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:04:09.584 INFO:teuthology.orchestra.run.smithi102.stdout: 0, 2021-05-25T12:04:09.584 INFO:teuthology.orchestra.run.smithi102.stdout: 1, 2021-05-25T12:04:09.584 INFO:teuthology.orchestra.run.smithi102.stdout: 2 2021-05-25T12:04:09.584 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:04:09.584 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 1, 2021-05-25T12:04:09.585 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:04:09.585 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:04:09.585 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:04:09.585 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:04:09.586 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:04:09.586 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:04:09.586 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:04:09.586 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:04:09.586 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:04:09.587 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:04:09.587 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:04:09.587 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:04:09.587 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:04:09.588 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:04:09.588 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:04:09.588 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:04:09.588 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:04:09.588 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:04:09.589 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:04:09.589 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:04:09.589 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:04:09.589 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:04:09.590 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:04:09.590 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:04:09.590 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:04:09.590 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:04:09.590 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:04:09.591 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:04:09.591 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:04:09.591 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:04:09.591 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:04:09.592 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:04:09.592 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:04:09.592 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:04:09.592 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:04:09.592 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:04:09.593 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:04:09.593 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:04:09.593 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:04:09.593 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:04:09.594 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:04:09.594 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:04:09.594 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:04:09.594 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:04:09.594 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:04:09.595 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:04:09.595 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:04:09.595 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:04:09.595 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:04:09.596 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:04:09.596 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:04:09.596 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:04:09.596 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:04:09.596 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:04:09.597 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:04:09.597 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:04:09.597 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:04:09.597 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:04:09.598 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:04:09.598 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:04:09.598 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:04:09.598 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:04:09.598 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:04:09.599 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:04:09.599 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:04:09.599 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:04:09.599 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:04:09.599 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:04:09.600 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:04:09.600 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:04:09.600 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:04:09.600 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:04:09.601 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:04:09.601 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:04:09.601 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:04:09.601 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:04:09.601 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:04:09.602 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:04:09.602 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:04:09.602 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:04:09.602 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:04:09.603 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:04:09.603 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:04:09.603 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:04:09.603 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:04:09.603 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:04:09.604 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:04:09.604 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:04:09.604 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:04:09.604 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:04:09.604 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:04:09.605 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:04:09.605 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:04:09.605 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:04:09.605 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:04:09.605 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:04:09.606 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:04:09.606 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:04:09.606 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:04:09.606 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:04:09.607 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:04:09.607 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:04:09.607 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:04:09.607 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:04:09.607 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:04:09.608 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:04:09.608 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:04:09.608 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:04:09.608 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:04:09.609 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:04:09.609 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:04:09.609 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:04:09.609 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:04:09.609 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:04:09.610 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:04:09.610 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:04:09.610 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:04:09.610 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:04:09.611 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:04:09.611 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:04:09.611 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:04:09.611 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:04:09.611 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 4 2021-05-25T12:04:09.612 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:04:09.612 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:04:09.612 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:04:09.612 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:04:09.612 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:04:09.615 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.505+0000 7f0115ef9700 10 librados: watch_flush enter 2021-05-25T12:04:09.615 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.505+0000 7f0115ef9700 10 librados: watch_flush exit 2021-05-25T12:04:09.615 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.505+0000 7f0115ef9700 20 client.184157.objecter shutdown clearing up homeless session... 2021-05-25T12:04:09.616 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.505+0000 7f0115ef9700 10 client.184157.objecter successfully canceled tick 2021-05-25T12:04:09.616 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.505+0000 7f0115ef9700 1 -- 172.21.15.102:0/3853891921 >> v1:172.21.15.181:6812/29582 conn(0x7f00f0033500 legacy=0x7f00f00359c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:09.616 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.505+0000 7f0115ef9700 10 monclient: shutdown 2021-05-25T12:04:09.616 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.505+0000 7f0115ef9700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:04:09.617 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.505+0000 7f0115ef9700 1 -- 172.21.15.102:0/3853891921 >> v1:172.21.15.102:6790/0 conn(0x7f0110126c10 legacy=0x7f0110142b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:09.617 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.505+0000 7f010ffff700 1 -- 172.21.15.102:0/3853891921 reap_dead start 2021-05-25T12:04:09.617 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.505+0000 7f010ffff700 1 -- 172.21.15.102:0/3853891921 reap_dead start 2021-05-25T12:04:09.617 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.505+0000 7f0115ef9700 1 -- 172.21.15.102:0/3853891921 shutdown_connections 2021-05-25T12:04:09.617 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.505+0000 7f0115ef9700 1 -- 172.21.15.102:0/3853891921 >> 172.21.15.102:0/3853891921 conn(0x7f011009b370 msgr2=0x7f01101254d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:04:09.618 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.505+0000 7f0115ef9700 1 -- 172.21.15.102:0/3853891921 shutdown_connections 2021-05-25T12:04:09.618 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.505+0000 7f0115ef9700 1 -- 172.21.15.102:0/3853891921 wait complete. 2021-05-25T12:04:09.618 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.505+0000 7f0115ef9700 1 librados: shutdown 2021-05-25T12:04:09.618 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2021-05-25T12:04:09.619 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2021-05-25T12:04:09.728 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.727+0000 7fbf62270700 10 monclient: get_monmap_and_config 2021-05-25T12:04:09.728 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.727+0000 7fbf62270700 10 monclient: build_initial_monmap 2021-05-25T12:04:09.729 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.727+0000 7fbf62270700 10 monclient: monmap: 2021-05-25T12:04:09.729 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:04:09.729 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:09.729 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:04:09.728073+0000 2021-05-25T12:04:09.730 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:04:09.728073+0000 2021-05-25T12:04:09.730 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:04:09.730 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:09.730 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:04:09.731 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:04:09.731 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:04:09.731 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:09.742 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.727+0000 7fbf62270700 1 Processor -- start 2021-05-25T12:04:09.742 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.727+0000 7fbf62270700 1 -- start start 2021-05-25T12:04:09.742 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.727+0000 7fbf62270700 10 monclient: init 2021-05-25T12:04:09.742 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.728+0000 7fbf62270700 10 monclient: _reopen_session rank -1 2021-05-25T12:04:09.743 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.728+0000 7fbf62270700 10 monclient: _add_conns ranks=[1,0,2] 2021-05-25T12:04:09.743 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.728+0000 7fbf62270700 10 monclient(hunting): picked mon.noname-c con 0x7fbf5c096c00 addr v1:172.21.15.181:6789/0 2021-05-25T12:04:09.743 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.728+0000 7fbf62270700 10 monclient(hunting): picked mon.noname-a con 0x7fbf5c0696b0 addr v1:172.21.15.102:6789/0 2021-05-25T12:04:09.743 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.728+0000 7fbf62270700 10 monclient(hunting): picked mon.noname-b con 0x7fbf5c1303a0 addr v1:172.21.15.102:6790/0 2021-05-25T12:04:09.743 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.728+0000 7fbf62270700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf5c068340 con 0x7fbf5c0696b0 2021-05-25T12:04:09.744 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.728+0000 7fbf62270700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf5c0982f0 con 0x7fbf5c096c00 2021-05-25T12:04:09.744 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.728+0000 7fbf62270700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf5c0677a0 con 0x7fbf5c1303a0 2021-05-25T12:04:09.744 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.728+0000 7fbf62270700 10 monclient(hunting): _renew_subs 2021-05-25T12:04:09.744 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.728+0000 7fbf62270700 10 monclient(hunting): authenticate will time out at 1050.325378s 2021-05-25T12:04:09.745 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.728+0000 7fbf5b7fe700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fbf5c0696b0 0x7fbf5c097be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60692/0 (socket says 172.21.15.102:60692) 2021-05-25T12:04:09.745 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.728+0000 7fbf5b7fe700 1 -- 172.21.15.102:0/424336033 learned_addr learned my addr 172.21.15.102:0/424336033 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:04:09.745 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.728+0000 7fbf6080d700 1 --1- 172.21.15.102:0/424336033 >> v1:172.21.15.102:6790/0 conn(0x7fbf5c1303a0 0x7fbf5c132790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:55572/0 (socket says 172.21.15.102:55572) 2021-05-25T12:04:09.745 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.728+0000 7fbf5affd700 1 -- 172.21.15.102:0/424336033 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2145668857 0 0) 0x7fbf5c0677a0 con 0x7fbf5c1303a0 2021-05-25T12:04:09.745 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.728+0000 7fbf5affd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:04:09.746 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.728+0000 7fbf5affd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:04:09.746 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.728+0000 7fbf5affd700 10 monclient(hunting): my global_id is 184160 2021-05-25T12:04:09.746 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.728+0000 7fbf5affd700 1 -- 172.21.15.102:0/424336033 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf44003270 con 0x7fbf5c1303a0 2021-05-25T12:04:09.746 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.728+0000 7fbf5affd700 1 -- 172.21.15.102:0/424336033 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2569315922 0 0) 0x7fbf5c068340 con 0x7fbf5c0696b0 2021-05-25T12:04:09.747 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.728+0000 7fbf5affd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:04:09.747 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.728+0000 7fbf5affd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:04:09.747 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.728+0000 7fbf5affd700 10 monclient(hunting): my global_id is 174432 2021-05-25T12:04:09.747 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.728+0000 7fbf5affd700 1 -- 172.21.15.102:0/424336033 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf5c0677a0 con 0x7fbf5c0696b0 2021-05-25T12:04:09.748 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.728+0000 7fbf5affd700 1 -- 172.21.15.102:0/424336033 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2765902282 0 0) 0x7fbf5c0677a0 con 0x7fbf5c0696b0 2021-05-25T12:04:09.748 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.728+0000 7fbf5affd700 1 -- 172.21.15.102:0/424336033 >> v1:172.21.15.102:6790/0 conn(0x7fbf5c1303a0 legacy=0x7fbf5c132790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:09.748 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.728+0000 7fbf5affd700 1 -- 172.21.15.102:0/424336033 >> v1:172.21.15.181:6789/0 conn(0x7fbf5c096c00 legacy=0x7fbf5c0974d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:09.748 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.728+0000 7fbf5affd700 10 monclient: _finish_hunting 0 2021-05-25T12:04:09.749 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.728+0000 7fbf5affd700 1 monclient: found mon.noname-a 2021-05-25T12:04:09.749 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.728+0000 7fbf5affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:04:09.749 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.728+0000 7fbf5affd700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T12:04:09.749 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.728+0000 7fbf5affd700 1 -- 172.21.15.102:0/424336033 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf5c067a40 con 0x7fbf5c0696b0 2021-05-25T12:04:09.750 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.729+0000 7fbf5affd700 10 monclient: _finish_auth 0 2021-05-25T12:04:09.750 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.729+0000 7fbf5affd700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:04:09.750 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.729+0000 7fbf62270700 5 monclient: authenticate success, global_id 174432 2021-05-25T12:04:09.750 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.729+0000 7fbf62270700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:04:09.751 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.729+0000 7fbf5affd700 1 -- 172.21.15.102:0/424336033 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fbf40001610 con 0x7fbf5c0696b0 2021-05-25T12:04:09.751 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.729+0000 7fbf5affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:09.751 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.729+0000 7fbf5affd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:04:09.751 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.729+0000 7fbf5affd700 10 monclient: dump: 2021-05-25T12:04:09.751 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:09.752 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:09.752 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:09.752 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:09.752 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:09.753 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:09.753 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:09.753 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:09.753 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:09.754 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:09.754 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.729+0000 7fbf62270700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:04:09.754 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.729+0000 7fbf5affd700 1 -- 172.21.15.102:0/424336033 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbf400017c0 con 0x7fbf5c0696b0 2021-05-25T12:04:09.754 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.729+0000 7fbf5affd700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:04:09.754 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.729+0000 7fbf62270700 10 monclient: get_monmap_and_config success 2021-05-25T12:04:09.755 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.729+0000 7fbf5affd700 1 -- 172.21.15.102:0/424336033 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fbf40001b50 con 0x7fbf5c0696b0 2021-05-25T12:04:09.755 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.729+0000 7fbf62270700 4 set_mon_vals no callback set 2021-05-25T12:04:09.755 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.729+0000 7fbf5affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:09.755 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.729+0000 7fbf5affd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:04:09.756 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.729+0000 7fbf5affd700 10 monclient: dump: 2021-05-25T12:04:09.756 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:09.756 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:09.756 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:09.757 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:09.757 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:09.757 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:09.757 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:09.757 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:09.758 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:09.758 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:09.758 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.729+0000 7fbf62270700 10 monclient: shutdown 2021-05-25T12:04:09.758 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.729+0000 7fbf62270700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:04:09.758 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.729+0000 7fbf62270700 1 -- 172.21.15.102:0/424336033 >> v1:172.21.15.102:6789/0 conn(0x7fbf5c0696b0 legacy=0x7fbf5c097be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:09.759 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.729+0000 7fbf62270700 1 -- 172.21.15.102:0/424336033 shutdown_connections 2021-05-25T12:04:09.759 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.729+0000 7fbf62270700 1 -- 172.21.15.102:0/424336033 >> 172.21.15.102:0/424336033 conn(0x7fbf5c09cb30 msgr2=0x7fbf5c09cf40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:04:09.759 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.729+0000 7fbf62270700 1 -- 172.21.15.102:0/424336033 shutdown_connections 2021-05-25T12:04:09.759 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.729+0000 7fbf62270700 1 -- 172.21.15.102:0/424336033 wait complete. 2021-05-25T12:04:09.760 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.730+0000 7fbf62270700 10 monclient: build_initial_monmap 2021-05-25T12:04:09.760 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.730+0000 7fbf62270700 10 monclient: monmap: 2021-05-25T12:04:09.760 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:04:09.760 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:04:09.760 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:04:09.761 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:04:09.761 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:04:09.761 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:04:09.761 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:04:09.762 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:04:09.762 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:04:09.762 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:09.762 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.730+0000 7fbf62270700 1 librados: starting msgr at 2021-05-25T12:04:09.762 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.730+0000 7fbf62270700 1 librados: starting objecter 2021-05-25T12:04:09.763 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.730+0000 7fbf62270700 1 Processor -- start 2021-05-25T12:04:09.763 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.730+0000 7fbf62270700 1 -- start start 2021-05-25T12:04:09.763 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.730+0000 7fbf62270700 1 librados: setting wanted keys 2021-05-25T12:04:09.763 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.730+0000 7fbf62270700 1 librados: calling monclient init 2021-05-25T12:04:09.763 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.730+0000 7fbf62270700 10 monclient: init 2021-05-25T12:04:09.764 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.730+0000 7fbf62270700 10 monclient: _reopen_session rank -1 2021-05-25T12:04:09.764 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.730+0000 7fbf62270700 10 monclient: _add_conns ranks=[0,1,2] 2021-05-25T12:04:09.764 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.730+0000 7fbf62270700 10 monclient(hunting): picked mon.noname-a con 0x7fbf5c0696b0 addr v1:172.21.15.102:6789/0 2021-05-25T12:04:09.764 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.730+0000 7fbf62270700 10 monclient(hunting): picked mon.noname-b con 0x7fbf5c096c00 addr v1:172.21.15.181:6789/0 2021-05-25T12:04:09.765 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.730+0000 7fbf62270700 10 monclient(hunting): picked mon.noname-c con 0x7fbf5c1303a0 addr v1:172.21.15.102:6790/0 2021-05-25T12:04:09.765 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.730+0000 7fbf62270700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf5c0969d0 con 0x7fbf5c0696b0 2021-05-25T12:04:09.765 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.730+0000 7fbf62270700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf5c139910 con 0x7fbf5c096c00 2021-05-25T12:04:09.765 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.731+0000 7fbf62270700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf5c116090 con 0x7fbf5c1303a0 2021-05-25T12:04:09.765 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.731+0000 7fbf62270700 10 monclient(hunting): _renew_subs 2021-05-25T12:04:09.766 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.731+0000 7fbf62270700 10 monclient(hunting): authenticate will time out at 1050.328014s 2021-05-25T12:04:09.766 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.731+0000 7fbf6080d700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fbf5c1303a0 0x7fbf5c140e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:55578/0 (socket says 172.21.15.102:55578) 2021-05-25T12:04:09.766 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.731+0000 7fbf5bfff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fbf5c0696b0 0x7fbf5c138af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60696/0 (socket says 172.21.15.102:60696) 2021-05-25T12:04:09.766 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.731+0000 7fbf6080d700 1 -- 172.21.15.102:0/3227843672 learned_addr learned my addr 172.21.15.102:0/3227843672 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:04:09.766 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.731+0000 7fbf4ffff700 10 client.?.objecter ms_handle_connect 0x7fbf5c1303a0 2021-05-25T12:04:09.767 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.731+0000 7fbf4ffff700 10 client.?.objecter resend_mon_ops 2021-05-25T12:04:09.767 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.731+0000 7fbf4ffff700 10 client.?.objecter ms_handle_connect 0x7fbf5c0696b0 2021-05-25T12:04:09.767 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.731+0000 7fbf4ffff700 10 client.?.objecter resend_mon_ops 2021-05-25T12:04:09.767 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.731+0000 7fbf4ffff700 10 client.?.objecter ms_handle_connect 0x7fbf5c096c00 2021-05-25T12:04:09.768 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.731+0000 7fbf4ffff700 10 client.?.objecter resend_mon_ops 2021-05-25T12:04:09.768 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.731+0000 7fbf4ffff700 1 -- 172.21.15.102:0/3227843672 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4140296643 0 0) 0x7fbf5c116090 con 0x7fbf5c1303a0 2021-05-25T12:04:09.768 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.731+0000 7fbf4ffff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:04:09.768 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.731+0000 7fbf4ffff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:04:09.768 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.731+0000 7fbf4ffff700 10 monclient(hunting): my global_id is 184163 2021-05-25T12:04:09.769 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.731+0000 7fbf4ffff700 1 -- 172.21.15.102:0/3227843672 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf38004c10 con 0x7fbf5c1303a0 2021-05-25T12:04:09.769 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.731+0000 7fbf4ffff700 1 -- 172.21.15.102:0/3227843672 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3392004680 0 0) 0x7fbf5c0969d0 con 0x7fbf5c0696b0 2021-05-25T12:04:09.769 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.731+0000 7fbf4ffff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:04:09.769 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.731+0000 7fbf4ffff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:04:09.769 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.731+0000 7fbf4ffff700 10 monclient(hunting): my global_id is 174435 2021-05-25T12:04:09.770 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.731+0000 7fbf4ffff700 1 -- 172.21.15.102:0/3227843672 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf5c116090 con 0x7fbf5c0696b0 2021-05-25T12:04:09.770 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.731+0000 7fbf4ffff700 1 -- 172.21.15.102:0/3227843672 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2574430994 0 0) 0x7fbf5c139910 con 0x7fbf5c096c00 2021-05-25T12:04:09.770 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.731+0000 7fbf4ffff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:04:09.770 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.731+0000 7fbf4ffff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:04:09.770 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.731+0000 7fbf4ffff700 10 monclient(hunting): my global_id is 95356 2021-05-25T12:04:09.771 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.731+0000 7fbf4ffff700 1 -- 172.21.15.102:0/3227843672 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf5c0969d0 con 0x7fbf5c096c00 2021-05-25T12:04:09.771 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.731+0000 7fbf4ffff700 1 -- 172.21.15.102:0/3227843672 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2802946363 0 0) 0x7fbf38004c10 con 0x7fbf5c1303a0 2021-05-25T12:04:09.771 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.731+0000 7fbf4ffff700 1 -- 172.21.15.102:0/3227843672 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf5c139910 con 0x7fbf5c1303a0 2021-05-25T12:04:09.771 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.731+0000 7fbf4ffff700 1 -- 172.21.15.102:0/3227843672 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fbf50001d20 con 0x7fbf5c1303a0 2021-05-25T12:04:09.771 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.731+0000 7fbf4ffff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:09.772 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.731+0000 7fbf4ffff700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:04:09.772 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.731+0000 7fbf4ffff700 10 monclient(hunting): dump: 2021-05-25T12:04:09.772 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:09.772 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:09.772 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:09.773 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:09.773 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:09.773 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:09.773 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:09.773 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:09.774 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:09.774 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:09.774 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.731+0000 7fbf4ffff700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:04:09.774 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.731+0000 7fbf4ffff700 1 -- 172.21.15.102:0/3227843672 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3642471557 0 0) 0x7fbf5c0969d0 con 0x7fbf5c096c00 2021-05-25T12:04:09.774 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.731+0000 7fbf4ffff700 1 -- 172.21.15.102:0/3227843672 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf38004c10 con 0x7fbf5c096c00 2021-05-25T12:04:09.775 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.731+0000 7fbf4ffff700 1 -- 172.21.15.102:0/3227843672 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fbf400015d0 con 0x7fbf5c096c00 2021-05-25T12:04:09.775 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.731+0000 7fbf4ffff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:09.775 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.731+0000 7fbf4ffff700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2021-05-25T12:04:09.775 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.731+0000 7fbf4ffff700 10 monclient(hunting): dump: 2021-05-25T12:04:09.775 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:09.776 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:09.776 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:09.776 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:09.776 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:09.776 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:09.777 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:09.777 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:09.777 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:09.777 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:09.777 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.731+0000 7fbf4ffff700 1 -- 172.21.15.102:0/3227843672 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 228658574 0 0) 0x7fbf5c139910 con 0x7fbf5c1303a0 2021-05-25T12:04:09.778 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.731+0000 7fbf4ffff700 1 -- 172.21.15.102:0/3227843672 >> v1:172.21.15.181:6789/0 conn(0x7fbf5c096c00 legacy=0x7fbf5c139200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:09.778 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.731+0000 7fbf4ffff700 1 -- 172.21.15.102:0/3227843672 >> v1:172.21.15.102:6789/0 conn(0x7fbf5c0696b0 legacy=0x7fbf5c138af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:09.778 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.731+0000 7fbf4ffff700 10 monclient: _finish_hunting 0 2021-05-25T12:04:09.778 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.731+0000 7fbf4ffff700 1 monclient: found mon.c 2021-05-25T12:04:09.778 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.731+0000 7fbf4ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:04:09.779 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.731+0000 7fbf4ffff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:04:09.779 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.731+0000 7fbf4ffff700 1 -- 172.21.15.102:0/3227843672 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf5c066d90 con 0x7fbf5c1303a0 2021-05-25T12:04:09.779 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.731+0000 7fbf4ffff700 10 monclient: _finish_auth 0 2021-05-25T12:04:09.779 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.731+0000 7fbf4ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:04:09.780 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.731+0000 7fbf62270700 5 monclient: authenticate success, global_id 184163 2021-05-25T12:04:09.780 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.731+0000 7fbf62270700 10 monclient: _renew_subs 2021-05-25T12:04:09.780 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.731+0000 7fbf62270700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:04:09.780 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.731+0000 7fbf62270700 1 -- 172.21.15.102:0/3227843672 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf5c12c4d0 con 0x7fbf5c1303a0 2021-05-25T12:04:09.780 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.731+0000 7fbf62270700 10 client.184163.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:04:09.781 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.731+0000 7fbf62270700 10 monclient: _renew_subs 2021-05-25T12:04:09.781 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.731+0000 7fbf62270700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:04:09.781 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.731+0000 7fbf62270700 1 -- 172.21.15.102:0/3227843672 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf5c12b9e0 con 0x7fbf5c1303a0 2021-05-25T12:04:09.781 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.731+0000 7fbf62270700 1 librados: init done 2021-05-25T12:04:09.781 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.732+0000 7fbf4ffff700 1 -- 172.21.15.102:0/3227843672 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbf500048b0 con 0x7fbf5c1303a0 2021-05-25T12:04:09.782 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.732+0000 7fbf4ffff700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:04:09.782 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.732+0000 7fbf4ffff700 1 -- 172.21.15.102:0/3227843672 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fbf50004bc0 con 0x7fbf5c1303a0 2021-05-25T12:04:09.782 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.732+0000 7fbf4ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:09.782 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.732+0000 7fbf4ffff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:04:09.783 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.732+0000 7fbf4ffff700 10 monclient: dump: 2021-05-25T12:04:09.783 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:09.783 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:09.783 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:09.783 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:09.784 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:09.784 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:09.784 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:09.784 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:09.785 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:09.785 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:09.785 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.732+0000 7fbf5a7fc700 4 set_mon_vals no callback set 2021-05-25T12:04:09.785 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.732+0000 7fbf4ffff700 1 -- 172.21.15.102:0/3227843672 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7fbf5000dee0 con 0x7fbf5c1303a0 2021-05-25T12:04:09.785 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.732+0000 7fbf62270700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:04:09.786 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.732+0000 7fbf62270700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:04:09.786 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.732+0000 7fbf62270700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:04:09.786 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.732+0000 7fbf62270700 1 -- 172.21.15.102:0/3227843672 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf3c004db0 con 0x7fbf5c1303a0 2021-05-25T12:04:09.786 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.732+0000 7fbf4ffff700 1 -- 172.21.15.102:0/3227843672 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(125..125 src has 1..125) v4 ==== 5802+0+0 (unknown 2341125478 0 0) 0x7fbf5003d930 con 0x7fbf5c1303a0 2021-05-25T12:04:09.787 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.732+0000 7fbf4ffff700 10 client.184163.objecter ms_dispatch 0x7fbf5c001550 osd_map(125..125 src has 1..125) v4 2021-05-25T12:04:09.787 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.732+0000 7fbf4ffff700 3 client.184163.objecter handle_osd_map got epochs [125,125] > 0 2021-05-25T12:04:09.787 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.732+0000 7fbf4ffff700 3 client.184163.objecter handle_osd_map decoding full epoch 125 2021-05-25T12:04:09.787 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.732+0000 7fbf4ffff700 20 client.184163.objecter dump_active .. 0 homeless 2021-05-25T12:04:09.787 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.732+0000 7fbf5bfff700 10 monclient: get_auth_request con 0x7fbf38033170 auth_method 0 2021-05-25T12:04:09.788 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.733+0000 7fbf4ffff700 10 client.184163.objecter ms_handle_connect 0x7fbf38033170 2021-05-25T12:04:09.788 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.736+0000 7fbf4ffff700 1 -- 172.21.15.102:0/3227843672 <== mon.2 v1:172.21.15.102:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7fbf50012dd0 con 0x7fbf5c1303a0 2021-05-25T12:04:09.788 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.736+0000 7fbf4ffff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:04:09.788 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.736+0000 7fbf4ffff700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:04:09.909 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: Running main() from gmock_main.cc 2021-05-25T12:04:09.910 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [==========] Running 40 tests from 2 test suites. 2021-05-25T12:04:09.910 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [----------] Global test environment set-up. 2021-05-25T12:04:09.910 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [----------] 24 tests from LibRadosAio 2021-05-25T12:04:09.910 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [ RUN ] LibRadosAio.TooBig 2021-05-25T12:04:09.911 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [ OK ] LibRadosAio.TooBig (16694 ms) 2021-05-25T12:04:09.911 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [ RUN ] LibRadosAio.SimpleWrite 2021-05-25T12:04:09.911 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [ OK ] LibRadosAio.SimpleWrite (3015 ms) 2021-05-25T12:04:09.911 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [ RUN ] LibRadosAio.WaitForSafe 2021-05-25T12:04:09.912 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [ OK ] LibRadosAio.WaitForSafe (2991 ms) 2021-05-25T12:04:09.912 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip 2021-05-25T12:04:09.912 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip (2665 ms) 2021-05-25T12:04:09.912 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip2 2021-05-25T12:04:09.912 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip2 (3006 ms) 2021-05-25T12:04:09.913 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip3 2021-05-25T12:04:09.913 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip3 (3003 ms) 2021-05-25T12:04:09.913 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripAppend 2021-05-25T12:04:09.913 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [ OK ] LibRadosAio.RoundTripAppend (3041 ms) 2021-05-25T12:04:09.914 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [ RUN ] LibRadosAio.RemoveTest 2021-05-25T12:04:09.915 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [ OK ] LibRadosAio.RemoveTest (3013 ms) 2021-05-25T12:04:09.915 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [ RUN ] LibRadosAio.XattrsRoundTrip 2021-05-25T12:04:09.915 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [ OK ] LibRadosAio.XattrsRoundTrip (27778 ms) 2021-05-25T12:04:09.915 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [ RUN ] LibRadosAio.RmXattr 2021-05-25T12:04:09.915 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [ OK ] LibRadosAio.RmXattr (3002 ms) 2021-05-25T12:04:09.916 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [ RUN ] LibRadosAio.XattrIter 2021-05-25T12:04:09.916 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [ OK ] LibRadosAio.XattrIter (2957 ms) 2021-05-25T12:04:09.916 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [ RUN ] LibRadosAio.IsComplete 2021-05-25T12:04:09.916 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [ OK ] LibRadosAio.IsComplete (2997 ms) 2021-05-25T12:04:09.917 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [ RUN ] LibRadosAio.IsSafe 2021-05-25T12:04:09.917 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [ OK ] LibRadosAio.IsSafe (3011 ms) 2021-05-25T12:04:09.917 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [ RUN ] LibRadosAio.ReturnValue 2021-05-25T12:04:09.917 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [ OK ] LibRadosAio.ReturnValue (2984 ms) 2021-05-25T12:04:09.917 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [ RUN ] LibRadosAio.Flush 2021-05-25T12:04:09.918 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [ OK ] LibRadosAio.Flush (3007 ms) 2021-05-25T12:04:09.918 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [ RUN ] LibRadosAio.FlushAsync 2021-05-25T12:04:09.918 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [ OK ] LibRadosAio.FlushAsync (33880 ms) 2021-05-25T12:04:09.918 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteFull 2021-05-25T12:04:09.919 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteFull (3006 ms) 2021-05-25T12:04:09.919 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteSame 2021-05-25T12:04:09.919 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteSame (3042 ms) 2021-05-25T12:04:09.919 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStat 2021-05-25T12:04:09.919 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [ OK ] LibRadosAio.SimpleStat (3002 ms) 2021-05-25T12:04:09.920 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStatNS 2021-05-25T12:04:09.920 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [ OK ] LibRadosAio.SimpleStatNS (27355 ms) 2021-05-25T12:04:09.920 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [ RUN ] LibRadosAio.StatRemove 2021-05-25T12:04:09.920 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [ OK ] LibRadosAio.StatRemove (3008 ms) 2021-05-25T12:04:09.920 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [ RUN ] LibRadosAio.ExecuteClass 2021-05-25T12:04:09.921 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [ OK ] LibRadosAio.ExecuteClass (3022 ms) 2021-05-25T12:04:09.921 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [ RUN ] LibRadosAio.MultiWrite 2021-05-25T12:04:09.921 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [ OK ] LibRadosAio.MultiWrite (2890 ms) 2021-05-25T12:04:09.921 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [ RUN ] LibRadosAio.AioUnlock 2021-05-25T12:04:09.921 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [ OK ] LibRadosAio.AioUnlock (3013 ms) 2021-05-25T12:04:09.922 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [----------] 24 tests from LibRadosAio (165382 ms total) 2021-05-25T12:04:09.922 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: 2021-05-25T12:04:09.922 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [----------] 16 tests from LibRadosAioEC 2021-05-25T12:04:09.922 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleWrite 2021-05-25T12:04:09.923 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleWrite (37810 ms) 2021-05-25T12:04:09.923 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [ RUN ] LibRadosAioEC.WaitForComplete 2021-05-25T12:04:09.923 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [ OK ] LibRadosAioEC.WaitForComplete (7036 ms) 2021-05-25T12:04:09.958 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.957+0000 7fbf62270700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2021-05-25T12:04:09.959 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.957+0000 7fbf62270700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2021-05-25T12:04:09.959 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.957+0000 7fbf62270700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:04:09.959 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.957+0000 7fbf62270700 1 -- 172.21.15.102:0/3227843672 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7fbf3c005870 con 0x7fbf5c1303a0 2021-05-25T12:04:09.960 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.958+0000 7fbf4ffff700 1 -- 172.21.15.102:0/3227843672 <== mon.2 v1:172.21.15.102:6790/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7fbf50012bf0 con 0x7fbf5c1303a0 2021-05-25T12:04:09.961 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.958+0000 7fbf4ffff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2021-05-25T12:04:09.961 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.958+0000 7fbf4ffff700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:04:09.962 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:04:09.958+0000 7fe1a9b6e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:04:09.962 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:04:09.958+0000 7fe1a9b6e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=28,config=2,health=10,kv=6,logm=54} crc {auth=1728945189,config=3140514009,health=1654430262,kv=1184736763,logm=182489732}) 2021-05-25T12:04:09.962 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:04:09.958+0000 7fe1a9b6e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=29,config=2,health=12,kv=6,logm=51} crc {auth=547706430,config=3140514009,health=2689956847,kv=1184736763,logm=805887561}) 2021-05-25T12:04:09.963 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:04:09.958+0000 7fe1a9b6e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:04:09.963 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:04:09.958+0000 7fe1a9b6e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=28,config=2,health=10,kv=6,logm=54} crc {auth=1728945189,config=3140514009,health=1654430262,kv=1184736763,logm=182489732}) 2021-05-25T12:04:09.963 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:04:09.958+0000 7fe1a9b6e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=23,config=2,health=12,kv=6,logm=57} crc {auth=862303670,config=3140514009,health=2689956847,kv=1184736764,logm=3849477107}) 2021-05-25T12:04:09.963 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:04:09.959+0000 7fe1a9b6e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:04:09.963 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:04:09.959+0000 7fe1a9b6e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=948463147}) 2021-05-25T12:04:09.964 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:04:09.959+0000 7fe1a9b6e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3812417754}) 2021-05-25T12:04:09.964 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:04:09.959+0000 7fe1a9b6e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:04:09.964 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:04:09.959+0000 7fe1a9b6e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=948463147}) 2021-05-25T12:04:09.964 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:04:09.959+0000 7fe1a9b6e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=487869568}) 2021-05-25T12:04:09.967 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.960+0000 7fbf62270700 10 librados: watch_flush enter 2021-05-25T12:04:09.967 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.960+0000 7fbf62270700 10 librados: watch_flush exit 2021-05-25T12:04:09.968 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.960+0000 7fbf62270700 20 client.184163.objecter shutdown clearing up homeless session... 2021-05-25T12:04:09.968 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.960+0000 7fbf62270700 10 client.184163.objecter successfully canceled tick 2021-05-25T12:04:09.968 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.960+0000 7fbf62270700 1 -- 172.21.15.102:0/3227843672 >> v1:172.21.15.181:6812/29582 conn(0x7fbf38033170 legacy=0x7fbf38035630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:09.968 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.960+0000 7fbf62270700 10 monclient: shutdown 2021-05-25T12:04:09.968 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.960+0000 7fbf62270700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:04:09.969 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.960+0000 7fbf62270700 1 -- 172.21.15.102:0/3227843672 >> v1:172.21.15.102:6790/0 conn(0x7fbf5c1303a0 legacy=0x7fbf5c140e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:09.969 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.960+0000 7fbf62270700 1 -- 172.21.15.102:0/3227843672 shutdown_connections 2021-05-25T12:04:09.969 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.960+0000 7fbf62270700 1 -- 172.21.15.102:0/3227843672 >> 172.21.15.102:0/3227843672 conn(0x7fbf5c09cb30 msgr2=0x7fbf5c12e6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:04:09.969 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.961+0000 7fbf62270700 1 -- 172.21.15.102:0/3227843672 shutdown_connections 2021-05-25T12:04:09.970 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.961+0000 7fbf62270700 1 -- 172.21.15.102:0/3227843672 wait complete. 2021-05-25T12:04:09.970 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:09.961+0000 7fbf62270700 1 librados: shutdown 2021-05-25T12:04:09.971 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:04:09.962+0000 7fe1a9b6e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:04:09.971 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:04:09.962+0000 7fe1a9b6e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1608364346}) 2021-05-25T12:04:09.971 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:04:09.962+0000 7fe1a9b6e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2208449507}) 2021-05-25T12:04:09.971 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:04:09.962+0000 7fe1a9b6e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:04:09.972 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:04:09.962+0000 7fe1a9b6e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1608364346}) 2021-05-25T12:04:09.972 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:04:09.962+0000 7fe1a9b6e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=911836484}) 2021-05-25T12:04:09.972 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:04:09.963+0000 7fe1a9b6e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:04:09.972 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:04:09.963+0000 7fe1a9b6e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=7,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=4,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=21,osdmap=37} crc {logm=4052380540,mdsmap=1531151339,mgr=2996579194,mgr_command_descs=3824783157,mgr_metadata=1267308364,mgrstat=3132428725,mon_config_key=4118394475,monmap=3809614708,osd_metadata=725478305,osd_pg_creating=171212092,osd_snap=2080021530,osdmap=3767155224}) 2021-05-25T12:04:09.972 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:04:09.963+0000 7fe1a9b6e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=7,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=4,monmap=2,osd_metadata=5,osd_pg_creating=1,osd_snap=23,osdmap=38} crc {logm=4052380540,mdsmap=1531151339,mgr=2996579194,mgr_command_descs=3824783157,mgr_metadata=1267308364,mgrstat=1185298094,mon_config_key=4118394475,monmap=3140514009,osd_metadata=4154602222,osd_pg_creating=171212092,osd_snap=899772989,osdmap=3640223155}) 2021-05-25T12:04:09.973 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:04:09.963+0000 7fe1a9b6e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:04:09.973 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:04:09.963+0000 7fe1a9b6e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=7,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=4,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=21,osdmap=37} crc {logm=4052380540,mdsmap=1531151339,mgr=2996579194,mgr_command_descs=3824783157,mgr_metadata=1267308364,mgrstat=3132428725,mon_config_key=4118394475,monmap=3809614708,osd_metadata=725478305,osd_pg_creating=171212092,osd_snap=2080021530,osdmap=3767155224}) 2021-05-25T12:04:09.973 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:04:09.963+0000 7fe1a9b6e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=6,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=4,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=24,osdmap=36} crc {logm=1873510492,mdsmap=1531151339,mgr=2996579194,mgr_command_descs=3824783157,mgr_metadata=1267308364,mgrstat=3132428725,mon_config_key=4118394475,monmap=3809614708,osd_metadata=927457006,osd_pg_creating=171212092,osd_snap=3213015543,osdmap=3837067530}) 2021-05-25T12:04:09.973 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:04:09.964+0000 7fe1a9b6e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:04:09.974 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:04:09.964+0000 7fe1a9b6e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2053576174}) 2021-05-25T12:04:09.974 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:04:09.964+0000 7fe1a9b6e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3400923757}) 2021-05-25T12:04:09.974 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:04:09.964+0000 7fe1a9b6e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:04:09.974 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:04:09.964+0000 7fe1a9b6e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2053576174}) 2021-05-25T12:04:09.974 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:04:09.964+0000 7fe1a9b6e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2117228956}) 2021-05-25T12:04:09.976 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:04:09.969+0000 7fe1a9b6e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:04:09.976 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:04:09.969+0000 7fe1a9b6e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=65} crc {osdmap=1392597811}) 2021-05-25T12:04:09.976 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:04:09.969+0000 7fe1a9b6e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=61} crc {osdmap=2072931977}) 2021-05-25T12:04:09.976 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:04:09.969+0000 7fe1a9b6e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:04:09.976 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:04:09.969+0000 7fe1a9b6e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=65} crc {osdmap=1392597811}) 2021-05-25T12:04:09.977 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:04:09.969+0000 7fe1a9b6e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=63} crc {osdmap=419334665}) 2021-05-25T12:04:09.977 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2021-05-25T12:04:10.978 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2021-05-25T12:04:10.979 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2021-05-25T12:04:11.119 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.093+0000 7f04ac0dd700 10 monclient: get_monmap_and_config 2021-05-25T12:04:11.120 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.093+0000 7f04ac0dd700 10 monclient: build_initial_monmap 2021-05-25T12:04:11.120 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.093+0000 7f04ac0dd700 10 monclient: monmap: 2021-05-25T12:04:11.120 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:04:11.121 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:11.121 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:04:11.094471+0000 2021-05-25T12:04:11.121 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:04:11.094471+0000 2021-05-25T12:04:11.122 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:04:11.122 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:11.122 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:04:11.123 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:04:11.123 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:04:11.123 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:11.124 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.094+0000 7f04ac0dd700 1 Processor -- start 2021-05-25T12:04:11.124 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.094+0000 7f04ac0dd700 1 -- start start 2021-05-25T12:04:11.124 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.094+0000 7f04ac0dd700 10 monclient: init 2021-05-25T12:04:11.124 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.094+0000 7f04ac0dd700 10 monclient: _reopen_session rank -1 2021-05-25T12:04:11.125 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.094+0000 7f04ac0dd700 10 monclient: _add_conns ranks=[1,2,0] 2021-05-25T12:04:11.125 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.094+0000 7f04ac0dd700 10 monclient(hunting): picked mon.noname-c con 0x7f04a4128d10 addr v1:172.21.15.181:6789/0 2021-05-25T12:04:11.125 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.094+0000 7f04ac0dd700 10 monclient(hunting): picked mon.noname-b con 0x7f04a4096810 addr v1:172.21.15.102:6790/0 2021-05-25T12:04:11.126 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.094+0000 7f04ac0dd700 10 monclient(hunting): picked mon.noname-a con 0x7f04a41321f0 addr v1:172.21.15.102:6789/0 2021-05-25T12:04:11.127 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.094+0000 7f04ac0dd700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04a4069b10 con 0x7f04a41321f0 2021-05-25T12:04:11.127 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.094+0000 7f04ac0dd700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04a40976c0 con 0x7f04a4128d10 2021-05-25T12:04:11.127 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.094+0000 7f04ac0dd700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04a4097960 con 0x7f04a4096810 2021-05-25T12:04:11.127 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.094+0000 7f04ac0dd700 10 monclient(hunting): _renew_subs 2021-05-25T12:04:11.128 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.094+0000 7f04ac0dd700 10 monclient(hunting): authenticate will time out at 1051.691886s 2021-05-25T12:04:11.128 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.095+0000 7f04aa67a700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f04a41321f0 0x7f04a41345e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60708/0 (socket says 172.21.15.102:60708) 2021-05-25T12:04:11.128 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.095+0000 7f04aa67a700 1 -- 172.21.15.102:0/958459200 learned_addr learned my addr 172.21.15.102:0/958459200 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:04:11.128 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.095+0000 7f04a9678700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f04a4096810 0x7f04a4130ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:55584/0 (socket says 172.21.15.102:55584) 2021-05-25T12:04:11.129 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.095+0000 7f04a8e77700 1 -- 172.21.15.102:0/958459200 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 199543093 0 0) 0x7f04a4069b10 con 0x7f04a41321f0 2021-05-25T12:04:11.129 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.095+0000 7f04a8e77700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:04:11.129 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.095+0000 7f04a8e77700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:04:11.129 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.095+0000 7f04a8e77700 10 monclient(hunting): my global_id is 174450 2021-05-25T12:04:11.129 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.095+0000 7f04a8e77700 1 -- 172.21.15.102:0/958459200 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f048c003270 con 0x7f04a41321f0 2021-05-25T12:04:11.130 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.095+0000 7f04a8e77700 1 -- 172.21.15.102:0/958459200 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4253956722 0 0) 0x7f04a4097960 con 0x7f04a4096810 2021-05-25T12:04:11.130 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.095+0000 7f04a8e77700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:04:11.130 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.095+0000 7f04a8e77700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:04:11.130 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.095+0000 7f04a8e77700 10 monclient(hunting): my global_id is 184175 2021-05-25T12:04:11.131 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.095+0000 7f04a8e77700 1 -- 172.21.15.102:0/958459200 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04a4069b10 con 0x7f04a4096810 2021-05-25T12:04:11.131 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.095+0000 7f04a8e77700 1 -- 172.21.15.102:0/958459200 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 986037355 0 0) 0x7f04a40976c0 con 0x7f04a4128d10 2021-05-25T12:04:11.131 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.095+0000 7f04a8e77700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:04:11.131 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.095+0000 7f04a8e77700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:04:11.131 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.095+0000 7f04a8e77700 10 monclient(hunting): my global_id is 95365 2021-05-25T12:04:11.132 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.095+0000 7f04a8e77700 1 -- 172.21.15.102:0/958459200 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04a4097960 con 0x7f04a4128d10 2021-05-25T12:04:11.132 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.095+0000 7f04a8e77700 1 -- 172.21.15.102:0/958459200 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2597274361 0 0) 0x7f048c003270 con 0x7f04a41321f0 2021-05-25T12:04:11.132 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.095+0000 7f04a8e77700 1 -- 172.21.15.102:0/958459200 >> v1:172.21.15.102:6790/0 conn(0x7f04a4096810 legacy=0x7f04a4130ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:11.132 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.095+0000 7f04a8e77700 1 -- 172.21.15.102:0/958459200 >> v1:172.21.15.181:6789/0 conn(0x7f04a4128d10 legacy=0x7f04a4068d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:11.133 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.095+0000 7f04a8e77700 10 monclient: _finish_hunting 0 2021-05-25T12:04:11.133 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.095+0000 7f04a8e77700 1 monclient: found mon.noname-a 2021-05-25T12:04:11.133 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.095+0000 7f04a8e77700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:04:11.133 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.095+0000 7f04a8e77700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T12:04:11.133 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.095+0000 7f04a8e77700 1 -- 172.21.15.102:0/958459200 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04a4097c00 con 0x7f04a41321f0 2021-05-25T12:04:11.134 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.095+0000 7f04a8e77700 10 monclient: _finish_auth 0 2021-05-25T12:04:11.134 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.095+0000 7f04a8e77700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:04:11.134 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.095+0000 7f04a8e77700 1 -- 172.21.15.102:0/958459200 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f04a0001990 con 0x7f04a41321f0 2021-05-25T12:04:11.134 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.095+0000 7f04ac0dd700 5 monclient: authenticate success, global_id 174450 2021-05-25T12:04:11.134 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.095+0000 7f04ac0dd700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:04:11.135 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.095+0000 7f04a8e77700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:11.135 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.095+0000 7f04a8e77700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:04:11.135 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.095+0000 7f04a8e77700 10 monclient: dump: 2021-05-25T12:04:11.135 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:11.136 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:11.136 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:11.136 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:11.136 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:11.136 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:11.137 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:11.137 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:11.137 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:11.137 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:11.137 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.095+0000 7f04ac0dd700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:04:11.138 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.095+0000 7f04a8e77700 1 -- 172.21.15.102:0/958459200 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f04a0001c00 con 0x7f04a41321f0 2021-05-25T12:04:11.138 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.095+0000 7f04a8e77700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:04:11.138 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.095+0000 7f04ac0dd700 10 monclient: get_monmap_and_config success 2021-05-25T12:04:11.138 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.095+0000 7f04a8e77700 1 -- 172.21.15.102:0/958459200 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f04a0001f90 con 0x7f04a41321f0 2021-05-25T12:04:11.139 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.095+0000 7f04a8e77700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:11.139 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.095+0000 7f04ac0dd700 4 set_mon_vals no callback set 2021-05-25T12:04:11.139 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.095+0000 7f04a8e77700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:04:11.139 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.095+0000 7f04a8e77700 10 monclient: dump: 2021-05-25T12:04:11.140 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:11.140 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:11.140 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:11.140 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:11.140 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:11.141 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:11.141 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:11.141 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:11.141 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:11.142 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:11.142 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.096+0000 7f04ac0dd700 10 monclient: shutdown 2021-05-25T12:04:11.142 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.096+0000 7f04ac0dd700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:04:11.142 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.096+0000 7f04ac0dd700 1 -- 172.21.15.102:0/958459200 >> v1:172.21.15.102:6789/0 conn(0x7f04a41321f0 legacy=0x7f04a41345e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:11.143 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.096+0000 7f04ac0dd700 1 -- 172.21.15.102:0/958459200 shutdown_connections 2021-05-25T12:04:11.143 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.096+0000 7f04ac0dd700 1 -- 172.21.15.102:0/958459200 >> 172.21.15.102:0/958459200 conn(0x7f04a409b920 msgr2=0x7f04a409bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:04:11.143 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.096+0000 7f04ac0dd700 1 -- 172.21.15.102:0/958459200 shutdown_connections 2021-05-25T12:04:11.143 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.096+0000 7f04ac0dd700 1 -- 172.21.15.102:0/958459200 wait complete. 2021-05-25T12:04:11.143 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.096+0000 7f04ac0dd700 10 monclient: build_initial_monmap 2021-05-25T12:04:11.144 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.096+0000 7f04ac0dd700 10 monclient: monmap: 2021-05-25T12:04:11.144 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:04:11.144 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:04:11.144 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:04:11.144 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:04:11.145 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:04:11.145 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:04:11.145 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:04:11.145 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:04:11.145 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:04:11.146 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:11.146 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.096+0000 7f04ac0dd700 1 librados: starting msgr at 2021-05-25T12:04:11.146 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.096+0000 7f04ac0dd700 1 librados: starting objecter 2021-05-25T12:04:11.146 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.097+0000 7f04ac0dd700 1 Processor -- start 2021-05-25T12:04:11.147 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.097+0000 7f04ac0dd700 1 -- start start 2021-05-25T12:04:11.147 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.097+0000 7f04ac0dd700 1 librados: setting wanted keys 2021-05-25T12:04:11.147 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.097+0000 7f04ac0dd700 1 librados: calling monclient init 2021-05-25T12:04:11.147 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.097+0000 7f04ac0dd700 10 monclient: init 2021-05-25T12:04:11.147 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.097+0000 7f04ac0dd700 10 monclient: _reopen_session rank -1 2021-05-25T12:04:11.148 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.097+0000 7f04ac0dd700 10 monclient: _add_conns ranks=[1,0,2] 2021-05-25T12:04:11.148 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.097+0000 7f04ac0dd700 10 monclient(hunting): picked mon.noname-b con 0x7f04a4096810 addr v1:172.21.15.181:6789/0 2021-05-25T12:04:11.148 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.097+0000 7f04ac0dd700 10 monclient(hunting): picked mon.noname-a con 0x7f04a4128d10 addr v1:172.21.15.102:6789/0 2021-05-25T12:04:11.148 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.097+0000 7f04ac0dd700 10 monclient(hunting): picked mon.noname-c con 0x7f04a41321f0 addr v1:172.21.15.102:6790/0 2021-05-25T12:04:11.148 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.097+0000 7f04ac0dd700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04a412f7f0 con 0x7f04a4128d10 2021-05-25T12:04:11.149 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.097+0000 7f04ac0dd700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04a412fa90 con 0x7f04a4096810 2021-05-25T12:04:11.149 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.097+0000 7f04ac0dd700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04a4130560 con 0x7f04a41321f0 2021-05-25T12:04:11.149 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.097+0000 7f04ac0dd700 10 monclient(hunting): _renew_subs 2021-05-25T12:04:11.149 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.097+0000 7f04ac0dd700 10 monclient(hunting): authenticate will time out at 1051.694483s 2021-05-25T12:04:11.149 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.097+0000 7f04aa67a700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f04a41321f0 0x7f04a4142db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:55592/0 (socket says 172.21.15.102:55592) 2021-05-25T12:04:11.150 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.097+0000 7f04a9678700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f04a4128d10 0x7f04a413f680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60712/0 (socket says 172.21.15.102:60712) 2021-05-25T12:04:11.150 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.097+0000 7f04aa67a700 1 -- 172.21.15.102:0/2920530150 learned_addr learned my addr 172.21.15.102:0/2920530150 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:04:11.150 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.097+0000 7f0491ffb700 10 client.?.objecter ms_handle_connect 0x7f04a41321f0 2021-05-25T12:04:11.150 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.097+0000 7f0491ffb700 10 client.?.objecter resend_mon_ops 2021-05-25T12:04:11.150 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.097+0000 7f0491ffb700 10 client.?.objecter ms_handle_connect 0x7f04a4128d10 2021-05-25T12:04:11.151 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.097+0000 7f0491ffb700 10 client.?.objecter resend_mon_ops 2021-05-25T12:04:11.151 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.097+0000 7f0491ffb700 10 client.?.objecter ms_handle_connect 0x7f04a4096810 2021-05-25T12:04:11.151 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.097+0000 7f0491ffb700 10 client.?.objecter resend_mon_ops 2021-05-25T12:04:11.151 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.097+0000 7f0491ffb700 1 -- 172.21.15.102:0/2920530150 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 959269266 0 0) 0x7f04a4130560 con 0x7f04a41321f0 2021-05-25T12:04:11.152 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.097+0000 7f0491ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:04:11.152 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.097+0000 7f0491ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:04:11.152 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.097+0000 7f0491ffb700 10 monclient(hunting): my global_id is 184178 2021-05-25T12:04:11.152 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.097+0000 7f0491ffb700 1 -- 172.21.15.102:0/2920530150 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0484004c10 con 0x7f04a41321f0 2021-05-25T12:04:11.152 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.097+0000 7f0491ffb700 1 -- 172.21.15.102:0/2920530150 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3639791273 0 0) 0x7f04a412f7f0 con 0x7f04a4128d10 2021-05-25T12:04:11.153 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.097+0000 7f0491ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:04:11.153 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.097+0000 7f0491ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:04:11.153 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.097+0000 7f0491ffb700 10 monclient(hunting): my global_id is 174453 2021-05-25T12:04:11.153 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.097+0000 7f0491ffb700 1 -- 172.21.15.102:0/2920530150 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04a4130560 con 0x7f04a4128d10 2021-05-25T12:04:11.153 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.097+0000 7f0491ffb700 1 -- 172.21.15.102:0/2920530150 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2368194045 0 0) 0x7f04a412fa90 con 0x7f04a4096810 2021-05-25T12:04:11.154 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.097+0000 7f0491ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:04:11.154 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.097+0000 7f0491ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:04:11.154 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.097+0000 7f0491ffb700 10 monclient(hunting): my global_id is 95368 2021-05-25T12:04:11.154 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.097+0000 7f0491ffb700 1 -- 172.21.15.102:0/2920530150 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04a412f7f0 con 0x7f04a4096810 2021-05-25T12:04:11.155 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.098+0000 7f0491ffb700 1 -- 172.21.15.102:0/2920530150 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1000079277 0 0) 0x7f04a4130560 con 0x7f04a4128d10 2021-05-25T12:04:11.155 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.098+0000 7f0491ffb700 1 -- 172.21.15.102:0/2920530150 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f04a412fa90 con 0x7f04a4128d10 2021-05-25T12:04:11.155 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.098+0000 7f0491ffb700 1 -- 172.21.15.102:0/2920530150 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f04940018c0 con 0x7f04a4128d10 2021-05-25T12:04:11.155 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.098+0000 7f0491ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:11.155 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.098+0000 7f0491ffb700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:04:11.156 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.098+0000 7f0491ffb700 10 monclient(hunting): dump: 2021-05-25T12:04:11.156 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:11.156 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:11.156 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:11.156 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:11.157 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:11.157 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:11.157 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:11.157 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:11.157 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:11.158 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:11.158 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.098+0000 7f0491ffb700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:04:11.158 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.098+0000 7f0491ffb700 1 -- 172.21.15.102:0/2920530150 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3653619867 0 0) 0x7f0484004c10 con 0x7f04a41321f0 2021-05-25T12:04:11.158 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.098+0000 7f0491ffb700 1 -- 172.21.15.102:0/2920530150 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f04a4130560 con 0x7f04a41321f0 2021-05-25T12:04:11.159 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.098+0000 7f0491ffb700 1 -- 172.21.15.102:0/2920530150 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f04a0001950 con 0x7f04a41321f0 2021-05-25T12:04:11.159 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.098+0000 7f0491ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:11.159 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.098+0000 7f0491ffb700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2021-05-25T12:04:11.159 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.098+0000 7f0491ffb700 10 monclient(hunting): dump: 2021-05-25T12:04:11.159 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:11.160 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:11.160 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:11.160 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:11.160 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:11.160 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:11.161 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:11.161 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:11.161 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:11.161 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:11.162 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.098+0000 7f0491ffb700 1 -- 172.21.15.102:0/2920530150 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2460582246 0 0) 0x7f04a412f7f0 con 0x7f04a4096810 2021-05-25T12:04:11.162 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.098+0000 7f0491ffb700 1 -- 172.21.15.102:0/2920530150 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0484004c10 con 0x7f04a4096810 2021-05-25T12:04:11.162 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.098+0000 7f0491ffb700 1 -- 172.21.15.102:0/2920530150 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f04980018c0 con 0x7f04a4096810 2021-05-25T12:04:11.162 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.098+0000 7f0491ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:11.162 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.098+0000 7f0491ffb700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2021-05-25T12:04:11.163 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.098+0000 7f0491ffb700 10 monclient(hunting): dump: 2021-05-25T12:04:11.163 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:11.163 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:11.163 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:11.164 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:11.164 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:11.164 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:11.164 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:11.164 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:11.165 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:11.165 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:11.165 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.098+0000 7f0491ffb700 1 -- 172.21.15.102:0/2920530150 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1362433253 0 0) 0x7f04a412fa90 con 0x7f04a4128d10 2021-05-25T12:04:11.165 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.098+0000 7f0491ffb700 1 -- 172.21.15.102:0/2920530150 >> v1:172.21.15.102:6790/0 conn(0x7f04a41321f0 legacy=0x7f04a4142db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:11.165 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.098+0000 7f0491ffb700 1 -- 172.21.15.102:0/2920530150 >> v1:172.21.15.181:6789/0 conn(0x7f04a4096810 legacy=0x7f04a412fe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:11.166 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.098+0000 7f0491ffb700 10 monclient: _finish_hunting 0 2021-05-25T12:04:11.166 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.098+0000 7f0491ffb700 1 monclient: found mon.a 2021-05-25T12:04:11.166 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.098+0000 7f0491ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:04:11.166 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.098+0000 7f0491ffb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:04:11.167 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.098+0000 7f0491ffb700 1 -- 172.21.15.102:0/2920530150 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04a412d8b0 con 0x7f04a4128d10 2021-05-25T12:04:11.167 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.098+0000 7f0491ffb700 10 monclient: _finish_auth 0 2021-05-25T12:04:11.167 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.098+0000 7f0491ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:04:11.167 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.098+0000 7f04ac0dd700 5 monclient: authenticate success, global_id 174453 2021-05-25T12:04:11.167 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.098+0000 7f04ac0dd700 10 monclient: _renew_subs 2021-05-25T12:04:11.168 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.098+0000 7f04ac0dd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:04:11.168 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.098+0000 7f04ac0dd700 1 -- 172.21.15.102:0/2920530150 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04a4131230 con 0x7f04a4128d10 2021-05-25T12:04:11.168 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.098+0000 7f0491ffb700 1 -- 172.21.15.102:0/2920530150 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0494004530 con 0x7f04a4128d10 2021-05-25T12:04:11.168 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.098+0000 7f0491ffb700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:04:11.169 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.098+0000 7f0491ffb700 1 -- 172.21.15.102:0/2920530150 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f04940048c0 con 0x7f04a4128d10 2021-05-25T12:04:11.169 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.098+0000 7f0491ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:11.169 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.098+0000 7f0491ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:04:11.169 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.098+0000 7f0491ffb700 10 monclient: dump: 2021-05-25T12:04:11.169 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:11.170 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:11.170 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:11.170 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:11.170 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:11.170 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:11.171 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:11.171 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:11.171 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:11.171 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:11.172 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.098+0000 7f04ac0dd700 10 client.174453.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:04:11.172 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.098+0000 7f0493fff700 4 set_mon_vals no callback set 2021-05-25T12:04:11.172 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.098+0000 7f04ac0dd700 10 monclient: _renew_subs 2021-05-25T12:04:11.172 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.098+0000 7f04ac0dd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:04:11.172 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.098+0000 7f04ac0dd700 1 -- 172.21.15.102:0/2920530150 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f04a4144690 con 0x7f04a4128d10 2021-05-25T12:04:11.173 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.098+0000 7f04ac0dd700 1 librados: init done 2021-05-25T12:04:11.173 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.099+0000 7f0491ffb700 1 -- 172.21.15.102:0/2920530150 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f0494004a20 con 0x7f04a4128d10 2021-05-25T12:04:11.173 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.099+0000 7f0491ffb700 1 -- 172.21.15.102:0/2920530150 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(127..127 src has 1..127) v4 ==== 5218+0+0 (unknown 2630921626 0 0) 0x7f049403d300 con 0x7f04a4128d10 2021-05-25T12:04:11.173 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.099+0000 7f0491ffb700 10 client.174453.objecter ms_dispatch 0x7f04a4001550 osd_map(127..127 src has 1..127) v4 2021-05-25T12:04:11.173 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.099+0000 7f0491ffb700 3 client.174453.objecter handle_osd_map got epochs [127,127] > 0 2021-05-25T12:04:11.174 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.099+0000 7f0491ffb700 3 client.174453.objecter handle_osd_map decoding full epoch 127 2021-05-25T12:04:11.174 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.099+0000 7f04ac0dd700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:04:11.174 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.099+0000 7f04ac0dd700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:04:11.174 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.099+0000 7f04ac0dd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:04:11.175 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.099+0000 7f0491ffb700 20 client.174453.objecter dump_active .. 0 homeless 2021-05-25T12:04:11.175 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.099+0000 7f04ac0dd700 1 -- 172.21.15.102:0/2920530150 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04a4095800 con 0x7f04a4128d10 2021-05-25T12:04:11.175 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.099+0000 7f04a9e79700 10 monclient: get_auth_request con 0x7f04840334a0 auth_method 0 2021-05-25T12:04:11.175 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.099+0000 7f0491ffb700 10 client.174453.objecter ms_handle_connect 0x7f04840334a0 2021-05-25T12:04:11.176 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.103+0000 7f0491ffb700 1 -- 172.21.15.102:0/2920530150 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7f0494012940 con 0x7f04a4128d10 2021-05-25T12:04:11.176 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.103+0000 7f0491ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:04:11.176 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.103+0000 7f0491ffb700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:04:11.318 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.311+0000 7f04ac0dd700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2021-05-25T12:04:11.318 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.311+0000 7f04ac0dd700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2021-05-25T12:04:11.319 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.311+0000 7f04ac0dd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:04:11.319 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.311+0000 7f04ac0dd700 1 -- 172.21.15.102:0/2920530150 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f04a4095ae0 con 0x7f04a4128d10 2021-05-25T12:04:11.319 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.311+0000 7f0491ffb700 1 -- 172.21.15.102:0/2920530150 <== mon.0 v1:172.21.15.102:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1342 (unknown 254746657 0 663237568) 0x7f0494012760 con 0x7f04a4128d10 2021-05-25T12:04:11.319 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.311+0000 7f0491ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2021-05-25T12:04:11.319 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.311+0000 7f0491ffb700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:04:11.320 INFO:teuthology.orchestra.run.smithi102.stdout:{"election_epoch":38,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"b","quorum_age":3,"features":{"quorum_con":"4540138303579357183","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"20ee91b6-8935-4b98-b868-85c03789ccc4","modified":"2021-05-25T11:59:11.011522Z","created":"2021-05-25T11:59:11.011522Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6790","nonce":0}]},"addr":"172.21.15.102:6790/0","public_addr":"172.21.15.102:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2021-05-25T12:04:11.322 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.312+0000 7f04ac0dd700 10 librados: watch_flush enter 2021-05-25T12:04:11.322 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.313+0000 7f04ac0dd700 10 librados: watch_flush exit 2021-05-25T12:04:11.322 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.313+0000 7f04ac0dd700 20 client.174453.objecter shutdown clearing up homeless session... 2021-05-25T12:04:11.323 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.313+0000 7f04ac0dd700 10 client.174453.objecter successfully canceled tick 2021-05-25T12:04:11.323 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.313+0000 7f04ac0dd700 1 -- 172.21.15.102:0/2920530150 >> v1:172.21.15.181:6812/29582 conn(0x7f04840334a0 legacy=0x7f0484035960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:11.323 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.313+0000 7f04ac0dd700 10 monclient: shutdown 2021-05-25T12:04:11.323 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.313+0000 7f04ac0dd700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:04:11.324 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.313+0000 7f04ac0dd700 1 -- 172.21.15.102:0/2920530150 >> v1:172.21.15.102:6789/0 conn(0x7f04a4128d10 legacy=0x7f04a413f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:11.324 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.313+0000 7f04ac0dd700 1 -- 172.21.15.102:0/2920530150 shutdown_connections 2021-05-25T12:04:11.324 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.313+0000 7f04ac0dd700 1 -- 172.21.15.102:0/2920530150 >> 172.21.15.102:0/2920530150 conn(0x7f04a409b920 msgr2=0x7f04a4062320 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:04:11.324 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.313+0000 7f04ac0dd700 1 -- 172.21.15.102:0/2920530150 shutdown_connections 2021-05-25T12:04:11.324 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.313+0000 7f04ac0dd700 1 -- 172.21.15.102:0/2920530150 wait complete. 2021-05-25T12:04:11.325 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.313+0000 7f04ac0dd700 1 librados: shutdown 2021-05-25T12:04:11.326 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2021-05-25T12:04:11.326 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2021-05-25T12:04:11.327 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2021-05-25T12:04:11.441 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.439+0000 7fb833c91700 10 monclient: get_monmap_and_config 2021-05-25T12:04:11.441 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.439+0000 7fb833c91700 10 monclient: build_initial_monmap 2021-05-25T12:04:11.442 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.439+0000 7fb833c91700 10 monclient: monmap: 2021-05-25T12:04:11.442 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:04:11.442 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:11.442 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:04:11.440671+0000 2021-05-25T12:04:11.443 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:04:11.440671+0000 2021-05-25T12:04:11.443 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:04:11.443 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:11.443 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:04:11.443 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:04:11.444 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:04:11.444 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:11.444 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.440+0000 7fb833c91700 1 Processor -- start 2021-05-25T12:04:11.444 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.440+0000 7fb833c91700 1 -- start start 2021-05-25T12:04:11.444 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.440+0000 7fb833c91700 10 monclient: init 2021-05-25T12:04:11.456 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.440+0000 7fb833c91700 10 monclient: _reopen_session rank -1 2021-05-25T12:04:11.457 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.440+0000 7fb833c91700 10 monclient: _add_conns ranks=[0,1,2] 2021-05-25T12:04:11.457 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.440+0000 7fb833c91700 10 monclient(hunting): picked mon.noname-a con 0x7fb82c090990 addr v1:172.21.15.102:6789/0 2021-05-25T12:04:11.457 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.440+0000 7fb833c91700 10 monclient(hunting): picked mon.noname-c con 0x7fb82c090d70 addr v1:172.21.15.181:6789/0 2021-05-25T12:04:11.457 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.440+0000 7fb833c91700 10 monclient(hunting): picked mon.noname-b con 0x7fb82c12e330 addr v1:172.21.15.102:6790/0 2021-05-25T12:04:11.458 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.440+0000 7fb833c91700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb82c096fa0 con 0x7fb82c090990 2021-05-25T12:04:11.458 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.440+0000 7fb833c91700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb82c0926a0 con 0x7fb82c090d70 2021-05-25T12:04:11.458 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.440+0000 7fb833c91700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb82c092940 con 0x7fb82c12e330 2021-05-25T12:04:11.458 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.440+0000 7fb833c91700 10 monclient(hunting): _renew_subs 2021-05-25T12:04:11.459 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.440+0000 7fb833c91700 10 monclient(hunting): authenticate will time out at 1052.037933s 2021-05-25T12:04:11.459 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.441+0000 7fb83222e700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fb82c12e330 0x7fb82c130720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:55600/0 (socket says 172.21.15.102:55600) 2021-05-25T12:04:11.459 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.441+0000 7fb831a2d700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fb82c090990 0x7fb82c093240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60720/0 (socket says 172.21.15.102:60720) 2021-05-25T12:04:11.459 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.441+0000 7fb83222e700 1 -- 172.21.15.102:0/1169811227 learned_addr learned my addr 172.21.15.102:0/1169811227 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:04:11.459 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.441+0000 7fb830a2b700 1 -- 172.21.15.102:0/1169811227 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1803982868 0 0) 0x7fb82c096fa0 con 0x7fb82c090990 2021-05-25T12:04:11.460 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.441+0000 7fb830a2b700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:04:11.460 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.441+0000 7fb830a2b700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:04:11.460 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.441+0000 7fb830a2b700 10 monclient(hunting): my global_id is 174456 2021-05-25T12:04:11.460 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.441+0000 7fb830a2b700 1 -- 172.21.15.102:0/1169811227 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb814003270 con 0x7fb82c090990 2021-05-25T12:04:11.461 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.441+0000 7fb830a2b700 1 -- 172.21.15.102:0/1169811227 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 532571133 0 0) 0x7fb82c092940 con 0x7fb82c12e330 2021-05-25T12:04:11.461 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.441+0000 7fb830a2b700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:04:11.461 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.441+0000 7fb830a2b700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:04:11.461 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.441+0000 7fb830a2b700 10 monclient(hunting): my global_id is 184181 2021-05-25T12:04:11.462 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.441+0000 7fb830a2b700 1 -- 172.21.15.102:0/1169811227 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb82c096fa0 con 0x7fb82c12e330 2021-05-25T12:04:11.462 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.441+0000 7fb830a2b700 1 -- 172.21.15.102:0/1169811227 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2642557285 0 0) 0x7fb82c0926a0 con 0x7fb82c090d70 2021-05-25T12:04:11.462 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.441+0000 7fb830a2b700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:04:11.462 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.441+0000 7fb830a2b700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:04:11.462 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.441+0000 7fb830a2b700 10 monclient(hunting): my global_id is 95371 2021-05-25T12:04:11.463 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.441+0000 7fb830a2b700 1 -- 172.21.15.102:0/1169811227 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb82c092940 con 0x7fb82c090d70 2021-05-25T12:04:11.463 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.441+0000 7fb830a2b700 1 -- 172.21.15.102:0/1169811227 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2330046561 0 0) 0x7fb814003270 con 0x7fb82c090990 2021-05-25T12:04:11.463 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.441+0000 7fb830a2b700 1 -- 172.21.15.102:0/1169811227 >> v1:172.21.15.102:6790/0 conn(0x7fb82c12e330 legacy=0x7fb82c130720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:11.463 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.441+0000 7fb830a2b700 1 -- 172.21.15.102:0/1169811227 >> v1:172.21.15.181:6789/0 conn(0x7fb82c090d70 legacy=0x7fb82c091200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:11.464 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.441+0000 7fb830a2b700 10 monclient: _finish_hunting 0 2021-05-25T12:04:11.464 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.441+0000 7fb830a2b700 1 monclient: found mon.noname-a 2021-05-25T12:04:11.464 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.441+0000 7fb830a2b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:04:11.464 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.441+0000 7fb830a2b700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T12:04:11.464 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.441+0000 7fb830a2b700 1 -- 172.21.15.102:0/1169811227 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb82c092be0 con 0x7fb82c090990 2021-05-25T12:04:11.465 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.441+0000 7fb830a2b700 10 monclient: _finish_auth 0 2021-05-25T12:04:11.465 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.441+0000 7fb830a2b700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:04:11.465 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.441+0000 7fb833c91700 5 monclient: authenticate success, global_id 174456 2021-05-25T12:04:11.465 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.441+0000 7fb830a2b700 1 -- 172.21.15.102:0/1169811227 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fb820001610 con 0x7fb82c090990 2021-05-25T12:04:11.465 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.441+0000 7fb830a2b700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:11.466 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.441+0000 7fb830a2b700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:04:11.466 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.441+0000 7fb830a2b700 10 monclient: dump: 2021-05-25T12:04:11.466 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:11.466 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:11.467 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:11.467 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:11.467 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:11.467 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:11.467 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:11.468 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:11.468 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:11.468 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:11.468 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.441+0000 7fb833c91700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:04:11.468 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.441+0000 7fb830a2b700 1 -- 172.21.15.102:0/1169811227 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb820001830 con 0x7fb82c090990 2021-05-25T12:04:11.469 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.441+0000 7fb830a2b700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:04:11.469 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.441+0000 7fb830a2b700 1 -- 172.21.15.102:0/1169811227 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fb820001b40 con 0x7fb82c090990 2021-05-25T12:04:11.469 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.441+0000 7fb833c91700 10 monclient: get_monmap_and_config success 2021-05-25T12:04:11.469 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.441+0000 7fb833c91700 4 set_mon_vals no callback set 2021-05-25T12:04:11.469 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.441+0000 7fb830a2b700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:11.470 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.441+0000 7fb830a2b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:04:11.470 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.441+0000 7fb830a2b700 10 monclient: dump: 2021-05-25T12:04:11.470 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:11.470 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:11.471 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:11.471 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:11.471 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:11.471 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:11.471 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:11.472 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:11.472 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:11.472 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:11.472 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.442+0000 7fb833c91700 10 monclient: shutdown 2021-05-25T12:04:11.472 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.442+0000 7fb833c91700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:04:11.473 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.442+0000 7fb833c91700 1 -- 172.21.15.102:0/1169811227 >> v1:172.21.15.102:6789/0 conn(0x7fb82c090990 legacy=0x7fb82c093240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:11.473 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.442+0000 7fb833c91700 1 -- 172.21.15.102:0/1169811227 shutdown_connections 2021-05-25T12:04:11.473 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.442+0000 7fb833c91700 1 -- 172.21.15.102:0/1169811227 >> 172.21.15.102:0/1169811227 conn(0x7fb82c09b370 msgr2=0x7fb82c097430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:04:11.473 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.442+0000 7fb833c91700 1 -- 172.21.15.102:0/1169811227 shutdown_connections 2021-05-25T12:04:11.473 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.442+0000 7fb833c91700 1 -- 172.21.15.102:0/1169811227 wait complete. 2021-05-25T12:04:11.474 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.442+0000 7fb833c91700 10 monclient: build_initial_monmap 2021-05-25T12:04:11.474 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.442+0000 7fb833c91700 10 monclient: monmap: 2021-05-25T12:04:11.474 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:04:11.474 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:04:11.475 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:04:11.475 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:04:11.475 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:04:11.475 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:04:11.475 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:04:11.476 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:04:11.476 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:04:11.476 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:11.476 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.442+0000 7fb833c91700 1 librados: starting msgr at 2021-05-25T12:04:11.477 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.442+0000 7fb833c91700 1 librados: starting objecter 2021-05-25T12:04:11.477 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.443+0000 7fb833c91700 1 Processor -- start 2021-05-25T12:04:11.477 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.443+0000 7fb833c91700 1 -- start start 2021-05-25T12:04:11.477 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.443+0000 7fb833c91700 1 librados: setting wanted keys 2021-05-25T12:04:11.477 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.443+0000 7fb833c91700 1 librados: calling monclient init 2021-05-25T12:04:11.478 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.443+0000 7fb833c91700 10 monclient: init 2021-05-25T12:04:11.478 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.443+0000 7fb833c91700 10 monclient: _reopen_session rank -1 2021-05-25T12:04:11.478 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.443+0000 7fb833c91700 10 monclient: _add_conns ranks=[2,0,1] 2021-05-25T12:04:11.478 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.443+0000 7fb833c91700 10 monclient(hunting): picked mon.noname-c con 0x7fb82c090990 addr v1:172.21.15.102:6790/0 2021-05-25T12:04:11.478 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.443+0000 7fb833c91700 10 monclient(hunting): picked mon.noname-a con 0x7fb82c090d70 addr v1:172.21.15.102:6789/0 2021-05-25T12:04:11.479 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.443+0000 7fb833c91700 10 monclient(hunting): picked mon.noname-b con 0x7fb82c12e330 addr v1:172.21.15.181:6789/0 2021-05-25T12:04:11.479 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.443+0000 7fb833c91700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb82c068b60 con 0x7fb82c090d70 2021-05-25T12:04:11.479 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.443+0000 7fb833c91700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb82c1398c0 con 0x7fb82c12e330 2021-05-25T12:04:11.479 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.443+0000 7fb833c91700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb82c112d90 con 0x7fb82c090990 2021-05-25T12:04:11.480 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.443+0000 7fb833c91700 10 monclient(hunting): _renew_subs 2021-05-25T12:04:11.480 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.443+0000 7fb833c91700 10 monclient(hunting): authenticate will time out at 1052.040483s 2021-05-25T12:04:11.480 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.443+0000 7fb831a2d700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fb82c090990 0x7fb82c138aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:55602/0 (socket says 172.21.15.102:55602) 2021-05-25T12:04:11.480 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.443+0000 7fb831a2d700 1 -- 172.21.15.102:0/2854213716 learned_addr learned my addr 172.21.15.102:0/2854213716 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:04:11.481 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.443+0000 7fb819ffb700 10 client.?.objecter ms_handle_connect 0x7fb82c090d70 2021-05-25T12:04:11.481 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.443+0000 7fb819ffb700 10 client.?.objecter resend_mon_ops 2021-05-25T12:04:11.481 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.443+0000 7fb819ffb700 10 client.?.objecter ms_handle_connect 0x7fb82c090990 2021-05-25T12:04:11.481 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.443+0000 7fb819ffb700 10 client.?.objecter resend_mon_ops 2021-05-25T12:04:11.481 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.443+0000 7fb819ffb700 1 -- 172.21.15.102:0/2854213716 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3459908155 0 0) 0x7fb82c112d90 con 0x7fb82c090990 2021-05-25T12:04:11.482 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.443+0000 7fb819ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:04:11.482 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.443+0000 7fb819ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:04:11.482 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.443+0000 7fb819ffb700 10 monclient(hunting): my global_id is 184184 2021-05-25T12:04:11.482 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.443+0000 7fb819ffb700 1 -- 172.21.15.102:0/2854213716 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb80c004c10 con 0x7fb82c090990 2021-05-25T12:04:11.482 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.443+0000 7fb819ffb700 1 -- 172.21.15.102:0/2854213716 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2149341043 0 0) 0x7fb82c068b60 con 0x7fb82c090d70 2021-05-25T12:04:11.483 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.443+0000 7fb819ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:04:11.483 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.443+0000 7fb819ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:04:11.483 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.443+0000 7fb819ffb700 10 monclient(hunting): my global_id is 174459 2021-05-25T12:04:11.483 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.443+0000 7fb819ffb700 1 -- 172.21.15.102:0/2854213716 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb82c112d90 con 0x7fb82c090d70 2021-05-25T12:04:11.484 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.443+0000 7fb819ffb700 10 client.?.objecter ms_handle_connect 0x7fb82c12e330 2021-05-25T12:04:11.484 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.443+0000 7fb819ffb700 10 client.?.objecter resend_mon_ops 2021-05-25T12:04:11.484 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.443+0000 7fb819ffb700 1 -- 172.21.15.102:0/2854213716 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2374579729 0 0) 0x7fb82c1398c0 con 0x7fb82c12e330 2021-05-25T12:04:11.484 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.443+0000 7fb819ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:04:11.484 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.443+0000 7fb819ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:04:11.485 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.443+0000 7fb819ffb700 10 monclient(hunting): my global_id is 95374 2021-05-25T12:04:11.485 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.443+0000 7fb819ffb700 1 -- 172.21.15.102:0/2854213716 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb82c068b60 con 0x7fb82c12e330 2021-05-25T12:04:11.485 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.443+0000 7fb819ffb700 1 -- 172.21.15.102:0/2854213716 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1960986830 0 0) 0x7fb80c004c10 con 0x7fb82c090990 2021-05-25T12:04:11.485 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.444+0000 7fb819ffb700 1 -- 172.21.15.102:0/2854213716 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb82c1398c0 con 0x7fb82c090990 2021-05-25T12:04:11.485 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.444+0000 7fb819ffb700 1 -- 172.21.15.102:0/2854213716 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fb820001510 con 0x7fb82c090990 2021-05-25T12:04:11.486 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.444+0000 7fb819ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:11.486 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.444+0000 7fb819ffb700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:04:11.486 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.444+0000 7fb819ffb700 10 monclient(hunting): dump: 2021-05-25T12:04:11.486 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:11.487 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:11.487 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:11.487 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:11.487 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:11.487 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:11.488 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:11.488 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:11.488 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:11.488 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:11.489 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.444+0000 7fb819ffb700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:04:11.489 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.444+0000 7fb819ffb700 1 -- 172.21.15.102:0/2854213716 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4175884400 0 0) 0x7fb82c112d90 con 0x7fb82c090d70 2021-05-25T12:04:11.489 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.444+0000 7fb819ffb700 1 -- 172.21.15.102:0/2854213716 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb80c004c10 con 0x7fb82c090d70 2021-05-25T12:04:11.489 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.444+0000 7fb819ffb700 1 -- 172.21.15.102:0/2854213716 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fb81c0018c0 con 0x7fb82c090d70 2021-05-25T12:04:11.489 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.444+0000 7fb819ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:11.490 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.444+0000 7fb819ffb700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2021-05-25T12:04:11.490 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.444+0000 7fb819ffb700 10 monclient(hunting): dump: 2021-05-25T12:04:11.490 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:11.490 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:11.490 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:11.491 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:11.491 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:11.491 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:11.491 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:11.492 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:11.492 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:11.492 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:11.492 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.444+0000 7fb819ffb700 1 -- 172.21.15.102:0/2854213716 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3241629262 0 0) 0x7fb82c1398c0 con 0x7fb82c090990 2021-05-25T12:04:11.492 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.444+0000 7fb819ffb700 1 -- 172.21.15.102:0/2854213716 >> v1:172.21.15.181:6789/0 conn(0x7fb82c12e330 legacy=0x7fb82c140df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:11.493 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.444+0000 7fb819ffb700 1 -- 172.21.15.102:0/2854213716 >> v1:172.21.15.102:6789/0 conn(0x7fb82c090d70 legacy=0x7fb82c1391b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:11.493 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.444+0000 7fb819ffb700 10 monclient: _finish_hunting 0 2021-05-25T12:04:11.493 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.444+0000 7fb819ffb700 1 monclient: found mon.c 2021-05-25T12:04:11.493 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.444+0000 7fb819ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:04:11.493 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.444+0000 7fb819ffb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:04:11.494 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.444+0000 7fb819ffb700 1 -- 172.21.15.102:0/2854213716 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb82c091fe0 con 0x7fb82c090990 2021-05-25T12:04:11.494 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.444+0000 7fb819ffb700 10 monclient: _finish_auth 0 2021-05-25T12:04:11.494 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.444+0000 7fb819ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:04:11.494 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.444+0000 7fb833c91700 5 monclient: authenticate success, global_id 184184 2021-05-25T12:04:11.495 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.444+0000 7fb833c91700 10 monclient: _renew_subs 2021-05-25T12:04:11.495 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.444+0000 7fb833c91700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:04:11.495 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.444+0000 7fb833c91700 1 -- 172.21.15.102:0/2854213716 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb82c1118c0 con 0x7fb82c090990 2021-05-25T12:04:11.495 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.444+0000 7fb819ffb700 1 -- 172.21.15.102:0/2854213716 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb8200047e0 con 0x7fb82c090990 2021-05-25T12:04:11.495 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.444+0000 7fb819ffb700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:04:11.496 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.444+0000 7fb819ffb700 1 -- 172.21.15.102:0/2854213716 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fb820004af0 con 0x7fb82c090990 2021-05-25T12:04:11.496 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.444+0000 7fb819ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:11.496 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.444+0000 7fb819ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:04:11.496 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.444+0000 7fb819ffb700 10 monclient: dump: 2021-05-25T12:04:11.496 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:11.497 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:11.497 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:11.497 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:11.497 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:11.498 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:11.498 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:11.498 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:11.498 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:11.498 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:11.499 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.444+0000 7fb833c91700 10 client.184184.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:04:11.499 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.444+0000 7fb833c91700 10 monclient: _renew_subs 2021-05-25T12:04:11.499 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.444+0000 7fb81bfff700 4 set_mon_vals no callback set 2021-05-25T12:04:11.499 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.444+0000 7fb833c91700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:04:11.500 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.444+0000 7fb833c91700 1 -- 172.21.15.102:0/2854213716 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb82c126920 con 0x7fb82c090990 2021-05-25T12:04:11.500 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.444+0000 7fb833c91700 1 librados: init done 2021-05-25T12:04:11.500 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.445+0000 7fb819ffb700 1 -- 172.21.15.102:0/2854213716 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7fb82000de90 con 0x7fb82c090990 2021-05-25T12:04:11.500 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.445+0000 7fb833c91700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:04:11.500 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.445+0000 7fb833c91700 1 -- 172.21.15.102:0/2854213716 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fb82c059730 con 0x7fb82c12a290 2021-05-25T12:04:11.501 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.445+0000 7fb819ffb700 1 -- 172.21.15.102:0/2854213716 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(127..127 src has 1..127) v4 ==== 5218+0+0 (unknown 2630921626 0 0) 0x7fb82003d4c0 con 0x7fb82c090990 2021-05-25T12:04:11.501 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.445+0000 7fb819ffb700 10 client.184184.objecter ms_dispatch 0x7fb82c001550 osd_map(127..127 src has 1..127) v4 2021-05-25T12:04:11.501 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.445+0000 7fb819ffb700 3 client.184184.objecter handle_osd_map got epochs [127,127] > 0 2021-05-25T12:04:11.501 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.445+0000 7fb819ffb700 3 client.184184.objecter handle_osd_map decoding full epoch 127 2021-05-25T12:04:11.501 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.445+0000 7fb819ffb700 20 client.184184.objecter dump_active .. 0 homeless 2021-05-25T12:04:11.502 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.445+0000 7fb819ffb700 10 client.184184.objecter ms_handle_connect 0x7fb82c12a290 2021-05-25T12:04:11.502 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.445+0000 7fb819ffb700 10 client.184184.objecter resend_mon_ops 2021-05-25T12:04:11.502 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.445+0000 7fb83122c700 10 monclient: get_auth_request con 0x7fb80c033520 auth_method 0 2021-05-25T12:04:11.502 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.445+0000 7fb819ffb700 1 -- 172.21.15.102:0/2854213716 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1951666910 0 0) 0x7fb82c059730 con 0x7fb82c12a290 2021-05-25T12:04:11.503 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.445+0000 7fb819ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:04:11.503 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.445+0000 7fb819ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:04:11.503 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.445+0000 7fb819ffb700 10 monclient(hunting): my global_id is 174462 2021-05-25T12:04:11.503 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.445+0000 7fb819ffb700 1 -- 172.21.15.102:0/2854213716 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb82c1398c0 con 0x7fb82c12a290 2021-05-25T12:04:11.504 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.445+0000 7fb819ffb700 1 -- 172.21.15.102:0/2854213716 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1895591113 0 0) 0x7fb82c1398c0 con 0x7fb82c12a290 2021-05-25T12:04:11.504 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.445+0000 7fb819ffb700 1 -- 172.21.15.102:0/2854213716 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb82c059730 con 0x7fb82c12a290 2021-05-25T12:04:11.504 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.445+0000 7fb819ffb700 10 client.184184.objecter ms_handle_connect 0x7fb80c033520 2021-05-25T12:04:11.504 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.445+0000 7fb819ffb700 1 -- 172.21.15.102:0/2854213716 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fb828001d50 con 0x7fb82c12a290 2021-05-25T12:04:11.504 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.445+0000 7fb819ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:11.505 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.445+0000 7fb819ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:04:11.505 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.445+0000 7fb819ffb700 10 monclient: dump: 2021-05-25T12:04:11.505 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:11.505 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:11.506 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:11.506 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:11.506 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:11.506 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:11.506 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:11.507 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:11.507 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:11.507 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:11.507 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.445+0000 7fb819ffb700 1 -- 172.21.15.102:0/2854213716 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3619251922 0 0) 0x7fb82c059730 con 0x7fb82c12a290 2021-05-25T12:04:11.507 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.445+0000 7fb819ffb700 1 -- 172.21.15.102:0/2854213716 --> v1:172.21.15.102:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb82c111ab0 con 0x7fb82c12a290 2021-05-25T12:04:11.508 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.446+0000 7fb819ffb700 1 -- 172.21.15.102:0/2854213716 <== mon.0 v1:172.21.15.102:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7fb828002080 con 0x7fb82c12a290 2021-05-25T12:04:11.508 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.446+0000 7fb819ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:04:11.508 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.446+0000 7fb819ffb700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:04:11.508 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.446+0000 7fb819ffb700 1 -- 172.21.15.102:0/2854213716 >> v1:172.21.15.102:6789/0 conn(0x7fb82c12a290 legacy=0x7fb82c12a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:11.508 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.446+0000 7fb819ffb700 1 -- 172.21.15.102:0/2854213716 >> v1:172.21.15.102:6789/0 conn(0x7fb82c12a290 legacy=0x7fb82c12a670 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:04:11.509 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.452+0000 7fb833c91700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:04:11.509 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.452+0000 7fb833c91700 1 -- 172.21.15.102:0/2854213716 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fb82c05d870 con 0x7fb82c027ce0 2021-05-25T12:04:11.509 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.452+0000 7fb819ffb700 10 client.184184.objecter ms_handle_connect 0x7fb82c027ce0 2021-05-25T12:04:11.509 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.452+0000 7fb819ffb700 10 client.184184.objecter resend_mon_ops 2021-05-25T12:04:11.509 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.453+0000 7fb819ffb700 1 -- 172.21.15.102:0/2854213716 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2507517973 0 0) 0x7fb82c05d870 con 0x7fb82c027ce0 2021-05-25T12:04:11.510 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.453+0000 7fb819ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:04:11.510 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.453+0000 7fb819ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:04:11.510 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.453+0000 7fb819ffb700 10 monclient(hunting): my global_id is 174465 2021-05-25T12:04:11.510 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.453+0000 7fb819ffb700 1 -- 172.21.15.102:0/2854213716 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb82c059730 con 0x7fb82c027ce0 2021-05-25T12:04:11.511 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.453+0000 7fb819ffb700 1 -- 172.21.15.102:0/2854213716 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2817848781 0 0) 0x7fb82c059730 con 0x7fb82c027ce0 2021-05-25T12:04:11.511 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.453+0000 7fb819ffb700 1 -- 172.21.15.102:0/2854213716 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb82c05d870 con 0x7fb82c027ce0 2021-05-25T12:04:11.511 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.453+0000 7fb819ffb700 1 -- 172.21.15.102:0/2854213716 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fb828004ce0 con 0x7fb82c027ce0 2021-05-25T12:04:11.511 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.453+0000 7fb819ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:11.511 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.453+0000 7fb819ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:04:11.512 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.453+0000 7fb819ffb700 10 monclient: dump: 2021-05-25T12:04:11.512 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:11.512 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:11.512 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:11.512 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:11.513 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:11.513 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:11.513 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:11.513 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:11.513 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:11.514 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:11.514 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.453+0000 7fb819ffb700 1 -- 172.21.15.102:0/2854213716 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3889344815 0 0) 0x7fb82c05d870 con 0x7fb82c027ce0 2021-05-25T12:04:11.514 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.453+0000 7fb819ffb700 1 -- 172.21.15.102:0/2854213716 --> v1:172.21.15.102:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fb82c14cbb0 con 0x7fb82c027ce0 2021-05-25T12:04:11.514 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.453+0000 7fb819ffb700 1 -- 172.21.15.102:0/2854213716 <== mon.0 v1:172.21.15.102:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4110 (unknown 0 0 243735252) 0x7fb828006020 con 0x7fb82c027ce0 2021-05-25T12:04:11.515 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.453+0000 7fb819ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:04:11.515 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.453+0000 7fb819ffb700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:04:11.515 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.453+0000 7fb819ffb700 1 -- 172.21.15.102:0/2854213716 >> v1:172.21.15.102:6789/0 conn(0x7fb82c027ce0 legacy=0x7fb82c12ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:11.515 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.453+0000 7fb819ffb700 1 -- 172.21.15.102:0/2854213716 >> v1:172.21.15.102:6789/0 conn(0x7fb82c027ce0 legacy=0x7fb82c12ad80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:04:11.516 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:04:11.516 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:04:11.516 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:04:11.516 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "peon", 2021-05-25T12:04:11.517 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 38, 2021-05-25T12:04:11.517 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:04:11.517 INFO:teuthology.orchestra.run.smithi102.stdout: 0, 2021-05-25T12:04:11.517 INFO:teuthology.orchestra.run.smithi102.stdout: 1, 2021-05-25T12:04:11.518 INFO:teuthology.orchestra.run.smithi102.stdout: 2 2021-05-25T12:04:11.518 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:04:11.518 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 3, 2021-05-25T12:04:11.518 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:04:11.518 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:04:11.519 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:04:11.519 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:04:11.519 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:04:11.519 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:04:11.520 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:04:11.520 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:04:11.520 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:04:11.520 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:04:11.520 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:04:11.521 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:04:11.521 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:04:11.521 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:04:11.521 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:04:11.522 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:04:11.522 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:04:11.522 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:04:11.522 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:04:11.522 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:04:11.523 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:04:11.523 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:04:11.523 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:04:11.523 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:04:11.523 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:04:11.524 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:04:11.524 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:04:11.524 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:04:11.524 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:04:11.524 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:04:11.525 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:04:11.525 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:04:11.525 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:04:11.525 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:04:11.526 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:04:11.526 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:04:11.526 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:04:11.526 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:04:11.526 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:04:11.527 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:04:11.527 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:04:11.527 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:04:11.527 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:04:11.528 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:04:11.528 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:04:11.528 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:04:11.528 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:04:11.528 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:04:11.529 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:04:11.529 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:04:11.529 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:04:11.529 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:04:11.529 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:04:11.530 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:04:11.530 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:04:11.530 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:04:11.530 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:04:11.531 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:04:11.531 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:04:11.531 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:04:11.531 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:04:11.531 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:04:11.532 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:04:11.532 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:04:11.532 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:04:11.532 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:04:11.532 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:04:11.533 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:04:11.533 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:04:11.533 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:04:11.533 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:04:11.533 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:04:11.534 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:04:11.534 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:04:11.534 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:04:11.534 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:04:11.535 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:04:11.535 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:04:11.535 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:04:11.535 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:04:11.536 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:04:11.536 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:04:11.536 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:04:11.536 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:04:11.536 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:04:11.537 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:04:11.537 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:04:11.537 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:04:11.537 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:04:11.537 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:04:11.538 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:04:11.538 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:04:11.538 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:04:11.538 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:04:11.539 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:04:11.539 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:04:11.539 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:04:11.539 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:04:11.539 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:04:11.540 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:04:11.540 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:04:11.540 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:04:11.540 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:04:11.541 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:04:11.541 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:04:11.541 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:04:11.541 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:04:11.541 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:04:11.542 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:04:11.542 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:04:11.542 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:04:11.542 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:04:11.543 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:04:11.543 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:04:11.543 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:04:11.543 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:04:11.543 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:04:11.544 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:04:11.544 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": [ 2021-05-25T12:04:11.544 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:04:11.544 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:04:11.544 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:04:11.545 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 3 2021-05-25T12:04:11.545 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:04:11.545 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:04:11.545 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:04:11.545 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:04:11.546 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:04:11.546 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:04:11.546 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 6 2021-05-25T12:04:11.546 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:04:11.547 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:04:11.547 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": [ 2021-05-25T12:04:11.547 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:04:11.547 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:04:11.547 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:04:11.548 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:04:11.548 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:04:11.548 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:04:11.548 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:04:11.548 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:04:11.549 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:04:11.551 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.453+0000 7fb833c91700 10 librados: watch_flush enter 2021-05-25T12:04:11.551 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.453+0000 7fb833c91700 10 librados: watch_flush exit 2021-05-25T12:04:11.551 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.453+0000 7fb833c91700 20 client.184184.objecter shutdown clearing up homeless session... 2021-05-25T12:04:11.552 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.453+0000 7fb833c91700 10 client.184184.objecter successfully canceled tick 2021-05-25T12:04:11.552 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.453+0000 7fb833c91700 1 -- 172.21.15.102:0/2854213716 >> v1:172.21.15.181:6812/29582 conn(0x7fb80c033520 legacy=0x7fb80c0359e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:11.552 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.454+0000 7fb833c91700 10 monclient: shutdown 2021-05-25T12:04:11.552 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.454+0000 7fb833c91700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:04:11.552 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.454+0000 7fb833c91700 1 -- 172.21.15.102:0/2854213716 >> v1:172.21.15.102:6790/0 conn(0x7fb82c090990 legacy=0x7fb82c138aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:11.553 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.454+0000 7fb83222e700 1 -- 172.21.15.102:0/2854213716 reap_dead start 2021-05-25T12:04:11.553 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.454+0000 7fb83222e700 1 -- 172.21.15.102:0/2854213716 reap_dead start 2021-05-25T12:04:11.553 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.454+0000 7fb833c91700 1 -- 172.21.15.102:0/2854213716 shutdown_connections 2021-05-25T12:04:11.553 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.454+0000 7fb833c91700 1 -- 172.21.15.102:0/2854213716 >> 172.21.15.102:0/2854213716 conn(0x7fb82c09b370 msgr2=0x7fb82c1292f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:04:11.553 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.454+0000 7fb833c91700 1 -- 172.21.15.102:0/2854213716 shutdown_connections 2021-05-25T12:04:11.554 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.454+0000 7fb833c91700 1 -- 172.21.15.102:0/2854213716 wait complete. 2021-05-25T12:04:11.554 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.454+0000 7fb833c91700 1 librados: shutdown 2021-05-25T12:04:11.554 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2021-05-25T12:04:11.664 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.662+0000 7ff74fa93700 10 monclient: get_monmap_and_config 2021-05-25T12:04:11.664 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.662+0000 7ff74fa93700 10 monclient: build_initial_monmap 2021-05-25T12:04:11.664 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.662+0000 7ff74fa93700 10 monclient: monmap: 2021-05-25T12:04:11.665 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:04:11.665 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:11.665 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:04:11.663764+0000 2021-05-25T12:04:11.665 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:04:11.663764+0000 2021-05-25T12:04:11.665 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:04:11.666 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:11.666 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:04:11.666 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:04:11.666 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:04:11.667 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:11.679 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.663+0000 7ff74fa93700 1 Processor -- start 2021-05-25T12:04:11.679 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.663+0000 7ff74fa93700 1 -- start start 2021-05-25T12:04:11.679 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.663+0000 7ff74fa93700 10 monclient: init 2021-05-25T12:04:11.680 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.663+0000 7ff74fa93700 10 monclient: _reopen_session rank -1 2021-05-25T12:04:11.680 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.664+0000 7ff74fa93700 10 monclient: _add_conns ranks=[1,2,0] 2021-05-25T12:04:11.680 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.664+0000 7ff74fa93700 10 monclient(hunting): picked mon.noname-c con 0x7ff748090990 addr v1:172.21.15.181:6789/0 2021-05-25T12:04:11.680 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.664+0000 7ff74fa93700 10 monclient(hunting): picked mon.noname-b con 0x7ff748090d70 addr v1:172.21.15.102:6790/0 2021-05-25T12:04:11.680 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.664+0000 7ff74fa93700 10 monclient(hunting): picked mon.noname-a con 0x7ff74812e330 addr v1:172.21.15.102:6789/0 2021-05-25T12:04:11.681 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.664+0000 7ff74fa93700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff748096fa0 con 0x7ff74812e330 2021-05-25T12:04:11.681 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.664+0000 7ff74fa93700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7480926a0 con 0x7ff748090990 2021-05-25T12:04:11.681 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.664+0000 7ff74fa93700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff748092940 con 0x7ff748090d70 2021-05-25T12:04:11.681 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.664+0000 7ff74fa93700 10 monclient(hunting): _renew_subs 2021-05-25T12:04:11.681 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.664+0000 7ff74fa93700 10 monclient(hunting): authenticate will time out at 1052.261124s 2021-05-25T12:04:11.682 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.664+0000 7ff74e030700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7ff74812e330 0x7ff748130720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60740/0 (socket says 172.21.15.102:60740) 2021-05-25T12:04:11.682 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.664+0000 7ff74e030700 1 -- 172.21.15.102:0/2934802977 learned_addr learned my addr 172.21.15.102:0/2934802977 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:04:11.682 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.664+0000 7ff74d02e700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7ff748090d70 0x7ff748091200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:55616/0 (socket says 172.21.15.102:55616) 2021-05-25T12:04:11.682 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.664+0000 7ff74c82d700 1 -- 172.21.15.102:0/2934802977 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3169663642 0 0) 0x7ff748092940 con 0x7ff748090d70 2021-05-25T12:04:11.683 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.664+0000 7ff74c82d700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:04:11.683 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.664+0000 7ff74c82d700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:04:11.683 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.664+0000 7ff74c82d700 10 monclient(hunting): my global_id is 184187 2021-05-25T12:04:11.683 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.664+0000 7ff74c82d700 1 -- 172.21.15.102:0/2934802977 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff730003270 con 0x7ff748090d70 2021-05-25T12:04:11.683 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.664+0000 7ff74c82d700 1 -- 172.21.15.102:0/2934802977 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 145424592 0 0) 0x7ff748096fa0 con 0x7ff74812e330 2021-05-25T12:04:11.684 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.664+0000 7ff74c82d700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:04:11.684 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.664+0000 7ff74c82d700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:04:11.684 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.664+0000 7ff74c82d700 10 monclient(hunting): my global_id is 174468 2021-05-25T12:04:11.684 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.664+0000 7ff74c82d700 1 -- 172.21.15.102:0/2934802977 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff748092940 con 0x7ff74812e330 2021-05-25T12:04:11.685 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.664+0000 7ff74c82d700 1 -- 172.21.15.102:0/2934802977 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1538328924 0 0) 0x7ff7480926a0 con 0x7ff748090990 2021-05-25T12:04:11.685 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.664+0000 7ff74c82d700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:04:11.685 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.664+0000 7ff74c82d700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:04:11.685 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.664+0000 7ff74c82d700 10 monclient(hunting): my global_id is 95377 2021-05-25T12:04:11.685 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.664+0000 7ff74c82d700 1 -- 172.21.15.102:0/2934802977 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff748096fa0 con 0x7ff748090990 2021-05-25T12:04:11.686 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.664+0000 7ff74c82d700 1 -- 172.21.15.102:0/2934802977 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1717452842 0 0) 0x7ff748092940 con 0x7ff74812e330 2021-05-25T12:04:11.686 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.664+0000 7ff74c82d700 1 -- 172.21.15.102:0/2934802977 >> v1:172.21.15.102:6790/0 conn(0x7ff748090d70 legacy=0x7ff748091200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:11.686 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.664+0000 7ff74c82d700 1 -- 172.21.15.102:0/2934802977 >> v1:172.21.15.181:6789/0 conn(0x7ff748090990 legacy=0x7ff748093240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:11.686 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.664+0000 7ff74c82d700 10 monclient: _finish_hunting 0 2021-05-25T12:04:11.687 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.664+0000 7ff74c82d700 1 monclient: found mon.noname-a 2021-05-25T12:04:11.687 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.664+0000 7ff74c82d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:04:11.687 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.664+0000 7ff74c82d700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T12:04:11.687 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.664+0000 7ff74c82d700 1 -- 172.21.15.102:0/2934802977 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff748092be0 con 0x7ff74812e330 2021-05-25T12:04:11.687 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.664+0000 7ff74c82d700 10 monclient: _finish_auth 0 2021-05-25T12:04:11.688 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.664+0000 7ff74c82d700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:04:11.688 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.664+0000 7ff74c82d700 1 -- 172.21.15.102:0/2934802977 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7ff744001970 con 0x7ff74812e330 2021-05-25T12:04:11.688 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.664+0000 7ff74fa93700 5 monclient: authenticate success, global_id 174468 2021-05-25T12:04:11.688 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.664+0000 7ff74fa93700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:04:11.689 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.664+0000 7ff74c82d700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:11.689 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.664+0000 7ff74c82d700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:04:11.689 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.664+0000 7ff74c82d700 10 monclient: dump: 2021-05-25T12:04:11.689 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:11.689 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:11.690 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:11.690 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:11.690 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:11.690 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:11.690 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:11.691 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:11.691 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:11.691 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:11.691 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.664+0000 7ff74fa93700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:04:11.691 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.664+0000 7ff74c82d700 1 -- 172.21.15.102:0/2934802977 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff744001ba0 con 0x7ff74812e330 2021-05-25T12:04:11.692 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.664+0000 7ff74c82d700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:04:11.692 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.664+0000 7ff74c82d700 1 -- 172.21.15.102:0/2934802977 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7ff744001f30 con 0x7ff74812e330 2021-05-25T12:04:11.692 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.664+0000 7ff74fa93700 10 monclient: get_monmap_and_config success 2021-05-25T12:04:11.692 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.664+0000 7ff74fa93700 4 set_mon_vals no callback set 2021-05-25T12:04:11.692 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.664+0000 7ff74c82d700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:11.693 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.664+0000 7ff74c82d700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:04:11.693 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.664+0000 7ff74c82d700 10 monclient: dump: 2021-05-25T12:04:11.693 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:11.693 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:11.694 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:11.694 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:11.694 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:11.694 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:11.694 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:11.695 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:11.695 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:11.695 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:11.695 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.665+0000 7ff74fa93700 10 monclient: shutdown 2021-05-25T12:04:11.696 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.665+0000 7ff74fa93700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:04:11.696 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.665+0000 7ff74fa93700 1 -- 172.21.15.102:0/2934802977 >> v1:172.21.15.102:6789/0 conn(0x7ff74812e330 legacy=0x7ff748130720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:11.696 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.665+0000 7ff74fa93700 1 -- 172.21.15.102:0/2934802977 shutdown_connections 2021-05-25T12:04:11.696 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.665+0000 7ff74fa93700 1 -- 172.21.15.102:0/2934802977 >> 172.21.15.102:0/2934802977 conn(0x7ff74809b370 msgr2=0x7ff748097430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:04:11.696 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.665+0000 7ff74fa93700 1 -- 172.21.15.102:0/2934802977 shutdown_connections 2021-05-25T12:04:11.697 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.665+0000 7ff74fa93700 1 -- 172.21.15.102:0/2934802977 wait complete. 2021-05-25T12:04:11.697 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.665+0000 7ff74fa93700 10 monclient: build_initial_monmap 2021-05-25T12:04:11.697 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.665+0000 7ff74fa93700 10 monclient: monmap: 2021-05-25T12:04:11.697 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:04:11.698 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:04:11.698 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:04:11.698 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:04:11.698 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:04:11.698 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:04:11.699 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:04:11.699 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:04:11.699 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:04:11.699 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:11.700 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.666+0000 7ff74fa93700 1 librados: starting msgr at 2021-05-25T12:04:11.700 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.666+0000 7ff74fa93700 1 librados: starting objecter 2021-05-25T12:04:11.700 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.666+0000 7ff74fa93700 1 Processor -- start 2021-05-25T12:04:11.700 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.666+0000 7ff74fa93700 1 -- start start 2021-05-25T12:04:11.701 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.666+0000 7ff74fa93700 1 librados: setting wanted keys 2021-05-25T12:04:11.701 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.666+0000 7ff74fa93700 1 librados: calling monclient init 2021-05-25T12:04:11.701 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.666+0000 7ff74fa93700 10 monclient: init 2021-05-25T12:04:11.701 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.666+0000 7ff74fa93700 10 monclient: _reopen_session rank -1 2021-05-25T12:04:11.701 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.666+0000 7ff74fa93700 10 monclient: _add_conns ranks=[0,1,2] 2021-05-25T12:04:11.702 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.666+0000 7ff74fa93700 10 monclient(hunting): picked mon.noname-a con 0x7ff748090990 addr v1:172.21.15.102:6789/0 2021-05-25T12:04:11.702 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.666+0000 7ff74fa93700 10 monclient(hunting): picked mon.noname-b con 0x7ff748090d70 addr v1:172.21.15.181:6789/0 2021-05-25T12:04:11.702 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.666+0000 7ff74fa93700 10 monclient(hunting): picked mon.noname-c con 0x7ff74812e330 addr v1:172.21.15.102:6790/0 2021-05-25T12:04:11.702 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.666+0000 7ff74fa93700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff748117960 con 0x7ff748090990 2021-05-25T12:04:11.702 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.666+0000 7ff74fa93700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff748117c00 con 0x7ff748090d70 2021-05-25T12:04:11.703 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.666+0000 7ff74fa93700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff748117ea0 con 0x7ff74812e330 2021-05-25T12:04:11.703 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.666+0000 7ff74fa93700 10 monclient(hunting): _renew_subs 2021-05-25T12:04:11.703 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.666+0000 7ff74fa93700 10 monclient(hunting): authenticate will time out at 1052.263657s 2021-05-25T12:04:11.703 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.666+0000 7ff74d82f700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7ff748090990 0x7ff74813ab20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60742/0 (socket says 172.21.15.102:60742) 2021-05-25T12:04:11.703 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.666+0000 7ff74d82f700 1 -- 172.21.15.102:0/2314187716 learned_addr learned my addr 172.21.15.102:0/2314187716 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:04:11.704 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.666+0000 7ff735ffb700 10 client.?.objecter ms_handle_connect 0x7ff748090990 2021-05-25T12:04:11.704 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.666+0000 7ff735ffb700 10 client.?.objecter resend_mon_ops 2021-05-25T12:04:11.704 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.666+0000 7ff735ffb700 10 client.?.objecter ms_handle_connect 0x7ff74812e330 2021-05-25T12:04:11.704 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.666+0000 7ff735ffb700 10 client.?.objecter resend_mon_ops 2021-05-25T12:04:11.705 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.666+0000 7ff735ffb700 1 -- 172.21.15.102:0/2314187716 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4173980400 0 0) 0x7ff748117960 con 0x7ff748090990 2021-05-25T12:04:11.705 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.666+0000 7ff735ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:04:11.705 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.666+0000 7ff735ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:04:11.705 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.666+0000 7ff735ffb700 10 monclient(hunting): my global_id is 174471 2021-05-25T12:04:11.705 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.666+0000 7ff735ffb700 1 -- 172.21.15.102:0/2314187716 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff728004c10 con 0x7ff748090990 2021-05-25T12:04:11.705 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.666+0000 7ff735ffb700 10 client.?.objecter ms_handle_connect 0x7ff748090d70 2021-05-25T12:04:11.706 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.666+0000 7ff735ffb700 10 client.?.objecter resend_mon_ops 2021-05-25T12:04:11.706 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.666+0000 7ff735ffb700 1 -- 172.21.15.102:0/2314187716 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 522819872 0 0) 0x7ff748117ea0 con 0x7ff74812e330 2021-05-25T12:04:11.706 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.666+0000 7ff735ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:04:11.706 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.666+0000 7ff735ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:04:11.707 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.666+0000 7ff735ffb700 10 monclient(hunting): my global_id is 184190 2021-05-25T12:04:11.707 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.666+0000 7ff735ffb700 1 -- 172.21.15.102:0/2314187716 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff748117960 con 0x7ff74812e330 2021-05-25T12:04:11.707 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.667+0000 7ff735ffb700 1 -- 172.21.15.102:0/2314187716 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3206913499 0 0) 0x7ff748117c00 con 0x7ff748090d70 2021-05-25T12:04:11.707 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.667+0000 7ff735ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:04:11.707 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.667+0000 7ff735ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:04:11.708 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.667+0000 7ff735ffb700 10 monclient(hunting): my global_id is 95380 2021-05-25T12:04:11.708 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.667+0000 7ff735ffb700 1 -- 172.21.15.102:0/2314187716 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff748117ea0 con 0x7ff748090d70 2021-05-25T12:04:11.708 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.667+0000 7ff735ffb700 1 -- 172.21.15.102:0/2314187716 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2083808327 0 0) 0x7ff728004c10 con 0x7ff748090990 2021-05-25T12:04:11.708 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.667+0000 7ff735ffb700 1 -- 172.21.15.102:0/2314187716 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff748117c00 con 0x7ff748090990 2021-05-25T12:04:11.708 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.667+0000 7ff735ffb700 1 -- 172.21.15.102:0/2314187716 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 271372395 0 0) 0x7ff748117960 con 0x7ff74812e330 2021-05-25T12:04:11.709 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.667+0000 7ff735ffb700 1 -- 172.21.15.102:0/2314187716 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff728004c10 con 0x7ff74812e330 2021-05-25T12:04:11.709 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.667+0000 7ff735ffb700 1 -- 172.21.15.102:0/2314187716 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7ff73c001980 con 0x7ff748090990 2021-05-25T12:04:11.709 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.667+0000 7ff735ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:11.709 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.667+0000 7ff735ffb700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:04:11.709 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.667+0000 7ff735ffb700 10 monclient(hunting): dump: 2021-05-25T12:04:11.710 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:11.710 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:11.710 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:11.710 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:11.710 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:11.711 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:11.711 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:11.711 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:11.711 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:11.711 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:11.712 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.667+0000 7ff735ffb700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:04:11.712 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.667+0000 7ff735ffb700 1 -- 172.21.15.102:0/2314187716 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7ff744001eb0 con 0x7ff74812e330 2021-05-25T12:04:11.712 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.667+0000 7ff735ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:11.712 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.667+0000 7ff735ffb700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2021-05-25T12:04:11.713 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.667+0000 7ff735ffb700 10 monclient(hunting): dump: 2021-05-25T12:04:11.713 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:11.713 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:11.713 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:11.713 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:11.714 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:11.714 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:11.714 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:11.714 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:11.714 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:11.715 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:11.715 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.667+0000 7ff735ffb700 1 -- 172.21.15.102:0/2314187716 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2289909864 0 0) 0x7ff748117ea0 con 0x7ff748090d70 2021-05-25T12:04:11.715 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.667+0000 7ff735ffb700 1 -- 172.21.15.102:0/2314187716 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff748117960 con 0x7ff748090d70 2021-05-25T12:04:11.715 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.667+0000 7ff735ffb700 1 -- 172.21.15.102:0/2314187716 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7ff738001a70 con 0x7ff748090d70 2021-05-25T12:04:11.716 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.667+0000 7ff735ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:11.716 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.667+0000 7ff735ffb700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2021-05-25T12:04:11.716 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.667+0000 7ff735ffb700 10 monclient(hunting): dump: 2021-05-25T12:04:11.716 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:11.716 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:11.717 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:11.717 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:11.717 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:11.717 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:11.718 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:11.718 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:11.718 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:11.718 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:11.718 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.667+0000 7ff735ffb700 1 -- 172.21.15.102:0/2314187716 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1397047066 0 0) 0x7ff748117c00 con 0x7ff748090990 2021-05-25T12:04:11.719 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.667+0000 7ff735ffb700 1 -- 172.21.15.102:0/2314187716 >> v1:172.21.15.102:6790/0 conn(0x7ff74812e330 legacy=0x7ff748142ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:11.719 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.667+0000 7ff735ffb700 1 -- 172.21.15.102:0/2314187716 >> v1:172.21.15.181:6789/0 conn(0x7ff748090d70 legacy=0x7ff748129450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:11.719 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.667+0000 7ff735ffb700 10 monclient: _finish_hunting 0 2021-05-25T12:04:11.719 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.667+0000 7ff735ffb700 1 monclient: found mon.a 2021-05-25T12:04:11.719 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.667+0000 7ff735ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:04:11.720 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.667+0000 7ff735ffb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:04:11.720 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.667+0000 7ff735ffb700 1 -- 172.21.15.102:0/2314187716 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff748115d30 con 0x7ff748090990 2021-05-25T12:04:11.720 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.667+0000 7ff735ffb700 10 monclient: _finish_auth 0 2021-05-25T12:04:11.720 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.667+0000 7ff735ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:04:11.721 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.667+0000 7ff74fa93700 5 monclient: authenticate success, global_id 174471 2021-05-25T12:04:11.721 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.667+0000 7ff74fa93700 10 monclient: _renew_subs 2021-05-25T12:04:11.721 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.667+0000 7ff74fa93700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:04:11.721 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.667+0000 7ff74fa93700 1 -- 172.21.15.102:0/2314187716 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff74813b230 con 0x7ff748090990 2021-05-25T12:04:11.722 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.667+0000 7ff735ffb700 1 -- 172.21.15.102:0/2314187716 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff73c002d00 con 0x7ff748090990 2021-05-25T12:04:11.722 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.667+0000 7ff735ffb700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:04:11.722 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.667+0000 7ff735ffb700 1 -- 172.21.15.102:0/2314187716 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7ff73c001fc0 con 0x7ff748090990 2021-05-25T12:04:11.722 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.667+0000 7ff735ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:11.722 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.667+0000 7ff735ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:04:11.723 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.667+0000 7ff735ffb700 10 monclient: dump: 2021-05-25T12:04:11.723 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:11.723 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:11.723 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:11.723 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:11.724 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:11.724 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:11.724 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:11.724 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:11.725 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:11.725 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:11.725 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.667+0000 7ff737fff700 4 set_mon_vals no callback set 2021-05-25T12:04:11.725 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.667+0000 7ff74fa93700 10 client.174471.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:04:11.725 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.667+0000 7ff74fa93700 10 monclient: _renew_subs 2021-05-25T12:04:11.726 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.667+0000 7ff74fa93700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:04:11.726 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.667+0000 7ff74fa93700 1 -- 172.21.15.102:0/2314187716 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff74813b4e0 con 0x7ff748090990 2021-05-25T12:04:11.726 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.667+0000 7ff74fa93700 1 librados: init done 2021-05-25T12:04:11.726 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.668+0000 7ff735ffb700 1 -- 172.21.15.102:0/2314187716 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7ff73c0021e0 con 0x7ff748090990 2021-05-25T12:04:11.727 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.668+0000 7ff735ffb700 1 -- 172.21.15.102:0/2314187716 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(127..127 src has 1..127) v4 ==== 5218+0+0 (unknown 2630921626 0 0) 0x7ff73c03d410 con 0x7ff748090990 2021-05-25T12:04:11.727 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.668+0000 7ff74fa93700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:04:11.727 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.668+0000 7ff735ffb700 10 client.174471.objecter ms_dispatch 0x7ff748001550 osd_map(127..127 src has 1..127) v4 2021-05-25T12:04:11.727 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.668+0000 7ff735ffb700 3 client.174471.objecter handle_osd_map got epochs [127,127] > 0 2021-05-25T12:04:11.727 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.668+0000 7ff735ffb700 3 client.174471.objecter handle_osd_map decoding full epoch 127 2021-05-25T12:04:11.728 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.668+0000 7ff74fa93700 1 -- 172.21.15.102:0/2314187716 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff74812a1a0 con 0x7ff748110f70 2021-05-25T12:04:11.728 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.668+0000 7ff735ffb700 20 client.174471.objecter dump_active .. 0 homeless 2021-05-25T12:04:11.728 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.668+0000 7ff735ffb700 10 client.174471.objecter ms_handle_connect 0x7ff748110f70 2021-05-25T12:04:11.728 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.668+0000 7ff74d02e700 10 monclient: get_auth_request con 0x7ff728033550 auth_method 0 2021-05-25T12:04:11.728 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.668+0000 7ff735ffb700 10 client.174471.objecter resend_mon_ops 2021-05-25T12:04:11.729 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.668+0000 7ff735ffb700 1 -- 172.21.15.102:0/2314187716 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2929001738 0 0) 0x7ff74812a1a0 con 0x7ff748110f70 2021-05-25T12:04:11.729 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.668+0000 7ff735ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:04:11.729 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.668+0000 7ff735ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:04:11.729 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.668+0000 7ff735ffb700 10 monclient(hunting): my global_id is 184193 2021-05-25T12:04:11.730 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.668+0000 7ff735ffb700 1 -- 172.21.15.102:0/2314187716 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff748117c00 con 0x7ff748110f70 2021-05-25T12:04:11.730 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.668+0000 7ff735ffb700 10 client.174471.objecter ms_handle_connect 0x7ff728033550 2021-05-25T12:04:11.730 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.668+0000 7ff735ffb700 1 -- 172.21.15.102:0/2314187716 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 860128027 0 0) 0x7ff748117c00 con 0x7ff748110f70 2021-05-25T12:04:11.730 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.668+0000 7ff735ffb700 1 -- 172.21.15.102:0/2314187716 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff74812a1a0 con 0x7ff748110f70 2021-05-25T12:04:11.730 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.668+0000 7ff735ffb700 1 -- 172.21.15.102:0/2314187716 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7ff7440050b0 con 0x7ff748110f70 2021-05-25T12:04:11.731 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.668+0000 7ff735ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:11.731 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.668+0000 7ff735ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:04:11.731 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.668+0000 7ff735ffb700 10 monclient: dump: 2021-05-25T12:04:11.731 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:11.731 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:11.732 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:11.732 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:11.732 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:11.732 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:11.733 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:11.733 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:11.733 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:11.733 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:11.733 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.669+0000 7ff735ffb700 1 -- 172.21.15.102:0/2314187716 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 317159439 0 0) 0x7ff74812a1a0 con 0x7ff748110f70 2021-05-25T12:04:11.734 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.669+0000 7ff735ffb700 1 -- 172.21.15.102:0/2314187716 --> v1:172.21.15.102:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff748059730 con 0x7ff748110f70 2021-05-25T12:04:11.734 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.669+0000 7ff735ffb700 1 -- 172.21.15.102:0/2314187716 <== mon.2 v1:172.21.15.102:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7ff744005320 con 0x7ff748110f70 2021-05-25T12:04:11.734 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.669+0000 7ff735ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:04:11.734 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.669+0000 7ff735ffb700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:04:11.734 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.669+0000 7ff735ffb700 1 -- 172.21.15.102:0/2314187716 >> v1:172.21.15.102:6790/0 conn(0x7ff748110f70 legacy=0x7ff748111350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:11.735 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.669+0000 7ff735ffb700 1 -- 172.21.15.102:0/2314187716 >> v1:172.21.15.102:6790/0 conn(0x7ff748110f70 legacy=0x7ff748111350 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:04:11.735 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.675+0000 7ff74fa93700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:04:11.735 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.675+0000 7ff74fa93700 1 -- 172.21.15.102:0/2314187716 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff748129e40 con 0x7ff748027ce0 2021-05-25T12:04:11.735 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.676+0000 7ff735ffb700 10 client.174471.objecter ms_handle_connect 0x7ff748027ce0 2021-05-25T12:04:11.736 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.676+0000 7ff735ffb700 10 client.174471.objecter resend_mon_ops 2021-05-25T12:04:11.736 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.676+0000 7ff735ffb700 1 -- 172.21.15.102:0/2314187716 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1679002505 0 0) 0x7ff748129e40 con 0x7ff748027ce0 2021-05-25T12:04:11.736 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.676+0000 7ff735ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:04:11.736 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.676+0000 7ff735ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:04:11.736 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.676+0000 7ff735ffb700 10 monclient(hunting): my global_id is 184196 2021-05-25T12:04:11.737 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.676+0000 7ff735ffb700 1 -- 172.21.15.102:0/2314187716 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff74812a1a0 con 0x7ff748027ce0 2021-05-25T12:04:11.737 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.676+0000 7ff735ffb700 1 -- 172.21.15.102:0/2314187716 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2078532439 0 0) 0x7ff74812a1a0 con 0x7ff748027ce0 2021-05-25T12:04:11.737 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.676+0000 7ff735ffb700 1 -- 172.21.15.102:0/2314187716 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff748129e40 con 0x7ff748027ce0 2021-05-25T12:04:11.737 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.676+0000 7ff735ffb700 1 -- 172.21.15.102:0/2314187716 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7ff7440059d0 con 0x7ff748027ce0 2021-05-25T12:04:11.738 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.676+0000 7ff735ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:11.738 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.676+0000 7ff735ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:04:11.738 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.676+0000 7ff735ffb700 10 monclient: dump: 2021-05-25T12:04:11.738 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:11.738 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:11.739 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:11.739 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:11.739 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:11.739 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:11.739 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:11.740 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:11.740 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:11.740 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:11.740 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.676+0000 7ff735ffb700 1 -- 172.21.15.102:0/2314187716 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2899717468 0 0) 0x7ff748129e40 con 0x7ff748027ce0 2021-05-25T12:04:11.740 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.676+0000 7ff735ffb700 1 -- 172.21.15.102:0/2314187716 --> v1:172.21.15.102:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ff74805d870 con 0x7ff748027ce0 2021-05-25T12:04:11.741 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.676+0000 7ff735ffb700 1 -- 172.21.15.102:0/2314187716 <== mon.2 v1:172.21.15.102:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3940 (unknown 0 0 3121422230) 0x7ff744005bc0 con 0x7ff748027ce0 2021-05-25T12:04:11.741 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.676+0000 7ff735ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:04:11.741 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.676+0000 7ff735ffb700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:04:11.741 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.676+0000 7ff735ffb700 1 -- 172.21.15.102:0/2314187716 >> v1:172.21.15.102:6790/0 conn(0x7ff748027ce0 legacy=0x7ff748148610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:11.741 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.676+0000 7ff735ffb700 1 -- 172.21.15.102:0/2314187716 >> v1:172.21.15.102:6790/0 conn(0x7ff748027ce0 legacy=0x7ff748148610 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:04:11.742 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:04:11.742 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:04:11.743 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:04:11.743 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "peon", 2021-05-25T12:04:11.743 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 38, 2021-05-25T12:04:11.743 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:04:11.743 INFO:teuthology.orchestra.run.smithi102.stdout: 0, 2021-05-25T12:04:11.744 INFO:teuthology.orchestra.run.smithi102.stdout: 1, 2021-05-25T12:04:11.744 INFO:teuthology.orchestra.run.smithi102.stdout: 2 2021-05-25T12:04:11.744 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:04:11.744 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 3, 2021-05-25T12:04:11.745 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:04:11.745 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:04:11.745 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:04:11.745 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:04:11.745 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:04:11.746 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:04:11.746 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:04:11.746 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:04:11.746 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:04:11.747 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:04:11.747 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:04:11.747 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:04:11.747 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:04:11.747 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:04:11.748 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:04:11.748 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:04:11.748 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:04:11.748 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:04:11.748 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:04:11.749 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:04:11.749 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:04:11.749 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:04:11.749 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:04:11.750 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:04:11.750 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:04:11.750 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:04:11.750 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:04:11.750 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:04:11.751 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:04:11.751 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:04:11.751 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:04:11.751 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:04:11.752 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:04:11.752 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:04:11.752 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:04:11.752 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:04:11.752 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:04:11.753 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:04:11.753 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:04:11.753 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:04:11.753 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:04:11.754 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:04:11.754 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:04:11.754 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:04:11.754 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:04:11.754 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:04:11.755 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:04:11.755 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:04:11.755 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:04:11.755 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:04:11.755 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:04:11.756 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:04:11.756 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:04:11.756 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:04:11.756 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:04:11.756 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:04:11.757 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:04:11.757 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:04:11.757 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:04:11.757 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:04:11.757 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:04:11.758 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:04:11.758 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:04:11.758 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:04:11.758 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:04:11.759 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:04:11.759 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:04:11.759 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:04:11.759 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:04:11.759 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:04:11.760 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:04:11.760 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:04:11.760 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:04:11.760 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:04:11.760 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:04:11.760 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:04:11.761 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:04:11.761 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:04:11.761 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:04:11.761 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:04:11.762 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:04:11.762 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:04:11.762 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:04:11.762 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:04:11.762 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:04:11.762 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:04:11.763 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:04:11.763 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:04:11.763 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:04:11.763 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:04:11.763 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:04:11.764 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:04:11.764 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:04:11.764 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:04:11.764 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:04:11.765 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:04:11.765 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:04:11.765 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:04:11.765 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:04:11.765 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:04:11.766 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:04:11.766 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:04:11.766 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:04:11.766 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:04:11.767 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:04:11.767 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:04:11.767 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:04:11.767 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:04:11.767 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:04:11.768 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:04:11.768 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:04:11.768 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:04:11.768 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:04:11.768 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:04:11.769 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:04:11.769 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:04:11.769 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:04:11.769 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:04:11.770 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": [ 2021-05-25T12:04:11.770 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:04:11.770 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:04:11.770 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:04:11.770 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 3 2021-05-25T12:04:11.771 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:04:11.771 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:04:11.771 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:04:11.771 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:04:11.771 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:04:11.772 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:04:11.772 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:04:11.772 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:04:11.772 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:04:11.773 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:04:11.773 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:04:11.773 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:04:11.775 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.676+0000 7ff74fa93700 10 librados: watch_flush enter 2021-05-25T12:04:11.776 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.676+0000 7ff74fa93700 10 librados: watch_flush exit 2021-05-25T12:04:11.776 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.677+0000 7ff74fa93700 20 client.174471.objecter shutdown clearing up homeless session... 2021-05-25T12:04:11.776 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.677+0000 7ff74fa93700 10 client.174471.objecter successfully canceled tick 2021-05-25T12:04:11.776 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.677+0000 7ff74fa93700 1 -- 172.21.15.102:0/2314187716 >> v1:172.21.15.181:6812/29582 conn(0x7ff728033550 legacy=0x7ff728035a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:11.777 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.677+0000 7ff74fa93700 10 monclient: shutdown 2021-05-25T12:04:11.777 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.677+0000 7ff74fa93700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:04:11.777 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.677+0000 7ff74fa93700 1 -- 172.21.15.102:0/2314187716 >> v1:172.21.15.102:6789/0 conn(0x7ff748090990 legacy=0x7ff74813ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:11.777 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.677+0000 7ff74e030700 1 -- 172.21.15.102:0/2314187716 reap_dead start 2021-05-25T12:04:11.778 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.677+0000 7ff74e030700 1 -- 172.21.15.102:0/2314187716 reap_dead start 2021-05-25T12:04:11.778 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.677+0000 7ff74fa93700 1 -- 172.21.15.102:0/2314187716 shutdown_connections 2021-05-25T12:04:11.778 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.677+0000 7ff74fa93700 1 -- 172.21.15.102:0/2314187716 >> 172.21.15.102:0/2314187716 conn(0x7ff74809b370 msgr2=0x7ff748068110 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:04:11.778 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.677+0000 7ff74fa93700 1 -- 172.21.15.102:0/2314187716 shutdown_connections 2021-05-25T12:04:11.778 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.677+0000 7ff74fa93700 1 -- 172.21.15.102:0/2314187716 wait complete. 2021-05-25T12:04:11.779 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.677+0000 7ff74fa93700 1 librados: shutdown 2021-05-25T12:04:11.779 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2021-05-25T12:04:11.889 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.888+0000 7f21a3155700 10 monclient: get_monmap_and_config 2021-05-25T12:04:11.890 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.888+0000 7f21a3155700 10 monclient: build_initial_monmap 2021-05-25T12:04:11.890 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.888+0000 7f21a3155700 10 monclient: monmap: 2021-05-25T12:04:11.890 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:04:11.890 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:11.890 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:04:11.889187+0000 2021-05-25T12:04:11.891 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:04:11.889187+0000 2021-05-25T12:04:11.891 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:04:11.891 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:11.891 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:04:11.892 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:04:11.892 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:04:11.892 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:11.904 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.888+0000 7f21a3155700 1 Processor -- start 2021-05-25T12:04:11.905 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.888+0000 7f21a3155700 1 -- start start 2021-05-25T12:04:11.905 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.888+0000 7f21a3155700 10 monclient: init 2021-05-25T12:04:11.905 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.889+0000 7f21a3155700 10 monclient: _reopen_session rank -1 2021-05-25T12:04:11.905 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.889+0000 7f21a3155700 10 monclient: _add_conns ranks=[0,2,1] 2021-05-25T12:04:11.906 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.889+0000 7f21a3155700 10 monclient(hunting): picked mon.noname-a con 0x7f219c090990 addr v1:172.21.15.102:6789/0 2021-05-25T12:04:11.906 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.889+0000 7f21a3155700 10 monclient(hunting): picked mon.noname-b con 0x7f219c090d70 addr v1:172.21.15.102:6790/0 2021-05-25T12:04:11.906 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.889+0000 7f21a3155700 10 monclient(hunting): picked mon.noname-c con 0x7f219c12e330 addr v1:172.21.15.181:6789/0 2021-05-25T12:04:11.906 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.889+0000 7f21a3155700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f219c096fa0 con 0x7f219c090990 2021-05-25T12:04:11.907 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.889+0000 7f21a3155700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f219c0926a0 con 0x7f219c12e330 2021-05-25T12:04:11.907 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.889+0000 7f21a3155700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f219c092940 con 0x7f219c090d70 2021-05-25T12:04:11.907 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.889+0000 7f21a3155700 10 monclient(hunting): _renew_subs 2021-05-25T12:04:11.907 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.889+0000 7f21a3155700 10 monclient(hunting): authenticate will time out at 1052.486449s 2021-05-25T12:04:11.907 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.889+0000 7f21a0ef1700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f219c090990 0x7f219c093240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60754/0 (socket says 172.21.15.102:60754) 2021-05-25T12:04:11.908 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.889+0000 7f21a0ef1700 1 -- 172.21.15.102:0/3571314233 learned_addr learned my addr 172.21.15.102:0/3571314233 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:04:11.908 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.889+0000 7f21937fe700 1 -- 172.21.15.102:0/3571314233 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1060029335 0 0) 0x7f219c096fa0 con 0x7f219c090990 2021-05-25T12:04:11.908 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.889+0000 7f21937fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:04:11.908 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.889+0000 7f21937fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:04:11.909 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.889+0000 7f21937fe700 10 monclient(hunting): my global_id is 174474 2021-05-25T12:04:11.909 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.889+0000 7f21937fe700 1 -- 172.21.15.102:0/3571314233 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2188003270 con 0x7f219c090990 2021-05-25T12:04:11.909 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.889+0000 7f21937fe700 1 -- 172.21.15.102:0/3571314233 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2841573397 0 0) 0x7f219c092940 con 0x7f219c090d70 2021-05-25T12:04:11.909 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.889+0000 7f21937fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:04:11.909 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.889+0000 7f21937fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:04:11.910 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.889+0000 7f21937fe700 10 monclient(hunting): my global_id is 184199 2021-05-25T12:04:11.910 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.889+0000 7f21937fe700 1 -- 172.21.15.102:0/3571314233 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f219c096fa0 con 0x7f219c090d70 2021-05-25T12:04:11.910 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.889+0000 7f21937fe700 1 -- 172.21.15.102:0/3571314233 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3816878178 0 0) 0x7f219c0926a0 con 0x7f219c12e330 2021-05-25T12:04:11.910 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.889+0000 7f21937fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:04:11.910 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.889+0000 7f21937fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:04:11.911 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.889+0000 7f21937fe700 10 monclient(hunting): my global_id is 95383 2021-05-25T12:04:11.911 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.889+0000 7f21937fe700 1 -- 172.21.15.102:0/3571314233 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f219c092940 con 0x7f219c12e330 2021-05-25T12:04:11.911 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.890+0000 7f21937fe700 1 -- 172.21.15.102:0/3571314233 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1878867654 0 0) 0x7f219c096fa0 con 0x7f219c090d70 2021-05-25T12:04:11.911 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.890+0000 7f21937fe700 1 -- 172.21.15.102:0/3571314233 >> v1:172.21.15.181:6789/0 conn(0x7f219c12e330 legacy=0x7f219c130720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:11.912 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.890+0000 7f21937fe700 1 -- 172.21.15.102:0/3571314233 >> v1:172.21.15.102:6789/0 conn(0x7f219c090990 legacy=0x7f219c093240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:11.912 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.890+0000 7f21937fe700 10 monclient: _finish_hunting 0 2021-05-25T12:04:11.912 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.890+0000 7f21937fe700 1 monclient: found mon.noname-b 2021-05-25T12:04:11.912 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.890+0000 7f21937fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:04:11.912 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.890+0000 7f21937fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T12:04:11.913 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.890+0000 7f21937fe700 1 -- 172.21.15.102:0/3571314233 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f219c092be0 con 0x7f219c090d70 2021-05-25T12:04:11.913 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.890+0000 7f21937fe700 10 monclient: _finish_auth 0 2021-05-25T12:04:11.913 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.890+0000 7f21937fe700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:04:11.913 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.890+0000 7f21a3155700 5 monclient: authenticate success, global_id 184199 2021-05-25T12:04:11.914 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.890+0000 7f21a3155700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:04:11.914 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.890+0000 7f21937fe700 1 -- 172.21.15.102:0/3571314233 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f21840014a0 con 0x7f219c090d70 2021-05-25T12:04:11.914 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.890+0000 7f21937fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:11.914 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.890+0000 7f21937fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:04:11.914 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.890+0000 7f21937fe700 10 monclient: dump: 2021-05-25T12:04:11.915 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:11.915 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:11.915 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:11.915 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:11.916 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:11.916 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:11.916 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:11.916 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:11.916 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:11.917 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:11.917 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.890+0000 7f21a3155700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:04:11.917 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.890+0000 7f21937fe700 1 -- 172.21.15.102:0/3571314233 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2184001750 con 0x7f219c090d70 2021-05-25T12:04:11.917 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.890+0000 7f21937fe700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:04:11.917 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.890+0000 7f21a3155700 10 monclient: get_monmap_and_config success 2021-05-25T12:04:11.917 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.890+0000 7f21937fe700 1 -- 172.21.15.102:0/3571314233 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f2184001ae0 con 0x7f219c090d70 2021-05-25T12:04:11.918 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.890+0000 7f21937fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:11.918 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.890+0000 7f21a3155700 4 set_mon_vals no callback set 2021-05-25T12:04:11.918 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.890+0000 7f21937fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:04:11.918 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.890+0000 7f21937fe700 10 monclient: dump: 2021-05-25T12:04:11.919 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:11.919 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:11.919 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:11.919 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:11.919 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:11.920 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:11.920 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:11.920 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:11.920 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:11.920 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:11.921 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.890+0000 7f21a3155700 10 monclient: shutdown 2021-05-25T12:04:11.921 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.890+0000 7f21a3155700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:04:11.921 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.890+0000 7f21a3155700 1 -- 172.21.15.102:0/3571314233 >> v1:172.21.15.102:6790/0 conn(0x7f219c090d70 legacy=0x7f219c091200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:11.921 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.890+0000 7f21a3155700 1 -- 172.21.15.102:0/3571314233 shutdown_connections 2021-05-25T12:04:11.921 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.890+0000 7f21a3155700 1 -- 172.21.15.102:0/3571314233 >> 172.21.15.102:0/3571314233 conn(0x7f219c09b370 msgr2=0x7f219c097430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:04:11.922 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.890+0000 7f21a3155700 1 -- 172.21.15.102:0/3571314233 shutdown_connections 2021-05-25T12:04:11.922 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.890+0000 7f21a3155700 1 -- 172.21.15.102:0/3571314233 wait complete. 2021-05-25T12:04:11.922 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.891+0000 7f21a3155700 10 monclient: build_initial_monmap 2021-05-25T12:04:11.922 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.891+0000 7f21a3155700 10 monclient: monmap: 2021-05-25T12:04:11.923 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:04:11.923 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:04:11.923 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:04:11.923 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:04:11.924 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:04:11.924 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:04:11.924 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:04:11.924 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:04:11.925 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:04:11.925 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:11.925 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.891+0000 7f21a3155700 1 librados: starting msgr at 2021-05-25T12:04:11.925 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.891+0000 7f21a3155700 1 librados: starting objecter 2021-05-25T12:04:11.925 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.891+0000 7f21a3155700 1 Processor -- start 2021-05-25T12:04:11.926 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.891+0000 7f21a3155700 1 -- start start 2021-05-25T12:04:11.926 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.891+0000 7f21a3155700 1 librados: setting wanted keys 2021-05-25T12:04:11.926 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.891+0000 7f21a3155700 1 librados: calling monclient init 2021-05-25T12:04:11.926 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.891+0000 7f21a3155700 10 monclient: init 2021-05-25T12:04:11.927 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.892+0000 7f21a3155700 10 monclient: _reopen_session rank -1 2021-05-25T12:04:11.927 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.892+0000 7f21a3155700 10 monclient: _add_conns ranks=[0,2,1] 2021-05-25T12:04:11.927 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.892+0000 7f21a3155700 10 monclient(hunting): picked mon.noname-a con 0x7f219c090990 addr v1:172.21.15.102:6789/0 2021-05-25T12:04:11.927 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.892+0000 7f21a3155700 10 monclient(hunting): picked mon.noname-c con 0x7f219c090d70 addr v1:172.21.15.102:6790/0 2021-05-25T12:04:11.927 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.892+0000 7f21a3155700 10 monclient(hunting): picked mon.noname-b con 0x7f219c12e330 addr v1:172.21.15.181:6789/0 2021-05-25T12:04:11.928 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.892+0000 7f21a3155700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f219c064b20 con 0x7f219c090990 2021-05-25T12:04:11.928 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.892+0000 7f21a3155700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f219c1134d0 con 0x7f219c12e330 2021-05-25T12:04:11.928 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.892+0000 7f21a3155700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f219c113770 con 0x7f219c090d70 2021-05-25T12:04:11.928 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.892+0000 7f21a3155700 10 monclient(hunting): _renew_subs 2021-05-25T12:04:11.928 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.892+0000 7f21a3155700 10 monclient(hunting): authenticate will time out at 1052.489234s 2021-05-25T12:04:11.929 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.892+0000 7f21a0ef1700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f219c090990 0x7f219c063d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60760/0 (socket says 172.21.15.102:60760) 2021-05-25T12:04:11.929 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.892+0000 7f21a0ef1700 1 -- 172.21.15.102:0/2998822868 learned_addr learned my addr 172.21.15.102:0/2998822868 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:04:11.929 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.892+0000 7f2190ff9700 10 client.?.objecter ms_handle_connect 0x7f219c090d70 2021-05-25T12:04:11.929 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.892+0000 7f2190ff9700 10 client.?.objecter resend_mon_ops 2021-05-25T12:04:11.929 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.892+0000 7f2190ff9700 10 client.?.objecter ms_handle_connect 0x7f219c090990 2021-05-25T12:04:11.930 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.892+0000 7f2190ff9700 10 client.?.objecter resend_mon_ops 2021-05-25T12:04:11.930 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.892+0000 7f2190ff9700 1 -- 172.21.15.102:0/2998822868 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 376068056 0 0) 0x7f219c113770 con 0x7f219c090d70 2021-05-25T12:04:11.930 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.892+0000 7f2190ff9700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:04:11.930 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.892+0000 7f2190ff9700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:04:11.931 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.892+0000 7f2190ff9700 10 monclient(hunting): my global_id is 184202 2021-05-25T12:04:11.931 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.892+0000 7f2190ff9700 1 -- 172.21.15.102:0/2998822868 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f217c004c10 con 0x7f219c090d70 2021-05-25T12:04:11.931 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.892+0000 7f2190ff9700 10 client.?.objecter ms_handle_connect 0x7f219c12e330 2021-05-25T12:04:11.931 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.892+0000 7f2190ff9700 10 client.?.objecter resend_mon_ops 2021-05-25T12:04:11.932 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.892+0000 7f2190ff9700 1 -- 172.21.15.102:0/2998822868 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1043268389 0 0) 0x7f219c064b20 con 0x7f219c090990 2021-05-25T12:04:11.932 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.892+0000 7f2190ff9700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:04:11.932 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.892+0000 7f2190ff9700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:04:11.932 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.892+0000 7f2190ff9700 10 monclient(hunting): my global_id is 174477 2021-05-25T12:04:11.932 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.892+0000 7f2190ff9700 1 -- 172.21.15.102:0/2998822868 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f219c113770 con 0x7f219c090990 2021-05-25T12:04:11.933 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.892+0000 7f2190ff9700 1 -- 172.21.15.102:0/2998822868 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 609774038 0 0) 0x7f219c1134d0 con 0x7f219c12e330 2021-05-25T12:04:11.933 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.892+0000 7f2190ff9700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:04:11.933 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.892+0000 7f2190ff9700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:04:11.933 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.892+0000 7f2190ff9700 10 monclient(hunting): my global_id is 95386 2021-05-25T12:04:11.933 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.892+0000 7f2190ff9700 1 -- 172.21.15.102:0/2998822868 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f219c064b20 con 0x7f219c12e330 2021-05-25T12:04:11.934 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.892+0000 7f2190ff9700 1 -- 172.21.15.102:0/2998822868 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 522203703 0 0) 0x7f217c004c10 con 0x7f219c090d70 2021-05-25T12:04:11.934 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.892+0000 7f2190ff9700 1 -- 172.21.15.102:0/2998822868 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f219c1134d0 con 0x7f219c090d70 2021-05-25T12:04:11.934 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.892+0000 7f2190ff9700 1 -- 172.21.15.102:0/2998822868 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f2184001560 con 0x7f219c090d70 2021-05-25T12:04:11.934 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.892+0000 7f2190ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:11.935 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.892+0000 7f2190ff9700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:04:11.935 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.892+0000 7f2190ff9700 10 monclient(hunting): dump: 2021-05-25T12:04:11.935 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:11.935 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:11.935 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:11.936 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:11.936 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:11.936 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:11.936 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:11.937 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:11.937 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:11.937 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:11.937 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.892+0000 7f2190ff9700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:04:11.937 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.892+0000 7f2190ff9700 1 -- 172.21.15.102:0/2998822868 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3008955204 0 0) 0x7f219c113770 con 0x7f219c090990 2021-05-25T12:04:11.938 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.892+0000 7f2190ff9700 1 -- 172.21.15.102:0/2998822868 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f217c004c10 con 0x7f219c090990 2021-05-25T12:04:11.938 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.892+0000 7f2190ff9700 1 -- 172.21.15.102:0/2998822868 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f218c0019f0 con 0x7f219c090990 2021-05-25T12:04:11.938 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.892+0000 7f2190ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:11.938 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.892+0000 7f2190ff9700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2021-05-25T12:04:11.939 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.892+0000 7f2190ff9700 10 monclient(hunting): dump: 2021-05-25T12:04:11.939 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:11.939 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:11.939 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:11.939 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:11.940 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:11.940 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:11.940 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:11.940 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:11.940 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:11.941 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:11.941 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.892+0000 7f2190ff9700 1 -- 172.21.15.102:0/2998822868 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 221964024 0 0) 0x7f219c064b20 con 0x7f219c12e330 2021-05-25T12:04:11.941 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.892+0000 7f2190ff9700 1 -- 172.21.15.102:0/2998822868 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f219c113770 con 0x7f219c12e330 2021-05-25T12:04:11.941 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.892+0000 7f2190ff9700 1 -- 172.21.15.102:0/2998822868 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f2198001c40 con 0x7f219c12e330 2021-05-25T12:04:11.942 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.892+0000 7f2190ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:11.942 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.892+0000 7f2190ff9700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2021-05-25T12:04:11.942 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.892+0000 7f2190ff9700 10 monclient(hunting): dump: 2021-05-25T12:04:11.942 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:11.942 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:11.943 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:11.943 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:11.943 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:11.944 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:11.944 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:11.944 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:11.944 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:11.944 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:11.945 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.892+0000 7f2190ff9700 1 -- 172.21.15.102:0/2998822868 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3387088916 0 0) 0x7f219c1134d0 con 0x7f219c090d70 2021-05-25T12:04:11.945 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.892+0000 7f2190ff9700 1 -- 172.21.15.102:0/2998822868 >> v1:172.21.15.181:6789/0 conn(0x7f219c12e330 legacy=0x7f219c147870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:11.945 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.892+0000 7f2190ff9700 1 -- 172.21.15.102:0/2998822868 >> v1:172.21.15.102:6789/0 conn(0x7f219c090990 legacy=0x7f219c063d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:11.945 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.892+0000 7f2190ff9700 10 monclient: _finish_hunting 0 2021-05-25T12:04:11.945 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.892+0000 7f2190ff9700 1 monclient: found mon.c 2021-05-25T12:04:11.946 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.892+0000 7f2190ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:04:11.946 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.892+0000 7f2190ff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:04:11.946 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.892+0000 7f2190ff9700 1 -- 172.21.15.102:0/2998822868 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f219c113a10 con 0x7f219c090d70 2021-05-25T12:04:11.946 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.892+0000 7f2190ff9700 10 monclient: _finish_auth 0 2021-05-25T12:04:11.947 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.892+0000 7f2190ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:04:11.947 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.892+0000 7f21a3155700 5 monclient: authenticate success, global_id 184202 2021-05-25T12:04:11.947 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.892+0000 7f21a3155700 10 monclient: _renew_subs 2021-05-25T12:04:11.947 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.893+0000 7f21a3155700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:04:11.947 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.893+0000 7f21a3155700 1 -- 172.21.15.102:0/2998822868 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f219c0657a0 con 0x7f219c090d70 2021-05-25T12:04:11.948 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.893+0000 7f21a3155700 10 client.184202.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:04:11.948 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.893+0000 7f21a3155700 10 monclient: _renew_subs 2021-05-25T12:04:11.948 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.893+0000 7f21a3155700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:04:11.948 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.893+0000 7f21a3155700 1 -- 172.21.15.102:0/2998822868 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f219c113f90 con 0x7f219c090d70 2021-05-25T12:04:11.948 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.893+0000 7f21a3155700 1 librados: init done 2021-05-25T12:04:11.949 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.893+0000 7f2190ff9700 1 -- 172.21.15.102:0/2998822868 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2184002240 con 0x7f219c090d70 2021-05-25T12:04:11.949 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.893+0000 7f2190ff9700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:04:11.949 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.893+0000 7f2190ff9700 1 -- 172.21.15.102:0/2998822868 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f21840025d0 con 0x7f219c090d70 2021-05-25T12:04:11.949 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.893+0000 7f2190ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:11.950 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.893+0000 7f2192ffd700 4 set_mon_vals no callback set 2021-05-25T12:04:11.950 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.893+0000 7f2190ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:04:11.950 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.893+0000 7f2190ff9700 10 monclient: dump: 2021-05-25T12:04:11.950 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:11.950 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:11.951 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:11.951 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:11.951 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:11.951 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:11.952 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:11.952 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:11.952 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:11.952 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:11.952 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.893+0000 7f21a3155700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:04:11.953 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.893+0000 7f21a3155700 1 -- 172.21.15.102:0/2998822868 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f21740065a0 con 0x7f2174002840 2021-05-25T12:04:11.953 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.893+0000 7f2190ff9700 1 -- 172.21.15.102:0/2998822868 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f21840027b0 con 0x7f219c090d70 2021-05-25T12:04:11.953 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.893+0000 7f2190ff9700 1 -- 172.21.15.102:0/2998822868 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(127..127 src has 1..127) v4 ==== 5218+0+0 (unknown 2630921626 0 0) 0x7f218403b030 con 0x7f219c090d70 2021-05-25T12:04:11.953 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.894+0000 7f2190ff9700 10 client.184202.objecter ms_dispatch 0x7f219c001550 osd_map(127..127 src has 1..127) v4 2021-05-25T12:04:11.954 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.894+0000 7f2190ff9700 3 client.184202.objecter handle_osd_map got epochs [127,127] > 0 2021-05-25T12:04:11.954 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.894+0000 7f2190ff9700 3 client.184202.objecter handle_osd_map decoding full epoch 127 2021-05-25T12:04:11.954 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.894+0000 7f2190ff9700 20 client.184202.objecter dump_active .. 0 homeless 2021-05-25T12:04:11.954 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.894+0000 7f2190ff9700 10 client.184202.objecter ms_handle_connect 0x7f2174002840 2021-05-25T12:04:11.954 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.894+0000 7f2190ff9700 10 client.184202.objecter resend_mon_ops 2021-05-25T12:04:11.955 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.894+0000 7f2190ff9700 1 -- 172.21.15.102:0/2998822868 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3348783176 0 0) 0x7f21740065a0 con 0x7f2174002840 2021-05-25T12:04:11.955 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.894+0000 7f2190ff9700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:04:11.955 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.894+0000 7f2190ff9700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:04:11.955 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.894+0000 7f2190ff9700 10 monclient(hunting): my global_id is 95389 2021-05-25T12:04:11.955 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.894+0000 7f2190ff9700 1 -- 172.21.15.102:0/2998822868 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f219c1134d0 con 0x7f2174002840 2021-05-25T12:04:11.956 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.894+0000 7f21a16f2700 10 monclient: get_auth_request con 0x7f217c033460 auth_method 0 2021-05-25T12:04:11.956 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.894+0000 7f2190ff9700 1 -- 172.21.15.102:0/2998822868 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3027571126 0 0) 0x7f219c1134d0 con 0x7f2174002840 2021-05-25T12:04:11.956 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.894+0000 7f2190ff9700 1 -- 172.21.15.102:0/2998822868 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f21740065a0 con 0x7f2174002840 2021-05-25T12:04:11.956 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.894+0000 7f2190ff9700 10 client.184202.objecter ms_handle_connect 0x7f217c033460 2021-05-25T12:04:11.957 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.894+0000 7f2190ff9700 1 -- 172.21.15.102:0/2998822868 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f218c0048c0 con 0x7f2174002840 2021-05-25T12:04:11.957 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.894+0000 7f2190ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:11.957 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.894+0000 7f2190ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T12:04:11.957 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.894+0000 7f2190ff9700 10 monclient: dump: 2021-05-25T12:04:11.957 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:11.958 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:11.958 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:11.958 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:11.958 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:11.958 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:11.959 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:11.959 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:11.959 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:11.959 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:11.959 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.894+0000 7f2190ff9700 1 -- 172.21.15.102:0/2998822868 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2070198201 0 0) 0x7f21740065a0 con 0x7f2174002840 2021-05-25T12:04:11.960 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.894+0000 7f2190ff9700 1 -- 172.21.15.102:0/2998822868 --> v1:172.21.15.181:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2174009e40 con 0x7f2174002840 2021-05-25T12:04:11.960 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.894+0000 7f2190ff9700 1 -- 172.21.15.102:0/2998822868 <== mon.1 v1:172.21.15.181:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7f218c004ce0 con 0x7f2174002840 2021-05-25T12:04:11.960 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.894+0000 7f2190ff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:04:11.960 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.894+0000 7f2190ff9700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:04:11.960 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.894+0000 7f2190ff9700 1 -- 172.21.15.102:0/2998822868 >> v1:172.21.15.181:6789/0 conn(0x7f2174002840 legacy=0x7f2174004ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:11.961 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.895+0000 7f2190ff9700 1 -- 172.21.15.102:0/2998822868 >> v1:172.21.15.181:6789/0 conn(0x7f2174002840 legacy=0x7f2174004ce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:04:11.961 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.901+0000 7f21a3155700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:04:11.961 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.901+0000 7f21a3155700 1 -- 172.21.15.102:0/2998822868 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f2174007510 con 0x7f21740068b0 2021-05-25T12:04:11.961 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.901+0000 7f2190ff9700 10 client.184202.objecter ms_handle_connect 0x7f21740068b0 2021-05-25T12:04:11.961 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.901+0000 7f2190ff9700 10 client.184202.objecter resend_mon_ops 2021-05-25T12:04:11.962 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.902+0000 7f2190ff9700 1 -- 172.21.15.102:0/2998822868 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1321932876 0 0) 0x7f2174007510 con 0x7f21740068b0 2021-05-25T12:04:11.962 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.902+0000 7f2190ff9700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:04:11.962 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.902+0000 7f2190ff9700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:04:11.962 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.902+0000 7f2190ff9700 10 monclient(hunting): my global_id is 95392 2021-05-25T12:04:11.963 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.902+0000 7f2190ff9700 1 -- 172.21.15.102:0/2998822868 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21740065a0 con 0x7f21740068b0 2021-05-25T12:04:11.963 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.902+0000 7f2190ff9700 1 -- 172.21.15.102:0/2998822868 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 969024675 0 0) 0x7f21740065a0 con 0x7f21740068b0 2021-05-25T12:04:11.963 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.902+0000 7f2190ff9700 1 -- 172.21.15.102:0/2998822868 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2174007510 con 0x7f21740068b0 2021-05-25T12:04:11.963 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.902+0000 7f2190ff9700 1 -- 172.21.15.102:0/2998822868 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f218c0052d0 con 0x7f21740068b0 2021-05-25T12:04:11.963 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.902+0000 7f2190ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:11.964 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.902+0000 7f2190ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T12:04:11.964 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.902+0000 7f2190ff9700 10 monclient: dump: 2021-05-25T12:04:11.964 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:11.964 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:11.965 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:11.965 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:11.965 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:11.965 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:11.965 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:11.966 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:11.966 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:11.966 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:11.966 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.902+0000 7f2190ff9700 1 -- 172.21.15.102:0/2998822868 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1616382336 0 0) 0x7f2174007510 con 0x7f21740068b0 2021-05-25T12:04:11.966 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.902+0000 7f2190ff9700 1 -- 172.21.15.102:0/2998822868 --> v1:172.21.15.181:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f2174008770 con 0x7f21740068b0 2021-05-25T12:04:11.967 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.902+0000 7f2190ff9700 1 -- 172.21.15.102:0/2998822868 <== mon.1 v1:172.21.15.181:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3772 (unknown 0 0 2919629096) 0x7f218c006550 con 0x7f21740068b0 2021-05-25T12:04:11.967 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.902+0000 7f2190ff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:04:11.967 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.902+0000 7f2190ff9700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:04:11.967 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.902+0000 7f2190ff9700 1 -- 172.21.15.102:0/2998822868 >> v1:172.21.15.181:6789/0 conn(0x7f21740068b0 legacy=0x7f2174006d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:11.968 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.902+0000 7f2190ff9700 1 -- 172.21.15.102:0/2998822868 >> v1:172.21.15.181:6789/0 conn(0x7f21740068b0 legacy=0x7f2174006d40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:04:11.968 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:04:11.968 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:04:11.968 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:04:11.969 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "leader", 2021-05-25T12:04:11.969 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 38, 2021-05-25T12:04:11.969 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:04:11.969 INFO:teuthology.orchestra.run.smithi102.stdout: 0, 2021-05-25T12:04:11.970 INFO:teuthology.orchestra.run.smithi102.stdout: 1, 2021-05-25T12:04:11.970 INFO:teuthology.orchestra.run.smithi102.stdout: 2 2021-05-25T12:04:11.970 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:04:11.970 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 4, 2021-05-25T12:04:11.971 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:04:11.971 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:04:11.971 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:04:11.971 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:04:11.971 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:04:11.972 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:04:11.972 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:04:11.972 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:04:11.972 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:04:11.973 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:04:11.973 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:04:11.973 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:04:11.973 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:04:11.973 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:04:11.974 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:04:11.974 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:04:11.974 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:04:11.974 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:04:11.975 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:04:11.975 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:04:11.975 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:04:11.975 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:04:11.975 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:04:11.976 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:04:11.976 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:04:11.976 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:04:11.976 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:04:11.977 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:04:11.977 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:04:11.977 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:04:11.977 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:04:11.978 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:04:11.978 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:04:11.978 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:04:11.978 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:04:11.978 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:04:11.979 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:04:11.979 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:04:11.979 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:04:11.979 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:04:11.980 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:04:11.980 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:04:11.980 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:04:11.980 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:04:11.980 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:04:11.981 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:04:11.981 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:04:11.981 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:04:11.981 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:04:11.981 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:04:11.982 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:04:11.982 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:04:11.982 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:04:11.982 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:04:11.983 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:04:11.983 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:04:11.983 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:04:11.983 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:04:11.983 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:04:11.984 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:04:11.984 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:04:11.984 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:04:11.984 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:04:11.985 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:04:11.985 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:04:11.985 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:04:11.985 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:04:11.985 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:04:11.986 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:04:11.986 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:04:11.986 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:04:11.986 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:04:11.987 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:04:11.987 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:04:11.987 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:04:11.987 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:04:11.987 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:04:11.988 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:04:11.988 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:04:11.988 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:04:11.988 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:04:11.989 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:04:11.989 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:04:11.989 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:04:11.989 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:04:11.989 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:04:11.990 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:04:11.990 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:04:11.990 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:04:11.990 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:04:11.990 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:04:11.991 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:04:11.991 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:04:11.991 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:04:11.991 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:04:11.992 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:04:11.992 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:04:11.992 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:04:11.992 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:04:11.992 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:04:11.993 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:04:11.993 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:04:11.993 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:04:11.993 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:04:11.993 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:04:11.994 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:04:11.994 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:04:11.994 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:04:11.994 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:04:11.995 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:04:11.995 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:04:11.995 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:04:11.995 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:04:11.995 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:04:11.996 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:04:11.996 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:04:11.996 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:04:11.996 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:04:11.997 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:04:11.997 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:04:11.997 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:04:11.997 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:04:11.997 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 2 2021-05-25T12:04:11.998 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:04:11.998 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:04:11.998 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:04:11.998 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:04:11.999 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:04:12.001 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.903+0000 7f21a3155700 10 librados: watch_flush enter 2021-05-25T12:04:12.001 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.903+0000 7f21a3155700 10 librados: watch_flush exit 2021-05-25T12:04:12.001 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.903+0000 7f21a3155700 20 client.184202.objecter shutdown clearing up homeless session... 2021-05-25T12:04:12.001 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.903+0000 7f21a3155700 10 client.184202.objecter successfully canceled tick 2021-05-25T12:04:12.002 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.903+0000 7f21a3155700 1 -- 172.21.15.102:0/2998822868 >> v1:172.21.15.181:6812/29582 conn(0x7f217c033460 legacy=0x7f217c035920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:12.002 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.903+0000 7f21a3155700 10 monclient: shutdown 2021-05-25T12:04:12.002 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.903+0000 7f21a3155700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:04:12.002 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.903+0000 7f21a3155700 1 -- 172.21.15.102:0/2998822868 >> v1:172.21.15.102:6790/0 conn(0x7f219c090d70 legacy=0x7f219c064410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:12.003 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.903+0000 7f21a16f2700 1 -- 172.21.15.102:0/2998822868 reap_dead start 2021-05-25T12:04:12.003 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.903+0000 7f21a16f2700 1 -- 172.21.15.102:0/2998822868 reap_dead start 2021-05-25T12:04:12.003 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.903+0000 7f21a3155700 1 -- 172.21.15.102:0/2998822868 shutdown_connections 2021-05-25T12:04:12.003 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.903+0000 7f21a3155700 1 -- 172.21.15.102:0/2998822868 >> 172.21.15.102:0/2998822868 conn(0x7f219c09b370 msgr2=0x7f219c0682c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:04:12.004 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.903+0000 7f21a3155700 1 -- 172.21.15.102:0/2998822868 shutdown_connections 2021-05-25T12:04:12.004 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.903+0000 7f21a3155700 1 -- 172.21.15.102:0/2998822868 wait complete. 2021-05-25T12:04:12.004 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:11.903+0000 7f21a3155700 1 librados: shutdown 2021-05-25T12:04:12.005 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2021-05-25T12:04:12.005 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['c'] 2021-05-25T12:04:12.005 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2021-05-25T12:04:12.006 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2021-05-25T12:04:12.006 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2021-05-25T12:04:12.006 INFO:teuthology.orchestra.run:waiting for 300 2021-05-25T12:04:12.018 INFO:tasks.ceph.mon.b:Stopped 2021-05-25T12:04:12.018 INFO:tasks.mon_thrash:Sending STOP to mon c 2021-05-25T12:04:12.019 INFO:tasks.ceph.mon.c:Sent signal 19 2021-05-25T12:04:12.019 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs to unfreeze mons 2021-05-25T12:04:32.021 INFO:tasks.mon_thrash:Sending CONT to mon c 2021-05-25T12:04:32.022 INFO:tasks.ceph.mon.c:Sent signal 18 2021-05-25T12:04:32.022 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2021-05-25T12:04:32.023 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2021-05-25T12:04:32.139 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:32.137+0000 7effd13cd700 10 monclient: get_monmap_and_config 2021-05-25T12:04:32.139 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:32.137+0000 7effd13cd700 10 monclient: build_initial_monmap 2021-05-25T12:04:32.140 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:32.137+0000 7effd13cd700 10 monclient: monmap: 2021-05-25T12:04:32.140 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:04:32.140 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:32.140 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:04:32.138728+0000 2021-05-25T12:04:32.141 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:04:32.138728+0000 2021-05-25T12:04:32.141 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:04:32.141 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:32.141 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:04:32.141 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:04:32.142 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:04:32.142 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:32.142 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:32.138+0000 7effd13cd700 1 Processor -- start 2021-05-25T12:04:32.142 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:32.138+0000 7effd13cd700 1 -- start start 2021-05-25T12:04:32.142 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:32.138+0000 7effd13cd700 10 monclient: init 2021-05-25T12:04:32.144 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:32.138+0000 7effd13cd700 10 monclient: _reopen_session rank -1 2021-05-25T12:04:32.144 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:32.138+0000 7effd13cd700 10 monclient: _add_conns ranks=[0,1,2] 2021-05-25T12:04:32.144 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:32.138+0000 7effd13cd700 10 monclient(hunting): picked mon.noname-a con 0x7effcc096cd0 addr v1:172.21.15.102:6789/0 2021-05-25T12:04:32.145 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:32.138+0000 7effd13cd700 10 monclient(hunting): picked mon.noname-c con 0x7effcc094000 addr v1:172.21.15.181:6789/0 2021-05-25T12:04:32.145 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:32.138+0000 7effd13cd700 10 monclient(hunting): picked mon.noname-b con 0x7effcc0921c0 addr v1:172.21.15.102:6790/0 2021-05-25T12:04:32.145 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:32.138+0000 7effd13cd700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effcc0931b0 con 0x7effcc096cd0 2021-05-25T12:04:32.145 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:32.138+0000 7effd13cd700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effcc093450 con 0x7effcc094000 2021-05-25T12:04:32.146 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:32.138+0000 7effd13cd700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effcc0936f0 con 0x7effcc0921c0 2021-05-25T12:04:32.146 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:32.138+0000 7effd13cd700 10 monclient(hunting): _renew_subs 2021-05-25T12:04:32.146 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:32.138+0000 7effd13cd700 10 monclient(hunting): authenticate will time out at 1072.735925s 2021-05-25T12:04:32.146 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:32.139+0000 7effca7fc700 1 -- >> v1:172.21.15.181:6789/0 conn(0x7effcc094000 legacy=0x7effcc0910a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2021-05-25T12:04:32.146 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:32.139+0000 7effcb7fe700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7effcc0921c0 0x7effcc131140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:55696/0 (socket says 172.21.15.102:55696) 2021-05-25T12:04:32.147 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:32.139+0000 7effcb7fe700 1 -- 172.21.15.102:0/2894030489 learned_addr learned my addr 172.21.15.102:0/2894030489 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:04:32.341 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:32.340+0000 7effca7fc700 1 -- 172.21.15.102:0/2894030489 >> v1:172.21.15.181:6789/0 conn(0x7effcc094000 legacy=0x7effcc0910a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2021-05-25T12:04:32.742 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:32.741+0000 7effca7fc700 1 -- 172.21.15.102:0/2894030489 >> v1:172.21.15.181:6789/0 conn(0x7effcc094000 legacy=0x7effcc0910a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2021-05-25T12:04:33.543 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:33.542+0000 7effca7fc700 1 -- 172.21.15.102:0/2894030489 >> v1:172.21.15.181:6789/0 conn(0x7effcc094000 legacy=0x7effcc0910a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2021-05-25T12:04:35.141 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:35.139+0000 7effc8ff9700 10 monclient(hunting): tick 2021-05-25T12:04:35.141 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:35.139+0000 7effc8ff9700 1 monclient(hunting): continuing hunt 2021-05-25T12:04:35.141 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:35.139+0000 7effc8ff9700 10 monclient(hunting): _reopen_session rank -1 2021-05-25T12:04:35.141 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:35.139+0000 7effc8ff9700 1 -- 172.21.15.102:0/2894030489 >> v1:172.21.15.102:6790/0 conn(0x7effcc0921c0 legacy=0x7effcc131140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:35.142 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:35.139+0000 7effc8ff9700 1 -- 172.21.15.102:0/2894030489 >> v1:172.21.15.181:6789/0 conn(0x7effcc094000 legacy=0x7effcc0910a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-05-25T12:04:35.142 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:35.139+0000 7effc8ff9700 1 -- 172.21.15.102:0/2894030489 >> v1:172.21.15.102:6789/0 conn(0x7effcc096cd0 legacy=0x7effcc090990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:35.142 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:35.139+0000 7effc8ff9700 10 monclient: _add_conns ranks=[2,0,1] 2021-05-25T12:04:35.142 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:35.139+0000 7effc8ff9700 10 monclient(hunting): picked mon.noname-b con 0x7effac0033e0 addr v1:172.21.15.102:6790/0 2021-05-25T12:04:35.143 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:35.139+0000 7effc8ff9700 10 monclient(hunting): picked mon.noname-a con 0x7effac007070 addr v1:172.21.15.102:6789/0 2021-05-25T12:04:35.143 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:35.139+0000 7effc8ff9700 10 monclient(hunting): picked mon.noname-c con 0x7effcc094000 addr v1:172.21.15.181:6789/0 2021-05-25T12:04:35.143 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:35.139+0000 7effc8ff9700 1 -- 172.21.15.102:0/2894030489 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effcc093ab0 con 0x7effac007070 2021-05-25T12:04:35.143 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:35.139+0000 7effc8ff9700 1 -- 172.21.15.102:0/2894030489 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effcc093450 con 0x7effcc094000 2021-05-25T12:04:35.144 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:35.139+0000 7effc8ff9700 1 -- 172.21.15.102:0/2894030489 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effac0105c0 con 0x7effac0033e0 2021-05-25T12:04:35.144 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:35.139+0000 7effc8ff9700 10 monclient(hunting): _renew_subs 2021-05-25T12:04:35.144 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:35.139+0000 7effcb7fe700 1 -- 172.21.15.102:0/2894030489 >> v1:172.21.15.181:6789/0 conn(0x7effcc094000 legacy=0x7effac00ce10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2021-05-25T12:04:35.341 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:35.340+0000 7effcb7fe700 1 -- 172.21.15.102:0/2894030489 >> v1:172.21.15.181:6789/0 conn(0x7effcc094000 legacy=0x7effac00ce10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2021-05-25T12:04:35.718 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:35.717+0000 7effca7fc700 1 -- 172.21.15.102:0/2894030489 >> v1:172.21.15.102:6789/0 conn(0x7effac007070 legacy=0x7effac009500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-05-25T12:04:35.719 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:35.717+0000 7effca7fc700 1 -- 172.21.15.102:0/2894030489 >> v1:172.21.15.102:6789/0 conn(0x7effac007070 legacy=0x7effac009500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-05-25T12:04:35.719 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:35.717+0000 7effca7fc700 1 --1- 172.21.15.102:0/2894030489 >> v1:172.21.15.102:6789/0 conn(0x7effac007070 0x7effac009500 :-1 s=OPENED pgs=412 cs=1 l=1).handle_message read tag failed 2021-05-25T12:04:35.719 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:35.717+0000 7effca7fc700 1 --1- 172.21.15.102:0/2894030489 >> v1:172.21.15.102:6789/0 conn(0x7effac007070 0x7effac009500 :-1 s=OPENED pgs=412 cs=1 l=1).fault on lossy channel, failing 2021-05-25T12:04:35.720 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:35.717+0000 7effc9ffb700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.102:6789/0 2021-05-25T12:04:35.742 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:35.741+0000 7effcb7fe700 1 -- 172.21.15.102:0/2894030489 >> v1:172.21.15.181:6789/0 conn(0x7effcc094000 legacy=0x7effac00ce10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2021-05-25T12:04:36.543 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:36.542+0000 7effcb7fe700 1 -- 172.21.15.102:0/2894030489 >> v1:172.21.15.181:6789/0 conn(0x7effcc094000 legacy=0x7effac00ce10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2021-05-25T12:04:37.029 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.027+0000 7effc9ffb700 1 -- 172.21.15.102:0/2894030489 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4111916562 0 0) 0x7effac0105c0 con 0x7effac0033e0 2021-05-25T12:04:37.029 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.027+0000 7effc9ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:04:37.029 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.027+0000 7effc9ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:04:37.030 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.027+0000 7effc9ffb700 10 monclient(hunting): my global_id is 184205 2021-05-25T12:04:37.030 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.027+0000 7effc9ffb700 1 -- 172.21.15.102:0/2894030489 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effb8003270 con 0x7effac0033e0 2021-05-25T12:04:37.031 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.029+0000 7effc9ffb700 1 -- 172.21.15.102:0/2894030489 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 787031375 0 0) 0x7effb8003270 con 0x7effac0033e0 2021-05-25T12:04:37.031 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.029+0000 7effc9ffb700 1 -- 172.21.15.102:0/2894030489 >> v1:172.21.15.181:6789/0 conn(0x7effcc094000 legacy=0x7effac00ce10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-05-25T12:04:37.032 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.029+0000 7effc9ffb700 1 -- 172.21.15.102:0/2894030489 >> v1:172.21.15.102:6789/0 conn(0x7effac007070 legacy=0x7effac009500 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:04:37.032 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.029+0000 7effc9ffb700 10 monclient: _finish_hunting 0 2021-05-25T12:04:37.032 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.029+0000 7effc9ffb700 1 monclient: found mon.noname-b 2021-05-25T12:04:37.032 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.029+0000 7effc9ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:04:37.033 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.029+0000 7effc9ffb700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T12:04:37.033 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.029+0000 7effc9ffb700 1 -- 172.21.15.102:0/2894030489 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effac0117d0 con 0x7effac0033e0 2021-05-25T12:04:37.033 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.029+0000 7effc9ffb700 10 monclient: _finish_auth 0 2021-05-25T12:04:37.033 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.029+0000 7effc9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:04:37.033 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.029+0000 7effd13cd700 5 monclient: authenticate success, global_id 184205 2021-05-25T12:04:37.034 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.029+0000 7effc9ffb700 1 -- 172.21.15.102:0/2894030489 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7effc0001590 con 0x7effac0033e0 2021-05-25T12:04:37.034 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.029+0000 7effd13cd700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:04:37.034 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.029+0000 7effc9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:37.034 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.029+0000 7effc9ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:04:37.035 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.029+0000 7effc9ffb700 10 monclient: dump: 2021-05-25T12:04:37.035 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:37.035 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:37.035 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:37.036 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:37.036 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:37.036 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:37.036 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:37.037 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:37.037 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:37.037 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:37.037 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.029+0000 7effd13cd700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:04:37.043 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.032+0000 7effc9ffb700 1 -- 172.21.15.102:0/2894030489 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7effc00017e0 con 0x7effac0033e0 2021-05-25T12:04:37.044 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.032+0000 7effc9ffb700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:04:37.044 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.032+0000 7effd13cd700 10 monclient: get_monmap_and_config success 2021-05-25T12:04:37.044 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.032+0000 7effd13cd700 4 set_mon_vals no callback set 2021-05-25T12:04:37.044 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.032+0000 7effc9ffb700 1 -- 172.21.15.102:0/2894030489 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7effc0001bb0 con 0x7effac0033e0 2021-05-25T12:04:37.045 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.032+0000 7effc9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:37.045 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.032+0000 7effc9ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:04:37.045 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.032+0000 7effc9ffb700 10 monclient: dump: 2021-05-25T12:04:37.045 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:37.045 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:37.046 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:37.046 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:37.046 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:37.046 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:37.047 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:37.047 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:37.047 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:37.047 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:37.048 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.033+0000 7effd13cd700 10 monclient: shutdown 2021-05-25T12:04:37.048 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.033+0000 7effd13cd700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:04:37.048 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.033+0000 7effd13cd700 1 -- 172.21.15.102:0/2894030489 >> v1:172.21.15.102:6790/0 conn(0x7effac0033e0 legacy=0x7effac005830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:37.048 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.033+0000 7effd13cd700 1 -- 172.21.15.102:0/2894030489 shutdown_connections 2021-05-25T12:04:37.048 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.033+0000 7effd13cd700 1 -- 172.21.15.102:0/2894030489 >> 172.21.15.102:0/2894030489 conn(0x7effcc09a910 msgr2=0x7effcc09ad20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:04:37.049 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.033+0000 7effd13cd700 1 -- 172.21.15.102:0/2894030489 shutdown_connections 2021-05-25T12:04:37.049 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.033+0000 7effd13cd700 1 -- 172.21.15.102:0/2894030489 wait complete. 2021-05-25T12:04:37.049 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.034+0000 7effd13cd700 10 monclient: build_initial_monmap 2021-05-25T12:04:37.049 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.034+0000 7effd13cd700 10 monclient: monmap: 2021-05-25T12:04:37.049 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:04:37.050 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:04:37.050 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:04:37.050 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:04:37.050 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:04:37.051 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:04:37.051 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:04:37.051 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:04:37.051 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:04:37.051 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:37.052 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.034+0000 7effd13cd700 1 librados: starting msgr at 2021-05-25T12:04:37.052 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.034+0000 7effd13cd700 1 librados: starting objecter 2021-05-25T12:04:37.052 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.034+0000 7effd13cd700 1 Processor -- start 2021-05-25T12:04:37.052 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.035+0000 7effd13cd700 1 -- start start 2021-05-25T12:04:37.052 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.035+0000 7effd13cd700 1 librados: setting wanted keys 2021-05-25T12:04:37.053 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.035+0000 7effd13cd700 1 librados: calling monclient init 2021-05-25T12:04:37.053 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.035+0000 7effd13cd700 10 monclient: init 2021-05-25T12:04:37.053 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.035+0000 7effd13cd700 10 monclient: _reopen_session rank -1 2021-05-25T12:04:37.053 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.035+0000 7effd13cd700 10 monclient: _add_conns ranks=[2,1,0] 2021-05-25T12:04:37.054 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.035+0000 7effd13cd700 10 monclient(hunting): picked mon.noname-c con 0x7effac0033e0 addr v1:172.21.15.102:6790/0 2021-05-25T12:04:37.054 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.035+0000 7effd13cd700 10 monclient(hunting): picked mon.noname-b con 0x7effac007070 addr v1:172.21.15.181:6789/0 2021-05-25T12:04:37.054 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.035+0000 7effd13cd700 10 monclient(hunting): picked mon.noname-a con 0x7effcc094000 addr v1:172.21.15.102:6789/0 2021-05-25T12:04:37.054 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.035+0000 7effd13cd700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effcc068f50 con 0x7effcc094000 2021-05-25T12:04:37.054 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.035+0000 7effca7fc700 1 -- >> v1:172.21.15.181:6789/0 conn(0x7effac007070 legacy=0x7effcc130e40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.181:6789/0 2021-05-25T12:04:37.055 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.035+0000 7effcb7fe700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7effcc094000 0x7effcc13f250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60914/0 (socket says 172.21.15.102:60914) 2021-05-25T12:04:37.055 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.035+0000 7effcb7fe700 1 -- 172.21.15.102:0/1330603685 learned_addr learned my addr 172.21.15.102:0/1330603685 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:04:37.055 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.035+0000 7effd13cd700 1 -- 172.21.15.102:0/1330603685 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effcc117d50 con 0x7effac007070 2021-05-25T12:04:37.055 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.035+0000 7effb2ffd700 10 client.?.objecter ms_handle_connect 0x7effcc094000 2021-05-25T12:04:37.056 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.035+0000 7effb2ffd700 10 client.?.objecter resend_mon_ops 2021-05-25T12:04:37.056 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.035+0000 7effd13cd700 1 -- 172.21.15.102:0/1330603685 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effcc0997f0 con 0x7effac0033e0 2021-05-25T12:04:37.056 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.035+0000 7effd13cd700 10 monclient(hunting): _renew_subs 2021-05-25T12:04:37.056 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.036+0000 7effd13cd700 10 monclient(hunting): authenticate will time out at 1077.633009s 2021-05-25T12:04:37.056 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.036+0000 7effb2ffd700 1 -- 172.21.15.102:0/1330603685 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2268345678 0 0) 0x7effcc068f50 con 0x7effcc094000 2021-05-25T12:04:37.057 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.036+0000 7effb2ffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:04:37.057 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.036+0000 7effb2ffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:04:37.057 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.036+0000 7effb2ffd700 10 monclient(hunting): my global_id is 174480 2021-05-25T12:04:37.057 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.036+0000 7effb2ffd700 1 -- 172.21.15.102:0/1330603685 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effac00a730 con 0x7effcc094000 2021-05-25T12:04:37.057 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.036+0000 7effb2ffd700 1 -- 172.21.15.102:0/1330603685 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3465709365 0 0) 0x7effac00a730 con 0x7effcc094000 2021-05-25T12:04:37.058 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.036+0000 7effb2ffd700 1 -- 172.21.15.102:0/1330603685 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7effcc068f50 con 0x7effcc094000 2021-05-25T12:04:37.058 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.036+0000 7effb2ffd700 1 -- 172.21.15.102:0/1330603685 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7effbc001bc0 con 0x7effcc094000 2021-05-25T12:04:37.058 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.036+0000 7effb2ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:37.058 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.036+0000 7effb2ffd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:04:37.059 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.036+0000 7effb2ffd700 10 monclient(hunting): dump: 2021-05-25T12:04:37.059 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:37.059 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:37.059 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:37.059 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:37.060 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:37.060 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:37.060 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:37.060 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:37.061 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:37.061 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:37.061 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.036+0000 7effb2ffd700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:04:37.061 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.036+0000 7effb2ffd700 1 -- 172.21.15.102:0/1330603685 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1297371110 0 0) 0x7effcc068f50 con 0x7effcc094000 2021-05-25T12:04:37.061 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.036+0000 7effb2ffd700 1 -- 172.21.15.102:0/1330603685 >> v1:172.21.15.102:6790/0 conn(0x7effac0033e0 legacy=0x7effcc137380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:37.062 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.036+0000 7effb2ffd700 1 -- 172.21.15.102:0/1330603685 >> v1:172.21.15.181:6789/0 conn(0x7effac007070 legacy=0x7effcc130e40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-05-25T12:04:37.062 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.036+0000 7effb2ffd700 10 monclient: _finish_hunting 0 2021-05-25T12:04:37.062 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.036+0000 7effb2ffd700 1 monclient: found mon.a 2021-05-25T12:04:37.062 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.036+0000 7effb2ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:04:37.063 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.036+0000 7effb2ffd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:04:37.063 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.036+0000 7effb2ffd700 1 -- 172.21.15.102:0/1330603685 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effcc131550 con 0x7effcc094000 2021-05-25T12:04:37.063 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.036+0000 7effb2ffd700 10 monclient: _finish_auth 0 2021-05-25T12:04:37.063 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.036+0000 7effb2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:04:37.063 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.036+0000 7effd13cd700 5 monclient: authenticate success, global_id 174480 2021-05-25T12:04:37.064 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.036+0000 7effd13cd700 10 monclient: _renew_subs 2021-05-25T12:04:37.064 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.036+0000 7effd13cd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:04:37.064 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.036+0000 7effd13cd700 1 -- 172.21.15.102:0/1330603685 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effcc117350 con 0x7effcc094000 2021-05-25T12:04:37.064 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.036+0000 7effd13cd700 10 client.174480.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:04:37.065 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.036+0000 7effd13cd700 10 monclient: _renew_subs 2021-05-25T12:04:37.065 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.036+0000 7effd13cd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:04:37.065 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.036+0000 7effd13cd700 1 -- 172.21.15.102:0/1330603685 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7effcc115d10 con 0x7effcc094000 2021-05-25T12:04:37.065 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.036+0000 7effd13cd700 1 librados: init done 2021-05-25T12:04:37.065 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.036+0000 7effb2ffd700 1 -- 172.21.15.102:0/1330603685 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7effbc004930 con 0x7effcc094000 2021-05-25T12:04:37.066 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.037+0000 7effb2ffd700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:04:37.066 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.037+0000 7effb2ffd700 1 -- 172.21.15.102:0/1330603685 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7effbc004c40 con 0x7effcc094000 2021-05-25T12:04:37.066 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.037+0000 7effb2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:37.066 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.037+0000 7effb2ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:04:37.067 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.037+0000 7effb2ffd700 10 monclient: dump: 2021-05-25T12:04:37.067 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:37.067 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:37.067 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:37.067 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:37.068 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:37.068 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:37.068 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:37.068 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:37.069 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:37.069 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:37.069 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.037+0000 7effc97fa700 4 set_mon_vals no callback set 2021-05-25T12:04:37.069 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.037+0000 7effb2ffd700 1 -- 172.21.15.102:0/1330603685 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7effbc004da0 con 0x7effcc094000 2021-05-25T12:04:37.069 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.037+0000 7effb2ffd700 1 -- 172.21.15.102:0/1330603685 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(128..128 src has 1..128) v4 ==== 6071+0+0 (unknown 1273967193 0 0) 0x7effbc03dc10 con 0x7effcc094000 2021-05-25T12:04:37.070 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.037+0000 7effb2ffd700 10 client.174480.objecter ms_dispatch 0x7effcc001550 osd_map(128..128 src has 1..128) v4 2021-05-25T12:04:37.070 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.037+0000 7effb2ffd700 3 client.174480.objecter handle_osd_map got epochs [128,128] > 0 2021-05-25T12:04:37.070 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.037+0000 7effb2ffd700 3 client.174480.objecter handle_osd_map decoding full epoch 128 2021-05-25T12:04:37.070 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.038+0000 7effb2ffd700 20 client.174480.objecter dump_active .. 0 homeless 2021-05-25T12:04:37.071 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.038+0000 7effd13cd700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:04:37.071 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.038+0000 7effd13cd700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:04:37.071 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.038+0000 7effd13cd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:04:37.071 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.038+0000 7effd13cd700 1 -- 172.21.15.102:0/1330603685 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effa8004db0 con 0x7effcc094000 2021-05-25T12:04:37.071 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.038+0000 7effcaffd700 10 monclient: get_auth_request con 0x7effac033f90 auth_method 0 2021-05-25T12:04:37.072 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.038+0000 7effb2ffd700 10 client.174480.objecter ms_handle_connect 0x7effac033f90 2021-05-25T12:04:37.072 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.042+0000 7effb2ffd700 1 -- 172.21.15.102:0/1330603685 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7effbc013170 con 0x7effcc094000 2021-05-25T12:04:37.072 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.042+0000 7effb2ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:04:37.072 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.042+0000 7effb2ffd700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:04:37.259 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.257+0000 7effd13cd700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2021-05-25T12:04:37.259 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.257+0000 7effd13cd700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2021-05-25T12:04:37.260 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.257+0000 7effd13cd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:04:37.260 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.257+0000 7effd13cd700 1 -- 172.21.15.102:0/1330603685 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7effa8005060 con 0x7effcc094000 2021-05-25T12:04:37.260 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.257+0000 7effb2ffd700 1 -- 172.21.15.102:0/1330603685 <== mon.0 v1:172.21.15.102:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1336 (unknown 254746657 0 2434773070) 0x7effbc012d60 con 0x7effcc094000 2021-05-25T12:04:37.261 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.257+0000 7effb2ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2021-05-25T12:04:37.261 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.257+0000 7effb2ffd700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:04:37.261 INFO:teuthology.orchestra.run.smithi102.stdout:{"election_epoch":40,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138303579357183","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"20ee91b6-8935-4b98-b868-85c03789ccc4","modified":"2021-05-25T11:59:11.011522Z","created":"2021-05-25T11:59:11.011522Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6790","nonce":0}]},"addr":"172.21.15.102:6790/0","public_addr":"172.21.15.102:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2021-05-25T12:04:37.263 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.259+0000 7effd13cd700 10 librados: watch_flush enter 2021-05-25T12:04:37.263 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.259+0000 7effd13cd700 10 librados: watch_flush exit 2021-05-25T12:04:37.263 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.259+0000 7effd13cd700 20 client.174480.objecter shutdown clearing up homeless session... 2021-05-25T12:04:37.264 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.259+0000 7effd13cd700 10 client.174480.objecter successfully canceled tick 2021-05-25T12:04:37.264 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.259+0000 7effd13cd700 1 -- 172.21.15.102:0/1330603685 >> v1:172.21.15.181:6812/29582 conn(0x7effac033f90 legacy=0x7effac036450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:37.264 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.259+0000 7effd13cd700 10 monclient: shutdown 2021-05-25T12:04:37.264 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.259+0000 7effd13cd700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:04:37.264 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.259+0000 7effd13cd700 1 -- 172.21.15.102:0/1330603685 >> v1:172.21.15.102:6789/0 conn(0x7effcc094000 legacy=0x7effcc13f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:37.265 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.259+0000 7effd13cd700 1 -- 172.21.15.102:0/1330603685 shutdown_connections 2021-05-25T12:04:37.265 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.259+0000 7effd13cd700 1 -- 172.21.15.102:0/1330603685 >> 172.21.15.102:0/1330603685 conn(0x7effcc09a910 msgr2=0x7effcc095aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:04:37.265 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.259+0000 7effd13cd700 1 -- 172.21.15.102:0/1330603685 shutdown_connections 2021-05-25T12:04:37.266 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.259+0000 7effd13cd700 1 -- 172.21.15.102:0/1330603685 wait complete. 2021-05-25T12:04:37.266 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.259+0000 7effd13cd700 1 librados: shutdown 2021-05-25T12:04:37.271 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2021-05-25T12:04:37.272 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2021-05-25T12:04:37.381 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.379+0000 7fc189dc1700 10 monclient: get_monmap_and_config 2021-05-25T12:04:37.381 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.379+0000 7fc189dc1700 10 monclient: build_initial_monmap 2021-05-25T12:04:37.381 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.379+0000 7fc189dc1700 10 monclient: monmap: 2021-05-25T12:04:37.382 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:04:37.382 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:37.382 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:04:37.380803+0000 2021-05-25T12:04:37.382 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:04:37.380803+0000 2021-05-25T12:04:37.383 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:04:37.383 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:37.383 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:04:37.383 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:04:37.384 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:04:37.384 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:37.397 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.380+0000 7fc189dc1700 1 Processor -- start 2021-05-25T12:04:37.397 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.380+0000 7fc189dc1700 1 -- start start 2021-05-25T12:04:37.397 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.380+0000 7fc189dc1700 10 monclient: init 2021-05-25T12:04:37.397 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.381+0000 7fc189dc1700 10 monclient: _reopen_session rank -1 2021-05-25T12:04:37.398 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.381+0000 7fc189dc1700 10 monclient: _add_conns ranks=[2,1,0] 2021-05-25T12:04:37.398 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.381+0000 7fc189dc1700 10 monclient(hunting): picked mon.noname-b con 0x7fc184090990 addr v1:172.21.15.102:6790/0 2021-05-25T12:04:37.398 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.381+0000 7fc189dc1700 10 monclient(hunting): picked mon.noname-c con 0x7fc184090d70 addr v1:172.21.15.181:6789/0 2021-05-25T12:04:37.398 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.381+0000 7fc189dc1700 10 monclient(hunting): picked mon.noname-a con 0x7fc18412e330 addr v1:172.21.15.102:6789/0 2021-05-25T12:04:37.398 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.381+0000 7fc189dc1700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc184096fa0 con 0x7fc18412e330 2021-05-25T12:04:37.399 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.381+0000 7fc189dc1700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1840926a0 con 0x7fc184090d70 2021-05-25T12:04:37.399 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.381+0000 7fc189dc1700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc184092940 con 0x7fc184090990 2021-05-25T12:04:37.399 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.381+0000 7fc189dc1700 10 monclient(hunting): _renew_subs 2021-05-25T12:04:37.399 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.381+0000 7fc189dc1700 10 monclient(hunting): authenticate will time out at 1077.978097s 2021-05-25T12:04:37.399 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.381+0000 7fc183fff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fc18412e330 0x7fc184130720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60918/0 (socket says 172.21.15.102:60918) 2021-05-25T12:04:37.400 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.381+0000 7fc183fff700 1 -- 172.21.15.102:0/904950576 learned_addr learned my addr 172.21.15.102:0/904950576 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:04:37.400 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.381+0000 7fc1837fe700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fc184090990 0x7fc184093240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:55796/0 (socket says 172.21.15.102:55796) 2021-05-25T12:04:37.400 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.381+0000 7fc182ffd700 1 -- 172.21.15.102:0/904950576 >> v1:172.21.15.181:6789/0 conn(0x7fc184090d70 legacy=0x7fc184091200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2021-05-25T12:04:37.400 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.381+0000 7fc1827fc700 1 -- 172.21.15.102:0/904950576 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1136974375 0 0) 0x7fc184096fa0 con 0x7fc18412e330 2021-05-25T12:04:37.401 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.381+0000 7fc1827fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:04:37.401 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.381+0000 7fc1827fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:04:37.401 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.381+0000 7fc1827fc700 10 monclient(hunting): my global_id is 174483 2021-05-25T12:04:37.401 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.381+0000 7fc1827fc700 1 -- 172.21.15.102:0/904950576 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc168003270 con 0x7fc18412e330 2021-05-25T12:04:37.401 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.381+0000 7fc1827fc700 1 -- 172.21.15.102:0/904950576 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 398606064 0 0) 0x7fc184092940 con 0x7fc184090990 2021-05-25T12:04:37.402 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.381+0000 7fc1827fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:04:37.402 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.381+0000 7fc1827fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:04:37.402 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.381+0000 7fc1827fc700 10 monclient(hunting): my global_id is 184208 2021-05-25T12:04:37.402 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.381+0000 7fc1827fc700 1 -- 172.21.15.102:0/904950576 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc184096fa0 con 0x7fc184090990 2021-05-25T12:04:37.402 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.381+0000 7fc1827fc700 1 -- 172.21.15.102:0/904950576 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1423679793 0 0) 0x7fc168003270 con 0x7fc18412e330 2021-05-25T12:04:37.403 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.381+0000 7fc1827fc700 1 -- 172.21.15.102:0/904950576 >> v1:172.21.15.102:6790/0 conn(0x7fc184090990 legacy=0x7fc184093240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:37.403 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.381+0000 7fc1827fc700 1 -- 172.21.15.102:0/904950576 >> v1:172.21.15.181:6789/0 conn(0x7fc184090d70 legacy=0x7fc184091200 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-05-25T12:04:37.403 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.381+0000 7fc1827fc700 10 monclient: _finish_hunting 0 2021-05-25T12:04:37.403 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.381+0000 7fc1827fc700 1 monclient: found mon.noname-a 2021-05-25T12:04:37.404 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.381+0000 7fc1827fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:04:37.404 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.381+0000 7fc1827fc700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T12:04:37.404 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.381+0000 7fc1827fc700 1 -- 172.21.15.102:0/904950576 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc184092be0 con 0x7fc18412e330 2021-05-25T12:04:37.404 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.381+0000 7fc1827fc700 10 monclient: _finish_auth 0 2021-05-25T12:04:37.404 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.381+0000 7fc1827fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:04:37.405 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.381+0000 7fc189dc1700 5 monclient: authenticate success, global_id 174483 2021-05-25T12:04:37.405 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.381+0000 7fc1827fc700 1 -- 172.21.15.102:0/904950576 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fc178001990 con 0x7fc18412e330 2021-05-25T12:04:37.405 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.381+0000 7fc1827fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:37.405 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.381+0000 7fc1827fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:04:37.406 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.381+0000 7fc1827fc700 10 monclient: dump: 2021-05-25T12:04:37.406 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:37.406 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:37.406 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:37.406 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:37.407 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:37.407 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:37.407 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:37.407 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:37.407 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:37.408 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:37.408 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.381+0000 7fc189dc1700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:04:37.408 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.381+0000 7fc1827fc700 1 -- 172.21.15.102:0/904950576 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc178001c00 con 0x7fc18412e330 2021-05-25T12:04:37.408 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.381+0000 7fc1827fc700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:04:37.408 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.381+0000 7fc1827fc700 1 -- 172.21.15.102:0/904950576 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fc178001f90 con 0x7fc18412e330 2021-05-25T12:04:37.409 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.381+0000 7fc189dc1700 10 monclient: get_monmap_and_config success 2021-05-25T12:04:37.409 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.381+0000 7fc189dc1700 4 set_mon_vals no callback set 2021-05-25T12:04:37.409 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.381+0000 7fc1827fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:37.409 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.381+0000 7fc1827fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:04:37.410 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.381+0000 7fc1827fc700 10 monclient: dump: 2021-05-25T12:04:37.410 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:37.410 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:37.410 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:37.410 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:37.411 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:37.411 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:37.411 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:37.411 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:37.411 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:37.412 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:37.412 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.382+0000 7fc189dc1700 10 monclient: shutdown 2021-05-25T12:04:37.412 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.382+0000 7fc189dc1700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:04:37.412 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.382+0000 7fc189dc1700 1 -- 172.21.15.102:0/904950576 >> v1:172.21.15.102:6789/0 conn(0x7fc18412e330 legacy=0x7fc184130720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:37.412 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.382+0000 7fc189dc1700 1 -- 172.21.15.102:0/904950576 shutdown_connections 2021-05-25T12:04:37.413 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.382+0000 7fc189dc1700 1 -- 172.21.15.102:0/904950576 >> 172.21.15.102:0/904950576 conn(0x7fc18409b370 msgr2=0x7fc184097430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:04:37.413 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.382+0000 7fc189dc1700 1 -- 172.21.15.102:0/904950576 shutdown_connections 2021-05-25T12:04:37.413 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.382+0000 7fc189dc1700 1 -- 172.21.15.102:0/904950576 wait complete. 2021-05-25T12:04:37.413 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.382+0000 7fc189dc1700 10 monclient: build_initial_monmap 2021-05-25T12:04:37.413 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.382+0000 7fc189dc1700 10 monclient: monmap: 2021-05-25T12:04:37.414 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:04:37.414 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:04:37.414 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:04:37.414 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:04:37.415 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:04:37.415 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:04:37.415 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:04:37.415 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:04:37.415 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:04:37.416 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:37.416 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.383+0000 7fc189dc1700 1 librados: starting msgr at 2021-05-25T12:04:37.416 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.383+0000 7fc189dc1700 1 librados: starting objecter 2021-05-25T12:04:37.416 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.383+0000 7fc189dc1700 1 Processor -- start 2021-05-25T12:04:37.416 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.383+0000 7fc189dc1700 1 -- start start 2021-05-25T12:04:37.417 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.383+0000 7fc189dc1700 1 librados: setting wanted keys 2021-05-25T12:04:37.417 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.383+0000 7fc189dc1700 1 librados: calling monclient init 2021-05-25T12:04:37.417 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.383+0000 7fc189dc1700 10 monclient: init 2021-05-25T12:04:37.417 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.383+0000 7fc189dc1700 10 monclient: _reopen_session rank -1 2021-05-25T12:04:37.417 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.383+0000 7fc189dc1700 10 monclient: _add_conns ranks=[0,1,2] 2021-05-25T12:04:37.418 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.383+0000 7fc189dc1700 10 monclient(hunting): picked mon.noname-a con 0x7fc184090990 addr v1:172.21.15.102:6789/0 2021-05-25T12:04:37.418 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.383+0000 7fc189dc1700 10 monclient(hunting): picked mon.noname-b con 0x7fc184090d70 addr v1:172.21.15.181:6789/0 2021-05-25T12:04:37.418 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.383+0000 7fc189dc1700 10 monclient(hunting): picked mon.noname-c con 0x7fc18412e330 addr v1:172.21.15.102:6790/0 2021-05-25T12:04:37.418 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.383+0000 7fc189dc1700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc18412a230 con 0x7fc184090990 2021-05-25T12:04:37.418 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.383+0000 7fc189dc1700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc184137760 con 0x7fc184090d70 2021-05-25T12:04:37.419 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.383+0000 7fc189dc1700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc184068890 con 0x7fc18412e330 2021-05-25T12:04:37.419 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.383+0000 7fc189dc1700 10 monclient(hunting): _renew_subs 2021-05-25T12:04:37.419 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.383+0000 7fc189dc1700 10 monclient(hunting): authenticate will time out at 1077.980755s 2021-05-25T12:04:37.419 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.383+0000 7fc1837fe700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fc184090990 0x7fc184136940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60920/0 (socket says 172.21.15.102:60920) 2021-05-25T12:04:37.420 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.383+0000 7fc1837fe700 1 -- 172.21.15.102:0/33133467 learned_addr learned my addr 172.21.15.102:0/33133467 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:04:37.420 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.383+0000 7fc182ffd700 1 -- 172.21.15.102:0/33133467 >> v1:172.21.15.181:6789/0 conn(0x7fc184090d70 legacy=0x7fc184137050 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.181:6789/0 2021-05-25T12:04:37.420 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.383+0000 7fc16f7fe700 10 client.?.objecter ms_handle_connect 0x7fc184090990 2021-05-25T12:04:37.420 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.383+0000 7fc16f7fe700 10 client.?.objecter resend_mon_ops 2021-05-25T12:04:37.420 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.383+0000 7fc16f7fe700 10 client.?.objecter ms_handle_connect 0x7fc18412e330 2021-05-25T12:04:37.421 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.383+0000 7fc16f7fe700 10 client.?.objecter resend_mon_ops 2021-05-25T12:04:37.421 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.384+0000 7fc16f7fe700 1 -- 172.21.15.102:0/33133467 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1489708280 0 0) 0x7fc18412a230 con 0x7fc184090990 2021-05-25T12:04:37.421 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.384+0000 7fc16f7fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:04:37.421 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.384+0000 7fc16f7fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:04:37.421 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.384+0000 7fc16f7fe700 10 monclient(hunting): my global_id is 174486 2021-05-25T12:04:37.422 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.384+0000 7fc16f7fe700 1 -- 172.21.15.102:0/33133467 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc164004c10 con 0x7fc184090990 2021-05-25T12:04:37.422 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.384+0000 7fc16f7fe700 1 -- 172.21.15.102:0/33133467 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1886089369 0 0) 0x7fc184068890 con 0x7fc18412e330 2021-05-25T12:04:37.422 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.384+0000 7fc16f7fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:04:37.422 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.384+0000 7fc16f7fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:04:37.422 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.384+0000 7fc16f7fe700 10 monclient(hunting): my global_id is 184211 2021-05-25T12:04:37.423 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.384+0000 7fc16f7fe700 1 -- 172.21.15.102:0/33133467 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc18412a230 con 0x7fc18412e330 2021-05-25T12:04:37.423 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.384+0000 7fc16f7fe700 1 -- 172.21.15.102:0/33133467 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 892999600 0 0) 0x7fc164004c10 con 0x7fc184090990 2021-05-25T12:04:37.423 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.384+0000 7fc16f7fe700 1 -- 172.21.15.102:0/33133467 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc184068890 con 0x7fc184090990 2021-05-25T12:04:37.423 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.384+0000 7fc16f7fe700 1 -- 172.21.15.102:0/33133467 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fc170001860 con 0x7fc184090990 2021-05-25T12:04:37.423 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.384+0000 7fc16f7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:37.424 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.384+0000 7fc16f7fe700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:04:37.424 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.384+0000 7fc16f7fe700 10 monclient(hunting): dump: 2021-05-25T12:04:37.424 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:37.424 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:37.424 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:37.425 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:37.425 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:37.425 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:37.425 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:37.425 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:37.426 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:37.426 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:37.426 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.384+0000 7fc16f7fe700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:04:37.426 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.384+0000 7fc16f7fe700 1 -- 172.21.15.102:0/33133467 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2148068096 0 0) 0x7fc18412a230 con 0x7fc18412e330 2021-05-25T12:04:37.427 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.384+0000 7fc16f7fe700 1 -- 172.21.15.102:0/33133467 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc164004c10 con 0x7fc18412e330 2021-05-25T12:04:37.427 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.384+0000 7fc16f7fe700 1 -- 172.21.15.102:0/33133467 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fc178001950 con 0x7fc18412e330 2021-05-25T12:04:37.427 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.384+0000 7fc16f7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:37.427 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.384+0000 7fc16f7fe700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2021-05-25T12:04:37.427 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.384+0000 7fc16f7fe700 10 monclient(hunting): dump: 2021-05-25T12:04:37.428 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:37.428 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:37.428 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:37.428 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:37.428 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:37.429 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:37.429 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:37.429 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:37.429 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:37.429 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:37.430 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.384+0000 7fc16f7fe700 1 -- 172.21.15.102:0/33133467 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 770276562 0 0) 0x7fc184068890 con 0x7fc184090990 2021-05-25T12:04:37.430 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.384+0000 7fc16f7fe700 1 -- 172.21.15.102:0/33133467 >> v1:172.21.15.102:6790/0 conn(0x7fc18412e330 legacy=0x7fc18413eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:37.430 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.384+0000 7fc16f7fe700 1 -- 172.21.15.102:0/33133467 >> v1:172.21.15.181:6789/0 conn(0x7fc184090d70 legacy=0x7fc184137050 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-05-25T12:04:37.430 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.384+0000 7fc16f7fe700 10 monclient: _finish_hunting 0 2021-05-25T12:04:37.430 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.384+0000 7fc16f7fe700 1 monclient: found mon.a 2021-05-25T12:04:37.431 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.384+0000 7fc16f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:04:37.431 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.384+0000 7fc16f7fe700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:04:37.431 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.384+0000 7fc16f7fe700 1 -- 172.21.15.102:0/33133467 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc184068b30 con 0x7fc184090990 2021-05-25T12:04:37.431 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.384+0000 7fc16f7fe700 10 monclient: _finish_auth 0 2021-05-25T12:04:37.431 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.384+0000 7fc16f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:04:37.432 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.384+0000 7fc189dc1700 5 monclient: authenticate success, global_id 174486 2021-05-25T12:04:37.432 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.384+0000 7fc189dc1700 10 monclient: _renew_subs 2021-05-25T12:04:37.432 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.384+0000 7fc189dc1700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:04:37.432 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.384+0000 7fc189dc1700 1 -- 172.21.15.102:0/33133467 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1840669b0 con 0x7fc184090990 2021-05-25T12:04:37.432 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.384+0000 7fc189dc1700 10 client.174486.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:04:37.433 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.384+0000 7fc189dc1700 10 monclient: _renew_subs 2021-05-25T12:04:37.433 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.384+0000 7fc189dc1700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:04:37.433 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.384+0000 7fc189dc1700 1 -- 172.21.15.102:0/33133467 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc184065050 con 0x7fc184090990 2021-05-25T12:04:37.433 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.384+0000 7fc189dc1700 1 librados: init done 2021-05-25T12:04:37.434 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.384+0000 7fc16f7fe700 1 -- 172.21.15.102:0/33133467 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc170004530 con 0x7fc184090990 2021-05-25T12:04:37.434 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.384+0000 7fc16f7fe700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:04:37.434 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.384+0000 7fc16f7fe700 1 -- 172.21.15.102:0/33133467 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fc170004840 con 0x7fc184090990 2021-05-25T12:04:37.434 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.384+0000 7fc16f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:37.434 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.384+0000 7fc16f7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:04:37.435 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.384+0000 7fc181ffb700 4 set_mon_vals no callback set 2021-05-25T12:04:37.435 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.384+0000 7fc16f7fe700 10 monclient: dump: 2021-05-25T12:04:37.435 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:37.435 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:37.435 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:37.436 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:37.436 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:37.436 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:37.436 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:37.437 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:37.437 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:37.437 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:37.437 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.385+0000 7fc189dc1700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:04:37.437 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.385+0000 7fc16f7fe700 1 -- 172.21.15.102:0/33133467 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7fc17000dbe0 con 0x7fc184090990 2021-05-25T12:04:37.438 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.385+0000 7fc189dc1700 1 -- 172.21.15.102:0/33133467 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fc15c0065a0 con 0x7fc15c002840 2021-05-25T12:04:37.438 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.385+0000 7fc16f7fe700 1 -- 172.21.15.102:0/33133467 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(128..128 src has 1..128) v4 ==== 6071+0+0 (unknown 1273967193 0 0) 0x7fc17003d790 con 0x7fc184090990 2021-05-25T12:04:37.438 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.385+0000 7fc16f7fe700 10 client.174486.objecter ms_dispatch 0x7fc184001550 osd_map(128..128 src has 1..128) v4 2021-05-25T12:04:37.438 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.385+0000 7fc16f7fe700 3 client.174486.objecter handle_osd_map got epochs [128,128] > 0 2021-05-25T12:04:37.438 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.385+0000 7fc16f7fe700 3 client.174486.objecter handle_osd_map decoding full epoch 128 2021-05-25T12:04:37.439 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.385+0000 7fc16f7fe700 20 client.174486.objecter dump_active .. 0 homeless 2021-05-25T12:04:37.439 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.385+0000 7fc16f7fe700 10 client.174486.objecter ms_handle_connect 0x7fc15c002840 2021-05-25T12:04:37.439 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.385+0000 7fc16f7fe700 10 client.174486.objecter resend_mon_ops 2021-05-25T12:04:37.439 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.385+0000 7fc16f7fe700 1 -- 172.21.15.102:0/33133467 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 521250529 0 0) 0x7fc15c0065a0 con 0x7fc15c002840 2021-05-25T12:04:37.439 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.385+0000 7fc16f7fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:04:37.440 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.385+0000 7fc16f7fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:04:37.440 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.385+0000 7fc16f7fe700 10 monclient(hunting): my global_id is 174489 2021-05-25T12:04:37.440 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.385+0000 7fc16f7fe700 1 -- 172.21.15.102:0/33133467 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc184068890 con 0x7fc15c002840 2021-05-25T12:04:37.440 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.385+0000 7fc183fff700 10 monclient: get_auth_request con 0x7fc1640332a0 auth_method 0 2021-05-25T12:04:37.440 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.385+0000 7fc16f7fe700 1 -- 172.21.15.102:0/33133467 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1774799881 0 0) 0x7fc184068890 con 0x7fc15c002840 2021-05-25T12:04:37.441 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.385+0000 7fc16f7fe700 1 -- 172.21.15.102:0/33133467 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc15c0065a0 con 0x7fc15c002840 2021-05-25T12:04:37.441 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.385+0000 7fc16f7fe700 1 -- 172.21.15.102:0/33133467 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fc174003d30 con 0x7fc15c002840 2021-05-25T12:04:37.441 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.385+0000 7fc16f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:37.441 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.385+0000 7fc16f7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:04:37.441 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.385+0000 7fc16f7fe700 10 monclient: dump: 2021-05-25T12:04:37.442 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:37.442 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:37.442 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:37.442 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:37.443 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:37.443 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:37.443 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:37.443 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:37.443 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:37.444 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:37.444 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.385+0000 7fc16f7fe700 1 -- 172.21.15.102:0/33133467 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1908063064 0 0) 0x7fc15c0065a0 con 0x7fc15c002840 2021-05-25T12:04:37.444 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.385+0000 7fc16f7fe700 1 -- 172.21.15.102:0/33133467 --> v1:172.21.15.102:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc15c009e40 con 0x7fc15c002840 2021-05-25T12:04:37.444 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.385+0000 7fc16f7fe700 10 client.174486.objecter ms_handle_connect 0x7fc1640332a0 2021-05-25T12:04:37.444 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.386+0000 7fc16f7fe700 1 -- 172.21.15.102:0/33133467 <== mon.0 v1:172.21.15.102:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7fc174006120 con 0x7fc15c002840 2021-05-25T12:04:37.445 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.386+0000 7fc16f7fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:04:37.445 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.386+0000 7fc16f7fe700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:04:37.445 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.386+0000 7fc16f7fe700 1 -- 172.21.15.102:0/33133467 >> v1:172.21.15.102:6789/0 conn(0x7fc15c002840 legacy=0x7fc15c004ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:37.445 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.386+0000 7fc16f7fe700 1 -- 172.21.15.102:0/33133467 >> v1:172.21.15.102:6789/0 conn(0x7fc15c002840 legacy=0x7fc15c004ce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:04:37.446 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.392+0000 7fc189dc1700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:04:37.446 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.392+0000 7fc189dc1700 1 -- 172.21.15.102:0/33133467 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fc15c009ad0 con 0x7fc15c008d90 2021-05-25T12:04:37.446 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.392+0000 7fc16f7fe700 10 client.174486.objecter ms_handle_connect 0x7fc15c008d90 2021-05-25T12:04:37.446 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.392+0000 7fc16f7fe700 10 client.174486.objecter resend_mon_ops 2021-05-25T12:04:37.446 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.392+0000 7fc16f7fe700 1 -- 172.21.15.102:0/33133467 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2170885088 0 0) 0x7fc15c009ad0 con 0x7fc15c008d90 2021-05-25T12:04:37.447 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.392+0000 7fc16f7fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:04:37.447 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.392+0000 7fc16f7fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:04:37.447 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.392+0000 7fc16f7fe700 10 monclient(hunting): my global_id is 174492 2021-05-25T12:04:37.447 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.392+0000 7fc16f7fe700 1 -- 172.21.15.102:0/33133467 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc15c0065a0 con 0x7fc15c008d90 2021-05-25T12:04:37.447 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.393+0000 7fc16f7fe700 1 -- 172.21.15.102:0/33133467 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1928568586 0 0) 0x7fc15c0065a0 con 0x7fc15c008d90 2021-05-25T12:04:37.448 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.393+0000 7fc16f7fe700 1 -- 172.21.15.102:0/33133467 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc15c009ad0 con 0x7fc15c008d90 2021-05-25T12:04:37.448 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.393+0000 7fc16f7fe700 1 -- 172.21.15.102:0/33133467 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fc1740045a0 con 0x7fc15c008d90 2021-05-25T12:04:37.448 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.393+0000 7fc16f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:37.448 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.393+0000 7fc16f7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:04:37.448 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.393+0000 7fc16f7fe700 10 monclient: dump: 2021-05-25T12:04:37.449 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:37.449 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:37.449 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:37.449 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:37.449 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:37.450 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:37.450 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:37.450 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:37.450 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:37.450 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:37.451 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.393+0000 7fc16f7fe700 1 -- 172.21.15.102:0/33133467 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1618060260 0 0) 0x7fc15c009ad0 con 0x7fc15c008d90 2021-05-25T12:04:37.451 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.393+0000 7fc16f7fe700 1 -- 172.21.15.102:0/33133467 --> v1:172.21.15.102:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fc15c008740 con 0x7fc15c008d90 2021-05-25T12:04:37.451 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.393+0000 7fc16f7fe700 1 -- 172.21.15.102:0/33133467 <== mon.0 v1:172.21.15.102:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4099 (unknown 0 0 3388224319) 0x7fc1740059a0 con 0x7fc15c008d90 2021-05-25T12:04:37.451 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.393+0000 7fc16f7fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:04:37.451 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.393+0000 7fc16f7fe700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:04:37.452 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.393+0000 7fc16f7fe700 1 -- 172.21.15.102:0/33133467 >> v1:172.21.15.102:6789/0 conn(0x7fc15c008d90 legacy=0x7fc15c009220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:37.452 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.393+0000 7fc16f7fe700 1 -- 172.21.15.102:0/33133467 >> v1:172.21.15.102:6789/0 conn(0x7fc15c008d90 legacy=0x7fc15c009220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:04:37.452 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:04:37.453 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:04:37.453 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:04:37.453 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "peon", 2021-05-25T12:04:37.453 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 40, 2021-05-25T12:04:37.454 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:04:37.454 INFO:teuthology.orchestra.run.smithi102.stdout: 0, 2021-05-25T12:04:37.454 INFO:teuthology.orchestra.run.smithi102.stdout: 2 2021-05-25T12:04:37.454 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:04:37.454 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 0, 2021-05-25T12:04:37.455 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:04:37.455 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:04:37.455 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:04:37.455 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:04:37.456 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:04:37.456 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:04:37.456 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:04:37.456 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:04:37.457 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:04:37.457 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:04:37.457 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:04:37.457 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:04:37.457 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:04:37.458 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:04:37.458 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:04:37.458 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:04:37.458 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:04:37.458 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:04:37.459 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:04:37.459 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:04:37.459 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:04:37.459 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:04:37.459 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:04:37.460 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:04:37.460 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:04:37.460 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:04:37.460 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:04:37.461 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:04:37.461 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:04:37.461 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:04:37.461 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:04:37.461 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:04:37.462 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:04:37.462 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:04:37.462 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:04:37.462 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:04:37.462 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:04:37.463 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:04:37.463 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:04:37.463 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:04:37.463 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:04:37.463 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:04:37.464 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:04:37.464 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:04:37.464 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:04:37.464 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:04:37.464 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:04:37.465 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:04:37.465 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:04:37.465 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:04:37.465 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:04:37.465 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:04:37.466 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:04:37.466 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:04:37.466 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:04:37.466 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:04:37.467 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:04:37.467 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:04:37.467 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:04:37.467 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:04:37.467 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:04:37.468 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:04:37.468 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:04:37.468 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:04:37.468 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:04:37.469 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:04:37.469 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:04:37.469 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:04:37.469 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:04:37.469 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:04:37.470 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:04:37.470 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:04:37.470 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:04:37.470 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:04:37.470 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:04:37.471 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:04:37.471 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:04:37.471 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:04:37.471 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:04:37.471 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:04:37.472 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:04:37.472 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:04:37.472 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:04:37.472 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:04:37.472 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:04:37.473 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:04:37.473 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:04:37.473 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:04:37.473 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:04:37.473 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:04:37.474 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:04:37.474 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:04:37.474 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:04:37.474 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:04:37.475 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:04:37.475 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:04:37.475 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:04:37.475 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:04:37.475 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:04:37.476 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:04:37.476 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:04:37.476 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:04:37.476 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:04:37.476 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:04:37.477 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:04:37.477 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:04:37.477 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:04:37.477 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:04:37.477 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:04:37.478 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:04:37.478 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:04:37.478 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:04:37.478 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:04:37.478 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:04:37.479 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:04:37.479 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:04:37.479 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:04:37.479 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:04:37.480 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": [ 2021-05-25T12:04:37.480 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:04:37.480 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:04:37.480 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:04:37.480 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 3 2021-05-25T12:04:37.481 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:04:37.481 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:04:37.481 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:04:37.481 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:04:37.481 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:04:37.482 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:04:37.482 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 8 2021-05-25T12:04:37.482 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:04:37.482 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:04:37.482 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": [ 2021-05-25T12:04:37.483 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:04:37.483 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:04:37.483 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:04:37.483 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:04:37.483 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:04:37.484 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:04:37.484 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:04:37.484 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:04:37.484 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:04:37.486 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.393+0000 7fc189dc1700 10 librados: watch_flush enter 2021-05-25T12:04:37.487 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.393+0000 7fc189dc1700 10 librados: watch_flush exit 2021-05-25T12:04:37.487 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.393+0000 7fc189dc1700 20 client.174486.objecter shutdown clearing up homeless session... 2021-05-25T12:04:37.487 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.393+0000 7fc189dc1700 10 client.174486.objecter successfully canceled tick 2021-05-25T12:04:37.487 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.393+0000 7fc189dc1700 1 -- 172.21.15.102:0/33133467 >> v1:172.21.15.181:6812/29582 conn(0x7fc1640332a0 legacy=0x7fc164035760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:37.488 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.393+0000 7fc189dc1700 10 monclient: shutdown 2021-05-25T12:04:37.488 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.393+0000 7fc189dc1700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:04:37.488 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.393+0000 7fc189dc1700 1 -- 172.21.15.102:0/33133467 >> v1:172.21.15.102:6789/0 conn(0x7fc184090990 legacy=0x7fc184136940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:37.488 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.393+0000 7fc183fff700 1 -- 172.21.15.102:0/33133467 reap_dead start 2021-05-25T12:04:37.489 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.393+0000 7fc183fff700 1 -- 172.21.15.102:0/33133467 reap_dead start 2021-05-25T12:04:37.489 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.393+0000 7fc189dc1700 1 -- 172.21.15.102:0/33133467 shutdown_connections 2021-05-25T12:04:37.489 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.393+0000 7fc189dc1700 1 -- 172.21.15.102:0/33133467 >> 172.21.15.102:0/33133467 conn(0x7fc18409b370 msgr2=0x7fc184129380 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:04:37.489 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.393+0000 7fc189dc1700 1 -- 172.21.15.102:0/33133467 shutdown_connections 2021-05-25T12:04:37.490 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.393+0000 7fc189dc1700 1 -- 172.21.15.102:0/33133467 wait complete. 2021-05-25T12:04:37.490 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.393+0000 7fc189dc1700 1 librados: shutdown 2021-05-25T12:04:37.490 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2021-05-25T12:04:37.600 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.598+0000 7f32f3fe4700 10 monclient: get_monmap_and_config 2021-05-25T12:04:37.600 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.598+0000 7f32f3fe4700 10 monclient: build_initial_monmap 2021-05-25T12:04:37.601 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.598+0000 7f32f3fe4700 10 monclient: monmap: 2021-05-25T12:04:37.601 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:04:37.601 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:37.601 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:04:37.599441+0000 2021-05-25T12:04:37.602 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:04:37.599441+0000 2021-05-25T12:04:37.602 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:04:37.602 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:37.602 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:04:37.602 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:04:37.603 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:04:37.603 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:37.603 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.599+0000 7f32f3fe4700 1 Processor -- start 2021-05-25T12:04:37.603 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.599+0000 7f32f3fe4700 1 -- start start 2021-05-25T12:04:37.604 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.599+0000 7f32f3fe4700 10 monclient: init 2021-05-25T12:04:37.615 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.599+0000 7f32f3fe4700 10 monclient: _reopen_session rank -1 2021-05-25T12:04:37.616 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.599+0000 7f32f3fe4700 10 monclient: _add_conns ranks=[1,2,0] 2021-05-25T12:04:37.616 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.599+0000 7f32f3fe4700 10 monclient(hunting): picked mon.noname-c con 0x7f32ec090990 addr v1:172.21.15.181:6789/0 2021-05-25T12:04:37.616 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.599+0000 7f32f3fe4700 10 monclient(hunting): picked mon.noname-b con 0x7f32ec090d70 addr v1:172.21.15.102:6790/0 2021-05-25T12:04:37.616 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.599+0000 7f32f3fe4700 10 monclient(hunting): picked mon.noname-a con 0x7f32ec12e330 addr v1:172.21.15.102:6789/0 2021-05-25T12:04:37.616 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.599+0000 7f32f3fe4700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32ec096fa0 con 0x7f32ec12e330 2021-05-25T12:04:37.617 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.599+0000 7f32f3fe4700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32ec0926a0 con 0x7f32ec090990 2021-05-25T12:04:37.617 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.599+0000 7f32f3fe4700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32ec092940 con 0x7f32ec090d70 2021-05-25T12:04:37.617 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.599+0000 7f32f3fe4700 10 monclient(hunting): _renew_subs 2021-05-25T12:04:37.617 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.599+0000 7f32f3fe4700 10 monclient(hunting): authenticate will time out at 1078.196810s 2021-05-25T12:04:37.618 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.599+0000 7f32f2581700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f32ec12e330 0x7f32ec130720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60936/0 (socket says 172.21.15.102:60936) 2021-05-25T12:04:37.618 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.599+0000 7f32f1d80700 1 -- >> v1:172.21.15.181:6789/0 conn(0x7f32ec090990 legacy=0x7f32ec093240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2021-05-25T12:04:37.618 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.599+0000 7f32f2581700 1 -- 172.21.15.102:0/4237184903 learned_addr learned my addr 172.21.15.102:0/4237184903 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:04:37.618 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.599+0000 7f32f157f700 1 --1- 172.21.15.102:0/4237184903 >> v1:172.21.15.102:6790/0 conn(0x7f32ec090d70 0x7f32ec091200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:55812/0 (socket says 172.21.15.102:55812) 2021-05-25T12:04:37.618 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.600+0000 7f32f0d7e700 1 -- 172.21.15.102:0/4237184903 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 74855143 0 0) 0x7f32ec096fa0 con 0x7f32ec12e330 2021-05-25T12:04:37.619 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.600+0000 7f32f0d7e700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:04:37.619 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.600+0000 7f32f0d7e700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:04:37.619 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.600+0000 7f32f0d7e700 10 monclient(hunting): my global_id is 174495 2021-05-25T12:04:37.619 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.600+0000 7f32f0d7e700 1 -- 172.21.15.102:0/4237184903 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32d4003270 con 0x7f32ec12e330 2021-05-25T12:04:37.620 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.600+0000 7f32f0d7e700 1 -- 172.21.15.102:0/4237184903 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4171332949 0 0) 0x7f32ec092940 con 0x7f32ec090d70 2021-05-25T12:04:37.620 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.600+0000 7f32f0d7e700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:04:37.620 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.600+0000 7f32f0d7e700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:04:37.620 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.600+0000 7f32f0d7e700 10 monclient(hunting): my global_id is 184214 2021-05-25T12:04:37.620 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.600+0000 7f32f0d7e700 1 -- 172.21.15.102:0/4237184903 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32ec096fa0 con 0x7f32ec090d70 2021-05-25T12:04:37.621 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.600+0000 7f32f0d7e700 1 -- 172.21.15.102:0/4237184903 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1085386860 0 0) 0x7f32d4003270 con 0x7f32ec12e330 2021-05-25T12:04:37.621 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.600+0000 7f32f0d7e700 1 -- 172.21.15.102:0/4237184903 >> v1:172.21.15.102:6790/0 conn(0x7f32ec090d70 legacy=0x7f32ec091200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:37.621 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.600+0000 7f32f0d7e700 1 -- 172.21.15.102:0/4237184903 >> v1:172.21.15.181:6789/0 conn(0x7f32ec090990 legacy=0x7f32ec093240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-05-25T12:04:37.621 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.600+0000 7f32f0d7e700 10 monclient: _finish_hunting 0 2021-05-25T12:04:37.621 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.600+0000 7f32f0d7e700 1 monclient: found mon.noname-a 2021-05-25T12:04:37.622 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.600+0000 7f32f0d7e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:04:37.622 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.600+0000 7f32f0d7e700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T12:04:37.622 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.600+0000 7f32f0d7e700 1 -- 172.21.15.102:0/4237184903 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32ec092be0 con 0x7f32ec12e330 2021-05-25T12:04:37.622 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.600+0000 7f32f0d7e700 10 monclient: _finish_auth 0 2021-05-25T12:04:37.622 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.600+0000 7f32f0d7e700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:04:37.623 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.600+0000 7f32f3fe4700 5 monclient: authenticate success, global_id 174495 2021-05-25T12:04:37.623 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.600+0000 7f32f0d7e700 1 -- 172.21.15.102:0/4237184903 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f32e8001990 con 0x7f32ec12e330 2021-05-25T12:04:37.623 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.600+0000 7f32f0d7e700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:37.623 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.600+0000 7f32f0d7e700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:04:37.624 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.600+0000 7f32f0d7e700 10 monclient: dump: 2021-05-25T12:04:37.624 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:37.624 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:37.624 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:37.624 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:37.625 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:37.625 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:37.625 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:37.625 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:37.625 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:37.626 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:37.626 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.600+0000 7f32f3fe4700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:04:37.626 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.600+0000 7f32f0d7e700 1 -- 172.21.15.102:0/4237184903 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f32e8001c00 con 0x7f32ec12e330 2021-05-25T12:04:37.626 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.600+0000 7f32f0d7e700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:04:37.627 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.600+0000 7f32f3fe4700 10 monclient: get_monmap_and_config success 2021-05-25T12:04:37.627 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.600+0000 7f32f0d7e700 1 -- 172.21.15.102:0/4237184903 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f32e8001f90 con 0x7f32ec12e330 2021-05-25T12:04:37.627 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.600+0000 7f32f3fe4700 4 set_mon_vals no callback set 2021-05-25T12:04:37.627 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.600+0000 7f32f0d7e700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:37.628 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.600+0000 7f32f0d7e700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:04:37.628 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.600+0000 7f32f0d7e700 10 monclient: dump: 2021-05-25T12:04:37.628 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:37.628 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:37.629 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:37.629 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:37.629 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:37.629 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:37.629 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:37.630 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:37.630 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:37.630 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:37.630 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.600+0000 7f32f3fe4700 10 monclient: shutdown 2021-05-25T12:04:37.631 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.600+0000 7f32f3fe4700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:04:37.631 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.600+0000 7f32f3fe4700 1 -- 172.21.15.102:0/4237184903 >> v1:172.21.15.102:6789/0 conn(0x7f32ec12e330 legacy=0x7f32ec130720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:37.631 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.601+0000 7f32f3fe4700 1 -- 172.21.15.102:0/4237184903 shutdown_connections 2021-05-25T12:04:37.631 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.601+0000 7f32f3fe4700 1 -- 172.21.15.102:0/4237184903 >> 172.21.15.102:0/4237184903 conn(0x7f32ec09b370 msgr2=0x7f32ec097430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:04:37.631 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.601+0000 7f32f3fe4700 1 -- 172.21.15.102:0/4237184903 shutdown_connections 2021-05-25T12:04:37.632 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.601+0000 7f32f3fe4700 1 -- 172.21.15.102:0/4237184903 wait complete. 2021-05-25T12:04:37.632 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.601+0000 7f32f3fe4700 10 monclient: build_initial_monmap 2021-05-25T12:04:37.632 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.601+0000 7f32f3fe4700 10 monclient: monmap: 2021-05-25T12:04:37.632 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:04:37.633 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:04:37.633 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:04:37.633 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:04:37.633 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:04:37.633 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:04:37.634 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:04:37.634 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:04:37.634 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:04:37.634 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:37.634 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.601+0000 7f32f3fe4700 1 librados: starting msgr at 2021-05-25T12:04:37.635 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.601+0000 7f32f3fe4700 1 librados: starting objecter 2021-05-25T12:04:37.635 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.601+0000 7f32f3fe4700 1 Processor -- start 2021-05-25T12:04:37.635 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.601+0000 7f32f3fe4700 1 -- start start 2021-05-25T12:04:37.635 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.601+0000 7f32f3fe4700 1 librados: setting wanted keys 2021-05-25T12:04:37.636 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.601+0000 7f32f3fe4700 1 librados: calling monclient init 2021-05-25T12:04:37.636 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.601+0000 7f32f3fe4700 10 monclient: init 2021-05-25T12:04:37.636 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.602+0000 7f32f3fe4700 10 monclient: _reopen_session rank -1 2021-05-25T12:04:37.636 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.602+0000 7f32f3fe4700 10 monclient: _add_conns ranks=[2,0,1] 2021-05-25T12:04:37.636 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.602+0000 7f32f3fe4700 10 monclient(hunting): picked mon.noname-c con 0x7f32ec090990 addr v1:172.21.15.102:6790/0 2021-05-25T12:04:37.637 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.602+0000 7f32f3fe4700 10 monclient(hunting): picked mon.noname-a con 0x7f32ec090d70 addr v1:172.21.15.102:6789/0 2021-05-25T12:04:37.637 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.602+0000 7f32f3fe4700 10 monclient(hunting): picked mon.noname-b con 0x7f32ec12e330 addr v1:172.21.15.181:6789/0 2021-05-25T12:04:37.637 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.602+0000 7f32f3fe4700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32ec113460 con 0x7f32ec090d70 2021-05-25T12:04:37.637 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.602+0000 7f32f3fe4700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32ec113700 con 0x7f32ec12e330 2021-05-25T12:04:37.638 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.602+0000 7f32f3fe4700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32ec1139a0 con 0x7f32ec090990 2021-05-25T12:04:37.638 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.602+0000 7f32f3fe4700 10 monclient(hunting): _renew_subs 2021-05-25T12:04:37.638 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.602+0000 7f32f3fe4700 10 monclient(hunting): authenticate will time out at 1078.199310s 2021-05-25T12:04:37.638 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.602+0000 7f32f157f700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f32ec090d70 0x7f32ec13b9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60938/0 (socket says 172.21.15.102:60938) 2021-05-25T12:04:37.638 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.602+0000 7f32f157f700 1 -- 172.21.15.102:0/3359587092 learned_addr learned my addr 172.21.15.102:0/3359587092 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:04:37.639 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.602+0000 7f32f2581700 1 -- 172.21.15.102:0/3359587092 >> v1:172.21.15.181:6789/0 conn(0x7f32ec12e330 legacy=0x7f32ec13f100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.181:6789/0 2021-05-25T12:04:37.639 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.602+0000 7f32d9ffb700 10 client.?.objecter ms_handle_connect 0x7f32ec090d70 2021-05-25T12:04:37.639 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.602+0000 7f32d9ffb700 10 client.?.objecter resend_mon_ops 2021-05-25T12:04:37.639 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.602+0000 7f32d9ffb700 10 client.?.objecter ms_handle_connect 0x7f32ec090990 2021-05-25T12:04:37.640 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.602+0000 7f32d9ffb700 10 client.?.objecter resend_mon_ops 2021-05-25T12:04:37.640 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.602+0000 7f32d9ffb700 1 -- 172.21.15.102:0/3359587092 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 224161663 0 0) 0x7f32ec1139a0 con 0x7f32ec090990 2021-05-25T12:04:37.640 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.602+0000 7f32d9ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:04:37.640 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.602+0000 7f32d9ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:04:37.640 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.602+0000 7f32d9ffb700 10 monclient(hunting): my global_id is 184217 2021-05-25T12:04:37.641 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.602+0000 7f32d9ffb700 1 -- 172.21.15.102:0/3359587092 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32cc004c10 con 0x7f32ec090990 2021-05-25T12:04:37.641 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.602+0000 7f32d9ffb700 1 -- 172.21.15.102:0/3359587092 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3927331921 0 0) 0x7f32ec113460 con 0x7f32ec090d70 2021-05-25T12:04:37.641 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.602+0000 7f32d9ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:04:37.641 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.602+0000 7f32d9ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:04:37.641 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.602+0000 7f32d9ffb700 10 monclient(hunting): my global_id is 174498 2021-05-25T12:04:37.642 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.602+0000 7f32d9ffb700 1 -- 172.21.15.102:0/3359587092 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32ec1139a0 con 0x7f32ec090d70 2021-05-25T12:04:37.642 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.602+0000 7f32d9ffb700 1 -- 172.21.15.102:0/3359587092 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3375436044 0 0) 0x7f32cc004c10 con 0x7f32ec090990 2021-05-25T12:04:37.642 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.602+0000 7f32d9ffb700 1 -- 172.21.15.102:0/3359587092 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f32ec113460 con 0x7f32ec090990 2021-05-25T12:04:37.642 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.602+0000 7f32d9ffb700 1 -- 172.21.15.102:0/3359587092 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f32dc003e50 con 0x7f32ec090990 2021-05-25T12:04:37.643 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.602+0000 7f32d9ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:37.643 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.602+0000 7f32d9ffb700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:04:37.643 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.602+0000 7f32d9ffb700 10 monclient(hunting): dump: 2021-05-25T12:04:37.643 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:37.643 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:37.644 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:37.644 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:37.644 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:37.644 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:37.645 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:37.645 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:37.645 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:37.645 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:37.645 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.602+0000 7f32d9ffb700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:04:37.646 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.602+0000 7f32d9ffb700 1 -- 172.21.15.102:0/3359587092 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1639845878 0 0) 0x7f32ec1139a0 con 0x7f32ec090d70 2021-05-25T12:04:37.646 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.602+0000 7f32d9ffb700 1 -- 172.21.15.102:0/3359587092 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f32cc004c10 con 0x7f32ec090d70 2021-05-25T12:04:37.646 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.602+0000 7f32d9ffb700 1 -- 172.21.15.102:0/3359587092 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f32e0001950 con 0x7f32ec090d70 2021-05-25T12:04:37.646 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.602+0000 7f32d9ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:37.646 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.602+0000 7f32d9ffb700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2021-05-25T12:04:37.647 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.602+0000 7f32d9ffb700 10 monclient(hunting): dump: 2021-05-25T12:04:37.647 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:37.647 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:37.647 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:37.647 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:37.648 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:37.648 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:37.648 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:37.648 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:37.648 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:37.649 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:37.649 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.602+0000 7f32d9ffb700 1 -- 172.21.15.102:0/3359587092 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3103583436 0 0) 0x7f32cc004c10 con 0x7f32ec090d70 2021-05-25T12:04:37.649 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.603+0000 7f32d9ffb700 1 -- 172.21.15.102:0/3359587092 >> v1:172.21.15.102:6790/0 conn(0x7f32ec090990 legacy=0x7f32ec128f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:37.649 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.603+0000 7f32d9ffb700 1 -- 172.21.15.102:0/3359587092 >> v1:172.21.15.181:6789/0 conn(0x7f32ec12e330 legacy=0x7f32ec13f100 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-05-25T12:04:37.649 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.603+0000 7f32d9ffb700 10 monclient: _finish_hunting 0 2021-05-25T12:04:37.650 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.603+0000 7f32d9ffb700 1 monclient: found mon.a 2021-05-25T12:04:37.650 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.603+0000 7f32d9ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:04:37.650 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.603+0000 7f32d9ffb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:04:37.650 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.603+0000 7f32d9ffb700 1 -- 172.21.15.102:0/3359587092 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32ec12a080 con 0x7f32ec090d70 2021-05-25T12:04:37.650 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.603+0000 7f32d9ffb700 10 monclient: _finish_auth 0 2021-05-25T12:04:37.651 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.603+0000 7f32d9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:04:37.651 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.603+0000 7f32f3fe4700 5 monclient: authenticate success, global_id 174498 2021-05-25T12:04:37.651 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.603+0000 7f32f3fe4700 10 monclient: _renew_subs 2021-05-25T12:04:37.651 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.603+0000 7f32f3fe4700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:04:37.652 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.603+0000 7f32f3fe4700 1 -- 172.21.15.102:0/3359587092 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32ec137520 con 0x7f32ec090d70 2021-05-25T12:04:37.652 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.603+0000 7f32d9ffb700 1 -- 172.21.15.102:0/3359587092 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f32e0004530 con 0x7f32ec090d70 2021-05-25T12:04:37.652 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.603+0000 7f32d9ffb700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:04:37.652 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.603+0000 7f32d9ffb700 1 -- 172.21.15.102:0/3359587092 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f32e0004840 con 0x7f32ec090d70 2021-05-25T12:04:37.652 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.603+0000 7f32d9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:37.653 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.603+0000 7f32d9ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:04:37.653 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.603+0000 7f32d9ffb700 10 monclient: dump: 2021-05-25T12:04:37.653 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:37.653 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:37.653 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:37.654 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:37.654 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:37.654 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:37.654 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:37.654 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:37.655 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:37.655 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:37.655 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.603+0000 7f32f3fe4700 10 client.174498.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:04:37.655 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.603+0000 7f32f3fe4700 10 monclient: _renew_subs 2021-05-25T12:04:37.655 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.603+0000 7f32dbfff700 4 set_mon_vals no callback set 2021-05-25T12:04:37.656 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.603+0000 7f32f3fe4700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:04:37.656 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.603+0000 7f32f3fe4700 1 -- 172.21.15.102:0/3359587092 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f32ec12ec20 con 0x7f32ec090d70 2021-05-25T12:04:37.656 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.603+0000 7f32f3fe4700 1 librados: init done 2021-05-25T12:04:37.656 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.603+0000 7f32d9ffb700 1 -- 172.21.15.102:0/3359587092 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f32e000db60 con 0x7f32ec090d70 2021-05-25T12:04:37.656 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.603+0000 7f32f3fe4700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:04:37.657 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.603+0000 7f32f3fe4700 1 -- 172.21.15.102:0/3359587092 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f32ec059730 con 0x7f32ec067a90 2021-05-25T12:04:37.657 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.603+0000 7f32d9ffb700 1 -- 172.21.15.102:0/3359587092 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(128..128 src has 1..128) v4 ==== 6071+0+0 (unknown 1273967193 0 0) 0x7f32e003d6d0 con 0x7f32ec090d70 2021-05-25T12:04:37.657 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.603+0000 7f32d9ffb700 10 client.174498.objecter ms_dispatch 0x7f32ec001550 osd_map(128..128 src has 1..128) v4 2021-05-25T12:04:37.657 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.603+0000 7f32d9ffb700 3 client.174498.objecter handle_osd_map got epochs [128,128] > 0 2021-05-25T12:04:37.658 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.603+0000 7f32d9ffb700 3 client.174498.objecter handle_osd_map decoding full epoch 128 2021-05-25T12:04:37.658 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.604+0000 7f32d9ffb700 20 client.174498.objecter dump_active .. 0 homeless 2021-05-25T12:04:37.658 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.604+0000 7f32d9ffb700 10 client.174498.objecter ms_handle_connect 0x7f32ec067a90 2021-05-25T12:04:37.658 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.604+0000 7f32d9ffb700 10 client.174498.objecter resend_mon_ops 2021-05-25T12:04:37.658 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.604+0000 7f32d9ffb700 1 -- 172.21.15.102:0/3359587092 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3712425323 0 0) 0x7f32ec059730 con 0x7f32ec067a90 2021-05-25T12:04:37.659 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.604+0000 7f32d9ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:04:37.659 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.604+0000 7f32d9ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:04:37.659 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.604+0000 7f32d9ffb700 10 monclient(hunting): my global_id is 184220 2021-05-25T12:04:37.659 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.604+0000 7f32d9ffb700 1 -- 172.21.15.102:0/3359587092 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32cc004c10 con 0x7f32ec067a90 2021-05-25T12:04:37.659 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.604+0000 7f32f1d80700 10 monclient: get_auth_request con 0x7f32cc035470 auth_method 0 2021-05-25T12:04:37.660 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.604+0000 7f32d9ffb700 1 -- 172.21.15.102:0/3359587092 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1740119409 0 0) 0x7f32cc004c10 con 0x7f32ec067a90 2021-05-25T12:04:37.660 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.604+0000 7f32d9ffb700 1 -- 172.21.15.102:0/3359587092 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f32ec059730 con 0x7f32ec067a90 2021-05-25T12:04:37.660 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.604+0000 7f32d9ffb700 1 -- 172.21.15.102:0/3359587092 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f32e8001590 con 0x7f32ec067a90 2021-05-25T12:04:37.660 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.604+0000 7f32d9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:37.661 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.604+0000 7f32d9ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:04:37.661 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.604+0000 7f32d9ffb700 10 monclient: dump: 2021-05-25T12:04:37.661 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:37.661 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:37.661 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:37.662 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:37.662 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:37.662 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:37.662 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:37.663 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:37.663 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:37.663 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:37.663 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.604+0000 7f32d9ffb700 1 -- 172.21.15.102:0/3359587092 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4209829730 0 0) 0x7f32ec059730 con 0x7f32ec067a90 2021-05-25T12:04:37.663 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.604+0000 7f32d9ffb700 1 -- 172.21.15.102:0/3359587092 --> v1:172.21.15.102:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f32ec112ce0 con 0x7f32ec067a90 2021-05-25T12:04:37.664 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.604+0000 7f32d9ffb700 10 client.174498.objecter ms_handle_connect 0x7f32cc035470 2021-05-25T12:04:37.664 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.604+0000 7f32d9ffb700 1 -- 172.21.15.102:0/3359587092 <== mon.2 v1:172.21.15.102:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7f32e8004970 con 0x7f32ec067a90 2021-05-25T12:04:37.664 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.604+0000 7f32d9ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:04:37.664 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.604+0000 7f32d9ffb700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:04:37.664 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.604+0000 7f32d9ffb700 1 -- 172.21.15.102:0/3359587092 >> v1:172.21.15.102:6790/0 conn(0x7f32ec067a90 legacy=0x7f32ec063d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:37.665 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.604+0000 7f32d9ffb700 1 -- 172.21.15.102:0/3359587092 >> v1:172.21.15.102:6790/0 conn(0x7f32ec067a90 legacy=0x7f32ec063d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:04:37.665 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.611+0000 7f32f3fe4700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:04:37.665 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.611+0000 7f32f3fe4700 1 -- 172.21.15.102:0/3359587092 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f32ec05d870 con 0x7f32ec027ce0 2021-05-25T12:04:37.665 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.611+0000 7f32d9ffb700 10 client.174498.objecter ms_handle_connect 0x7f32ec027ce0 2021-05-25T12:04:37.665 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.611+0000 7f32d9ffb700 10 client.174498.objecter resend_mon_ops 2021-05-25T12:04:37.666 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.611+0000 7f32d9ffb700 1 -- 172.21.15.102:0/3359587092 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 974401632 0 0) 0x7f32ec05d870 con 0x7f32ec027ce0 2021-05-25T12:04:37.666 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.611+0000 7f32d9ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:04:37.666 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.611+0000 7f32d9ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:04:37.666 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.611+0000 7f32d9ffb700 10 monclient(hunting): my global_id is 184223 2021-05-25T12:04:37.667 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.611+0000 7f32d9ffb700 1 -- 172.21.15.102:0/3359587092 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32ec059730 con 0x7f32ec027ce0 2021-05-25T12:04:37.667 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.611+0000 7f32d9ffb700 1 -- 172.21.15.102:0/3359587092 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2300300572 0 0) 0x7f32ec059730 con 0x7f32ec027ce0 2021-05-25T12:04:37.667 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.611+0000 7f32d9ffb700 1 -- 172.21.15.102:0/3359587092 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f32ec05d870 con 0x7f32ec027ce0 2021-05-25T12:04:37.667 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.611+0000 7f32d9ffb700 1 -- 172.21.15.102:0/3359587092 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f32e8001a50 con 0x7f32ec027ce0 2021-05-25T12:04:37.667 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.611+0000 7f32d9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:04:37.668 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.611+0000 7f32d9ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:04:37.668 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.611+0000 7f32d9ffb700 10 monclient: dump: 2021-05-25T12:04:37.668 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:04:37.668 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:04:37.668 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:37.669 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:04:37.669 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:04:37.669 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:04:37.669 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:04:37.669 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:04:37.670 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:04:37.670 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:04:37.670 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.611+0000 7f32d9ffb700 1 -- 172.21.15.102:0/3359587092 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2715936930 0 0) 0x7f32ec05d870 con 0x7f32ec027ce0 2021-05-25T12:04:37.670 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.611+0000 7f32d9ffb700 1 -- 172.21.15.102:0/3359587092 --> v1:172.21.15.102:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f32ec064b20 con 0x7f32ec027ce0 2021-05-25T12:04:37.671 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.612+0000 7f32d9ffb700 1 -- 172.21.15.102:0/3359587092 <== mon.2 v1:172.21.15.102:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3931 (unknown 0 0 2162226577) 0x7f32e8006350 con 0x7f32ec027ce0 2021-05-25T12:04:37.671 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.612+0000 7f32d9ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:04:37.671 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.612+0000 7f32d9ffb700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:04:37.671 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.612+0000 7f32d9ffb700 1 -- 172.21.15.102:0/3359587092 >> v1:172.21.15.102:6790/0 conn(0x7f32ec027ce0 legacy=0x7f32ec064410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:37.671 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.612+0000 7f32d9ffb700 1 -- 172.21.15.102:0/3359587092 >> v1:172.21.15.102:6790/0 conn(0x7f32ec027ce0 legacy=0x7f32ec064410 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:04:37.672 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:04:37.672 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:04:37.672 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:04:37.673 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "leader", 2021-05-25T12:04:37.673 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 40, 2021-05-25T12:04:37.673 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:04:37.673 INFO:teuthology.orchestra.run.smithi102.stdout: 0, 2021-05-25T12:04:37.673 INFO:teuthology.orchestra.run.smithi102.stdout: 2 2021-05-25T12:04:37.674 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:04:37.674 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 0, 2021-05-25T12:04:37.674 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:04:37.674 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:04:37.675 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:04:37.675 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:04:37.675 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:04:37.675 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:04:37.675 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:04:37.676 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:04:37.676 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:04:37.676 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:04:37.676 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:04:37.677 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:04:37.677 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:04:37.677 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:04:37.677 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:04:37.677 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:04:37.678 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:04:37.678 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:04:37.678 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:04:37.678 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:04:37.678 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:04:37.679 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:04:37.679 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:04:37.679 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:04:37.679 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:04:37.680 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:04:37.680 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:04:37.680 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:04:37.680 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:04:37.681 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:04:37.681 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:04:37.681 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:04:37.681 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:04:37.682 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:04:37.682 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:04:37.682 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:04:37.682 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:04:37.682 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:04:37.683 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:04:37.683 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:04:37.683 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:04:37.683 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:04:37.683 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:04:37.684 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:04:37.684 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:04:37.684 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:04:37.684 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:04:37.684 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:04:37.685 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:04:37.685 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:04:37.685 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:04:37.685 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:04:37.685 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:04:37.686 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:04:37.686 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:04:37.686 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:04:37.686 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:04:37.686 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:04:37.687 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:04:37.687 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:04:37.687 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:04:37.687 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:04:37.687 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:04:37.688 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:04:37.688 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:04:37.688 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:04:37.688 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:04:37.689 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:04:37.689 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:04:37.689 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:04:37.689 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:04:37.689 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:04:37.690 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:04:37.690 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:04:37.690 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:04:37.690 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:04:37.691 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:04:37.691 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:04:37.691 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:04:37.691 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:04:37.691 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:04:37.692 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:04:37.692 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:04:37.692 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:04:37.692 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:04:37.692 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:04:37.693 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:04:37.693 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:04:37.693 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:04:37.693 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:04:37.693 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:04:37.694 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:04:37.694 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:04:37.694 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:04:37.694 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:04:37.694 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:04:37.695 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:04:37.695 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:04:37.695 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:04:37.695 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:04:37.695 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:04:37.696 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:04:37.696 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:04:37.696 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:04:37.696 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:04:37.696 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:04:37.697 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:04:37.697 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:04:37.697 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:04:37.697 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:04:37.697 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:04:37.698 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:04:37.698 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:04:37.698 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:04:37.698 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:04:37.699 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:04:37.699 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:04:37.699 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:04:37.699 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": [ 2021-05-25T12:04:37.699 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:04:37.700 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:04:37.700 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:04:37.700 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 3 2021-05-25T12:04:37.700 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:04:37.700 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:04:37.701 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:04:37.701 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:04:37.701 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:04:37.701 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:04:37.701 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:04:37.702 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:04:37.702 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:04:37.702 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:04:37.702 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:04:37.702 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:04:37.705 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.612+0000 7f32f3fe4700 10 librados: watch_flush enter 2021-05-25T12:04:37.705 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.612+0000 7f32f3fe4700 10 librados: watch_flush exit 2021-05-25T12:04:37.705 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.612+0000 7f32f3fe4700 20 client.174498.objecter shutdown clearing up homeless session... 2021-05-25T12:04:37.706 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.612+0000 7f32f3fe4700 10 client.174498.objecter successfully canceled tick 2021-05-25T12:04:37.706 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.612+0000 7f32f3fe4700 1 -- 172.21.15.102:0/3359587092 >> v1:172.21.15.181:6812/29582 conn(0x7f32cc035470 legacy=0x7f32cc037930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:37.706 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.612+0000 7f32f3fe4700 10 monclient: shutdown 2021-05-25T12:04:37.706 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.612+0000 7f32f3fe4700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:04:37.707 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.612+0000 7f32f3fe4700 1 -- 172.21.15.102:0/3359587092 >> v1:172.21.15.102:6789/0 conn(0x7f32ec090d70 legacy=0x7f32ec13b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:04:37.707 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.612+0000 7f32f2581700 1 -- 172.21.15.102:0/3359587092 reap_dead start 2021-05-25T12:04:37.707 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.612+0000 7f32f2581700 1 -- 172.21.15.102:0/3359587092 reap_dead start 2021-05-25T12:04:37.707 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.612+0000 7f32f3fe4700 1 -- 172.21.15.102:0/3359587092 shutdown_connections 2021-05-25T12:04:37.707 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.612+0000 7f32f3fe4700 1 -- 172.21.15.102:0/3359587092 >> 172.21.15.102:0/3359587092 conn(0x7f32ec09b370 msgr2=0x7f32ec093a60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:04:37.708 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.612+0000 7f32f3fe4700 1 -- 172.21.15.102:0/3359587092 shutdown_connections 2021-05-25T12:04:37.708 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.612+0000 7f32f3fe4700 1 -- 172.21.15.102:0/3359587092 wait complete. 2021-05-25T12:04:37.708 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:04:37.612+0000 7f32f3fe4700 1 librados: shutdown 2021-05-25T12:04:37.709 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2021-05-25T12:04:39.043 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [ RUN ] LibRadosAio api_aio_pp: Running main() from gmock_main.cc 2021-05-25T12:04:39.043 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [==========] Running 53 tests from 4 test suites. 2021-05-25T12:04:39.043 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [----------] Global test environment set-up. 2021-05-25T12:04:39.044 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [----------] 28 tests from LibRadosAio 2021-05-25T12:04:39.044 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ RUN ] LibRadosAio.TooBigPP 2021-05-25T12:04:39.044 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ OK ] LibRadosAio.TooBigPP (17703 ms) 2021-05-25T12:04:39.044 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolQuotaPP 2021-05-25T12:04:39.045 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolQuotaPP (19721 ms) 2021-05-25T12:04:39.045 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleWritePP 2021-05-25T12:04:39.045 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleWritePP (30782 ms) 2021-05-25T12:04:39.045 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ RUN ] LibRadosAio.WaitForSafePP 2021-05-25T12:04:39.045 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ OK ] LibRadosAio.WaitForSafePP (2966 ms) 2021-05-25T12:04:39.046 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP 2021-05-25T12:04:39.046 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP (2986 ms) 2021-05-25T12:04:39.046 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP2 2021-05-25T12:04:39.046 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP2 (3013 ms) 2021-05-25T12:04:39.046 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP3 2021-05-25T12:04:39.047 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP3 (2984 ms) 2021-05-25T12:04:39.047 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripSparseReadPP 2021-05-25T12:04:39.047 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripSparseReadPP (3009 ms) 2021-05-25T12:04:39.047 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsCompletePP 2021-05-25T12:04:39.048 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ OK ] LibRadosAio.IsCompletePP (17179 ms) 2021-05-25T12:04:39.048 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsSafePP 2021-05-25T12:04:39.048 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ OK ] LibRadosAio.IsSafePP (2878 ms) 2021-05-25T12:04:39.048 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ RUN ] LibRadosAio.ReturnValuePP 2021-05-25T12:04:39.048 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ OK ] LibRadosAio.ReturnValuePP (3002 ms) 2021-05-25T12:04:39.049 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushPP 2021-05-25T12:04:39.049 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushPP (3007 ms) 2021-05-25T12:04:39.049 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushAsyncPP 2021-05-25T12:04:39.049 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushAsyncPP (7811 ms) 2021-05-25T12:04:39.049 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP 2021-05-25T12:04:39.050 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP (3008 ms) 2021-05-25T12:04:39.050 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP2 2021-05-25T12:04:39.050 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP2 (3037 ms) 2021-05-25T12:04:39.050 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP 2021-05-25T12:04:39.050 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP (3007 ms) 2021-05-25T12:04:39.051 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP2 2021-05-25T12:04:39.051 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP2 (27354 ms) 2021-05-25T12:04:39.051 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPPNS 2021-05-25T12:04:39.051 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPPNS (3007 ms) 2021-05-25T12:04:39.052 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPP 2021-05-25T12:04:39.052 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPP (3019 ms) 2021-05-25T12:04:39.052 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ RUN ] LibRadosAio.StatRemovePP 2021-05-25T12:04:39.052 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ OK ] LibRadosAio.StatRemovePP (2891 ms) 2021-05-25T12:04:39.052 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ RUN ] LibRadosAio.ExecuteClassPP 2021-05-25T12:04:39.053 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ OK ] LibRadosAio.ExecuteClassPP (3016 ms) 2021-05-25T12:04:39.053 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ RUN ] LibRadosAio.OmapPP 2021-05-25T12:04:39.053 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ OK ] LibRadosAio.OmapPP (3018 ms) 2021-05-25T12:04:39.053 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiWritePP 2021-05-25T12:04:39.053 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiWritePP (2960 ms) 2021-05-25T12:04:39.054 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ RUN ] LibRadosAio.AioUnlockPP 2021-05-25T12:04:39.054 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ OK ] LibRadosAio.AioUnlockPP (33861 ms) 2021-05-25T12:04:39.054 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripAppendPP 2021-05-25T12:04:39.054 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripAppendPP (2998 ms) 2021-05-25T12:04:39.055 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ RUN ] LibRadosAio.RacingRemovePP 2021-05-25T12:04:39.055 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ OK ] LibRadosAio.RacingRemovePP (3017 ms) 2021-05-25T12:04:39.055 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP 2021-05-25T12:04:43.057 INFO:tasks.ceph.osd.3.smithi181.stderr:2021-05-25T12:04:43.053+0000 7f14e787b700 -1 osd.3 pg_epoch: 134 pg[58.4( v 130'88 (0'0,130'88] local-lis/les=30/31 n=7 ec=30/30 lis/c=30/30 les/c/f=31/31/0 sis=30) [3,1] r=0 lpr=30 crt=130'88 lcod 129'86 mlcod 129'86 active+clean+snaptrim trimq=[e~1]] removing snap head 2021-05-25T12:04:57.710 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2021-05-25T12:04:57.710 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2021-05-25T12:04:57.711 INFO:tasks.ceph.mon.b:Restarting daemon 2021-05-25T12:04:57.711 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2021-05-25T12:04:57.714 INFO:tasks.ceph.mon.b:Started 2021-05-25T12:04:57.715 INFO:tasks.mon_thrash:Sending STOP to mon c 2021-05-25T12:04:57.715 INFO:tasks.ceph.mon.c:Sent signal 19 2021-05-25T12:04:57.716 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs to unfreeze mons 2021-05-25T12:05:15.744 INFO:tasks.ceph.osd.0.smithi102.stderr:2021-05-25T12:05:15.741+0000 7f38dd34a700 -1 osd.0 pg_epoch: 153 pg[58.13( v 149'71 (0'0,149'71] local-lis/les=106/107 n=6 ec=30/30 lis/c=106/106 les/c/f=107/109/0 sis=106) [0,2] r=0 lpr=106 crt=149'71 lcod 148'69 mlcod 148'69 active+clean+snaptrim trimq=[11~1]] removing snap head 2021-05-25T12:05:17.716 INFO:tasks.mon_thrash:Sending CONT to mon c 2021-05-25T12:05:17.717 INFO:tasks.ceph.mon.c:Sent signal 18 2021-05-25T12:05:17.717 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2021-05-25T12:05:17.718 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2021-05-25T12:05:17.867 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.845+0000 7f0850795700 10 monclient: get_monmap_and_config 2021-05-25T12:05:17.868 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.845+0000 7f0850795700 10 monclient: build_initial_monmap 2021-05-25T12:05:17.868 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.845+0000 7f0850795700 10 monclient: monmap: 2021-05-25T12:05:17.868 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:05:17.869 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:17.869 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:05:17.846305+0000 2021-05-25T12:05:17.870 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:05:17.846305+0000 2021-05-25T12:05:17.870 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:05:17.870 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:17.871 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:05:17.871 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:05:17.872 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:05:17.872 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:17.873 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.845+0000 7f0850795700 1 Processor -- start 2021-05-25T12:05:17.873 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.845+0000 7f0850795700 1 -- start start 2021-05-25T12:05:17.873 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.845+0000 7f0850795700 10 monclient: init 2021-05-25T12:05:17.874 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.846+0000 7f0850795700 10 monclient: _reopen_session rank -1 2021-05-25T12:05:17.874 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.846+0000 7f0850795700 10 monclient: _add_conns ranks=[0,2,1] 2021-05-25T12:05:17.875 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.846+0000 7f0850795700 10 monclient(hunting): picked mon.noname-a con 0x7f0848090990 addr v1:172.21.15.102:6789/0 2021-05-25T12:05:17.875 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.846+0000 7f0850795700 10 monclient(hunting): picked mon.noname-b con 0x7f0848090d70 addr v1:172.21.15.102:6790/0 2021-05-25T12:05:17.876 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.846+0000 7f0850795700 10 monclient(hunting): picked mon.noname-c con 0x7f084812e330 addr v1:172.21.15.181:6789/0 2021-05-25T12:05:17.876 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.846+0000 7f0850795700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0848096fa0 con 0x7f0848090990 2021-05-25T12:05:17.876 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.846+0000 7f0850795700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08480926a0 con 0x7f084812e330 2021-05-25T12:05:17.877 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.846+0000 7f0850795700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0848092940 con 0x7f0848090d70 2021-05-25T12:05:17.877 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.846+0000 7f0850795700 10 monclient(hunting): _renew_subs 2021-05-25T12:05:17.878 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.846+0000 7f0850795700 10 monclient(hunting): authenticate will time out at 1118.443628s 2021-05-25T12:05:17.878 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.846+0000 7f084dd30700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f0848090d70 0x7f0848091200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:55888/0 (socket says 172.21.15.102:55888) 2021-05-25T12:05:17.879 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.846+0000 7f084e531700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f0848090990 0x7f0848093240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:32780/0 (socket says 172.21.15.102:32780) 2021-05-25T12:05:17.879 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.846+0000 7f084dd30700 1 -- 172.21.15.102:0/920012284 learned_addr learned my addr 172.21.15.102:0/920012284 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:05:17.879 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.847+0000 7f084d52f700 1 -- 172.21.15.102:0/920012284 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3754953685 0 0) 0x7f0848096fa0 con 0x7f0848090990 2021-05-25T12:05:17.880 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.847+0000 7f084d52f700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:17.880 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.847+0000 7f084d52f700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:17.881 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.847+0000 7f084d52f700 10 monclient(hunting): my global_id is 174561 2021-05-25T12:05:17.881 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.847+0000 7f084d52f700 1 -- 172.21.15.102:0/920012284 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0830003270 con 0x7f0848090990 2021-05-25T12:05:17.882 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.847+0000 7f084d52f700 1 -- 172.21.15.102:0/920012284 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3953280351 0 0) 0x7f0830003270 con 0x7f0848090990 2021-05-25T12:05:17.882 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.847+0000 7f084d52f700 1 -- 172.21.15.102:0/920012284 >> v1:172.21.15.102:6790/0 conn(0x7f0848090d70 legacy=0x7f0848091200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:17.882 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.847+0000 7f084d52f700 1 -- 172.21.15.102:0/920012284 >> v1:172.21.15.181:6789/0 conn(0x7f084812e330 legacy=0x7f0848130720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:17.883 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.847+0000 7f084d52f700 10 monclient: _finish_hunting 0 2021-05-25T12:05:17.883 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.847+0000 7f084d52f700 1 monclient: found mon.noname-a 2021-05-25T12:05:17.884 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.847+0000 7f084d52f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:05:17.884 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.847+0000 7f084d52f700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T12:05:17.885 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.847+0000 7f084d52f700 1 -- 172.21.15.102:0/920012284 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0848092be0 con 0x7f0848090990 2021-05-25T12:05:17.885 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.847+0000 7f084d52f700 10 monclient: _finish_auth 0 2021-05-25T12:05:17.885 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.847+0000 7f084d52f700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:05:17.886 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.847+0000 7f0850795700 5 monclient: authenticate success, global_id 174561 2021-05-25T12:05:17.886 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.847+0000 7f084d52f700 1 -- 172.21.15.102:0/920012284 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f0838001610 con 0x7f0848090990 2021-05-25T12:05:17.887 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.847+0000 7f084d52f700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:17.887 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.847+0000 7f084d52f700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:05:17.888 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.847+0000 7f084d52f700 10 monclient: dump: 2021-05-25T12:05:17.888 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:17.888 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:17.889 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:17.889 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:17.890 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:17.890 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:17.891 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:17.891 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:17.891 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:17.892 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:17.892 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.847+0000 7f0850795700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:05:17.893 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.847+0000 7f084d52f700 1 -- 172.21.15.102:0/920012284 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f08380018c0 con 0x7f0848090990 2021-05-25T12:05:17.893 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.847+0000 7f084d52f700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:05:17.893 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.847+0000 7f084d52f700 1 -- 172.21.15.102:0/920012284 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f0838001c90 con 0x7f0848090990 2021-05-25T12:05:17.894 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.847+0000 7f0850795700 10 monclient: get_monmap_and_config success 2021-05-25T12:05:17.894 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.847+0000 7f0850795700 4 set_mon_vals no callback set 2021-05-25T12:05:17.895 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.847+0000 7f084d52f700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:17.895 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.847+0000 7f084d52f700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:05:17.896 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.847+0000 7f084d52f700 10 monclient: dump: 2021-05-25T12:05:17.896 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:17.896 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:17.897 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:17.897 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:17.898 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:17.898 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:17.899 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:17.899 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:17.899 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:17.900 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:17.900 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.847+0000 7f0850795700 10 monclient: shutdown 2021-05-25T12:05:17.901 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.847+0000 7f0850795700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:05:17.901 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.847+0000 7f0850795700 1 -- 172.21.15.102:0/920012284 >> v1:172.21.15.102:6789/0 conn(0x7f0848090990 legacy=0x7f0848093240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:17.902 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.848+0000 7f0850795700 1 -- 172.21.15.102:0/920012284 shutdown_connections 2021-05-25T12:05:17.902 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.848+0000 7f0850795700 1 -- 172.21.15.102:0/920012284 >> 172.21.15.102:0/920012284 conn(0x7f084809b370 msgr2=0x7f0848097430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:05:17.902 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.848+0000 7f0850795700 1 -- 172.21.15.102:0/920012284 shutdown_connections 2021-05-25T12:05:17.903 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.848+0000 7f0850795700 1 -- 172.21.15.102:0/920012284 wait complete. 2021-05-25T12:05:17.903 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.848+0000 7f0850795700 10 monclient: build_initial_monmap 2021-05-25T12:05:17.904 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.848+0000 7f0850795700 10 monclient: monmap: 2021-05-25T12:05:17.904 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:05:17.905 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:05:17.905 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:05:17.905 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:05:17.906 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:05:17.906 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:05:17.907 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:05:17.907 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:05:17.907 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:05:17.908 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:17.908 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.848+0000 7f0850795700 1 librados: starting msgr at 2021-05-25T12:05:17.908 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.848+0000 7f0850795700 1 librados: starting objecter 2021-05-25T12:05:17.908 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.848+0000 7f0850795700 1 Processor -- start 2021-05-25T12:05:17.909 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.849+0000 7f0850795700 1 -- start start 2021-05-25T12:05:17.909 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.849+0000 7f0850795700 1 librados: setting wanted keys 2021-05-25T12:05:17.909 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.849+0000 7f0850795700 1 librados: calling monclient init 2021-05-25T12:05:17.909 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.849+0000 7f0850795700 10 monclient: init 2021-05-25T12:05:17.909 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.849+0000 7f0850795700 10 monclient: _reopen_session rank -1 2021-05-25T12:05:17.910 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.849+0000 7f0850795700 10 monclient: _add_conns ranks=[0,1,2] 2021-05-25T12:05:17.910 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.849+0000 7f0850795700 10 monclient(hunting): picked mon.noname-a con 0x7f0848090990 addr v1:172.21.15.102:6789/0 2021-05-25T12:05:17.910 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.849+0000 7f0850795700 10 monclient(hunting): picked mon.noname-b con 0x7f0848090d70 addr v1:172.21.15.181:6789/0 2021-05-25T12:05:17.910 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.849+0000 7f0850795700 10 monclient(hunting): picked mon.noname-c con 0x7f084812e330 addr v1:172.21.15.102:6790/0 2021-05-25T12:05:17.910 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.849+0000 7f0850795700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0848091fe0 con 0x7f0848090990 2021-05-25T12:05:17.911 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.849+0000 7f0850795700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08480664e0 con 0x7f0848090d70 2021-05-25T12:05:17.911 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.849+0000 7f0850795700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0848066780 con 0x7f084812e330 2021-05-25T12:05:17.911 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.849+0000 7f0850795700 10 monclient(hunting): _renew_subs 2021-05-25T12:05:17.911 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.849+0000 7f0850795700 10 monclient(hunting): authenticate will time out at 1118.446521s 2021-05-25T12:05:17.912 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.849+0000 7f084e531700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f0848090990 0x7f084813ab20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:32784/0 (socket says 172.21.15.102:32784) 2021-05-25T12:05:17.912 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.849+0000 7f084e531700 1 -- 172.21.15.102:0/2117356951 learned_addr learned my addr 172.21.15.102:0/2117356951 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:05:17.912 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.849+0000 7f08367fc700 10 client.?.objecter ms_handle_connect 0x7f0848090990 2021-05-25T12:05:17.912 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.849+0000 7f08367fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:05:17.912 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.849+0000 7f08367fc700 10 client.?.objecter ms_handle_connect 0x7f084812e330 2021-05-25T12:05:17.913 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.849+0000 7f08367fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:05:17.913 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.849+0000 7f08367fc700 10 client.?.objecter ms_handle_connect 0x7f0848090d70 2021-05-25T12:05:17.913 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.849+0000 7f08367fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:05:17.913 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.849+0000 7f08367fc700 1 -- 172.21.15.102:0/2117356951 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 979201428 0 0) 0x7f0848091fe0 con 0x7f0848090990 2021-05-25T12:05:17.914 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.849+0000 7f08367fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:17.914 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.849+0000 7f08367fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:17.914 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.849+0000 7f08367fc700 10 monclient(hunting): my global_id is 174564 2021-05-25T12:05:17.914 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.849+0000 7f08367fc700 1 -- 172.21.15.102:0/2117356951 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0828004c10 con 0x7f0848090990 2021-05-25T12:05:17.914 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.849+0000 7f08367fc700 1 -- 172.21.15.102:0/2117356951 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3470712325 0 0) 0x7f08480664e0 con 0x7f0848090d70 2021-05-25T12:05:17.915 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.849+0000 7f08367fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:17.915 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.849+0000 7f08367fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:17.915 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.849+0000 7f08367fc700 10 monclient(hunting): my global_id is 194101 2021-05-25T12:05:17.915 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.849+0000 7f08367fc700 1 -- 172.21.15.102:0/2117356951 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0848091fe0 con 0x7f0848090d70 2021-05-25T12:05:17.915 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.850+0000 7f08367fc700 1 -- 172.21.15.102:0/2117356951 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4113240788 0 0) 0x7f0828004c10 con 0x7f0848090990 2021-05-25T12:05:17.916 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.850+0000 7f08367fc700 1 -- 172.21.15.102:0/2117356951 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f08480664e0 con 0x7f0848090990 2021-05-25T12:05:17.916 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.850+0000 7f08367fc700 1 -- 172.21.15.102:0/2117356951 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f0838001cd0 con 0x7f0848090990 2021-05-25T12:05:17.916 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.850+0000 7f08367fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:17.916 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.850+0000 7f08367fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:05:17.917 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.850+0000 7f08367fc700 10 monclient(hunting): dump: 2021-05-25T12:05:17.917 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:17.917 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:17.917 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:17.917 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:17.918 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:17.918 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:17.918 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:17.918 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:17.918 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:17.919 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:17.919 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.850+0000 7f08367fc700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:05:17.919 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.850+0000 7f08367fc700 1 -- 172.21.15.102:0/2117356951 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2601164091 0 0) 0x7f0848091fe0 con 0x7f0848090d70 2021-05-25T12:05:17.919 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.850+0000 7f08367fc700 1 -- 172.21.15.102:0/2117356951 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0828004c10 con 0x7f0848090d70 2021-05-25T12:05:17.920 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.850+0000 7f08367fc700 1 -- 172.21.15.102:0/2117356951 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f083c0017f0 con 0x7f0848090d70 2021-05-25T12:05:17.920 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.850+0000 7f08367fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:17.920 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.850+0000 7f08367fc700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2021-05-25T12:05:17.920 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.850+0000 7f08367fc700 10 monclient(hunting): dump: 2021-05-25T12:05:17.920 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:17.921 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:17.921 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:17.921 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:17.921 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:17.921 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:17.922 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:17.922 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:17.922 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:17.922 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:17.923 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.850+0000 7f08367fc700 1 -- 172.21.15.102:0/2117356951 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1095762821 0 0) 0x7f08480664e0 con 0x7f0848090990 2021-05-25T12:05:17.923 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.850+0000 7f08367fc700 1 -- 172.21.15.102:0/2117356951 >> v1:172.21.15.102:6790/0 conn(0x7f084812e330 legacy=0x7f0848142eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:17.923 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.850+0000 7f08367fc700 1 -- 172.21.15.102:0/2117356951 >> v1:172.21.15.181:6789/0 conn(0x7f0848090d70 legacy=0x7f0848126490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:17.923 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.850+0000 7f08367fc700 10 monclient: _finish_hunting 0 2021-05-25T12:05:17.923 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.850+0000 7f08367fc700 1 monclient: found mon.a 2021-05-25T12:05:17.924 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.850+0000 7f08367fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:05:17.924 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.850+0000 7f08367fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:05:17.924 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.850+0000 7f08367fc700 1 -- 172.21.15.102:0/2117356951 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0848068930 con 0x7f0848090990 2021-05-25T12:05:17.924 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.850+0000 7f08367fc700 10 monclient: _finish_auth 0 2021-05-25T12:05:17.924 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.850+0000 7f08367fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:05:17.925 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.850+0000 7f0850795700 5 monclient: authenticate success, global_id 174564 2021-05-25T12:05:17.925 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.850+0000 7f0850795700 10 monclient: _renew_subs 2021-05-25T12:05:17.925 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.850+0000 7f0850795700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:05:17.925 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.850+0000 7f0850795700 1 -- 172.21.15.102:0/2117356951 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f084813b230 con 0x7f0848090990 2021-05-25T12:05:17.926 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.850+0000 7f0850795700 10 client.174564.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:05:17.926 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.850+0000 7f0850795700 10 monclient: _renew_subs 2021-05-25T12:05:17.926 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.850+0000 7f0850795700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:05:17.926 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.850+0000 7f0850795700 1 -- 172.21.15.102:0/2117356951 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0848126d60 con 0x7f0848090990 2021-05-25T12:05:17.926 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.850+0000 7f0850795700 1 librados: init done 2021-05-25T12:05:17.927 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.850+0000 7f08367fc700 1 -- 172.21.15.102:0/2117356951 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f08380049b0 con 0x7f0848090990 2021-05-25T12:05:17.927 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.850+0000 7f08367fc700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:05:17.927 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.850+0000 7f08367fc700 1 -- 172.21.15.102:0/2117356951 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f0838004d40 con 0x7f0848090990 2021-05-25T12:05:17.927 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.850+0000 7f08367fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:17.927 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.850+0000 7f084cd2e700 4 set_mon_vals no callback set 2021-05-25T12:05:17.928 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.850+0000 7f08367fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:05:17.928 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.850+0000 7f08367fc700 10 monclient: dump: 2021-05-25T12:05:17.928 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:17.928 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:17.929 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:17.929 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:17.929 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:17.929 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:17.929 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:17.930 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:17.930 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:17.930 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:17.930 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.851+0000 7f08367fc700 1 -- 172.21.15.102:0/2117356951 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f0838004ea0 con 0x7f0848090990 2021-05-25T12:05:17.930 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.851+0000 7f0850795700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:05:17.931 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.851+0000 7f0850795700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:05:17.931 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.851+0000 7f0850795700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:05:17.931 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.851+0000 7f08367fc700 1 -- 172.21.15.102:0/2117356951 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(155..155 src has 1..155) v4 ==== 6444+0+0 (unknown 721441627 0 0) 0x7f083803db90 con 0x7f0848090990 2021-05-25T12:05:17.931 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.851+0000 7f08367fc700 10 client.174564.objecter ms_dispatch 0x7f0848001550 osd_map(155..155 src has 1..155) v4 2021-05-25T12:05:17.932 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.851+0000 7f08367fc700 3 client.174564.objecter handle_osd_map got epochs [155,155] > 0 2021-05-25T12:05:17.932 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.851+0000 7f08367fc700 3 client.174564.objecter handle_osd_map decoding full epoch 155 2021-05-25T12:05:17.932 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.851+0000 7f0850795700 1 -- 172.21.15.102:0/2117356951 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f082c004db0 con 0x7f0848090990 2021-05-25T12:05:17.932 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.851+0000 7f08367fc700 20 client.174564.objecter dump_active .. 0 homeless 2021-05-25T12:05:17.932 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.851+0000 7f084dd30700 10 monclient: get_auth_request con 0x7f08280332f0 auth_method 0 2021-05-25T12:05:17.933 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.851+0000 7f08367fc700 10 client.174564.objecter ms_handle_connect 0x7f08280332f0 2021-05-25T12:05:17.933 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.855+0000 7f08367fc700 1 -- 172.21.15.102:0/2117356951 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7f08380064c0 con 0x7f0848090990 2021-05-25T12:05:17.933 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.855+0000 7f08367fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:05:17.933 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:17.855+0000 7f08367fc700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:05:18.061 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:18.059+0000 7f0850795700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2021-05-25T12:05:18.061 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:18.059+0000 7f0850795700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2021-05-25T12:05:18.062 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:18.059+0000 7f0850795700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:05:18.062 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:18.059+0000 7f0850795700 1 -- 172.21.15.102:0/2117356951 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f082c005060 con 0x7f0848090990 2021-05-25T12:05:18.062 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:18.060+0000 7f08367fc700 1 -- 172.21.15.102:0/2117356951 <== mon.0 v1:172.21.15.102:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1336 (unknown 254746657 0 2030886462) 0x7f0838012e00 con 0x7f0848090990 2021-05-25T12:05:18.062 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:18.060+0000 7f08367fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2021-05-25T12:05:18.063 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:18.060+0000 7f08367fc700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:05:18.063 INFO:teuthology.orchestra.run.smithi102.stdout:{"election_epoch":42,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540138303579357183","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"20ee91b6-8935-4b98-b868-85c03789ccc4","modified":"2021-05-25T11:59:11.011522Z","created":"2021-05-25T11:59:11.011522Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6790","nonce":0}]},"addr":"172.21.15.102:6790/0","public_addr":"172.21.15.102:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2021-05-25T12:05:18.064 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:18.061+0000 7f0850795700 10 librados: watch_flush enter 2021-05-25T12:05:18.065 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:18.061+0000 7f0850795700 10 librados: watch_flush exit 2021-05-25T12:05:18.065 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:18.061+0000 7f0850795700 20 client.174564.objecter shutdown clearing up homeless session... 2021-05-25T12:05:18.065 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:18.061+0000 7f0850795700 10 client.174564.objecter successfully canceled tick 2021-05-25T12:05:18.065 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:18.061+0000 7f0850795700 1 -- 172.21.15.102:0/2117356951 >> v1:172.21.15.181:6812/29582 conn(0x7f08280332f0 legacy=0x7f08280357b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:18.066 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:18.061+0000 7f0850795700 10 monclient: shutdown 2021-05-25T12:05:18.066 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:18.061+0000 7f0850795700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:05:18.066 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:18.061+0000 7f0850795700 1 -- 172.21.15.102:0/2117356951 >> v1:172.21.15.102:6789/0 conn(0x7f0848090990 legacy=0x7f084813ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:18.066 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:18.061+0000 7f0850795700 1 -- 172.21.15.102:0/2117356951 shutdown_connections 2021-05-25T12:05:18.067 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:18.061+0000 7f0850795700 1 -- 172.21.15.102:0/2117356951 >> 172.21.15.102:0/2117356951 conn(0x7f084809b370 msgr2=0x7f08481292f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:05:18.067 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:18.061+0000 7f0850795700 1 -- 172.21.15.102:0/2117356951 shutdown_connections 2021-05-25T12:05:18.067 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:18.061+0000 7f0850795700 1 -- 172.21.15.102:0/2117356951 wait complete. 2021-05-25T12:05:18.067 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:18.061+0000 7f0850795700 1 librados: shutdown 2021-05-25T12:05:21.075 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2021-05-25T12:05:21.192 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.190+0000 7f4dd7ae7700 10 monclient: get_monmap_and_config 2021-05-25T12:05:21.192 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.190+0000 7f4dd7ae7700 10 monclient: build_initial_monmap 2021-05-25T12:05:21.192 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.190+0000 7f4dd7ae7700 10 monclient: monmap: 2021-05-25T12:05:21.193 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:05:21.193 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:21.193 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:05:21.191427+0000 2021-05-25T12:05:21.193 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:05:21.191427+0000 2021-05-25T12:05:21.193 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:05:21.194 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:21.194 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:05:21.194 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:05:21.194 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:05:21.195 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:21.195 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.191+0000 7f4dd7ae7700 1 Processor -- start 2021-05-25T12:05:21.195 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.191+0000 7f4dd7ae7700 1 -- start start 2021-05-25T12:05:21.195 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.191+0000 7f4dd7ae7700 10 monclient: init 2021-05-25T12:05:21.205 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.191+0000 7f4dd7ae7700 10 monclient: _reopen_session rank -1 2021-05-25T12:05:21.206 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.191+0000 7f4dd7ae7700 10 monclient: _add_conns ranks=[0,2,1] 2021-05-25T12:05:21.206 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.191+0000 7f4dd7ae7700 10 monclient(hunting): picked mon.noname-a con 0x7f4dd0090990 addr v1:172.21.15.102:6789/0 2021-05-25T12:05:21.206 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.191+0000 7f4dd7ae7700 10 monclient(hunting): picked mon.noname-b con 0x7f4dd0090d70 addr v1:172.21.15.102:6790/0 2021-05-25T12:05:21.207 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.191+0000 7f4dd7ae7700 10 monclient(hunting): picked mon.noname-c con 0x7f4dd012e330 addr v1:172.21.15.181:6789/0 2021-05-25T12:05:21.207 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.191+0000 7f4dd7ae7700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dd0096fa0 con 0x7f4dd0090990 2021-05-25T12:05:21.207 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.191+0000 7f4dd7ae7700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dd00926a0 con 0x7f4dd012e330 2021-05-25T12:05:21.207 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.191+0000 7f4dd7ae7700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dd0092940 con 0x7f4dd0090d70 2021-05-25T12:05:21.208 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.191+0000 7f4dd7ae7700 10 monclient(hunting): _renew_subs 2021-05-25T12:05:21.208 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.191+0000 7f4dd7ae7700 10 monclient(hunting): authenticate will time out at 1121.788823s 2021-05-25T12:05:21.208 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.191+0000 7f4dd5082700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f4dd0090d70 0x7f4dd0091200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:55902/0 (socket says 172.21.15.102:55902) 2021-05-25T12:05:21.208 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.191+0000 7f4dd5883700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f4dd0090990 0x7f4dd0093240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:32792/0 (socket says 172.21.15.102:32792) 2021-05-25T12:05:21.208 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.191+0000 7f4dd5082700 1 -- 172.21.15.102:0/1556796593 learned_addr learned my addr 172.21.15.102:0/1556796593 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:05:21.209 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.192+0000 7f4dd4881700 1 -- 172.21.15.102:0/1556796593 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 406733435 0 0) 0x7f4dd0096fa0 con 0x7f4dd0090990 2021-05-25T12:05:21.209 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.192+0000 7f4dd4881700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:21.209 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.192+0000 7f4dd4881700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:21.209 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.192+0000 7f4dd4881700 10 monclient(hunting): my global_id is 174567 2021-05-25T12:05:21.210 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.192+0000 7f4dd4881700 1 -- 172.21.15.102:0/1556796593 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4db8004380 con 0x7f4dd0090990 2021-05-25T12:05:21.210 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.192+0000 7f4dd4881700 1 -- 172.21.15.102:0/1556796593 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4290047570 0 0) 0x7f4dd00926a0 con 0x7f4dd012e330 2021-05-25T12:05:21.210 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.192+0000 7f4dd4881700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:21.210 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.192+0000 7f4dd4881700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:21.211 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.192+0000 7f4dd4881700 10 monclient(hunting): my global_id is 194104 2021-05-25T12:05:21.211 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.192+0000 7f4dd4881700 1 -- 172.21.15.102:0/1556796593 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4dd0096fa0 con 0x7f4dd012e330 2021-05-25T12:05:21.211 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.192+0000 7f4dd4881700 1 -- 172.21.15.102:0/1556796593 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3730501774 0 0) 0x7f4db8004380 con 0x7f4dd0090990 2021-05-25T12:05:21.211 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.192+0000 7f4dd4881700 1 -- 172.21.15.102:0/1556796593 >> v1:172.21.15.102:6790/0 conn(0x7f4dd0090d70 legacy=0x7f4dd0091200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:21.211 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.192+0000 7f4dd4881700 1 -- 172.21.15.102:0/1556796593 >> v1:172.21.15.181:6789/0 conn(0x7f4dd012e330 legacy=0x7f4dd0130720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:21.212 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.192+0000 7f4dd4881700 10 monclient: _finish_hunting 0 2021-05-25T12:05:21.212 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.192+0000 7f4dd4881700 1 monclient: found mon.noname-a 2021-05-25T12:05:21.212 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.192+0000 7f4dd4881700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:05:21.212 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.192+0000 7f4dd4881700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T12:05:21.213 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.192+0000 7f4dd4881700 1 -- 172.21.15.102:0/1556796593 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dd0092be0 con 0x7f4dd0090990 2021-05-25T12:05:21.213 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.192+0000 7f4dd4881700 10 monclient: _finish_auth 0 2021-05-25T12:05:21.213 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.192+0000 7f4dd4881700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:05:21.214 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.192+0000 7f4dd7ae7700 5 monclient: authenticate success, global_id 174567 2021-05-25T12:05:21.214 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.192+0000 7f4dd4881700 1 -- 172.21.15.102:0/1556796593 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f4dc4001590 con 0x7f4dd0090990 2021-05-25T12:05:21.214 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.192+0000 7f4dd4881700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:21.214 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.192+0000 7f4dd4881700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:05:21.214 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.192+0000 7f4dd4881700 10 monclient: dump: 2021-05-25T12:05:21.215 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:21.215 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:21.215 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:21.215 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:21.216 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:21.216 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:21.216 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:21.216 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:21.217 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:21.217 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:21.217 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.192+0000 7f4dd7ae7700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:05:21.217 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.192+0000 7f4dd4881700 1 -- 172.21.15.102:0/1556796593 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4dc40017b0 con 0x7f4dd0090990 2021-05-25T12:05:21.217 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.192+0000 7f4dd4881700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:05:21.218 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.192+0000 7f4dd7ae7700 10 monclient: get_monmap_and_config success 2021-05-25T12:05:21.218 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.192+0000 7f4dd4881700 1 -- 172.21.15.102:0/1556796593 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f4dc4001ac0 con 0x7f4dd0090990 2021-05-25T12:05:21.218 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.192+0000 7f4dd4881700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:21.218 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.192+0000 7f4dd7ae7700 4 set_mon_vals no callback set 2021-05-25T12:05:21.219 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.192+0000 7f4dd4881700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:05:21.219 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.192+0000 7f4dd4881700 10 monclient: dump: 2021-05-25T12:05:21.219 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:21.219 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:21.219 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:21.220 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:21.220 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:21.220 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:21.220 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:21.220 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:21.221 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:21.221 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:21.221 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.193+0000 7f4dd7ae7700 10 monclient: shutdown 2021-05-25T12:05:21.221 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.193+0000 7f4dd7ae7700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:05:21.221 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.193+0000 7f4dd7ae7700 1 -- 172.21.15.102:0/1556796593 >> v1:172.21.15.102:6789/0 conn(0x7f4dd0090990 legacy=0x7f4dd0093240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:21.222 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.193+0000 7f4dd7ae7700 1 -- 172.21.15.102:0/1556796593 shutdown_connections 2021-05-25T12:05:21.222 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.193+0000 7f4dd7ae7700 1 -- 172.21.15.102:0/1556796593 >> 172.21.15.102:0/1556796593 conn(0x7f4dd009b370 msgr2=0x7f4dd0097430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:05:21.222 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.193+0000 7f4dd7ae7700 1 -- 172.21.15.102:0/1556796593 shutdown_connections 2021-05-25T12:05:21.223 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.193+0000 7f4dd7ae7700 1 -- 172.21.15.102:0/1556796593 wait complete. 2021-05-25T12:05:21.223 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.193+0000 7f4dd7ae7700 10 monclient: build_initial_monmap 2021-05-25T12:05:21.223 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.193+0000 7f4dd7ae7700 10 monclient: monmap: 2021-05-25T12:05:21.223 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:05:21.223 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:05:21.224 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:05:21.224 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:05:21.224 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:05:21.224 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:05:21.225 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:05:21.225 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:05:21.225 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:05:21.225 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:21.225 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.193+0000 7f4dd7ae7700 1 librados: starting msgr at 2021-05-25T12:05:21.226 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.193+0000 7f4dd7ae7700 1 librados: starting objecter 2021-05-25T12:05:21.226 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.193+0000 7f4dd7ae7700 1 Processor -- start 2021-05-25T12:05:21.226 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.193+0000 7f4dd7ae7700 1 -- start start 2021-05-25T12:05:21.226 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.193+0000 7f4dd7ae7700 1 librados: setting wanted keys 2021-05-25T12:05:21.226 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.193+0000 7f4dd7ae7700 1 librados: calling monclient init 2021-05-25T12:05:21.227 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.193+0000 7f4dd7ae7700 10 monclient: init 2021-05-25T12:05:21.227 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.194+0000 7f4dd7ae7700 10 monclient: _reopen_session rank -1 2021-05-25T12:05:21.227 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.194+0000 7f4dd7ae7700 10 monclient: _add_conns ranks=[2,0,1] 2021-05-25T12:05:21.227 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.194+0000 7f4dd7ae7700 10 monclient(hunting): picked mon.noname-c con 0x7f4dd0090990 addr v1:172.21.15.102:6790/0 2021-05-25T12:05:21.228 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.194+0000 7f4dd7ae7700 10 monclient(hunting): picked mon.noname-a con 0x7f4dd0090d70 addr v1:172.21.15.102:6789/0 2021-05-25T12:05:21.228 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.194+0000 7f4dd7ae7700 10 monclient(hunting): picked mon.noname-b con 0x7f4dd012e330 addr v1:172.21.15.181:6789/0 2021-05-25T12:05:21.228 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.194+0000 7f4dd7ae7700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dd0067fa0 con 0x7f4dd0090d70 2021-05-25T12:05:21.228 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.194+0000 7f4dd7ae7700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dd011a2a0 con 0x7f4dd012e330 2021-05-25T12:05:21.228 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.194+0000 7f4dd7ae7700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dd011a540 con 0x7f4dd0090990 2021-05-25T12:05:21.229 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.194+0000 7f4dd7ae7700 10 monclient(hunting): _renew_subs 2021-05-25T12:05:21.229 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.194+0000 7f4dd7ae7700 10 monclient(hunting): authenticate will time out at 1121.791402s 2021-05-25T12:05:21.229 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.194+0000 7f4dd5883700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f4dd0090990 0x7f4dd012baf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:55908/0 (socket says 172.21.15.102:55908) 2021-05-25T12:05:21.229 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.194+0000 7f4dd5883700 1 -- 172.21.15.102:0/3580758270 learned_addr learned my addr 172.21.15.102:0/3580758270 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:05:21.229 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.194+0000 7f4dd5082700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f4dd0090d70 0x7f4dd0140170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:32800/0 (socket says 172.21.15.102:32800) 2021-05-25T12:05:21.230 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.194+0000 7f4dbdffb700 10 client.?.objecter ms_handle_connect 0x7f4dd0090990 2021-05-25T12:05:21.230 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.194+0000 7f4dbdffb700 10 client.?.objecter resend_mon_ops 2021-05-25T12:05:21.230 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.194+0000 7f4dbdffb700 10 client.?.objecter ms_handle_connect 0x7f4dd0090d70 2021-05-25T12:05:21.230 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.194+0000 7f4dbdffb700 10 client.?.objecter resend_mon_ops 2021-05-25T12:05:21.231 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.194+0000 7f4dbdffb700 1 -- 172.21.15.102:0/3580758270 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1208433736 0 0) 0x7f4dd0067fa0 con 0x7f4dd0090d70 2021-05-25T12:05:21.231 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.194+0000 7f4dbdffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:21.231 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.194+0000 7f4dbdffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:21.231 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.194+0000 7f4dbdffb700 10 monclient(hunting): my global_id is 174570 2021-05-25T12:05:21.232 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.194+0000 7f4dbdffb700 1 -- 172.21.15.102:0/3580758270 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4db0004c10 con 0x7f4dd0090d70 2021-05-25T12:05:21.232 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.194+0000 7f4dbdffb700 10 client.?.objecter ms_handle_connect 0x7f4dd012e330 2021-05-25T12:05:21.232 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.194+0000 7f4dbdffb700 10 client.?.objecter resend_mon_ops 2021-05-25T12:05:21.232 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.194+0000 7f4dbdffb700 1 -- 172.21.15.102:0/3580758270 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3477886325 0 0) 0x7f4db0004c10 con 0x7f4dd0090d70 2021-05-25T12:05:21.233 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.194+0000 7f4dbdffb700 1 -- 172.21.15.102:0/3580758270 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4dd0067fa0 con 0x7f4dd0090d70 2021-05-25T12:05:21.233 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.194+0000 7f4dbdffb700 1 -- 172.21.15.102:0/3580758270 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f4dc00017f0 con 0x7f4dd0090d70 2021-05-25T12:05:21.233 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.194+0000 7f4dbdffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:21.233 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.194+0000 7f4dbdffb700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:05:21.233 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.194+0000 7f4dbdffb700 10 monclient(hunting): dump: 2021-05-25T12:05:21.234 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:21.234 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:21.234 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:21.234 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:21.235 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:21.235 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:21.235 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:21.235 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:21.235 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:21.236 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:21.236 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.194+0000 7f4dbdffb700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:05:21.236 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.195+0000 7f4dbdffb700 1 -- 172.21.15.102:0/3580758270 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1271436228 0 0) 0x7f4dd011a2a0 con 0x7f4dd012e330 2021-05-25T12:05:21.236 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.195+0000 7f4dbdffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:21.236 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.195+0000 7f4dbdffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:21.237 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.195+0000 7f4dbdffb700 10 monclient(hunting): my global_id is 194107 2021-05-25T12:05:21.237 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.195+0000 7f4dbdffb700 1 -- 172.21.15.102:0/3580758270 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4db0004c10 con 0x7f4dd012e330 2021-05-25T12:05:21.237 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.195+0000 7f4dbdffb700 1 -- 172.21.15.102:0/3580758270 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 532294265 0 0) 0x7f4dd0067fa0 con 0x7f4dd0090d70 2021-05-25T12:05:21.237 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.195+0000 7f4dbdffb700 1 -- 172.21.15.102:0/3580758270 >> v1:172.21.15.102:6790/0 conn(0x7f4dd0090990 legacy=0x7f4dd012baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:21.238 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.195+0000 7f4dbdffb700 1 -- 172.21.15.102:0/3580758270 >> v1:172.21.15.181:6789/0 conn(0x7f4dd012e330 legacy=0x7f4dd01438a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:21.238 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.195+0000 7f4dbdffb700 10 monclient: _finish_hunting 0 2021-05-25T12:05:21.238 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.195+0000 7f4dbdffb700 1 monclient: found mon.a 2021-05-25T12:05:21.238 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.195+0000 7f4dbdffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:05:21.238 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.195+0000 7f4dbdffb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:05:21.239 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.195+0000 7f4dbdffb700 1 -- 172.21.15.102:0/3580758270 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dd01190f0 con 0x7f4dd0090d70 2021-05-25T12:05:21.239 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.195+0000 7f4dbdffb700 10 monclient: _finish_auth 0 2021-05-25T12:05:21.239 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.195+0000 7f4dbdffb700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:05:21.239 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.195+0000 7f4dd7ae7700 5 monclient: authenticate success, global_id 174570 2021-05-25T12:05:21.239 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.195+0000 7f4dd7ae7700 10 monclient: _renew_subs 2021-05-25T12:05:21.240 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.195+0000 7f4dd7ae7700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:05:21.240 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.195+0000 7f4dd7ae7700 1 -- 172.21.15.102:0/3580758270 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dd012c200 con 0x7f4dd0090d70 2021-05-25T12:05:21.240 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.195+0000 7f4dbdffb700 1 -- 172.21.15.102:0/3580758270 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4dc0004530 con 0x7f4dd0090d70 2021-05-25T12:05:21.240 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.195+0000 7f4dbdffb700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:05:21.240 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.195+0000 7f4dbdffb700 1 -- 172.21.15.102:0/3580758270 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f4dc0004840 con 0x7f4dd0090d70 2021-05-25T12:05:21.241 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.195+0000 7f4dbdffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:21.241 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.195+0000 7f4dbdffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:05:21.241 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.195+0000 7f4dbdffb700 10 monclient: dump: 2021-05-25T12:05:21.241 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:21.242 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:21.242 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:21.242 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:21.242 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:21.242 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:21.243 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:21.243 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:21.243 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:21.243 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:21.244 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.195+0000 7f4dd7ae7700 10 client.174570.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:05:21.244 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.195+0000 7f4dbffff700 4 set_mon_vals no callback set 2021-05-25T12:05:21.244 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.195+0000 7f4dd7ae7700 10 monclient: _renew_subs 2021-05-25T12:05:21.244 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.195+0000 7f4dd7ae7700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:05:21.245 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.195+0000 7f4dd7ae7700 1 -- 172.21.15.102:0/3580758270 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dd013afa0 con 0x7f4dd0090d70 2021-05-25T12:05:21.245 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.195+0000 7f4dd7ae7700 1 librados: init done 2021-05-25T12:05:21.245 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.196+0000 7f4dbdffb700 1 -- 172.21.15.102:0/3580758270 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f4dc0004a20 con 0x7f4dd0090d70 2021-05-25T12:05:21.245 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.196+0000 7f4dd7ae7700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:05:21.245 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.196+0000 7f4dd7ae7700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:05:21.246 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.196+0000 7f4dd7ae7700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:05:21.246 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.196+0000 7f4dd7ae7700 1 -- 172.21.15.102:0/3580758270 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dd0059730 con 0x7f4dd0090d70 2021-05-25T12:05:21.246 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.196+0000 7f4dbdffb700 1 -- 172.21.15.102:0/3580758270 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(158..158 src has 1..158) v4 ==== 6444+0+0 (unknown 1381816456 0 0) 0x7f4dc003da90 con 0x7f4dd0090d70 2021-05-25T12:05:21.246 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.196+0000 7f4dbdffb700 10 client.174570.objecter ms_dispatch 0x7f4dd0001550 osd_map(158..158 src has 1..158) v4 2021-05-25T12:05:21.246 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.196+0000 7f4dbdffb700 3 client.174570.objecter handle_osd_map got epochs [158,158] > 0 2021-05-25T12:05:21.247 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.196+0000 7f4dbdffb700 3 client.174570.objecter handle_osd_map decoding full epoch 158 2021-05-25T12:05:21.247 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.196+0000 7f4dbdffb700 20 client.174570.objecter dump_active .. 0 homeless 2021-05-25T12:05:21.247 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.196+0000 7f4dd5883700 10 monclient: get_auth_request con 0x7f4db00331e0 auth_method 0 2021-05-25T12:05:21.247 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.196+0000 7f4dbdffb700 10 client.174570.objecter ms_handle_connect 0x7f4db00331e0 2021-05-25T12:05:21.247 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.200+0000 7f4dbdffb700 1 -- 172.21.15.102:0/3580758270 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7f4dc0005220 con 0x7f4dd0090d70 2021-05-25T12:05:21.248 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.200+0000 7f4dbdffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:05:21.248 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.200+0000 7f4dbdffb700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:05:21.409 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.407+0000 7f4dd7ae7700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2021-05-25T12:05:21.409 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.407+0000 7f4dd7ae7700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2021-05-25T12:05:21.409 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.407+0000 7f4dd7ae7700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:05:21.410 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.407+0000 7f4dd7ae7700 1 -- 172.21.15.102:0/3580758270 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f4dd0096500 con 0x7f4dd0090d70 2021-05-25T12:05:21.410 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.408+0000 7f4dbdffb700 1 -- 172.21.15.102:0/3580758270 <== mon.0 v1:172.21.15.102:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1336 (unknown 254746657 0 3428697453) 0x7f4dc0012930 con 0x7f4dd0090d70 2021-05-25T12:05:21.410 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.408+0000 7f4dbdffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2021-05-25T12:05:21.411 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.408+0000 7f4dbdffb700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:05:21.411 INFO:teuthology.orchestra.run.smithi102.stdout:{"election_epoch":42,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540138303579357183","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"20ee91b6-8935-4b98-b868-85c03789ccc4","modified":"2021-05-25T11:59:11.011522Z","created":"2021-05-25T11:59:11.011522Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6790","nonce":0}]},"addr":"172.21.15.102:6790/0","public_addr":"172.21.15.102:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2021-05-25T12:05:21.413 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.409+0000 7f4dd7ae7700 10 librados: watch_flush enter 2021-05-25T12:05:21.413 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.409+0000 7f4dd7ae7700 10 librados: watch_flush exit 2021-05-25T12:05:21.413 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.409+0000 7f4dd7ae7700 20 client.174570.objecter shutdown clearing up homeless session... 2021-05-25T12:05:21.413 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.409+0000 7f4dd7ae7700 10 client.174570.objecter successfully canceled tick 2021-05-25T12:05:21.414 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.409+0000 7f4dd7ae7700 1 -- 172.21.15.102:0/3580758270 >> v1:172.21.15.181:6812/29582 conn(0x7f4db00331e0 legacy=0x7f4db00356a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:21.414 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.409+0000 7f4dd7ae7700 10 monclient: shutdown 2021-05-25T12:05:21.414 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.409+0000 7f4dd7ae7700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:05:21.414 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.409+0000 7f4dd7ae7700 1 -- 172.21.15.102:0/3580758270 >> v1:172.21.15.102:6789/0 conn(0x7f4dd0090d70 legacy=0x7f4dd0140170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:21.415 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.409+0000 7f4dd7ae7700 1 -- 172.21.15.102:0/3580758270 shutdown_connections 2021-05-25T12:05:21.415 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.409+0000 7f4dd7ae7700 1 -- 172.21.15.102:0/3580758270 >> 172.21.15.102:0/3580758270 conn(0x7f4dd009b370 msgr2=0x7f4dd009a040 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:05:21.415 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.409+0000 7f4dd7ae7700 1 -- 172.21.15.102:0/3580758270 shutdown_connections 2021-05-25T12:05:21.415 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.409+0000 7f4dd7ae7700 1 -- 172.21.15.102:0/3580758270 wait complete. 2021-05-25T12:05:21.415 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:21.409+0000 7f4dd7ae7700 1 librados: shutdown 2021-05-25T12:05:24.422 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2021-05-25T12:05:24.545 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.543+0000 7f7d3cefa700 10 monclient: get_monmap_and_config 2021-05-25T12:05:24.545 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.543+0000 7f7d3cefa700 10 monclient: build_initial_monmap 2021-05-25T12:05:24.545 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.543+0000 7f7d3cefa700 10 monclient: monmap: 2021-05-25T12:05:24.546 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:05:24.546 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:24.546 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:05:24.544394+0000 2021-05-25T12:05:24.546 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:05:24.544394+0000 2021-05-25T12:05:24.547 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:05:24.547 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:24.547 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:05:24.547 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:05:24.548 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:05:24.548 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:24.548 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.543+0000 7f7d3cefa700 1 Processor -- start 2021-05-25T12:05:24.548 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.544+0000 7f7d3cefa700 1 -- start start 2021-05-25T12:05:24.548 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.544+0000 7f7d3cefa700 10 monclient: init 2021-05-25T12:05:24.558 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.544+0000 7f7d3cefa700 10 monclient: _reopen_session rank -1 2021-05-25T12:05:24.558 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.544+0000 7f7d3cefa700 10 monclient: _add_conns ranks=[2,1,0] 2021-05-25T12:05:24.558 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.544+0000 7f7d3cefa700 10 monclient(hunting): picked mon.noname-b con 0x7f7d38093fc0 addr v1:172.21.15.102:6790/0 2021-05-25T12:05:24.559 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.544+0000 7f7d3cefa700 10 monclient(hunting): picked mon.noname-c con 0x7f7d380692d0 addr v1:172.21.15.181:6789/0 2021-05-25T12:05:24.559 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.544+0000 7f7d3cefa700 10 monclient(hunting): picked mon.noname-a con 0x7f7d3812f130 addr v1:172.21.15.102:6789/0 2021-05-25T12:05:24.559 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.544+0000 7f7d3cefa700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d38097160 con 0x7f7d3812f130 2021-05-25T12:05:24.559 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.544+0000 7f7d3cefa700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d38092fe0 con 0x7f7d380692d0 2021-05-25T12:05:24.559 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.544+0000 7f7d3cefa700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d38093280 con 0x7f7d38093fc0 2021-05-25T12:05:24.560 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.544+0000 7f7d3cefa700 10 monclient(hunting): _renew_subs 2021-05-25T12:05:24.560 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.544+0000 7f7d3cefa700 10 monclient(hunting): authenticate will time out at 1125.141719s 2021-05-25T12:05:24.560 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.544+0000 7f7d36d9d700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f7d3812f130 0x7f7d38131520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:32810/0 (socket says 172.21.15.102:32810) 2021-05-25T12:05:24.560 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.544+0000 7f7d36d9d700 1 -- 172.21.15.102:0/257773962 learned_addr learned my addr 172.21.15.102:0/257773962 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:05:24.561 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.545+0000 7f7d3559a700 1 -- 172.21.15.102:0/257773962 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1866334453 0 0) 0x7f7d38092fe0 con 0x7f7d380692d0 2021-05-25T12:05:24.561 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.545+0000 7f7d3559a700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:24.561 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.545+0000 7f7d3559a700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:24.561 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.545+0000 7f7d3559a700 10 monclient(hunting): my global_id is 194110 2021-05-25T12:05:24.561 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.545+0000 7f7d3559a700 1 -- 172.21.15.102:0/257773962 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d1c003270 con 0x7f7d380692d0 2021-05-25T12:05:24.562 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.545+0000 7f7d3559a700 1 -- 172.21.15.102:0/257773962 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2444392928 0 0) 0x7f7d1c003270 con 0x7f7d380692d0 2021-05-25T12:05:24.562 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.545+0000 7f7d3559a700 1 -- 172.21.15.102:0/257773962 >> v1:172.21.15.102:6790/0 conn(0x7f7d38093fc0 legacy=0x7f7d38096340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:24.562 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.545+0000 7f7d3559a700 1 -- 172.21.15.102:0/257773962 >> v1:172.21.15.102:6789/0 conn(0x7f7d3812f130 legacy=0x7f7d38131520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:24.562 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.545+0000 7f7d3559a700 10 monclient: _finish_hunting 0 2021-05-25T12:05:24.562 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.545+0000 7f7d3559a700 1 monclient: found mon.noname-c 2021-05-25T12:05:24.563 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.545+0000 7f7d3559a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:05:24.563 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.545+0000 7f7d3559a700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.181:6789/0 2021-05-25T12:05:24.563 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.545+0000 7f7d3559a700 1 -- 172.21.15.102:0/257773962 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d38093520 con 0x7f7d380692d0 2021-05-25T12:05:24.563 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.545+0000 7f7d3559a700 10 monclient: _finish_auth 0 2021-05-25T12:05:24.563 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.545+0000 7f7d3559a700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:05:24.564 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.545+0000 7f7d3559a700 1 -- 172.21.15.102:0/257773962 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f7d20001610 con 0x7f7d380692d0 2021-05-25T12:05:24.564 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.545+0000 7f7d3cefa700 5 monclient: authenticate success, global_id 194110 2021-05-25T12:05:24.564 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.545+0000 7f7d3cefa700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:05:24.564 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.545+0000 7f7d3559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:24.564 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.545+0000 7f7d3559a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:05:24.565 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.545+0000 7f7d3559a700 10 monclient: dump: 2021-05-25T12:05:24.565 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:24.565 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:24.565 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:24.565 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:24.566 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:24.566 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:24.566 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:24.566 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:24.566 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:24.567 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:24.567 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.545+0000 7f7d3cefa700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:05:24.567 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.545+0000 7f7d3559a700 1 -- 172.21.15.102:0/257773962 <== mon.1 v1:172.21.15.181:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7d200018c0 con 0x7f7d380692d0 2021-05-25T12:05:24.567 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.545+0000 7f7d3559a700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:05:24.567 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.545+0000 7f7d3cefa700 10 monclient: get_monmap_and_config success 2021-05-25T12:05:24.568 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.545+0000 7f7d3cefa700 4 set_mon_vals no callback set 2021-05-25T12:05:24.568 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.545+0000 7f7d3559a700 1 -- 172.21.15.102:0/257773962 <== mon.1 v1:172.21.15.181:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f7d20001c90 con 0x7f7d380692d0 2021-05-25T12:05:24.568 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.545+0000 7f7d3559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:24.568 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.545+0000 7f7d3559a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T12:05:24.568 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.545+0000 7f7d3559a700 10 monclient: dump: 2021-05-25T12:05:24.569 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:24.569 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:24.569 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:24.569 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:24.569 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:24.570 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:24.570 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:24.570 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:24.570 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:24.570 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:24.571 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.546+0000 7f7d3cefa700 10 monclient: shutdown 2021-05-25T12:05:24.571 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.546+0000 7f7d3cefa700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:05:24.571 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.546+0000 7f7d3cefa700 1 -- 172.21.15.102:0/257773962 >> v1:172.21.15.181:6789/0 conn(0x7f7d380692d0 legacy=0x7f7d38096a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:24.571 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.546+0000 7f7d3cefa700 1 -- 172.21.15.102:0/257773962 shutdown_connections 2021-05-25T12:05:24.571 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.546+0000 7f7d3cefa700 1 -- 172.21.15.102:0/257773962 >> 172.21.15.102:0/257773962 conn(0x7f7d3809b370 msgr2=0x7f7d38097430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:05:24.572 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.546+0000 7f7d3cefa700 1 -- 172.21.15.102:0/257773962 shutdown_connections 2021-05-25T12:05:24.572 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.546+0000 7f7d3cefa700 1 -- 172.21.15.102:0/257773962 wait complete. 2021-05-25T12:05:24.572 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.546+0000 7f7d3cefa700 10 monclient: build_initial_monmap 2021-05-25T12:05:24.572 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.546+0000 7f7d3cefa700 10 monclient: monmap: 2021-05-25T12:05:24.572 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:05:24.573 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:05:24.573 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:05:24.573 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:05:24.573 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:05:24.574 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:05:24.574 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:05:24.574 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:05:24.574 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:05:24.575 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:24.575 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.546+0000 7f7d3cefa700 1 librados: starting msgr at 2021-05-25T12:05:24.575 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.546+0000 7f7d3cefa700 1 librados: starting objecter 2021-05-25T12:05:24.575 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.547+0000 7f7d3cefa700 1 Processor -- start 2021-05-25T12:05:24.575 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.547+0000 7f7d3cefa700 1 -- start start 2021-05-25T12:05:24.576 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.547+0000 7f7d3cefa700 1 librados: setting wanted keys 2021-05-25T12:05:24.576 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.547+0000 7f7d3cefa700 1 librados: calling monclient init 2021-05-25T12:05:24.576 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.547+0000 7f7d3cefa700 10 monclient: init 2021-05-25T12:05:24.576 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.547+0000 7f7d3cefa700 10 monclient: _reopen_session rank -1 2021-05-25T12:05:24.576 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.547+0000 7f7d3cefa700 10 monclient: _add_conns ranks=[2,0,1] 2021-05-25T12:05:24.577 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.547+0000 7f7d3cefa700 10 monclient(hunting): picked mon.noname-c con 0x7f7d380692d0 addr v1:172.21.15.102:6790/0 2021-05-25T12:05:24.577 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.547+0000 7f7d3cefa700 10 monclient(hunting): picked mon.noname-a con 0x7f7d38093fc0 addr v1:172.21.15.102:6789/0 2021-05-25T12:05:24.577 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.547+0000 7f7d3cefa700 10 monclient(hunting): picked mon.noname-b con 0x7f7d3812f130 addr v1:172.21.15.181:6789/0 2021-05-25T12:05:24.577 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.547+0000 7f7d3cefa700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d38092350 con 0x7f7d38093fc0 2021-05-25T12:05:24.577 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.547+0000 7f7d3cefa700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d38068090 con 0x7f7d3812f130 2021-05-25T12:05:24.578 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.547+0000 7f7d3cefa700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d38140430 con 0x7f7d380692d0 2021-05-25T12:05:24.578 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.547+0000 7f7d3cefa700 10 monclient(hunting): _renew_subs 2021-05-25T12:05:24.578 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.547+0000 7f7d3cefa700 10 monclient(hunting): authenticate will time out at 1125.144522s 2021-05-25T12:05:24.578 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.547+0000 7f7d3659c700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f7d380692d0 0x7f7d3813f610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:55922/0 (socket says 172.21.15.102:55922) 2021-05-25T12:05:24.579 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.547+0000 7f7d3659c700 1 -- 172.21.15.102:0/2264407759 learned_addr learned my addr 172.21.15.102:0/2264407759 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:05:24.579 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.547+0000 7f7d35d9b700 1 --1- 172.21.15.102:0/2264407759 >> v1:172.21.15.102:6789/0 conn(0x7f7d38093fc0 0x7f7d3813fd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:32814/0 (socket says 172.21.15.102:32814) 2021-05-25T12:05:24.579 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.547+0000 7f7d267fc700 10 client.?.objecter ms_handle_connect 0x7f7d380692d0 2021-05-25T12:05:24.579 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.547+0000 7f7d267fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:05:24.579 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.547+0000 7f7d267fc700 10 client.?.objecter ms_handle_connect 0x7f7d38093fc0 2021-05-25T12:05:24.580 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.547+0000 7f7d267fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:05:24.580 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.547+0000 7f7d267fc700 10 client.?.objecter ms_handle_connect 0x7f7d3812f130 2021-05-25T12:05:24.580 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.547+0000 7f7d267fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:05:24.580 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.548+0000 7f7d267fc700 1 -- 172.21.15.102:0/2264407759 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2311635140 0 0) 0x7f7d38068090 con 0x7f7d3812f130 2021-05-25T12:05:24.581 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.548+0000 7f7d267fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:24.581 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.548+0000 7f7d267fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:24.581 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.548+0000 7f7d267fc700 10 monclient(hunting): my global_id is 194113 2021-05-25T12:05:24.581 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.548+0000 7f7d267fc700 1 -- 172.21.15.102:0/2264407759 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d14004c10 con 0x7f7d3812f130 2021-05-25T12:05:24.581 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.548+0000 7f7d267fc700 1 -- 172.21.15.102:0/2264407759 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3340422077 0 0) 0x7f7d14004c10 con 0x7f7d3812f130 2021-05-25T12:05:24.582 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.548+0000 7f7d267fc700 1 -- 172.21.15.102:0/2264407759 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7d38068090 con 0x7f7d3812f130 2021-05-25T12:05:24.582 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.548+0000 7f7d267fc700 1 -- 172.21.15.102:0/2264407759 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f7d28001af0 con 0x7f7d3812f130 2021-05-25T12:05:24.582 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.548+0000 7f7d267fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:24.582 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.548+0000 7f7d267fc700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:05:24.583 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.548+0000 7f7d267fc700 10 monclient(hunting): dump: 2021-05-25T12:05:24.583 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:24.583 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:24.583 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:24.583 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:24.584 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:24.584 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:24.584 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:24.584 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:24.585 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:24.585 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:24.585 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.548+0000 7f7d267fc700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:05:24.585 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.548+0000 7f7d267fc700 1 -- 172.21.15.102:0/2264407759 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3426742930 0 0) 0x7f7d38068090 con 0x7f7d3812f130 2021-05-25T12:05:24.585 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.548+0000 7f7d267fc700 1 -- 172.21.15.102:0/2264407759 >> v1:172.21.15.102:6790/0 conn(0x7f7d380692d0 legacy=0x7f7d3813f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:24.586 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.548+0000 7f7d267fc700 1 -- 172.21.15.102:0/2264407759 >> v1:172.21.15.102:6789/0 conn(0x7f7d38093fc0 legacy=0x7f7d3813fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:24.586 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.548+0000 7f7d267fc700 10 monclient: _finish_hunting 0 2021-05-25T12:05:24.586 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.548+0000 7f7d267fc700 1 monclient: found mon.b 2021-05-25T12:05:24.586 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.548+0000 7f7d267fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:05:24.587 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.548+0000 7f7d267fc700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T12:05:24.587 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.548+0000 7f7d267fc700 1 -- 172.21.15.102:0/2264407759 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d38111a60 con 0x7f7d3812f130 2021-05-25T12:05:24.587 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.548+0000 7f7d267fc700 10 monclient: _finish_auth 0 2021-05-25T12:05:24.587 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.548+0000 7f7d267fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:05:24.587 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.548+0000 7f7d3cefa700 5 monclient: authenticate success, global_id 194113 2021-05-25T12:05:24.588 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.548+0000 7f7d3cefa700 10 monclient: _renew_subs 2021-05-25T12:05:24.588 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.548+0000 7f7d3cefa700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T12:05:24.588 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.548+0000 7f7d3cefa700 1 -- 172.21.15.102:0/2264407759 --> v1:172.21.15.181:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d38066830 con 0x7f7d3812f130 2021-05-25T12:05:24.588 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.548+0000 7f7d3cefa700 10 client.194113.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:05:24.588 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.548+0000 7f7d3cefa700 10 monclient: _renew_subs 2021-05-25T12:05:24.588 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.548+0000 7f7d3cefa700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T12:05:24.589 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.548+0000 7f7d3cefa700 1 -- 172.21.15.102:0/2264407759 --> v1:172.21.15.181:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d38067c20 con 0x7f7d3812f130 2021-05-25T12:05:24.589 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.548+0000 7f7d3cefa700 1 librados: init done 2021-05-25T12:05:24.589 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.548+0000 7f7d267fc700 1 -- 172.21.15.102:0/2264407759 <== mon.1 v1:172.21.15.181:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7d280048b0 con 0x7f7d3812f130 2021-05-25T12:05:24.589 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.548+0000 7f7d267fc700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:05:24.589 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.548+0000 7f7d267fc700 1 -- 172.21.15.102:0/2264407759 <== mon.1 v1:172.21.15.181:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f7d28004bc0 con 0x7f7d3812f130 2021-05-25T12:05:24.590 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.548+0000 7f7d267fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:24.590 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.548+0000 7f7d267fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T12:05:24.590 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.548+0000 7f7d267fc700 10 monclient: dump: 2021-05-25T12:05:24.590 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:24.591 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:24.591 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:24.591 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:24.591 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:24.591 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:24.592 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:24.592 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:24.592 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:24.592 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:24.592 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.548+0000 7f7d34d99700 4 set_mon_vals no callback set 2021-05-25T12:05:24.593 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.549+0000 7f7d267fc700 1 -- 172.21.15.102:0/2264407759 <== mon.1 v1:172.21.15.181:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f7d28004de0 con 0x7f7d3812f130 2021-05-25T12:05:24.593 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.549+0000 7f7d3cefa700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:05:24.593 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.549+0000 7f7d3cefa700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:05:24.593 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.549+0000 7f7d3cefa700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T12:05:24.593 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.549+0000 7f7d3cefa700 1 -- 172.21.15.102:0/2264407759 --> v1:172.21.15.181:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d38059730 con 0x7f7d3812f130 2021-05-25T12:05:24.594 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.549+0000 7f7d267fc700 1 -- 172.21.15.102:0/2264407759 <== mon.1 v1:172.21.15.181:6789/0 8 ==== osd_map(160..160 src has 1..160) v4 ==== 6444+0+0 (unknown 3124456911 0 0) 0x7f7d2803dc80 con 0x7f7d3812f130 2021-05-25T12:05:24.594 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.549+0000 7f7d267fc700 10 client.194113.objecter ms_dispatch 0x7f7d38001550 osd_map(160..160 src has 1..160) v4 2021-05-25T12:05:24.594 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.549+0000 7f7d267fc700 3 client.194113.objecter handle_osd_map got epochs [160,160] > 0 2021-05-25T12:05:24.594 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.549+0000 7f7d267fc700 3 client.194113.objecter handle_osd_map decoding full epoch 160 2021-05-25T12:05:24.594 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.549+0000 7f7d267fc700 20 client.194113.objecter dump_active .. 0 homeless 2021-05-25T12:05:24.595 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.549+0000 7f7d3659c700 10 monclient: get_auth_request con 0x7f7d14033160 auth_method 0 2021-05-25T12:05:24.595 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.550+0000 7f7d267fc700 10 client.194113.objecter ms_handle_connect 0x7f7d14033160 2021-05-25T12:05:24.595 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.554+0000 7f7d267fc700 1 -- 172.21.15.102:0/2264407759 <== mon.1 v1:172.21.15.181:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7f7d28005590 con 0x7f7d3812f130 2021-05-25T12:05:24.595 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.554+0000 7f7d267fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:05:24.595 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.554+0000 7f7d267fc700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:05:24.764 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.763+0000 7f7d3cefa700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2021-05-25T12:05:24.765 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.763+0000 7f7d3cefa700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2021-05-25T12:05:24.765 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.763+0000 7f7d3cefa700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T12:05:24.765 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.763+0000 7f7d3cefa700 1 -- 172.21.15.102:0/2264407759 --> v1:172.21.15.181:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f7d3805d870 con 0x7f7d3812f130 2021-05-25T12:05:24.766 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.764+0000 7f7d267fc700 1 -- 172.21.15.102:0/2264407759 <== mon.1 v1:172.21.15.181:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1337 (unknown 254746657 0 595150392) 0x7f7d28012da0 con 0x7f7d3812f130 2021-05-25T12:05:24.766 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.764+0000 7f7d267fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2021-05-25T12:05:24.766 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.764+0000 7f7d267fc700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:05:24.767 INFO:teuthology.orchestra.run.smithi102.stdout:{"election_epoch":42,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540138303579357183","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"20ee91b6-8935-4b98-b868-85c03789ccc4","modified":"2021-05-25T11:59:11.011522Z","created":"2021-05-25T11:59:11.011522Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6790","nonce":0}]},"addr":"172.21.15.102:6790/0","public_addr":"172.21.15.102:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2021-05-25T12:05:24.768 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.765+0000 7f7d3cefa700 10 librados: watch_flush enter 2021-05-25T12:05:24.769 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.765+0000 7f7d3cefa700 10 librados: watch_flush exit 2021-05-25T12:05:24.769 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.765+0000 7f7d3cefa700 20 client.194113.objecter shutdown clearing up homeless session... 2021-05-25T12:05:24.769 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.765+0000 7f7d3cefa700 10 client.194113.objecter successfully canceled tick 2021-05-25T12:05:24.769 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.765+0000 7f7d3cefa700 1 -- 172.21.15.102:0/2264407759 >> v1:172.21.15.181:6812/29582 conn(0x7f7d14033160 legacy=0x7f7d14035620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:24.770 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.765+0000 7f7d3cefa700 10 monclient: shutdown 2021-05-25T12:05:24.770 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.765+0000 7f7d3cefa700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:05:24.770 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.765+0000 7f7d3cefa700 1 -- 172.21.15.102:0/2264407759 >> v1:172.21.15.181:6789/0 conn(0x7f7d3812f130 legacy=0x7f7d38147a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:24.770 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.765+0000 7f7d3cefa700 1 -- 172.21.15.102:0/2264407759 shutdown_connections 2021-05-25T12:05:24.771 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.765+0000 7f7d3cefa700 1 -- 172.21.15.102:0/2264407759 >> 172.21.15.102:0/2264407759 conn(0x7f7d3809b370 msgr2=0x7f7d3809a020 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:05:24.771 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.765+0000 7f7d3cefa700 1 -- 172.21.15.102:0/2264407759 shutdown_connections 2021-05-25T12:05:24.771 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.765+0000 7f7d3cefa700 1 -- 172.21.15.102:0/2264407759 wait complete. 2021-05-25T12:05:24.771 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:24.765+0000 7f7d3cefa700 1 librados: shutdown 2021-05-25T12:05:27.719 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:05:27.718+0000 7f5d1054b700 -1 mon.c@2(electing) e1 get_health_metrics reporting 2 slow ops, oldest is mon_command({"prefix": "osd pool create", "pool": "test-rados-api-smithi181-32842-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi181-32842-30"} v 0) 2021-05-25T12:05:27.778 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2021-05-25T12:05:27.901 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:27.900+0000 7fd4448a2700 10 monclient: get_monmap_and_config 2021-05-25T12:05:27.902 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:27.900+0000 7fd4448a2700 10 monclient: build_initial_monmap 2021-05-25T12:05:27.902 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:27.900+0000 7fd4448a2700 10 monclient: monmap: 2021-05-25T12:05:27.902 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:05:27.903 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:27.903 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:05:27.901180+0000 2021-05-25T12:05:27.903 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:05:27.901180+0000 2021-05-25T12:05:27.903 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:05:27.904 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:27.904 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:05:27.904 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:05:27.904 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:05:27.904 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:27.905 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:27.900+0000 7fd4448a2700 1 Processor -- start 2021-05-25T12:05:27.905 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:27.900+0000 7fd4448a2700 1 -- start start 2021-05-25T12:05:27.905 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:27.900+0000 7fd4448a2700 10 monclient: init 2021-05-25T12:05:27.906 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:27.901+0000 7fd4448a2700 10 monclient: _reopen_session rank -1 2021-05-25T12:05:27.907 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:27.901+0000 7fd4448a2700 10 monclient: _add_conns ranks=[2,0,1] 2021-05-25T12:05:27.907 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:27.901+0000 7fd4448a2700 10 monclient(hunting): picked mon.noname-b con 0x7fd440093d90 addr v1:172.21.15.102:6790/0 2021-05-25T12:05:27.907 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:27.901+0000 7fd4448a2700 10 monclient(hunting): picked mon.noname-a con 0x7fd440132900 addr v1:172.21.15.102:6789/0 2021-05-25T12:05:27.907 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:27.901+0000 7fd4448a2700 10 monclient(hunting): picked mon.noname-c con 0x7fd440136410 addr v1:172.21.15.181:6789/0 2021-05-25T12:05:27.908 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:27.901+0000 7fd4448a2700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd440091ed0 con 0x7fd440132900 2021-05-25T12:05:27.908 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:27.901+0000 7fd4448a2700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd440092170 con 0x7fd440136410 2021-05-25T12:05:27.908 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:27.901+0000 7fd4448a2700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd440092410 con 0x7fd440093d90 2021-05-25T12:05:27.908 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:27.901+0000 7fd4448a2700 10 monclient(hunting): _renew_subs 2021-05-25T12:05:27.909 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:27.901+0000 7fd4448a2700 10 monclient(hunting): authenticate will time out at 1128.498521s 2021-05-25T12:05:27.909 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:27.901+0000 7fd43dd9b700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fd440132900 0x7fd440134cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:32822/0 (socket says 172.21.15.102:32822) 2021-05-25T12:05:27.909 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:27.901+0000 7fd43dd9b700 1 -- 172.21.15.102:0/3304217414 learned_addr learned my addr 172.21.15.102:0/3304217414 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:05:27.909 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:27.901+0000 7fd43e59c700 1 --1- 172.21.15.102:0/3304217414 >> v1:172.21.15.102:6790/0 conn(0x7fd440093d90 0x7fd4400691c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:55930/0 (socket says 172.21.15.102:55930) 2021-05-25T12:05:28.760 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.751+0000 7fd43d59a700 1 -- 172.21.15.102:0/3304217414 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2351693222 0 0) 0x7fd440091ed0 con 0x7fd440132900 2021-05-25T12:05:28.761 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.751+0000 7fd43d59a700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:28.761 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.751+0000 7fd43d59a700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:28.761 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.751+0000 7fd43d59a700 10 monclient(hunting): my global_id is 174573 2021-05-25T12:05:28.762 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.751+0000 7fd43d59a700 1 -- 172.21.15.102:0/3304217414 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd424003270 con 0x7fd440132900 2021-05-25T12:05:28.762 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.752+0000 7fd43d59a700 1 -- 172.21.15.102:0/3304217414 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3680853305 0 0) 0x7fd424003270 con 0x7fd440132900 2021-05-25T12:05:28.762 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.752+0000 7fd43d59a700 1 -- 172.21.15.102:0/3304217414 >> v1:172.21.15.102:6790/0 conn(0x7fd440093d90 legacy=0x7fd4400691c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:28.762 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.752+0000 7fd43d59a700 1 -- 172.21.15.102:0/3304217414 >> v1:172.21.15.181:6789/0 conn(0x7fd440136410 legacy=0x7fd440138800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:28.763 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.752+0000 7fd43d59a700 10 monclient: _finish_hunting 0 2021-05-25T12:05:28.763 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.752+0000 7fd43d59a700 1 monclient: found mon.noname-a 2021-05-25T12:05:28.763 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.752+0000 7fd43d59a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:05:28.763 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.752+0000 7fd43d59a700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T12:05:28.763 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.752+0000 7fd43d59a700 1 -- 172.21.15.102:0/3304217414 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd440094170 con 0x7fd440132900 2021-05-25T12:05:28.764 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.752+0000 7fd43d59a700 10 monclient: _finish_auth 0 2021-05-25T12:05:28.764 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.752+0000 7fd43d59a700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:05:28.764 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.752+0000 7fd43d59a700 1 -- 172.21.15.102:0/3304217414 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fd428001610 con 0x7fd440132900 2021-05-25T12:05:28.764 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.752+0000 7fd43d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:28.765 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.752+0000 7fd43d59a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:05:28.765 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.752+0000 7fd43d59a700 10 monclient: dump: 2021-05-25T12:05:28.765 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:28.765 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:28.765 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:28.766 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:28.766 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:28.766 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:28.766 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:28.766 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:28.767 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:28.767 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:28.767 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.752+0000 7fd4448a2700 5 monclient: authenticate success, global_id 174573 2021-05-25T12:05:28.767 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.752+0000 7fd4448a2700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:05:28.768 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.752+0000 7fd43d59a700 1 -- 172.21.15.102:0/3304217414 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd428001930 con 0x7fd440132900 2021-05-25T12:05:28.768 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.752+0000 7fd43d59a700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:05:28.768 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.752+0000 7fd4448a2700 10 monclient: get_monmap_and_config success 2021-05-25T12:05:28.768 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.752+0000 7fd43d59a700 1 -- 172.21.15.102:0/3304217414 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fd428001cc0 con 0x7fd440132900 2021-05-25T12:05:28.768 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.752+0000 7fd43d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:28.769 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.752+0000 7fd4448a2700 4 set_mon_vals no callback set 2021-05-25T12:05:28.769 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.752+0000 7fd43d59a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:05:28.769 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.752+0000 7fd43d59a700 10 monclient: dump: 2021-05-25T12:05:28.769 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:28.769 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:28.770 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:28.770 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:28.770 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:28.770 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:28.771 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:28.771 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:28.771 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:28.771 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:28.771 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.753+0000 7fd4448a2700 10 monclient: shutdown 2021-05-25T12:05:28.772 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.753+0000 7fd4448a2700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:05:28.772 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.753+0000 7fd4448a2700 1 -- 172.21.15.102:0/3304217414 >> v1:172.21.15.102:6789/0 conn(0x7fd440132900 legacy=0x7fd440134cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:28.772 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.753+0000 7fd4448a2700 1 -- 172.21.15.102:0/3304217414 shutdown_connections 2021-05-25T12:05:28.772 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.753+0000 7fd4448a2700 1 -- 172.21.15.102:0/3304217414 >> 172.21.15.102:0/3304217414 conn(0x7fd44009b370 msgr2=0x7fd440097430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:05:28.772 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.753+0000 7fd4448a2700 1 -- 172.21.15.102:0/3304217414 shutdown_connections 2021-05-25T12:05:28.773 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.753+0000 7fd4448a2700 1 -- 172.21.15.102:0/3304217414 wait complete. 2021-05-25T12:05:28.773 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.754+0000 7fd4448a2700 10 monclient: build_initial_monmap 2021-05-25T12:05:28.773 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.754+0000 7fd4448a2700 10 monclient: monmap: 2021-05-25T12:05:28.773 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:05:28.774 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:05:28.774 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:05:28.774 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:05:28.774 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:05:28.774 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:05:28.775 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:05:28.775 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:05:28.775 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:05:28.775 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:28.775 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.754+0000 7fd4448a2700 1 librados: starting msgr at 2021-05-25T12:05:28.776 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.754+0000 7fd4448a2700 1 librados: starting objecter 2021-05-25T12:05:28.776 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.754+0000 7fd4448a2700 1 Processor -- start 2021-05-25T12:05:28.776 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.754+0000 7fd4448a2700 1 -- start start 2021-05-25T12:05:28.776 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.754+0000 7fd4448a2700 1 librados: setting wanted keys 2021-05-25T12:05:28.777 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.754+0000 7fd4448a2700 1 librados: calling monclient init 2021-05-25T12:05:28.777 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.754+0000 7fd4448a2700 10 monclient: init 2021-05-25T12:05:28.777 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.754+0000 7fd4448a2700 10 monclient: _reopen_session rank -1 2021-05-25T12:05:28.777 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.754+0000 7fd4448a2700 10 monclient: _add_conns ranks=[0,2,1] 2021-05-25T12:05:28.777 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.754+0000 7fd4448a2700 10 monclient(hunting): picked mon.noname-a con 0x7fd440093d90 addr v1:172.21.15.102:6789/0 2021-05-25T12:05:28.778 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.755+0000 7fd4448a2700 10 monclient(hunting): picked mon.noname-c con 0x7fd440132900 addr v1:172.21.15.102:6790/0 2021-05-25T12:05:28.778 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.755+0000 7fd4448a2700 10 monclient(hunting): picked mon.noname-b con 0x7fd440136410 addr v1:172.21.15.181:6789/0 2021-05-25T12:05:28.778 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.755+0000 7fd4448a2700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd440068aa0 con 0x7fd440093d90 2021-05-25T12:05:28.778 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.755+0000 7fd4448a2700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd440119780 con 0x7fd440136410 2021-05-25T12:05:28.778 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.755+0000 7fd4448a2700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd440066c40 con 0x7fd440132900 2021-05-25T12:05:28.779 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.755+0000 7fd4448a2700 10 monclient(hunting): _renew_subs 2021-05-25T12:05:28.779 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.755+0000 7fd4448a2700 10 monclient(hunting): authenticate will time out at 1129.352118s 2021-05-25T12:05:28.779 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.755+0000 7fd43e59c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fd440093d90 0x7fd440065980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:32826/0 (socket says 172.21.15.102:32826) 2021-05-25T12:05:28.779 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.755+0000 7fd43e59c700 1 -- 172.21.15.102:0/3523259659 learned_addr learned my addr 172.21.15.102:0/3523259659 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:05:28.780 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.755+0000 7fd42e7fc700 10 client.?.objecter ms_handle_connect 0x7fd440132900 2021-05-25T12:05:28.780 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.755+0000 7fd42e7fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:05:28.780 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.755+0000 7fd42e7fc700 10 client.?.objecter ms_handle_connect 0x7fd440093d90 2021-05-25T12:05:28.780 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.755+0000 7fd42e7fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:05:28.781 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.755+0000 7fd42e7fc700 10 client.?.objecter ms_handle_connect 0x7fd440136410 2021-05-25T12:05:28.781 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.755+0000 7fd42e7fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:05:28.781 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.755+0000 7fd42e7fc700 1 -- 172.21.15.102:0/3523259659 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2806458836 0 0) 0x7fd440068aa0 con 0x7fd440093d90 2021-05-25T12:05:28.781 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.755+0000 7fd42e7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:28.781 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.755+0000 7fd42e7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:28.782 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.755+0000 7fd42e7fc700 10 monclient(hunting): my global_id is 174576 2021-05-25T12:05:28.782 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.755+0000 7fd42e7fc700 1 -- 172.21.15.102:0/3523259659 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd41c004c10 con 0x7fd440093d90 2021-05-25T12:05:28.782 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.755+0000 7fd42e7fc700 1 -- 172.21.15.102:0/3523259659 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3839201022 0 0) 0x7fd41c004c10 con 0x7fd440093d90 2021-05-25T12:05:28.782 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.755+0000 7fd42e7fc700 1 -- 172.21.15.102:0/3523259659 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd440068aa0 con 0x7fd440093d90 2021-05-25T12:05:28.783 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.756+0000 7fd42e7fc700 1 -- 172.21.15.102:0/3523259659 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fd4340017b0 con 0x7fd440093d90 2021-05-25T12:05:28.783 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.756+0000 7fd42e7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:28.783 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.756+0000 7fd42e7fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:05:28.783 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.756+0000 7fd42e7fc700 10 monclient(hunting): dump: 2021-05-25T12:05:28.783 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:28.784 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:28.784 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:28.784 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:28.784 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:28.784 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:28.785 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:28.785 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:28.785 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:28.785 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:28.786 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.756+0000 7fd42e7fc700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:05:28.786 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.756+0000 7fd42e7fc700 1 -- 172.21.15.102:0/3523259659 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4091122652 0 0) 0x7fd440068aa0 con 0x7fd440093d90 2021-05-25T12:05:28.786 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.756+0000 7fd42e7fc700 1 -- 172.21.15.102:0/3523259659 >> v1:172.21.15.102:6790/0 conn(0x7fd440132900 legacy=0x7fd440144830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:28.786 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.756+0000 7fd42e7fc700 1 -- 172.21.15.102:0/3523259659 >> v1:172.21.15.181:6789/0 conn(0x7fd440136410 legacy=0x7fd440147f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:28.786 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.756+0000 7fd42e7fc700 10 monclient: _finish_hunting 0 2021-05-25T12:05:28.787 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.756+0000 7fd42e7fc700 1 monclient: found mon.a 2021-05-25T12:05:28.787 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.756+0000 7fd42e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:05:28.787 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.756+0000 7fd42e7fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:05:28.787 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.756+0000 7fd42e7fc700 1 -- 172.21.15.102:0/3523259659 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd440067870 con 0x7fd440093d90 2021-05-25T12:05:28.788 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.756+0000 7fd42e7fc700 10 monclient: _finish_auth 0 2021-05-25T12:05:28.788 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.756+0000 7fd42e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:05:28.788 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.756+0000 7fd4448a2700 5 monclient: authenticate success, global_id 174576 2021-05-25T12:05:28.788 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.756+0000 7fd4448a2700 10 monclient: _renew_subs 2021-05-25T12:05:28.788 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.756+0000 7fd4448a2700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:05:28.789 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.756+0000 7fd4448a2700 1 -- 172.21.15.102:0/3523259659 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd440066090 con 0x7fd440093d90 2021-05-25T12:05:28.789 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.756+0000 7fd4448a2700 10 client.174576.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:05:28.789 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.756+0000 7fd4448a2700 10 monclient: _renew_subs 2021-05-25T12:05:28.789 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.756+0000 7fd4448a2700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:05:28.790 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.756+0000 7fd4448a2700 1 -- 172.21.15.102:0/3523259659 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4400650c0 con 0x7fd440093d90 2021-05-25T12:05:28.790 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.756+0000 7fd4448a2700 1 librados: init done 2021-05-25T12:05:28.790 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.757+0000 7fd42e7fc700 1 -- 172.21.15.102:0/3523259659 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd434004530 con 0x7fd440093d90 2021-05-25T12:05:28.790 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.757+0000 7fd42e7fc700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:05:28.790 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.757+0000 7fd42e7fc700 1 -- 172.21.15.102:0/3523259659 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fd4340048c0 con 0x7fd440093d90 2021-05-25T12:05:28.791 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.757+0000 7fd42e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:28.791 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.757+0000 7fd42e7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:05:28.791 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.757+0000 7fd43cd99700 4 set_mon_vals no callback set 2021-05-25T12:05:28.791 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.757+0000 7fd42e7fc700 10 monclient: dump: 2021-05-25T12:05:28.792 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:28.792 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:28.792 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:28.792 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:28.792 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:28.793 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:28.793 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:28.793 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:28.793 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:28.793 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:28.794 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.757+0000 7fd42e7fc700 1 -- 172.21.15.102:0/3523259659 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7fd434004a20 con 0x7fd440093d90 2021-05-25T12:05:28.794 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.757+0000 7fd4448a2700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:05:28.794 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.757+0000 7fd42e7fc700 1 -- 172.21.15.102:0/3523259659 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(160..160 src has 1..160) v4 ==== 6444+0+0 (unknown 3124456911 0 0) 0x7fd43403d790 con 0x7fd440093d90 2021-05-25T12:05:28.794 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.757+0000 7fd42e7fc700 10 client.174576.objecter ms_dispatch 0x7fd440001550 osd_map(160..160 src has 1..160) v4 2021-05-25T12:05:28.795 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.757+0000 7fd42e7fc700 3 client.174576.objecter handle_osd_map got epochs [160,160] > 0 2021-05-25T12:05:28.795 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.757+0000 7fd42e7fc700 3 client.174576.objecter handle_osd_map decoding full epoch 160 2021-05-25T12:05:28.795 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.757+0000 7fd42e7fc700 20 client.174576.objecter dump_active .. 0 homeless 2021-05-25T12:05:28.795 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.758+0000 7fd4448a2700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:05:28.795 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.758+0000 7fd4448a2700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:05:28.796 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.758+0000 7fd4448a2700 1 -- 172.21.15.102:0/3523259659 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd420004db0 con 0x7fd440093d90 2021-05-25T12:05:28.796 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.758+0000 7fd43dd9b700 10 monclient: get_auth_request con 0x7fd41c033220 auth_method 0 2021-05-25T12:05:28.796 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.758+0000 7fd42e7fc700 10 client.174576.objecter ms_handle_connect 0x7fd41c033220 2021-05-25T12:05:28.797 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.764+0000 7fd42e7fc700 1 -- 172.21.15.102:0/3523259659 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7fd434012c90 con 0x7fd440093d90 2021-05-25T12:05:28.797 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.764+0000 7fd42e7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:05:28.797 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.764+0000 7fd42e7fc700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:05:28.987 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.985+0000 7fd4448a2700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2021-05-25T12:05:28.987 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.985+0000 7fd4448a2700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2021-05-25T12:05:28.987 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.985+0000 7fd4448a2700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:05:28.988 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.985+0000 7fd4448a2700 1 -- 172.21.15.102:0/3523259659 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fd420005060 con 0x7fd440093d90 2021-05-25T12:05:28.988 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.986+0000 7fd42e7fc700 1 -- 172.21.15.102:0/3523259659 <== mon.0 v1:172.21.15.102:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1336 (unknown 254746657 0 657976693) 0x7fd434012880 con 0x7fd440093d90 2021-05-25T12:05:28.988 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.986+0000 7fd42e7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2021-05-25T12:05:28.988 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.986+0000 7fd42e7fc700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:05:28.989 INFO:teuthology.orchestra.run.smithi102.stdout:{"election_epoch":44,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138303579357183","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"20ee91b6-8935-4b98-b868-85c03789ccc4","modified":"2021-05-25T11:59:11.011522Z","created":"2021-05-25T11:59:11.011522Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6790","nonce":0}]},"addr":"172.21.15.102:6790/0","public_addr":"172.21.15.102:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2021-05-25T12:05:28.990 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.987+0000 7fd4448a2700 10 librados: watch_flush enter 2021-05-25T12:05:28.991 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.987+0000 7fd4448a2700 10 librados: watch_flush exit 2021-05-25T12:05:28.991 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.987+0000 7fd4448a2700 20 client.174576.objecter shutdown clearing up homeless session... 2021-05-25T12:05:28.991 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.987+0000 7fd4448a2700 10 client.174576.objecter successfully canceled tick 2021-05-25T12:05:28.991 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.987+0000 7fd4448a2700 1 -- 172.21.15.102:0/3523259659 >> v1:172.21.15.181:6812/29582 conn(0x7fd41c033220 legacy=0x7fd41c0356e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:28.991 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.987+0000 7fd4448a2700 10 monclient: shutdown 2021-05-25T12:05:28.992 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.987+0000 7fd4448a2700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:05:28.992 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.987+0000 7fd4448a2700 1 -- 172.21.15.102:0/3523259659 >> v1:172.21.15.102:6789/0 conn(0x7fd440093d90 legacy=0x7fd440065980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:28.992 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.987+0000 7fd4448a2700 1 -- 172.21.15.102:0/3523259659 shutdown_connections 2021-05-25T12:05:28.992 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.987+0000 7fd4448a2700 1 -- 172.21.15.102:0/3523259659 >> 172.21.15.102:0/3523259659 conn(0x7fd44009b370 msgr2=0x7fd440068f60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:05:28.992 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.987+0000 7fd4448a2700 1 -- 172.21.15.102:0/3523259659 shutdown_connections 2021-05-25T12:05:28.993 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.987+0000 7fd4448a2700 1 -- 172.21.15.102:0/3523259659 wait complete. 2021-05-25T12:05:28.993 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:28.987+0000 7fd4448a2700 1 librados: shutdown 2021-05-25T12:05:32.001 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2021-05-25T12:05:32.138 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.136+0000 7f80dadb6700 10 monclient: get_monmap_and_config 2021-05-25T12:05:32.139 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.136+0000 7f80dadb6700 10 monclient: build_initial_monmap 2021-05-25T12:05:32.139 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.136+0000 7f80dadb6700 10 monclient: monmap: 2021-05-25T12:05:32.139 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:05:32.139 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:32.140 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:05:32.137633+0000 2021-05-25T12:05:32.140 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:05:32.137633+0000 2021-05-25T12:05:32.140 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:05:32.140 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:32.141 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:05:32.141 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:05:32.141 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:05:32.141 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:32.142 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.137+0000 7f80dadb6700 1 Processor -- start 2021-05-25T12:05:32.142 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.137+0000 7f80dadb6700 1 -- start start 2021-05-25T12:05:32.142 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.137+0000 7f80dadb6700 10 monclient: init 2021-05-25T12:05:32.151 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.137+0000 7f80dadb6700 10 monclient: _reopen_session rank -1 2021-05-25T12:05:32.152 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.137+0000 7f80dadb6700 10 monclient: _add_conns ranks=[2,1,0] 2021-05-25T12:05:32.152 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.137+0000 7f80dadb6700 10 monclient(hunting): picked mon.noname-b con 0x7f80d4069320 addr v1:172.21.15.102:6790/0 2021-05-25T12:05:32.152 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.137+0000 7f80dadb6700 10 monclient(hunting): picked mon.noname-c con 0x7f80d4096400 addr v1:172.21.15.181:6789/0 2021-05-25T12:05:32.152 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.137+0000 7f80dadb6700 10 monclient(hunting): picked mon.noname-a con 0x7f80d4131da0 addr v1:172.21.15.102:6789/0 2021-05-25T12:05:32.153 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.137+0000 7f80dadb6700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80d4069700 con 0x7f80d4131da0 2021-05-25T12:05:32.153 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.137+0000 7f80dadb6700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80d4129220 con 0x7f80d4096400 2021-05-25T12:05:32.153 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.137+0000 7f80dadb6700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80d4094990 con 0x7f80d4069320 2021-05-25T12:05:32.153 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.137+0000 7f80dadb6700 10 monclient(hunting): _renew_subs 2021-05-25T12:05:32.153 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.137+0000 7f80dadb6700 10 monclient(hunting): authenticate will time out at 1132.734925s 2021-05-25T12:05:32.154 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.138+0000 7f80d9db4700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f80d4069320 0x7f80d4128b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:55944/0 (socket says 172.21.15.102:55944) 2021-05-25T12:05:32.154 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.138+0000 7f80d9db4700 1 -- 172.21.15.102:0/3680944203 learned_addr learned my addr 172.21.15.102:0/3680944203 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:05:32.154 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.138+0000 7f80da5b5700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f80d4131da0 0x7f80d4134190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:32836/0 (socket says 172.21.15.102:32836) 2021-05-25T12:05:32.154 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.138+0000 7f80d8db2700 1 -- 172.21.15.102:0/3680944203 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4012139460 0 0) 0x7f80d4094990 con 0x7f80d4069320 2021-05-25T12:05:32.155 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.138+0000 7f80d8db2700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:32.155 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.138+0000 7f80d8db2700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:32.155 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.138+0000 7f80d8db2700 10 monclient(hunting): my global_id is 184298 2021-05-25T12:05:32.155 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.138+0000 7f80d8db2700 1 -- 172.21.15.102:0/3680944203 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80bc003270 con 0x7f80d4069320 2021-05-25T12:05:32.156 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.138+0000 7f80d8db2700 1 -- 172.21.15.102:0/3680944203 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2449196972 0 0) 0x7f80d00014e0 con 0x7f80d4131da0 2021-05-25T12:05:32.156 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.138+0000 7f80d8db2700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:32.156 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.138+0000 7f80d8db2700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:32.156 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.138+0000 7f80d8db2700 10 monclient(hunting): my global_id is 174591 2021-05-25T12:05:32.156 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.138+0000 7f80d8db2700 1 -- 172.21.15.102:0/3680944203 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80d4094990 con 0x7f80d4131da0 2021-05-25T12:05:32.157 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.138+0000 7f80d8db2700 1 -- 172.21.15.102:0/3680944203 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2676208808 0 0) 0x7f80d4094990 con 0x7f80d4131da0 2021-05-25T12:05:32.157 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.138+0000 7f80d8db2700 1 -- 172.21.15.102:0/3680944203 >> v1:172.21.15.102:6790/0 conn(0x7f80d4069320 legacy=0x7f80d4128b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:32.157 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.138+0000 7f80d8db2700 1 -- 172.21.15.102:0/3680944203 >> v1:172.21.15.181:6789/0 conn(0x7f80d4096400 legacy=0x7f80d4094280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:32.157 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.138+0000 7f80d8db2700 10 monclient: _finish_hunting 0 2021-05-25T12:05:32.158 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.138+0000 7f80d8db2700 1 monclient: found mon.noname-a 2021-05-25T12:05:32.158 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.138+0000 7f80d8db2700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:05:32.158 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.138+0000 7f80d8db2700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T12:05:32.158 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.138+0000 7f80d8db2700 1 -- 172.21.15.102:0/3680944203 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80d4090b10 con 0x7f80d4131da0 2021-05-25T12:05:32.158 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.138+0000 7f80d8db2700 10 monclient: _finish_auth 0 2021-05-25T12:05:32.159 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.138+0000 7f80d8db2700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:05:32.159 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.138+0000 7f80dadb6700 5 monclient: authenticate success, global_id 174591 2021-05-25T12:05:32.159 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.138+0000 7f80d8db2700 1 -- 172.21.15.102:0/3680944203 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f80d00019d0 con 0x7f80d4131da0 2021-05-25T12:05:32.159 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.138+0000 7f80dadb6700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:05:32.160 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.138+0000 7f80d8db2700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:32.160 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.138+0000 7f80d8db2700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:05:32.160 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.138+0000 7f80d8db2700 10 monclient: dump: 2021-05-25T12:05:32.160 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:32.161 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:32.161 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:32.161 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:32.161 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:32.161 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:32.162 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:32.162 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:32.162 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:32.162 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:32.163 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.138+0000 7f80dadb6700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:05:32.163 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.138+0000 7f80d8db2700 1 -- 172.21.15.102:0/3680944203 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f80d0001b80 con 0x7f80d4131da0 2021-05-25T12:05:32.163 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.138+0000 7f80d8db2700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:05:32.163 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.138+0000 7f80d8db2700 1 -- 172.21.15.102:0/3680944203 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f80d0001f10 con 0x7f80d4131da0 2021-05-25T12:05:32.164 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.138+0000 7f80dadb6700 10 monclient: get_monmap_and_config success 2021-05-25T12:05:32.164 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.138+0000 7f80dadb6700 4 set_mon_vals no callback set 2021-05-25T12:05:32.164 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.138+0000 7f80d8db2700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:32.164 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.138+0000 7f80d8db2700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:05:32.164 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.138+0000 7f80d8db2700 10 monclient: dump: 2021-05-25T12:05:32.165 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:32.165 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:32.165 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:32.166 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:32.166 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:32.166 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:32.166 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:32.167 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:32.167 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:32.167 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:32.167 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.139+0000 7f80dadb6700 10 monclient: shutdown 2021-05-25T12:05:32.167 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.139+0000 7f80dadb6700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:05:32.168 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.139+0000 7f80dadb6700 1 -- 172.21.15.102:0/3680944203 >> v1:172.21.15.102:6789/0 conn(0x7f80d4131da0 legacy=0x7f80d4134190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:32.168 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.139+0000 7f80dadb6700 1 -- 172.21.15.102:0/3680944203 shutdown_connections 2021-05-25T12:05:32.168 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.139+0000 7f80dadb6700 1 -- 172.21.15.102:0/3680944203 >> 172.21.15.102:0/3680944203 conn(0x7f80d40974b0 msgr2=0x7f80d40978c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:05:32.168 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.139+0000 7f80dadb6700 1 -- 172.21.15.102:0/3680944203 shutdown_connections 2021-05-25T12:05:32.169 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.139+0000 7f80dadb6700 1 -- 172.21.15.102:0/3680944203 wait complete. 2021-05-25T12:05:32.169 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.139+0000 7f80dadb6700 10 monclient: build_initial_monmap 2021-05-25T12:05:32.169 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.139+0000 7f80dadb6700 10 monclient: monmap: 2021-05-25T12:05:32.169 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:05:32.169 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:05:32.170 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:05:32.170 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:05:32.170 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:05:32.170 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:05:32.170 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:05:32.171 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:05:32.171 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:05:32.171 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:32.171 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.140+0000 7f80dadb6700 1 librados: starting msgr at 2021-05-25T12:05:32.172 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.140+0000 7f80dadb6700 1 librados: starting objecter 2021-05-25T12:05:32.172 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.140+0000 7f80dadb6700 1 Processor -- start 2021-05-25T12:05:32.172 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.140+0000 7f80dadb6700 1 -- start start 2021-05-25T12:05:32.172 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.140+0000 7f80dadb6700 1 librados: setting wanted keys 2021-05-25T12:05:32.172 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.140+0000 7f80dadb6700 1 librados: calling monclient init 2021-05-25T12:05:32.173 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.140+0000 7f80dadb6700 10 monclient: init 2021-05-25T12:05:32.173 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.140+0000 7f80dadb6700 10 monclient: _reopen_session rank -1 2021-05-25T12:05:32.173 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.140+0000 7f80dadb6700 10 monclient: _add_conns ranks=[2,1,0] 2021-05-25T12:05:32.173 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.140+0000 7f80dadb6700 10 monclient(hunting): picked mon.noname-c con 0x7f80d4069320 addr v1:172.21.15.102:6790/0 2021-05-25T12:05:32.174 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.140+0000 7f80dadb6700 10 monclient(hunting): picked mon.noname-b con 0x7f80d4096400 addr v1:172.21.15.181:6789/0 2021-05-25T12:05:32.174 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.141+0000 7f80dadb6700 10 monclient(hunting): picked mon.noname-a con 0x7f80d4131da0 addr v1:172.21.15.102:6789/0 2021-05-25T12:05:32.174 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.141+0000 7f80da5b5700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f80d4131da0 0x7f80d4147ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:32844/0 (socket says 172.21.15.102:32844) 2021-05-25T12:05:32.174 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.141+0000 7f80da5b5700 1 -- 172.21.15.102:0/2578091782 learned_addr learned my addr 172.21.15.102:0/2578091782 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:05:32.175 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.141+0000 7f80dadb6700 1 -- 172.21.15.102:0/2578091782 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80d4067560 con 0x7f80d4131da0 2021-05-25T12:05:32.175 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.141+0000 7f80c1ffb700 10 client.?.objecter ms_handle_connect 0x7f80d4131da0 2021-05-25T12:05:32.175 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.141+0000 7f80c1ffb700 10 client.?.objecter resend_mon_ops 2021-05-25T12:05:32.175 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.141+0000 7f80dadb6700 1 -- 172.21.15.102:0/2578091782 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80d4064db0 con 0x7f80d4096400 2021-05-25T12:05:32.176 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.141+0000 7f80c1ffb700 10 client.?.objecter ms_handle_connect 0x7f80d4096400 2021-05-25T12:05:32.176 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.141+0000 7f80c1ffb700 10 client.?.objecter resend_mon_ops 2021-05-25T12:05:32.176 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.141+0000 7f80dadb6700 1 -- 172.21.15.102:0/2578091782 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80d4067230 con 0x7f80d4069320 2021-05-25T12:05:32.176 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.141+0000 7f80dadb6700 10 monclient(hunting): _renew_subs 2021-05-25T12:05:32.177 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.141+0000 7f80dadb6700 10 monclient(hunting): authenticate will time out at 1132.738341s 2021-05-25T12:05:32.177 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.141+0000 7f80c1ffb700 1 -- 172.21.15.102:0/2578091782 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1655625151 0 0) 0x7f80d4067560 con 0x7f80d4131da0 2021-05-25T12:05:32.177 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.141+0000 7f80c1ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:32.177 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.141+0000 7f80c1ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:32.177 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.141+0000 7f80c1ffb700 10 monclient(hunting): my global_id is 174594 2021-05-25T12:05:32.178 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.141+0000 7f80c1ffb700 1 -- 172.21.15.102:0/2578091782 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80b4004c10 con 0x7f80d4131da0 2021-05-25T12:05:32.178 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.141+0000 7f80c1ffb700 10 client.?.objecter ms_handle_connect 0x7f80d4069320 2021-05-25T12:05:32.178 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.141+0000 7f80c1ffb700 10 client.?.objecter resend_mon_ops 2021-05-25T12:05:32.178 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.141+0000 7f80c1ffb700 1 -- 172.21.15.102:0/2578091782 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4665918 0 0) 0x7f80b4004c10 con 0x7f80d4131da0 2021-05-25T12:05:32.179 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.142+0000 7f80c1ffb700 1 -- 172.21.15.102:0/2578091782 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f80d4067560 con 0x7f80d4131da0 2021-05-25T12:05:32.179 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.142+0000 7f80c1ffb700 1 -- 172.21.15.102:0/2578091782 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f80d00019d0 con 0x7f80d4131da0 2021-05-25T12:05:32.179 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.142+0000 7f80c1ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:32.179 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.142+0000 7f80c1ffb700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:05:32.179 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.142+0000 7f80c1ffb700 10 monclient(hunting): dump: 2021-05-25T12:05:32.180 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:32.180 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:32.180 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:32.180 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:32.181 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:32.181 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:32.181 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:32.181 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:32.182 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:32.182 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:32.182 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.142+0000 7f80c1ffb700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:05:32.182 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.142+0000 7f80c1ffb700 1 -- 172.21.15.102:0/2578091782 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1109767076 0 0) 0x7f80d4067230 con 0x7f80d4069320 2021-05-25T12:05:32.183 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.142+0000 7f80c1ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:32.183 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.142+0000 7f80c1ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:32.183 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.142+0000 7f80c1ffb700 10 monclient(hunting): my global_id is 184301 2021-05-25T12:05:32.183 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.142+0000 7f80c1ffb700 1 -- 172.21.15.102:0/2578091782 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80b4004c10 con 0x7f80d4069320 2021-05-25T12:05:32.184 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.142+0000 7f80c1ffb700 1 -- 172.21.15.102:0/2578091782 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3098497233 0 0) 0x7f80b4004c10 con 0x7f80d4069320 2021-05-25T12:05:32.184 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.142+0000 7f80c1ffb700 1 -- 172.21.15.102:0/2578091782 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f80d4067230 con 0x7f80d4069320 2021-05-25T12:05:32.184 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.142+0000 7f80c1ffb700 1 -- 172.21.15.102:0/2578091782 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f80c80016a0 con 0x7f80d4069320 2021-05-25T12:05:32.184 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.142+0000 7f80c1ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:32.184 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.142+0000 7f80c1ffb700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2021-05-25T12:05:32.185 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.142+0000 7f80c1ffb700 10 monclient(hunting): dump: 2021-05-25T12:05:32.185 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:32.185 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:32.185 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:32.185 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:32.186 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:32.186 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:32.186 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:32.186 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:32.186 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:32.187 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:32.187 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.142+0000 7f80c1ffb700 1 -- 172.21.15.102:0/2578091782 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2130949465 0 0) 0x7f80d4067230 con 0x7f80d4069320 2021-05-25T12:05:32.187 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.142+0000 7f80c1ffb700 1 -- 172.21.15.102:0/2578091782 >> v1:172.21.15.181:6789/0 conn(0x7f80d4096400 legacy=0x7f80d40646a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:32.187 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.142+0000 7f80c1ffb700 1 -- 172.21.15.102:0/2578091782 >> v1:172.21.15.102:6789/0 conn(0x7f80d4131da0 legacy=0x7f80d4147ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:32.187 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.142+0000 7f80c1ffb700 10 monclient: _finish_hunting 0 2021-05-25T12:05:32.188 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.142+0000 7f80c1ffb700 1 monclient: found mon.c 2021-05-25T12:05:32.188 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.142+0000 7f80c1ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:05:32.188 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.142+0000 7f80c1ffb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:05:32.188 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.142+0000 7f80c1ffb700 1 -- 172.21.15.102:0/2578091782 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80d4067790 con 0x7f80d4069320 2021-05-25T12:05:32.189 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.142+0000 7f80c1ffb700 10 monclient: _finish_auth 0 2021-05-25T12:05:32.189 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.142+0000 7f80c1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:05:32.189 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.142+0000 7f80dadb6700 5 monclient: authenticate success, global_id 184301 2021-05-25T12:05:32.189 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.142+0000 7f80dadb6700 10 monclient: _renew_subs 2021-05-25T12:05:32.189 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.142+0000 7f80dadb6700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:05:32.190 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.142+0000 7f80dadb6700 1 -- 172.21.15.102:0/2578091782 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80d4067a30 con 0x7f80d4069320 2021-05-25T12:05:32.190 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.142+0000 7f80c1ffb700 1 -- 172.21.15.102:0/2578091782 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f80c8004530 con 0x7f80d4069320 2021-05-25T12:05:32.190 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.142+0000 7f80c1ffb700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:05:32.190 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.142+0000 7f80c1ffb700 1 -- 172.21.15.102:0/2578091782 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f80c8004840 con 0x7f80d4069320 2021-05-25T12:05:32.191 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.142+0000 7f80c1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:32.191 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.142+0000 7f80c1ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:05:32.191 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.142+0000 7f80c1ffb700 10 monclient: dump: 2021-05-25T12:05:32.191 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:32.191 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:32.192 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:32.192 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:32.192 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:32.192 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:32.192 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:32.193 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:32.193 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:32.193 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:32.193 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.142+0000 7f80c3fff700 4 set_mon_vals no callback set 2021-05-25T12:05:32.193 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.142+0000 7f80dadb6700 10 client.184301.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:05:32.194 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.142+0000 7f80dadb6700 10 monclient: _renew_subs 2021-05-25T12:05:32.194 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.142+0000 7f80dadb6700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:05:32.194 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.142+0000 7f80dadb6700 1 -- 172.21.15.102:0/2578091782 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f80d4065be0 con 0x7f80d4069320 2021-05-25T12:05:32.194 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.143+0000 7f80dadb6700 1 librados: init done 2021-05-25T12:05:32.194 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.143+0000 7f80c1ffb700 1 -- 172.21.15.102:0/2578091782 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f80c800db60 con 0x7f80d4069320 2021-05-25T12:05:32.195 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.143+0000 7f80c1ffb700 1 -- 172.21.15.102:0/2578091782 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(164..164 src has 1..164) v4 ==== 7748+0+0 (unknown 272760688 0 0) 0x7f80c803dc80 con 0x7f80d4069320 2021-05-25T12:05:32.195 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.143+0000 7f80c1ffb700 10 client.184301.objecter ms_dispatch 0x7f80d4001550 osd_map(164..164 src has 1..164) v4 2021-05-25T12:05:32.195 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.143+0000 7f80c1ffb700 3 client.184301.objecter handle_osd_map got epochs [164,164] > 0 2021-05-25T12:05:32.195 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.143+0000 7f80c1ffb700 3 client.184301.objecter handle_osd_map decoding full epoch 164 2021-05-25T12:05:32.196 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.144+0000 7f80c1ffb700 20 client.184301.objecter dump_active .. 0 homeless 2021-05-25T12:05:32.196 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.144+0000 7f80d95b3700 10 monclient: get_auth_request con 0x7f80b4033370 auth_method 0 2021-05-25T12:05:32.196 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.144+0000 7f80dadb6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:05:32.196 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.144+0000 7f80dadb6700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:05:32.197 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.144+0000 7f80dadb6700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:05:32.197 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.144+0000 7f80dadb6700 1 -- 172.21.15.102:0/2578091782 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80d4059730 con 0x7f80d4069320 2021-05-25T12:05:32.197 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.144+0000 7f80c1ffb700 10 client.184301.objecter ms_handle_connect 0x7f80b4033370 2021-05-25T12:05:32.197 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.148+0000 7f80c1ffb700 1 -- 172.21.15.102:0/2578091782 <== mon.2 v1:172.21.15.102:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7f80c80128f0 con 0x7f80d4069320 2021-05-25T12:05:32.197 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.148+0000 7f80c1ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:05:32.198 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.148+0000 7f80c1ffb700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:05:32.438 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.436+0000 7f80dadb6700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2021-05-25T12:05:32.438 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.436+0000 7f80dadb6700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2021-05-25T12:05:32.439 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.436+0000 7f80dadb6700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:05:32.439 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.436+0000 7f80dadb6700 1 -- 172.21.15.102:0/2578091782 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f80d4066760 con 0x7f80d4069320 2021-05-25T12:05:32.439 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.437+0000 7f80c1ffb700 1 -- 172.21.15.102:0/2578091782 <== mon.2 v1:172.21.15.102:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1336 (unknown 254746657 0 3387854181) 0x7f80c8012710 con 0x7f80d4069320 2021-05-25T12:05:32.439 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.437+0000 7f80c1ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2021-05-25T12:05:32.440 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.437+0000 7f80c1ffb700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:05:32.440 INFO:teuthology.orchestra.run.smithi102.stdout:{"election_epoch":44,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"c","quorum_age":3,"features":{"quorum_con":"4540138303579357183","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"20ee91b6-8935-4b98-b868-85c03789ccc4","modified":"2021-05-25T11:59:11.011522Z","created":"2021-05-25T11:59:11.011522Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6790","nonce":0}]},"addr":"172.21.15.102:6790/0","public_addr":"172.21.15.102:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2021-05-25T12:05:32.442 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.438+0000 7f80dadb6700 10 librados: watch_flush enter 2021-05-25T12:05:32.442 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.438+0000 7f80dadb6700 10 librados: watch_flush exit 2021-05-25T12:05:32.442 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.438+0000 7f80dadb6700 20 client.184301.objecter shutdown clearing up homeless session... 2021-05-25T12:05:32.442 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.438+0000 7f80dadb6700 10 client.184301.objecter successfully canceled tick 2021-05-25T12:05:32.442 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.438+0000 7f80dadb6700 1 -- 172.21.15.102:0/2578091782 >> v1:172.21.15.181:6812/29582 conn(0x7f80b4033370 legacy=0x7f80b4035830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:32.443 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.438+0000 7f80dadb6700 10 monclient: shutdown 2021-05-25T12:05:32.443 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.438+0000 7f80dadb6700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:05:32.443 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.438+0000 7f80dadb6700 1 -- 172.21.15.102:0/2578091782 >> v1:172.21.15.102:6790/0 conn(0x7f80d4069320 legacy=0x7f80d4063f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:32.443 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.438+0000 7f80dadb6700 1 -- 172.21.15.102:0/2578091782 shutdown_connections 2021-05-25T12:05:32.444 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.438+0000 7f80dadb6700 1 -- 172.21.15.102:0/2578091782 >> 172.21.15.102:0/2578091782 conn(0x7f80d40974b0 msgr2=0x7f80d40999b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:05:32.444 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.438+0000 7f80dadb6700 1 -- 172.21.15.102:0/2578091782 shutdown_connections 2021-05-25T12:05:32.444 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.438+0000 7f80dadb6700 1 -- 172.21.15.102:0/2578091782 wait complete. 2021-05-25T12:05:32.444 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:32.438+0000 7f80dadb6700 1 librados: shutdown 2021-05-25T12:05:35.452 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2021-05-25T12:05:35.583 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.581+0000 7f4114ae3700 10 monclient: get_monmap_and_config 2021-05-25T12:05:35.583 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.581+0000 7f4114ae3700 10 monclient: build_initial_monmap 2021-05-25T12:05:35.584 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.581+0000 7f4114ae3700 10 monclient: monmap: 2021-05-25T12:05:35.584 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:05:35.584 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:35.585 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:05:35.582561+0000 2021-05-25T12:05:35.585 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:05:35.582561+0000 2021-05-25T12:05:35.585 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:05:35.585 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:35.585 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:05:35.586 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:05:35.586 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:05:35.586 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:35.586 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.582+0000 7f4114ae3700 1 Processor -- start 2021-05-25T12:05:35.587 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.582+0000 7f4114ae3700 1 -- start start 2021-05-25T12:05:35.587 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.582+0000 7f4114ae3700 10 monclient: init 2021-05-25T12:05:35.596 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.583+0000 7f4114ae3700 10 monclient: _reopen_session rank -1 2021-05-25T12:05:35.596 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.583+0000 7f4114ae3700 10 monclient: _add_conns ranks=[2,0,1] 2021-05-25T12:05:35.597 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.583+0000 7f4114ae3700 10 monclient(hunting): picked mon.noname-b con 0x7f4110069620 addr v1:172.21.15.102:6790/0 2021-05-25T12:05:35.597 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.583+0000 7f4114ae3700 10 monclient(hunting): picked mon.noname-a con 0x7f411012b100 addr v1:172.21.15.102:6789/0 2021-05-25T12:05:35.597 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.583+0000 7f4114ae3700 10 monclient(hunting): picked mon.noname-c con 0x7f411012ec10 addr v1:172.21.15.181:6789/0 2021-05-25T12:05:35.597 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.583+0000 7f4114ae3700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4110093070 con 0x7f411012b100 2021-05-25T12:05:35.598 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.583+0000 7f4114ae3700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4110093310 con 0x7f411012ec10 2021-05-25T12:05:35.598 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.583+0000 7f4114ae3700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41100935b0 con 0x7f4110069620 2021-05-25T12:05:35.598 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.583+0000 7f4114ae3700 10 monclient(hunting): _renew_subs 2021-05-25T12:05:35.598 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.583+0000 7f4114ae3700 10 monclient(hunting): authenticate will time out at 1136.180403s 2021-05-25T12:05:35.598 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.583+0000 7f410e59c700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f4110069620 0x7f4110093c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:55960/0 (socket says 172.21.15.102:55960) 2021-05-25T12:05:35.599 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.583+0000 7f410e59c700 1 -- 172.21.15.102:0/776629447 learned_addr learned my addr 172.21.15.102:0/776629447 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:05:35.599 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.583+0000 7f410d59a700 1 -- 172.21.15.102:0/776629447 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2692934996 0 0) 0x7f41100935b0 con 0x7f4110069620 2021-05-25T12:05:35.599 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.583+0000 7f410d59a700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:35.599 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.583+0000 7f410d59a700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:35.600 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.583+0000 7f410d59a700 10 monclient(hunting): my global_id is 184310 2021-05-25T12:05:35.600 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.583+0000 7f410d59a700 1 -- 172.21.15.102:0/776629447 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40f4003270 con 0x7f4110069620 2021-05-25T12:05:35.600 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.584+0000 7f410d59a700 1 -- 172.21.15.102:0/776629447 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1760993739 0 0) 0x7f4110093310 con 0x7f411012ec10 2021-05-25T12:05:35.600 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.584+0000 7f410d59a700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:35.601 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.584+0000 7f410d59a700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:35.601 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.584+0000 7f410d59a700 10 monclient(hunting): my global_id is 194116 2021-05-25T12:05:35.601 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.584+0000 7f410d59a700 1 -- 172.21.15.102:0/776629447 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41100935b0 con 0x7f411012ec10 2021-05-25T12:05:35.601 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.584+0000 7f410d59a700 1 -- 172.21.15.102:0/776629447 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4231417689 0 0) 0x7f4110093070 con 0x7f411012b100 2021-05-25T12:05:35.601 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.584+0000 7f410d59a700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:35.602 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.584+0000 7f410d59a700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:35.602 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.584+0000 7f410d59a700 10 monclient(hunting): my global_id is 174603 2021-05-25T12:05:35.602 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.584+0000 7f410d59a700 1 -- 172.21.15.102:0/776629447 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4110093310 con 0x7f411012b100 2021-05-25T12:05:35.602 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.584+0000 7f410d59a700 1 -- 172.21.15.102:0/776629447 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3576019426 0 0) 0x7f40f4003270 con 0x7f4110069620 2021-05-25T12:05:35.602 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.584+0000 7f410d59a700 1 -- 172.21.15.102:0/776629447 >> v1:172.21.15.181:6789/0 conn(0x7f411012ec10 legacy=0x7f4110131000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:35.603 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.584+0000 7f410d59a700 1 -- 172.21.15.102:0/776629447 >> v1:172.21.15.102:6789/0 conn(0x7f411012b100 legacy=0x7f411012d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:35.603 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.584+0000 7f410d59a700 10 monclient: _finish_hunting 0 2021-05-25T12:05:35.603 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.584+0000 7f410d59a700 1 monclient: found mon.noname-b 2021-05-25T12:05:35.603 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.584+0000 7f410d59a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:05:35.603 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.584+0000 7f410d59a700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T12:05:35.604 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.584+0000 7f410d59a700 1 -- 172.21.15.102:0/776629447 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4110093850 con 0x7f4110069620 2021-05-25T12:05:35.604 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.584+0000 7f410d59a700 10 monclient: _finish_auth 0 2021-05-25T12:05:35.604 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.584+0000 7f410d59a700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:05:35.604 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.584+0000 7f410d59a700 1 -- 172.21.15.102:0/776629447 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f4104001610 con 0x7f4110069620 2021-05-25T12:05:35.604 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.584+0000 7f410d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:35.605 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.584+0000 7f410d59a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:05:35.605 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.584+0000 7f410d59a700 10 monclient: dump: 2021-05-25T12:05:35.605 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:35.605 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:35.605 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:35.606 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:35.606 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:35.606 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:35.606 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:35.607 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:35.607 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:35.607 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:35.607 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.586+0000 7f410d59a700 1 -- 172.21.15.102:0/776629447 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4104001830 con 0x7f4110069620 2021-05-25T12:05:35.607 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.586+0000 7f410d59a700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:05:35.608 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.586+0000 7f410d59a700 1 -- 172.21.15.102:0/776629447 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f4104001b40 con 0x7f4110069620 2021-05-25T12:05:35.608 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.586+0000 7f410d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:35.608 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.586+0000 7f410d59a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:05:35.608 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.586+0000 7f410d59a700 10 monclient: dump: 2021-05-25T12:05:35.608 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:35.609 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:35.609 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:35.609 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:35.609 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:35.610 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:35.610 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:35.610 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:35.610 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:35.610 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:35.611 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.587+0000 7f4114ae3700 5 monclient: authenticate success, global_id 184310 2021-05-25T12:05:35.611 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.587+0000 7f4114ae3700 10 monclient: get_monmap_and_config success 2021-05-25T12:05:35.611 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.587+0000 7f4114ae3700 4 set_mon_vals no callback set 2021-05-25T12:05:35.611 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.587+0000 7f4114ae3700 10 monclient: shutdown 2021-05-25T12:05:35.611 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.587+0000 7f4114ae3700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:05:35.612 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.587+0000 7f4114ae3700 1 -- 172.21.15.102:0/776629447 >> v1:172.21.15.102:6790/0 conn(0x7f4110069620 legacy=0x7f4110093c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:35.612 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.587+0000 7f4114ae3700 1 -- 172.21.15.102:0/776629447 shutdown_connections 2021-05-25T12:05:35.612 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.587+0000 7f4114ae3700 1 -- 172.21.15.102:0/776629447 >> 172.21.15.102:0/776629447 conn(0x7f411009a910 msgr2=0x7f411009ad20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:05:35.612 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.587+0000 7f4114ae3700 1 -- 172.21.15.102:0/776629447 shutdown_connections 2021-05-25T12:05:35.612 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.587+0000 7f4114ae3700 1 -- 172.21.15.102:0/776629447 wait complete. 2021-05-25T12:05:35.613 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.588+0000 7f4114ae3700 10 monclient: build_initial_monmap 2021-05-25T12:05:35.613 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.588+0000 7f4114ae3700 10 monclient: monmap: 2021-05-25T12:05:35.613 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:05:35.613 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:05:35.613 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:05:35.614 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:05:35.614 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:05:35.614 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:05:35.614 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:05:35.615 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:05:35.615 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:05:35.615 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:35.615 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.588+0000 7f4114ae3700 1 librados: starting msgr at 2021-05-25T12:05:35.615 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.588+0000 7f4114ae3700 1 librados: starting objecter 2021-05-25T12:05:35.616 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.588+0000 7f4114ae3700 1 Processor -- start 2021-05-25T12:05:35.616 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.588+0000 7f4114ae3700 1 -- start start 2021-05-25T12:05:35.616 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.588+0000 7f4114ae3700 1 librados: setting wanted keys 2021-05-25T12:05:35.616 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.588+0000 7f4114ae3700 1 librados: calling monclient init 2021-05-25T12:05:35.617 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.588+0000 7f4114ae3700 10 monclient: init 2021-05-25T12:05:35.617 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.589+0000 7f4114ae3700 10 monclient: _reopen_session rank -1 2021-05-25T12:05:35.617 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.589+0000 7f4114ae3700 10 monclient: _add_conns ranks=[1,0,2] 2021-05-25T12:05:35.617 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.589+0000 7f4114ae3700 10 monclient(hunting): picked mon.noname-b con 0x7f4110069620 addr v1:172.21.15.181:6789/0 2021-05-25T12:05:35.617 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.589+0000 7f4114ae3700 10 monclient(hunting): picked mon.noname-a con 0x7f411012b100 addr v1:172.21.15.102:6789/0 2021-05-25T12:05:35.618 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.589+0000 7f4114ae3700 10 monclient(hunting): picked mon.noname-c con 0x7f411012ec10 addr v1:172.21.15.102:6790/0 2021-05-25T12:05:35.618 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.589+0000 7f4114ae3700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f411013f870 con 0x7f411012b100 2021-05-25T12:05:35.618 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.589+0000 7f4114ae3700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4110067440 con 0x7f4110069620 2021-05-25T12:05:35.618 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.589+0000 7f4114ae3700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41100676e0 con 0x7f411012ec10 2021-05-25T12:05:35.618 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.589+0000 7f4114ae3700 10 monclient(hunting): _renew_subs 2021-05-25T12:05:35.619 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.589+0000 7f410dd9b700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f411012b100 0x7f411012a2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:32858/0 (socket says 172.21.15.102:32858) 2021-05-25T12:05:35.619 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.589+0000 7f410dd9b700 1 -- 172.21.15.102:0/3706852240 learned_addr learned my addr 172.21.15.102:0/3706852240 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:05:35.619 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.589+0000 7f40fe7fc700 10 client.?.objecter ms_handle_connect 0x7f411012ec10 2021-05-25T12:05:35.619 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.589+0000 7f40fe7fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:05:35.619 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.589+0000 7f40fe7fc700 10 client.?.objecter ms_handle_connect 0x7f411012b100 2021-05-25T12:05:35.620 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.589+0000 7f40fe7fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:05:35.620 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.589+0000 7f4114ae3700 10 monclient(hunting): authenticate will time out at 1136.186263s 2021-05-25T12:05:35.620 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.589+0000 7f40fe7fc700 1 -- 172.21.15.102:0/3706852240 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 872707893 0 0) 0x7f41100676e0 con 0x7f411012ec10 2021-05-25T12:05:35.620 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.589+0000 7f40fe7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:35.620 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.589+0000 7f40fe7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:35.621 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.589+0000 7f40fe7fc700 10 monclient(hunting): my global_id is 184313 2021-05-25T12:05:35.621 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.589+0000 7f40fe7fc700 1 -- 172.21.15.102:0/3706852240 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40ec004c10 con 0x7f411012ec10 2021-05-25T12:05:35.621 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.589+0000 7f40fe7fc700 10 client.?.objecter ms_handle_connect 0x7f4110069620 2021-05-25T12:05:35.621 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.589+0000 7f40fe7fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:05:35.622 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.589+0000 7f40fe7fc700 1 -- 172.21.15.102:0/3706852240 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1009990121 0 0) 0x7f4110093310 con 0x7f411012b100 2021-05-25T12:05:35.622 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.589+0000 7f40fe7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:35.622 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.589+0000 7f40fe7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:35.622 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.589+0000 7f40fe7fc700 10 monclient(hunting): my global_id is 174606 2021-05-25T12:05:35.622 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.589+0000 7f40fe7fc700 1 -- 172.21.15.102:0/3706852240 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41100676e0 con 0x7f411012b100 2021-05-25T12:05:35.623 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.589+0000 7f40fe7fc700 1 -- 172.21.15.102:0/3706852240 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1089577586 0 0) 0x7f4110067440 con 0x7f4110069620 2021-05-25T12:05:35.623 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.589+0000 7f40fe7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:35.623 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.589+0000 7f40fe7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:35.623 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.589+0000 7f40fe7fc700 10 monclient(hunting): my global_id is 194119 2021-05-25T12:05:35.623 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.589+0000 7f40fe7fc700 1 -- 172.21.15.102:0/3706852240 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4110093310 con 0x7f4110069620 2021-05-25T12:05:35.624 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.589+0000 7f40fe7fc700 1 -- 172.21.15.102:0/3706852240 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3671504958 0 0) 0x7f40ec004c10 con 0x7f411012ec10 2021-05-25T12:05:35.624 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.589+0000 7f40fe7fc700 1 -- 172.21.15.102:0/3706852240 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f40ec006480 con 0x7f411012ec10 2021-05-25T12:05:35.624 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.589+0000 7f40fe7fc700 1 -- 172.21.15.102:0/3706852240 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 13274092 0 0) 0x7f41100676e0 con 0x7f411012b100 2021-05-25T12:05:35.624 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.589+0000 7f40fe7fc700 1 -- 172.21.15.102:0/3706852240 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f40ec004c10 con 0x7f411012b100 2021-05-25T12:05:35.624 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.589+0000 7f40fe7fc700 1 -- 172.21.15.102:0/3706852240 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f4100001c40 con 0x7f411012ec10 2021-05-25T12:05:35.625 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.589+0000 7f40fe7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:35.625 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.589+0000 7f40fe7fc700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:05:35.625 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.589+0000 7f40fe7fc700 10 monclient(hunting): dump: 2021-05-25T12:05:35.625 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:35.626 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:35.626 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:35.626 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:35.626 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:35.627 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:35.627 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:35.627 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:35.627 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:35.627 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:35.628 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.590+0000 7f40fe7fc700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:05:35.628 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.590+0000 7f40fe7fc700 1 -- 172.21.15.102:0/3706852240 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f40f8001830 con 0x7f411012b100 2021-05-25T12:05:35.628 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.590+0000 7f40fe7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:35.628 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.590+0000 7f40fe7fc700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2021-05-25T12:05:35.628 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.590+0000 7f40fe7fc700 10 monclient(hunting): dump: 2021-05-25T12:05:35.629 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:35.629 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:35.629 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:35.629 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:35.630 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:35.630 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:35.630 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:35.630 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:35.630 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:35.631 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:35.631 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.590+0000 7f40fe7fc700 1 -- 172.21.15.102:0/3706852240 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2919252616 0 0) 0x7f4110093310 con 0x7f4110069620 2021-05-25T12:05:35.631 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.590+0000 7f40fe7fc700 1 -- 172.21.15.102:0/3706852240 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f41100676e0 con 0x7f4110069620 2021-05-25T12:05:35.631 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.590+0000 7f40fe7fc700 1 -- 172.21.15.102:0/3706852240 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f4104001510 con 0x7f4110069620 2021-05-25T12:05:35.631 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.590+0000 7f40fe7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:35.632 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.590+0000 7f40fe7fc700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2021-05-25T12:05:35.632 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.590+0000 7f40fe7fc700 10 monclient(hunting): dump: 2021-05-25T12:05:35.632 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:35.632 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:35.633 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:35.633 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:35.633 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:35.633 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:35.633 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:35.634 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:35.634 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:35.634 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:35.634 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.590+0000 7f40fe7fc700 1 -- 172.21.15.102:0/3706852240 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1193280168 0 0) 0x7f40ec004c10 con 0x7f411012b100 2021-05-25T12:05:35.634 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.590+0000 7f40fe7fc700 1 -- 172.21.15.102:0/3706852240 >> v1:172.21.15.102:6790/0 conn(0x7f411012ec10 legacy=0x7f41101474b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:35.635 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.590+0000 7f40fe7fc700 1 -- 172.21.15.102:0/3706852240 >> v1:172.21.15.181:6789/0 conn(0x7f4110069620 legacy=0x7f411013f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:35.635 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.590+0000 7f40fe7fc700 10 monclient: _finish_hunting 0 2021-05-25T12:05:35.635 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.590+0000 7f40fe7fc700 1 monclient: found mon.a 2021-05-25T12:05:35.635 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.590+0000 7f40fe7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:05:35.636 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.590+0000 7f40fe7fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:05:35.636 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.590+0000 7f40fe7fc700 1 -- 172.21.15.102:0/3706852240 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4110067980 con 0x7f411012b100 2021-05-25T12:05:35.636 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.590+0000 7f40fe7fc700 10 monclient: _finish_auth 0 2021-05-25T12:05:35.636 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.590+0000 7f40fe7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:05:35.636 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.590+0000 7f4114ae3700 5 monclient: authenticate success, global_id 174606 2021-05-25T12:05:35.637 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.590+0000 7f4114ae3700 10 monclient: _renew_subs 2021-05-25T12:05:35.637 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.590+0000 7f4114ae3700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:05:35.637 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.590+0000 7f4114ae3700 1 -- 172.21.15.102:0/3706852240 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4110065810 con 0x7f411012b100 2021-05-25T12:05:35.637 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.590+0000 7f4114ae3700 10 client.174606.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:05:35.638 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.590+0000 7f4114ae3700 10 monclient: _renew_subs 2021-05-25T12:05:35.638 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.590+0000 7f4114ae3700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:05:35.638 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.590+0000 7f4114ae3700 1 -- 172.21.15.102:0/3706852240 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f411013fea0 con 0x7f411012b100 2021-05-25T12:05:35.638 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.590+0000 7f4114ae3700 1 librados: init done 2021-05-25T12:05:35.638 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.590+0000 7f40fe7fc700 1 -- 172.21.15.102:0/3706852240 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f40f80019d0 con 0x7f411012b100 2021-05-25T12:05:35.639 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.590+0000 7f40fe7fc700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:05:35.639 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.590+0000 7f40fe7fc700 1 -- 172.21.15.102:0/3706852240 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f40f8001d60 con 0x7f411012b100 2021-05-25T12:05:35.639 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.590+0000 7f40fe7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:35.639 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.590+0000 7f40fe7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:05:35.639 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.590+0000 7f410cd99700 4 set_mon_vals no callback set 2021-05-25T12:05:35.640 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.590+0000 7f40fe7fc700 10 monclient: dump: 2021-05-25T12:05:35.640 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:35.640 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:35.640 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:35.641 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:35.641 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:35.641 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:35.641 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:35.641 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:35.642 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:35.642 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:35.642 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.590+0000 7f40fe7fc700 1 -- 172.21.15.102:0/3706852240 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f40f8001ec0 con 0x7f411012b100 2021-05-25T12:05:35.642 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.591+0000 7f40fe7fc700 1 -- 172.21.15.102:0/3706852240 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(167..167 src has 1..167) v4 ==== 7701+0+0 (unknown 260224446 0 0) 0x7f40f803dbb0 con 0x7f411012b100 2021-05-25T12:05:35.643 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.591+0000 7f40fe7fc700 10 client.174606.objecter ms_dispatch 0x7f4110001550 osd_map(167..167 src has 1..167) v4 2021-05-25T12:05:35.643 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.591+0000 7f40fe7fc700 3 client.174606.objecter handle_osd_map got epochs [167,167] > 0 2021-05-25T12:05:35.643 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.591+0000 7f40fe7fc700 3 client.174606.objecter handle_osd_map decoding full epoch 167 2021-05-25T12:05:35.643 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.591+0000 7f40fe7fc700 20 client.174606.objecter dump_active .. 0 homeless 2021-05-25T12:05:35.644 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.591+0000 7f410e59c700 10 monclient: get_auth_request con 0x7f40ec033790 auth_method 0 2021-05-25T12:05:35.644 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.591+0000 7f40fe7fc700 10 client.174606.objecter ms_handle_connect 0x7f40ec033790 2021-05-25T12:05:35.644 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.591+0000 7f4114ae3700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:05:35.644 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.591+0000 7f4114ae3700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:05:35.644 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.591+0000 7f4114ae3700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:05:35.645 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.591+0000 7f4114ae3700 1 -- 172.21.15.102:0/3706852240 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40f0004db0 con 0x7f411012b100 2021-05-25T12:05:35.645 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.596+0000 7f40fe7fc700 1 -- 172.21.15.102:0/3706852240 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7f40f8002650 con 0x7f411012b100 2021-05-25T12:05:35.646 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.596+0000 7f40fe7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:05:35.646 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.596+0000 7f40fe7fc700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:05:35.815 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.813+0000 7f4114ae3700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2021-05-25T12:05:35.815 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.813+0000 7f4114ae3700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2021-05-25T12:05:35.815 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.813+0000 7f4114ae3700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:05:35.816 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.813+0000 7f4114ae3700 1 -- 172.21.15.102:0/3706852240 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f40f0005060 con 0x7f411012b100 2021-05-25T12:05:35.816 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.813+0000 7f40fe7fc700 1 -- 172.21.15.102:0/3706852240 <== mon.0 v1:172.21.15.102:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1342 (unknown 254746657 0 606658015) 0x7f40f8012a70 con 0x7f411012b100 2021-05-25T12:05:35.816 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.813+0000 7f40fe7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2021-05-25T12:05:35.816 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.813+0000 7f40fe7fc700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:05:35.817 INFO:teuthology.orchestra.run.smithi102.stdout:{"election_epoch":46,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":3,"features":{"quorum_con":"4540138303579357183","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"20ee91b6-8935-4b98-b868-85c03789ccc4","modified":"2021-05-25T11:59:11.011522Z","created":"2021-05-25T11:59:11.011522Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6790","nonce":0}]},"addr":"172.21.15.102:6790/0","public_addr":"172.21.15.102:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2021-05-25T12:05:35.818 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.814+0000 7f4114ae3700 10 librados: watch_flush enter 2021-05-25T12:05:35.818 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.814+0000 7f4114ae3700 10 librados: watch_flush exit 2021-05-25T12:05:35.819 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.814+0000 7f4114ae3700 20 client.174606.objecter shutdown clearing up homeless session... 2021-05-25T12:05:35.819 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.814+0000 7f4114ae3700 10 client.174606.objecter successfully canceled tick 2021-05-25T12:05:35.819 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.815+0000 7f4114ae3700 1 -- 172.21.15.102:0/3706852240 >> v1:172.21.15.181:6812/29582 conn(0x7f40ec033790 legacy=0x7f40ec035c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:35.819 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.815+0000 7f4114ae3700 10 monclient: shutdown 2021-05-25T12:05:35.820 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.815+0000 7f4114ae3700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:05:35.820 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.815+0000 7f4114ae3700 1 -- 172.21.15.102:0/3706852240 >> v1:172.21.15.102:6789/0 conn(0x7f411012b100 legacy=0x7f411012a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:35.820 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.815+0000 7f4114ae3700 1 -- 172.21.15.102:0/3706852240 shutdown_connections 2021-05-25T12:05:35.820 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.815+0000 7f4114ae3700 1 -- 172.21.15.102:0/3706852240 >> 172.21.15.102:0/3706852240 conn(0x7f411009a910 msgr2=0x7f41100956a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:05:35.820 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.815+0000 7f4114ae3700 1 -- 172.21.15.102:0/3706852240 shutdown_connections 2021-05-25T12:05:35.821 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.815+0000 7f4114ae3700 1 -- 172.21.15.102:0/3706852240 wait complete. 2021-05-25T12:05:35.821 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.815+0000 7f4114ae3700 1 librados: shutdown 2021-05-25T12:05:35.827 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2021-05-25T12:05:35.827 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2021-05-25T12:05:35.942 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.941+0000 7f340aa5c700 10 monclient: get_monmap_and_config 2021-05-25T12:05:35.943 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.941+0000 7f340aa5c700 10 monclient: build_initial_monmap 2021-05-25T12:05:35.943 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.941+0000 7f340aa5c700 10 monclient: monmap: 2021-05-25T12:05:35.943 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:05:35.943 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:35.944 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:05:35.942151+0000 2021-05-25T12:05:35.944 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:05:35.942151+0000 2021-05-25T12:05:35.944 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:05:35.944 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:35.945 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:05:35.945 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:05:35.945 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:05:35.945 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:35.958 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.941+0000 7f340aa5c700 1 Processor -- start 2021-05-25T12:05:35.959 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.942+0000 7f340aa5c700 1 -- start start 2021-05-25T12:05:35.959 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.942+0000 7f340aa5c700 10 monclient: init 2021-05-25T12:05:35.959 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.942+0000 7f340aa5c700 10 monclient: _reopen_session rank -1 2021-05-25T12:05:35.959 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.942+0000 7f340aa5c700 10 monclient: _add_conns ranks=[2,0,1] 2021-05-25T12:05:35.960 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.942+0000 7f340aa5c700 10 monclient(hunting): picked mon.noname-b con 0x7f3404090990 addr v1:172.21.15.102:6790/0 2021-05-25T12:05:35.960 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.942+0000 7f340aa5c700 10 monclient(hunting): picked mon.noname-a con 0x7f3404090d70 addr v1:172.21.15.102:6789/0 2021-05-25T12:05:35.960 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.942+0000 7f340aa5c700 10 monclient(hunting): picked mon.noname-c con 0x7f340412e330 addr v1:172.21.15.181:6789/0 2021-05-25T12:05:35.960 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.942+0000 7f340aa5c700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3404096fa0 con 0x7f3404090d70 2021-05-25T12:05:35.960 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.942+0000 7f340aa5c700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34040926a0 con 0x7f340412e330 2021-05-25T12:05:35.961 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.942+0000 7f340aa5c700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3404092940 con 0x7f3404090990 2021-05-25T12:05:35.961 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.942+0000 7f340aa5c700 10 monclient(hunting): _renew_subs 2021-05-25T12:05:35.961 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.942+0000 7f340aa5c700 10 monclient(hunting): authenticate will time out at 1136.539507s 2021-05-25T12:05:35.961 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.942+0000 7f34037fe700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f3404090d70 0x7f3404091200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:32866/0 (socket says 172.21.15.102:32866) 2021-05-25T12:05:35.962 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.942+0000 7f34037fe700 1 -- 172.21.15.102:0/3339866408 learned_addr learned my addr 172.21.15.102:0/3339866408 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:05:35.962 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.942+0000 7f3403fff700 1 --1- 172.21.15.102:0/3339866408 >> v1:172.21.15.102:6790/0 conn(0x7f3404090990 0x7f3404093240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:55974/0 (socket says 172.21.15.102:55974) 2021-05-25T12:05:35.962 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.942+0000 7f3402ffd700 1 -- 172.21.15.102:0/3339866408 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1827453509 0 0) 0x7f3404092940 con 0x7f3404090990 2021-05-25T12:05:35.962 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.942+0000 7f3402ffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:35.962 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.942+0000 7f3402ffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:35.963 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.942+0000 7f3402ffd700 10 monclient(hunting): my global_id is 184316 2021-05-25T12:05:35.963 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.942+0000 7f3402ffd700 1 -- 172.21.15.102:0/3339866408 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33e8003270 con 0x7f3404090990 2021-05-25T12:05:35.963 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.942+0000 7f3402ffd700 1 -- 172.21.15.102:0/3339866408 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1042335951 0 0) 0x7f3404096fa0 con 0x7f3404090d70 2021-05-25T12:05:35.963 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.942+0000 7f3402ffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:35.964 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.942+0000 7f3402ffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:35.964 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.942+0000 7f3402ffd700 10 monclient(hunting): my global_id is 174609 2021-05-25T12:05:35.964 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.942+0000 7f3402ffd700 1 -- 172.21.15.102:0/3339866408 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3404092940 con 0x7f3404090d70 2021-05-25T12:05:35.964 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.943+0000 7f3402ffd700 1 -- 172.21.15.102:0/3339866408 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3240203977 0 0) 0x7f34040926a0 con 0x7f340412e330 2021-05-25T12:05:35.964 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.943+0000 7f3402ffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:35.965 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.943+0000 7f3402ffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:35.965 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.943+0000 7f3402ffd700 10 monclient(hunting): my global_id is 194122 2021-05-25T12:05:35.965 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.943+0000 7f3402ffd700 1 -- 172.21.15.102:0/3339866408 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3404096fa0 con 0x7f340412e330 2021-05-25T12:05:35.965 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.943+0000 7f3402ffd700 1 -- 172.21.15.102:0/3339866408 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1816582308 0 0) 0x7f33e8003270 con 0x7f3404090990 2021-05-25T12:05:35.966 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.943+0000 7f3402ffd700 1 -- 172.21.15.102:0/3339866408 >> v1:172.21.15.181:6789/0 conn(0x7f340412e330 legacy=0x7f3404130720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:35.966 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.943+0000 7f3402ffd700 1 -- 172.21.15.102:0/3339866408 >> v1:172.21.15.102:6789/0 conn(0x7f3404090d70 legacy=0x7f3404091200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:35.966 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.943+0000 7f3402ffd700 10 monclient: _finish_hunting 0 2021-05-25T12:05:35.966 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.943+0000 7f3402ffd700 1 monclient: found mon.noname-b 2021-05-25T12:05:35.966 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.943+0000 7f3402ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:05:35.967 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.943+0000 7f3402ffd700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T12:05:35.967 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.943+0000 7f3402ffd700 1 -- 172.21.15.102:0/3339866408 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3404092be0 con 0x7f3404090990 2021-05-25T12:05:35.967 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.943+0000 7f3402ffd700 10 monclient: _finish_auth 0 2021-05-25T12:05:35.967 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.943+0000 7f3402ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:05:35.967 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.943+0000 7f3402ffd700 1 -- 172.21.15.102:0/3339866408 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f33f4001610 con 0x7f3404090990 2021-05-25T12:05:35.968 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.943+0000 7f3402ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:35.968 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.943+0000 7f3402ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:05:35.968 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.943+0000 7f3402ffd700 10 monclient: dump: 2021-05-25T12:05:35.968 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:35.969 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:35.969 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:35.969 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:35.969 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:35.970 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:35.970 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:35.970 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:35.970 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:35.971 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:35.971 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.943+0000 7f340aa5c700 5 monclient: authenticate success, global_id 184316 2021-05-25T12:05:35.971 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.943+0000 7f340aa5c700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:05:35.971 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.943+0000 7f3402ffd700 1 -- 172.21.15.102:0/3339866408 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f33f4001800 con 0x7f3404090990 2021-05-25T12:05:35.971 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.943+0000 7f3402ffd700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:05:35.972 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.943+0000 7f340aa5c700 10 monclient: get_monmap_and_config success 2021-05-25T12:05:35.972 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.943+0000 7f3402ffd700 1 -- 172.21.15.102:0/3339866408 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f33f4001b90 con 0x7f3404090990 2021-05-25T12:05:35.972 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.943+0000 7f340aa5c700 4 set_mon_vals no callback set 2021-05-25T12:05:35.972 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.943+0000 7f3402ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:35.972 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.943+0000 7f3402ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:05:35.973 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.943+0000 7f3402ffd700 10 monclient: dump: 2021-05-25T12:05:35.973 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:35.973 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:35.973 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:35.974 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:35.974 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:35.974 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:35.974 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:35.974 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:35.975 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:35.975 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:35.975 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.943+0000 7f340aa5c700 10 monclient: shutdown 2021-05-25T12:05:35.975 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.943+0000 7f340aa5c700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:05:35.975 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.943+0000 7f340aa5c700 1 -- 172.21.15.102:0/3339866408 >> v1:172.21.15.102:6790/0 conn(0x7f3404090990 legacy=0x7f3404093240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:35.976 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.943+0000 7f340aa5c700 1 -- 172.21.15.102:0/3339866408 shutdown_connections 2021-05-25T12:05:35.976 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.943+0000 7f340aa5c700 1 -- 172.21.15.102:0/3339866408 >> 172.21.15.102:0/3339866408 conn(0x7f340409b370 msgr2=0x7f3404097430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:05:35.976 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.943+0000 7f340aa5c700 1 -- 172.21.15.102:0/3339866408 shutdown_connections 2021-05-25T12:05:35.976 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.943+0000 7f340aa5c700 1 -- 172.21.15.102:0/3339866408 wait complete. 2021-05-25T12:05:35.977 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.944+0000 7f340aa5c700 10 monclient: build_initial_monmap 2021-05-25T12:05:35.977 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.944+0000 7f340aa5c700 10 monclient: monmap: 2021-05-25T12:05:35.977 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:05:35.977 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:05:35.977 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:05:35.978 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:05:35.978 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:05:35.978 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:05:35.978 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:05:35.982 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:05:35.982 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:05:35.982 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:35.983 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.944+0000 7f340aa5c700 1 librados: starting msgr at 2021-05-25T12:05:35.983 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.944+0000 7f340aa5c700 1 librados: starting objecter 2021-05-25T12:05:35.983 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.944+0000 7f340aa5c700 1 Processor -- start 2021-05-25T12:05:35.983 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.944+0000 7f340aa5c700 1 -- start start 2021-05-25T12:05:35.984 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.944+0000 7f340aa5c700 1 librados: setting wanted keys 2021-05-25T12:05:35.984 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.944+0000 7f340aa5c700 1 librados: calling monclient init 2021-05-25T12:05:35.984 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.944+0000 7f340aa5c700 10 monclient: init 2021-05-25T12:05:35.984 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.945+0000 7f340aa5c700 10 monclient: _reopen_session rank -1 2021-05-25T12:05:35.984 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.945+0000 7f340aa5c700 10 monclient: _add_conns ranks=[0,1,2] 2021-05-25T12:05:35.985 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.945+0000 7f340aa5c700 10 monclient(hunting): picked mon.noname-a con 0x7f3404090990 addr v1:172.21.15.102:6789/0 2021-05-25T12:05:35.985 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.945+0000 7f340aa5c700 10 monclient(hunting): picked mon.noname-b con 0x7f3404090d70 addr v1:172.21.15.181:6789/0 2021-05-25T12:05:35.985 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.945+0000 7f340aa5c700 10 monclient(hunting): picked mon.noname-c con 0x7f340412e330 addr v1:172.21.15.102:6790/0 2021-05-25T12:05:35.985 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.945+0000 7f340aa5c700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f340412a100 con 0x7f3404090990 2021-05-25T12:05:35.985 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.945+0000 7f340aa5c700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3404125be0 con 0x7f3404090d70 2021-05-25T12:05:35.986 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.945+0000 7f340aa5c700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3404111ca0 con 0x7f340412e330 2021-05-25T12:05:35.986 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.945+0000 7f340aa5c700 10 monclient(hunting): _renew_subs 2021-05-25T12:05:35.986 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.945+0000 7f340aa5c700 10 monclient(hunting): authenticate will time out at 1136.542281s 2021-05-25T12:05:35.986 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.945+0000 7f3403fff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f3404090990 0x7f34041254d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:32870/0 (socket says 172.21.15.102:32870) 2021-05-25T12:05:35.986 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.945+0000 7f3403fff700 1 -- 172.21.15.102:0/510981782 learned_addr learned my addr 172.21.15.102:0/510981782 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:05:35.987 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.945+0000 7f33effff700 10 client.?.objecter ms_handle_connect 0x7f3404090990 2021-05-25T12:05:35.987 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.945+0000 7f33effff700 10 client.?.objecter resend_mon_ops 2021-05-25T12:05:35.987 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.945+0000 7f33effff700 10 client.?.objecter ms_handle_connect 0x7f340412e330 2021-05-25T12:05:35.987 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.945+0000 7f33effff700 10 client.?.objecter resend_mon_ops 2021-05-25T12:05:35.988 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.945+0000 7f33effff700 10 client.?.objecter ms_handle_connect 0x7f3404090d70 2021-05-25T12:05:35.988 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.945+0000 7f33effff700 10 client.?.objecter resend_mon_ops 2021-05-25T12:05:35.988 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.945+0000 7f33effff700 1 -- 172.21.15.102:0/510981782 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1288328967 0 0) 0x7f340412a100 con 0x7f3404090990 2021-05-25T12:05:35.988 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.945+0000 7f33effff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:35.988 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.945+0000 7f33effff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:35.989 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.945+0000 7f33effff700 10 monclient(hunting): my global_id is 174612 2021-05-25T12:05:35.989 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.945+0000 7f33effff700 1 -- 172.21.15.102:0/510981782 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33e4004c10 con 0x7f3404090990 2021-05-25T12:05:35.989 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.945+0000 7f33effff700 1 -- 172.21.15.102:0/510981782 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 806758431 0 0) 0x7f3404111ca0 con 0x7f340412e330 2021-05-25T12:05:35.989 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.945+0000 7f33effff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:35.989 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.945+0000 7f33effff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:35.990 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.945+0000 7f33effff700 10 monclient(hunting): my global_id is 184319 2021-05-25T12:05:35.990 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.945+0000 7f33effff700 1 -- 172.21.15.102:0/510981782 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f340412a100 con 0x7f340412e330 2021-05-25T12:05:35.990 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.945+0000 7f33effff700 1 -- 172.21.15.102:0/510981782 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 645358966 0 0) 0x7f3404125be0 con 0x7f3404090d70 2021-05-25T12:05:35.990 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.945+0000 7f33effff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:35.991 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.945+0000 7f33effff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:35.991 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.945+0000 7f33effff700 10 monclient(hunting): my global_id is 194125 2021-05-25T12:05:35.991 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.945+0000 7f33effff700 1 -- 172.21.15.102:0/510981782 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3404111ca0 con 0x7f3404090d70 2021-05-25T12:05:35.991 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.945+0000 7f33effff700 1 -- 172.21.15.102:0/510981782 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2025448598 0 0) 0x7f33e4004c10 con 0x7f3404090990 2021-05-25T12:05:35.991 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.945+0000 7f33effff700 1 -- 172.21.15.102:0/510981782 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3404125be0 con 0x7f3404090990 2021-05-25T12:05:35.992 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.945+0000 7f33effff700 1 -- 172.21.15.102:0/510981782 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f33f4001b50 con 0x7f3404090990 2021-05-25T12:05:35.992 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.945+0000 7f33effff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:35.992 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.945+0000 7f33effff700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:05:35.992 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.945+0000 7f33effff700 10 monclient(hunting): dump: 2021-05-25T12:05:35.993 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:35.993 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:35.993 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:35.993 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:35.993 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:35.994 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:35.994 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:35.994 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:35.994 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:35.995 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:35.995 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.945+0000 7f33effff700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:05:35.995 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.945+0000 7f33effff700 1 -- 172.21.15.102:0/510981782 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3838735501 0 0) 0x7f3404111ca0 con 0x7f3404090d70 2021-05-25T12:05:35.995 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.945+0000 7f33effff700 1 -- 172.21.15.102:0/510981782 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f33e4004c10 con 0x7f3404090d70 2021-05-25T12:05:35.995 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.945+0000 7f33effff700 1 -- 172.21.15.102:0/510981782 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f33f00018d0 con 0x7f3404090d70 2021-05-25T12:05:35.996 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.945+0000 7f33effff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:35.996 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.945+0000 7f33effff700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2021-05-25T12:05:35.996 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.945+0000 7f33effff700 10 monclient(hunting): dump: 2021-05-25T12:05:35.996 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:35.996 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:35.997 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:35.997 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:35.997 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:35.997 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:35.998 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:35.998 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:35.998 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:35.998 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:35.998 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.945+0000 7f33effff700 1 -- 172.21.15.102:0/510981782 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2990931584 0 0) 0x7f340412a100 con 0x7f340412e330 2021-05-25T12:05:35.999 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.945+0000 7f33effff700 1 -- 172.21.15.102:0/510981782 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3404111ca0 con 0x7f340412e330 2021-05-25T12:05:35.999 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.945+0000 7f33effff700 1 -- 172.21.15.102:0/510981782 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f33f8001b80 con 0x7f340412e330 2021-05-25T12:05:35.999 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.945+0000 7f33effff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:35.999 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.945+0000 7f33effff700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2021-05-25T12:05:36.000 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.945+0000 7f33effff700 10 monclient(hunting): dump: 2021-05-25T12:05:36.000 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:36.000 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:36.000 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:36.000 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:36.001 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:36.001 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:36.001 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:36.001 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:36.001 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:36.002 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:36.002 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.945+0000 7f33effff700 1 -- 172.21.15.102:0/510981782 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2133918830 0 0) 0x7f3404125be0 con 0x7f3404090990 2021-05-25T12:05:36.002 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.945+0000 7f33effff700 1 -- 172.21.15.102:0/510981782 >> v1:172.21.15.102:6790/0 conn(0x7f340412e330 legacy=0x7f3404141670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:36.002 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.945+0000 7f33effff700 1 -- 172.21.15.102:0/510981782 >> v1:172.21.15.181:6789/0 conn(0x7f3404090d70 legacy=0x7f340413df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:36.003 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.945+0000 7f33effff700 10 monclient: _finish_hunting 0 2021-05-25T12:05:36.003 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.945+0000 7f33effff700 1 monclient: found mon.a 2021-05-25T12:05:36.003 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.945+0000 7f33effff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:05:36.003 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.945+0000 7f33effff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:05:36.003 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.945+0000 7f33effff700 1 -- 172.21.15.102:0/510981782 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3404111f40 con 0x7f3404090990 2021-05-25T12:05:36.004 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.945+0000 7f33effff700 10 monclient: _finish_auth 0 2021-05-25T12:05:36.004 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.945+0000 7f33effff700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:05:36.004 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.945+0000 7f340aa5c700 5 monclient: authenticate success, global_id 174612 2021-05-25T12:05:36.004 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.945+0000 7f340aa5c700 10 monclient: _renew_subs 2021-05-25T12:05:36.005 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.945+0000 7f340aa5c700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:05:36.005 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.945+0000 7f340aa5c700 1 -- 172.21.15.102:0/510981782 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34040689e0 con 0x7f3404090990 2021-05-25T12:05:36.005 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.946+0000 7f340aa5c700 10 client.174612.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:05:36.005 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.946+0000 7f340aa5c700 10 monclient: _renew_subs 2021-05-25T12:05:36.005 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.946+0000 7f340aa5c700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:05:36.006 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.946+0000 7f340aa5c700 1 -- 172.21.15.102:0/510981782 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3404139890 con 0x7f3404090990 2021-05-25T12:05:36.006 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.946+0000 7f340aa5c700 1 librados: init done 2021-05-25T12:05:36.006 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.946+0000 7f33effff700 1 -- 172.21.15.102:0/510981782 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f33f40048a0 con 0x7f3404090990 2021-05-25T12:05:36.006 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.946+0000 7f33effff700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:05:36.006 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.946+0000 7f33effff700 1 -- 172.21.15.102:0/510981782 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f33f4004c30 con 0x7f3404090990 2021-05-25T12:05:36.007 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.946+0000 7f33effff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:36.007 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.946+0000 7f33effff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:05:36.007 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.946+0000 7f34027fc700 4 set_mon_vals no callback set 2021-05-25T12:05:36.007 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.946+0000 7f33effff700 10 monclient: dump: 2021-05-25T12:05:36.008 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:36.008 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:36.008 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:36.008 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:36.008 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:36.009 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:36.009 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:36.009 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:36.009 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:36.010 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:36.010 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.946+0000 7f340aa5c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:05:36.010 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.946+0000 7f340aa5c700 1 -- 172.21.15.102:0/510981782 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f33dc0065a0 con 0x7f33dc002840 2021-05-25T12:05:36.010 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.946+0000 7f33effff700 1 -- 172.21.15.102:0/510981782 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f33f4004e10 con 0x7f3404090990 2021-05-25T12:05:36.010 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.946+0000 7f33effff700 10 client.174612.objecter ms_handle_connect 0x7f33dc002840 2021-05-25T12:05:36.011 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.946+0000 7f33effff700 10 client.174612.objecter resend_mon_ops 2021-05-25T12:05:36.011 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.946+0000 7f33effff700 1 -- 172.21.15.102:0/510981782 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(168..168 src has 1..168) v4 ==== 7138+0+0 (unknown 3442091228 0 0) 0x7f33f403de30 con 0x7f3404090990 2021-05-25T12:05:36.011 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.946+0000 7f33effff700 10 client.174612.objecter ms_dispatch 0x7f3404001550 osd_map(168..168 src has 1..168) v4 2021-05-25T12:05:36.011 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.946+0000 7f33effff700 3 client.174612.objecter handle_osd_map got epochs [168,168] > 0 2021-05-25T12:05:36.012 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.946+0000 7f33effff700 3 client.174612.objecter handle_osd_map decoding full epoch 168 2021-05-25T12:05:36.012 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.947+0000 7f33effff700 20 client.174612.objecter dump_active .. 0 homeless 2021-05-25T12:05:36.012 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.947+0000 7f33effff700 1 -- 172.21.15.102:0/510981782 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1415968245 0 0) 0x7f33dc0065a0 con 0x7f33dc002840 2021-05-25T12:05:36.012 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.947+0000 7f33effff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:36.013 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.947+0000 7f33effff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:36.013 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.947+0000 7f33effff700 10 monclient(hunting): my global_id is 174615 2021-05-25T12:05:36.013 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.947+0000 7f33effff700 1 -- 172.21.15.102:0/510981782 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3404125be0 con 0x7f33dc002840 2021-05-25T12:05:36.013 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.947+0000 7f3408ff9700 10 monclient: get_auth_request con 0x7f33e40335a0 auth_method 0 2021-05-25T12:05:36.013 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.947+0000 7f33effff700 1 -- 172.21.15.102:0/510981782 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 232882312 0 0) 0x7f3404125be0 con 0x7f33dc002840 2021-05-25T12:05:36.014 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.947+0000 7f33effff700 1 -- 172.21.15.102:0/510981782 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f33dc0065a0 con 0x7f33dc002840 2021-05-25T12:05:36.014 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.947+0000 7f33effff700 1 -- 172.21.15.102:0/510981782 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f33f00046d0 con 0x7f33dc002840 2021-05-25T12:05:36.014 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.947+0000 7f33effff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:36.014 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.947+0000 7f33effff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:05:36.014 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.947+0000 7f33effff700 10 monclient: dump: 2021-05-25T12:05:36.015 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:36.015 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:36.015 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:36.015 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:36.016 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:36.016 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:36.016 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:36.016 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:36.016 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:36.017 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:36.017 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.947+0000 7f33effff700 10 client.174612.objecter ms_handle_connect 0x7f33e40335a0 2021-05-25T12:05:36.017 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.947+0000 7f33effff700 1 -- 172.21.15.102:0/510981782 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2394210726 0 0) 0x7f33dc0065a0 con 0x7f33dc002840 2021-05-25T12:05:36.017 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.947+0000 7f33effff700 1 -- 172.21.15.102:0/510981782 --> v1:172.21.15.102:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f33dc009e40 con 0x7f33dc002840 2021-05-25T12:05:36.017 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.947+0000 7f33effff700 1 -- 172.21.15.102:0/510981782 <== mon.0 v1:172.21.15.102:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7f33f0006a10 con 0x7f33dc002840 2021-05-25T12:05:36.018 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.947+0000 7f33effff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:05:36.018 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.947+0000 7f33effff700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:05:36.018 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.947+0000 7f33effff700 1 -- 172.21.15.102:0/510981782 >> v1:172.21.15.102:6789/0 conn(0x7f33dc002840 legacy=0x7f33dc004ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:36.018 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.947+0000 7f33effff700 1 -- 172.21.15.102:0/510981782 >> v1:172.21.15.102:6789/0 conn(0x7f33dc002840 legacy=0x7f33dc004ce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:05:36.018 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.954+0000 7f340aa5c700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:05:36.019 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.954+0000 7f340aa5c700 1 -- 172.21.15.102:0/510981782 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f33dc007510 con 0x7f33dc0068b0 2021-05-25T12:05:36.019 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.954+0000 7f33effff700 10 client.174612.objecter ms_handle_connect 0x7f33dc0068b0 2021-05-25T12:05:36.019 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.954+0000 7f33effff700 10 client.174612.objecter resend_mon_ops 2021-05-25T12:05:36.019 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.954+0000 7f33effff700 1 -- 172.21.15.102:0/510981782 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 693311690 0 0) 0x7f33dc007510 con 0x7f33dc0068b0 2021-05-25T12:05:36.020 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.954+0000 7f33effff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:36.020 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.954+0000 7f33effff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:36.020 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.954+0000 7f33effff700 10 monclient(hunting): my global_id is 174618 2021-05-25T12:05:36.020 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.954+0000 7f33effff700 1 -- 172.21.15.102:0/510981782 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33dc0065a0 con 0x7f33dc0068b0 2021-05-25T12:05:36.020 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.954+0000 7f33effff700 1 -- 172.21.15.102:0/510981782 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2742821793 0 0) 0x7f33dc0065a0 con 0x7f33dc0068b0 2021-05-25T12:05:36.021 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.954+0000 7f33effff700 1 -- 172.21.15.102:0/510981782 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f33dc007510 con 0x7f33dc0068b0 2021-05-25T12:05:36.021 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.954+0000 7f33effff700 1 -- 172.21.15.102:0/510981782 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f33f0004e70 con 0x7f33dc0068b0 2021-05-25T12:05:36.021 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.954+0000 7f33effff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:36.021 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.954+0000 7f33effff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:05:36.022 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.954+0000 7f33effff700 10 monclient: dump: 2021-05-25T12:05:36.022 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:36.022 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:36.022 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:36.022 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:36.023 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:36.023 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:36.023 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:36.023 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:36.024 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:36.024 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:36.024 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.954+0000 7f33effff700 1 -- 172.21.15.102:0/510981782 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4119865881 0 0) 0x7f33dc007510 con 0x7f33dc0068b0 2021-05-25T12:05:36.024 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.954+0000 7f33effff700 1 -- 172.21.15.102:0/510981782 --> v1:172.21.15.102:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f33dc008770 con 0x7f33dc0068b0 2021-05-25T12:05:36.024 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.955+0000 7f33effff700 1 -- 172.21.15.102:0/510981782 <== mon.0 v1:172.21.15.102:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3940 (unknown 0 0 2627930286) 0x7f33f00060e0 con 0x7f33dc0068b0 2021-05-25T12:05:36.025 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.955+0000 7f33effff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:05:36.025 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.955+0000 7f33effff700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:05:36.025 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.955+0000 7f33effff700 1 -- 172.21.15.102:0/510981782 >> v1:172.21.15.102:6789/0 conn(0x7f33dc0068b0 legacy=0x7f33dc006d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:36.025 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.955+0000 7f33effff700 1 -- 172.21.15.102:0/510981782 >> v1:172.21.15.102:6789/0 conn(0x7f33dc0068b0 legacy=0x7f33dc006d40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:05:36.026 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:05:36.026 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:05:36.026 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:05:36.026 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "peon", 2021-05-25T12:05:36.027 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 46, 2021-05-25T12:05:36.027 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:05:36.027 INFO:teuthology.orchestra.run.smithi102.stdout: 0, 2021-05-25T12:05:36.027 INFO:teuthology.orchestra.run.smithi102.stdout: 1, 2021-05-25T12:05:36.027 INFO:teuthology.orchestra.run.smithi102.stdout: 2 2021-05-25T12:05:36.028 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:05:36.028 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 3, 2021-05-25T12:05:36.028 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:05:36.028 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:05:36.028 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:05:36.029 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:05:36.029 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:05:36.029 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:05:36.029 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:05:36.030 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:05:36.030 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:05:36.030 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:05:36.030 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:05:36.031 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:05:36.031 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:05:36.031 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:05:36.031 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:05:36.031 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:05:36.032 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:05:36.032 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:05:36.032 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:05:36.032 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:05:36.032 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:05:36.033 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:05:36.033 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:05:36.033 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:05:36.033 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:05:36.033 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:05:36.034 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:05:36.034 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:05:36.034 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:05:36.034 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:05:36.034 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:05:36.035 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:05:36.035 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:05:36.035 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:05:36.036 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:05:36.036 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:05:36.036 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:05:36.036 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:05:36.037 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:05:36.037 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:05:36.037 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:05:36.037 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:05:36.037 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:05:36.038 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:05:36.038 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:05:36.038 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:05:36.038 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:05:36.038 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:05:36.039 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:05:36.039 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:05:36.039 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:05:36.039 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:05:36.040 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:05:36.040 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:05:36.040 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:05:36.040 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:05:36.040 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:05:36.041 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:05:36.041 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:05:36.041 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:05:36.041 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:05:36.041 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:05:36.042 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:05:36.042 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:05:36.042 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:05:36.042 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:05:36.042 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:05:36.043 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:05:36.043 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:05:36.043 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:05:36.043 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:05:36.043 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:05:36.044 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:05:36.044 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:05:36.044 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:05:36.044 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:05:36.044 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:05:36.045 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:05:36.045 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:05:36.045 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:05:36.045 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:05:36.045 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:05:36.046 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:05:36.046 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:05:36.046 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:05:36.046 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:05:36.046 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:05:36.047 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:05:36.047 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:05:36.047 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:05:36.047 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:05:36.047 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:05:36.048 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:05:36.048 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:05:36.048 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:05:36.048 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:05:36.049 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:05:36.049 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:05:36.049 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:05:36.049 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:05:36.049 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:05:36.050 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:05:36.050 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:05:36.050 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:05:36.050 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:05:36.051 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:05:36.051 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:05:36.051 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:05:36.051 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:05:36.051 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:05:36.052 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:05:36.052 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:05:36.052 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:05:36.052 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:05:36.052 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:05:36.053 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:05:36.053 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:05:36.053 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:05:36.053 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": [ 2021-05-25T12:05:36.053 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:05:36.054 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:05:36.054 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:05:36.054 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 2 2021-05-25T12:05:36.054 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:05:36.054 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:05:36.055 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:05:36.055 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:05:36.055 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:05:36.055 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:05:36.055 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 8 2021-05-25T12:05:36.056 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:05:36.056 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:05:36.056 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:05:36.056 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:05:36.056 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:05:36.058 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.955+0000 7f340aa5c700 10 librados: watch_flush enter 2021-05-25T12:05:36.059 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.955+0000 7f340aa5c700 10 librados: watch_flush exit 2021-05-25T12:05:36.059 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.955+0000 7f340aa5c700 20 client.174612.objecter shutdown clearing up homeless session... 2021-05-25T12:05:36.059 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.955+0000 7f340aa5c700 10 client.174612.objecter successfully canceled tick 2021-05-25T12:05:36.059 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.955+0000 7f340aa5c700 1 -- 172.21.15.102:0/510981782 >> v1:172.21.15.181:6812/29582 conn(0x7f33e40335a0 legacy=0x7f33e4035a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:36.060 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.955+0000 7f340aa5c700 10 monclient: shutdown 2021-05-25T12:05:36.060 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.955+0000 7f340aa5c700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:05:36.060 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.955+0000 7f340aa5c700 1 -- 172.21.15.102:0/510981782 >> v1:172.21.15.102:6789/0 conn(0x7f3404090990 legacy=0x7f34041254d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:36.060 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.955+0000 7f3408ff9700 1 -- 172.21.15.102:0/510981782 reap_dead start 2021-05-25T12:05:36.060 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.955+0000 7f3408ff9700 1 -- 172.21.15.102:0/510981782 reap_dead start 2021-05-25T12:05:36.061 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.955+0000 7f340aa5c700 1 -- 172.21.15.102:0/510981782 shutdown_connections 2021-05-25T12:05:36.061 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.955+0000 7f340aa5c700 1 -- 172.21.15.102:0/510981782 >> 172.21.15.102:0/510981782 conn(0x7f340409b370 msgr2=0x7f34041294b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:05:36.061 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.955+0000 7f340aa5c700 1 -- 172.21.15.102:0/510981782 shutdown_connections 2021-05-25T12:05:36.061 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.955+0000 7f340aa5c700 1 -- 172.21.15.102:0/510981782 wait complete. 2021-05-25T12:05:36.061 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:35.955+0000 7f340aa5c700 1 librados: shutdown 2021-05-25T12:05:36.062 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2021-05-25T12:05:36.186 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.173+0000 7f2fa1987700 10 monclient: get_monmap_and_config 2021-05-25T12:05:36.186 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.173+0000 7f2fa1987700 10 monclient: build_initial_monmap 2021-05-25T12:05:36.186 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.173+0000 7f2fa1987700 10 monclient: monmap: 2021-05-25T12:05:36.187 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:05:36.187 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:36.187 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:05:36.173943+0000 2021-05-25T12:05:36.187 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:05:36.173943+0000 2021-05-25T12:05:36.188 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:05:36.188 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:36.188 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:05:36.188 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:05:36.189 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:05:36.189 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:36.189 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.173+0000 7f2fa1987700 1 Processor -- start 2021-05-25T12:05:36.189 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.173+0000 7f2fa1987700 1 -- start start 2021-05-25T12:05:36.189 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.173+0000 7f2fa1987700 10 monclient: init 2021-05-25T12:05:36.190 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.174+0000 7f2fa1987700 10 monclient: _reopen_session rank -1 2021-05-25T12:05:36.190 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.174+0000 7f2fa1987700 10 monclient: _add_conns ranks=[1,0,2] 2021-05-25T12:05:36.190 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.174+0000 7f2fa1987700 10 monclient(hunting): picked mon.noname-c con 0x7f2f9c098210 addr v1:172.21.15.181:6789/0 2021-05-25T12:05:36.190 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.174+0000 7f2fa1987700 10 monclient(hunting): picked mon.noname-a con 0x7f2f9c12bd00 addr v1:172.21.15.102:6789/0 2021-05-25T12:05:36.191 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.174+0000 7f2fa1987700 10 monclient(hunting): picked mon.noname-b con 0x7f2f9c12c190 addr v1:172.21.15.102:6790/0 2021-05-25T12:05:36.191 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.174+0000 7f2fa1987700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f9c099b20 con 0x7f2f9c12bd00 2021-05-25T12:05:36.191 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.174+0000 7f2fa1987700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f9c0985f0 con 0x7f2f9c098210 2021-05-25T12:05:36.191 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.174+0000 7f2fa1987700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f9c09acd0 con 0x7f2f9c12c190 2021-05-25T12:05:36.191 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.174+0000 7f2fa1987700 10 monclient(hunting): _renew_subs 2021-05-25T12:05:36.192 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.174+0000 7f2fa1987700 10 monclient(hunting): authenticate will time out at 1136.771108s 2021-05-25T12:05:36.192 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.174+0000 7f2f9a7fc700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f2f9c12bd00 0x7f2f9c130920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:32884/0 (socket says 172.21.15.102:32884) 2021-05-25T12:05:36.192 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.174+0000 7f2f9a7fc700 1 -- 172.21.15.102:0/2244463564 learned_addr learned my addr 172.21.15.102:0/2244463564 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:05:36.192 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.174+0000 7f2f99ffb700 1 -- 172.21.15.102:0/2244463564 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2320976186 0 0) 0x7f2f9c099b20 con 0x7f2f9c12bd00 2021-05-25T12:05:36.193 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.174+0000 7f2f99ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:36.193 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.174+0000 7f2f99ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:36.193 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.174+0000 7f2f99ffb700 10 monclient(hunting): my global_id is 174621 2021-05-25T12:05:36.193 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.174+0000 7f2f99ffb700 1 -- 172.21.15.102:0/2244463564 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f88003270 con 0x7f2f9c12bd00 2021-05-25T12:05:36.193 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.174+0000 7f2f99ffb700 1 -- 172.21.15.102:0/2244463564 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 561336233 0 0) 0x7f2f9c09acd0 con 0x7f2f9c12c190 2021-05-25T12:05:36.194 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.174+0000 7f2f99ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:36.194 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.174+0000 7f2f99ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:36.194 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.174+0000 7f2f99ffb700 10 monclient(hunting): my global_id is 184322 2021-05-25T12:05:36.194 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.174+0000 7f2f99ffb700 1 -- 172.21.15.102:0/2244463564 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f9c099b20 con 0x7f2f9c12c190 2021-05-25T12:05:36.195 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.174+0000 7f2f99ffb700 1 -- 172.21.15.102:0/2244463564 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3780211685 0 0) 0x7f2f90001110 con 0x7f2f9c098210 2021-05-25T12:05:36.195 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.174+0000 7f2f99ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:36.195 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.174+0000 7f2f99ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:36.195 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.174+0000 7f2f99ffb700 10 monclient(hunting): my global_id is 194128 2021-05-25T12:05:36.195 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.174+0000 7f2f99ffb700 1 -- 172.21.15.102:0/2244463564 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f9c09acd0 con 0x7f2f9c098210 2021-05-25T12:05:36.196 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.174+0000 7f2f99ffb700 1 -- 172.21.15.102:0/2244463564 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 227983772 0 0) 0x7f2f88003270 con 0x7f2f9c12bd00 2021-05-25T12:05:36.196 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.174+0000 7f2f99ffb700 1 -- 172.21.15.102:0/2244463564 >> v1:172.21.15.102:6790/0 conn(0x7f2f9c12c190 legacy=0x7f2f9c134050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:36.196 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.174+0000 7f2f99ffb700 1 -- 172.21.15.102:0/2244463564 >> v1:172.21.15.181:6789/0 conn(0x7f2f9c098210 legacy=0x7f2f9c12b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:36.196 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.174+0000 7f2f99ffb700 10 monclient: _finish_hunting 0 2021-05-25T12:05:36.197 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.174+0000 7f2f99ffb700 1 monclient: found mon.noname-a 2021-05-25T12:05:36.197 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.174+0000 7f2f99ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:05:36.197 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.174+0000 7f2f99ffb700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T12:05:36.197 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.174+0000 7f2f99ffb700 1 -- 172.21.15.102:0/2244463564 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f9c09af70 con 0x7f2f9c12bd00 2021-05-25T12:05:36.197 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.174+0000 7f2f99ffb700 10 monclient: _finish_auth 0 2021-05-25T12:05:36.198 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.174+0000 7f2f99ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:05:36.198 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.174+0000 7f2f99ffb700 1 -- 172.21.15.102:0/2244463564 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f2f84001610 con 0x7f2f9c12bd00 2021-05-25T12:05:36.198 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.174+0000 7f2f99ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:36.198 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.174+0000 7f2f99ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:05:36.199 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.174+0000 7f2f99ffb700 10 monclient: dump: 2021-05-25T12:05:36.199 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:36.199 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:36.199 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:36.200 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:36.200 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:36.200 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:36.200 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:36.200 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:36.201 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:36.201 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:36.201 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.174+0000 7f2fa1987700 5 monclient: authenticate success, global_id 174621 2021-05-25T12:05:36.201 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.174+0000 7f2fa1987700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:05:36.201 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.174+0000 7f2f99ffb700 1 -- 172.21.15.102:0/2244463564 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2f84001880 con 0x7f2f9c12bd00 2021-05-25T12:05:36.201 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.174+0000 7f2f99ffb700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:05:36.202 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.174+0000 7f2f99ffb700 1 -- 172.21.15.102:0/2244463564 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f2f84001c10 con 0x7f2f9c12bd00 2021-05-25T12:05:36.202 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.174+0000 7f2fa1987700 10 monclient: get_monmap_and_config success 2021-05-25T12:05:36.202 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.174+0000 7f2fa1987700 4 set_mon_vals no callback set 2021-05-25T12:05:36.202 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.174+0000 7f2f99ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:36.202 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.174+0000 7f2f99ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:05:36.203 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.174+0000 7f2f99ffb700 10 monclient: dump: 2021-05-25T12:05:36.203 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:36.203 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:36.203 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:36.203 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:36.204 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:36.204 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:36.204 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:36.204 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:36.205 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:36.205 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:36.205 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.175+0000 7f2fa1987700 10 monclient: shutdown 2021-05-25T12:05:36.205 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.175+0000 7f2fa1987700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:05:36.205 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.175+0000 7f2fa1987700 1 -- 172.21.15.102:0/2244463564 >> v1:172.21.15.102:6789/0 conn(0x7f2f9c12bd00 legacy=0x7f2f9c130920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:36.206 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.175+0000 7f2fa1987700 1 -- 172.21.15.102:0/2244463564 shutdown_connections 2021-05-25T12:05:36.206 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.175+0000 7f2fa1987700 1 -- 172.21.15.102:0/2244463564 >> 172.21.15.102:0/2244463564 conn(0x7f2f9c09e9d0 msgr2=0x7f2f9c09ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:05:36.206 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.175+0000 7f2fa1987700 1 -- 172.21.15.102:0/2244463564 shutdown_connections 2021-05-25T12:05:36.206 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.175+0000 7f2fa1987700 1 -- 172.21.15.102:0/2244463564 wait complete. 2021-05-25T12:05:36.206 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.176+0000 7f2fa1987700 10 monclient: build_initial_monmap 2021-05-25T12:05:36.207 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.176+0000 7f2fa1987700 10 monclient: monmap: 2021-05-25T12:05:36.207 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:05:36.207 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:05:36.207 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:05:36.207 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:05:36.207 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:05:36.208 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:05:36.208 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:05:36.208 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:05:36.208 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:05:36.208 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:36.209 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.176+0000 7f2fa1987700 1 librados: starting msgr at 2021-05-25T12:05:36.209 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.176+0000 7f2fa1987700 1 librados: starting objecter 2021-05-25T12:05:36.209 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.176+0000 7f2fa1987700 1 Processor -- start 2021-05-25T12:05:36.209 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.176+0000 7f2fa1987700 1 -- start start 2021-05-25T12:05:36.209 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.176+0000 7f2fa1987700 1 librados: setting wanted keys 2021-05-25T12:05:36.210 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.176+0000 7f2fa1987700 1 librados: calling monclient init 2021-05-25T12:05:36.210 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.176+0000 7f2fa1987700 10 monclient: init 2021-05-25T12:05:36.210 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.176+0000 7f2fa1987700 10 monclient: _reopen_session rank -1 2021-05-25T12:05:36.210 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.176+0000 7f2fa1987700 10 monclient: _add_conns ranks=[0,2,1] 2021-05-25T12:05:36.211 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.176+0000 7f2fa1987700 10 monclient(hunting): picked mon.noname-a con 0x7f2f9c098210 addr v1:172.21.15.102:6789/0 2021-05-25T12:05:36.211 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.176+0000 7f2fa1987700 10 monclient(hunting): picked mon.noname-c con 0x7f2f9c12bd00 addr v1:172.21.15.102:6790/0 2021-05-25T12:05:36.211 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.176+0000 7f2fa1987700 10 monclient(hunting): picked mon.noname-b con 0x7f2f9c12c190 addr v1:172.21.15.181:6789/0 2021-05-25T12:05:36.211 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.176+0000 7f2fa1987700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f9c09a220 con 0x7f2f9c098210 2021-05-25T12:05:36.211 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.176+0000 7f2fa1987700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f9c097470 con 0x7f2f9c12c190 2021-05-25T12:05:36.212 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.176+0000 7f2fa1987700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f9c12aec0 con 0x7f2f9c12bd00 2021-05-25T12:05:36.212 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.176+0000 7f2fa1987700 10 monclient(hunting): _renew_subs 2021-05-25T12:05:36.212 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.176+0000 7f2fa1987700 10 monclient(hunting): authenticate will time out at 1136.773915s 2021-05-25T12:05:36.212 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.176+0000 7f2f9affd700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f2f9c098210 0x7f2f9c128160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:32888/0 (socket says 172.21.15.102:32888) 2021-05-25T12:05:36.212 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.176+0000 7f2f9affd700 1 -- 172.21.15.102:0/208633949 learned_addr learned my addr 172.21.15.102:0/208633949 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:05:36.213 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.177+0000 7f2f82ffd700 10 client.?.objecter ms_handle_connect 0x7f2f9c098210 2021-05-25T12:05:36.213 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.177+0000 7f2f82ffd700 10 client.?.objecter resend_mon_ops 2021-05-25T12:05:36.213 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.177+0000 7f2f82ffd700 10 client.?.objecter ms_handle_connect 0x7f2f9c12bd00 2021-05-25T12:05:36.213 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.177+0000 7f2f82ffd700 10 client.?.objecter resend_mon_ops 2021-05-25T12:05:36.213 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.177+0000 7f2f82ffd700 10 client.?.objecter ms_handle_connect 0x7f2f9c12c190 2021-05-25T12:05:36.214 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.177+0000 7f2f82ffd700 10 client.?.objecter resend_mon_ops 2021-05-25T12:05:36.214 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.177+0000 7f2f82ffd700 1 -- 172.21.15.102:0/208633949 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3642547001 0 0) 0x7f2f9c09a220 con 0x7f2f9c098210 2021-05-25T12:05:36.214 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.177+0000 7f2f82ffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:36.214 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.177+0000 7f2f82ffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:36.214 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.177+0000 7f2f82ffd700 10 monclient(hunting): my global_id is 174624 2021-05-25T12:05:36.215 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.177+0000 7f2f82ffd700 1 -- 172.21.15.102:0/208633949 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f7c004c10 con 0x7f2f9c098210 2021-05-25T12:05:36.215 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.177+0000 7f2f82ffd700 1 -- 172.21.15.102:0/208633949 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2313352305 0 0) 0x7f2f9c12aec0 con 0x7f2f9c12bd00 2021-05-25T12:05:36.215 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.177+0000 7f2f82ffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:36.215 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.177+0000 7f2f82ffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:36.215 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.177+0000 7f2f82ffd700 10 monclient(hunting): my global_id is 184325 2021-05-25T12:05:36.216 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.177+0000 7f2f82ffd700 1 -- 172.21.15.102:0/208633949 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f9c09a220 con 0x7f2f9c12bd00 2021-05-25T12:05:36.216 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.177+0000 7f2f82ffd700 1 -- 172.21.15.102:0/208633949 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3510131317 0 0) 0x7f2f9c097470 con 0x7f2f9c12c190 2021-05-25T12:05:36.216 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.177+0000 7f2f82ffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:36.216 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.177+0000 7f2f82ffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:36.217 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.177+0000 7f2f82ffd700 10 monclient(hunting): my global_id is 194131 2021-05-25T12:05:36.217 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.177+0000 7f2f82ffd700 1 -- 172.21.15.102:0/208633949 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f9c12aec0 con 0x7f2f9c12c190 2021-05-25T12:05:36.217 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.177+0000 7f2f82ffd700 1 -- 172.21.15.102:0/208633949 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2316497970 0 0) 0x7f2f7c004c10 con 0x7f2f9c098210 2021-05-25T12:05:36.217 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.177+0000 7f2f82ffd700 1 -- 172.21.15.102:0/208633949 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2f9c097470 con 0x7f2f9c098210 2021-05-25T12:05:36.217 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.177+0000 7f2f82ffd700 1 -- 172.21.15.102:0/208633949 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f2f900018b0 con 0x7f2f9c098210 2021-05-25T12:05:36.218 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.177+0000 7f2f82ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:36.218 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.177+0000 7f2f82ffd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:05:36.218 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.177+0000 7f2f82ffd700 10 monclient(hunting): dump: 2021-05-25T12:05:36.218 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:36.218 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:36.219 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:36.219 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:36.219 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:36.219 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:36.219 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:36.220 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:36.220 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:36.220 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:36.220 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.177+0000 7f2f82ffd700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:05:36.220 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.177+0000 7f2f82ffd700 1 -- 172.21.15.102:0/208633949 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1328955532 0 0) 0x7f2f9c12aec0 con 0x7f2f9c12c190 2021-05-25T12:05:36.221 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.177+0000 7f2f82ffd700 1 -- 172.21.15.102:0/208633949 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2f7c004c10 con 0x7f2f9c12c190 2021-05-25T12:05:36.221 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.177+0000 7f2f82ffd700 1 -- 172.21.15.102:0/208633949 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f2f8c001d50 con 0x7f2f9c12c190 2021-05-25T12:05:36.221 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.177+0000 7f2f82ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:36.221 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.177+0000 7f2f82ffd700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2021-05-25T12:05:36.221 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.177+0000 7f2f82ffd700 10 monclient(hunting): dump: 2021-05-25T12:05:36.222 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:36.222 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:36.222 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:36.222 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:36.222 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:36.223 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:36.223 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:36.223 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:36.223 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:36.223 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:36.224 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.177+0000 7f2f82ffd700 1 -- 172.21.15.102:0/208633949 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3855605078 0 0) 0x7f2f9c09a220 con 0x7f2f9c12bd00 2021-05-25T12:05:36.224 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.177+0000 7f2f82ffd700 1 -- 172.21.15.102:0/208633949 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2f9c12aec0 con 0x7f2f9c12bd00 2021-05-25T12:05:36.224 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.177+0000 7f2f82ffd700 1 -- 172.21.15.102:0/208633949 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f2f840015d0 con 0x7f2f9c12bd00 2021-05-25T12:05:36.224 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.177+0000 7f2f82ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:36.224 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.177+0000 7f2f82ffd700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2021-05-25T12:05:36.225 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.177+0000 7f2f82ffd700 10 monclient(hunting): dump: 2021-05-25T12:05:36.225 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:36.225 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:36.225 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:36.225 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:36.226 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:36.226 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:36.226 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:36.226 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:36.226 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:36.227 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:36.227 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.177+0000 7f2f82ffd700 1 -- 172.21.15.102:0/208633949 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2106398563 0 0) 0x7f2f9c097470 con 0x7f2f9c098210 2021-05-25T12:05:36.227 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.177+0000 7f2f82ffd700 1 -- 172.21.15.102:0/208633949 >> v1:172.21.15.102:6790/0 conn(0x7f2f9c12bd00 legacy=0x7f2f9c140000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:36.227 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.177+0000 7f2f82ffd700 1 -- 172.21.15.102:0/208633949 >> v1:172.21.15.181:6789/0 conn(0x7f2f9c12c190 legacy=0x7f2f9c143730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:36.228 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.177+0000 7f2f82ffd700 10 monclient: _finish_hunting 0 2021-05-25T12:05:36.228 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.177+0000 7f2f82ffd700 1 monclient: found mon.a 2021-05-25T12:05:36.228 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.177+0000 7f2f82ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:05:36.228 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.177+0000 7f2f82ffd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:05:36.228 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.177+0000 7f2f82ffd700 1 -- 172.21.15.102:0/208633949 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f9c115ba0 con 0x7f2f9c098210 2021-05-25T12:05:36.229 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.177+0000 7f2f82ffd700 10 monclient: _finish_auth 0 2021-05-25T12:05:36.229 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.177+0000 7f2f82ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:05:36.229 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.177+0000 7f2fa1987700 5 monclient: authenticate success, global_id 174624 2021-05-25T12:05:36.229 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.177+0000 7f2fa1987700 10 monclient: _renew_subs 2021-05-25T12:05:36.229 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.177+0000 7f2fa1987700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:05:36.230 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.177+0000 7f2fa1987700 1 -- 172.21.15.102:0/208633949 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f9c115050 con 0x7f2f9c098210 2021-05-25T12:05:36.230 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.177+0000 7f2fa1987700 10 client.174624.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:05:36.230 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.177+0000 7f2fa1987700 10 monclient: _renew_subs 2021-05-25T12:05:36.230 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.177+0000 7f2fa1987700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:05:36.230 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.177+0000 7f2fa1987700 1 -- 172.21.15.102:0/208633949 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f9c128a30 con 0x7f2f9c098210 2021-05-25T12:05:36.231 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.177+0000 7f2fa1987700 1 librados: init done 2021-05-25T12:05:36.231 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.177+0000 7f2f82ffd700 1 -- 172.21.15.102:0/208633949 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2f90001b20 con 0x7f2f9c098210 2021-05-25T12:05:36.231 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.177+0000 7f2f82ffd700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:05:36.231 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.177+0000 7f2f82ffd700 1 -- 172.21.15.102:0/208633949 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f2f90001eb0 con 0x7f2f9c098210 2021-05-25T12:05:36.231 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.177+0000 7f2f82ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:36.232 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.177+0000 7f2f82ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:05:36.232 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.177+0000 7f2f997fa700 4 set_mon_vals no callback set 2021-05-25T12:05:36.232 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.177+0000 7f2f82ffd700 10 monclient: dump: 2021-05-25T12:05:36.232 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:36.232 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:36.233 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:36.233 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:36.233 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:36.233 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:36.233 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:36.234 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:36.234 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:36.234 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:36.234 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.178+0000 7f2f82ffd700 1 -- 172.21.15.102:0/208633949 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f2f90002090 con 0x7f2f9c098210 2021-05-25T12:05:36.234 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.178+0000 7f2fa1987700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:05:36.234 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.178+0000 7f2fa1987700 1 -- 172.21.15.102:0/208633949 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f2f740065a0 con 0x7f2f74002840 2021-05-25T12:05:36.235 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.178+0000 7f2f82ffd700 1 -- 172.21.15.102:0/208633949 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(168..168 src has 1..168) v4 ==== 7138+0+0 (unknown 3442091228 0 0) 0x7f2f9003da90 con 0x7f2f9c098210 2021-05-25T12:05:36.235 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.178+0000 7f2f82ffd700 10 client.174624.objecter ms_dispatch 0x7f2f9c001550 osd_map(168..168 src has 1..168) v4 2021-05-25T12:05:36.235 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.178+0000 7f2f82ffd700 3 client.174624.objecter handle_osd_map got epochs [168,168] > 0 2021-05-25T12:05:36.235 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.178+0000 7f2f82ffd700 3 client.174624.objecter handle_osd_map decoding full epoch 168 2021-05-25T12:05:36.235 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.178+0000 7f2f82ffd700 20 client.174624.objecter dump_active .. 0 homeless 2021-05-25T12:05:36.236 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.178+0000 7f2f82ffd700 10 client.174624.objecter ms_handle_connect 0x7f2f74002840 2021-05-25T12:05:36.236 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.178+0000 7f2f82ffd700 10 client.174624.objecter resend_mon_ops 2021-05-25T12:05:36.236 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.178+0000 7f2f82ffd700 1 -- 172.21.15.102:0/208633949 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3338022962 0 0) 0x7f2f740065a0 con 0x7f2f74002840 2021-05-25T12:05:36.236 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.178+0000 7f2f82ffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:36.237 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.178+0000 7f2f82ffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:36.237 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.178+0000 7f2f82ffd700 10 monclient(hunting): my global_id is 184328 2021-05-25T12:05:36.237 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.178+0000 7f2f82ffd700 1 -- 172.21.15.102:0/208633949 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f9c097470 con 0x7f2f74002840 2021-05-25T12:05:36.237 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.178+0000 7f2f9a7fc700 10 monclient: get_auth_request con 0x7f2f7c033550 auth_method 0 2021-05-25T12:05:36.237 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.178+0000 7f2f82ffd700 1 -- 172.21.15.102:0/208633949 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3130911794 0 0) 0x7f2f9c097470 con 0x7f2f74002840 2021-05-25T12:05:36.238 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.179+0000 7f2f82ffd700 1 -- 172.21.15.102:0/208633949 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2f740065a0 con 0x7f2f74002840 2021-05-25T12:05:36.238 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.179+0000 7f2f82ffd700 1 -- 172.21.15.102:0/208633949 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f2f8c002340 con 0x7f2f74002840 2021-05-25T12:05:36.238 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.179+0000 7f2f82ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:36.238 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.179+0000 7f2f82ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:05:36.238 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.179+0000 7f2f82ffd700 10 monclient: dump: 2021-05-25T12:05:36.239 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:36.239 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:36.239 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:36.239 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:36.240 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:36.240 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:36.240 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:36.240 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:36.240 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:36.241 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:36.241 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.179+0000 7f2f82ffd700 1 -- 172.21.15.102:0/208633949 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1124993595 0 0) 0x7f2f740065a0 con 0x7f2f74002840 2021-05-25T12:05:36.241 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.179+0000 7f2f82ffd700 1 -- 172.21.15.102:0/208633949 --> v1:172.21.15.102:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2f74009e40 con 0x7f2f74002840 2021-05-25T12:05:36.241 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.179+0000 7f2f82ffd700 10 client.174624.objecter ms_handle_connect 0x7f2f7c033550 2021-05-25T12:05:36.242 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.179+0000 7f2f82ffd700 1 -- 172.21.15.102:0/208633949 <== mon.2 v1:172.21.15.102:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7f2f8c004650 con 0x7f2f74002840 2021-05-25T12:05:36.242 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.179+0000 7f2f82ffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:05:36.242 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.179+0000 7f2f82ffd700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:05:36.242 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.179+0000 7f2f82ffd700 1 -- 172.21.15.102:0/208633949 >> v1:172.21.15.102:6790/0 conn(0x7f2f74002840 legacy=0x7f2f74004ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:36.242 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.179+0000 7f2f82ffd700 1 -- 172.21.15.102:0/208633949 >> v1:172.21.15.102:6790/0 conn(0x7f2f74002840 legacy=0x7f2f74004ce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:05:36.244 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.186+0000 7f2fa1987700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:05:36.245 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.186+0000 7f2fa1987700 1 -- 172.21.15.102:0/208633949 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f2f74009ad0 con 0x7f2f74008d90 2021-05-25T12:05:36.245 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.186+0000 7f2f82ffd700 10 client.174624.objecter ms_handle_connect 0x7f2f74008d90 2021-05-25T12:05:36.245 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.186+0000 7f2f82ffd700 10 client.174624.objecter resend_mon_ops 2021-05-25T12:05:36.245 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.186+0000 7f2f82ffd700 1 -- 172.21.15.102:0/208633949 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2641155885 0 0) 0x7f2f74009ad0 con 0x7f2f74008d90 2021-05-25T12:05:36.246 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.186+0000 7f2f82ffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:36.246 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.186+0000 7f2f82ffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:36.246 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.186+0000 7f2f82ffd700 10 monclient(hunting): my global_id is 184331 2021-05-25T12:05:36.246 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.186+0000 7f2f82ffd700 1 -- 172.21.15.102:0/208633949 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f740065a0 con 0x7f2f74008d90 2021-05-25T12:05:36.247 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.186+0000 7f2f82ffd700 1 -- 172.21.15.102:0/208633949 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1467852205 0 0) 0x7f2f740065a0 con 0x7f2f74008d90 2021-05-25T12:05:36.247 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.186+0000 7f2f82ffd700 1 -- 172.21.15.102:0/208633949 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2f74009ad0 con 0x7f2f74008d90 2021-05-25T12:05:36.247 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.186+0000 7f2f82ffd700 1 -- 172.21.15.102:0/208633949 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f2f8c002b70 con 0x7f2f74008d90 2021-05-25T12:05:36.247 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.186+0000 7f2f82ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:36.247 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.186+0000 7f2f82ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:05:36.248 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.186+0000 7f2f82ffd700 10 monclient: dump: 2021-05-25T12:05:36.248 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:36.248 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:36.248 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:36.248 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:36.249 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:36.249 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:36.249 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:36.249 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:36.249 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:36.250 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:36.250 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.186+0000 7f2f82ffd700 1 -- 172.21.15.102:0/208633949 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1588812382 0 0) 0x7f2f74009ad0 con 0x7f2f74008d90 2021-05-25T12:05:36.250 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.186+0000 7f2f82ffd700 1 -- 172.21.15.102:0/208633949 --> v1:172.21.15.102:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f2f74008740 con 0x7f2f74008d90 2021-05-25T12:05:36.250 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.186+0000 7f2f82ffd700 1 -- 172.21.15.102:0/208633949 <== mon.2 v1:172.21.15.102:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3942 (unknown 0 0 3584305227) 0x7f2f8c004900 con 0x7f2f74008d90 2021-05-25T12:05:36.251 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.186+0000 7f2f82ffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:05:36.251 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.186+0000 7f2f82ffd700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:05:36.251 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.186+0000 7f2f82ffd700 1 -- 172.21.15.102:0/208633949 >> v1:172.21.15.102:6790/0 conn(0x7f2f74008d90 legacy=0x7f2f74009220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:36.251 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.186+0000 7f2f82ffd700 1 -- 172.21.15.102:0/208633949 >> v1:172.21.15.102:6790/0 conn(0x7f2f74008d90 legacy=0x7f2f74009220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:05:36.252 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:05:36.252 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:05:36.252 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:05:36.252 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "leader", 2021-05-25T12:05:36.252 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 46, 2021-05-25T12:05:36.253 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:05:36.253 INFO:teuthology.orchestra.run.smithi102.stdout: 0, 2021-05-25T12:05:36.253 INFO:teuthology.orchestra.run.smithi102.stdout: 1, 2021-05-25T12:05:36.253 INFO:teuthology.orchestra.run.smithi102.stdout: 2 2021-05-25T12:05:36.254 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:05:36.254 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 3, 2021-05-25T12:05:36.254 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:05:36.254 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:05:36.254 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:05:36.255 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:05:36.255 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:05:36.255 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:05:36.255 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:05:36.256 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:05:36.256 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:05:36.256 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:05:36.256 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:05:36.256 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:05:36.257 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:05:36.257 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:05:36.257 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:05:36.257 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:05:36.258 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:05:36.258 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:05:36.258 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:05:36.258 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:05:36.258 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:05:36.259 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:05:36.259 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:05:36.259 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:05:36.259 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:05:36.259 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:05:36.260 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:05:36.260 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:05:36.260 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:05:36.260 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:05:36.261 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:05:36.261 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:05:36.261 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:05:36.261 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:05:36.261 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:05:36.262 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:05:36.262 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:05:36.262 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:05:36.262 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:05:36.263 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:05:36.263 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:05:36.263 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:05:36.263 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:05:36.263 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:05:36.264 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:05:36.264 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:05:36.264 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:05:36.264 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:05:36.264 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:05:36.265 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:05:36.265 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:05:36.265 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:05:36.265 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:05:36.265 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:05:36.266 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:05:36.266 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:05:36.266 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:05:36.266 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:05:36.267 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:05:36.267 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:05:36.267 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:05:36.267 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:05:36.267 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:05:36.268 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:05:36.268 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:05:36.268 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:05:36.268 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:05:36.269 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:05:36.269 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:05:36.269 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:05:36.269 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:05:36.269 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:05:36.270 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:05:36.270 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:05:36.270 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:05:36.270 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:05:36.271 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:05:36.271 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:05:36.271 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:05:36.271 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:05:36.271 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:05:36.272 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:05:36.272 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:05:36.272 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:05:36.272 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:05:36.272 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:05:36.273 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:05:36.273 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:05:36.273 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:05:36.273 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:05:36.273 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:05:36.274 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:05:36.274 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:05:36.274 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:05:36.274 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:05:36.275 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:05:36.275 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:05:36.275 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:05:36.275 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:05:36.276 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:05:36.276 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:05:36.276 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:05:36.276 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:05:36.276 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:05:36.277 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:05:36.277 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:05:36.277 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:05:36.277 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:05:36.278 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:05:36.278 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:05:36.278 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:05:36.278 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:05:36.278 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:05:36.279 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:05:36.279 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:05:36.279 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:05:36.279 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:05:36.280 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:05:36.280 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:05:36.280 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:05:36.280 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:05:36.281 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:05:36.281 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 4 2021-05-25T12:05:36.281 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:05:36.281 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:05:36.281 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": [ 2021-05-25T12:05:36.282 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:05:36.282 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:05:36.282 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:05:36.282 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:05:36.282 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:05:36.283 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:05:36.283 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:05:36.283 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:05:36.283 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:05:36.286 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.187+0000 7f2fa1987700 10 librados: watch_flush enter 2021-05-25T12:05:36.286 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.187+0000 7f2fa1987700 10 librados: watch_flush exit 2021-05-25T12:05:36.286 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.187+0000 7f2fa1987700 20 client.174624.objecter shutdown clearing up homeless session... 2021-05-25T12:05:36.286 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.187+0000 7f2fa1987700 10 client.174624.objecter successfully canceled tick 2021-05-25T12:05:36.287 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.187+0000 7f2fa1987700 1 -- 172.21.15.102:0/208633949 >> v1:172.21.15.181:6812/29582 conn(0x7f2f7c033550 legacy=0x7f2f7c035a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:36.287 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.187+0000 7f2fa1987700 10 monclient: shutdown 2021-05-25T12:05:36.287 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.187+0000 7f2fa1987700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:05:36.287 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.187+0000 7f2fa1987700 1 -- 172.21.15.102:0/208633949 >> v1:172.21.15.102:6789/0 conn(0x7f2f9c098210 legacy=0x7f2f9c128160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:36.287 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.187+0000 7f2f9b7fe700 1 -- 172.21.15.102:0/208633949 reap_dead start 2021-05-25T12:05:36.288 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.187+0000 7f2f9b7fe700 1 -- 172.21.15.102:0/208633949 reap_dead start 2021-05-25T12:05:36.288 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.187+0000 7f2fa1987700 1 -- 172.21.15.102:0/208633949 shutdown_connections 2021-05-25T12:05:36.288 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.187+0000 7f2fa1987700 1 -- 172.21.15.102:0/208633949 >> 172.21.15.102:0/208633949 conn(0x7f2f9c09e9d0 msgr2=0x7f2f9c12b540 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:05:36.288 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.187+0000 7f2fa1987700 1 -- 172.21.15.102:0/208633949 shutdown_connections 2021-05-25T12:05:36.289 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.187+0000 7f2fa1987700 1 -- 172.21.15.102:0/208633949 wait complete. 2021-05-25T12:05:36.289 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.187+0000 7f2fa1987700 1 librados: shutdown 2021-05-25T12:05:36.289 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2021-05-25T12:05:36.400 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.398+0000 7f0ec2994700 10 monclient: get_monmap_and_config 2021-05-25T12:05:36.400 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.398+0000 7f0ec2994700 10 monclient: build_initial_monmap 2021-05-25T12:05:36.400 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.398+0000 7f0ec2994700 10 monclient: monmap: 2021-05-25T12:05:36.400 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:05:36.401 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:36.401 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:05:36.399462+0000 2021-05-25T12:05:36.401 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:05:36.399462+0000 2021-05-25T12:05:36.401 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:05:36.402 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:36.402 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:05:36.402 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:05:36.402 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:05:36.402 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:36.403 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.399+0000 7f0ec2994700 1 Processor -- start 2021-05-25T12:05:36.416 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.399+0000 7f0ec2994700 1 -- start start 2021-05-25T12:05:36.417 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.399+0000 7f0ec2994700 10 monclient: init 2021-05-25T12:05:36.417 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.399+0000 7f0ec2994700 10 monclient: _reopen_session rank -1 2021-05-25T12:05:36.417 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.399+0000 7f0ec2994700 10 monclient: _add_conns ranks=[1,2,0] 2021-05-25T12:05:36.417 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.399+0000 7f0ec2994700 10 monclient(hunting): picked mon.noname-c con 0x7f0ebc098930 addr v1:172.21.15.181:6789/0 2021-05-25T12:05:36.418 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.399+0000 7f0ec2994700 10 monclient(hunting): picked mon.noname-b con 0x7f0ebc12bca0 addr v1:172.21.15.102:6790/0 2021-05-25T12:05:36.418 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.399+0000 7f0ec2994700 10 monclient(hunting): picked mon.noname-a con 0x7f0ebc12c150 addr v1:172.21.15.102:6789/0 2021-05-25T12:05:36.418 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.399+0000 7f0ec2994700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ebc099b20 con 0x7f0ebc12c150 2021-05-25T12:05:36.418 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.399+0000 7f0ec2994700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ebc09a3a0 con 0x7f0ebc098930 2021-05-25T12:05:36.419 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.399+0000 7f0ec2994700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ebc09ac60 con 0x7f0ebc12bca0 2021-05-25T12:05:36.419 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.399+0000 7f0ec2994700 10 monclient(hunting): _renew_subs 2021-05-25T12:05:36.419 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.399+0000 7f0ec2994700 10 monclient(hunting): authenticate will time out at 1136.996796s 2021-05-25T12:05:36.419 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.399+0000 7f0ec1191700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f0ebc12bca0 0x7f0ebc1308c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:56012/0 (socket says 172.21.15.102:56012) 2021-05-25T12:05:36.419 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.399+0000 7f0ec2193700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f0ebc12c150 0x7f0ebc133ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:32904/0 (socket says 172.21.15.102:32904) 2021-05-25T12:05:36.420 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.399+0000 7f0ec1191700 1 -- 172.21.15.102:0/101643333 learned_addr learned my addr 172.21.15.102:0/101643333 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:05:36.420 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.400+0000 7f0ec0990700 1 -- 172.21.15.102:0/101643333 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1055155722 0 0) 0x7f0ebc09ac60 con 0x7f0ebc12bca0 2021-05-25T12:05:36.420 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.400+0000 7f0ec0990700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:36.420 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.400+0000 7f0ec0990700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:36.421 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.400+0000 7f0ec0990700 10 monclient(hunting): my global_id is 184334 2021-05-25T12:05:36.421 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.400+0000 7f0ec0990700 1 -- 172.21.15.102:0/101643333 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ea4003270 con 0x7f0ebc12bca0 2021-05-25T12:05:36.421 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.400+0000 7f0ec0990700 1 -- 172.21.15.102:0/101643333 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1493560018 0 0) 0x7f0ebc099b20 con 0x7f0ebc12c150 2021-05-25T12:05:36.421 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.400+0000 7f0ec0990700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:36.421 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.400+0000 7f0ec0990700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:36.422 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.400+0000 7f0ec0990700 10 monclient(hunting): my global_id is 174627 2021-05-25T12:05:36.422 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.400+0000 7f0ec0990700 1 -- 172.21.15.102:0/101643333 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ebc09ac60 con 0x7f0ebc12c150 2021-05-25T12:05:36.422 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.400+0000 7f0ec0990700 1 -- 172.21.15.102:0/101643333 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3512779198 0 0) 0x7f0ebc09a3a0 con 0x7f0ebc098930 2021-05-25T12:05:36.422 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.400+0000 7f0ec0990700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:36.422 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.400+0000 7f0ec0990700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:36.423 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.400+0000 7f0ec0990700 10 monclient(hunting): my global_id is 194134 2021-05-25T12:05:36.423 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.400+0000 7f0ec0990700 1 -- 172.21.15.102:0/101643333 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ebc099b20 con 0x7f0ebc098930 2021-05-25T12:05:36.423 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.400+0000 7f0ec0990700 1 -- 172.21.15.102:0/101643333 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1300850999 0 0) 0x7f0ea4003270 con 0x7f0ebc12bca0 2021-05-25T12:05:36.423 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.400+0000 7f0ec0990700 1 -- 172.21.15.102:0/101643333 >> v1:172.21.15.181:6789/0 conn(0x7f0ebc098930 legacy=0x7f0ebc12b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:36.423 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.400+0000 7f0ec0990700 1 -- 172.21.15.102:0/101643333 >> v1:172.21.15.102:6789/0 conn(0x7f0ebc12c150 legacy=0x7f0ebc133ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:36.424 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.400+0000 7f0ec0990700 10 monclient: _finish_hunting 0 2021-05-25T12:05:36.424 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.400+0000 7f0ec0990700 1 monclient: found mon.noname-b 2021-05-25T12:05:36.424 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.400+0000 7f0ec0990700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:05:36.424 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.400+0000 7f0ec0990700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T12:05:36.424 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.400+0000 7f0ec0990700 1 -- 172.21.15.102:0/101643333 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ebc09af00 con 0x7f0ebc12bca0 2021-05-25T12:05:36.425 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.400+0000 7f0ec0990700 10 monclient: _finish_auth 0 2021-05-25T12:05:36.425 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.400+0000 7f0ec0990700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:05:36.425 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.400+0000 7f0ec0990700 1 -- 172.21.15.102:0/101643333 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f0eac001610 con 0x7f0ebc12bca0 2021-05-25T12:05:36.425 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.400+0000 7f0ec2994700 5 monclient: authenticate success, global_id 184334 2021-05-25T12:05:36.425 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.400+0000 7f0ec2994700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:05:36.426 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.400+0000 7f0ec0990700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:36.426 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.400+0000 7f0ec0990700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:05:36.426 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.400+0000 7f0ec0990700 10 monclient: dump: 2021-05-25T12:05:36.426 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:36.426 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:36.427 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:36.427 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:36.427 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:36.427 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:36.427 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:36.428 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:36.428 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:36.428 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:36.428 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.400+0000 7f0ec2994700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:05:36.428 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.400+0000 7f0ec0990700 1 -- 172.21.15.102:0/101643333 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0eac001880 con 0x7f0ebc12bca0 2021-05-25T12:05:36.429 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.400+0000 7f0ec0990700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:05:36.429 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.400+0000 7f0ec2994700 10 monclient: get_monmap_and_config success 2021-05-25T12:05:36.429 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.400+0000 7f0ec0990700 1 -- 172.21.15.102:0/101643333 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f0eac001c10 con 0x7f0ebc12bca0 2021-05-25T12:05:36.429 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.400+0000 7f0ec0990700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:36.429 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.400+0000 7f0ec2994700 4 set_mon_vals no callback set 2021-05-25T12:05:36.430 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.400+0000 7f0ec0990700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:05:36.430 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.400+0000 7f0ec0990700 10 monclient: dump: 2021-05-25T12:05:36.430 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:36.430 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:36.430 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:36.431 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:36.431 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:36.431 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:36.431 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:36.431 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:36.432 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:36.432 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:36.432 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.400+0000 7f0ec2994700 10 monclient: shutdown 2021-05-25T12:05:36.432 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.400+0000 7f0ec2994700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:05:36.432 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.400+0000 7f0ec2994700 1 -- 172.21.15.102:0/101643333 >> v1:172.21.15.102:6790/0 conn(0x7f0ebc12bca0 legacy=0x7f0ebc1308c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:36.433 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.401+0000 7f0ec2994700 1 -- 172.21.15.102:0/101643333 shutdown_connections 2021-05-25T12:05:36.433 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.401+0000 7f0ec2994700 1 -- 172.21.15.102:0/101643333 >> 172.21.15.102:0/101643333 conn(0x7f0ebc09e9d0 msgr2=0x7f0ebc09ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:05:36.433 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.401+0000 7f0ec2994700 1 -- 172.21.15.102:0/101643333 shutdown_connections 2021-05-25T12:05:36.433 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.401+0000 7f0ec2994700 1 -- 172.21.15.102:0/101643333 wait complete. 2021-05-25T12:05:36.433 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.401+0000 7f0ec2994700 10 monclient: build_initial_monmap 2021-05-25T12:05:36.434 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.401+0000 7f0ec2994700 10 monclient: monmap: 2021-05-25T12:05:36.434 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:05:36.434 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:05:36.434 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:05:36.435 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:05:36.435 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:05:36.435 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:05:36.435 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:05:36.435 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:05:36.436 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:05:36.436 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:36.436 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.401+0000 7f0ec2994700 1 librados: starting msgr at 2021-05-25T12:05:36.436 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.401+0000 7f0ec2994700 1 librados: starting objecter 2021-05-25T12:05:36.436 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.401+0000 7f0ec2994700 1 Processor -- start 2021-05-25T12:05:36.437 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.401+0000 7f0ec2994700 1 -- start start 2021-05-25T12:05:36.437 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.401+0000 7f0ec2994700 1 librados: setting wanted keys 2021-05-25T12:05:36.437 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.401+0000 7f0ec2994700 1 librados: calling monclient init 2021-05-25T12:05:36.437 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.401+0000 7f0ec2994700 10 monclient: init 2021-05-25T12:05:36.437 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.402+0000 7f0ec2994700 10 monclient: _reopen_session rank -1 2021-05-25T12:05:36.438 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.402+0000 7f0ec2994700 10 monclient: _add_conns ranks=[1,0,2] 2021-05-25T12:05:36.438 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.402+0000 7f0ec2994700 10 monclient(hunting): picked mon.noname-b con 0x7f0ebc098930 addr v1:172.21.15.181:6789/0 2021-05-25T12:05:36.438 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.402+0000 7f0ec2994700 10 monclient(hunting): picked mon.noname-a con 0x7f0ebc12bca0 addr v1:172.21.15.102:6789/0 2021-05-25T12:05:36.438 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.402+0000 7f0ec2994700 10 monclient(hunting): picked mon.noname-c con 0x7f0ebc12c150 addr v1:172.21.15.102:6790/0 2021-05-25T12:05:36.438 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.402+0000 7f0ec2994700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ebc126060 con 0x7f0ebc12bca0 2021-05-25T12:05:36.439 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.402+0000 7f0ec2994700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ebc124190 con 0x7f0ebc098930 2021-05-25T12:05:36.439 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.402+0000 7f0ec2994700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ebc123780 con 0x7f0ebc12c150 2021-05-25T12:05:36.439 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.402+0000 7f0ec2994700 10 monclient(hunting): _renew_subs 2021-05-25T12:05:36.439 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.402+0000 7f0ec2994700 10 monclient(hunting): authenticate will time out at 1136.999443s 2021-05-25T12:05:36.439 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.402+0000 7f0ec2193700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f0ebc12c150 0x7f0ebc143590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:56020/0 (socket says 172.21.15.102:56020) 2021-05-25T12:05:36.440 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.402+0000 7f0ec2193700 1 -- 172.21.15.102:0/3869695838 learned_addr learned my addr 172.21.15.102:0/3869695838 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:05:36.440 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.402+0000 7f0ea9ffb700 10 client.?.objecter ms_handle_connect 0x7f0ebc12c150 2021-05-25T12:05:36.440 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.402+0000 7f0ea9ffb700 10 client.?.objecter resend_mon_ops 2021-05-25T12:05:36.440 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.402+0000 7f0ea9ffb700 10 client.?.objecter ms_handle_connect 0x7f0ebc12bca0 2021-05-25T12:05:36.440 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.402+0000 7f0ea9ffb700 10 client.?.objecter resend_mon_ops 2021-05-25T12:05:36.441 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.402+0000 7f0ea9ffb700 10 client.?.objecter ms_handle_connect 0x7f0ebc098930 2021-05-25T12:05:36.441 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.402+0000 7f0ea9ffb700 10 client.?.objecter resend_mon_ops 2021-05-25T12:05:36.441 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.402+0000 7f0ea9ffb700 1 -- 172.21.15.102:0/3869695838 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 323871941 0 0) 0x7f0ebc123780 con 0x7f0ebc12c150 2021-05-25T12:05:36.441 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.402+0000 7f0ea9ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:36.441 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.402+0000 7f0ea9ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:36.442 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.402+0000 7f0ea9ffb700 10 monclient(hunting): my global_id is 184337 2021-05-25T12:05:36.442 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.402+0000 7f0ea9ffb700 1 -- 172.21.15.102:0/3869695838 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e9c004c10 con 0x7f0ebc12c150 2021-05-25T12:05:36.442 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.402+0000 7f0ea9ffb700 1 -- 172.21.15.102:0/3869695838 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1664267530 0 0) 0x7f0ebc126060 con 0x7f0ebc12bca0 2021-05-25T12:05:36.442 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.402+0000 7f0ea9ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:36.443 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.402+0000 7f0ea9ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:36.443 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.402+0000 7f0ea9ffb700 10 monclient(hunting): my global_id is 174630 2021-05-25T12:05:36.443 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.402+0000 7f0ea9ffb700 1 -- 172.21.15.102:0/3869695838 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ebc123780 con 0x7f0ebc12bca0 2021-05-25T12:05:36.443 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.402+0000 7f0ea9ffb700 1 -- 172.21.15.102:0/3869695838 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3899605650 0 0) 0x7f0ebc124190 con 0x7f0ebc098930 2021-05-25T12:05:36.443 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.402+0000 7f0ea9ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:36.444 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.402+0000 7f0ea9ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:36.444 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.402+0000 7f0ea9ffb700 10 monclient(hunting): my global_id is 194137 2021-05-25T12:05:36.444 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.402+0000 7f0ea9ffb700 1 -- 172.21.15.102:0/3869695838 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ebc126060 con 0x7f0ebc098930 2021-05-25T12:05:36.444 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.402+0000 7f0ea9ffb700 1 -- 172.21.15.102:0/3869695838 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1886784880 0 0) 0x7f0ebc126060 con 0x7f0ebc098930 2021-05-25T12:05:36.444 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.403+0000 7f0ea9ffb700 1 -- 172.21.15.102:0/3869695838 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0ebc124190 con 0x7f0ebc098930 2021-05-25T12:05:36.445 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.403+0000 7f0ea9ffb700 1 -- 172.21.15.102:0/3869695838 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2963910706 0 0) 0x7f0e9c004c10 con 0x7f0ebc12c150 2021-05-25T12:05:36.445 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.403+0000 7f0ea9ffb700 1 -- 172.21.15.102:0/3869695838 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0ebc126060 con 0x7f0ebc12c150 2021-05-25T12:05:36.445 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.403+0000 7f0ea9ffb700 1 -- 172.21.15.102:0/3869695838 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f0eb0001880 con 0x7f0ebc098930 2021-05-25T12:05:36.445 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.403+0000 7f0ea9ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:36.445 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.403+0000 7f0ea9ffb700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:05:36.446 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.403+0000 7f0ea9ffb700 10 monclient(hunting): dump: 2021-05-25T12:05:36.446 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:36.446 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:36.446 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:36.446 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:36.447 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:36.447 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:36.447 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:36.447 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:36.448 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:36.448 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:36.448 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.403+0000 7f0ea9ffb700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:05:36.448 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.403+0000 7f0ea9ffb700 1 -- 172.21.15.102:0/3869695838 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2042278438 0 0) 0x7f0ebc123780 con 0x7f0ebc12bca0 2021-05-25T12:05:36.448 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.403+0000 7f0ea9ffb700 1 -- 172.21.15.102:0/3869695838 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0e9c004c10 con 0x7f0ebc12bca0 2021-05-25T12:05:36.449 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.403+0000 7f0ea9ffb700 1 -- 172.21.15.102:0/3869695838 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f0eb8001c80 con 0x7f0ebc12c150 2021-05-25T12:05:36.449 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.403+0000 7f0ea9ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:36.449 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.403+0000 7f0ea9ffb700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2021-05-25T12:05:36.449 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.403+0000 7f0ea9ffb700 10 monclient(hunting): dump: 2021-05-25T12:05:36.449 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:36.450 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:36.450 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:36.450 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:36.450 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:36.450 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:36.451 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:36.451 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:36.451 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:36.451 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:36.451 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.403+0000 7f0ea9ffb700 1 -- 172.21.15.102:0/3869695838 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f0eac0015d0 con 0x7f0ebc12bca0 2021-05-25T12:05:36.452 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.403+0000 7f0ea9ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:36.452 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.403+0000 7f0ea9ffb700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2021-05-25T12:05:36.452 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.403+0000 7f0ea9ffb700 10 monclient(hunting): dump: 2021-05-25T12:05:36.452 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:36.452 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:36.453 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:36.453 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:36.453 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:36.453 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:36.453 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:36.454 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:36.454 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:36.454 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:36.454 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.403+0000 7f0ea9ffb700 1 -- 172.21.15.102:0/3869695838 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3650835711 0 0) 0x7f0ebc124190 con 0x7f0ebc098930 2021-05-25T12:05:36.454 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.403+0000 7f0ea9ffb700 1 -- 172.21.15.102:0/3869695838 >> v1:172.21.15.102:6790/0 conn(0x7f0ebc12c150 legacy=0x7f0ebc143590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:36.455 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.403+0000 7f0ea9ffb700 1 -- 172.21.15.102:0/3869695838 >> v1:172.21.15.102:6789/0 conn(0x7f0ebc12bca0 legacy=0x7f0ebc13fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:36.455 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.403+0000 7f0ea9ffb700 10 monclient: _finish_hunting 0 2021-05-25T12:05:36.455 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.403+0000 7f0ea9ffb700 1 monclient: found mon.b 2021-05-25T12:05:36.455 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.403+0000 7f0ea9ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:05:36.455 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.403+0000 7f0ea9ffb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T12:05:36.456 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.403+0000 7f0ea9ffb700 1 -- 172.21.15.102:0/3869695838 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ebc123910 con 0x7f0ebc098930 2021-05-25T12:05:36.456 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.403+0000 7f0ea9ffb700 10 monclient: _finish_auth 0 2021-05-25T12:05:36.456 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.403+0000 7f0ea9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:05:36.456 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.403+0000 7f0ec2994700 5 monclient: authenticate success, global_id 194137 2021-05-25T12:05:36.456 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.403+0000 7f0ec2994700 10 monclient: _renew_subs 2021-05-25T12:05:36.457 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.403+0000 7f0ec2994700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T12:05:36.457 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.403+0000 7f0ec2994700 1 -- 172.21.15.102:0/3869695838 --> v1:172.21.15.181:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ebc126cb0 con 0x7f0ebc098930 2021-05-25T12:05:36.457 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.403+0000 7f0ea9ffb700 1 -- 172.21.15.102:0/3869695838 <== mon.1 v1:172.21.15.181:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0eb0001a20 con 0x7f0ebc098930 2021-05-25T12:05:36.457 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.403+0000 7f0ea9ffb700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:05:36.457 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.403+0000 7f0ea9ffb700 1 -- 172.21.15.102:0/3869695838 <== mon.1 v1:172.21.15.181:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f0eb0001db0 con 0x7f0ebc098930 2021-05-25T12:05:36.458 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.403+0000 7f0ea9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:36.458 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.403+0000 7f0ea9ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T12:05:36.458 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.403+0000 7f0ea9ffb700 10 monclient: dump: 2021-05-25T12:05:36.458 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:36.458 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:36.459 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:36.459 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:36.459 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:36.459 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:36.460 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:36.460 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:36.460 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:36.460 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:36.460 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.403+0000 7f0ec2994700 10 client.194137.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:05:36.461 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.403+0000 7f0ec2994700 10 monclient: _renew_subs 2021-05-25T12:05:36.461 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.403+0000 7f0eabfff700 4 set_mon_vals no callback set 2021-05-25T12:05:36.461 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.403+0000 7f0ec2994700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T12:05:36.461 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.403+0000 7f0ec2994700 1 -- 172.21.15.102:0/3869695838 --> v1:172.21.15.181:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ebc127120 con 0x7f0ebc098930 2021-05-25T12:05:36.461 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.403+0000 7f0ec2994700 1 librados: init done 2021-05-25T12:05:36.462 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.404+0000 7f0ec2994700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:05:36.462 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.404+0000 7f0ec2994700 1 -- 172.21.15.102:0/3869695838 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f0ebc124e00 con 0x7f0ebc128bb0 2021-05-25T12:05:36.462 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.404+0000 7f0ea9ffb700 1 -- 172.21.15.102:0/3869695838 <== mon.1 v1:172.21.15.181:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f0eb0001fd0 con 0x7f0ebc098930 2021-05-25T12:05:36.462 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.404+0000 7f0ea9ffb700 1 -- 172.21.15.102:0/3869695838 <== mon.1 v1:172.21.15.181:6789/0 8 ==== osd_map(168..168 src has 1..168) v4 ==== 7138+0+0 (unknown 3442091228 0 0) 0x7f0eb003b480 con 0x7f0ebc098930 2021-05-25T12:05:36.463 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.404+0000 7f0ea9ffb700 10 client.194137.objecter ms_dispatch 0x7f0ebc001550 osd_map(168..168 src has 1..168) v4 2021-05-25T12:05:36.463 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.404+0000 7f0ea9ffb700 3 client.194137.objecter handle_osd_map got epochs [168,168] > 0 2021-05-25T12:05:36.463 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.404+0000 7f0ea9ffb700 3 client.194137.objecter handle_osd_map decoding full epoch 168 2021-05-25T12:05:36.463 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.404+0000 7f0ea9ffb700 20 client.194137.objecter dump_active .. 0 homeless 2021-05-25T12:05:36.463 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.404+0000 7f0ea9ffb700 10 client.194137.objecter ms_handle_connect 0x7f0ebc128bb0 2021-05-25T12:05:36.464 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.404+0000 7f0ea9ffb700 10 client.194137.objecter resend_mon_ops 2021-05-25T12:05:36.464 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.404+0000 7f0ec2193700 10 monclient: get_auth_request con 0x7f0e9c033590 auth_method 0 2021-05-25T12:05:36.464 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.404+0000 7f0ea9ffb700 1 -- 172.21.15.102:0/3869695838 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 266287877 0 0) 0x7f0ebc124e00 con 0x7f0ebc128bb0 2021-05-25T12:05:36.464 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.404+0000 7f0ea9ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:36.464 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.404+0000 7f0ea9ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:36.465 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.404+0000 7f0ea9ffb700 10 monclient(hunting): my global_id is 194140 2021-05-25T12:05:36.465 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.404+0000 7f0ea9ffb700 1 -- 172.21.15.102:0/3869695838 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ebc124190 con 0x7f0ebc128bb0 2021-05-25T12:05:36.465 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.404+0000 7f0ea9ffb700 10 client.194137.objecter ms_handle_connect 0x7f0e9c033590 2021-05-25T12:05:36.465 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.404+0000 7f0ea9ffb700 1 -- 172.21.15.102:0/3869695838 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 755486297 0 0) 0x7f0ebc124190 con 0x7f0ebc128bb0 2021-05-25T12:05:36.465 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.404+0000 7f0ea9ffb700 1 -- 172.21.15.102:0/3869695838 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0ebc124e00 con 0x7f0ebc128bb0 2021-05-25T12:05:36.466 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.404+0000 7f0ea9ffb700 1 -- 172.21.15.102:0/3869695838 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f0eac0049c0 con 0x7f0ebc128bb0 2021-05-25T12:05:36.466 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.404+0000 7f0ea9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:36.466 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.404+0000 7f0ea9ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T12:05:36.466 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.404+0000 7f0ea9ffb700 10 monclient: dump: 2021-05-25T12:05:36.466 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:36.467 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:36.467 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:36.467 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:36.467 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:36.467 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:36.468 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:36.468 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:36.468 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:36.468 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:36.468 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.404+0000 7f0ea9ffb700 1 -- 172.21.15.102:0/3869695838 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2190678835 0 0) 0x7f0ebc124e00 con 0x7f0ebc128bb0 2021-05-25T12:05:36.469 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.404+0000 7f0ea9ffb700 1 -- 172.21.15.102:0/3869695838 --> v1:172.21.15.181:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0ebc12a160 con 0x7f0ebc128bb0 2021-05-25T12:05:36.469 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.405+0000 7f0ea9ffb700 1 -- 172.21.15.102:0/3869695838 <== mon.1 v1:172.21.15.181:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7f0eac006f80 con 0x7f0ebc128bb0 2021-05-25T12:05:36.469 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.405+0000 7f0ea9ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:05:36.469 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.405+0000 7f0ea9ffb700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:05:36.469 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.405+0000 7f0ea9ffb700 1 -- 172.21.15.102:0/3869695838 >> v1:172.21.15.181:6789/0 conn(0x7f0ebc128bb0 legacy=0x7f0ebc093e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:36.470 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.405+0000 7f0ea9ffb700 1 -- 172.21.15.102:0/3869695838 >> v1:172.21.15.181:6789/0 conn(0x7f0ebc128bb0 legacy=0x7f0ebc093e10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:05:36.470 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.411+0000 7f0ec2994700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:05:36.470 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.411+0000 7f0ec2994700 1 -- 172.21.15.102:0/3869695838 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f0ebc125e50 con 0x7f0ebc027ce0 2021-05-25T12:05:36.470 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.412+0000 7f0ea9ffb700 10 client.194137.objecter ms_handle_connect 0x7f0ebc027ce0 2021-05-25T12:05:36.471 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.412+0000 7f0ea9ffb700 10 client.194137.objecter resend_mon_ops 2021-05-25T12:05:36.471 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.412+0000 7f0ea9ffb700 1 -- 172.21.15.102:0/3869695838 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3006671806 0 0) 0x7f0ebc125e50 con 0x7f0ebc027ce0 2021-05-25T12:05:36.471 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.412+0000 7f0ea9ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:36.471 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.412+0000 7f0ea9ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:36.472 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.412+0000 7f0ea9ffb700 10 monclient(hunting): my global_id is 194143 2021-05-25T12:05:36.472 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.412+0000 7f0ea9ffb700 1 -- 172.21.15.102:0/3869695838 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ebc124e00 con 0x7f0ebc027ce0 2021-05-25T12:05:36.472 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.412+0000 7f0ea9ffb700 1 -- 172.21.15.102:0/3869695838 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3380061916 0 0) 0x7f0ebc124e00 con 0x7f0ebc027ce0 2021-05-25T12:05:36.472 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.412+0000 7f0ea9ffb700 1 -- 172.21.15.102:0/3869695838 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0ebc125e50 con 0x7f0ebc027ce0 2021-05-25T12:05:36.472 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.412+0000 7f0ea9ffb700 1 -- 172.21.15.102:0/3869695838 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f0eac005450 con 0x7f0ebc027ce0 2021-05-25T12:05:36.473 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.412+0000 7f0ea9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:36.473 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.412+0000 7f0ea9ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T12:05:36.473 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.412+0000 7f0ea9ffb700 10 monclient: dump: 2021-05-25T12:05:36.473 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:36.473 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:36.474 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:36.474 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:36.474 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:36.474 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:36.474 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:36.475 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:36.475 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:36.475 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:36.475 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.412+0000 7f0ea9ffb700 1 -- 172.21.15.102:0/3869695838 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4253999746 0 0) 0x7f0ebc125e50 con 0x7f0ebc027ce0 2021-05-25T12:05:36.475 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.412+0000 7f0ea9ffb700 1 -- 172.21.15.102:0/3869695838 --> v1:172.21.15.181:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f0ebc059730 con 0x7f0ebc027ce0 2021-05-25T12:05:36.476 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.412+0000 7f0ea9ffb700 1 -- 172.21.15.102:0/3869695838 <== mon.1 v1:172.21.15.181:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3940 (unknown 0 0 630920011) 0x7f0eac006810 con 0x7f0ebc027ce0 2021-05-25T12:05:36.476 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.412+0000 7f0ea9ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:05:36.476 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.412+0000 7f0ea9ffb700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:05:36.476 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.412+0000 7f0ea9ffb700 1 -- 172.21.15.102:0/3869695838 >> v1:172.21.15.181:6789/0 conn(0x7f0ebc027ce0 legacy=0x7f0ebc094520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:36.476 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.412+0000 7f0ea9ffb700 1 -- 172.21.15.102:0/3869695838 >> v1:172.21.15.181:6789/0 conn(0x7f0ebc027ce0 legacy=0x7f0ebc094520 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:05:36.477 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:05:36.477 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:05:36.477 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:05:36.478 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "peon", 2021-05-25T12:05:36.478 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 46, 2021-05-25T12:05:36.478 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:05:36.478 INFO:teuthology.orchestra.run.smithi102.stdout: 0, 2021-05-25T12:05:36.478 INFO:teuthology.orchestra.run.smithi102.stdout: 1, 2021-05-25T12:05:36.479 INFO:teuthology.orchestra.run.smithi102.stdout: 2 2021-05-25T12:05:36.479 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:05:36.479 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 3, 2021-05-25T12:05:36.479 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:05:36.479 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:05:36.480 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:05:36.480 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:05:36.480 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:05:36.480 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:05:36.480 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:05:36.481 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:05:36.481 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:05:36.481 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:05:36.481 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:05:36.482 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:05:36.482 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:05:36.482 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:05:36.482 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:05:36.482 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:05:36.483 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:05:36.483 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:05:36.483 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:05:36.483 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:05:36.483 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:05:36.484 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:05:36.484 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:05:36.484 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:05:36.484 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:05:36.484 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:05:36.485 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:05:36.485 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:05:36.485 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:05:36.485 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:05:36.485 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:05:36.486 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:05:36.486 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:05:36.486 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:05:36.486 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:05:36.486 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:05:36.487 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:05:36.487 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:05:36.487 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:05:36.487 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:05:36.487 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:05:36.488 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:05:36.488 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:05:36.488 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:05:36.488 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:05:36.488 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:05:36.489 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:05:36.489 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:05:36.489 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:05:36.489 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:05:36.489 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:05:36.490 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:05:36.490 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:05:36.490 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:05:36.490 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:05:36.490 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:05:36.491 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:05:36.491 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:05:36.491 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:05:36.491 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:05:36.491 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:05:36.492 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:05:36.492 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:05:36.492 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:05:36.492 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:05:36.492 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:05:36.493 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:05:36.493 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:05:36.493 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:05:36.493 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:05:36.494 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:05:36.494 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:05:36.494 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:05:36.494 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:05:36.495 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:05:36.495 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:05:36.495 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:05:36.495 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:05:36.495 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:05:36.496 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:05:36.496 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:05:36.496 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:05:36.496 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:05:36.496 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:05:36.497 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:05:36.497 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:05:36.497 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:05:36.497 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:05:36.497 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:05:36.498 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:05:36.498 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:05:36.498 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:05:36.498 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:05:36.499 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:05:36.499 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:05:36.499 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:05:36.499 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:05:36.499 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:05:36.500 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:05:36.500 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:05:36.500 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:05:36.500 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:05:36.500 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:05:36.501 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:05:36.501 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:05:36.501 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:05:36.501 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:05:36.502 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:05:36.502 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:05:36.502 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:05:36.502 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:05:36.502 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:05:36.503 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:05:36.503 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:05:36.503 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:05:36.503 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:05:36.503 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:05:36.504 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:05:36.504 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": [ 2021-05-25T12:05:36.504 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:05:36.504 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:05:36.505 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:05:36.505 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 4 2021-05-25T12:05:36.505 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:05:36.505 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:05:36.505 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:05:36.506 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:05:36.506 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:05:36.506 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:05:36.506 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 2 2021-05-25T12:05:36.507 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:05:36.507 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:05:36.507 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:05:36.507 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:05:36.507 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:05:36.509 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.413+0000 7f0ec2994700 10 librados: watch_flush enter 2021-05-25T12:05:36.510 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.413+0000 7f0ec2994700 10 librados: watch_flush exit 2021-05-25T12:05:36.510 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.413+0000 7f0ec2994700 20 client.194137.objecter shutdown clearing up homeless session... 2021-05-25T12:05:36.510 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.413+0000 7f0ec2994700 10 client.194137.objecter successfully canceled tick 2021-05-25T12:05:36.510 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.413+0000 7f0ec2994700 1 -- 172.21.15.102:0/3869695838 >> v1:172.21.15.181:6812/29582 conn(0x7f0e9c033590 legacy=0x7f0e9c035a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:36.511 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.413+0000 7f0ec2994700 10 monclient: shutdown 2021-05-25T12:05:36.511 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.413+0000 7f0ec2994700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:05:36.511 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.413+0000 7f0ec2994700 1 -- 172.21.15.102:0/3869695838 >> v1:172.21.15.181:6789/0 conn(0x7f0ebc098930 legacy=0x7f0ebc1284a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:36.511 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.413+0000 7f0ec2193700 1 -- 172.21.15.102:0/3869695838 reap_dead start 2021-05-25T12:05:36.511 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.413+0000 7f0ec2193700 1 -- 172.21.15.102:0/3869695838 reap_dead start 2021-05-25T12:05:36.512 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.413+0000 7f0ec2994700 1 -- 172.21.15.102:0/3869695838 shutdown_connections 2021-05-25T12:05:36.512 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.413+0000 7f0ec2994700 1 -- 172.21.15.102:0/3869695838 >> 172.21.15.102:0/3869695838 conn(0x7f0ebc09e9d0 msgr2=0x7f0ebc12a820 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:05:36.512 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.413+0000 7f0ec2994700 1 -- 172.21.15.102:0/3869695838 shutdown_connections 2021-05-25T12:05:36.512 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.413+0000 7f0ec2994700 1 -- 172.21.15.102:0/3869695838 wait complete. 2021-05-25T12:05:36.512 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.413+0000 7f0ec2994700 1 librados: shutdown 2021-05-25T12:05:36.513 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2021-05-25T12:05:36.513 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2021-05-25T12:05:36.623 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.621+0000 7f0e2359e700 10 monclient: get_monmap_and_config 2021-05-25T12:05:36.624 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.621+0000 7f0e2359e700 10 monclient: build_initial_monmap 2021-05-25T12:05:36.624 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.621+0000 7f0e2359e700 10 monclient: monmap: 2021-05-25T12:05:36.624 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:05:36.624 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:36.625 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:05:36.623157+0000 2021-05-25T12:05:36.625 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:05:36.623157+0000 2021-05-25T12:05:36.625 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:05:36.625 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:36.626 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:05:36.626 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:05:36.626 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:05:36.626 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:36.626 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.621+0000 7f0e2359e700 1 Processor -- start 2021-05-25T12:05:36.627 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.621+0000 7f0e2359e700 1 -- start start 2021-05-25T12:05:36.627 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.621+0000 7f0e2359e700 10 monclient: init 2021-05-25T12:05:36.637 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.622+0000 7f0e2359e700 10 monclient: _reopen_session rank -1 2021-05-25T12:05:36.637 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.622+0000 7f0e2359e700 10 monclient: _add_conns ranks=[0,1,2] 2021-05-25T12:05:36.637 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.622+0000 7f0e2359e700 10 monclient(hunting): picked mon.noname-a con 0x7f0e24095c40 addr v1:172.21.15.102:6789/0 2021-05-25T12:05:36.637 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.622+0000 7f0e2359e700 10 monclient(hunting): picked mon.noname-c con 0x7f0e24096020 addr v1:172.21.15.181:6789/0 2021-05-25T12:05:36.638 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.622+0000 7f0e2359e700 10 monclient(hunting): picked mon.noname-b con 0x7f0e2412e5c0 addr v1:172.21.15.102:6790/0 2021-05-25T12:05:36.638 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.622+0000 7f0e2359e700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e24092990 con 0x7f0e24095c40 2021-05-25T12:05:36.638 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.622+0000 7f0e2359e700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e24092c30 con 0x7f0e24096020 2021-05-25T12:05:36.638 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.622+0000 7f0e2359e700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e24092ed0 con 0x7f0e2412e5c0 2021-05-25T12:05:36.639 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.622+0000 7f0e2359e700 10 monclient(hunting): _renew_subs 2021-05-25T12:05:36.639 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.622+0000 7f0e2359e700 10 monclient(hunting): authenticate will time out at 1137.220505s 2021-05-25T12:05:36.639 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.622+0000 7f0e2259c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f0e24095c40 0x7f0e24093290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:32918/0 (socket says 172.21.15.102:32918) 2021-05-25T12:05:36.639 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.622+0000 7f0e22d9d700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f0e2412e5c0 0x7f0e241309b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:56032/0 (socket says 172.21.15.102:56032) 2021-05-25T12:05:36.640 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.622+0000 7f0e2259c700 1 -- 172.21.15.102:0/3922088955 learned_addr learned my addr 172.21.15.102:0/3922088955 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:05:36.640 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.622+0000 7f0e2159a700 1 -- 172.21.15.102:0/3922088955 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4024363957 0 0) 0x7f0e24092ed0 con 0x7f0e2412e5c0 2021-05-25T12:05:36.640 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.622+0000 7f0e2159a700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:36.640 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.622+0000 7f0e2159a700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:36.640 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.622+0000 7f0e2159a700 10 monclient(hunting): my global_id is 184340 2021-05-25T12:05:36.641 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.622+0000 7f0e2159a700 1 -- 172.21.15.102:0/3922088955 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e08003270 con 0x7f0e2412e5c0 2021-05-25T12:05:36.641 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.622+0000 7f0e2159a700 1 -- 172.21.15.102:0/3922088955 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3739851559 0 0) 0x7f0e24092990 con 0x7f0e24095c40 2021-05-25T12:05:36.641 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.622+0000 7f0e2159a700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:36.641 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.622+0000 7f0e2159a700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:36.641 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.622+0000 7f0e2159a700 10 monclient(hunting): my global_id is 174633 2021-05-25T12:05:36.642 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.622+0000 7f0e2159a700 1 -- 172.21.15.102:0/3922088955 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e24092ed0 con 0x7f0e24095c40 2021-05-25T12:05:36.642 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.622+0000 7f0e2159a700 1 -- 172.21.15.102:0/3922088955 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1752410937 0 0) 0x7f0e24092c30 con 0x7f0e24096020 2021-05-25T12:05:36.642 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.622+0000 7f0e2159a700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:36.642 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.622+0000 7f0e2159a700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:36.642 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.622+0000 7f0e2159a700 10 monclient(hunting): my global_id is 194146 2021-05-25T12:05:36.643 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.622+0000 7f0e2159a700 1 -- 172.21.15.102:0/3922088955 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e24092990 con 0x7f0e24096020 2021-05-25T12:05:36.643 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.623+0000 7f0e2159a700 1 -- 172.21.15.102:0/3922088955 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2797166642 0 0) 0x7f0e08003270 con 0x7f0e2412e5c0 2021-05-25T12:05:36.643 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.623+0000 7f0e2159a700 1 -- 172.21.15.102:0/3922088955 >> v1:172.21.15.181:6789/0 conn(0x7f0e24096020 legacy=0x7f0e240964d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:36.643 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.623+0000 7f0e2159a700 1 -- 172.21.15.102:0/3922088955 >> v1:172.21.15.102:6789/0 conn(0x7f0e24095c40 legacy=0x7f0e24093290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:36.643 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.623+0000 7f0e2159a700 10 monclient: _finish_hunting 0 2021-05-25T12:05:36.644 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.623+0000 7f0e2159a700 1 monclient: found mon.noname-b 2021-05-25T12:05:36.644 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.623+0000 7f0e2159a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:05:36.644 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.623+0000 7f0e2159a700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T12:05:36.644 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.623+0000 7f0e2159a700 1 -- 172.21.15.102:0/3922088955 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e24092430 con 0x7f0e2412e5c0 2021-05-25T12:05:36.644 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.623+0000 7f0e2159a700 10 monclient: _finish_auth 0 2021-05-25T12:05:36.645 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.623+0000 7f0e2159a700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:05:36.645 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.623+0000 7f0e2359e700 5 monclient: authenticate success, global_id 184340 2021-05-25T12:05:36.645 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.623+0000 7f0e2359e700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:05:36.645 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.623+0000 7f0e2159a700 1 -- 172.21.15.102:0/3922088955 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f0e14001910 con 0x7f0e2412e5c0 2021-05-25T12:05:36.645 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.623+0000 7f0e2159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:36.646 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.623+0000 7f0e2159a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:05:36.646 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.623+0000 7f0e2159a700 10 monclient: dump: 2021-05-25T12:05:36.646 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:36.646 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:36.647 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:36.647 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:36.647 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:36.647 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:36.647 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:36.648 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:36.648 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:36.648 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:36.648 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.623+0000 7f0e2359e700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:05:36.648 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.623+0000 7f0e2159a700 1 -- 172.21.15.102:0/3922088955 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0e14001b00 con 0x7f0e2412e5c0 2021-05-25T12:05:36.649 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.623+0000 7f0e2159a700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:05:36.649 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.623+0000 7f0e2359e700 10 monclient: get_monmap_and_config success 2021-05-25T12:05:36.649 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.623+0000 7f0e2159a700 1 -- 172.21.15.102:0/3922088955 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f0e14001e90 con 0x7f0e2412e5c0 2021-05-25T12:05:36.649 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.623+0000 7f0e2359e700 4 set_mon_vals no callback set 2021-05-25T12:05:36.649 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.623+0000 7f0e2159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:36.650 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.623+0000 7f0e2159a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:05:36.650 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.623+0000 7f0e2159a700 10 monclient: dump: 2021-05-25T12:05:36.650 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:36.650 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:36.650 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:36.651 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:36.651 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:36.651 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:36.651 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:36.651 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:36.652 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:36.652 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:36.652 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.623+0000 7f0e2359e700 10 monclient: shutdown 2021-05-25T12:05:36.652 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.623+0000 7f0e2359e700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:05:36.652 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.623+0000 7f0e2359e700 1 -- 172.21.15.102:0/3922088955 >> v1:172.21.15.102:6790/0 conn(0x7f0e2412e5c0 legacy=0x7f0e241309b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:36.653 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.623+0000 7f0e2359e700 1 -- 172.21.15.102:0/3922088955 shutdown_connections 2021-05-25T12:05:36.653 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.623+0000 7f0e2359e700 1 -- 172.21.15.102:0/3922088955 >> 172.21.15.102:0/3922088955 conn(0x7f0e240974b0 msgr2=0x7f0e240978c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:05:36.653 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.623+0000 7f0e2359e700 1 -- 172.21.15.102:0/3922088955 shutdown_connections 2021-05-25T12:05:36.653 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.623+0000 7f0e2359e700 1 -- 172.21.15.102:0/3922088955 wait complete. 2021-05-25T12:05:36.653 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.624+0000 7f0e2359e700 10 monclient: build_initial_monmap 2021-05-25T12:05:36.654 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.624+0000 7f0e2359e700 10 monclient: monmap: 2021-05-25T12:05:36.654 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:05:36.654 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:05:36.654 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:05:36.654 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:05:36.655 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:05:36.655 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:05:36.655 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:05:36.655 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:05:36.655 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:05:36.656 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:36.656 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.624+0000 7f0e2359e700 1 librados: starting msgr at 2021-05-25T12:05:36.656 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.624+0000 7f0e2359e700 1 librados: starting objecter 2021-05-25T12:05:36.656 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.624+0000 7f0e2359e700 1 Processor -- start 2021-05-25T12:05:36.656 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.624+0000 7f0e2359e700 1 -- start start 2021-05-25T12:05:36.657 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.624+0000 7f0e2359e700 1 librados: setting wanted keys 2021-05-25T12:05:36.657 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.624+0000 7f0e2359e700 1 librados: calling monclient init 2021-05-25T12:05:36.657 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.624+0000 7f0e2359e700 10 monclient: init 2021-05-25T12:05:36.657 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.624+0000 7f0e2359e700 10 monclient: _reopen_session rank -1 2021-05-25T12:05:36.657 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.624+0000 7f0e2359e700 10 monclient: _add_conns ranks=[0,2,1] 2021-05-25T12:05:36.658 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.624+0000 7f0e2359e700 10 monclient(hunting): picked mon.noname-a con 0x7f0e24095c40 addr v1:172.21.15.102:6789/0 2021-05-25T12:05:36.658 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.624+0000 7f0e2359e700 10 monclient(hunting): picked mon.noname-c con 0x7f0e24096020 addr v1:172.21.15.102:6790/0 2021-05-25T12:05:36.658 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.624+0000 7f0e2359e700 10 monclient(hunting): picked mon.noname-b con 0x7f0e2412e5c0 addr v1:172.21.15.181:6789/0 2021-05-25T12:05:36.658 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.625+0000 7f0e2359e700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e24114870 con 0x7f0e24095c40 2021-05-25T12:05:36.658 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.625+0000 7f0e2359e700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e24116e80 con 0x7f0e2412e5c0 2021-05-25T12:05:36.659 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.625+0000 7f0e2359e700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e24117120 con 0x7f0e24096020 2021-05-25T12:05:36.659 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.625+0000 7f0e2359e700 10 monclient(hunting): _renew_subs 2021-05-25T12:05:36.659 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.625+0000 7f0e2359e700 10 monclient(hunting): authenticate will time out at 1137.223031s 2021-05-25T12:05:36.659 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.625+0000 7f0e2259c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f0e24095c40 0x7f0e2413af50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:32924/0 (socket says 172.21.15.102:32924) 2021-05-25T12:05:36.660 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.625+0000 7f0e2259c700 1 -- 172.21.15.102:0/2022675484 learned_addr learned my addr 172.21.15.102:0/2022675484 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:05:36.660 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.625+0000 7f0e0e7fc700 10 client.?.objecter ms_handle_connect 0x7f0e24095c40 2021-05-25T12:05:36.660 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.625+0000 7f0e0e7fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:05:36.660 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.625+0000 7f0e0e7fc700 10 client.?.objecter ms_handle_connect 0x7f0e24096020 2021-05-25T12:05:36.660 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.625+0000 7f0e0e7fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:05:36.661 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.625+0000 7f0e0e7fc700 10 client.?.objecter ms_handle_connect 0x7f0e2412e5c0 2021-05-25T12:05:36.661 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.625+0000 7f0e0e7fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:05:36.661 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.625+0000 7f0e0e7fc700 1 -- 172.21.15.102:0/2022675484 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2515052064 0 0) 0x7f0e24114870 con 0x7f0e24095c40 2021-05-25T12:05:36.661 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.625+0000 7f0e0e7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:36.661 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.625+0000 7f0e0e7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:36.662 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.625+0000 7f0e0e7fc700 10 monclient(hunting): my global_id is 174636 2021-05-25T12:05:36.662 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.625+0000 7f0e0e7fc700 1 -- 172.21.15.102:0/2022675484 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e00004c10 con 0x7f0e24095c40 2021-05-25T12:05:36.662 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.625+0000 7f0e0e7fc700 1 -- 172.21.15.102:0/2022675484 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3772085941 0 0) 0x7f0e24117120 con 0x7f0e24096020 2021-05-25T12:05:36.662 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.625+0000 7f0e0e7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:36.662 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.625+0000 7f0e0e7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:36.663 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.625+0000 7f0e0e7fc700 10 monclient(hunting): my global_id is 184343 2021-05-25T12:05:36.663 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.625+0000 7f0e0e7fc700 1 -- 172.21.15.102:0/2022675484 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e24114870 con 0x7f0e24096020 2021-05-25T12:05:36.663 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.625+0000 7f0e0e7fc700 1 -- 172.21.15.102:0/2022675484 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1299793336 0 0) 0x7f0e24116e80 con 0x7f0e2412e5c0 2021-05-25T12:05:36.663 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.625+0000 7f0e0e7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:36.663 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.625+0000 7f0e0e7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:36.664 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.625+0000 7f0e0e7fc700 10 monclient(hunting): my global_id is 194149 2021-05-25T12:05:36.664 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.625+0000 7f0e0e7fc700 1 -- 172.21.15.102:0/2022675484 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e24117120 con 0x7f0e2412e5c0 2021-05-25T12:05:36.664 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.625+0000 7f0e0e7fc700 1 -- 172.21.15.102:0/2022675484 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4120445577 0 0) 0x7f0e00004c10 con 0x7f0e24095c40 2021-05-25T12:05:36.664 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.625+0000 7f0e0e7fc700 1 -- 172.21.15.102:0/2022675484 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0e24116e80 con 0x7f0e24095c40 2021-05-25T12:05:36.664 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.625+0000 7f0e0e7fc700 1 -- 172.21.15.102:0/2022675484 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f0e10001a50 con 0x7f0e24095c40 2021-05-25T12:05:36.665 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.625+0000 7f0e0e7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:36.665 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.625+0000 7f0e0e7fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:05:36.665 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.625+0000 7f0e0e7fc700 10 monclient(hunting): dump: 2021-05-25T12:05:36.665 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:36.665 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:36.666 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:36.666 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:36.666 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:36.666 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:36.666 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:36.667 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:36.667 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:36.667 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:36.667 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.625+0000 7f0e0e7fc700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:05:36.667 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.625+0000 7f0e0e7fc700 1 -- 172.21.15.102:0/2022675484 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1026552940 0 0) 0x7f0e24114870 con 0x7f0e24096020 2021-05-25T12:05:36.668 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.625+0000 7f0e0e7fc700 1 -- 172.21.15.102:0/2022675484 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0e00004c10 con 0x7f0e24096020 2021-05-25T12:05:36.668 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.625+0000 7f0e0e7fc700 1 -- 172.21.15.102:0/2022675484 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2445825757 0 0) 0x7f0e24117120 con 0x7f0e2412e5c0 2021-05-25T12:05:36.668 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.625+0000 7f0e0e7fc700 1 -- 172.21.15.102:0/2022675484 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0e24114870 con 0x7f0e2412e5c0 2021-05-25T12:05:36.668 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.625+0000 7f0e0e7fc700 1 -- 172.21.15.102:0/2022675484 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f0e18001840 con 0x7f0e24096020 2021-05-25T12:05:36.668 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.625+0000 7f0e0e7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:36.669 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.625+0000 7f0e0e7fc700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2021-05-25T12:05:36.669 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.625+0000 7f0e0e7fc700 10 monclient(hunting): dump: 2021-05-25T12:05:36.669 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:36.669 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:36.669 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:36.670 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:36.670 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:36.670 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:36.670 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:36.670 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:36.671 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:36.671 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:36.671 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.625+0000 7f0e0e7fc700 1 -- 172.21.15.102:0/2022675484 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f0e14001c70 con 0x7f0e2412e5c0 2021-05-25T12:05:36.671 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.625+0000 7f0e0e7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:36.672 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.625+0000 7f0e0e7fc700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2021-05-25T12:05:36.672 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.625+0000 7f0e0e7fc700 10 monclient(hunting): dump: 2021-05-25T12:05:36.672 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:36.672 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:36.672 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:36.673 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:36.673 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:36.673 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:36.673 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:36.674 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:36.674 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:36.674 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:36.674 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.625+0000 7f0e0e7fc700 1 -- 172.21.15.102:0/2022675484 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2358438331 0 0) 0x7f0e24116e80 con 0x7f0e24095c40 2021-05-25T12:05:36.674 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.625+0000 7f0e0e7fc700 1 -- 172.21.15.102:0/2022675484 >> v1:172.21.15.102:6790/0 conn(0x7f0e24096020 legacy=0x7f0e241286d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:36.675 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.625+0000 7f0e0e7fc700 1 -- 172.21.15.102:0/2022675484 >> v1:172.21.15.181:6789/0 conn(0x7f0e2412e5c0 legacy=0x7f0e24143300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:36.675 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.625+0000 7f0e0e7fc700 10 monclient: _finish_hunting 0 2021-05-25T12:05:36.675 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.625+0000 7f0e0e7fc700 1 monclient: found mon.a 2021-05-25T12:05:36.675 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.625+0000 7f0e0e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:05:36.675 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.625+0000 7f0e0e7fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:05:36.676 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.625+0000 7f0e0e7fc700 1 -- 172.21.15.102:0/2022675484 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e24115a90 con 0x7f0e24095c40 2021-05-25T12:05:36.676 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.625+0000 7f0e0e7fc700 10 monclient: _finish_auth 0 2021-05-25T12:05:36.676 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.625+0000 7f0e0e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:05:36.676 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.625+0000 7f0e2359e700 5 monclient: authenticate success, global_id 174636 2021-05-25T12:05:36.676 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.625+0000 7f0e2359e700 10 monclient: _renew_subs 2021-05-25T12:05:36.677 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.625+0000 7f0e2359e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:05:36.677 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.625+0000 7f0e2359e700 1 -- 172.21.15.102:0/2022675484 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e2413b660 con 0x7f0e24095c40 2021-05-25T12:05:36.677 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.625+0000 7f0e2359e700 10 client.174636.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:05:36.677 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.625+0000 7f0e2359e700 10 monclient: _renew_subs 2021-05-25T12:05:36.677 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.625+0000 7f0e2359e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:05:36.677 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.625+0000 7f0e2359e700 1 -- 172.21.15.102:0/2022675484 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e24128fa0 con 0x7f0e24095c40 2021-05-25T12:05:36.678 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.625+0000 7f0e2359e700 1 librados: init done 2021-05-25T12:05:36.678 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.625+0000 7f0e0e7fc700 1 -- 172.21.15.102:0/2022675484 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0e10001d80 con 0x7f0e24095c40 2021-05-25T12:05:36.678 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.625+0000 7f0e0e7fc700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:05:36.678 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.625+0000 7f0e0e7fc700 1 -- 172.21.15.102:0/2022675484 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f0e10004a00 con 0x7f0e24095c40 2021-05-25T12:05:36.679 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.625+0000 7f0e0e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:36.679 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.625+0000 7f0e20d99700 4 set_mon_vals no callback set 2021-05-25T12:05:36.679 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.625+0000 7f0e0e7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:05:36.679 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.625+0000 7f0e0e7fc700 10 monclient: dump: 2021-05-25T12:05:36.679 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:36.680 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:36.680 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:36.680 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:36.680 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:36.680 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:36.681 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:36.681 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:36.681 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:36.681 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:36.681 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.626+0000 7f0e0e7fc700 1 -- 172.21.15.102:0/2022675484 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f0e1000dda0 con 0x7f0e24095c40 2021-05-25T12:05:36.682 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.626+0000 7f0e2359e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:05:36.682 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.626+0000 7f0e2359e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:05:36.682 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.626+0000 7f0e2359e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:05:36.682 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.626+0000 7f0e2359e700 1 -- 172.21.15.102:0/2022675484 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e04004db0 con 0x7f0e24095c40 2021-05-25T12:05:36.682 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.626+0000 7f0e0e7fc700 1 -- 172.21.15.102:0/2022675484 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(168..168 src has 1..168) v4 ==== 7138+0+0 (unknown 3442091228 0 0) 0x7f0e1003dd60 con 0x7f0e24095c40 2021-05-25T12:05:36.683 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.626+0000 7f0e0e7fc700 10 client.174636.objecter ms_dispatch 0x7f0e24001550 osd_map(168..168 src has 1..168) v4 2021-05-25T12:05:36.683 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.626+0000 7f0e0e7fc700 3 client.174636.objecter handle_osd_map got epochs [168,168] > 0 2021-05-25T12:05:36.683 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.626+0000 7f0e0e7fc700 3 client.174636.objecter handle_osd_map decoding full epoch 168 2021-05-25T12:05:36.683 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.626+0000 7f0e0e7fc700 20 client.174636.objecter dump_active .. 0 homeless 2021-05-25T12:05:36.683 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.626+0000 7f0e21d9b700 10 monclient: get_auth_request con 0x7f0e00033570 auth_method 0 2021-05-25T12:05:36.684 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.627+0000 7f0e0e7fc700 10 client.174636.objecter ms_handle_connect 0x7f0e00033570 2021-05-25T12:05:36.684 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.630+0000 7f0e0e7fc700 1 -- 172.21.15.102:0/2022675484 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7f0e10012eb0 con 0x7f0e24095c40 2021-05-25T12:05:36.684 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.630+0000 7f0e0e7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:05:36.684 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.630+0000 7f0e0e7fc700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:05:36.864 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.862+0000 7f0e2359e700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2021-05-25T12:05:36.865 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.862+0000 7f0e2359e700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2021-05-25T12:05:36.865 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.862+0000 7f0e2359e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:05:36.865 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.862+0000 7f0e2359e700 1 -- 172.21.15.102:0/2022675484 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f0e04005870 con 0x7f0e24095c40 2021-05-25T12:05:36.866 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.863+0000 7f0e0e7fc700 1 -- 172.21.15.102:0/2022675484 <== mon.0 v1:172.21.15.102:6789/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7f0e10012aa0 con 0x7f0e24095c40 2021-05-25T12:05:36.866 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.863+0000 7f0e0e7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2021-05-25T12:05:36.867 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.863+0000 7f0e0e7fc700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:05:36.868 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:05:36.863+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:05:36.868 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:05:36.863+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=30,config=2,health=12,kv=6,logm=50} crc {auth=2974206599,config=3140514009,health=1738252192,kv=1184736763,logm=3466679210}) 2021-05-25T12:05:36.869 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:05:36.864+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=30,config=2,health=12,kv=6,logm=50} crc {auth=2241472568,config=3140514009,health=1738252192,kv=1184736763,logm=3024915335}) 2021-05-25T12:05:36.869 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:05:36.864+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:05:36.869 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:05:36.864+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=30,config=2,health=12,kv=6,logm=50} crc {auth=2974206599,config=3140514009,health=1738252192,kv=1184736763,logm=3466679210}) 2021-05-25T12:05:36.869 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:05:36.864+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=30,config=2,health=11,kv=6,logm=51} crc {auth=417708199,config=3140514009,health=2143125366,kv=1184736763,logm=1001100514}) 2021-05-25T12:05:36.870 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:05:36.864+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:05:36.870 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:05:36.864+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2838426249}) 2021-05-25T12:05:36.870 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:05:36.864+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3210543572}) 2021-05-25T12:05:36.870 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:05:36.864+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:05:36.870 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:05:36.864+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2838426249}) 2021-05-25T12:05:36.871 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:05:36.865+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4046345528}) 2021-05-25T12:05:36.872 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.865+0000 7f0e2359e700 10 librados: watch_flush enter 2021-05-25T12:05:36.873 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.865+0000 7f0e2359e700 10 librados: watch_flush exit 2021-05-25T12:05:36.873 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.865+0000 7f0e2359e700 20 client.174636.objecter shutdown clearing up homeless session... 2021-05-25T12:05:36.873 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.865+0000 7f0e2359e700 10 client.174636.objecter successfully canceled tick 2021-05-25T12:05:36.873 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.865+0000 7f0e2359e700 1 -- 172.21.15.102:0/2022675484 >> v1:172.21.15.181:6812/29582 conn(0x7f0e00033570 legacy=0x7f0e00035a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:36.873 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.865+0000 7f0e2359e700 10 monclient: shutdown 2021-05-25T12:05:36.874 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.865+0000 7f0e2359e700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:05:36.874 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.865+0000 7f0e2359e700 1 -- 172.21.15.102:0/2022675484 >> v1:172.21.15.102:6789/0 conn(0x7f0e24095c40 legacy=0x7f0e2413af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:36.874 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.865+0000 7f0e2359e700 1 -- 172.21.15.102:0/2022675484 shutdown_connections 2021-05-25T12:05:36.874 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.865+0000 7f0e2359e700 1 -- 172.21.15.102:0/2022675484 >> 172.21.15.102:0/2022675484 conn(0x7f0e240974b0 msgr2=0x7f0e2409a130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:05:36.875 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.865+0000 7f0e2359e700 1 -- 172.21.15.102:0/2022675484 shutdown_connections 2021-05-25T12:05:36.875 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.865+0000 7f0e2359e700 1 -- 172.21.15.102:0/2022675484 wait complete. 2021-05-25T12:05:36.875 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:36.865+0000 7f0e2359e700 1 librados: shutdown 2021-05-25T12:05:36.878 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:05:36.867+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:05:36.878 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:05:36.867+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1273027874}) 2021-05-25T12:05:36.878 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:05:36.867+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2756293999}) 2021-05-25T12:05:36.878 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:05:36.867+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:05:36.879 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:05:36.867+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1273027874}) 2021-05-25T12:05:36.879 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:05:36.867+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1339617649}) 2021-05-25T12:05:36.879 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:05:36.869+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:05:36.879 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:05:36.869+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=79,mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10} crc {logm=1516448776,mdsmap=3585761405,mgr=2996579194,mgr_command_descs=3824783157,mgr_metadata=1267308364,mgrstat=2702107619}) 2021-05-25T12:05:36.880 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:05:36.869+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=85,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=3} crc {logm=3812330895,mdsmap=1531151339,mgr=2996579194,mgr_command_descs=3824783157,mgr_metadata=1267308364,mgrstat=1713436065}) 2021-05-25T12:05:36.880 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:05:36.869+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:05:36.880 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:05:36.869+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=79,mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10} crc {logm=1516448776,mdsmap=3585761405,mgr=2996579194,mgr_command_descs=3824783157,mgr_metadata=1267308364,mgrstat=2702107619}) 2021-05-25T12:05:36.880 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:05:36.869+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=81,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=7} crc {logm=2362912692,mdsmap=1531151339,mgr=2996579194,mgr_command_descs=3824783157,mgr_metadata=1267308364,mgrstat=3847899108}) 2021-05-25T12:05:36.880 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:05:36.870+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:05:36.881 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:05:36.870+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgrstat=3,mon_config_key=3,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=26,osdmap=58} crc {mgrstat=231615299,mon_config_key=2093747287,monmap=3809614708,osd_metadata=725478305,osd_pg_creating=224012297,osd_snap=3637687236,osdmap=4178583955}) 2021-05-25T12:05:36.881 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:05:36.870+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=3,mon_config_key=4,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=26,osdmap=58} crc {mgrstat=231615299,mon_config_key=4118394475,monmap=2855167571,osd_metadata=101247178,osd_pg_creating=224012297,osd_snap=3679267829,osdmap=1853755859}) 2021-05-25T12:05:36.881 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:05:36.870+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:05:36.881 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:05:36.870+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgrstat=3,mon_config_key=3,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=26,osdmap=58} crc {mgrstat=231615299,mon_config_key=2093747287,monmap=3809614708,osd_metadata=725478305,osd_pg_creating=224012297,osd_snap=3637687236,osdmap=4178583955}) 2021-05-25T12:05:36.882 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:05:36.870+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgrstat=3,mon_config_key=3,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=28,osdmap=56} crc {mgrstat=231615299,mon_config_key=1945171658,monmap=3809614708,osd_metadata=725478305,osd_pg_creating=224012297,osd_snap=621851679,osdmap=250658408}) 2021-05-25T12:05:36.882 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:05:36.871+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:05:36.882 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:05:36.871+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=951467623}) 2021-05-25T12:05:36.882 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:05:36.871+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=483130373}) 2021-05-25T12:05:36.882 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:05:36.871+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:05:36.883 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:05:36.871+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=951467623}) 2021-05-25T12:05:36.883 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:05:36.871+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1978310730}) 2021-05-25T12:05:36.883 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:05:36.872+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:05:36.883 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:05:36.873+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2874503303}) 2021-05-25T12:05:36.884 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:05:36.873+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3631567402}) 2021-05-25T12:05:36.884 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:05:36.873+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:05:36.884 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:05:36.873+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2874503303}) 2021-05-25T12:05:36.884 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:05:36.873+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=95901983}) 2021-05-25T12:05:36.885 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2021-05-25T12:05:37.887 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2021-05-25T12:05:37.888 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2021-05-25T12:05:38.003 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.000+0000 7fe1e6bc8700 10 monclient: get_monmap_and_config 2021-05-25T12:05:38.003 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.000+0000 7fe1e6bc8700 10 monclient: build_initial_monmap 2021-05-25T12:05:38.003 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.000+0000 7fe1e6bc8700 10 monclient: monmap: 2021-05-25T12:05:38.004 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:05:38.004 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:38.004 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:05:38.002636+0000 2021-05-25T12:05:38.004 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:05:38.002636+0000 2021-05-25T12:05:38.005 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:05:38.005 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:38.005 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:05:38.005 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:05:38.005 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:05:38.006 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:38.006 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.001+0000 7fe1e6bc8700 1 Processor -- start 2021-05-25T12:05:38.015 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.001+0000 7fe1e6bc8700 1 -- start start 2021-05-25T12:05:38.016 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.001+0000 7fe1e6bc8700 10 monclient: init 2021-05-25T12:05:38.016 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.001+0000 7fe1e6bc8700 10 monclient: _reopen_session rank -1 2021-05-25T12:05:38.016 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.001+0000 7fe1e6bc8700 10 monclient: _add_conns ranks=[0,1,2] 2021-05-25T12:05:38.016 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.001+0000 7fe1e6bc8700 10 monclient(hunting): picked mon.noname-a con 0x7fe1e0063ea0 addr v1:172.21.15.102:6789/0 2021-05-25T12:05:38.017 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.001+0000 7fe1e6bc8700 10 monclient(hunting): picked mon.noname-c con 0x7fe1e0098060 addr v1:172.21.15.181:6789/0 2021-05-25T12:05:38.017 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.001+0000 7fe1e6bc8700 10 monclient(hunting): picked mon.noname-b con 0x7fe1e00984e0 addr v1:172.21.15.102:6790/0 2021-05-25T12:05:38.017 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.001+0000 7fe1e6bc8700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1e0064280 con 0x7fe1e0063ea0 2021-05-25T12:05:38.017 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.001+0000 7fe1e6bc8700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1e0099e10 con 0x7fe1e0098060 2021-05-25T12:05:38.017 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.001+0000 7fe1e6bc8700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1e009af10 con 0x7fe1e00984e0 2021-05-25T12:05:38.018 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.001+0000 7fe1e6bc8700 10 monclient(hunting): _renew_subs 2021-05-25T12:05:38.018 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.001+0000 7fe1e6bc8700 10 monclient(hunting): authenticate will time out at 1138.599945s 2021-05-25T12:05:38.018 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.002+0000 7fe1e5bc6700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fe1e0063ea0 0x7fe1e01287c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:32940/0 (socket says 172.21.15.102:32940) 2021-05-25T12:05:38.018 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.002+0000 7fe1e63c7700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fe1e00984e0 0x7fe1e0134230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:56054/0 (socket says 172.21.15.102:56054) 2021-05-25T12:05:38.018 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.002+0000 7fe1e5bc6700 1 -- 172.21.15.102:0/1228881966 learned_addr learned my addr 172.21.15.102:0/1228881966 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:05:38.019 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.002+0000 7fe1e4bc4700 1 -- 172.21.15.102:0/1228881966 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 249519503 0 0) 0x7fe1e009af10 con 0x7fe1e00984e0 2021-05-25T12:05:38.019 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.002+0000 7fe1e4bc4700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:38.019 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.002+0000 7fe1e4bc4700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:38.019 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.002+0000 7fe1e4bc4700 10 monclient(hunting): my global_id is 184349 2021-05-25T12:05:38.020 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.002+0000 7fe1e4bc4700 1 -- 172.21.15.102:0/1228881966 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1c8003270 con 0x7fe1e00984e0 2021-05-25T12:05:38.020 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.002+0000 7fe1e4bc4700 1 -- 172.21.15.102:0/1228881966 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2620775686 0 0) 0x7fe1e0064280 con 0x7fe1e0063ea0 2021-05-25T12:05:38.020 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.002+0000 7fe1e4bc4700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:38.020 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.002+0000 7fe1e4bc4700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:38.020 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.002+0000 7fe1e4bc4700 10 monclient(hunting): my global_id is 174645 2021-05-25T12:05:38.021 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.002+0000 7fe1e4bc4700 1 -- 172.21.15.102:0/1228881966 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1e009af10 con 0x7fe1e0063ea0 2021-05-25T12:05:38.021 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.002+0000 7fe1e4bc4700 1 -- 172.21.15.102:0/1228881966 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4134256955 0 0) 0x7fe1e0099e10 con 0x7fe1e0098060 2021-05-25T12:05:38.021 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.002+0000 7fe1e4bc4700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:38.021 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.002+0000 7fe1e4bc4700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:38.022 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.002+0000 7fe1e4bc4700 10 monclient(hunting): my global_id is 194158 2021-05-25T12:05:38.022 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.002+0000 7fe1e4bc4700 1 -- 172.21.15.102:0/1228881966 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1e0064280 con 0x7fe1e0098060 2021-05-25T12:05:38.022 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.002+0000 7fe1e4bc4700 1 -- 172.21.15.102:0/1228881966 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 594470992 0 0) 0x7fe1e009af10 con 0x7fe1e0063ea0 2021-05-25T12:05:38.022 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.002+0000 7fe1e4bc4700 1 -- 172.21.15.102:0/1228881966 >> v1:172.21.15.102:6790/0 conn(0x7fe1e00984e0 legacy=0x7fe1e0134230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:38.022 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.002+0000 7fe1e4bc4700 1 -- 172.21.15.102:0/1228881966 >> v1:172.21.15.181:6789/0 conn(0x7fe1e0098060 legacy=0x7fe1e0130b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:38.023 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.002+0000 7fe1e4bc4700 10 monclient: _finish_hunting 0 2021-05-25T12:05:38.023 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.002+0000 7fe1e4bc4700 1 monclient: found mon.noname-a 2021-05-25T12:05:38.023 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.002+0000 7fe1e4bc4700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:05:38.023 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.002+0000 7fe1e4bc4700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T12:05:38.023 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.002+0000 7fe1e4bc4700 1 -- 172.21.15.102:0/1228881966 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1e009b1b0 con 0x7fe1e0063ea0 2021-05-25T12:05:38.024 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.002+0000 7fe1e4bc4700 10 monclient: _finish_auth 0 2021-05-25T12:05:38.024 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.002+0000 7fe1e4bc4700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:05:38.024 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.002+0000 7fe1e6bc8700 5 monclient: authenticate success, global_id 174645 2021-05-25T12:05:38.024 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.002+0000 7fe1e4bc4700 1 -- 172.21.15.102:0/1228881966 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fe1d40015f0 con 0x7fe1e0063ea0 2021-05-25T12:05:38.025 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.002+0000 7fe1e6bc8700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:05:38.025 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.002+0000 7fe1e4bc4700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:38.025 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.002+0000 7fe1e4bc4700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:05:38.025 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.002+0000 7fe1e4bc4700 10 monclient: dump: 2021-05-25T12:05:38.026 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:38.026 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:38.026 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:38.026 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:38.027 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:38.027 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:38.027 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:38.027 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:38.027 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:38.028 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:38.028 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.002+0000 7fe1e6bc8700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:05:38.028 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.002+0000 7fe1e4bc4700 1 -- 172.21.15.102:0/1228881966 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe1d4001820 con 0x7fe1e0063ea0 2021-05-25T12:05:38.028 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.002+0000 7fe1e4bc4700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:05:38.028 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.002+0000 7fe1e6bc8700 10 monclient: get_monmap_and_config success 2021-05-25T12:05:38.029 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.002+0000 7fe1e4bc4700 1 -- 172.21.15.102:0/1228881966 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fe1d4001bb0 con 0x7fe1e0063ea0 2021-05-25T12:05:38.029 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.002+0000 7fe1e6bc8700 4 set_mon_vals no callback set 2021-05-25T12:05:38.029 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.002+0000 7fe1e4bc4700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:38.029 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.002+0000 7fe1e4bc4700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:05:38.029 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.002+0000 7fe1e4bc4700 10 monclient: dump: 2021-05-25T12:05:38.030 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:38.030 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:38.030 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:38.030 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:38.031 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:38.031 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:38.031 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:38.031 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:38.031 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:38.032 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:38.032 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.003+0000 7fe1e6bc8700 10 monclient: shutdown 2021-05-25T12:05:38.032 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.003+0000 7fe1e6bc8700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:05:38.032 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.003+0000 7fe1e6bc8700 1 -- 172.21.15.102:0/1228881966 >> v1:172.21.15.102:6789/0 conn(0x7fe1e0063ea0 legacy=0x7fe1e01287c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:38.032 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.003+0000 7fe1e6bc8700 1 -- 172.21.15.102:0/1228881966 shutdown_connections 2021-05-25T12:05:38.033 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.003+0000 7fe1e6bc8700 1 -- 172.21.15.102:0/1228881966 >> 172.21.15.102:0/1228881966 conn(0x7fe1e009eab0 msgr2=0x7fe1e009eec0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:05:38.033 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.003+0000 7fe1e6bc8700 1 -- 172.21.15.102:0/1228881966 shutdown_connections 2021-05-25T12:05:38.033 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.003+0000 7fe1e6bc8700 1 -- 172.21.15.102:0/1228881966 wait complete. 2021-05-25T12:05:38.033 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.003+0000 7fe1e6bc8700 10 monclient: build_initial_monmap 2021-05-25T12:05:38.034 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.003+0000 7fe1e6bc8700 10 monclient: monmap: 2021-05-25T12:05:38.034 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:05:38.034 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:05:38.034 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:05:38.034 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:05:38.035 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:05:38.035 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:05:38.035 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:05:38.035 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:05:38.035 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:05:38.036 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:38.036 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.003+0000 7fe1e6bc8700 1 librados: starting msgr at 2021-05-25T12:05:38.036 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.003+0000 7fe1e6bc8700 1 librados: starting objecter 2021-05-25T12:05:38.036 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.004+0000 7fe1e6bc8700 1 Processor -- start 2021-05-25T12:05:38.036 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.004+0000 7fe1e6bc8700 1 -- start start 2021-05-25T12:05:38.037 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.004+0000 7fe1e6bc8700 1 librados: setting wanted keys 2021-05-25T12:05:38.037 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.004+0000 7fe1e6bc8700 1 librados: calling monclient init 2021-05-25T12:05:38.037 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.004+0000 7fe1e6bc8700 10 monclient: init 2021-05-25T12:05:38.037 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.004+0000 7fe1e6bc8700 10 monclient: _reopen_session rank -1 2021-05-25T12:05:38.038 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.004+0000 7fe1e6bc8700 10 monclient: _add_conns ranks=[0,1,2] 2021-05-25T12:05:38.038 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.004+0000 7fe1e6bc8700 10 monclient(hunting): picked mon.noname-a con 0x7fe1e0063ea0 addr v1:172.21.15.102:6789/0 2021-05-25T12:05:38.038 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.004+0000 7fe1e5bc6700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fe1e0063ea0 0x7fe1e0127b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:32946/0 (socket says 172.21.15.102:32946) 2021-05-25T12:05:38.038 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.004+0000 7fe1e6bc8700 10 monclient(hunting): picked mon.noname-b con 0x7fe1e0098060 addr v1:172.21.15.181:6789/0 2021-05-25T12:05:38.038 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.004+0000 7fe1e5bc6700 1 -- 172.21.15.102:0/2894561478 learned_addr learned my addr 172.21.15.102:0/2894561478 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:05:38.039 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.004+0000 7fe1e6bc8700 10 monclient(hunting): picked mon.noname-c con 0x7fe1e00984e0 addr v1:172.21.15.102:6790/0 2021-05-25T12:05:38.039 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.004+0000 7fe1e6bc8700 1 -- 172.21.15.102:0/2894561478 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1e0126ff0 con 0x7fe1e0063ea0 2021-05-25T12:05:38.039 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.004+0000 7fe1e6bc8700 1 -- 172.21.15.102:0/2894561478 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1e0126110 con 0x7fe1e0098060 2021-05-25T12:05:38.039 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.004+0000 7fe1e6bc8700 1 -- 172.21.15.102:0/2894561478 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1e0127220 con 0x7fe1e00984e0 2021-05-25T12:05:38.040 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.004+0000 7fe1e6bc8700 10 monclient(hunting): _renew_subs 2021-05-25T12:05:38.040 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.004+0000 7fe1e6bc8700 10 monclient(hunting): authenticate will time out at 1138.602709s 2021-05-25T12:05:38.040 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.004+0000 7fe1cdffb700 10 client.?.objecter ms_handle_connect 0x7fe1e0063ea0 2021-05-25T12:05:38.040 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.004+0000 7fe1cdffb700 10 client.?.objecter resend_mon_ops 2021-05-25T12:05:38.040 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.004+0000 7fe1cdffb700 10 client.?.objecter ms_handle_connect 0x7fe1e00984e0 2021-05-25T12:05:38.041 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.004+0000 7fe1cdffb700 10 client.?.objecter resend_mon_ops 2021-05-25T12:05:38.041 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.004+0000 7fe1cdffb700 1 -- 172.21.15.102:0/2894561478 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3113070698 0 0) 0x7fe1e0126ff0 con 0x7fe1e0063ea0 2021-05-25T12:05:38.041 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.004+0000 7fe1cdffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:38.041 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.004+0000 7fe1cdffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:38.041 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.004+0000 7fe1cdffb700 10 monclient(hunting): my global_id is 174648 2021-05-25T12:05:38.041 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.004+0000 7fe1cdffb700 1 -- 172.21.15.102:0/2894561478 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1c0004c10 con 0x7fe1e0063ea0 2021-05-25T12:05:38.042 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.004+0000 7fe1cdffb700 1 -- 172.21.15.102:0/2894561478 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2146796377 0 0) 0x7fe1e0127220 con 0x7fe1e00984e0 2021-05-25T12:05:38.042 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.004+0000 7fe1cdffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:38.042 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.004+0000 7fe1cdffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:38.042 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.004+0000 7fe1cdffb700 10 monclient(hunting): my global_id is 184352 2021-05-25T12:05:38.043 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.004+0000 7fe1cdffb700 1 -- 172.21.15.102:0/2894561478 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1e0126ff0 con 0x7fe1e00984e0 2021-05-25T12:05:38.043 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.004+0000 7fe1cdffb700 10 client.?.objecter ms_handle_connect 0x7fe1e0098060 2021-05-25T12:05:38.043 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.004+0000 7fe1cdffb700 10 client.?.objecter resend_mon_ops 2021-05-25T12:05:38.043 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.005+0000 7fe1cdffb700 1 -- 172.21.15.102:0/2894561478 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2106331796 0 0) 0x7fe1c0004c10 con 0x7fe1e0063ea0 2021-05-25T12:05:38.043 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.005+0000 7fe1cdffb700 1 -- 172.21.15.102:0/2894561478 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe1e0127220 con 0x7fe1e0063ea0 2021-05-25T12:05:38.044 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.005+0000 7fe1cdffb700 1 -- 172.21.15.102:0/2894561478 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fe1d4001c30 con 0x7fe1e0063ea0 2021-05-25T12:05:38.044 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.005+0000 7fe1cdffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:38.044 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.005+0000 7fe1cdffb700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:05:38.044 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.005+0000 7fe1cdffb700 10 monclient(hunting): dump: 2021-05-25T12:05:38.045 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:38.045 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:38.045 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:38.045 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:38.046 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:38.046 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:38.046 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:38.046 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:38.046 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:38.047 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:38.047 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.005+0000 7fe1cdffb700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:05:38.047 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.005+0000 7fe1cdffb700 1 -- 172.21.15.102:0/2894561478 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3863372508 0 0) 0x7fe1e0126ff0 con 0x7fe1e00984e0 2021-05-25T12:05:38.048 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.005+0000 7fe1cdffb700 1 -- 172.21.15.102:0/2894561478 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe1c0004c10 con 0x7fe1e00984e0 2021-05-25T12:05:38.048 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.005+0000 7fe1cdffb700 1 -- 172.21.15.102:0/2894561478 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fe1dc001540 con 0x7fe1e00984e0 2021-05-25T12:05:38.048 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.005+0000 7fe1cdffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:38.048 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.005+0000 7fe1cdffb700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2021-05-25T12:05:38.048 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.005+0000 7fe1cdffb700 10 monclient(hunting): dump: 2021-05-25T12:05:38.049 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:38.049 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:38.049 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:38.049 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:38.049 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:38.050 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:38.050 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:38.050 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:38.050 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:38.050 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:38.051 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.005+0000 7fe1cdffb700 1 -- 172.21.15.102:0/2894561478 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3753351402 0 0) 0x7fe1e0127220 con 0x7fe1e0063ea0 2021-05-25T12:05:38.051 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.005+0000 7fe1cdffb700 1 -- 172.21.15.102:0/2894561478 >> v1:172.21.15.102:6790/0 conn(0x7fe1e00984e0 legacy=0x7fe1e01430c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:38.051 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.005+0000 7fe1cdffb700 1 -- 172.21.15.102:0/2894561478 >> v1:172.21.15.181:6789/0 conn(0x7fe1e0098060 legacy=0x7fe1e0120520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:38.051 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.005+0000 7fe1cdffb700 10 monclient: _finish_hunting 0 2021-05-25T12:05:38.051 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.005+0000 7fe1cdffb700 1 monclient: found mon.a 2021-05-25T12:05:38.052 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.005+0000 7fe1cdffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:05:38.052 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.005+0000 7fe1cdffb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:05:38.052 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.005+0000 7fe1cdffb700 1 -- 172.21.15.102:0/2894561478 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1e01274c0 con 0x7fe1e0063ea0 2021-05-25T12:05:38.052 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.005+0000 7fe1cdffb700 10 monclient: _finish_auth 0 2021-05-25T12:05:38.052 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.005+0000 7fe1cdffb700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:05:38.053 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.005+0000 7fe1e6bc8700 5 monclient: authenticate success, global_id 174648 2021-05-25T12:05:38.053 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.005+0000 7fe1e6bc8700 10 monclient: _renew_subs 2021-05-25T12:05:38.053 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.005+0000 7fe1e6bc8700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:05:38.053 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.005+0000 7fe1e6bc8700 1 -- 172.21.15.102:0/2894561478 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1e0125350 con 0x7fe1e0063ea0 2021-05-25T12:05:38.054 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.005+0000 7fe1cdffb700 1 -- 172.21.15.102:0/2894561478 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe1d4004960 con 0x7fe1e0063ea0 2021-05-25T12:05:38.054 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.005+0000 7fe1cdffb700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:05:38.054 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.005+0000 7fe1cdffb700 1 -- 172.21.15.102:0/2894561478 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fe1d4004c70 con 0x7fe1e0063ea0 2021-05-25T12:05:38.054 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.005+0000 7fe1cdffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:38.054 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.005+0000 7fe1cdffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:05:38.055 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.005+0000 7fe1cdffb700 10 monclient: dump: 2021-05-25T12:05:38.055 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:38.055 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:38.055 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:38.056 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:38.056 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:38.056 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:38.056 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:38.056 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:38.057 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:38.057 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:38.057 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.005+0000 7fe1cffff700 4 set_mon_vals no callback set 2021-05-25T12:05:38.057 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.005+0000 7fe1e6bc8700 10 client.174648.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:05:38.057 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.005+0000 7fe1e6bc8700 10 monclient: _renew_subs 2021-05-25T12:05:38.058 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.005+0000 7fe1e6bc8700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:05:38.058 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.005+0000 7fe1e6bc8700 1 -- 172.21.15.102:0/2894561478 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1e0120df0 con 0x7fe1e0063ea0 2021-05-25T12:05:38.058 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.005+0000 7fe1e6bc8700 1 librados: init done 2021-05-25T12:05:38.058 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.006+0000 7fe1cdffb700 1 -- 172.21.15.102:0/2894561478 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7fe1d4004e50 con 0x7fe1e0063ea0 2021-05-25T12:05:38.059 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.006+0000 7fe1cdffb700 1 -- 172.21.15.102:0/2894561478 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(170..170 src has 1..170) v4 ==== 6493+0+0 (unknown 2891575432 0 0) 0x7fe1d403dcb0 con 0x7fe1e0063ea0 2021-05-25T12:05:38.059 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.006+0000 7fe1cdffb700 10 client.174648.objecter ms_dispatch 0x7fe1e0001550 osd_map(170..170 src has 1..170) v4 2021-05-25T12:05:38.059 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.006+0000 7fe1cdffb700 3 client.174648.objecter handle_osd_map got epochs [170,170] > 0 2021-05-25T12:05:38.059 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.006+0000 7fe1cdffb700 3 client.174648.objecter handle_osd_map decoding full epoch 170 2021-05-25T12:05:38.059 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.006+0000 7fe1cdffb700 20 client.174648.objecter dump_active .. 0 homeless 2021-05-25T12:05:38.060 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.006+0000 7fe1e53c5700 10 monclient: get_auth_request con 0x7fe1c0033310 auth_method 0 2021-05-25T12:05:38.060 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.006+0000 7fe1e6bc8700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:05:38.060 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.006+0000 7fe1e6bc8700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:05:38.060 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.006+0000 7fe1e6bc8700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:05:38.060 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.006+0000 7fe1e6bc8700 1 -- 172.21.15.102:0/2894561478 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1e01210a0 con 0x7fe1e0063ea0 2021-05-25T12:05:38.061 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.006+0000 7fe1cdffb700 10 client.174648.objecter ms_handle_connect 0x7fe1c0033310 2021-05-25T12:05:38.061 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.010+0000 7fe1cdffb700 1 -- 172.21.15.102:0/2894561478 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7fe1e01210a0 con 0x7fe1e0063ea0 2021-05-25T12:05:38.061 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.010+0000 7fe1cdffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:05:38.061 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.010+0000 7fe1cdffb700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:05:38.231 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.228+0000 7fe1e6bc8700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2021-05-25T12:05:38.231 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.228+0000 7fe1e6bc8700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2021-05-25T12:05:38.231 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.228+0000 7fe1e6bc8700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:05:38.231 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.228+0000 7fe1e6bc8700 1 -- 172.21.15.102:0/2894561478 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fe1e01210a0 con 0x7fe1e0063ea0 2021-05-25T12:05:38.232 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.228+0000 7fe1cdffb700 1 -- 172.21.15.102:0/2894561478 <== mon.0 v1:172.21.15.102:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1342 (unknown 254746657 0 4235619086) 0x7fe1e01210a0 con 0x7fe1e0063ea0 2021-05-25T12:05:38.232 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.228+0000 7fe1cdffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2021-05-25T12:05:38.232 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.228+0000 7fe1cdffb700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:05:38.233 INFO:teuthology.orchestra.run.smithi102.stdout:{"election_epoch":46,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":5,"features":{"quorum_con":"4540138303579357183","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"20ee91b6-8935-4b98-b868-85c03789ccc4","modified":"2021-05-25T11:59:11.011522Z","created":"2021-05-25T11:59:11.011522Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6790","nonce":0}]},"addr":"172.21.15.102:6790/0","public_addr":"172.21.15.102:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2021-05-25T12:05:38.234 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.230+0000 7fe1e6bc8700 10 librados: watch_flush enter 2021-05-25T12:05:38.234 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.230+0000 7fe1e6bc8700 10 librados: watch_flush exit 2021-05-25T12:05:38.234 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.230+0000 7fe1e6bc8700 20 client.174648.objecter shutdown clearing up homeless session... 2021-05-25T12:05:38.235 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.230+0000 7fe1e6bc8700 10 client.174648.objecter successfully canceled tick 2021-05-25T12:05:38.235 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.230+0000 7fe1e6bc8700 1 -- 172.21.15.102:0/2894561478 >> v1:172.21.15.181:6812/29582 conn(0x7fe1c0033310 legacy=0x7fe1c00357d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:38.235 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.230+0000 7fe1e6bc8700 10 monclient: shutdown 2021-05-25T12:05:38.235 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.230+0000 7fe1e6bc8700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:05:38.236 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.230+0000 7fe1e6bc8700 1 -- 172.21.15.102:0/2894561478 >> v1:172.21.15.102:6789/0 conn(0x7fe1e0063ea0 legacy=0x7fe1e0127b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:38.236 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.230+0000 7fe1e6bc8700 1 -- 172.21.15.102:0/2894561478 shutdown_connections 2021-05-25T12:05:38.236 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.230+0000 7fe1e6bc8700 1 -- 172.21.15.102:0/2894561478 >> 172.21.15.102:0/2894561478 conn(0x7fe1e009eab0 msgr2=0x7fe1e009c8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:05:38.236 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.230+0000 7fe1e6bc8700 1 -- 172.21.15.102:0/2894561478 shutdown_connections 2021-05-25T12:05:38.236 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.230+0000 7fe1e6bc8700 1 -- 172.21.15.102:0/2894561478 wait complete. 2021-05-25T12:05:38.237 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.230+0000 7fe1e6bc8700 1 librados: shutdown 2021-05-25T12:05:38.243 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2021-05-25T12:05:38.244 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2021-05-25T12:05:38.244 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2021-05-25T12:05:38.353 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.351+0000 7f90b9081700 10 monclient: get_monmap_and_config 2021-05-25T12:05:38.353 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.351+0000 7f90b9081700 10 monclient: build_initial_monmap 2021-05-25T12:05:38.353 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.351+0000 7f90b9081700 10 monclient: monmap: 2021-05-25T12:05:38.354 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:05:38.354 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:38.354 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:05:38.352940+0000 2021-05-25T12:05:38.354 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:05:38.352940+0000 2021-05-25T12:05:38.354 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:05:38.355 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:38.355 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:05:38.355 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:05:38.355 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:05:38.356 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:38.367 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.351+0000 7f90b9081700 1 Processor -- start 2021-05-25T12:05:38.367 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.351+0000 7f90b9081700 1 -- start start 2021-05-25T12:05:38.368 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.351+0000 7f90b9081700 10 monclient: init 2021-05-25T12:05:38.368 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.351+0000 7f90b9081700 10 monclient: _reopen_session rank -1 2021-05-25T12:05:38.368 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.351+0000 7f90b9081700 10 monclient: _add_conns ranks=[2,0,1] 2021-05-25T12:05:38.368 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.351+0000 7f90b9081700 10 monclient(hunting): picked mon.noname-b con 0x7f90b4090990 addr v1:172.21.15.102:6790/0 2021-05-25T12:05:38.369 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.351+0000 7f90b9081700 10 monclient(hunting): picked mon.noname-a con 0x7f90b4090d70 addr v1:172.21.15.102:6789/0 2021-05-25T12:05:38.369 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.352+0000 7f90b9081700 10 monclient(hunting): picked mon.noname-c con 0x7f90b412e330 addr v1:172.21.15.181:6789/0 2021-05-25T12:05:38.369 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.352+0000 7f90b9081700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90b4096fa0 con 0x7f90b4090d70 2021-05-25T12:05:38.369 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.352+0000 7f90b9081700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90b40926a0 con 0x7f90b412e330 2021-05-25T12:05:38.369 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.352+0000 7f90b9081700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90b4092940 con 0x7f90b4090990 2021-05-25T12:05:38.370 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.352+0000 7f90b9081700 10 monclient(hunting): _renew_subs 2021-05-25T12:05:38.370 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.352+0000 7f90b9081700 10 monclient(hunting): authenticate will time out at 1138.950040s 2021-05-25T12:05:38.370 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.352+0000 7f90b2d9d700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f90b4090990 0x7f90b4093240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:56064/0 (socket says 172.21.15.102:56064) 2021-05-25T12:05:38.370 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.352+0000 7f90b2d9d700 1 -- 172.21.15.102:0/1307459169 learned_addr learned my addr 172.21.15.102:0/1307459169 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:05:38.371 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.352+0000 7f90b259c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f90b4090d70 0x7f90b4091200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:32954/0 (socket says 172.21.15.102:32954) 2021-05-25T12:05:38.371 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.352+0000 7f90b1d9b700 1 -- 172.21.15.102:0/1307459169 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 479562645 0 0) 0x7f90b4092940 con 0x7f90b4090990 2021-05-25T12:05:38.371 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.352+0000 7f90b1d9b700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:38.371 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.352+0000 7f90b1d9b700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:38.372 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.352+0000 7f90b1d9b700 10 monclient(hunting): my global_id is 184355 2021-05-25T12:05:38.372 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.352+0000 7f90b1d9b700 1 -- 172.21.15.102:0/1307459169 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90a0003270 con 0x7f90b4090990 2021-05-25T12:05:38.372 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.352+0000 7f90b1d9b700 1 -- 172.21.15.102:0/1307459169 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3190655714 0 0) 0x7f90b4096fa0 con 0x7f90b4090d70 2021-05-25T12:05:38.372 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.352+0000 7f90b1d9b700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:38.372 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.352+0000 7f90b1d9b700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:38.373 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.352+0000 7f90b1d9b700 10 monclient(hunting): my global_id is 174651 2021-05-25T12:05:38.373 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.352+0000 7f90b1d9b700 1 -- 172.21.15.102:0/1307459169 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90b4092940 con 0x7f90b4090d70 2021-05-25T12:05:38.373 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.352+0000 7f90b1d9b700 1 -- 172.21.15.102:0/1307459169 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3495337816 0 0) 0x7f90a0003270 con 0x7f90b4090990 2021-05-25T12:05:38.373 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.352+0000 7f90b1d9b700 1 -- 172.21.15.102:0/1307459169 >> v1:172.21.15.181:6789/0 conn(0x7f90b412e330 legacy=0x7f90b4130720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:38.373 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.352+0000 7f90b1d9b700 1 -- 172.21.15.102:0/1307459169 >> v1:172.21.15.102:6789/0 conn(0x7f90b4090d70 legacy=0x7f90b4091200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:38.374 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.352+0000 7f90b1d9b700 10 monclient: _finish_hunting 0 2021-05-25T12:05:38.374 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.352+0000 7f90b1d9b700 1 monclient: found mon.noname-b 2021-05-25T12:05:38.374 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.352+0000 7f90b1d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:05:38.374 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.352+0000 7f90b1d9b700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T12:05:38.375 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.352+0000 7f90b1d9b700 1 -- 172.21.15.102:0/1307459169 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90b4092be0 con 0x7f90b4090990 2021-05-25T12:05:38.375 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.352+0000 7f90b1d9b700 10 monclient: _finish_auth 0 2021-05-25T12:05:38.375 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.352+0000 7f90b1d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:05:38.375 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.352+0000 7f90b9081700 5 monclient: authenticate success, global_id 184355 2021-05-25T12:05:38.375 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.352+0000 7f90b1d9b700 1 -- 172.21.15.102:0/1307459169 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f90a8001610 con 0x7f90b4090990 2021-05-25T12:05:38.376 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.352+0000 7f90b1d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:38.376 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.352+0000 7f90b1d9b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:05:38.376 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.352+0000 7f90b1d9b700 10 monclient: dump: 2021-05-25T12:05:38.376 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:38.376 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:38.377 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:38.377 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:38.377 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:38.377 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:38.378 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:38.378 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:38.378 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:38.378 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:38.379 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.352+0000 7f90b9081700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:05:38.379 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.352+0000 7f90b1d9b700 1 -- 172.21.15.102:0/1307459169 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f90a80017c0 con 0x7f90b4090990 2021-05-25T12:05:38.379 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.352+0000 7f90b1d9b700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:05:38.379 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.352+0000 7f90b9081700 10 monclient: get_monmap_and_config success 2021-05-25T12:05:38.380 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.352+0000 7f90b1d9b700 1 -- 172.21.15.102:0/1307459169 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f90a8001b50 con 0x7f90b4090990 2021-05-25T12:05:38.380 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.352+0000 7f90b1d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:38.380 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.352+0000 7f90b9081700 4 set_mon_vals no callback set 2021-05-25T12:05:38.380 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.352+0000 7f90b1d9b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:05:38.380 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.352+0000 7f90b1d9b700 10 monclient: dump: 2021-05-25T12:05:38.381 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:38.381 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:38.381 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:38.381 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:38.382 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:38.382 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:38.382 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:38.382 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:38.382 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:38.383 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:38.383 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.353+0000 7f90b9081700 10 monclient: shutdown 2021-05-25T12:05:38.383 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.353+0000 7f90b9081700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:05:38.383 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.353+0000 7f90b9081700 1 -- 172.21.15.102:0/1307459169 >> v1:172.21.15.102:6790/0 conn(0x7f90b4090990 legacy=0x7f90b4093240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:38.384 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.353+0000 7f90b9081700 1 -- 172.21.15.102:0/1307459169 shutdown_connections 2021-05-25T12:05:38.384 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.353+0000 7f90b9081700 1 -- 172.21.15.102:0/1307459169 >> 172.21.15.102:0/1307459169 conn(0x7f90b409b370 msgr2=0x7f90b4097430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:05:38.384 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.353+0000 7f90b9081700 1 -- 172.21.15.102:0/1307459169 shutdown_connections 2021-05-25T12:05:38.384 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.353+0000 7f90b9081700 1 -- 172.21.15.102:0/1307459169 wait complete. 2021-05-25T12:05:38.384 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.353+0000 7f90b9081700 10 monclient: build_initial_monmap 2021-05-25T12:05:38.385 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.353+0000 7f90b9081700 10 monclient: monmap: 2021-05-25T12:05:38.385 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:05:38.385 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:05:38.385 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:05:38.385 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:05:38.386 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:05:38.386 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:05:38.386 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:05:38.386 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:05:38.387 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:05:38.387 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:38.387 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.354+0000 7f90b9081700 1 librados: starting msgr at 2021-05-25T12:05:38.387 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.354+0000 7f90b9081700 1 librados: starting objecter 2021-05-25T12:05:38.388 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.354+0000 7f90b9081700 1 Processor -- start 2021-05-25T12:05:38.388 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.354+0000 7f90b9081700 1 -- start start 2021-05-25T12:05:38.388 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.354+0000 7f90b9081700 1 librados: setting wanted keys 2021-05-25T12:05:38.388 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.354+0000 7f90b9081700 1 librados: calling monclient init 2021-05-25T12:05:38.389 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.354+0000 7f90b9081700 10 monclient: init 2021-05-25T12:05:38.389 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.354+0000 7f90b9081700 10 monclient: _reopen_session rank -1 2021-05-25T12:05:38.389 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.354+0000 7f90b9081700 10 monclient: _add_conns ranks=[2,1,0] 2021-05-25T12:05:38.389 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.354+0000 7f90b9081700 10 monclient(hunting): picked mon.noname-c con 0x7f90b4090990 addr v1:172.21.15.102:6790/0 2021-05-25T12:05:38.390 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.354+0000 7f90b9081700 10 monclient(hunting): picked mon.noname-b con 0x7f90b4090d70 addr v1:172.21.15.181:6789/0 2021-05-25T12:05:38.390 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.354+0000 7f90b9081700 10 monclient(hunting): picked mon.noname-a con 0x7f90b412e330 addr v1:172.21.15.102:6789/0 2021-05-25T12:05:38.390 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.354+0000 7f90b9081700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90b4137620 con 0x7f90b412e330 2021-05-25T12:05:38.390 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.354+0000 7f90b9081700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90b40676f0 con 0x7f90b4090d70 2021-05-25T12:05:38.390 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.354+0000 7f90b9081700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90b4067990 con 0x7f90b4090990 2021-05-25T12:05:38.391 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.354+0000 7f90b9081700 10 monclient(hunting): _renew_subs 2021-05-25T12:05:38.391 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.354+0000 7f90b9081700 10 monclient(hunting): authenticate will time out at 1138.952734s 2021-05-25T12:05:38.391 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.354+0000 7f90b2d9d700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f90b4090990 0x7f90b4136800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:56072/0 (socket says 172.21.15.102:56072) 2021-05-25T12:05:38.391 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.354+0000 7f90b359e700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f90b412e330 0x7f90b413ed80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:32964/0 (socket says 172.21.15.102:32964) 2021-05-25T12:05:38.392 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.354+0000 7f90b2d9d700 1 -- 172.21.15.102:0/296105262 learned_addr learned my addr 172.21.15.102:0/296105262 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:05:38.392 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.354+0000 7f909affd700 10 client.?.objecter ms_handle_connect 0x7f90b412e330 2021-05-25T12:05:38.392 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.354+0000 7f909affd700 10 client.?.objecter resend_mon_ops 2021-05-25T12:05:38.392 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.354+0000 7f909affd700 10 client.?.objecter ms_handle_connect 0x7f90b4090990 2021-05-25T12:05:38.392 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.354+0000 7f909affd700 10 client.?.objecter resend_mon_ops 2021-05-25T12:05:38.393 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.355+0000 7f909affd700 1 -- 172.21.15.102:0/296105262 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2496229008 0 0) 0x7f90b4137620 con 0x7f90b412e330 2021-05-25T12:05:38.393 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.355+0000 7f909affd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:38.393 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.355+0000 7f909affd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:38.393 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.355+0000 7f909affd700 10 monclient(hunting): my global_id is 174654 2021-05-25T12:05:38.394 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.355+0000 7f909affd700 1 -- 172.21.15.102:0/296105262 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9094004c10 con 0x7f90b412e330 2021-05-25T12:05:38.394 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.355+0000 7f909affd700 1 -- 172.21.15.102:0/296105262 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2941062515 0 0) 0x7f90b4067990 con 0x7f90b4090990 2021-05-25T12:05:38.394 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.355+0000 7f909affd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:38.394 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.355+0000 7f909affd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:38.394 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.355+0000 7f909affd700 10 monclient(hunting): my global_id is 184358 2021-05-25T12:05:38.395 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.355+0000 7f909affd700 1 -- 172.21.15.102:0/296105262 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90b4137620 con 0x7f90b4090990 2021-05-25T12:05:38.395 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.355+0000 7f909affd700 10 client.?.objecter ms_handle_connect 0x7f90b4090d70 2021-05-25T12:05:38.395 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.355+0000 7f909affd700 10 client.?.objecter resend_mon_ops 2021-05-25T12:05:38.395 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.355+0000 7f909affd700 1 -- 172.21.15.102:0/296105262 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2154279932 0 0) 0x7f90b40676f0 con 0x7f90b4090d70 2021-05-25T12:05:38.396 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.355+0000 7f909affd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:38.396 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.355+0000 7f909affd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:38.396 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.355+0000 7f909affd700 10 monclient(hunting): my global_id is 194164 2021-05-25T12:05:38.396 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.355+0000 7f909affd700 1 -- 172.21.15.102:0/296105262 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90b4067990 con 0x7f90b4090d70 2021-05-25T12:05:38.397 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.355+0000 7f909affd700 1 -- 172.21.15.102:0/296105262 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2606653563 0 0) 0x7f9094004c10 con 0x7f90b412e330 2021-05-25T12:05:38.397 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.355+0000 7f909affd700 1 -- 172.21.15.102:0/296105262 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f90b40676f0 con 0x7f90b412e330 2021-05-25T12:05:38.397 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.355+0000 7f909affd700 1 -- 172.21.15.102:0/296105262 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3359731152 0 0) 0x7f90b4137620 con 0x7f90b4090990 2021-05-25T12:05:38.397 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.355+0000 7f909affd700 1 -- 172.21.15.102:0/296105262 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9094004c10 con 0x7f90b4090990 2021-05-25T12:05:38.398 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.355+0000 7f909affd700 1 -- 172.21.15.102:0/296105262 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f90a4001bc0 con 0x7f90b412e330 2021-05-25T12:05:38.398 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.355+0000 7f909affd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:38.398 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.355+0000 7f909affd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:05:38.398 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.355+0000 7f909affd700 10 monclient(hunting): dump: 2021-05-25T12:05:38.398 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:38.399 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:38.399 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:38.399 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:38.399 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:38.400 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:38.400 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:38.400 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:38.400 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:38.401 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:38.401 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.355+0000 7f909affd700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:05:38.401 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.355+0000 7f909affd700 1 -- 172.21.15.102:0/296105262 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f90a80015d0 con 0x7f90b4090990 2021-05-25T12:05:38.401 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.355+0000 7f909affd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:38.401 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.355+0000 7f909affd700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2021-05-25T12:05:38.402 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.355+0000 7f909affd700 10 monclient(hunting): dump: 2021-05-25T12:05:38.402 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:38.402 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:38.402 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:38.403 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:38.403 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:38.403 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:38.403 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:38.403 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:38.404 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:38.404 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:38.404 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.355+0000 7f909affd700 1 -- 172.21.15.102:0/296105262 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2803503485 0 0) 0x7f90b40676f0 con 0x7f90b412e330 2021-05-25T12:05:38.404 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.355+0000 7f909affd700 1 -- 172.21.15.102:0/296105262 >> v1:172.21.15.102:6790/0 conn(0x7f90b4090990 legacy=0x7f90b4136800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:38.405 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.355+0000 7f909affd700 1 -- 172.21.15.102:0/296105262 >> v1:172.21.15.181:6789/0 conn(0x7f90b4090d70 legacy=0x7f90b4136f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:38.405 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.355+0000 7f909affd700 10 monclient: _finish_hunting 0 2021-05-25T12:05:38.405 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.355+0000 7f909affd700 1 monclient: found mon.a 2021-05-25T12:05:38.405 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.355+0000 7f909affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:05:38.406 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.355+0000 7f909affd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:05:38.406 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.355+0000 7f909affd700 1 -- 172.21.15.102:0/296105262 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90b4065840 con 0x7f90b412e330 2021-05-25T12:05:38.406 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.355+0000 7f909affd700 10 monclient: _finish_auth 0 2021-05-25T12:05:38.406 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.355+0000 7f909affd700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:05:38.407 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.355+0000 7f90b9081700 5 monclient: authenticate success, global_id 174654 2021-05-25T12:05:38.407 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.355+0000 7f90b9081700 10 monclient: _renew_subs 2021-05-25T12:05:38.407 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.355+0000 7f90b9081700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:05:38.407 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.355+0000 7f90b9081700 1 -- 172.21.15.102:0/296105262 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90b4067ff0 con 0x7f90b412e330 2021-05-25T12:05:38.407 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.355+0000 7f90b9081700 10 client.174654.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:05:38.408 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.355+0000 7f90b9081700 10 monclient: _renew_subs 2021-05-25T12:05:38.408 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.355+0000 7f90b9081700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:05:38.408 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.355+0000 7f90b9081700 1 -- 172.21.15.102:0/296105262 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f90b4063ec0 con 0x7f90b412e330 2021-05-25T12:05:38.408 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.355+0000 7f90b9081700 1 librados: init done 2021-05-25T12:05:38.408 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.355+0000 7f909affd700 1 -- 172.21.15.102:0/296105262 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f90a40021b0 con 0x7f90b412e330 2021-05-25T12:05:38.409 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.355+0000 7f909affd700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:05:38.409 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.355+0000 7f909affd700 1 -- 172.21.15.102:0/296105262 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f90a4002540 con 0x7f90b412e330 2021-05-25T12:05:38.409 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.355+0000 7f909affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:38.409 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.355+0000 7f90b159a700 4 set_mon_vals no callback set 2021-05-25T12:05:38.410 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.355+0000 7f909affd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:05:38.410 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.355+0000 7f909affd700 10 monclient: dump: 2021-05-25T12:05:38.410 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:38.410 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:38.410 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:38.411 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:38.411 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:38.411 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:38.411 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:38.412 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:38.412 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:38.412 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:38.412 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.356+0000 7f90b9081700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:05:38.413 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.356+0000 7f90b9081700 1 -- 172.21.15.102:0/296105262 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f908c0065a0 con 0x7f908c002840 2021-05-25T12:05:38.413 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.356+0000 7f909affd700 1 -- 172.21.15.102:0/296105262 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f90a4002720 con 0x7f90b412e330 2021-05-25T12:05:38.413 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.356+0000 7f909affd700 10 client.174654.objecter ms_handle_connect 0x7f908c002840 2021-05-25T12:05:38.413 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.356+0000 7f909affd700 10 client.174654.objecter resend_mon_ops 2021-05-25T12:05:38.413 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.356+0000 7f909affd700 1 -- 172.21.15.102:0/296105262 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(170..170 src has 1..170) v4 ==== 6493+0+0 (unknown 2891575432 0 0) 0x7f90a403dbe0 con 0x7f90b412e330 2021-05-25T12:05:38.414 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.356+0000 7f909affd700 10 client.174654.objecter ms_dispatch 0x7f90b4001550 osd_map(170..170 src has 1..170) v4 2021-05-25T12:05:38.414 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.356+0000 7f909affd700 3 client.174654.objecter handle_osd_map got epochs [170,170] > 0 2021-05-25T12:05:38.414 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.356+0000 7f909affd700 3 client.174654.objecter handle_osd_map decoding full epoch 170 2021-05-25T12:05:38.414 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.356+0000 7f909affd700 20 client.174654.objecter dump_active .. 0 homeless 2021-05-25T12:05:38.415 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.356+0000 7f909affd700 1 -- 172.21.15.102:0/296105262 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2466476551 0 0) 0x7f908c0065a0 con 0x7f908c002840 2021-05-25T12:05:38.415 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.356+0000 7f909affd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:38.415 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.356+0000 7f909affd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:38.415 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.356+0000 7f909affd700 10 monclient(hunting): my global_id is 174657 2021-05-25T12:05:38.416 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.356+0000 7f909affd700 1 -- 172.21.15.102:0/296105262 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90b40676f0 con 0x7f908c002840 2021-05-25T12:05:38.416 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.356+0000 7f90b259c700 10 monclient: get_auth_request con 0x7f9094033420 auth_method 0 2021-05-25T12:05:38.416 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.356+0000 7f909affd700 1 -- 172.21.15.102:0/296105262 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2804391967 0 0) 0x7f90b40676f0 con 0x7f908c002840 2021-05-25T12:05:38.416 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.356+0000 7f909affd700 1 -- 172.21.15.102:0/296105262 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f908c0065a0 con 0x7f908c002840 2021-05-25T12:05:38.416 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.356+0000 7f909affd700 1 -- 172.21.15.102:0/296105262 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f90a8004750 con 0x7f908c002840 2021-05-25T12:05:38.417 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.356+0000 7f909affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:38.417 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.356+0000 7f909affd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:05:38.417 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.356+0000 7f909affd700 10 monclient: dump: 2021-05-25T12:05:38.417 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:38.418 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:38.418 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:38.418 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:38.418 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:38.418 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:38.419 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:38.419 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:38.419 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:38.419 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:38.420 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.356+0000 7f909affd700 1 -- 172.21.15.102:0/296105262 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1434566345 0 0) 0x7f908c0065a0 con 0x7f908c002840 2021-05-25T12:05:38.420 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.356+0000 7f909affd700 1 -- 172.21.15.102:0/296105262 --> v1:172.21.15.102:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f908c009e40 con 0x7f908c002840 2021-05-25T12:05:38.420 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.356+0000 7f909affd700 10 client.174654.objecter ms_handle_connect 0x7f9094033420 2021-05-25T12:05:38.420 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.357+0000 7f909affd700 1 -- 172.21.15.102:0/296105262 <== mon.0 v1:172.21.15.102:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7f90a8006b00 con 0x7f908c002840 2021-05-25T12:05:38.420 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.357+0000 7f909affd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:05:38.421 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.357+0000 7f909affd700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:05:38.421 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.357+0000 7f909affd700 1 -- 172.21.15.102:0/296105262 >> v1:172.21.15.102:6789/0 conn(0x7f908c002840 legacy=0x7f908c004ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:38.421 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.357+0000 7f909affd700 1 -- 172.21.15.102:0/296105262 >> v1:172.21.15.102:6789/0 conn(0x7f908c002840 legacy=0x7f908c004ce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:05:38.421 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.363+0000 7f90b9081700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:05:38.421 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.363+0000 7f90b9081700 1 -- 172.21.15.102:0/296105262 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f908c007540 con 0x7f908c0068e0 2021-05-25T12:05:38.422 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.363+0000 7f909affd700 10 client.174654.objecter ms_handle_connect 0x7f908c0068e0 2021-05-25T12:05:38.422 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.363+0000 7f909affd700 10 client.174654.objecter resend_mon_ops 2021-05-25T12:05:38.422 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.364+0000 7f909affd700 1 -- 172.21.15.102:0/296105262 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2196784153 0 0) 0x7f908c007540 con 0x7f908c0068e0 2021-05-25T12:05:38.422 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.364+0000 7f909affd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:38.423 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.364+0000 7f909affd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:38.423 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.364+0000 7f909affd700 10 monclient(hunting): my global_id is 174660 2021-05-25T12:05:38.423 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.364+0000 7f909affd700 1 -- 172.21.15.102:0/296105262 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f908c0065a0 con 0x7f908c0068e0 2021-05-25T12:05:38.423 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.364+0000 7f909affd700 1 -- 172.21.15.102:0/296105262 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 660713181 0 0) 0x7f908c0065a0 con 0x7f908c0068e0 2021-05-25T12:05:38.424 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.364+0000 7f909affd700 1 -- 172.21.15.102:0/296105262 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f908c007540 con 0x7f908c0068e0 2021-05-25T12:05:38.424 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.364+0000 7f909affd700 1 -- 172.21.15.102:0/296105262 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f90a8004f40 con 0x7f908c0068e0 2021-05-25T12:05:38.424 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.364+0000 7f909affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:38.424 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.364+0000 7f909affd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:05:38.424 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.364+0000 7f909affd700 10 monclient: dump: 2021-05-25T12:05:38.425 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:38.425 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:38.425 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:38.425 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:38.425 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:38.426 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:38.426 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:38.426 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:38.426 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:38.427 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:38.427 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.364+0000 7f909affd700 1 -- 172.21.15.102:0/296105262 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3461104640 0 0) 0x7f908c007540 con 0x7f908c0068e0 2021-05-25T12:05:38.427 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.364+0000 7f909affd700 1 -- 172.21.15.102:0/296105262 --> v1:172.21.15.102:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f908c008770 con 0x7f908c0068e0 2021-05-25T12:05:38.427 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.364+0000 7f909affd700 1 -- 172.21.15.102:0/296105262 <== mon.0 v1:172.21.15.102:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3940 (unknown 0 0 2381883871) 0x7f90a8006400 con 0x7f908c0068e0 2021-05-25T12:05:38.428 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.364+0000 7f909affd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:05:38.428 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.364+0000 7f909affd700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:05:38.428 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.364+0000 7f909affd700 1 -- 172.21.15.102:0/296105262 >> v1:172.21.15.102:6789/0 conn(0x7f908c0068e0 legacy=0x7f908c006d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:38.428 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.364+0000 7f909affd700 1 -- 172.21.15.102:0/296105262 >> v1:172.21.15.102:6789/0 conn(0x7f908c0068e0 legacy=0x7f908c006d70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:05:38.429 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:05:38.429 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:05:38.429 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:05:38.429 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "peon", 2021-05-25T12:05:38.430 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 46, 2021-05-25T12:05:38.430 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:05:38.430 INFO:teuthology.orchestra.run.smithi102.stdout: 0, 2021-05-25T12:05:38.430 INFO:teuthology.orchestra.run.smithi102.stdout: 1, 2021-05-25T12:05:38.430 INFO:teuthology.orchestra.run.smithi102.stdout: 2 2021-05-25T12:05:38.431 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:05:38.431 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 5, 2021-05-25T12:05:38.431 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:05:38.431 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:05:38.432 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:05:38.432 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:05:38.432 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:05:38.432 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:05:38.432 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:05:38.433 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:05:38.433 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:05:38.433 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:05:38.433 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:05:38.433 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:05:38.434 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:05:38.434 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:05:38.434 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:05:38.434 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:05:38.434 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:05:38.435 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:05:38.435 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:05:38.435 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:05:38.435 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:05:38.436 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:05:38.436 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:05:38.436 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:05:38.436 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:05:38.436 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:05:38.437 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:05:38.437 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:05:38.437 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:05:38.437 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:05:38.437 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:05:38.438 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:05:38.438 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:05:38.438 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:05:38.438 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:05:38.439 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:05:38.439 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:05:38.439 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:05:38.439 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:05:38.439 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:05:38.440 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:05:38.440 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:05:38.440 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:05:38.440 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:05:38.440 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:05:38.441 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:05:38.441 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:05:38.441 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:05:38.441 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:05:38.441 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:05:38.442 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:05:38.442 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:05:38.442 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:05:38.442 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:05:38.442 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:05:38.443 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:05:38.443 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:05:38.443 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:05:38.443 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:05:38.443 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:05:38.444 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:05:38.444 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:05:38.444 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:05:38.444 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:05:38.445 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:05:38.445 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:05:38.445 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:05:38.445 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:05:38.445 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:05:38.446 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:05:38.446 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:05:38.446 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:05:38.446 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:05:38.446 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:05:38.447 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:05:38.447 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:05:38.447 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:05:38.447 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:05:38.447 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:05:38.448 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:05:38.448 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:05:38.448 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:05:38.448 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:05:38.448 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:05:38.448 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:05:38.449 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:05:38.449 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:05:38.449 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:05:38.449 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:05:38.450 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:05:38.450 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:05:38.450 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:05:38.450 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:05:38.450 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:05:38.451 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:05:38.451 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:05:38.451 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:05:38.451 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:05:38.451 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:05:38.452 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:05:38.452 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:05:38.452 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:05:38.452 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:05:38.452 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:05:38.453 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:05:38.453 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:05:38.453 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:05:38.453 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:05:38.453 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:05:38.454 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:05:38.454 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:05:38.454 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:05:38.454 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:05:38.455 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:05:38.455 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:05:38.455 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:05:38.455 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:05:38.455 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:05:38.456 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": [ 2021-05-25T12:05:38.456 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:05:38.456 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:05:38.456 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:05:38.457 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 2 2021-05-25T12:05:38.457 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:05:38.457 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:05:38.457 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:05:38.457 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:05:38.458 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:05:38.458 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:05:38.458 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 7 2021-05-25T12:05:38.458 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:05:38.458 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:05:38.459 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:05:38.459 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:05:38.459 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:05:38.461 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.364+0000 7f90b9081700 10 librados: watch_flush enter 2021-05-25T12:05:38.461 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.364+0000 7f90b9081700 10 librados: watch_flush exit 2021-05-25T12:05:38.462 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.364+0000 7f90b9081700 20 client.174654.objecter shutdown clearing up homeless session... 2021-05-25T12:05:38.462 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.364+0000 7f90b9081700 10 client.174654.objecter successfully canceled tick 2021-05-25T12:05:38.462 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.364+0000 7f90b9081700 1 -- 172.21.15.102:0/296105262 >> v1:172.21.15.181:6812/29582 conn(0x7f9094033420 legacy=0x7f90940358e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:38.462 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.364+0000 7f90b9081700 10 monclient: shutdown 2021-05-25T12:05:38.463 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.364+0000 7f90b9081700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:05:38.463 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.364+0000 7f90b9081700 1 -- 172.21.15.102:0/296105262 >> v1:172.21.15.102:6789/0 conn(0x7f90b412e330 legacy=0x7f90b413ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:38.463 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.364+0000 7f90b359e700 1 -- 172.21.15.102:0/296105262 reap_dead start 2021-05-25T12:05:38.463 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.364+0000 7f90b359e700 1 -- 172.21.15.102:0/296105262 reap_dead start 2021-05-25T12:05:38.464 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.364+0000 7f90b9081700 1 -- 172.21.15.102:0/296105262 shutdown_connections 2021-05-25T12:05:38.464 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.364+0000 7f90b9081700 1 -- 172.21.15.102:0/296105262 >> 172.21.15.102:0/296105262 conn(0x7f90b409b370 msgr2=0x7f90b41294b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:05:38.464 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.364+0000 7f90b9081700 1 -- 172.21.15.102:0/296105262 shutdown_connections 2021-05-25T12:05:38.464 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.365+0000 7f90b9081700 1 -- 172.21.15.102:0/296105262 wait complete. 2021-05-25T12:05:38.465 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.365+0000 7f90b9081700 1 librados: shutdown 2021-05-25T12:05:38.465 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2021-05-25T12:05:38.575 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.572+0000 7f48c7ba3700 10 monclient: get_monmap_and_config 2021-05-25T12:05:38.576 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.572+0000 7f48c7ba3700 10 monclient: build_initial_monmap 2021-05-25T12:05:38.576 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.572+0000 7f48c7ba3700 10 monclient: monmap: 2021-05-25T12:05:38.576 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:05:38.576 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:38.577 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:05:38.574849+0000 2021-05-25T12:05:38.577 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:05:38.574849+0000 2021-05-25T12:05:38.577 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:05:38.577 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:38.578 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:05:38.578 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:05:38.578 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:05:38.578 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:38.579 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.573+0000 7f48c7ba3700 1 Processor -- start 2021-05-25T12:05:38.579 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.573+0000 7f48c7ba3700 1 -- start start 2021-05-25T12:05:38.579 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.573+0000 7f48c7ba3700 10 monclient: init 2021-05-25T12:05:38.593 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.574+0000 7f48c7ba3700 10 monclient: _reopen_session rank -1 2021-05-25T12:05:38.593 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.574+0000 7f48c7ba3700 10 monclient: _add_conns ranks=[0,2,1] 2021-05-25T12:05:38.594 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.574+0000 7f48c7ba3700 10 monclient(hunting): picked mon.noname-a con 0x7f48c0068f60 addr v1:172.21.15.102:6789/0 2021-05-25T12:05:38.594 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.574+0000 7f48c7ba3700 10 monclient(hunting): picked mon.noname-b con 0x7f48c0096800 addr v1:172.21.15.102:6790/0 2021-05-25T12:05:38.594 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.574+0000 7f48c7ba3700 10 monclient(hunting): picked mon.noname-c con 0x7f48c0131ae0 addr v1:172.21.15.181:6789/0 2021-05-25T12:05:38.594 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.574+0000 7f48c7ba3700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48c0129040 con 0x7f48c0068f60 2021-05-25T12:05:38.595 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.574+0000 7f48c7ba3700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48c0097360 con 0x7f48c0131ae0 2021-05-25T12:05:38.595 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.574+0000 7f48c7ba3700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48c0069a50 con 0x7f48c0096800 2021-05-25T12:05:38.595 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.574+0000 7f48c7ba3700 10 monclient(hunting): _renew_subs 2021-05-25T12:05:38.595 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.574+0000 7f48c7ba3700 10 monclient(hunting): authenticate will time out at 1139.172123s 2021-05-25T12:05:38.596 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.574+0000 7f48c593f700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f48c0068f60 0x7f48c0128930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:32972/0 (socket says 172.21.15.102:32972) 2021-05-25T12:05:38.596 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.574+0000 7f48c513e700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f48c0096800 0x7f48c0069340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:56082/0 (socket says 172.21.15.102:56082) 2021-05-25T12:05:38.596 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.574+0000 7f48c593f700 1 -- 172.21.15.102:0/2360459274 learned_addr learned my addr 172.21.15.102:0/2360459274 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:05:38.596 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.574+0000 7f48c493d700 1 -- 172.21.15.102:0/2360459274 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 577314183 0 0) 0x7f48c0069a50 con 0x7f48c0096800 2021-05-25T12:05:38.596 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.574+0000 7f48c493d700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:38.597 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.574+0000 7f48c493d700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:38.597 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.574+0000 7f48c493d700 10 monclient(hunting): my global_id is 184361 2021-05-25T12:05:38.597 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.574+0000 7f48c493d700 1 -- 172.21.15.102:0/2360459274 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48a8003270 con 0x7f48c0096800 2021-05-25T12:05:38.597 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.574+0000 7f48c493d700 1 -- 172.21.15.102:0/2360459274 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 425677273 0 0) 0x7f48c0129040 con 0x7f48c0068f60 2021-05-25T12:05:38.598 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.574+0000 7f48c493d700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:38.598 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.574+0000 7f48c493d700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:38.598 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.574+0000 7f48c493d700 10 monclient(hunting): my global_id is 174663 2021-05-25T12:05:38.598 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.574+0000 7f48c493d700 1 -- 172.21.15.102:0/2360459274 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48c0069a50 con 0x7f48c0068f60 2021-05-25T12:05:38.598 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.574+0000 7f48c493d700 1 -- 172.21.15.102:0/2360459274 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1588060121 0 0) 0x7f48c0097360 con 0x7f48c0131ae0 2021-05-25T12:05:38.599 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.574+0000 7f48c493d700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:38.599 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.574+0000 7f48c493d700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:38.599 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.574+0000 7f48c493d700 10 monclient(hunting): my global_id is 194167 2021-05-25T12:05:38.599 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.574+0000 7f48c493d700 1 -- 172.21.15.102:0/2360459274 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48c0129040 con 0x7f48c0131ae0 2021-05-25T12:05:38.600 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.574+0000 7f48c493d700 1 -- 172.21.15.102:0/2360459274 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 871103794 0 0) 0x7f48a8003270 con 0x7f48c0096800 2021-05-25T12:05:38.600 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.574+0000 7f48c493d700 1 -- 172.21.15.102:0/2360459274 >> v1:172.21.15.181:6789/0 conn(0x7f48c0131ae0 legacy=0x7f48c0133ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:38.600 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.574+0000 7f48c493d700 1 -- 172.21.15.102:0/2360459274 >> v1:172.21.15.102:6789/0 conn(0x7f48c0068f60 legacy=0x7f48c0128930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:38.600 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.574+0000 7f48c493d700 10 monclient: _finish_hunting 0 2021-05-25T12:05:38.601 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.574+0000 7f48c493d700 1 monclient: found mon.noname-b 2021-05-25T12:05:38.601 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.574+0000 7f48c493d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:05:38.601 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.574+0000 7f48c493d700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T12:05:38.601 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.574+0000 7f48c493d700 1 -- 172.21.15.102:0/2360459274 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48c0061ce0 con 0x7f48c0096800 2021-05-25T12:05:38.602 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.574+0000 7f48c493d700 10 monclient: _finish_auth 0 2021-05-25T12:05:38.602 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.574+0000 7f48c493d700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:05:38.602 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.574+0000 7f48c493d700 1 -- 172.21.15.102:0/2360459274 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f48b4001610 con 0x7f48c0096800 2021-05-25T12:05:38.602 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.574+0000 7f48c7ba3700 5 monclient: authenticate success, global_id 184361 2021-05-25T12:05:38.602 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.574+0000 7f48c7ba3700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:05:38.603 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.574+0000 7f48c493d700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:38.603 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.574+0000 7f48c493d700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:05:38.603 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.574+0000 7f48c493d700 10 monclient: dump: 2021-05-25T12:05:38.603 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:38.604 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:38.604 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:38.604 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:38.604 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:38.605 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:38.605 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:38.605 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:38.605 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:38.606 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:38.606 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.574+0000 7f48c7ba3700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:05:38.606 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.574+0000 7f48c493d700 1 -- 172.21.15.102:0/2360459274 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f48b40018f0 con 0x7f48c0096800 2021-05-25T12:05:38.606 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.574+0000 7f48c493d700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:05:38.607 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.574+0000 7f48c7ba3700 10 monclient: get_monmap_and_config success 2021-05-25T12:05:38.607 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.574+0000 7f48c493d700 1 -- 172.21.15.102:0/2360459274 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f48b4001c80 con 0x7f48c0096800 2021-05-25T12:05:38.607 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.574+0000 7f48c493d700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:38.607 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.574+0000 7f48c7ba3700 4 set_mon_vals no callback set 2021-05-25T12:05:38.608 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.574+0000 7f48c493d700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:05:38.608 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.574+0000 7f48c493d700 10 monclient: dump: 2021-05-25T12:05:38.608 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:38.608 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:38.609 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:38.609 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:38.609 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:38.609 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:38.610 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:38.610 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:38.610 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:38.610 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:38.611 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.575+0000 7f48c7ba3700 10 monclient: shutdown 2021-05-25T12:05:38.611 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.575+0000 7f48c7ba3700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:05:38.611 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.575+0000 7f48c7ba3700 1 -- 172.21.15.102:0/2360459274 >> v1:172.21.15.102:6790/0 conn(0x7f48c0096800 legacy=0x7f48c0069340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:38.611 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.575+0000 7f48c7ba3700 1 -- 172.21.15.102:0/2360459274 shutdown_connections 2021-05-25T12:05:38.612 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.575+0000 7f48c7ba3700 1 -- 172.21.15.102:0/2360459274 >> 172.21.15.102:0/2360459274 conn(0x7f48c009b920 msgr2=0x7f48c009bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:05:38.612 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.575+0000 7f48c7ba3700 1 -- 172.21.15.102:0/2360459274 shutdown_connections 2021-05-25T12:05:38.612 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.575+0000 7f48c7ba3700 1 -- 172.21.15.102:0/2360459274 wait complete. 2021-05-25T12:05:38.612 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.575+0000 7f48c7ba3700 10 monclient: build_initial_monmap 2021-05-25T12:05:38.613 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.575+0000 7f48c7ba3700 10 monclient: monmap: 2021-05-25T12:05:38.613 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:05:38.613 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:05:38.613 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:05:38.614 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:05:38.614 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:05:38.614 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:05:38.614 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:05:38.614 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:05:38.615 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:05:38.615 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:38.615 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.576+0000 7f48c7ba3700 1 librados: starting msgr at 2021-05-25T12:05:38.615 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.576+0000 7f48c7ba3700 1 librados: starting objecter 2021-05-25T12:05:38.616 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.576+0000 7f48c7ba3700 1 Processor -- start 2021-05-25T12:05:38.616 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.576+0000 7f48c7ba3700 1 -- start start 2021-05-25T12:05:38.616 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.576+0000 7f48c7ba3700 1 librados: setting wanted keys 2021-05-25T12:05:38.616 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.576+0000 7f48c7ba3700 1 librados: calling monclient init 2021-05-25T12:05:38.617 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.576+0000 7f48c7ba3700 10 monclient: init 2021-05-25T12:05:38.617 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.576+0000 7f48c7ba3700 10 monclient: _reopen_session rank -1 2021-05-25T12:05:38.617 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.576+0000 7f48c7ba3700 10 monclient: _add_conns ranks=[0,2,1] 2021-05-25T12:05:38.617 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.576+0000 7f48c7ba3700 10 monclient(hunting): picked mon.noname-a con 0x7f48c0068f60 addr v1:172.21.15.102:6789/0 2021-05-25T12:05:38.618 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.576+0000 7f48c7ba3700 10 monclient(hunting): picked mon.noname-c con 0x7f48c0096800 addr v1:172.21.15.102:6790/0 2021-05-25T12:05:38.618 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.576+0000 7f48c7ba3700 10 monclient(hunting): picked mon.noname-b con 0x7f48c0131ae0 addr v1:172.21.15.181:6789/0 2021-05-25T12:05:38.618 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.576+0000 7f48c7ba3700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48c0121350 con 0x7f48c0068f60 2021-05-25T12:05:38.618 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.576+0000 7f48c7ba3700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48c0123290 con 0x7f48c0131ae0 2021-05-25T12:05:38.619 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.576+0000 7f48c7ba3700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48c0096ca0 con 0x7f48c0096800 2021-05-25T12:05:38.619 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.576+0000 7f48c7ba3700 10 monclient(hunting): _renew_subs 2021-05-25T12:05:38.619 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.576+0000 7f48c7ba3700 10 monclient(hunting): authenticate will time out at 1139.174656s 2021-05-25T12:05:38.619 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.576+0000 7f48c593f700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f48c0068f60 0x7f48c01207b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:32978/0 (socket says 172.21.15.102:32978) 2021-05-25T12:05:38.619 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.576+0000 7f48c593f700 1 -- 172.21.15.102:0/1605917423 learned_addr learned my addr 172.21.15.102:0/1605917423 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:05:38.620 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.576+0000 7f48adffb700 10 client.?.objecter ms_handle_connect 0x7f48c0068f60 2021-05-25T12:05:38.620 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.576+0000 7f48adffb700 10 client.?.objecter resend_mon_ops 2021-05-25T12:05:38.620 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.576+0000 7f48adffb700 10 client.?.objecter ms_handle_connect 0x7f48c0096800 2021-05-25T12:05:38.620 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.576+0000 7f48adffb700 10 client.?.objecter resend_mon_ops 2021-05-25T12:05:38.621 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.576+0000 7f48adffb700 1 -- 172.21.15.102:0/1605917423 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1754117385 0 0) 0x7f48c0121350 con 0x7f48c0068f60 2021-05-25T12:05:38.621 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.576+0000 7f48adffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:38.621 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.576+0000 7f48adffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:38.621 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.576+0000 7f48adffb700 10 monclient(hunting): my global_id is 174666 2021-05-25T12:05:38.621 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.576+0000 7f48adffb700 1 -- 172.21.15.102:0/1605917423 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48a0004c10 con 0x7f48c0068f60 2021-05-25T12:05:38.622 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.576+0000 7f48adffb700 1 -- 172.21.15.102:0/1605917423 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 12864541 0 0) 0x7f48c0096ca0 con 0x7f48c0096800 2021-05-25T12:05:38.622 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.576+0000 7f48adffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:38.622 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.576+0000 7f48adffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:38.622 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.576+0000 7f48adffb700 10 monclient(hunting): my global_id is 184364 2021-05-25T12:05:38.623 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.576+0000 7f48adffb700 1 -- 172.21.15.102:0/1605917423 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48c0121350 con 0x7f48c0096800 2021-05-25T12:05:38.623 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.577+0000 7f48adffb700 10 client.?.objecter ms_handle_connect 0x7f48c0131ae0 2021-05-25T12:05:38.623 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.577+0000 7f48adffb700 10 client.?.objecter resend_mon_ops 2021-05-25T12:05:38.623 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.577+0000 7f48adffb700 1 -- 172.21.15.102:0/1605917423 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3695156597 0 0) 0x7f48a0004c10 con 0x7f48c0068f60 2021-05-25T12:05:38.624 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.577+0000 7f48adffb700 1 -- 172.21.15.102:0/1605917423 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f48c0096ca0 con 0x7f48c0068f60 2021-05-25T12:05:38.624 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.577+0000 7f48adffb700 1 -- 172.21.15.102:0/1605917423 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2132349097 0 0) 0x7f48c0123290 con 0x7f48c0131ae0 2021-05-25T12:05:38.624 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.577+0000 7f48adffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:38.624 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.577+0000 7f48adffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:38.624 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.577+0000 7f48adffb700 10 monclient(hunting): my global_id is 194170 2021-05-25T12:05:38.625 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.577+0000 7f48adffb700 1 -- 172.21.15.102:0/1605917423 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48a0004c10 con 0x7f48c0131ae0 2021-05-25T12:05:38.625 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.577+0000 7f48adffb700 1 -- 172.21.15.102:0/1605917423 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f48b0001920 con 0x7f48c0068f60 2021-05-25T12:05:38.625 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.577+0000 7f48adffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:38.625 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.577+0000 7f48adffb700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:05:38.626 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.577+0000 7f48adffb700 10 monclient(hunting): dump: 2021-05-25T12:05:38.626 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:38.626 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:38.626 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:38.626 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:38.627 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:38.627 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:38.627 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:38.627 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:38.628 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:38.628 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:38.628 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.577+0000 7f48adffb700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:05:38.628 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.577+0000 7f48adffb700 1 -- 172.21.15.102:0/1605917423 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 975383292 0 0) 0x7f48c0121350 con 0x7f48c0096800 2021-05-25T12:05:38.628 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.577+0000 7f48adffb700 1 -- 172.21.15.102:0/1605917423 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f48c0123290 con 0x7f48c0096800 2021-05-25T12:05:38.629 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.577+0000 7f48adffb700 1 -- 172.21.15.102:0/1605917423 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f48b40015d0 con 0x7f48c0096800 2021-05-25T12:05:38.629 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.577+0000 7f48adffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:38.629 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.577+0000 7f48adffb700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2021-05-25T12:05:38.629 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.577+0000 7f48adffb700 10 monclient(hunting): dump: 2021-05-25T12:05:38.630 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:38.630 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:38.630 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:38.630 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:38.631 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:38.631 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:38.631 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:38.631 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:38.631 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:38.632 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:38.632 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.577+0000 7f48adffb700 1 -- 172.21.15.102:0/1605917423 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2146770315 0 0) 0x7f48c0096ca0 con 0x7f48c0068f60 2021-05-25T12:05:38.632 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.577+0000 7f48adffb700 1 -- 172.21.15.102:0/1605917423 >> v1:172.21.15.102:6790/0 conn(0x7f48c0096800 legacy=0x7f48c01256e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:38.632 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.577+0000 7f48adffb700 1 -- 172.21.15.102:0/1605917423 >> v1:172.21.15.181:6789/0 conn(0x7f48c0131ae0 legacy=0x7f48c0143130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:38.633 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.577+0000 7f48adffb700 10 monclient: _finish_hunting 0 2021-05-25T12:05:38.633 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.577+0000 7f48adffb700 1 monclient: found mon.a 2021-05-25T12:05:38.633 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.577+0000 7f48adffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:05:38.633 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.577+0000 7f48adffb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:05:38.634 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.577+0000 7f48adffb700 1 -- 172.21.15.102:0/1605917423 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48c0135020 con 0x7f48c0068f60 2021-05-25T12:05:38.634 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.577+0000 7f48adffb700 10 monclient: _finish_auth 0 2021-05-25T12:05:38.634 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.577+0000 7f48adffb700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:05:38.634 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.577+0000 7f48c7ba3700 5 monclient: authenticate success, global_id 174666 2021-05-25T12:05:38.634 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.577+0000 7f48c7ba3700 10 monclient: _renew_subs 2021-05-25T12:05:38.635 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.577+0000 7f48c7ba3700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:05:38.635 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.577+0000 7f48c7ba3700 1 -- 172.21.15.102:0/1605917423 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48c0125df0 con 0x7f48c0068f60 2021-05-25T12:05:38.635 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.577+0000 7f48adffb700 1 -- 172.21.15.102:0/1605917423 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f48b0001c90 con 0x7f48c0068f60 2021-05-25T12:05:38.635 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.577+0000 7f48adffb700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:05:38.636 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.577+0000 7f48adffb700 1 -- 172.21.15.102:0/1605917423 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f48b0004980 con 0x7f48c0068f60 2021-05-25T12:05:38.636 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.577+0000 7f48adffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:38.636 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.577+0000 7f48adffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:05:38.636 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.577+0000 7f48adffb700 10 monclient: dump: 2021-05-25T12:05:38.636 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:38.637 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:38.637 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:38.637 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:38.637 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:38.637 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:38.638 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:38.638 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:38.638 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:38.638 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:38.639 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.577+0000 7f48affff700 4 set_mon_vals no callback set 2021-05-25T12:05:38.639 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.577+0000 7f48c7ba3700 10 client.174666.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:05:38.639 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.577+0000 7f48c7ba3700 10 monclient: _renew_subs 2021-05-25T12:05:38.639 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.577+0000 7f48c7ba3700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:05:38.640 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.577+0000 7f48c7ba3700 1 -- 172.21.15.102:0/1605917423 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f48c00680b0 con 0x7f48c0068f60 2021-05-25T12:05:38.640 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.577+0000 7f48c7ba3700 1 librados: init done 2021-05-25T12:05:38.640 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.578+0000 7f48adffb700 1 -- 172.21.15.102:0/1605917423 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f48b000dca0 con 0x7f48c0068f60 2021-05-25T12:05:38.640 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.578+0000 7f48c7ba3700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:05:38.640 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.578+0000 7f48c7ba3700 1 -- 172.21.15.102:0/1605917423 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f48c0059730 con 0x7f48c009a310 2021-05-25T12:05:38.641 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.578+0000 7f48adffb700 1 -- 172.21.15.102:0/1605917423 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(170..170 src has 1..170) v4 ==== 6493+0+0 (unknown 2891575432 0 0) 0x7f48b003d990 con 0x7f48c0068f60 2021-05-25T12:05:38.641 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.578+0000 7f48adffb700 10 client.174666.objecter ms_dispatch 0x7f48c0001550 osd_map(170..170 src has 1..170) v4 2021-05-25T12:05:38.641 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.578+0000 7f48adffb700 3 client.174666.objecter handle_osd_map got epochs [170,170] > 0 2021-05-25T12:05:38.642 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.578+0000 7f48adffb700 3 client.174666.objecter handle_osd_map decoding full epoch 170 2021-05-25T12:05:38.642 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.578+0000 7f48adffb700 20 client.174666.objecter dump_active .. 0 homeless 2021-05-25T12:05:38.642 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.578+0000 7f48adffb700 10 client.174666.objecter ms_handle_connect 0x7f48c009a310 2021-05-25T12:05:38.642 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.578+0000 7f48adffb700 10 client.174666.objecter resend_mon_ops 2021-05-25T12:05:38.643 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.578+0000 7f48adffb700 1 -- 172.21.15.102:0/1605917423 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 559828278 0 0) 0x7f48c0059730 con 0x7f48c009a310 2021-05-25T12:05:38.643 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.578+0000 7f48adffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:38.643 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.578+0000 7f48adffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:38.643 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.578+0000 7f48adffb700 10 monclient(hunting): my global_id is 184367 2021-05-25T12:05:38.643 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.578+0000 7f48adffb700 1 -- 172.21.15.102:0/1605917423 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48c0096ca0 con 0x7f48c009a310 2021-05-25T12:05:38.644 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.578+0000 7f48c6140700 10 monclient: get_auth_request con 0x7f48a00331b0 auth_method 0 2021-05-25T12:05:38.644 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.578+0000 7f48adffb700 1 -- 172.21.15.102:0/1605917423 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1782397917 0 0) 0x7f48c0096ca0 con 0x7f48c009a310 2021-05-25T12:05:38.644 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.578+0000 7f48adffb700 1 -- 172.21.15.102:0/1605917423 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f48c0059730 con 0x7f48c009a310 2021-05-25T12:05:38.644 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.578+0000 7f48adffb700 1 -- 172.21.15.102:0/1605917423 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f48b4004770 con 0x7f48c009a310 2021-05-25T12:05:38.645 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.578+0000 7f48adffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:38.645 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.578+0000 7f48adffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:05:38.645 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.578+0000 7f48adffb700 10 monclient: dump: 2021-05-25T12:05:38.645 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:38.645 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:38.646 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:38.646 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:38.646 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:38.646 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:38.647 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:38.647 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:38.647 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:38.647 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:38.647 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.578+0000 7f48adffb700 10 client.174666.objecter ms_handle_connect 0x7f48a00331b0 2021-05-25T12:05:38.648 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.578+0000 7f48adffb700 1 -- 172.21.15.102:0/1605917423 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3879304751 0 0) 0x7f48c0059730 con 0x7f48c009a310 2021-05-25T12:05:38.648 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.578+0000 7f48adffb700 1 -- 172.21.15.102:0/1605917423 --> v1:172.21.15.102:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f48c0068360 con 0x7f48c009a310 2021-05-25T12:05:38.648 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.579+0000 7f48adffb700 1 -- 172.21.15.102:0/1605917423 <== mon.2 v1:172.21.15.102:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7f48b4006b40 con 0x7f48c009a310 2021-05-25T12:05:38.648 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.579+0000 7f48adffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:05:38.648 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.579+0000 7f48adffb700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:05:38.649 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.579+0000 7f48adffb700 1 -- 172.21.15.102:0/1605917423 >> v1:172.21.15.102:6790/0 conn(0x7f48c009a310 legacy=0x7f48c009a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:38.649 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.579+0000 7f48adffb700 1 -- 172.21.15.102:0/1605917423 >> v1:172.21.15.102:6790/0 conn(0x7f48c009a310 legacy=0x7f48c009a6f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:05:38.649 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.585+0000 7f48c7ba3700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:05:38.649 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.585+0000 7f48c7ba3700 1 -- 172.21.15.102:0/1605917423 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f48c005d790 con 0x7f48c0027ce0 2021-05-25T12:05:38.650 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.585+0000 7f48adffb700 10 client.174666.objecter ms_handle_connect 0x7f48c0027ce0 2021-05-25T12:05:38.650 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.585+0000 7f48adffb700 10 client.174666.objecter resend_mon_ops 2021-05-25T12:05:38.650 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.586+0000 7f48adffb700 1 -- 172.21.15.102:0/1605917423 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4124681646 0 0) 0x7f48c005d790 con 0x7f48c0027ce0 2021-05-25T12:05:38.650 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.586+0000 7f48adffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:38.651 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.586+0000 7f48adffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:38.651 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.586+0000 7f48adffb700 10 monclient(hunting): my global_id is 184370 2021-05-25T12:05:38.651 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.586+0000 7f48adffb700 1 -- 172.21.15.102:0/1605917423 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48c0059730 con 0x7f48c0027ce0 2021-05-25T12:05:38.651 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.586+0000 7f48adffb700 1 -- 172.21.15.102:0/1605917423 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3748313897 0 0) 0x7f48c0059730 con 0x7f48c0027ce0 2021-05-25T12:05:38.652 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.586+0000 7f48adffb700 1 -- 172.21.15.102:0/1605917423 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f48c005d790 con 0x7f48c0027ce0 2021-05-25T12:05:38.652 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.586+0000 7f48adffb700 1 -- 172.21.15.102:0/1605917423 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f48b4004fe0 con 0x7f48c0027ce0 2021-05-25T12:05:38.652 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.586+0000 7f48adffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:38.652 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.586+0000 7f48adffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:05:38.652 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.586+0000 7f48adffb700 10 monclient: dump: 2021-05-25T12:05:38.653 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:38.653 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:38.653 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:38.653 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:38.654 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:38.654 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:38.654 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:38.654 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:38.655 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:38.655 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:38.655 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.586+0000 7f48adffb700 1 -- 172.21.15.102:0/1605917423 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1464588054 0 0) 0x7f48c005d790 con 0x7f48c0027ce0 2021-05-25T12:05:38.655 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.586+0000 7f48adffb700 1 -- 172.21.15.102:0/1605917423 --> v1:172.21.15.102:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f48c0135430 con 0x7f48c0027ce0 2021-05-25T12:05:38.655 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.586+0000 7f48adffb700 1 -- 172.21.15.102:0/1605917423 <== mon.2 v1:172.21.15.102:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3942 (unknown 0 0 3424373420) 0x7f48b40063c0 con 0x7f48c0027ce0 2021-05-25T12:05:38.656 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.586+0000 7f48adffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:05:38.656 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.586+0000 7f48adffb700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:05:38.656 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.586+0000 7f48adffb700 1 -- 172.21.15.102:0/1605917423 >> v1:172.21.15.102:6790/0 conn(0x7f48c0027ce0 legacy=0x7f48c0148870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:38.656 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.586+0000 7f48adffb700 1 -- 172.21.15.102:0/1605917423 >> v1:172.21.15.102:6790/0 conn(0x7f48c0027ce0 legacy=0x7f48c0148870 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:05:38.657 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:05:38.657 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:05:38.657 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:05:38.658 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "leader", 2021-05-25T12:05:38.658 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 46, 2021-05-25T12:05:38.658 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:05:38.659 INFO:teuthology.orchestra.run.smithi102.stdout: 0, 2021-05-25T12:05:38.659 INFO:teuthology.orchestra.run.smithi102.stdout: 1, 2021-05-25T12:05:38.659 INFO:teuthology.orchestra.run.smithi102.stdout: 2 2021-05-25T12:05:38.659 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:05:38.659 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 5, 2021-05-25T12:05:38.660 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:05:38.660 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:05:38.660 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:05:38.660 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:05:38.660 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:05:38.661 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:05:38.661 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:05:38.661 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:05:38.661 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:05:38.662 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:05:38.662 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:05:38.662 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:05:38.662 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:05:38.663 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:05:38.663 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:05:38.663 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:05:38.663 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:05:38.663 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:05:38.664 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:05:38.664 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:05:38.664 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:05:38.664 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:05:38.665 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:05:38.665 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:05:38.665 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:05:38.665 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:05:38.665 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:05:38.666 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:05:38.666 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:05:38.666 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:05:38.666 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:05:38.667 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:05:38.667 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:05:38.667 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:05:38.667 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:05:38.668 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:05:38.668 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:05:38.668 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:05:38.668 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:05:38.668 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:05:38.669 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:05:38.669 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:05:38.669 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:05:38.669 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:05:38.670 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:05:38.670 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:05:38.670 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:05:38.670 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:05:38.671 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:05:38.671 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:05:38.671 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:05:38.671 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:05:38.671 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:05:38.672 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:05:38.672 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:05:38.672 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:05:38.672 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:05:38.673 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:05:38.673 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:05:38.673 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:05:38.673 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:05:38.673 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:05:38.674 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:05:38.674 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:05:38.674 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:05:38.674 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:05:38.675 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:05:38.675 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:05:38.675 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:05:38.675 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:05:38.675 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:05:38.676 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:05:38.676 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:05:38.676 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:05:38.676 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:05:38.677 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:05:38.677 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:05:38.677 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:05:38.677 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:05:38.677 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:05:38.678 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:05:38.678 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:05:38.678 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:05:38.678 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:05:38.679 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:05:38.679 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:05:38.679 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:05:38.679 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:05:38.679 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:05:38.680 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:05:38.680 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:05:38.680 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:05:38.680 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:05:38.681 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:05:38.681 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:05:38.681 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:05:38.681 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:05:38.681 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:05:38.682 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:05:38.682 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:05:38.682 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:05:38.682 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:05:38.683 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:05:38.683 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:05:38.683 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:05:38.683 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:05:38.683 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:05:38.684 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:05:38.684 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:05:38.684 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:05:38.684 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:05:38.685 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:05:38.685 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:05:38.685 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:05:38.685 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:05:38.686 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:05:38.686 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:05:38.686 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:05:38.686 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:05:38.687 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:05:38.687 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:05:38.687 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:05:38.687 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 2 2021-05-25T12:05:38.687 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:05:38.688 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:05:38.688 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": [ 2021-05-25T12:05:38.688 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:05:38.688 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:05:38.688 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:05:38.689 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:05:38.689 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:05:38.689 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:05:38.689 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:05:38.690 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:05:38.690 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:05:38.692 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.586+0000 7f48c7ba3700 10 librados: watch_flush enter 2021-05-25T12:05:38.692 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.586+0000 7f48c7ba3700 10 librados: watch_flush exit 2021-05-25T12:05:38.693 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.586+0000 7f48c7ba3700 20 client.174666.objecter shutdown clearing up homeless session... 2021-05-25T12:05:38.693 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.586+0000 7f48c7ba3700 10 client.174666.objecter successfully canceled tick 2021-05-25T12:05:38.693 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.586+0000 7f48c7ba3700 1 -- 172.21.15.102:0/1605917423 >> v1:172.21.15.181:6812/29582 conn(0x7f48a00331b0 legacy=0x7f48a0035670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:38.693 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.586+0000 7f48c7ba3700 10 monclient: shutdown 2021-05-25T12:05:38.694 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.586+0000 7f48c7ba3700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:05:38.694 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.586+0000 7f48c7ba3700 1 -- 172.21.15.102:0/1605917423 >> v1:172.21.15.102:6789/0 conn(0x7f48c0068f60 legacy=0x7f48c01207b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:38.694 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.586+0000 7f48c6140700 1 -- 172.21.15.102:0/1605917423 reap_dead start 2021-05-25T12:05:38.694 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.586+0000 7f48c6140700 1 -- 172.21.15.102:0/1605917423 reap_dead start 2021-05-25T12:05:38.695 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.586+0000 7f48c7ba3700 1 -- 172.21.15.102:0/1605917423 shutdown_connections 2021-05-25T12:05:38.695 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.586+0000 7f48c7ba3700 1 -- 172.21.15.102:0/1605917423 >> 172.21.15.102:0/1605917423 conn(0x7f48c009b920 msgr2=0x7f48c0126a20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:05:38.695 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.587+0000 7f48c7ba3700 1 -- 172.21.15.102:0/1605917423 shutdown_connections 2021-05-25T12:05:38.695 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.587+0000 7f48c7ba3700 1 -- 172.21.15.102:0/1605917423 wait complete. 2021-05-25T12:05:38.696 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.587+0000 7f48c7ba3700 1 librados: shutdown 2021-05-25T12:05:38.696 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2021-05-25T12:05:38.814 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.811+0000 7fa0affff700 10 monclient: get_monmap_and_config 2021-05-25T12:05:38.814 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.811+0000 7fa0affff700 10 monclient: build_initial_monmap 2021-05-25T12:05:38.815 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.811+0000 7fa0affff700 10 monclient: monmap: 2021-05-25T12:05:38.815 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:05:38.815 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:38.815 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:05:38.813463+0000 2021-05-25T12:05:38.816 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:05:38.813463+0000 2021-05-25T12:05:38.816 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:05:38.816 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:38.816 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:05:38.817 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:05:38.817 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:05:38.817 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:38.828 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.813+0000 7fa0affff700 1 Processor -- start 2021-05-25T12:05:38.828 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.813+0000 7fa0affff700 1 -- start start 2021-05-25T12:05:38.828 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.813+0000 7fa0affff700 10 monclient: init 2021-05-25T12:05:38.829 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.813+0000 7fa0affff700 10 monclient: _reopen_session rank -1 2021-05-25T12:05:38.829 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.813+0000 7fa0affff700 10 monclient: _add_conns ranks=[1,2,0] 2021-05-25T12:05:38.829 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.813+0000 7fa0affff700 10 monclient(hunting): picked mon.noname-c con 0x7fa0b0098930 addr v1:172.21.15.181:6789/0 2021-05-25T12:05:38.829 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.813+0000 7fa0affff700 10 monclient(hunting): picked mon.noname-b con 0x7fa0b012bca0 addr v1:172.21.15.102:6790/0 2021-05-25T12:05:38.830 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.813+0000 7fa0affff700 10 monclient(hunting): picked mon.noname-a con 0x7fa0b012c150 addr v1:172.21.15.102:6789/0 2021-05-25T12:05:38.830 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.813+0000 7fa0affff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0b0099b20 con 0x7fa0b012c150 2021-05-25T12:05:38.830 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.813+0000 7fa0affff700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0b009a3a0 con 0x7fa0b0098930 2021-05-25T12:05:38.830 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.813+0000 7fa0affff700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0b009ac60 con 0x7fa0b012bca0 2021-05-25T12:05:38.831 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.813+0000 7fa0affff700 10 monclient(hunting): _renew_subs 2021-05-25T12:05:38.831 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.813+0000 7fa0affff700 10 monclient(hunting): authenticate will time out at 1139.411975s 2021-05-25T12:05:38.831 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.814+0000 7fa0af7fe700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fa0b012c150 0x7fa0b0133ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:32992/0 (socket says 172.21.15.102:32992) 2021-05-25T12:05:38.831 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.814+0000 7fa0af7fe700 1 -- 172.21.15.102:0/3034740542 learned_addr learned my addr 172.21.15.102:0/3034740542 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:05:38.831 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.814+0000 7fa0adffb700 1 -- 172.21.15.102:0/3034740542 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 708141208 0 0) 0x7fa0b009ac60 con 0x7fa0b012bca0 2021-05-25T12:05:38.832 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.814+0000 7fa0adffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:38.832 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.814+0000 7fa0adffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:38.832 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.814+0000 7fa0adffb700 10 monclient(hunting): my global_id is 184373 2021-05-25T12:05:38.832 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.814+0000 7fa0adffb700 1 -- 172.21.15.102:0/3034740542 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa09c003270 con 0x7fa0b012bca0 2021-05-25T12:05:38.833 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.814+0000 7fa0adffb700 1 -- 172.21.15.102:0/3034740542 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 591545389 0 0) 0x7fa0b0099b20 con 0x7fa0b012c150 2021-05-25T12:05:38.833 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.814+0000 7fa0adffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:38.833 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.814+0000 7fa0adffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:38.833 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.814+0000 7fa0adffb700 10 monclient(hunting): my global_id is 174669 2021-05-25T12:05:38.834 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.814+0000 7fa0adffb700 1 -- 172.21.15.102:0/3034740542 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0b009ac60 con 0x7fa0b012c150 2021-05-25T12:05:38.834 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.815+0000 7fa0adffb700 1 -- 172.21.15.102:0/3034740542 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1642186815 0 0) 0x7fa0b009ac60 con 0x7fa0b012c150 2021-05-25T12:05:38.834 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.815+0000 7fa0adffb700 1 -- 172.21.15.102:0/3034740542 >> v1:172.21.15.102:6790/0 conn(0x7fa0b012bca0 legacy=0x7fa0b01308c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:38.834 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.815+0000 7fa0adffb700 1 -- 172.21.15.102:0/3034740542 >> v1:172.21.15.181:6789/0 conn(0x7fa0b0098930 legacy=0x7fa0b012b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:38.834 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.815+0000 7fa0adffb700 10 monclient: _finish_hunting 0 2021-05-25T12:05:38.835 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.815+0000 7fa0adffb700 1 monclient: found mon.noname-a 2021-05-25T12:05:38.835 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.815+0000 7fa0adffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:05:38.835 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.815+0000 7fa0adffb700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T12:05:38.835 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.815+0000 7fa0adffb700 1 -- 172.21.15.102:0/3034740542 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0b009af00 con 0x7fa0b012c150 2021-05-25T12:05:38.836 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.815+0000 7fa0adffb700 10 monclient: _finish_auth 0 2021-05-25T12:05:38.836 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.815+0000 7fa0adffb700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:05:38.836 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.815+0000 7fa0adffb700 1 -- 172.21.15.102:0/3034740542 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fa0a0001970 con 0x7fa0b012c150 2021-05-25T12:05:38.836 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.815+0000 7fa0adffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:38.836 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.815+0000 7fa0adffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:05:38.837 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.815+0000 7fa0adffb700 10 monclient: dump: 2021-05-25T12:05:38.837 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:38.837 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:38.837 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:38.838 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:38.838 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:38.838 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:38.838 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:38.838 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:38.839 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:38.839 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:38.839 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.815+0000 7fa0affff700 5 monclient: authenticate success, global_id 174669 2021-05-25T12:05:38.839 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.815+0000 7fa0affff700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:05:38.840 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.815+0000 7fa0adffb700 1 -- 172.21.15.102:0/3034740542 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa0a0001b90 con 0x7fa0b012c150 2021-05-25T12:05:38.840 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.815+0000 7fa0adffb700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:05:38.840 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.815+0000 7fa0affff700 10 monclient: get_monmap_and_config success 2021-05-25T12:05:38.840 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.815+0000 7fa0adffb700 1 -- 172.21.15.102:0/3034740542 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fa0a0001ea0 con 0x7fa0b012c150 2021-05-25T12:05:38.840 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.815+0000 7fa0affff700 4 set_mon_vals no callback set 2021-05-25T12:05:38.841 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.815+0000 7fa0adffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:38.841 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.815+0000 7fa0adffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:05:38.841 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.815+0000 7fa0adffb700 10 monclient: dump: 2021-05-25T12:05:38.841 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:38.842 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:38.842 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:38.842 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:38.842 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:38.842 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:38.843 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:38.843 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:38.843 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:38.843 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:38.844 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.816+0000 7fa0affff700 10 monclient: shutdown 2021-05-25T12:05:38.844 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.816+0000 7fa0affff700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:05:38.844 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.816+0000 7fa0affff700 1 -- 172.21.15.102:0/3034740542 >> v1:172.21.15.102:6789/0 conn(0x7fa0b012c150 legacy=0x7fa0b0133ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:38.844 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.816+0000 7fa0affff700 1 -- 172.21.15.102:0/3034740542 shutdown_connections 2021-05-25T12:05:38.845 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.816+0000 7fa0affff700 1 -- 172.21.15.102:0/3034740542 >> 172.21.15.102:0/3034740542 conn(0x7fa0b009e9d0 msgr2=0x7fa0b009ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:05:38.845 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.816+0000 7fa0affff700 1 -- 172.21.15.102:0/3034740542 shutdown_connections 2021-05-25T12:05:38.845 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.816+0000 7fa0affff700 1 -- 172.21.15.102:0/3034740542 wait complete. 2021-05-25T12:05:38.845 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.817+0000 7fa0affff700 10 monclient: build_initial_monmap 2021-05-25T12:05:38.845 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.817+0000 7fa0affff700 10 monclient: monmap: 2021-05-25T12:05:38.846 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:05:38.846 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:05:38.846 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:05:38.846 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:05:38.847 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:05:38.847 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:05:38.847 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:05:38.847 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:05:38.848 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:05:38.848 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:38.848 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.817+0000 7fa0affff700 1 librados: starting msgr at 2021-05-25T12:05:38.848 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.817+0000 7fa0affff700 1 librados: starting objecter 2021-05-25T12:05:38.848 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.817+0000 7fa0affff700 1 Processor -- start 2021-05-25T12:05:38.849 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.817+0000 7fa0affff700 1 -- start start 2021-05-25T12:05:38.849 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.817+0000 7fa0affff700 1 librados: setting wanted keys 2021-05-25T12:05:38.849 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.817+0000 7fa0affff700 1 librados: calling monclient init 2021-05-25T12:05:38.849 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.817+0000 7fa0affff700 10 monclient: init 2021-05-25T12:05:38.850 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.818+0000 7fa0affff700 10 monclient: _reopen_session rank -1 2021-05-25T12:05:38.850 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.818+0000 7fa0affff700 10 monclient: _add_conns ranks=[1,0,2] 2021-05-25T12:05:38.850 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.818+0000 7fa0affff700 10 monclient(hunting): picked mon.noname-b con 0x7fa0b0098930 addr v1:172.21.15.181:6789/0 2021-05-25T12:05:38.850 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.818+0000 7fa0affff700 10 monclient(hunting): picked mon.noname-a con 0x7fa0b012bca0 addr v1:172.21.15.102:6789/0 2021-05-25T12:05:38.851 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.818+0000 7fa0affff700 10 monclient(hunting): picked mon.noname-c con 0x7fa0b012c150 addr v1:172.21.15.102:6790/0 2021-05-25T12:05:38.851 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.818+0000 7fa0affff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0b013b7f0 con 0x7fa0b012bca0 2021-05-25T12:05:38.851 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.818+0000 7fa0affff700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0b01304b0 con 0x7fa0b0098930 2021-05-25T12:05:38.851 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.818+0000 7fa0affff700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0b012ee20 con 0x7fa0b012c150 2021-05-25T12:05:38.852 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.818+0000 7fa0affff700 10 monclient(hunting): _renew_subs 2021-05-25T12:05:38.852 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.818+0000 7fa0affff700 10 monclient(hunting): authenticate will time out at 1139.416254s 2021-05-25T12:05:38.852 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.818+0000 7fa0ae7fc700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fa0b012bca0 0x7fa0b0134990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:32998/0 (socket says 172.21.15.102:32998) 2021-05-25T12:05:38.852 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.818+0000 7fa0af7fe700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fa0b012c150 0x7fa0b0142cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:56110/0 (socket says 172.21.15.102:56110) 2021-05-25T12:05:38.852 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.818+0000 7fa0ae7fc700 1 -- 172.21.15.102:0/4137244525 learned_addr learned my addr 172.21.15.102:0/4137244525 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:05:38.853 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.818+0000 7fa096ffd700 10 client.?.objecter ms_handle_connect 0x7fa0b012c150 2021-05-25T12:05:38.853 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.818+0000 7fa096ffd700 10 client.?.objecter resend_mon_ops 2021-05-25T12:05:38.853 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.818+0000 7fa096ffd700 10 client.?.objecter ms_handle_connect 0x7fa0b012bca0 2021-05-25T12:05:38.853 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.818+0000 7fa096ffd700 10 client.?.objecter resend_mon_ops 2021-05-25T12:05:38.854 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.818+0000 7fa096ffd700 10 client.?.objecter ms_handle_connect 0x7fa0b0098930 2021-05-25T12:05:38.854 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.818+0000 7fa096ffd700 10 client.?.objecter resend_mon_ops 2021-05-25T12:05:38.854 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.818+0000 7fa096ffd700 1 -- 172.21.15.102:0/4137244525 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 487049329 0 0) 0x7fa0b012ee20 con 0x7fa0b012c150 2021-05-25T12:05:38.854 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.818+0000 7fa096ffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:38.855 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.818+0000 7fa096ffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:38.855 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.818+0000 7fa096ffd700 10 monclient(hunting): my global_id is 184376 2021-05-25T12:05:38.855 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.818+0000 7fa096ffd700 1 -- 172.21.15.102:0/4137244525 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa090004c10 con 0x7fa0b012c150 2021-05-25T12:05:38.855 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.818+0000 7fa096ffd700 1 -- 172.21.15.102:0/4137244525 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3969132416 0 0) 0x7fa0b013b7f0 con 0x7fa0b012bca0 2021-05-25T12:05:38.855 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.818+0000 7fa096ffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:38.856 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.818+0000 7fa096ffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:38.856 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.818+0000 7fa096ffd700 10 monclient(hunting): my global_id is 174672 2021-05-25T12:05:38.856 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.818+0000 7fa096ffd700 1 -- 172.21.15.102:0/4137244525 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0b012ee20 con 0x7fa0b012bca0 2021-05-25T12:05:38.856 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.818+0000 7fa096ffd700 1 -- 172.21.15.102:0/4137244525 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 79617505 0 0) 0x7fa0b01304b0 con 0x7fa0b0098930 2021-05-25T12:05:38.856 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.818+0000 7fa096ffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:38.857 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.818+0000 7fa096ffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:38.857 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.818+0000 7fa096ffd700 10 monclient(hunting): my global_id is 194176 2021-05-25T12:05:38.857 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.818+0000 7fa096ffd700 1 -- 172.21.15.102:0/4137244525 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0b013b7f0 con 0x7fa0b0098930 2021-05-25T12:05:38.857 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.818+0000 7fa096ffd700 1 -- 172.21.15.102:0/4137244525 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4234788550 0 0) 0x7fa090004c10 con 0x7fa0b012c150 2021-05-25T12:05:38.857 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.818+0000 7fa096ffd700 1 -- 172.21.15.102:0/4137244525 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa0b01304b0 con 0x7fa0b012c150 2021-05-25T12:05:38.858 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.818+0000 7fa096ffd700 1 -- 172.21.15.102:0/4137244525 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fa0a0001870 con 0x7fa0b012c150 2021-05-25T12:05:38.858 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.818+0000 7fa096ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:38.858 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.818+0000 7fa096ffd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:05:38.858 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.818+0000 7fa096ffd700 10 monclient(hunting): dump: 2021-05-25T12:05:38.859 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:38.859 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:38.859 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:38.859 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:38.859 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:38.860 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:38.860 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:38.860 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:38.860 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:38.861 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:38.861 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.818+0000 7fa096ffd700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:05:38.861 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.818+0000 7fa096ffd700 1 -- 172.21.15.102:0/4137244525 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3001087631 0 0) 0x7fa0b012ee20 con 0x7fa0b012bca0 2021-05-25T12:05:38.861 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.818+0000 7fa096ffd700 1 -- 172.21.15.102:0/4137244525 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa090004c10 con 0x7fa0b012bca0 2021-05-25T12:05:38.861 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.818+0000 7fa096ffd700 1 -- 172.21.15.102:0/4137244525 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fa0980019f0 con 0x7fa0b012bca0 2021-05-25T12:05:38.862 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.818+0000 7fa096ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:38.862 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.818+0000 7fa096ffd700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2021-05-25T12:05:38.862 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.818+0000 7fa096ffd700 10 monclient(hunting): dump: 2021-05-25T12:05:38.862 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:38.863 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:38.863 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:38.863 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:38.863 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:38.863 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:38.864 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:38.864 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:38.864 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:38.864 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:38.865 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.819+0000 7fa096ffd700 1 -- 172.21.15.102:0/4137244525 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 301626100 0 0) 0x7fa0b01304b0 con 0x7fa0b012c150 2021-05-25T12:05:38.865 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.819+0000 7fa096ffd700 1 -- 172.21.15.102:0/4137244525 >> v1:172.21.15.181:6789/0 conn(0x7fa0b0098930 legacy=0x7fa0b013a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:38.865 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.819+0000 7fa096ffd700 1 -- 172.21.15.102:0/4137244525 >> v1:172.21.15.102:6789/0 conn(0x7fa0b012bca0 legacy=0x7fa0b0134990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:38.865 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.819+0000 7fa096ffd700 10 monclient: _finish_hunting 0 2021-05-25T12:05:38.865 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.819+0000 7fa096ffd700 1 monclient: found mon.c 2021-05-25T12:05:38.866 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.819+0000 7fa096ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:05:38.866 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.819+0000 7fa096ffd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:05:38.866 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.819+0000 7fa096ffd700 1 -- 172.21.15.102:0/4137244525 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0b012e3b0 con 0x7fa0b012c150 2021-05-25T12:05:38.867 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.819+0000 7fa096ffd700 10 monclient: _finish_auth 0 2021-05-25T12:05:38.867 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.819+0000 7fa096ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:05:38.867 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.819+0000 7fa0affff700 5 monclient: authenticate success, global_id 184376 2021-05-25T12:05:38.867 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.819+0000 7fa0affff700 10 monclient: _renew_subs 2021-05-25T12:05:38.867 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.819+0000 7fa0affff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:05:38.868 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.819+0000 7fa0affff700 1 -- 172.21.15.102:0/4137244525 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0b013b080 con 0x7fa0b012c150 2021-05-25T12:05:38.868 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.819+0000 7fa0affff700 10 client.184376.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:05:38.868 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.819+0000 7fa0affff700 10 monclient: _renew_subs 2021-05-25T12:05:38.868 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.819+0000 7fa0affff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:05:38.868 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.819+0000 7fa0affff700 1 -- 172.21.15.102:0/4137244525 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0b01350a0 con 0x7fa0b012c150 2021-05-25T12:05:38.869 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.819+0000 7fa0affff700 1 librados: init done 2021-05-25T12:05:38.869 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.819+0000 7fa096ffd700 1 -- 172.21.15.102:0/4137244525 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa0a0004b60 con 0x7fa0b012c150 2021-05-25T12:05:38.869 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.819+0000 7fa096ffd700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:05:38.869 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.819+0000 7fa096ffd700 1 -- 172.21.15.102:0/4137244525 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fa0a0004e70 con 0x7fa0b012c150 2021-05-25T12:05:38.870 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.819+0000 7fa096ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:38.870 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.819+0000 7fa0ad7fa700 4 set_mon_vals no callback set 2021-05-25T12:05:38.870 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.819+0000 7fa096ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:05:38.870 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.819+0000 7fa096ffd700 10 monclient: dump: 2021-05-25T12:05:38.871 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:38.871 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:38.871 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:38.871 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:38.871 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:38.872 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:38.872 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:38.872 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:38.872 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:38.873 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:38.873 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.820+0000 7fa0affff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:05:38.873 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.820+0000 7fa0affff700 1 -- 172.21.15.102:0/4137244525 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fa0880065a0 con 0x7fa088002840 2021-05-25T12:05:38.873 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.820+0000 7fa096ffd700 1 -- 172.21.15.102:0/4137244525 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7fa0a000e190 con 0x7fa0b012c150 2021-05-25T12:05:38.874 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.820+0000 7fa096ffd700 10 client.184376.objecter ms_handle_connect 0x7fa088002840 2021-05-25T12:05:38.874 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.820+0000 7fa096ffd700 10 client.184376.objecter resend_mon_ops 2021-05-25T12:05:38.874 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.820+0000 7fa096ffd700 1 -- 172.21.15.102:0/4137244525 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(171..171 src has 1..171) v4 ==== 6209+0+0 (unknown 228058464 0 0) 0x7fa0a003dcd0 con 0x7fa0b012c150 2021-05-25T12:05:38.874 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.820+0000 7fa096ffd700 10 client.184376.objecter ms_dispatch 0x7fa0b0001550 osd_map(171..171 src has 1..171) v4 2021-05-25T12:05:38.874 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.820+0000 7fa096ffd700 3 client.184376.objecter handle_osd_map got epochs [171,171] > 0 2021-05-25T12:05:38.875 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.820+0000 7fa096ffd700 3 client.184376.objecter handle_osd_map decoding full epoch 171 2021-05-25T12:05:38.875 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.820+0000 7fa096ffd700 20 client.184376.objecter dump_active .. 0 homeless 2021-05-25T12:05:38.875 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.820+0000 7fa096ffd700 1 -- 172.21.15.102:0/4137244525 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3227238241 0 0) 0x7fa0880065a0 con 0x7fa088002840 2021-05-25T12:05:38.875 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.820+0000 7fa096ffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:38.876 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.820+0000 7fa096ffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:38.876 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.820+0000 7fa096ffd700 10 monclient(hunting): my global_id is 194179 2021-05-25T12:05:38.876 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.820+0000 7fa096ffd700 1 -- 172.21.15.102:0/4137244525 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0b01304b0 con 0x7fa088002840 2021-05-25T12:05:38.876 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.820+0000 7fa0ae7fc700 10 monclient: get_auth_request con 0x7fa0900333d0 auth_method 0 2021-05-25T12:05:38.876 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.820+0000 7fa096ffd700 1 -- 172.21.15.102:0/4137244525 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3114445088 0 0) 0x7fa0b01304b0 con 0x7fa088002840 2021-05-25T12:05:38.877 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.820+0000 7fa096ffd700 1 -- 172.21.15.102:0/4137244525 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa0880065a0 con 0x7fa088002840 2021-05-25T12:05:38.877 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.820+0000 7fa096ffd700 1 -- 172.21.15.102:0/4137244525 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fa0a4001810 con 0x7fa088002840 2021-05-25T12:05:38.877 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.820+0000 7fa096ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:38.877 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.820+0000 7fa096ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T12:05:38.878 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.820+0000 7fa096ffd700 10 monclient: dump: 2021-05-25T12:05:38.878 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:38.878 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:38.878 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:38.879 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:38.879 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:38.879 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:38.879 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:38.880 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:38.880 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:38.880 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:38.880 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.821+0000 7fa096ffd700 10 client.184376.objecter ms_handle_connect 0x7fa0900333d0 2021-05-25T12:05:38.880 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.821+0000 7fa096ffd700 1 -- 172.21.15.102:0/4137244525 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4151431837 0 0) 0x7fa0880065a0 con 0x7fa088002840 2021-05-25T12:05:38.881 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.821+0000 7fa096ffd700 1 -- 172.21.15.102:0/4137244525 --> v1:172.21.15.181:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa088009e40 con 0x7fa088002840 2021-05-25T12:05:38.881 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.821+0000 7fa096ffd700 1 -- 172.21.15.102:0/4137244525 <== mon.1 v1:172.21.15.181:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7fa0a4002d10 con 0x7fa088002840 2021-05-25T12:05:38.881 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.821+0000 7fa096ffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:05:38.881 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.821+0000 7fa096ffd700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:05:38.882 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.821+0000 7fa096ffd700 1 -- 172.21.15.102:0/4137244525 >> v1:172.21.15.181:6789/0 conn(0x7fa088002840 legacy=0x7fa088004ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:38.882 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.821+0000 7fa096ffd700 1 -- 172.21.15.102:0/4137244525 >> v1:172.21.15.181:6789/0 conn(0x7fa088002840 legacy=0x7fa088004ce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:05:38.884 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.828+0000 7fa0affff700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:05:38.884 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.828+0000 7fa0affff700 1 -- 172.21.15.102:0/4137244525 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fa088007540 con 0x7fa0880068e0 2021-05-25T12:05:38.884 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.829+0000 7fa096ffd700 10 client.184376.objecter ms_handle_connect 0x7fa0880068e0 2021-05-25T12:05:38.885 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.829+0000 7fa096ffd700 10 client.184376.objecter resend_mon_ops 2021-05-25T12:05:38.885 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.829+0000 7fa096ffd700 1 -- 172.21.15.102:0/4137244525 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2113220902 0 0) 0x7fa088007540 con 0x7fa0880068e0 2021-05-25T12:05:38.885 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.829+0000 7fa096ffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:38.885 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.829+0000 7fa096ffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:38.886 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.829+0000 7fa096ffd700 10 monclient(hunting): my global_id is 194182 2021-05-25T12:05:38.886 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.829+0000 7fa096ffd700 1 -- 172.21.15.102:0/4137244525 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0880065a0 con 0x7fa0880068e0 2021-05-25T12:05:38.886 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.829+0000 7fa096ffd700 1 -- 172.21.15.102:0/4137244525 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 82763399 0 0) 0x7fa0880065a0 con 0x7fa0880068e0 2021-05-25T12:05:38.886 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.829+0000 7fa096ffd700 1 -- 172.21.15.102:0/4137244525 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa088007540 con 0x7fa0880068e0 2021-05-25T12:05:38.887 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.829+0000 7fa096ffd700 1 -- 172.21.15.102:0/4137244525 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fa0a4002040 con 0x7fa0880068e0 2021-05-25T12:05:38.887 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.829+0000 7fa096ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:38.887 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.829+0000 7fa096ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T12:05:38.887 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.829+0000 7fa096ffd700 10 monclient: dump: 2021-05-25T12:05:38.888 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:38.888 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:38.888 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:38.888 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:38.888 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:38.889 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:38.889 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:38.889 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:38.889 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:38.889 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:38.890 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.829+0000 7fa096ffd700 1 -- 172.21.15.102:0/4137244525 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3983936340 0 0) 0x7fa088007540 con 0x7fa0880068e0 2021-05-25T12:05:38.890 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.829+0000 7fa096ffd700 1 -- 172.21.15.102:0/4137244525 --> v1:172.21.15.181:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa088008770 con 0x7fa0880068e0 2021-05-25T12:05:38.890 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.829+0000 7fa096ffd700 1 -- 172.21.15.102:0/4137244525 <== mon.1 v1:172.21.15.181:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3940 (unknown 0 0 2152781978) 0x7fa0a4002400 con 0x7fa0880068e0 2021-05-25T12:05:38.890 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.829+0000 7fa096ffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:05:38.891 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.829+0000 7fa096ffd700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:05:38.891 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.829+0000 7fa096ffd700 1 -- 172.21.15.102:0/4137244525 >> v1:172.21.15.181:6789/0 conn(0x7fa0880068e0 legacy=0x7fa088006d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:38.891 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.829+0000 7fa096ffd700 1 -- 172.21.15.102:0/4137244525 >> v1:172.21.15.181:6789/0 conn(0x7fa0880068e0 legacy=0x7fa088006d70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:05:38.892 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:05:38.892 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:05:38.892 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:05:38.892 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "peon", 2021-05-25T12:05:38.893 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 46, 2021-05-25T12:05:38.893 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:05:38.893 INFO:teuthology.orchestra.run.smithi102.stdout: 0, 2021-05-25T12:05:38.893 INFO:teuthology.orchestra.run.smithi102.stdout: 1, 2021-05-25T12:05:38.893 INFO:teuthology.orchestra.run.smithi102.stdout: 2 2021-05-25T12:05:38.894 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:05:38.894 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 6, 2021-05-25T12:05:38.894 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:05:38.894 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:05:38.894 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:05:38.895 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:05:38.895 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:05:38.895 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:05:38.895 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:05:38.896 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:05:38.896 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:05:38.896 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:05:38.896 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:05:38.897 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:05:38.897 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:05:38.897 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:05:38.897 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:05:38.898 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:05:38.898 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:05:38.898 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:05:38.898 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:05:38.899 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:05:38.899 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:05:38.899 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:05:38.899 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:05:38.899 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:05:38.900 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:05:38.900 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:05:38.900 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:05:38.900 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:05:38.901 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:05:38.901 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:05:38.901 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:05:38.901 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:05:38.902 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:05:38.902 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:05:38.902 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:05:38.902 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:05:38.903 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:05:38.903 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:05:38.903 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:05:38.903 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:05:38.903 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:05:38.904 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:05:38.904 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:05:38.904 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:05:38.904 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:05:38.905 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:05:38.905 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:05:38.905 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:05:38.905 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:05:38.906 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:05:38.906 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:05:38.906 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:05:38.906 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:05:38.907 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:05:38.907 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:05:38.907 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:05:38.907 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:05:38.908 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:05:38.908 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:05:38.908 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:05:38.908 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:05:38.908 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:05:38.909 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:05:38.909 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:05:38.909 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:05:38.909 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:05:38.910 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:05:38.910 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:05:38.910 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:05:38.910 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:05:38.910 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:05:38.911 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:05:38.911 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:05:38.911 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:05:38.911 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:05:38.912 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:05:38.912 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:05:38.912 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:05:38.912 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:05:38.912 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:05:38.913 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:05:38.913 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:05:38.913 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:05:38.913 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:05:38.914 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:05:38.914 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:05:38.914 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:05:38.914 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:05:38.915 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:05:38.915 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:05:38.915 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:05:38.915 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:05:38.915 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:05:38.916 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:05:38.916 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:05:38.916 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:05:38.916 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:05:38.917 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:05:38.917 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:05:38.917 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:05:38.917 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:05:38.917 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:05:38.918 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:05:38.918 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:05:38.918 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:05:38.918 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:05:38.919 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:05:38.919 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:05:38.919 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:05:38.919 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:05:38.920 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:05:38.920 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:05:38.920 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:05:38.920 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:05:38.921 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:05:38.921 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:05:38.921 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:05:38.921 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:05:38.922 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": [ 2021-05-25T12:05:38.922 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:05:38.922 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:05:38.922 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:05:38.923 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 4 2021-05-25T12:05:38.923 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:05:38.923 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:05:38.923 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:05:38.924 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:05:38.924 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:05:38.924 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:05:38.924 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 2 2021-05-25T12:05:38.924 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:05:38.925 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:05:38.925 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:05:38.925 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:05:38.925 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:05:38.928 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.830+0000 7fa0affff700 10 librados: watch_flush enter 2021-05-25T12:05:38.928 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.830+0000 7fa0affff700 10 librados: watch_flush exit 2021-05-25T12:05:38.928 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.830+0000 7fa0affff700 20 client.184376.objecter shutdown clearing up homeless session... 2021-05-25T12:05:38.929 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.830+0000 7fa0affff700 10 client.184376.objecter successfully canceled tick 2021-05-25T12:05:38.929 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.830+0000 7fa0affff700 1 -- 172.21.15.102:0/4137244525 >> v1:172.21.15.181:6812/29582 conn(0x7fa0900333d0 legacy=0x7fa090035890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:38.929 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.830+0000 7fa0affff700 10 monclient: shutdown 2021-05-25T12:05:38.929 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.830+0000 7fa0affff700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:05:38.929 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.830+0000 7fa0affff700 1 -- 172.21.15.102:0/4137244525 >> v1:172.21.15.102:6790/0 conn(0x7fa0b012c150 legacy=0x7fa0b0142cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:38.930 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.830+0000 7fa0af7fe700 1 -- 172.21.15.102:0/4137244525 reap_dead start 2021-05-25T12:05:38.930 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.830+0000 7fa0af7fe700 1 -- 172.21.15.102:0/4137244525 reap_dead start 2021-05-25T12:05:38.930 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.830+0000 7fa0affff700 1 -- 172.21.15.102:0/4137244525 shutdown_connections 2021-05-25T12:05:38.930 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.830+0000 7fa0affff700 1 -- 172.21.15.102:0/4137244525 >> 172.21.15.102:0/4137244525 conn(0x7fa0b009e9d0 msgr2=0x7fa0b0129fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:05:38.931 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.830+0000 7fa0affff700 1 -- 172.21.15.102:0/4137244525 shutdown_connections 2021-05-25T12:05:38.931 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.830+0000 7fa0affff700 1 -- 172.21.15.102:0/4137244525 wait complete. 2021-05-25T12:05:38.931 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:38.830+0000 7fa0affff700 1 librados: shutdown 2021-05-25T12:05:38.932 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2021-05-25T12:05:38.932 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2021-05-25T12:05:38.932 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2021-05-25T12:05:38.933 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2021-05-25T12:05:38.933 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2021-05-25T12:05:38.934 INFO:teuthology.orchestra.run:waiting for 300 2021-05-25T12:05:38.944 INFO:tasks.ceph.mon.a:Stopped 2021-05-25T12:05:38.945 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2021-05-25T12:05:38.945 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2021-05-25T12:05:39.093 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.091+0000 7fe448cc4700 10 monclient: get_monmap_and_config 2021-05-25T12:05:39.093 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.091+0000 7fe448cc4700 10 monclient: build_initial_monmap 2021-05-25T12:05:39.093 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.091+0000 7fe448cc4700 10 monclient: monmap: 2021-05-25T12:05:39.094 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:05:39.094 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:39.094 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:05:39.092913+0000 2021-05-25T12:05:39.094 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:05:39.092913+0000 2021-05-25T12:05:39.095 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:05:39.095 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:39.095 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:05:39.095 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:05:39.096 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:05:39.096 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:39.105 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.091+0000 7fe448cc4700 1 Processor -- start 2021-05-25T12:05:39.106 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.091+0000 7fe448cc4700 1 -- start start 2021-05-25T12:05:39.106 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.091+0000 7fe448cc4700 10 monclient: init 2021-05-25T12:05:39.106 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.092+0000 7fe448cc4700 10 monclient: _reopen_session rank -1 2021-05-25T12:05:39.106 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.092+0000 7fe448cc4700 10 monclient: _add_conns ranks=[1,0,2] 2021-05-25T12:05:39.107 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.092+0000 7fe448cc4700 10 monclient(hunting): picked mon.noname-c con 0x7fe444096630 addr v1:172.21.15.181:6789/0 2021-05-25T12:05:39.107 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.092+0000 7fe448cc4700 10 monclient(hunting): picked mon.noname-a con 0x7fe444093bd0 addr v1:172.21.15.102:6789/0 2021-05-25T12:05:39.107 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.092+0000 7fe448cc4700 10 monclient(hunting): picked mon.noname-b con 0x7fe444096a30 addr v1:172.21.15.102:6790/0 2021-05-25T12:05:39.107 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.092+0000 7fe448cc4700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4440630c0 con 0x7fe444093bd0 2021-05-25T12:05:39.108 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.092+0000 7fe448cc4700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe444064320 con 0x7fe444096630 2021-05-25T12:05:39.108 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.092+0000 7fe448cc4700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4440645c0 con 0x7fe444096a30 2021-05-25T12:05:39.108 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.092+0000 7fe448cc4700 10 monclient(hunting): _renew_subs 2021-05-25T12:05:39.108 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.092+0000 7fe448cc4700 10 monclient(hunting): authenticate will time out at 1139.690211s 2021-05-25T12:05:39.109 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.092+0000 7fe441d9b700 1 -- >> v1:172.21.15.102:6789/0 conn(0x7fe444093bd0 legacy=0x7fe4441308f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:05:39.109 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.092+0000 7fe442d9d700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fe444096a30 0x7fe444134020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:56134/0 (socket says 172.21.15.102:56134) 2021-05-25T12:05:39.109 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.092+0000 7fe442d9d700 1 -- 172.21.15.102:0/3738777351 learned_addr learned my addr 172.21.15.102:0/3738777351 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:05:39.109 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.092+0000 7fe44159a700 1 -- 172.21.15.102:0/3738777351 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 118848847 0 0) 0x7fe4440645c0 con 0x7fe444096a30 2021-05-25T12:05:39.109 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.092+0000 7fe44159a700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:39.110 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.092+0000 7fe44159a700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:39.110 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.092+0000 7fe44159a700 10 monclient(hunting): my global_id is 184379 2021-05-25T12:05:39.110 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.092+0000 7fe44159a700 1 -- 172.21.15.102:0/3738777351 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe428003270 con 0x7fe444096a30 2021-05-25T12:05:39.110 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.092+0000 7fe44159a700 1 -- 172.21.15.102:0/3738777351 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2239088363 0 0) 0x7fe444064320 con 0x7fe444096630 2021-05-25T12:05:39.111 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.092+0000 7fe44159a700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:39.111 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.092+0000 7fe44159a700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:39.111 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.092+0000 7fe44159a700 10 monclient(hunting): my global_id is 194185 2021-05-25T12:05:39.111 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.092+0000 7fe44159a700 1 -- 172.21.15.102:0/3738777351 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4440645c0 con 0x7fe444096630 2021-05-25T12:05:39.111 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.092+0000 7fe44159a700 1 -- 172.21.15.102:0/3738777351 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 987294679 0 0) 0x7fe428003270 con 0x7fe444096a30 2021-05-25T12:05:39.112 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.092+0000 7fe44159a700 1 -- 172.21.15.102:0/3738777351 >> v1:172.21.15.181:6789/0 conn(0x7fe444096630 legacy=0x7fe4440697d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:39.112 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.092+0000 7fe44159a700 1 -- 172.21.15.102:0/3738777351 >> v1:172.21.15.102:6789/0 conn(0x7fe444093bd0 legacy=0x7fe4441308f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-05-25T12:05:39.112 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.092+0000 7fe44159a700 10 monclient: _finish_hunting 0 2021-05-25T12:05:39.112 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.092+0000 7fe44159a700 1 monclient: found mon.noname-b 2021-05-25T12:05:39.113 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.092+0000 7fe44159a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:05:39.113 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.092+0000 7fe44159a700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T12:05:39.113 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.092+0000 7fe44159a700 1 -- 172.21.15.102:0/3738777351 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe444064860 con 0x7fe444096a30 2021-05-25T12:05:39.113 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.092+0000 7fe44159a700 10 monclient: _finish_auth 0 2021-05-25T12:05:39.113 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.092+0000 7fe44159a700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:05:39.114 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.092+0000 7fe44159a700 1 -- 172.21.15.102:0/3738777351 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fe434001990 con 0x7fe444096a30 2021-05-25T12:05:39.114 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.092+0000 7fe448cc4700 5 monclient: authenticate success, global_id 184379 2021-05-25T12:05:39.114 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.092+0000 7fe448cc4700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:05:39.114 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.092+0000 7fe44159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:39.115 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.092+0000 7fe44159a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:05:39.115 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.092+0000 7fe44159a700 10 monclient: dump: 2021-05-25T12:05:39.115 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:39.115 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:39.115 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:39.116 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:39.116 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:39.116 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:39.116 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:39.116 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:39.117 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:39.117 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:39.117 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.092+0000 7fe448cc4700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:05:39.117 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.092+0000 7fe44159a700 1 -- 172.21.15.102:0/3738777351 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe434001bb0 con 0x7fe444096a30 2021-05-25T12:05:39.117 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.092+0000 7fe44159a700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:05:39.118 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.092+0000 7fe448cc4700 10 monclient: get_monmap_and_config success 2021-05-25T12:05:39.118 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.092+0000 7fe44159a700 1 -- 172.21.15.102:0/3738777351 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fe434001ec0 con 0x7fe444096a30 2021-05-25T12:05:39.118 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.092+0000 7fe448cc4700 4 set_mon_vals no callback set 2021-05-25T12:05:39.118 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.092+0000 7fe44159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:39.119 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.092+0000 7fe44159a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:05:39.119 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.092+0000 7fe44159a700 10 monclient: dump: 2021-05-25T12:05:39.119 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:39.119 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:39.119 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:39.120 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:39.120 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:39.120 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:39.120 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:39.120 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:39.121 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:39.121 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:39.121 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.093+0000 7fe448cc4700 10 monclient: shutdown 2021-05-25T12:05:39.121 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.093+0000 7fe448cc4700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:05:39.121 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.093+0000 7fe448cc4700 1 -- 172.21.15.102:0/3738777351 >> v1:172.21.15.102:6790/0 conn(0x7fe444096a30 legacy=0x7fe444134020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:39.122 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.093+0000 7fe448cc4700 1 -- 172.21.15.102:0/3738777351 shutdown_connections 2021-05-25T12:05:39.122 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.093+0000 7fe448cc4700 1 -- 172.21.15.102:0/3738777351 >> 172.21.15.102:0/3738777351 conn(0x7fe44409cd10 msgr2=0x7fe44409d120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:05:39.122 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.093+0000 7fe448cc4700 1 -- 172.21.15.102:0/3738777351 shutdown_connections 2021-05-25T12:05:39.122 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.093+0000 7fe448cc4700 1 -- 172.21.15.102:0/3738777351 wait complete. 2021-05-25T12:05:39.123 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.093+0000 7fe448cc4700 10 monclient: build_initial_monmap 2021-05-25T12:05:39.123 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.093+0000 7fe448cc4700 10 monclient: monmap: 2021-05-25T12:05:39.123 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:05:39.123 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:05:39.123 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:05:39.124 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:05:39.124 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:05:39.124 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:05:39.124 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:05:39.124 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:05:39.125 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:05:39.125 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:39.125 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.094+0000 7fe448cc4700 1 librados: starting msgr at 2021-05-25T12:05:39.125 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.094+0000 7fe448cc4700 1 librados: starting objecter 2021-05-25T12:05:39.125 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.094+0000 7fe448cc4700 1 Processor -- start 2021-05-25T12:05:39.126 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.094+0000 7fe448cc4700 1 -- start start 2021-05-25T12:05:39.126 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.094+0000 7fe448cc4700 1 librados: setting wanted keys 2021-05-25T12:05:39.126 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.094+0000 7fe448cc4700 1 librados: calling monclient init 2021-05-25T12:05:39.126 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.094+0000 7fe448cc4700 10 monclient: init 2021-05-25T12:05:39.127 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.094+0000 7fe448cc4700 10 monclient: _reopen_session rank -1 2021-05-25T12:05:39.127 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.094+0000 7fe448cc4700 10 monclient: _add_conns ranks=[1,2,0] 2021-05-25T12:05:39.127 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.094+0000 7fe448cc4700 10 monclient(hunting): picked mon.noname-b con 0x7fe444093bd0 addr v1:172.21.15.181:6789/0 2021-05-25T12:05:39.127 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.094+0000 7fe448cc4700 10 monclient(hunting): picked mon.noname-c con 0x7fe444096630 addr v1:172.21.15.102:6790/0 2021-05-25T12:05:39.127 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.094+0000 7fe448cc4700 10 monclient(hunting): picked mon.noname-a con 0x7fe444096a30 addr v1:172.21.15.102:6789/0 2021-05-25T12:05:39.128 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.094+0000 7fe448cc4700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe444132310 con 0x7fe444096a30 2021-05-25T12:05:39.128 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.094+0000 7fe448cc4700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe44413bbb0 con 0x7fe444093bd0 2021-05-25T12:05:39.128 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.094+0000 7fe448cc4700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe444134100 con 0x7fe444096630 2021-05-25T12:05:39.128 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.094+0000 7fe448cc4700 10 monclient(hunting): _renew_subs 2021-05-25T12:05:39.129 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.094+0000 7fe448cc4700 10 monclient(hunting): authenticate will time out at 1139.692671s 2021-05-25T12:05:39.129 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.094+0000 7fe442d9d700 1 -- >> v1:172.21.15.102:6789/0 conn(0x7fe444096a30 legacy=0x7fe4441432f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:05:39.129 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.094+0000 7fe441d9b700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fe444096630 0x7fe44413b4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:56138/0 (socket says 172.21.15.102:56138) 2021-05-25T12:05:39.129 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.094+0000 7fe441d9b700 1 -- 172.21.15.102:0/3974098820 learned_addr learned my addr 172.21.15.102:0/3974098820 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:05:39.130 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.094+0000 7fe42e7fc700 10 client.?.objecter ms_handle_connect 0x7fe444096630 2021-05-25T12:05:39.130 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.094+0000 7fe42e7fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:05:39.130 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.094+0000 7fe42e7fc700 1 -- 172.21.15.102:0/3974098820 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3115728949 0 0) 0x7fe444134100 con 0x7fe444096630 2021-05-25T12:05:39.130 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.094+0000 7fe42e7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:39.131 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.094+0000 7fe42e7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:39.131 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.094+0000 7fe42e7fc700 10 monclient(hunting): my global_id is 184382 2021-05-25T12:05:39.131 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.094+0000 7fe42e7fc700 1 -- 172.21.15.102:0/3974098820 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe420004c10 con 0x7fe444096630 2021-05-25T12:05:39.131 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.094+0000 7fe42e7fc700 10 client.?.objecter ms_handle_connect 0x7fe444093bd0 2021-05-25T12:05:39.132 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.094+0000 7fe42e7fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:05:39.132 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.095+0000 7fe42e7fc700 1 -- 172.21.15.102:0/3974098820 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 133837491 0 0) 0x7fe44413bbb0 con 0x7fe444093bd0 2021-05-25T12:05:39.132 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.095+0000 7fe42e7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:39.132 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.095+0000 7fe42e7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:39.132 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.095+0000 7fe42e7fc700 10 monclient(hunting): my global_id is 194188 2021-05-25T12:05:39.133 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.095+0000 7fe42e7fc700 1 -- 172.21.15.102:0/3974098820 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe444134100 con 0x7fe444093bd0 2021-05-25T12:05:39.133 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.095+0000 7fe42e7fc700 1 -- 172.21.15.102:0/3974098820 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2768290819 0 0) 0x7fe420004c10 con 0x7fe444096630 2021-05-25T12:05:39.133 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.095+0000 7fe42e7fc700 1 -- 172.21.15.102:0/3974098820 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe44413bbb0 con 0x7fe444096630 2021-05-25T12:05:39.133 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.095+0000 7fe42e7fc700 1 -- 172.21.15.102:0/3974098820 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fe430003eb0 con 0x7fe444096630 2021-05-25T12:05:39.134 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.095+0000 7fe42e7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:39.134 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.095+0000 7fe42e7fc700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:05:39.134 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.095+0000 7fe42e7fc700 10 monclient(hunting): dump: 2021-05-25T12:05:39.134 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:39.134 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:39.135 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:39.135 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:39.135 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:39.135 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:39.136 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:39.136 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:39.136 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:39.136 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:39.136 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.095+0000 7fe42e7fc700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:05:39.137 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.095+0000 7fe42e7fc700 1 -- 172.21.15.102:0/3974098820 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 472277439 0 0) 0x7fe44413bbb0 con 0x7fe444096630 2021-05-25T12:05:39.137 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.095+0000 7fe42e7fc700 1 -- 172.21.15.102:0/3974098820 >> v1:172.21.15.181:6789/0 conn(0x7fe444093bd0 legacy=0x7fe44413ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:39.137 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.095+0000 7fe42e7fc700 1 -- 172.21.15.102:0/3974098820 >> v1:172.21.15.102:6789/0 conn(0x7fe444096a30 legacy=0x7fe4441432f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-05-25T12:05:39.137 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.095+0000 7fe42e7fc700 10 monclient: _finish_hunting 0 2021-05-25T12:05:39.137 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.095+0000 7fe42e7fc700 1 monclient: found mon.c 2021-05-25T12:05:39.138 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.095+0000 7fe42e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:05:39.138 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.095+0000 7fe42e7fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:05:39.138 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.095+0000 7fe42e7fc700 1 -- 172.21.15.102:0/3974098820 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4441343a0 con 0x7fe444096630 2021-05-25T12:05:39.138 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.095+0000 7fe42e7fc700 10 monclient: _finish_auth 0 2021-05-25T12:05:39.139 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.095+0000 7fe42e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:05:39.139 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.095+0000 7fe448cc4700 5 monclient: authenticate success, global_id 184382 2021-05-25T12:05:39.139 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.095+0000 7fe448cc4700 10 monclient: _renew_subs 2021-05-25T12:05:39.139 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.095+0000 7fe448cc4700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:05:39.140 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.095+0000 7fe448cc4700 1 -- 172.21.15.102:0/3974098820 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe444096ef0 con 0x7fe444096630 2021-05-25T12:05:39.140 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.095+0000 7fe448cc4700 10 client.184382.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:05:39.140 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.095+0000 7fe448cc4700 10 monclient: _renew_subs 2021-05-25T12:05:39.140 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.095+0000 7fe448cc4700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:05:39.140 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.095+0000 7fe42e7fc700 1 -- 172.21.15.102:0/3974098820 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe4300040d0 con 0x7fe444096630 2021-05-25T12:05:39.141 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.095+0000 7fe448cc4700 1 -- 172.21.15.102:0/3974098820 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4441306c0 con 0x7fe444096630 2021-05-25T12:05:39.141 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.095+0000 7fe448cc4700 1 librados: init done 2021-05-25T12:05:39.141 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.095+0000 7fe42e7fc700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:05:39.141 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.095+0000 7fe42e7fc700 1 -- 172.21.15.102:0/3974098820 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fe4300043e0 con 0x7fe444096630 2021-05-25T12:05:39.142 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.095+0000 7fe42e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:39.142 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.095+0000 7fe42e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:05:39.142 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.095+0000 7fe440d99700 4 set_mon_vals no callback set 2021-05-25T12:05:39.142 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.095+0000 7fe42e7fc700 10 monclient: dump: 2021-05-25T12:05:39.143 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:39.143 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:39.143 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:39.143 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:39.143 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:39.144 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:39.144 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:39.144 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:39.144 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:39.145 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:39.145 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.095+0000 7fe42e7fc700 1 -- 172.21.15.102:0/3974098820 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7fe4300045c0 con 0x7fe444096630 2021-05-25T12:05:39.145 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.096+0000 7fe42e7fc700 1 -- 172.21.15.102:0/3974098820 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(171..171 src has 1..171) v4 ==== 6209+0+0 (unknown 228058464 0 0) 0x7fe43003d830 con 0x7fe444096630 2021-05-25T12:05:39.145 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.096+0000 7fe42e7fc700 10 client.184382.objecter ms_dispatch 0x7fe444001550 osd_map(171..171 src has 1..171) v4 2021-05-25T12:05:39.145 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.096+0000 7fe42e7fc700 3 client.184382.objecter handle_osd_map got epochs [171,171] > 0 2021-05-25T12:05:39.146 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.096+0000 7fe42e7fc700 3 client.184382.objecter handle_osd_map decoding full epoch 171 2021-05-25T12:05:39.146 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.096+0000 7fe42e7fc700 20 client.184382.objecter dump_active .. 0 homeless 2021-05-25T12:05:39.146 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.096+0000 7fe448cc4700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:05:39.146 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.096+0000 7fe448cc4700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:05:39.146 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.096+0000 7fe448cc4700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:05:39.147 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.096+0000 7fe448cc4700 1 -- 172.21.15.102:0/3974098820 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe444059730 con 0x7fe444096630 2021-05-25T12:05:39.147 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.096+0000 7fe44259c700 10 monclient: get_auth_request con 0x7fe4200331d0 auth_method 0 2021-05-25T12:05:39.147 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.096+0000 7fe42e7fc700 10 client.184382.objecter ms_handle_connect 0x7fe4200331d0 2021-05-25T12:05:39.147 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.100+0000 7fe42e7fc700 1 -- 172.21.15.102:0/3974098820 <== mon.2 v1:172.21.15.102:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7fe430004da0 con 0x7fe444096630 2021-05-25T12:05:39.147 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.100+0000 7fe42e7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:05:39.148 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.100+0000 7fe42e7fc700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:05:39.312 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.309+0000 7fe448cc4700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2021-05-25T12:05:39.312 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.309+0000 7fe448cc4700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2021-05-25T12:05:39.312 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.309+0000 7fe448cc4700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:05:39.313 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.309+0000 7fe448cc4700 1 -- 172.21.15.102:0/3974098820 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fe444135740 con 0x7fe444096630 2021-05-25T12:05:39.313 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.309+0000 7fe42e7fc700 1 -- 172.21.15.102:0/3974098820 <== mon.2 v1:172.21.15.102:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1342 (unknown 254746657 0 313461534) 0x7fe430012ce0 con 0x7fe444096630 2021-05-25T12:05:39.313 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.309+0000 7fe42e7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2021-05-25T12:05:39.313 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.309+0000 7fe42e7fc700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:05:39.314 INFO:teuthology.orchestra.run.smithi102.stdout:{"election_epoch":46,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":6,"features":{"quorum_con":"4540138303579357183","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"20ee91b6-8935-4b98-b868-85c03789ccc4","modified":"2021-05-25T11:59:11.011522Z","created":"2021-05-25T11:59:11.011522Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6790","nonce":0}]},"addr":"172.21.15.102:6790/0","public_addr":"172.21.15.102:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2021-05-25T12:05:39.315 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.311+0000 7fe448cc4700 10 librados: watch_flush enter 2021-05-25T12:05:39.316 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.311+0000 7fe448cc4700 10 librados: watch_flush exit 2021-05-25T12:05:39.316 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.311+0000 7fe448cc4700 20 client.184382.objecter shutdown clearing up homeless session... 2021-05-25T12:05:39.316 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.311+0000 7fe448cc4700 10 client.184382.objecter successfully canceled tick 2021-05-25T12:05:39.316 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.311+0000 7fe448cc4700 1 -- 172.21.15.102:0/3974098820 >> v1:172.21.15.181:6812/29582 conn(0x7fe4200331d0 legacy=0x7fe420035690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:39.317 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.311+0000 7fe448cc4700 10 monclient: shutdown 2021-05-25T12:05:39.317 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.311+0000 7fe448cc4700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:05:39.317 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.311+0000 7fe448cc4700 1 -- 172.21.15.102:0/3974098820 >> v1:172.21.15.102:6790/0 conn(0x7fe444096630 legacy=0x7fe44413b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:39.317 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.311+0000 7fe448cc4700 1 -- 172.21.15.102:0/3974098820 shutdown_connections 2021-05-25T12:05:39.317 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.311+0000 7fe448cc4700 1 -- 172.21.15.102:0/3974098820 >> 172.21.15.102:0/3974098820 conn(0x7fe44409cd10 msgr2=0x7fe444094cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:05:39.318 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.311+0000 7fe448cc4700 1 -- 172.21.15.102:0/3974098820 shutdown_connections 2021-05-25T12:05:39.318 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.311+0000 7fe448cc4700 1 -- 172.21.15.102:0/3974098820 wait complete. 2021-05-25T12:05:39.318 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:39.311+0000 7fe448cc4700 1 librados: shutdown 2021-05-25T12:05:42.325 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2021-05-25T12:05:42.443 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.442+0000 7fb35ee53700 10 monclient: get_monmap_and_config 2021-05-25T12:05:42.444 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.442+0000 7fb35ee53700 10 monclient: build_initial_monmap 2021-05-25T12:05:42.444 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.442+0000 7fb35ee53700 10 monclient: monmap: 2021-05-25T12:05:42.444 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:05:42.445 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:42.445 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:05:42.443384+0000 2021-05-25T12:05:42.445 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:05:42.443384+0000 2021-05-25T12:05:42.445 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:05:42.445 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:42.446 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:05:42.446 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:05:42.446 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:05:42.446 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:42.454 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.443+0000 7fb35ee53700 1 Processor -- start 2021-05-25T12:05:42.454 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.443+0000 7fb35ee53700 1 -- start start 2021-05-25T12:05:42.455 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.443+0000 7fb35ee53700 10 monclient: init 2021-05-25T12:05:42.455 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.443+0000 7fb35ee53700 10 monclient: _reopen_session rank -1 2021-05-25T12:05:42.455 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.443+0000 7fb35ee53700 10 monclient: _add_conns ranks=[0,1,2] 2021-05-25T12:05:42.455 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.443+0000 7fb35ee53700 10 monclient(hunting): picked mon.noname-a con 0x7fb358090990 addr v1:172.21.15.102:6789/0 2021-05-25T12:05:42.455 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.443+0000 7fb35ee53700 10 monclient(hunting): picked mon.noname-c con 0x7fb358090d70 addr v1:172.21.15.181:6789/0 2021-05-25T12:05:42.456 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.443+0000 7fb35ee53700 10 monclient(hunting): picked mon.noname-b con 0x7fb35812e330 addr v1:172.21.15.102:6790/0 2021-05-25T12:05:42.456 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.443+0000 7fb35ee53700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb358096fa0 con 0x7fb358090990 2021-05-25T12:05:42.456 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.443+0000 7fb35ee53700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3580926a0 con 0x7fb358090d70 2021-05-25T12:05:42.456 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.443+0000 7fb35ee53700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb358092940 con 0x7fb35812e330 2021-05-25T12:05:42.456 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.443+0000 7fb35ee53700 10 monclient(hunting): _renew_subs 2021-05-25T12:05:42.457 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.443+0000 7fb35ee53700 10 monclient(hunting): authenticate will time out at 1143.040812s 2021-05-25T12:05:42.457 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.443+0000 7fb35cbef700 1 -- >> v1:172.21.15.102:6789/0 conn(0x7fb358090990 legacy=0x7fb358093240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:05:42.457 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.443+0000 7fb35d3f0700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fb35812e330 0x7fb358130720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:56156/0 (socket says 172.21.15.102:56156) 2021-05-25T12:05:42.457 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.443+0000 7fb35d3f0700 1 -- 172.21.15.102:0/3342579155 learned_addr learned my addr 172.21.15.102:0/3342579155 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:05:42.457 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.443+0000 7fb34f7fe700 1 -- 172.21.15.102:0/3342579155 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2670508250 0 0) 0x7fb358092940 con 0x7fb35812e330 2021-05-25T12:05:42.458 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.443+0000 7fb34f7fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:42.458 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.443+0000 7fb34f7fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:42.458 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.443+0000 7fb34f7fe700 10 monclient(hunting): my global_id is 184385 2021-05-25T12:05:42.458 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.443+0000 7fb34f7fe700 1 -- 172.21.15.102:0/3342579155 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb344003270 con 0x7fb35812e330 2021-05-25T12:05:42.458 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.443+0000 7fb34f7fe700 1 -- 172.21.15.102:0/3342579155 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1027039818 0 0) 0x7fb344003270 con 0x7fb35812e330 2021-05-25T12:05:42.459 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.443+0000 7fb34f7fe700 1 -- 172.21.15.102:0/3342579155 >> v1:172.21.15.181:6789/0 conn(0x7fb358090d70 legacy=0x7fb358091200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:42.459 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.443+0000 7fb34f7fe700 1 -- 172.21.15.102:0/3342579155 >> v1:172.21.15.102:6789/0 conn(0x7fb358090990 legacy=0x7fb358093240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-05-25T12:05:42.459 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.443+0000 7fb34f7fe700 10 monclient: _finish_hunting 0 2021-05-25T12:05:42.459 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.443+0000 7fb34f7fe700 1 monclient: found mon.noname-b 2021-05-25T12:05:42.459 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.443+0000 7fb34f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:05:42.460 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.443+0000 7fb34f7fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T12:05:42.460 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.443+0000 7fb34f7fe700 1 -- 172.21.15.102:0/3342579155 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb358092be0 con 0x7fb35812e330 2021-05-25T12:05:42.460 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.443+0000 7fb34f7fe700 10 monclient: _finish_auth 0 2021-05-25T12:05:42.460 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.443+0000 7fb34f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:05:42.460 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.443+0000 7fb34f7fe700 1 -- 172.21.15.102:0/3342579155 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fb354001990 con 0x7fb35812e330 2021-05-25T12:05:42.461 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.443+0000 7fb35ee53700 5 monclient: authenticate success, global_id 184385 2021-05-25T12:05:42.461 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.443+0000 7fb35ee53700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:05:42.461 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.443+0000 7fb34f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:42.461 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.443+0000 7fb34f7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:05:42.461 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.443+0000 7fb34f7fe700 10 monclient: dump: 2021-05-25T12:05:42.461 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:42.462 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:42.462 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:42.462 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:42.462 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:42.462 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:42.463 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:42.463 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:42.463 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:42.463 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:42.463 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.443+0000 7fb35ee53700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:05:42.464 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.443+0000 7fb34f7fe700 1 -- 172.21.15.102:0/3342579155 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb354001c40 con 0x7fb35812e330 2021-05-25T12:05:42.464 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.443+0000 7fb34f7fe700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:05:42.464 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.443+0000 7fb35ee53700 10 monclient: get_monmap_and_config success 2021-05-25T12:05:42.464 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.443+0000 7fb34f7fe700 1 -- 172.21.15.102:0/3342579155 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fb354001fd0 con 0x7fb35812e330 2021-05-25T12:05:42.465 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.443+0000 7fb35ee53700 4 set_mon_vals no callback set 2021-05-25T12:05:42.465 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.443+0000 7fb34f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:42.465 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.443+0000 7fb34f7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:05:42.465 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.443+0000 7fb34f7fe700 10 monclient: dump: 2021-05-25T12:05:42.465 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:42.466 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:42.466 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:42.466 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:42.466 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:42.466 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:42.467 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:42.467 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:42.467 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:42.467 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:42.467 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.443+0000 7fb35ee53700 10 monclient: shutdown 2021-05-25T12:05:42.468 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.443+0000 7fb35ee53700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:05:42.468 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.443+0000 7fb35ee53700 1 -- 172.21.15.102:0/3342579155 >> v1:172.21.15.102:6790/0 conn(0x7fb35812e330 legacy=0x7fb358130720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:42.468 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.443+0000 7fb35ee53700 1 -- 172.21.15.102:0/3342579155 shutdown_connections 2021-05-25T12:05:42.468 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.443+0000 7fb35ee53700 1 -- 172.21.15.102:0/3342579155 >> 172.21.15.102:0/3342579155 conn(0x7fb35809b370 msgr2=0x7fb358097430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:05:42.468 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.444+0000 7fb35ee53700 1 -- 172.21.15.102:0/3342579155 shutdown_connections 2021-05-25T12:05:42.469 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.444+0000 7fb35ee53700 1 -- 172.21.15.102:0/3342579155 wait complete. 2021-05-25T12:05:42.469 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.444+0000 7fb35ee53700 10 monclient: build_initial_monmap 2021-05-25T12:05:42.469 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.444+0000 7fb35ee53700 10 monclient: monmap: 2021-05-25T12:05:42.469 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:05:42.469 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:05:42.470 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:05:42.470 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:05:42.470 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:05:42.470 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:05:42.470 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:05:42.471 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:05:42.471 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:05:42.471 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:42.471 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.444+0000 7fb35ee53700 1 librados: starting msgr at 2021-05-25T12:05:42.471 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.444+0000 7fb35ee53700 1 librados: starting objecter 2021-05-25T12:05:42.472 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.444+0000 7fb35ee53700 1 Processor -- start 2021-05-25T12:05:42.472 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.445+0000 7fb35ee53700 1 -- start start 2021-05-25T12:05:42.472 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.445+0000 7fb35ee53700 1 librados: setting wanted keys 2021-05-25T12:05:42.472 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.445+0000 7fb35ee53700 1 librados: calling monclient init 2021-05-25T12:05:42.472 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.445+0000 7fb35ee53700 10 monclient: init 2021-05-25T12:05:42.473 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.445+0000 7fb35ee53700 10 monclient: _reopen_session rank -1 2021-05-25T12:05:42.473 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.445+0000 7fb35ee53700 10 monclient: _add_conns ranks=[2,1,0] 2021-05-25T12:05:42.473 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.445+0000 7fb35ee53700 10 monclient(hunting): picked mon.noname-c con 0x7fb358090990 addr v1:172.21.15.102:6790/0 2021-05-25T12:05:42.473 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.445+0000 7fb35ee53700 10 monclient(hunting): picked mon.noname-b con 0x7fb358090d70 addr v1:172.21.15.181:6789/0 2021-05-25T12:05:42.474 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.445+0000 7fb35ee53700 10 monclient(hunting): picked mon.noname-a con 0x7fb35812e330 addr v1:172.21.15.102:6789/0 2021-05-25T12:05:42.474 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.445+0000 7fb35ee53700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb358128c90 con 0x7fb35812e330 2021-05-25T12:05:42.474 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.445+0000 7fb35ee53700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3581127b0 con 0x7fb358090d70 2021-05-25T12:05:42.474 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.445+0000 7fb35ee53700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb358112a50 con 0x7fb358090990 2021-05-25T12:05:42.474 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.445+0000 7fb35ee53700 10 monclient(hunting): _renew_subs 2021-05-25T12:05:42.475 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.445+0000 7fb35ee53700 10 monclient(hunting): authenticate will time out at 1143.043525s 2021-05-25T12:05:42.475 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.445+0000 7fb35d3f0700 1 -- >> v1:172.21.15.102:6789/0 conn(0x7fb35812e330 legacy=0x7fb35813ef60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:05:42.475 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.445+0000 7fb35cbef700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fb358090990 0x7fb358128580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:56158/0 (socket says 172.21.15.102:56158) 2021-05-25T12:05:42.475 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.445+0000 7fb35cbef700 1 -- 172.21.15.102:0/1857500916 learned_addr learned my addr 172.21.15.102:0/1857500916 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:05:42.475 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.445+0000 7fb34cff9700 10 client.?.objecter ms_handle_connect 0x7fb358090990 2021-05-25T12:05:42.476 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.445+0000 7fb34cff9700 10 client.?.objecter resend_mon_ops 2021-05-25T12:05:42.476 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.445+0000 7fb34cff9700 1 -- 172.21.15.102:0/1857500916 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3510325213 0 0) 0x7fb358112a50 con 0x7fb358090990 2021-05-25T12:05:42.476 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.445+0000 7fb34cff9700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:42.476 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.445+0000 7fb34cff9700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:42.476 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.445+0000 7fb34cff9700 10 monclient(hunting): my global_id is 184388 2021-05-25T12:05:42.477 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.445+0000 7fb34cff9700 1 -- 172.21.15.102:0/1857500916 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb338004c10 con 0x7fb358090990 2021-05-25T12:05:42.477 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.445+0000 7fb34cff9700 10 client.?.objecter ms_handle_connect 0x7fb358090d70 2021-05-25T12:05:42.477 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.445+0000 7fb34cff9700 10 client.?.objecter resend_mon_ops 2021-05-25T12:05:42.477 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.446+0000 7fb34cff9700 1 -- 172.21.15.102:0/1857500916 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4001894308 0 0) 0x7fb338004c10 con 0x7fb358090990 2021-05-25T12:05:42.477 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.446+0000 7fb34cff9700 1 -- 172.21.15.102:0/1857500916 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb358112a50 con 0x7fb358090990 2021-05-25T12:05:42.478 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.446+0000 7fb34cff9700 1 -- 172.21.15.102:0/1857500916 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fb348003eb0 con 0x7fb358090990 2021-05-25T12:05:42.478 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.446+0000 7fb34cff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:42.478 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.446+0000 7fb34cff9700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:05:42.478 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.446+0000 7fb34cff9700 10 monclient(hunting): dump: 2021-05-25T12:05:42.478 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:42.479 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:42.479 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:42.479 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:42.479 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:42.479 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:42.480 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:42.480 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:42.480 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:42.480 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:42.480 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.446+0000 7fb34cff9700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:05:42.481 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.446+0000 7fb34cff9700 1 -- 172.21.15.102:0/1857500916 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3279592875 0 0) 0x7fb358112a50 con 0x7fb358090990 2021-05-25T12:05:42.481 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.446+0000 7fb34cff9700 1 -- 172.21.15.102:0/1857500916 >> v1:172.21.15.181:6789/0 conn(0x7fb358090d70 legacy=0x7fb358136bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:42.481 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.446+0000 7fb34cff9700 1 -- 172.21.15.102:0/1857500916 >> v1:172.21.15.102:6789/0 conn(0x7fb35812e330 legacy=0x7fb35813ef60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-05-25T12:05:42.481 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.446+0000 7fb34cff9700 10 monclient: _finish_hunting 0 2021-05-25T12:05:42.481 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.446+0000 7fb34cff9700 1 monclient: found mon.c 2021-05-25T12:05:42.482 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.446+0000 7fb34cff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:05:42.482 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.446+0000 7fb34cff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:05:42.482 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.446+0000 7fb34cff9700 1 -- 172.21.15.102:0/1857500916 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb358112cf0 con 0x7fb358090990 2021-05-25T12:05:42.482 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.446+0000 7fb34cff9700 10 monclient: _finish_auth 0 2021-05-25T12:05:42.482 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.446+0000 7fb34cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:05:42.483 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.446+0000 7fb35ee53700 5 monclient: authenticate success, global_id 184388 2021-05-25T12:05:42.483 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.446+0000 7fb35ee53700 10 monclient: _renew_subs 2021-05-25T12:05:42.483 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.446+0000 7fb35ee53700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:05:42.483 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.446+0000 7fb35ee53700 1 -- 172.21.15.102:0/1857500916 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3581372c0 con 0x7fb358090990 2021-05-25T12:05:42.483 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.446+0000 7fb35ee53700 10 client.184388.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:05:42.484 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.446+0000 7fb35ee53700 10 monclient: _renew_subs 2021-05-25T12:05:42.484 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.446+0000 7fb35ee53700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:05:42.484 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.446+0000 7fb35ee53700 1 -- 172.21.15.102:0/1857500916 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb358111b00 con 0x7fb358090990 2021-05-25T12:05:42.484 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.446+0000 7fb35ee53700 1 librados: init done 2021-05-25T12:05:42.484 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.446+0000 7fb34cff9700 1 -- 172.21.15.102:0/1857500916 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb348004160 con 0x7fb358090990 2021-05-25T12:05:42.485 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.446+0000 7fb34cff9700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:05:42.485 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.446+0000 7fb34cff9700 1 -- 172.21.15.102:0/1857500916 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fb3480044f0 con 0x7fb358090990 2021-05-25T12:05:42.485 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.446+0000 7fb34cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:42.485 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.446+0000 7fb34effd700 4 set_mon_vals no callback set 2021-05-25T12:05:42.485 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.446+0000 7fb34cff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:05:42.486 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.446+0000 7fb34cff9700 10 monclient: dump: 2021-05-25T12:05:42.486 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:42.486 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:42.486 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:42.486 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:42.487 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:42.487 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:42.487 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:42.487 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:42.487 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:42.488 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:42.488 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.447+0000 7fb34cff9700 1 -- 172.21.15.102:0/1857500916 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7fb3480046d0 con 0x7fb358090990 2021-05-25T12:05:42.488 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.447+0000 7fb35ee53700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:05:42.488 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.447+0000 7fb35ee53700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:05:42.489 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.447+0000 7fb35ee53700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:05:42.489 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.447+0000 7fb34cff9700 1 -- 172.21.15.102:0/1857500916 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(171..171 src has 1..171) v4 ==== 6209+0+0 (unknown 228058464 0 0) 0x7fb34803d940 con 0x7fb358090990 2021-05-25T12:05:42.489 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.447+0000 7fb34cff9700 10 client.184388.objecter ms_dispatch 0x7fb358001550 osd_map(171..171 src has 1..171) v4 2021-05-25T12:05:42.489 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.447+0000 7fb35ee53700 1 -- 172.21.15.102:0/1857500916 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb330004db0 con 0x7fb358090990 2021-05-25T12:05:42.489 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.447+0000 7fb34cff9700 3 client.184388.objecter handle_osd_map got epochs [171,171] > 0 2021-05-25T12:05:42.490 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.447+0000 7fb34cff9700 3 client.184388.objecter handle_osd_map decoding full epoch 171 2021-05-25T12:05:42.490 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.447+0000 7fb34cff9700 20 client.184388.objecter dump_active .. 0 homeless 2021-05-25T12:05:42.490 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.447+0000 7fb34ffff700 10 monclient: get_auth_request con 0x7fb3380330b0 auth_method 0 2021-05-25T12:05:42.490 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.447+0000 7fb34cff9700 10 client.184388.objecter ms_handle_connect 0x7fb3380330b0 2021-05-25T12:05:42.490 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.451+0000 7fb34cff9700 1 -- 172.21.15.102:0/1857500916 <== mon.2 v1:172.21.15.102:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7fb348004eb0 con 0x7fb358090990 2021-05-25T12:05:42.491 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.451+0000 7fb34cff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:05:42.491 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.451+0000 7fb34cff9700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:05:42.662 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.659+0000 7fb35ee53700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2021-05-25T12:05:42.662 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.659+0000 7fb35ee53700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2021-05-25T12:05:42.662 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.659+0000 7fb35ee53700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:05:42.663 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.659+0000 7fb35ee53700 1 -- 172.21.15.102:0/1857500916 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fb330005060 con 0x7fb358090990 2021-05-25T12:05:42.663 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.659+0000 7fb34cff9700 1 -- 172.21.15.102:0/1857500916 <== mon.2 v1:172.21.15.102:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1342 (unknown 254746657 0 1227321437) 0x7fb348012dd0 con 0x7fb358090990 2021-05-25T12:05:42.663 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.659+0000 7fb34cff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2021-05-25T12:05:42.663 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.659+0000 7fb34cff9700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:05:42.664 INFO:teuthology.orchestra.run.smithi102.stdout:{"election_epoch":46,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":9,"features":{"quorum_con":"4540138303579357183","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"20ee91b6-8935-4b98-b868-85c03789ccc4","modified":"2021-05-25T11:59:11.011522Z","created":"2021-05-25T11:59:11.011522Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6790","nonce":0}]},"addr":"172.21.15.102:6790/0","public_addr":"172.21.15.102:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2021-05-25T12:05:42.665 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.661+0000 7fb35ee53700 10 librados: watch_flush enter 2021-05-25T12:05:42.666 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.661+0000 7fb35ee53700 10 librados: watch_flush exit 2021-05-25T12:05:42.666 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.661+0000 7fb35ee53700 20 client.184388.objecter shutdown clearing up homeless session... 2021-05-25T12:05:42.666 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.661+0000 7fb35ee53700 10 client.184388.objecter successfully canceled tick 2021-05-25T12:05:42.666 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.661+0000 7fb35ee53700 1 -- 172.21.15.102:0/1857500916 >> v1:172.21.15.181:6812/29582 conn(0x7fb3380330b0 legacy=0x7fb338035570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:42.667 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.661+0000 7fb35ee53700 10 monclient: shutdown 2021-05-25T12:05:42.667 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.661+0000 7fb35ee53700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:05:42.667 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.661+0000 7fb35ee53700 1 -- 172.21.15.102:0/1857500916 >> v1:172.21.15.102:6790/0 conn(0x7fb358090990 legacy=0x7fb358128580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:42.667 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.661+0000 7fb35ee53700 1 -- 172.21.15.102:0/1857500916 shutdown_connections 2021-05-25T12:05:42.667 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.661+0000 7fb35ee53700 1 -- 172.21.15.102:0/1857500916 >> 172.21.15.102:0/1857500916 conn(0x7fb35809b370 msgr2=0x7fb358129660 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:05:42.668 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.661+0000 7fb35ee53700 1 -- 172.21.15.102:0/1857500916 shutdown_connections 2021-05-25T12:05:42.668 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.661+0000 7fb35ee53700 1 -- 172.21.15.102:0/1857500916 wait complete. 2021-05-25T12:05:42.668 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:42.661+0000 7fb35ee53700 1 librados: shutdown 2021-05-25T12:05:45.676 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2021-05-25T12:05:45.798 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:45.796+0000 7f18e11d6700 10 monclient: get_monmap_and_config 2021-05-25T12:05:45.799 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:45.796+0000 7f18e11d6700 10 monclient: build_initial_monmap 2021-05-25T12:05:45.799 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:45.796+0000 7f18e11d6700 10 monclient: monmap: 2021-05-25T12:05:45.799 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:05:45.799 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:45.800 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:05:45.797075+0000 2021-05-25T12:05:45.800 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:05:45.797075+0000 2021-05-25T12:05:45.800 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:05:45.800 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:45.800 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:05:45.801 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:05:45.801 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:05:45.801 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:45.801 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:45.796+0000 7f18e11d6700 1 Processor -- start 2021-05-25T12:05:45.802 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:45.796+0000 7f18e11d6700 1 -- start start 2021-05-25T12:05:45.802 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:45.796+0000 7f18e11d6700 10 monclient: init 2021-05-25T12:05:45.802 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:45.797+0000 7f18e11d6700 10 monclient: _reopen_session rank -1 2021-05-25T12:05:45.802 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:45.797+0000 7f18e11d6700 10 monclient: _add_conns ranks=[0,1,2] 2021-05-25T12:05:45.802 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:45.797+0000 7f18e11d6700 10 monclient(hunting): picked mon.noname-a con 0x7f18dc128ec0 addr v1:172.21.15.102:6789/0 2021-05-25T12:05:45.803 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:45.797+0000 7f18e11d6700 10 monclient(hunting): picked mon.noname-c con 0x7f18dc094280 addr v1:172.21.15.181:6789/0 2021-05-25T12:05:45.803 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:45.797+0000 7f18e11d6700 10 monclient(hunting): picked mon.noname-b con 0x7f18dc094730 addr v1:172.21.15.102:6790/0 2021-05-25T12:05:45.803 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:45.797+0000 7f18e11d6700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18dc128ae0 con 0x7f18dc128ec0 2021-05-25T12:05:45.803 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:45.797+0000 7f18e11d6700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18dc069730 con 0x7f18dc094280 2021-05-25T12:05:45.804 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:45.797+0000 7f18e11d6700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18dc090b10 con 0x7f18dc094730 2021-05-25T12:05:45.804 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:45.797+0000 7f18e11d6700 10 monclient(hunting): _renew_subs 2021-05-25T12:05:45.804 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:45.797+0000 7f18e11d6700 10 monclient(hunting): authenticate will time out at 1146.394213s 2021-05-25T12:05:45.804 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:45.797+0000 7f18dad9d700 1 -- >> v1:172.21.15.102:6789/0 conn(0x7f18dc128ec0 legacy=0x7f18dc1283d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:05:45.804 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:45.797+0000 7f18db59e700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f18dc094730 0x7f18dc134350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:56170/0 (socket says 172.21.15.102:56170) 2021-05-25T12:05:45.805 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:45.797+0000 7f18db59e700 1 -- 172.21.15.102:0/1886462964 learned_addr learned my addr 172.21.15.102:0/1886462964 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:05:45.999 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:45.998+0000 7f18dad9d700 1 -- 172.21.15.102:0/1886462964 >> v1:172.21.15.102:6789/0 conn(0x7f18dc128ec0 legacy=0x7f18dc1283d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:05:46.400 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:46.399+0000 7f18dad9d700 1 -- 172.21.15.102:0/1886462964 >> v1:172.21.15.102:6789/0 conn(0x7f18dc128ec0 legacy=0x7f18dc1283d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:05:47.201 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:47.200+0000 7f18dad9d700 1 -- 172.21.15.102:0/1886462964 >> v1:172.21.15.102:6789/0 conn(0x7f18dc128ec0 legacy=0x7f18dc1283d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:05:48.799 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:48.797+0000 7f18d8d99700 10 monclient(hunting): tick 2021-05-25T12:05:48.799 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:48.797+0000 7f18d8d99700 1 monclient(hunting): continuing hunt 2021-05-25T12:05:48.799 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:48.797+0000 7f18d8d99700 10 monclient(hunting): _reopen_session rank -1 2021-05-25T12:05:48.800 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:48.797+0000 7f18d8d99700 1 -- 172.21.15.102:0/1886462964 >> v1:172.21.15.102:6790/0 conn(0x7f18dc094730 legacy=0x7f18dc134350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:48.800 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:48.797+0000 7f18d8d99700 1 -- 172.21.15.102:0/1886462964 >> v1:172.21.15.181:6789/0 conn(0x7f18dc094280 legacy=0x7f18dc130c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:48.800 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:48.797+0000 7f18d8d99700 1 -- 172.21.15.102:0/1886462964 >> v1:172.21.15.102:6789/0 conn(0x7f18dc128ec0 legacy=0x7f18dc1283d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-05-25T12:05:48.800 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:48.797+0000 7f18d8d99700 10 monclient: _add_conns ranks=[1,2,0] 2021-05-25T12:05:48.800 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:48.797+0000 7f18d8d99700 10 monclient(hunting): picked mon.noname-c con 0x7f18bc0033e0 addr v1:172.21.15.181:6789/0 2021-05-25T12:05:48.801 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:48.797+0000 7f18d8d99700 10 monclient(hunting): picked mon.noname-b con 0x7f18dc094730 addr v1:172.21.15.102:6790/0 2021-05-25T12:05:48.801 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:48.797+0000 7f18d8d99700 10 monclient(hunting): picked mon.noname-a con 0x7f18dc128ec0 addr v1:172.21.15.102:6789/0 2021-05-25T12:05:48.801 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:48.797+0000 7f18d8d99700 1 -- 172.21.15.102:0/1886462964 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18dc090db0 con 0x7f18dc128ec0 2021-05-25T12:05:48.801 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:48.797+0000 7f18d8d99700 1 -- 172.21.15.102:0/1886462964 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18dc128ae0 con 0x7f18bc0033e0 2021-05-25T12:05:48.801 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:48.797+0000 7f18d8d99700 1 -- 172.21.15.102:0/1886462964 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18bc0100c0 con 0x7f18dc094730 2021-05-25T12:05:48.802 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:48.797+0000 7f18d8d99700 10 monclient(hunting): _renew_subs 2021-05-25T12:05:48.802 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:48.797+0000 7f18db59e700 1 -- 172.21.15.102:0/1886462964 >> v1:172.21.15.102:6789/0 conn(0x7f18dc128ec0 legacy=0x7f18bc00c8f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:05:48.999 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:48.998+0000 7f18db59e700 1 -- 172.21.15.102:0/1886462964 >> v1:172.21.15.102:6789/0 conn(0x7f18dc128ec0 legacy=0x7f18bc00c8f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:05:49.400 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:49.399+0000 7f18db59e700 1 -- 172.21.15.102:0/1886462964 >> v1:172.21.15.102:6789/0 conn(0x7f18dc128ec0 legacy=0x7f18bc00c8f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:05:50.201 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:50.200+0000 7f18db59e700 1 -- 172.21.15.102:0/1886462964 >> v1:172.21.15.102:6789/0 conn(0x7f18dc128ec0 legacy=0x7f18bc00c8f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:05:51.799 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:51.797+0000 7f18d8d99700 10 monclient(hunting): tick 2021-05-25T12:05:51.800 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:51.797+0000 7f18d8d99700 1 monclient(hunting): continuing hunt 2021-05-25T12:05:51.800 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:51.797+0000 7f18d8d99700 10 monclient(hunting): _reopen_session rank -1 2021-05-25T12:05:51.800 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:51.797+0000 7f18d8d99700 1 -- 172.21.15.102:0/1886462964 >> v1:172.21.15.102:6790/0 conn(0x7f18dc094730 legacy=0x7f18bc0090f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:51.800 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:51.797+0000 7f18d8d99700 1 -- 172.21.15.102:0/1886462964 >> v1:172.21.15.181:6789/0 conn(0x7f18bc0033e0 legacy=0x7f18bc005830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:51.801 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:51.797+0000 7f18d8d99700 1 -- 172.21.15.102:0/1886462964 >> v1:172.21.15.102:6789/0 conn(0x7f18dc128ec0 legacy=0x7f18bc00c8f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-05-25T12:05:51.801 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:51.797+0000 7f18d8d99700 10 monclient: _add_conns ranks=[2,0,1] 2021-05-25T12:05:51.801 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:51.797+0000 7f18d8d99700 10 monclient(hunting): picked mon.noname-b con 0x7f18bc00f140 addr v1:172.21.15.102:6790/0 2021-05-25T12:05:51.801 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:51.797+0000 7f18d8d99700 10 monclient(hunting): picked mon.noname-a con 0x7f18dc128ec0 addr v1:172.21.15.102:6789/0 2021-05-25T12:05:51.802 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:51.797+0000 7f18d8d99700 10 monclient(hunting): picked mon.noname-c con 0x7f18bc0033e0 addr v1:172.21.15.181:6789/0 2021-05-25T12:05:51.802 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:51.797+0000 7f18d8d99700 1 -- 172.21.15.102:0/1886462964 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18bc011330 con 0x7f18dc128ec0 2021-05-25T12:05:51.802 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:51.797+0000 7f18d8d99700 1 -- 172.21.15.102:0/1886462964 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18dc090db0 con 0x7f18bc0033e0 2021-05-25T12:05:51.802 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:51.797+0000 7f18d8d99700 1 -- 172.21.15.102:0/1886462964 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18bc00fe30 con 0x7f18bc00f140 2021-05-25T12:05:51.803 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:51.797+0000 7f18da59c700 1 -- 172.21.15.102:0/1886462964 >> v1:172.21.15.102:6789/0 conn(0x7f18dc128ec0 legacy=0x7f18bc00c8f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:05:51.803 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:51.797+0000 7f18d8d99700 10 monclient(hunting): _renew_subs 2021-05-25T12:05:51.999 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:51.998+0000 7f18da59c700 1 -- 172.21.15.102:0/1886462964 >> v1:172.21.15.102:6789/0 conn(0x7f18dc128ec0 legacy=0x7f18bc00c8f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:05:52.400 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:52.399+0000 7f18da59c700 1 -- 172.21.15.102:0/1886462964 >> v1:172.21.15.102:6789/0 conn(0x7f18dc128ec0 legacy=0x7f18bc00c8f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:05:53.201 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.200+0000 7f18da59c700 1 -- 172.21.15.102:0/1886462964 >> v1:172.21.15.102:6789/0 conn(0x7f18dc128ec0 legacy=0x7f18bc00c8f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:05:53.834 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.833+0000 7f18d9d9b700 1 -- 172.21.15.102:0/1886462964 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3013428818 0 0) 0x7f18dc090db0 con 0x7f18bc0033e0 2021-05-25T12:05:53.835 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.833+0000 7f18d9d9b700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:53.835 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.833+0000 7f18d9d9b700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:53.835 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.833+0000 7f18d9d9b700 10 monclient(hunting): my global_id is 194191 2021-05-25T12:05:53.835 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.833+0000 7f18d9d9b700 1 -- 172.21.15.102:0/1886462964 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18c8003270 con 0x7f18bc0033e0 2021-05-25T12:05:53.845 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.844+0000 7f18d9d9b700 1 -- 172.21.15.102:0/1886462964 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4261566999 0 0) 0x7f18bc00fe30 con 0x7f18bc00f140 2021-05-25T12:05:53.845 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.844+0000 7f18d9d9b700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:53.846 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.844+0000 7f18d9d9b700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:53.846 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.844+0000 7f18d9d9b700 10 monclient(hunting): my global_id is 184391 2021-05-25T12:05:53.846 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.844+0000 7f18d9d9b700 1 -- 172.21.15.102:0/1886462964 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18dc090db0 con 0x7f18bc00f140 2021-05-25T12:05:53.848 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.846+0000 7f18d9d9b700 1 -- 172.21.15.102:0/1886462964 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2264827542 0 0) 0x7f18dc090db0 con 0x7f18bc00f140 2021-05-25T12:05:53.848 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.846+0000 7f18d9d9b700 1 -- 172.21.15.102:0/1886462964 >> v1:172.21.15.181:6789/0 conn(0x7f18bc0033e0 legacy=0x7f18bc005830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:53.848 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.846+0000 7f18d9d9b700 1 -- 172.21.15.102:0/1886462964 >> v1:172.21.15.102:6789/0 conn(0x7f18dc128ec0 legacy=0x7f18bc00c8f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-05-25T12:05:53.849 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.846+0000 7f18d9d9b700 10 monclient: _finish_hunting 0 2021-05-25T12:05:53.849 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.846+0000 7f18d9d9b700 1 monclient: found mon.noname-b 2021-05-25T12:05:53.849 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.846+0000 7f18d9d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:05:53.849 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.846+0000 7f18d9d9b700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T12:05:53.850 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.846+0000 7f18d9d9b700 1 -- 172.21.15.102:0/1886462964 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18bc0080b0 con 0x7f18bc00f140 2021-05-25T12:05:53.850 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.846+0000 7f18d9d9b700 10 monclient: _finish_auth 0 2021-05-25T12:05:53.850 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.846+0000 7f18d9d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:05:53.850 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.846+0000 7f18d9d9b700 1 -- 172.21.15.102:0/1886462964 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f18d0003f20 con 0x7f18bc00f140 2021-05-25T12:05:53.850 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.846+0000 7f18d9d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:53.851 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.846+0000 7f18d9d9b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:05:53.851 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.846+0000 7f18d9d9b700 10 monclient: dump: 2021-05-25T12:05:53.851 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:53.851 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:53.852 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:53.852 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:53.852 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:53.852 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:53.852 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:53.853 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:53.853 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:53.853 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:53.854 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.847+0000 7f18e11d6700 5 monclient: authenticate success, global_id 184391 2021-05-25T12:05:53.854 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.847+0000 7f18e11d6700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:05:53.856 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.847+0000 7f18d9d9b700 1 -- 172.21.15.102:0/1886462964 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f18d0004140 con 0x7f18bc00f140 2021-05-25T12:05:53.857 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.847+0000 7f18d9d9b700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:05:53.857 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.847+0000 7f18d9d9b700 1 -- 172.21.15.102:0/1886462964 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f18d0004510 con 0x7f18bc00f140 2021-05-25T12:05:53.857 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.847+0000 7f18d9d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:53.857 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.847+0000 7f18d9d9b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:05:53.857 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.847+0000 7f18d9d9b700 10 monclient: dump: 2021-05-25T12:05:53.858 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:53.858 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:53.858 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:53.858 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:53.859 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:53.859 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:53.859 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:53.859 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:53.859 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:53.860 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:53.860 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.847+0000 7f18e11d6700 10 monclient: get_monmap_and_config success 2021-05-25T12:05:53.860 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.847+0000 7f18e11d6700 4 set_mon_vals no callback set 2021-05-25T12:05:53.860 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.848+0000 7f18e11d6700 10 monclient: shutdown 2021-05-25T12:05:53.861 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.848+0000 7f18e11d6700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:05:53.861 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.848+0000 7f18e11d6700 1 -- 172.21.15.102:0/1886462964 >> v1:172.21.15.102:6790/0 conn(0x7f18bc00f140 legacy=0x7f18bc00f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:53.861 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.849+0000 7f18e11d6700 1 -- 172.21.15.102:0/1886462964 shutdown_connections 2021-05-25T12:05:53.861 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.849+0000 7f18e11d6700 1 -- 172.21.15.102:0/1886462964 >> 172.21.15.102:0/1886462964 conn(0x7f18dc09b370 msgr2=0x7f18dc097430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:05:53.862 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.849+0000 7f18e11d6700 1 -- 172.21.15.102:0/1886462964 shutdown_connections 2021-05-25T12:05:53.862 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.849+0000 7f18e11d6700 1 -- 172.21.15.102:0/1886462964 wait complete. 2021-05-25T12:05:53.862 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.850+0000 7f18e11d6700 10 monclient: build_initial_monmap 2021-05-25T12:05:53.862 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.850+0000 7f18e11d6700 10 monclient: monmap: 2021-05-25T12:05:53.862 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:05:53.863 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:05:53.863 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:05:53.863 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:05:53.864 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:05:53.864 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:05:53.864 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:05:53.864 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:05:53.865 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:05:53.865 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:53.865 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.850+0000 7f18e11d6700 1 librados: starting msgr at 2021-05-25T12:05:53.865 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.850+0000 7f18e11d6700 1 librados: starting objecter 2021-05-25T12:05:53.866 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.850+0000 7f18e11d6700 1 Processor -- start 2021-05-25T12:05:53.866 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.851+0000 7f18e11d6700 1 -- start start 2021-05-25T12:05:53.866 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.851+0000 7f18e11d6700 1 librados: setting wanted keys 2021-05-25T12:05:53.866 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.851+0000 7f18e11d6700 1 librados: calling monclient init 2021-05-25T12:05:53.867 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.851+0000 7f18e11d6700 10 monclient: init 2021-05-25T12:05:53.867 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.852+0000 7f18e11d6700 10 monclient: _reopen_session rank -1 2021-05-25T12:05:53.867 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.852+0000 7f18e11d6700 10 monclient: _add_conns ranks=[1,0,2] 2021-05-25T12:05:53.867 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.852+0000 7f18e11d6700 10 monclient(hunting): picked mon.noname-b con 0x7f18bc0033e0 addr v1:172.21.15.181:6789/0 2021-05-25T12:05:53.868 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.852+0000 7f18e11d6700 10 monclient(hunting): picked mon.noname-a con 0x7f18bc00f140 addr v1:172.21.15.102:6789/0 2021-05-25T12:05:53.868 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.852+0000 7f18e11d6700 10 monclient(hunting): picked mon.noname-c con 0x7f18dc128ec0 addr v1:172.21.15.102:6790/0 2021-05-25T12:05:53.868 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.852+0000 7f18e11d6700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18dc13bcf0 con 0x7f18bc00f140 2021-05-25T12:05:53.868 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.852+0000 7f18e11d6700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18dc1128f0 con 0x7f18bc0033e0 2021-05-25T12:05:53.868 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.852+0000 7f18e11d6700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18dc112b90 con 0x7f18dc128ec0 2021-05-25T12:05:53.869 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.852+0000 7f18e11d6700 10 monclient(hunting): _renew_subs 2021-05-25T12:05:53.869 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.852+0000 7f18e11d6700 10 monclient(hunting): authenticate will time out at 1154.449651s 2021-05-25T12:05:53.869 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.852+0000 7f18da59c700 1 -- >> v1:172.21.15.102:6789/0 conn(0x7f18bc00f140 legacy=0x7f18dc140170 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:05:53.869 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.852+0000 7f18dad9d700 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f18bc0033e0 0x7f18dc125150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.102:46978/0 (socket says 172.21.15.102:46978) 2021-05-25T12:05:53.869 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.852+0000 7f18dad9d700 1 -- 172.21.15.102:0/2552467788 learned_addr learned my addr 172.21.15.102:0/2552467788 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:05:53.870 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.853+0000 7f18c2ffd700 10 client.?.objecter ms_handle_connect 0x7f18bc0033e0 2021-05-25T12:05:53.870 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.853+0000 7f18c2ffd700 10 client.?.objecter resend_mon_ops 2021-05-25T12:05:53.870 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.853+0000 7f18c2ffd700 10 client.?.objecter ms_handle_connect 0x7f18dc128ec0 2021-05-25T12:05:53.870 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.853+0000 7f18c2ffd700 10 client.?.objecter resend_mon_ops 2021-05-25T12:05:53.887 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.883+0000 7f18c2ffd700 1 -- 172.21.15.102:0/2552467788 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1088771241 0 0) 0x7f18dc1128f0 con 0x7f18bc0033e0 2021-05-25T12:05:53.887 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.883+0000 7f18c2ffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:53.888 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.883+0000 7f18c2ffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:53.888 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.883+0000 7f18c2ffd700 10 monclient(hunting): my global_id is 194194 2021-05-25T12:05:53.888 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.883+0000 7f18c2ffd700 1 -- 172.21.15.102:0/2552467788 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18bc00fe30 con 0x7f18bc0033e0 2021-05-25T12:05:53.888 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.883+0000 7f18c2ffd700 1 -- 172.21.15.102:0/2552467788 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2901752297 0 0) 0x7f18bc00fe30 con 0x7f18bc0033e0 2021-05-25T12:05:53.888 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.883+0000 7f18c2ffd700 1 -- 172.21.15.102:0/2552467788 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f18dc1128f0 con 0x7f18bc0033e0 2021-05-25T12:05:53.889 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.883+0000 7f18c2ffd700 1 -- 172.21.15.102:0/2552467788 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f18d0004450 con 0x7f18bc0033e0 2021-05-25T12:05:53.889 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.883+0000 7f18c2ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:53.889 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.883+0000 7f18c2ffd700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:05:53.889 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.883+0000 7f18c2ffd700 10 monclient(hunting): dump: 2021-05-25T12:05:53.889 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:53.890 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:53.890 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:53.890 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:53.890 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:53.891 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:53.891 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:53.891 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:53.891 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:53.891 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:53.892 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.883+0000 7f18c2ffd700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:05:53.892 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.883+0000 7f18c2ffd700 1 -- 172.21.15.102:0/2552467788 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1402158736 0 0) 0x7f18dc1128f0 con 0x7f18bc0033e0 2021-05-25T12:05:53.892 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.883+0000 7f18c2ffd700 1 -- 172.21.15.102:0/2552467788 >> v1:172.21.15.102:6790/0 conn(0x7f18dc128ec0 legacy=0x7f18dc1438a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:53.892 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.883+0000 7f18c2ffd700 1 -- 172.21.15.102:0/2552467788 >> v1:172.21.15.102:6789/0 conn(0x7f18bc00f140 legacy=0x7f18dc140170 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-05-25T12:05:53.892 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.883+0000 7f18c2ffd700 10 monclient: _finish_hunting 0 2021-05-25T12:05:53.893 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.883+0000 7f18c2ffd700 1 monclient: found mon.b 2021-05-25T12:05:53.893 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.883+0000 7f18c2ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:05:53.893 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.883+0000 7f18c2ffd700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T12:05:53.893 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.883+0000 7f18c2ffd700 1 -- 172.21.15.102:0/2552467788 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18dc112050 con 0x7f18bc0033e0 2021-05-25T12:05:53.894 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.883+0000 7f18c2ffd700 10 monclient: _finish_auth 0 2021-05-25T12:05:53.894 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.883+0000 7f18c2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:05:53.894 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.883+0000 7f18e11d6700 5 monclient: authenticate success, global_id 194194 2021-05-25T12:05:53.894 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.883+0000 7f18e11d6700 10 monclient: _renew_subs 2021-05-25T12:05:53.895 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.883+0000 7f18e11d6700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T12:05:53.895 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.883+0000 7f18e11d6700 1 -- 172.21.15.102:0/2552467788 --> v1:172.21.15.181:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18dc12c1b0 con 0x7f18bc0033e0 2021-05-25T12:05:53.895 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.883+0000 7f18e11d6700 10 client.194194.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:05:53.895 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.883+0000 7f18e11d6700 10 monclient: _renew_subs 2021-05-25T12:05:53.896 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.883+0000 7f18e11d6700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T12:05:53.896 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.883+0000 7f18e11d6700 1 -- 172.21.15.102:0/2552467788 --> v1:172.21.15.181:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f18dc130480 con 0x7f18bc0033e0 2021-05-25T12:05:53.896 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.883+0000 7f18e11d6700 1 librados: init done 2021-05-25T12:05:53.896 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.883+0000 7f18c2ffd700 1 -- 172.21.15.102:0/2552467788 <== mon.1 v1:172.21.15.181:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f18d0004ab0 con 0x7f18bc0033e0 2021-05-25T12:05:53.897 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.883+0000 7f18c2ffd700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:05:53.897 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.883+0000 7f18c2ffd700 1 -- 172.21.15.102:0/2552467788 <== mon.1 v1:172.21.15.181:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f18d0004dc0 con 0x7f18bc0033e0 2021-05-25T12:05:53.897 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.883+0000 7f18c2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:53.897 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.883+0000 7f18c2ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T12:05:53.897 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.883+0000 7f18c2ffd700 10 monclient: dump: 2021-05-25T12:05:53.898 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:53.898 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:53.898 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:53.898 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:53.899 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:53.899 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:53.899 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:53.899 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:53.899 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:53.900 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:53.900 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.883+0000 7f18d959a700 4 set_mon_vals no callback set 2021-05-25T12:05:53.900 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.884+0000 7f18c2ffd700 1 -- 172.21.15.102:0/2552467788 <== mon.1 v1:172.21.15.181:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f18d0004fe0 con 0x7f18bc0033e0 2021-05-25T12:05:53.900 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.884+0000 7f18e11d6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:05:53.900 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.884+0000 7f18e11d6700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:05:53.901 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.884+0000 7f18e11d6700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T12:05:53.901 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.884+0000 7f18e11d6700 1 -- 172.21.15.102:0/2552467788 --> v1:172.21.15.181:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18dc059730 con 0x7f18bc0033e0 2021-05-25T12:05:53.901 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.884+0000 7f18c2ffd700 1 -- 172.21.15.102:0/2552467788 <== mon.1 v1:172.21.15.181:6789/0 8 ==== osd_map(172..172 src has 1..172) v4 ==== 5856+0+0 (unknown 973787454 0 0) 0x7f18d003dba0 con 0x7f18bc0033e0 2021-05-25T12:05:53.901 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.884+0000 7f18c2ffd700 10 client.194194.objecter ms_dispatch 0x7f18dc001550 osd_map(172..172 src has 1..172) v4 2021-05-25T12:05:53.902 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.884+0000 7f18c2ffd700 3 client.194194.objecter handle_osd_map got epochs [172,172] > 0 2021-05-25T12:05:53.902 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.884+0000 7f18c2ffd700 3 client.194194.objecter handle_osd_map decoding full epoch 172 2021-05-25T12:05:53.902 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.885+0000 7f18c2ffd700 20 client.194194.objecter dump_active .. 0 homeless 2021-05-25T12:05:53.902 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.885+0000 7f18da59c700 10 monclient: get_auth_request con 0x7f18bc033ef0 auth_method 0 2021-05-25T12:05:53.902 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.885+0000 7f18c2ffd700 10 client.194194.objecter ms_handle_connect 0x7f18bc033ef0 2021-05-25T12:05:53.903 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.889+0000 7f18c2ffd700 1 -- 172.21.15.102:0/2552467788 <== mon.1 v1:172.21.15.181:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7f18d00131f0 con 0x7f18bc0033e0 2021-05-25T12:05:53.903 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.889+0000 7f18c2ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:05:53.904 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:53.889+0000 7f18c2ffd700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:05:54.106 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.104+0000 7f18e11d6700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2021-05-25T12:05:54.106 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.104+0000 7f18e11d6700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2021-05-25T12:05:54.106 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.104+0000 7f18e11d6700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T12:05:54.106 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.104+0000 7f18e11d6700 1 -- 172.21.15.102:0/2552467788 --> v1:172.21.15.181:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f18dc097e20 con 0x7f18bc0033e0 2021-05-25T12:05:54.107 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.104+0000 7f18c2ffd700 1 -- 172.21.15.102:0/2552467788 <== mon.1 v1:172.21.15.181:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1336 (unknown 254746657 0 1238638840) 0x7f18d0012de0 con 0x7f18bc0033e0 2021-05-25T12:05:54.107 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.104+0000 7f18c2ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2021-05-25T12:05:54.107 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.104+0000 7f18c2ffd700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:05:54.108 INFO:teuthology.orchestra.run.smithi102.stdout:{"election_epoch":50,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138303579357183","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"20ee91b6-8935-4b98-b868-85c03789ccc4","modified":"2021-05-25T11:59:11.011522Z","created":"2021-05-25T11:59:11.011522Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6790","nonce":0}]},"addr":"172.21.15.102:6790/0","public_addr":"172.21.15.102:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2021-05-25T12:05:54.109 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.106+0000 7f18e11d6700 10 librados: watch_flush enter 2021-05-25T12:05:54.110 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.106+0000 7f18e11d6700 10 librados: watch_flush exit 2021-05-25T12:05:54.110 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.106+0000 7f18e11d6700 20 client.194194.objecter shutdown clearing up homeless session... 2021-05-25T12:05:54.110 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.106+0000 7f18e11d6700 10 client.194194.objecter successfully canceled tick 2021-05-25T12:05:54.110 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.106+0000 7f18e11d6700 1 -- 172.21.15.102:0/2552467788 >> v1:172.21.15.181:6812/29582 conn(0x7f18bc033ef0 legacy=0x7f18bc0363b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:54.111 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.106+0000 7f18e11d6700 10 monclient: shutdown 2021-05-25T12:05:54.111 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.106+0000 7f18e11d6700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:05:54.111 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.106+0000 7f18e11d6700 1 -- 172.21.15.102:0/2552467788 >> v1:172.21.15.181:6789/0 conn(0x7f18bc0033e0 legacy=0x7f18dc125150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:54.111 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.106+0000 7f18e11d6700 1 -- 172.21.15.102:0/2552467788 shutdown_connections 2021-05-25T12:05:54.112 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.106+0000 7f18e11d6700 1 -- 172.21.15.102:0/2552467788 >> 172.21.15.102:0/2552467788 conn(0x7f18dc09b370 msgr2=0x7f18dc1281a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:05:54.112 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.106+0000 7f18e11d6700 1 -- 172.21.15.102:0/2552467788 shutdown_connections 2021-05-25T12:05:54.112 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.106+0000 7f18e11d6700 1 -- 172.21.15.102:0/2552467788 wait complete. 2021-05-25T12:05:54.112 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.106+0000 7f18e11d6700 1 librados: shutdown 2021-05-25T12:05:54.119 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2021-05-25T12:05:54.119 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2021-05-25T12:05:54.229 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.228+0000 7f918cec2700 10 monclient: get_monmap_and_config 2021-05-25T12:05:54.230 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.228+0000 7f918cec2700 10 monclient: build_initial_monmap 2021-05-25T12:05:54.230 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.228+0000 7f918cec2700 10 monclient: monmap: 2021-05-25T12:05:54.230 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:05:54.231 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:54.231 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:05:54.228951+0000 2021-05-25T12:05:54.232 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:05:54.228951+0000 2021-05-25T12:05:54.232 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:05:54.232 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:54.232 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:05:54.233 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:05:54.233 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:05:54.233 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:54.233 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.228+0000 7f918cec2700 1 Processor -- start 2021-05-25T12:05:54.233 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.228+0000 7f918cec2700 1 -- start start 2021-05-25T12:05:54.234 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.228+0000 7f918cec2700 10 monclient: init 2021-05-25T12:05:54.248 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.229+0000 7f918cec2700 10 monclient: _reopen_session rank -1 2021-05-25T12:05:54.248 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.229+0000 7f918cec2700 10 monclient: _add_conns ranks=[0,1,2] 2021-05-25T12:05:54.248 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.229+0000 7f918cec2700 10 monclient(hunting): picked mon.noname-a con 0x7f9188090990 addr v1:172.21.15.102:6789/0 2021-05-25T12:05:54.249 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.229+0000 7f918cec2700 10 monclient(hunting): picked mon.noname-c con 0x7f9188090d70 addr v1:172.21.15.181:6789/0 2021-05-25T12:05:54.249 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.229+0000 7f918cec2700 10 monclient(hunting): picked mon.noname-b con 0x7f918812e330 addr v1:172.21.15.102:6790/0 2021-05-25T12:05:54.249 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.229+0000 7f918cec2700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9188096fa0 con 0x7f9188090990 2021-05-25T12:05:54.249 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.229+0000 7f918cec2700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91880926a0 con 0x7f9188090d70 2021-05-25T12:05:54.250 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.229+0000 7f918cec2700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9188092940 con 0x7f918812e330 2021-05-25T12:05:54.250 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.229+0000 7f918cec2700 10 monclient(hunting): _renew_subs 2021-05-25T12:05:54.250 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.229+0000 7f918cec2700 10 monclient(hunting): authenticate will time out at 1154.826292s 2021-05-25T12:05:54.250 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.229+0000 7f918659c700 1 -- >> v1:172.21.15.102:6789/0 conn(0x7f9188090990 legacy=0x7f9188093240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:05:54.251 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.229+0000 7f9186d9d700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f918812e330 0x7f9188130720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:56218/0 (socket says 172.21.15.102:56218) 2021-05-25T12:05:54.251 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.229+0000 7f9186d9d700 1 -- 172.21.15.102:0/942332612 learned_addr learned my addr 172.21.15.102:0/942332612 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:05:54.251 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.229+0000 7f918559a700 1 -- 172.21.15.102:0/942332612 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2314834207 0 0) 0x7f91880926a0 con 0x7f9188090d70 2021-05-25T12:05:54.251 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.229+0000 7f918559a700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:54.252 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.229+0000 7f918559a700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:54.252 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.229+0000 7f918559a700 10 monclient(hunting): my global_id is 194203 2021-05-25T12:05:54.252 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.229+0000 7f918559a700 1 -- 172.21.15.102:0/942332612 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f916c003270 con 0x7f9188090d70 2021-05-25T12:05:54.252 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.229+0000 7f918559a700 1 -- 172.21.15.102:0/942332612 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 84342410 0 0) 0x7f9188092940 con 0x7f918812e330 2021-05-25T12:05:54.253 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.229+0000 7f918559a700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:54.253 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.229+0000 7f918559a700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:54.253 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.229+0000 7f918559a700 10 monclient(hunting): my global_id is 184394 2021-05-25T12:05:54.253 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.229+0000 7f918559a700 1 -- 172.21.15.102:0/942332612 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91880926a0 con 0x7f918812e330 2021-05-25T12:05:54.253 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.229+0000 7f918559a700 1 -- 172.21.15.102:0/942332612 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 874270764 0 0) 0x7f916c003270 con 0x7f9188090d70 2021-05-25T12:05:54.254 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.229+0000 7f918559a700 1 -- 172.21.15.102:0/942332612 >> v1:172.21.15.102:6790/0 conn(0x7f918812e330 legacy=0x7f9188130720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:54.254 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.229+0000 7f918559a700 1 -- 172.21.15.102:0/942332612 >> v1:172.21.15.102:6789/0 conn(0x7f9188090990 legacy=0x7f9188093240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-05-25T12:05:54.254 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.229+0000 7f918559a700 10 monclient: _finish_hunting 0 2021-05-25T12:05:54.254 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.229+0000 7f918559a700 1 monclient: found mon.noname-c 2021-05-25T12:05:54.255 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.229+0000 7f918559a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:05:54.255 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.229+0000 7f918559a700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.181:6789/0 2021-05-25T12:05:54.255 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.229+0000 7f918559a700 1 -- 172.21.15.102:0/942332612 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9188092be0 con 0x7f9188090d70 2021-05-25T12:05:54.255 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.229+0000 7f918559a700 10 monclient: _finish_auth 0 2021-05-25T12:05:54.255 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.229+0000 7f918559a700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:05:54.256 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.229+0000 7f918cec2700 5 monclient: authenticate success, global_id 194203 2021-05-25T12:05:54.256 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.229+0000 7f918559a700 1 -- 172.21.15.102:0/942332612 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f9170001610 con 0x7f9188090d70 2021-05-25T12:05:54.256 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.229+0000 7f918cec2700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:05:54.256 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.229+0000 7f918559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:54.257 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.229+0000 7f918559a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:05:54.257 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.229+0000 7f918559a700 10 monclient: dump: 2021-05-25T12:05:54.257 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:54.257 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:54.257 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:54.258 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:54.258 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:54.258 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:54.258 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:54.258 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:54.259 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:54.259 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:54.259 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.229+0000 7f918cec2700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:05:54.259 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.229+0000 7f918559a700 1 -- 172.21.15.102:0/942332612 <== mon.1 v1:172.21.15.181:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9170001880 con 0x7f9188090d70 2021-05-25T12:05:54.260 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.229+0000 7f918559a700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:05:54.260 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.229+0000 7f918cec2700 10 monclient: get_monmap_and_config success 2021-05-25T12:05:54.260 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.229+0000 7f918559a700 1 -- 172.21.15.102:0/942332612 <== mon.1 v1:172.21.15.181:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f9170001c10 con 0x7f9188090d70 2021-05-25T12:05:54.260 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.229+0000 7f918559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:54.260 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.229+0000 7f918cec2700 4 set_mon_vals no callback set 2021-05-25T12:05:54.261 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.229+0000 7f918559a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T12:05:54.261 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.229+0000 7f918559a700 10 monclient: dump: 2021-05-25T12:05:54.261 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:54.261 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:54.262 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:54.262 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:54.262 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:54.262 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:54.262 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:54.263 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:54.263 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:54.263 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:54.263 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.230+0000 7f918cec2700 10 monclient: shutdown 2021-05-25T12:05:54.264 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.230+0000 7f918cec2700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:05:54.264 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.230+0000 7f918cec2700 1 -- 172.21.15.102:0/942332612 >> v1:172.21.15.181:6789/0 conn(0x7f9188090d70 legacy=0x7f9188091200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:54.264 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.230+0000 7f918cec2700 1 -- 172.21.15.102:0/942332612 shutdown_connections 2021-05-25T12:05:54.264 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.230+0000 7f918cec2700 1 -- 172.21.15.102:0/942332612 >> 172.21.15.102:0/942332612 conn(0x7f918809b370 msgr2=0x7f9188097430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:05:54.264 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.230+0000 7f918cec2700 1 -- 172.21.15.102:0/942332612 shutdown_connections 2021-05-25T12:05:54.265 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.230+0000 7f918cec2700 1 -- 172.21.15.102:0/942332612 wait complete. 2021-05-25T12:05:54.265 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.231+0000 7f918cec2700 10 monclient: build_initial_monmap 2021-05-25T12:05:54.265 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.231+0000 7f918cec2700 10 monclient: monmap: 2021-05-25T12:05:54.265 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:05:54.265 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:05:54.266 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:05:54.266 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:05:54.266 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:05:54.266 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:05:54.267 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:05:54.267 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:05:54.267 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:05:54.267 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:54.267 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.231+0000 7f918cec2700 1 librados: starting msgr at 2021-05-25T12:05:54.268 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.231+0000 7f918cec2700 1 librados: starting objecter 2021-05-25T12:05:54.268 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.231+0000 7f918cec2700 1 Processor -- start 2021-05-25T12:05:54.268 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.231+0000 7f918cec2700 1 -- start start 2021-05-25T12:05:54.268 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.231+0000 7f918cec2700 1 librados: setting wanted keys 2021-05-25T12:05:54.269 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.231+0000 7f918cec2700 1 librados: calling monclient init 2021-05-25T12:05:54.269 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.231+0000 7f918cec2700 10 monclient: init 2021-05-25T12:05:54.269 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.231+0000 7f918cec2700 10 monclient: _reopen_session rank -1 2021-05-25T12:05:54.269 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.231+0000 7f918cec2700 10 monclient: _add_conns ranks=[2,0,1] 2021-05-25T12:05:54.269 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.231+0000 7f918cec2700 10 monclient(hunting): picked mon.noname-c con 0x7f9188090990 addr v1:172.21.15.102:6790/0 2021-05-25T12:05:54.270 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.231+0000 7f918cec2700 10 monclient(hunting): picked mon.noname-a con 0x7f9188090d70 addr v1:172.21.15.102:6789/0 2021-05-25T12:05:54.270 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.231+0000 7f918cec2700 10 monclient(hunting): picked mon.noname-b con 0x7f918812e330 addr v1:172.21.15.181:6789/0 2021-05-25T12:05:54.270 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.231+0000 7f918cec2700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f918812a1a0 con 0x7f9188090d70 2021-05-25T12:05:54.270 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.231+0000 7f918cec2700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9188068630 con 0x7f918812e330 2021-05-25T12:05:54.270 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.231+0000 7f918cec2700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91880688d0 con 0x7f9188090990 2021-05-25T12:05:54.271 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.231+0000 7f918cec2700 10 monclient(hunting): _renew_subs 2021-05-25T12:05:54.271 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.231+0000 7f918cec2700 10 monclient(hunting): authenticate will time out at 1154.828922s 2021-05-25T12:05:54.271 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.231+0000 7f9185d9b700 1 -- >> v1:172.21.15.102:6789/0 conn(0x7f9188090d70 legacy=0x7f9188126490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:05:54.271 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.232+0000 7f918659c700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f9188090990 0x7f918813ab90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:56220/0 (socket says 172.21.15.102:56220) 2021-05-25T12:05:54.272 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.232+0000 7f918659c700 1 -- 172.21.15.102:0/224840123 learned_addr learned my addr 172.21.15.102:0/224840123 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:05:54.272 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.232+0000 7f91767fc700 10 client.?.objecter ms_handle_connect 0x7f9188090990 2021-05-25T12:05:54.272 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.232+0000 7f91767fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:05:54.272 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.232+0000 7f91767fc700 10 client.?.objecter ms_handle_connect 0x7f918812e330 2021-05-25T12:05:54.272 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.232+0000 7f91767fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:05:54.273 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.232+0000 7f91767fc700 1 -- 172.21.15.102:0/224840123 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 784147235 0 0) 0x7f91880688d0 con 0x7f9188090990 2021-05-25T12:05:54.273 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.232+0000 7f91767fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:54.273 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.232+0000 7f91767fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:54.273 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.232+0000 7f91767fc700 10 monclient(hunting): my global_id is 184397 2021-05-25T12:05:54.273 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.232+0000 7f91767fc700 1 -- 172.21.15.102:0/224840123 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9164004c10 con 0x7f9188090990 2021-05-25T12:05:54.274 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.232+0000 7f91767fc700 1 -- 172.21.15.102:0/224840123 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1036099320 0 0) 0x7f9188068630 con 0x7f918812e330 2021-05-25T12:05:54.274 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.232+0000 7f91767fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:54.274 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.232+0000 7f91767fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:54.274 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.232+0000 7f91767fc700 10 monclient(hunting): my global_id is 194206 2021-05-25T12:05:54.275 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.232+0000 7f91767fc700 1 -- 172.21.15.102:0/224840123 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91880688d0 con 0x7f918812e330 2021-05-25T12:05:54.275 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.232+0000 7f91767fc700 1 -- 172.21.15.102:0/224840123 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2431821134 0 0) 0x7f9164004c10 con 0x7f9188090990 2021-05-25T12:05:54.275 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.232+0000 7f91767fc700 1 -- 172.21.15.102:0/224840123 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9188068630 con 0x7f9188090990 2021-05-25T12:05:54.275 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.232+0000 7f91767fc700 1 -- 172.21.15.102:0/224840123 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2035450706 0 0) 0x7f91880688d0 con 0x7f918812e330 2021-05-25T12:05:54.275 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.232+0000 7f91767fc700 1 -- 172.21.15.102:0/224840123 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9164004c10 con 0x7f918812e330 2021-05-25T12:05:54.276 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.232+0000 7f91767fc700 1 -- 172.21.15.102:0/224840123 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f9178001c40 con 0x7f918812e330 2021-05-25T12:05:54.276 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.232+0000 7f91767fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:54.276 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.232+0000 7f91767fc700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:05:54.276 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.232+0000 7f91767fc700 10 monclient(hunting): dump: 2021-05-25T12:05:54.276 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:54.277 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:54.277 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:54.277 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:54.277 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:54.278 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:54.278 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:54.278 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:54.278 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:54.278 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:54.279 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.232+0000 7f91767fc700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:05:54.279 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.232+0000 7f91767fc700 1 -- 172.21.15.102:0/224840123 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f917c004fc0 con 0x7f9188090990 2021-05-25T12:05:54.279 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.232+0000 7f91767fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:54.279 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.232+0000 7f91767fc700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2021-05-25T12:05:54.280 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.232+0000 7f91767fc700 10 monclient(hunting): dump: 2021-05-25T12:05:54.280 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:54.280 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:54.280 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:54.280 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:54.281 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:54.281 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:54.281 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:54.281 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:54.281 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:54.282 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:54.282 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.232+0000 7f91767fc700 1 -- 172.21.15.102:0/224840123 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1248015003 0 0) 0x7f9188068630 con 0x7f9188090990 2021-05-25T12:05:54.282 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.232+0000 7f91767fc700 1 -- 172.21.15.102:0/224840123 >> v1:172.21.15.181:6789/0 conn(0x7f918812e330 legacy=0x7f9188142f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:54.282 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.232+0000 7f91767fc700 1 -- 172.21.15.102:0/224840123 >> v1:172.21.15.102:6789/0 conn(0x7f9188090d70 legacy=0x7f9188126490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-05-25T12:05:54.282 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.232+0000 7f91767fc700 10 monclient: _finish_hunting 0 2021-05-25T12:05:54.283 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.232+0000 7f91767fc700 1 monclient: found mon.c 2021-05-25T12:05:54.283 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.232+0000 7f91767fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:05:54.283 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.232+0000 7f91767fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:05:54.283 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.232+0000 7f91767fc700 1 -- 172.21.15.102:0/224840123 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91880667d0 con 0x7f9188090990 2021-05-25T12:05:54.284 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.232+0000 7f91767fc700 10 monclient: _finish_auth 0 2021-05-25T12:05:54.284 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.232+0000 7f91767fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:05:54.284 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.232+0000 7f918cec2700 5 monclient: authenticate success, global_id 184397 2021-05-25T12:05:54.284 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.232+0000 7f918cec2700 10 monclient: _renew_subs 2021-05-25T12:05:54.284 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.232+0000 7f918cec2700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:05:54.285 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.232+0000 7f918cec2700 1 -- 172.21.15.102:0/224840123 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f918813b2a0 con 0x7f9188090990 2021-05-25T12:05:54.285 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.232+0000 7f918cec2700 10 client.184397.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:05:54.285 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.232+0000 7f918cec2700 10 monclient: _renew_subs 2021-05-25T12:05:54.285 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.232+0000 7f918cec2700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:05:54.285 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.232+0000 7f918cec2700 1 -- 172.21.15.102:0/224840123 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9188126ba0 con 0x7f9188090990 2021-05-25T12:05:54.286 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.232+0000 7f918cec2700 1 librados: init done 2021-05-25T12:05:54.286 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.232+0000 7f91767fc700 1 -- 172.21.15.102:0/224840123 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f917c003d40 con 0x7f9188090990 2021-05-25T12:05:54.286 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.232+0000 7f91767fc700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:05:54.286 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.232+0000 7f91767fc700 1 -- 172.21.15.102:0/224840123 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f917c0040d0 con 0x7f9188090990 2021-05-25T12:05:54.287 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.232+0000 7f91767fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:54.287 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.232+0000 7f9184d99700 4 set_mon_vals no callback set 2021-05-25T12:05:54.287 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.232+0000 7f91767fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:05:54.287 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.232+0000 7f91767fc700 10 monclient: dump: 2021-05-25T12:05:54.287 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:54.288 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:54.288 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:54.288 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:54.288 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:54.288 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:54.289 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:54.289 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:54.289 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:54.289 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:54.289 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.233+0000 7f918cec2700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:05:54.290 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.233+0000 7f918cec2700 1 -- 172.21.15.102:0/224840123 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f91680065a0 con 0x7f9168002840 2021-05-25T12:05:54.290 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.233+0000 7f91767fc700 1 -- 172.21.15.102:0/224840123 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f917c0042b0 con 0x7f9188090990 2021-05-25T12:05:54.290 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.233+0000 7f91767fc700 10 client.184397.objecter ms_handle_connect 0x7f9168002840 2021-05-25T12:05:54.290 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.233+0000 7f91767fc700 10 client.184397.objecter resend_mon_ops 2021-05-25T12:05:54.290 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.233+0000 7f91767fc700 1 -- 172.21.15.102:0/224840123 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(172..172 src has 1..172) v4 ==== 5856+0+0 (unknown 973787454 0 0) 0x7f917c03d760 con 0x7f9188090990 2021-05-25T12:05:54.291 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.233+0000 7f91767fc700 10 client.184397.objecter ms_dispatch 0x7f9188001550 osd_map(172..172 src has 1..172) v4 2021-05-25T12:05:54.291 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.233+0000 7f91767fc700 3 client.184397.objecter handle_osd_map got epochs [172,172] > 0 2021-05-25T12:05:54.291 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.233+0000 7f91767fc700 3 client.184397.objecter handle_osd_map decoding full epoch 172 2021-05-25T12:05:54.291 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.233+0000 7f91767fc700 20 client.184397.objecter dump_active .. 0 homeless 2021-05-25T12:05:54.292 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.233+0000 7f91767fc700 1 -- 172.21.15.102:0/224840123 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2293957059 0 0) 0x7f91680065a0 con 0x7f9168002840 2021-05-25T12:05:54.292 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.233+0000 7f91767fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:54.292 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.233+0000 7f91767fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:54.292 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.233+0000 7f91767fc700 10 monclient(hunting): my global_id is 184400 2021-05-25T12:05:54.292 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.233+0000 7f91767fc700 1 -- 172.21.15.102:0/224840123 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9188068630 con 0x7f9168002840 2021-05-25T12:05:54.293 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.233+0000 7f9186d9d700 10 monclient: get_auth_request con 0x7f9164033250 auth_method 0 2021-05-25T12:05:54.293 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.233+0000 7f91767fc700 1 -- 172.21.15.102:0/224840123 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2387853257 0 0) 0x7f9188068630 con 0x7f9168002840 2021-05-25T12:05:54.293 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.234+0000 7f91767fc700 1 -- 172.21.15.102:0/224840123 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f91680065a0 con 0x7f9168002840 2021-05-25T12:05:54.293 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.234+0000 7f91767fc700 10 client.184397.objecter ms_handle_connect 0x7f9164033250 2021-05-25T12:05:54.294 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.234+0000 7f91767fc700 1 -- 172.21.15.102:0/224840123 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f9170001b90 con 0x7f9168002840 2021-05-25T12:05:54.294 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.234+0000 7f91767fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:54.294 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.234+0000 7f91767fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:05:54.294 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.234+0000 7f91767fc700 10 monclient: dump: 2021-05-25T12:05:54.294 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:54.295 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:54.295 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:54.295 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:54.295 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:54.295 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:54.296 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:54.296 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:54.296 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:54.296 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:54.296 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.234+0000 7f91767fc700 1 -- 172.21.15.102:0/224840123 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3963760206 0 0) 0x7f91680065a0 con 0x7f9168002840 2021-05-25T12:05:54.297 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.234+0000 7f91767fc700 1 -- 172.21.15.102:0/224840123 --> v1:172.21.15.102:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9168009e40 con 0x7f9168002840 2021-05-25T12:05:54.297 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.234+0000 7f91767fc700 1 -- 172.21.15.102:0/224840123 <== mon.2 v1:172.21.15.102:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7f91700044c0 con 0x7f9168002840 2021-05-25T12:05:54.297 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.234+0000 7f91767fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:05:54.297 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.234+0000 7f91767fc700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:05:54.297 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.234+0000 7f91767fc700 1 -- 172.21.15.102:0/224840123 >> v1:172.21.15.102:6790/0 conn(0x7f9168002840 legacy=0x7f9168004ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:54.298 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.234+0000 7f91767fc700 1 -- 172.21.15.102:0/224840123 >> v1:172.21.15.102:6790/0 conn(0x7f9168002840 legacy=0x7f9168004ce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:05:54.298 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.241+0000 7f918cec2700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:05:54.298 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.241+0000 7f918cec2700 1 -- 172.21.15.102:0/224840123 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f9168009b00 con 0x7f9168008dc0 2021-05-25T12:05:54.298 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.241+0000 7f91767fc700 10 client.184397.objecter ms_handle_connect 0x7f9168008dc0 2021-05-25T12:05:54.299 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.241+0000 7f91767fc700 10 client.184397.objecter resend_mon_ops 2021-05-25T12:05:54.299 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.241+0000 7f91767fc700 1 -- 172.21.15.102:0/224840123 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2822775717 0 0) 0x7f9168009b00 con 0x7f9168008dc0 2021-05-25T12:05:54.299 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.241+0000 7f91767fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:54.299 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.241+0000 7f91767fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:54.299 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.241+0000 7f91767fc700 10 monclient(hunting): my global_id is 184403 2021-05-25T12:05:54.300 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.241+0000 7f91767fc700 1 -- 172.21.15.102:0/224840123 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91680065a0 con 0x7f9168008dc0 2021-05-25T12:05:54.300 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.241+0000 7f91767fc700 1 -- 172.21.15.102:0/224840123 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1113460704 0 0) 0x7f91680065a0 con 0x7f9168008dc0 2021-05-25T12:05:54.300 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.241+0000 7f91767fc700 1 -- 172.21.15.102:0/224840123 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9168009b00 con 0x7f9168008dc0 2021-05-25T12:05:54.300 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.241+0000 7f91767fc700 1 -- 172.21.15.102:0/224840123 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f9170004b30 con 0x7f9168008dc0 2021-05-25T12:05:54.300 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.241+0000 7f91767fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:54.301 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.241+0000 7f91767fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:05:54.301 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.241+0000 7f91767fc700 10 monclient: dump: 2021-05-25T12:05:54.301 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:54.301 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:54.302 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:54.302 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:54.302 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:54.302 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:54.302 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:54.303 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:54.303 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:54.303 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:54.303 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.241+0000 7f91767fc700 1 -- 172.21.15.102:0/224840123 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4196727505 0 0) 0x7f9168009b00 con 0x7f9168008dc0 2021-05-25T12:05:54.303 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.241+0000 7f91767fc700 1 -- 172.21.15.102:0/224840123 --> v1:172.21.15.102:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f9168008740 con 0x7f9168008dc0 2021-05-25T12:05:54.304 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.241+0000 7f91767fc700 1 -- 172.21.15.102:0/224840123 <== mon.2 v1:172.21.15.102:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4101 (unknown 0 0 1707597124) 0x7f9170004d80 con 0x7f9168008dc0 2021-05-25T12:05:54.304 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.241+0000 7f91767fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:05:54.304 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.241+0000 7f91767fc700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:05:54.304 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.241+0000 7f91767fc700 1 -- 172.21.15.102:0/224840123 >> v1:172.21.15.102:6790/0 conn(0x7f9168008dc0 legacy=0x7f9168009250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:54.305 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.241+0000 7f91767fc700 1 -- 172.21.15.102:0/224840123 >> v1:172.21.15.102:6790/0 conn(0x7f9168008dc0 legacy=0x7f9168009250 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:05:54.305 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:05:54.305 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:05:54.305 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:05:54.306 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "leader", 2021-05-25T12:05:54.306 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 50, 2021-05-25T12:05:54.306 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:05:54.306 INFO:teuthology.orchestra.run.smithi102.stdout: 1, 2021-05-25T12:05:54.307 INFO:teuthology.orchestra.run.smithi102.stdout: 2 2021-05-25T12:05:54.307 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:05:54.307 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 0, 2021-05-25T12:05:54.307 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:05:54.307 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:05:54.308 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:05:54.308 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:05:54.308 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:05:54.308 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:05:54.308 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:05:54.309 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:05:54.309 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:05:54.309 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:05:54.309 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:05:54.310 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:05:54.310 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:05:54.310 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:05:54.310 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:05:54.310 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:05:54.311 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:05:54.311 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:05:54.311 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:05:54.311 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:05:54.312 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:05:54.312 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:05:54.312 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:05:54.312 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:05:54.312 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:05:54.313 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:05:54.313 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:05:54.313 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:05:54.313 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:05:54.313 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:05:54.314 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:05:54.314 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:05:54.314 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:05:54.314 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:05:54.314 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:05:54.315 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:05:54.315 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:05:54.315 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:05:54.315 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:05:54.316 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:05:54.316 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:05:54.316 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:05:54.316 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:05:54.316 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:05:54.317 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:05:54.317 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:05:54.317 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:05:54.317 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:05:54.317 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:05:54.318 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:05:54.318 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:05:54.318 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:05:54.318 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:05:54.319 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:05:54.319 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:05:54.319 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:05:54.319 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:05:54.319 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:05:54.320 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:05:54.320 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:05:54.320 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:05:54.320 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:05:54.320 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:05:54.321 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:05:54.321 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:05:54.321 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:05:54.321 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:05:54.322 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:05:54.322 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:05:54.322 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:05:54.322 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:05:54.322 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:05:54.323 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:05:54.323 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:05:54.323 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:05:54.323 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:05:54.323 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:05:54.324 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:05:54.324 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:05:54.324 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:05:54.324 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:05:54.324 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:05:54.325 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:05:54.325 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:05:54.325 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:05:54.325 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:05:54.326 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:05:54.326 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:05:54.326 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:05:54.326 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:05:54.326 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:05:54.327 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:05:54.327 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:05:54.327 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:05:54.327 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:05:54.327 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:05:54.328 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:05:54.328 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:05:54.328 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:05:54.328 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:05:54.329 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:05:54.329 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:05:54.329 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:05:54.329 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:05:54.329 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:05:54.330 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:05:54.330 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:05:54.330 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:05:54.330 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:05:54.330 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:05:54.331 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:05:54.331 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:05:54.331 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:05:54.331 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:05:54.331 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:05:54.332 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:05:54.332 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:05:54.332 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:05:54.332 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": [ 2021-05-25T12:05:54.333 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:05:54.333 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:05:54.333 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:05:54.333 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 2 2021-05-25T12:05:54.333 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:05:54.334 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:05:54.334 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:05:54.334 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:05:54.334 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:05:54.334 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:05:54.335 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 3 2021-05-25T12:05:54.335 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:05:54.335 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:05:54.335 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": [ 2021-05-25T12:05:54.335 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:05:54.336 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:05:54.336 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:05:54.336 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:05:54.336 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:05:54.336 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:05:54.337 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:05:54.337 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:05:54.337 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:05:54.339 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.242+0000 7f918cec2700 10 librados: watch_flush enter 2021-05-25T12:05:54.340 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.242+0000 7f918cec2700 10 librados: watch_flush exit 2021-05-25T12:05:54.340 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.242+0000 7f918cec2700 20 client.184397.objecter shutdown clearing up homeless session... 2021-05-25T12:05:54.340 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.242+0000 7f918cec2700 10 client.184397.objecter successfully canceled tick 2021-05-25T12:05:54.340 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.242+0000 7f918cec2700 1 -- 172.21.15.102:0/224840123 >> v1:172.21.15.181:6812/29582 conn(0x7f9164033250 legacy=0x7f9164035710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:54.341 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.242+0000 7f918cec2700 10 monclient: shutdown 2021-05-25T12:05:54.341 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.242+0000 7f918cec2700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:05:54.341 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.242+0000 7f918cec2700 1 -- 172.21.15.102:0/224840123 >> v1:172.21.15.102:6790/0 conn(0x7f9188090990 legacy=0x7f918813ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:54.341 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.242+0000 7f9186d9d700 1 -- 172.21.15.102:0/224840123 reap_dead start 2021-05-25T12:05:54.342 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.242+0000 7f9186d9d700 1 -- 172.21.15.102:0/224840123 reap_dead start 2021-05-25T12:05:54.342 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.242+0000 7f918cec2700 1 -- 172.21.15.102:0/224840123 shutdown_connections 2021-05-25T12:05:54.342 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.242+0000 7f918cec2700 1 -- 172.21.15.102:0/224840123 >> 172.21.15.102:0/224840123 conn(0x7f918809b370 msgr2=0x7f91881292f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:05:54.342 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.242+0000 7f918cec2700 1 -- 172.21.15.102:0/224840123 shutdown_connections 2021-05-25T12:05:54.342 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.242+0000 7f918cec2700 1 -- 172.21.15.102:0/224840123 wait complete. 2021-05-25T12:05:54.343 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.242+0000 7f918cec2700 1 librados: shutdown 2021-05-25T12:05:54.343 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2021-05-25T12:05:54.454 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.453+0000 7f561b420700 10 monclient: get_monmap_and_config 2021-05-25T12:05:54.455 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.453+0000 7f561b420700 10 monclient: build_initial_monmap 2021-05-25T12:05:54.455 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.453+0000 7f561b420700 10 monclient: monmap: 2021-05-25T12:05:54.455 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:05:54.455 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:54.456 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:05:54.453926+0000 2021-05-25T12:05:54.456 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:05:54.453926+0000 2021-05-25T12:05:54.456 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:05:54.456 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:54.457 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:05:54.457 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:05:54.457 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:05:54.457 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:54.457 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.453+0000 7f561b420700 1 Processor -- start 2021-05-25T12:05:54.458 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.453+0000 7f561b420700 1 -- start start 2021-05-25T12:05:54.458 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.453+0000 7f561b420700 10 monclient: init 2021-05-25T12:05:54.470 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.454+0000 7f561b420700 10 monclient: _reopen_session rank -1 2021-05-25T12:05:54.470 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.454+0000 7f561b420700 10 monclient: _add_conns ranks=[1,2,0] 2021-05-25T12:05:54.470 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.454+0000 7f561b420700 10 monclient(hunting): picked mon.noname-c con 0x7f5614090990 addr v1:172.21.15.181:6789/0 2021-05-25T12:05:54.470 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.454+0000 7f561b420700 10 monclient(hunting): picked mon.noname-b con 0x7f5614090d70 addr v1:172.21.15.102:6790/0 2021-05-25T12:05:54.471 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.454+0000 7f561b420700 10 monclient(hunting): picked mon.noname-a con 0x7f561412e330 addr v1:172.21.15.102:6789/0 2021-05-25T12:05:54.471 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.454+0000 7f561b420700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5614096fa0 con 0x7f561412e330 2021-05-25T12:05:54.471 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.454+0000 7f561b420700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56140926a0 con 0x7f5614090990 2021-05-25T12:05:54.471 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.454+0000 7f561b420700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5614092940 con 0x7f5614090d70 2021-05-25T12:05:54.472 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.454+0000 7f561b420700 10 monclient(hunting): _renew_subs 2021-05-25T12:05:54.472 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.454+0000 7f561b420700 10 monclient(hunting): authenticate will time out at 1155.051170s 2021-05-25T12:05:54.472 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.454+0000 7f56199bd700 1 -- >> v1:172.21.15.102:6789/0 conn(0x7f561412e330 legacy=0x7f5614130720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:05:54.472 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.454+0000 7f56189bb700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f5614090d70 0x7f5614091200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:56234/0 (socket says 172.21.15.102:56234) 2021-05-25T12:05:54.472 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.454+0000 7f56189bb700 1 -- 172.21.15.102:0/1406941372 learned_addr learned my addr 172.21.15.102:0/1406941372 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:05:54.473 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.454+0000 7f5603fff700 1 -- 172.21.15.102:0/1406941372 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3897421677 0 0) 0x7f5614092940 con 0x7f5614090d70 2021-05-25T12:05:54.473 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.454+0000 7f5603fff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:54.473 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.454+0000 7f5603fff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:54.473 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.454+0000 7f5603fff700 10 monclient(hunting): my global_id is 184406 2021-05-25T12:05:54.473 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.454+0000 7f5603fff700 1 -- 172.21.15.102:0/1406941372 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55fc003270 con 0x7f5614090d70 2021-05-25T12:05:54.474 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.454+0000 7f5603fff700 1 -- 172.21.15.102:0/1406941372 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1159853566 0 0) 0x7f56140926a0 con 0x7f5614090990 2021-05-25T12:05:54.474 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.454+0000 7f5603fff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:54.474 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.454+0000 7f5603fff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:54.474 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.454+0000 7f5603fff700 10 monclient(hunting): my global_id is 194209 2021-05-25T12:05:54.475 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.454+0000 7f5603fff700 1 -- 172.21.15.102:0/1406941372 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5614092940 con 0x7f5614090990 2021-05-25T12:05:54.475 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.454+0000 7f5603fff700 1 -- 172.21.15.102:0/1406941372 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2909931165 0 0) 0x7f55fc003270 con 0x7f5614090d70 2021-05-25T12:05:54.475 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.454+0000 7f5603fff700 1 -- 172.21.15.102:0/1406941372 >> v1:172.21.15.181:6789/0 conn(0x7f5614090990 legacy=0x7f5614093240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:54.475 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.454+0000 7f5603fff700 1 -- 172.21.15.102:0/1406941372 >> v1:172.21.15.102:6789/0 conn(0x7f561412e330 legacy=0x7f5614130720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-05-25T12:05:54.475 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.454+0000 7f5603fff700 10 monclient: _finish_hunting 0 2021-05-25T12:05:54.476 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.454+0000 7f5603fff700 1 monclient: found mon.noname-b 2021-05-25T12:05:54.476 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.454+0000 7f5603fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:05:54.476 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.454+0000 7f5603fff700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T12:05:54.476 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.454+0000 7f5603fff700 1 -- 172.21.15.102:0/1406941372 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5614092be0 con 0x7f5614090d70 2021-05-25T12:05:54.477 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.454+0000 7f5603fff700 10 monclient: _finish_auth 0 2021-05-25T12:05:54.477 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.454+0000 7f5603fff700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:05:54.477 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.454+0000 7f561b420700 5 monclient: authenticate success, global_id 184406 2021-05-25T12:05:54.477 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.454+0000 7f5603fff700 1 -- 172.21.15.102:0/1406941372 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f5604001610 con 0x7f5614090d70 2021-05-25T12:05:54.477 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.454+0000 7f561b420700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:05:54.478 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.454+0000 7f5603fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:54.478 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.454+0000 7f5603fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:05:54.478 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.454+0000 7f5603fff700 10 monclient: dump: 2021-05-25T12:05:54.478 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:54.479 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:54.479 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:54.479 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:54.479 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:54.479 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:54.480 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:54.480 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:54.480 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:54.480 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:54.480 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.454+0000 7f561b420700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:05:54.481 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.454+0000 7f5603fff700 1 -- 172.21.15.102:0/1406941372 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5604001800 con 0x7f5614090d70 2021-05-25T12:05:54.481 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.454+0000 7f5603fff700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:05:54.481 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.454+0000 7f561b420700 10 monclient: get_monmap_and_config success 2021-05-25T12:05:54.481 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.454+0000 7f5603fff700 1 -- 172.21.15.102:0/1406941372 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f5604001b90 con 0x7f5614090d70 2021-05-25T12:05:54.481 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.454+0000 7f561b420700 4 set_mon_vals no callback set 2021-05-25T12:05:54.482 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.454+0000 7f5603fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:54.482 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.454+0000 7f5603fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:05:54.482 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.454+0000 7f5603fff700 10 monclient: dump: 2021-05-25T12:05:54.482 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:54.482 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:54.483 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:54.483 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:54.483 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:54.483 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:54.484 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:54.484 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:54.484 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:54.484 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:54.484 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.455+0000 7f561b420700 10 monclient: shutdown 2021-05-25T12:05:54.485 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.455+0000 7f561b420700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:05:54.485 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.455+0000 7f561b420700 1 -- 172.21.15.102:0/1406941372 >> v1:172.21.15.102:6790/0 conn(0x7f5614090d70 legacy=0x7f5614091200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:54.485 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.455+0000 7f561b420700 1 -- 172.21.15.102:0/1406941372 shutdown_connections 2021-05-25T12:05:54.485 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.455+0000 7f561b420700 1 -- 172.21.15.102:0/1406941372 >> 172.21.15.102:0/1406941372 conn(0x7f561409b370 msgr2=0x7f5614097430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:05:54.485 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.455+0000 7f561b420700 1 -- 172.21.15.102:0/1406941372 shutdown_connections 2021-05-25T12:05:54.486 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.455+0000 7f561b420700 1 -- 172.21.15.102:0/1406941372 wait complete. 2021-05-25T12:05:54.486 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.455+0000 7f561b420700 10 monclient: build_initial_monmap 2021-05-25T12:05:54.486 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.455+0000 7f561b420700 10 monclient: monmap: 2021-05-25T12:05:54.486 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:05:54.487 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:05:54.487 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:05:54.487 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:05:54.487 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:05:54.487 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:05:54.488 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:05:54.488 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:05:54.488 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:05:54.488 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:54.488 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.456+0000 7f561b420700 1 librados: starting msgr at 2021-05-25T12:05:54.489 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.456+0000 7f561b420700 1 librados: starting objecter 2021-05-25T12:05:54.489 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.456+0000 7f561b420700 1 Processor -- start 2021-05-25T12:05:54.489 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.456+0000 7f561b420700 1 -- start start 2021-05-25T12:05:54.489 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.456+0000 7f561b420700 1 librados: setting wanted keys 2021-05-25T12:05:54.489 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.456+0000 7f561b420700 1 librados: calling monclient init 2021-05-25T12:05:54.490 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.456+0000 7f561b420700 10 monclient: init 2021-05-25T12:05:54.490 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.456+0000 7f561b420700 10 monclient: _reopen_session rank -1 2021-05-25T12:05:54.490 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.456+0000 7f561b420700 10 monclient: _add_conns ranks=[2,0,1] 2021-05-25T12:05:54.490 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.456+0000 7f561b420700 10 monclient(hunting): picked mon.noname-c con 0x7f5614090990 addr v1:172.21.15.102:6790/0 2021-05-25T12:05:54.491 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.456+0000 7f561b420700 10 monclient(hunting): picked mon.noname-a con 0x7f5614090d70 addr v1:172.21.15.102:6789/0 2021-05-25T12:05:54.491 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.456+0000 7f561b420700 10 monclient(hunting): picked mon.noname-b con 0x7f561412e330 addr v1:172.21.15.181:6789/0 2021-05-25T12:05:54.491 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.456+0000 7f561b420700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56141184e0 con 0x7f5614090d70 2021-05-25T12:05:54.491 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.456+0000 7f561b420700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5614118780 con 0x7f561412e330 2021-05-25T12:05:54.491 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.456+0000 7f561b420700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5614118a20 con 0x7f5614090990 2021-05-25T12:05:54.492 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.456+0000 7f561b420700 10 monclient(hunting): _renew_subs 2021-05-25T12:05:54.492 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.456+0000 7f561b420700 10 monclient(hunting): authenticate will time out at 1155.053841s 2021-05-25T12:05:54.492 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.456+0000 7f56189bb700 1 -- >> v1:172.21.15.102:6789/0 conn(0x7f5614090d70 legacy=0x7f5614140110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:05:54.492 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.456+0000 7f56191bc700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f5614090990 0x7f561413b4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:56238/0 (socket says 172.21.15.102:56238) 2021-05-25T12:05:54.492 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.456+0000 7f56191bc700 1 -- 172.21.15.102:0/833059499 learned_addr learned my addr 172.21.15.102:0/833059499 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:05:54.493 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.456+0000 7f56017fa700 10 client.?.objecter ms_handle_connect 0x7f5614090990 2021-05-25T12:05:54.493 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.456+0000 7f56017fa700 10 client.?.objecter resend_mon_ops 2021-05-25T12:05:54.493 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.457+0000 7f56017fa700 10 client.?.objecter ms_handle_connect 0x7f561412e330 2021-05-25T12:05:54.493 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.457+0000 7f56017fa700 10 client.?.objecter resend_mon_ops 2021-05-25T12:05:54.494 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.457+0000 7f56017fa700 1 -- 172.21.15.102:0/833059499 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2600300276 0 0) 0x7f5614118a20 con 0x7f5614090990 2021-05-25T12:05:54.494 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.457+0000 7f56017fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:54.494 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.457+0000 7f56017fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:54.494 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.457+0000 7f56017fa700 10 monclient(hunting): my global_id is 184409 2021-05-25T12:05:54.494 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.457+0000 7f56017fa700 1 -- 172.21.15.102:0/833059499 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55f4004c10 con 0x7f5614090990 2021-05-25T12:05:54.495 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.457+0000 7f56017fa700 1 -- 172.21.15.102:0/833059499 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2630062217 0 0) 0x7f5614118780 con 0x7f561412e330 2021-05-25T12:05:54.495 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.457+0000 7f56017fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:54.495 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.457+0000 7f56017fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:54.495 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.457+0000 7f56017fa700 10 monclient(hunting): my global_id is 194212 2021-05-25T12:05:54.496 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.457+0000 7f56017fa700 1 -- 172.21.15.102:0/833059499 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5614118a20 con 0x7f561412e330 2021-05-25T12:05:54.496 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.457+0000 7f56017fa700 1 -- 172.21.15.102:0/833059499 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3615276686 0 0) 0x7f55f4004c10 con 0x7f5614090990 2021-05-25T12:05:54.496 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.457+0000 7f56017fa700 1 -- 172.21.15.102:0/833059499 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5614118780 con 0x7f5614090990 2021-05-25T12:05:54.496 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.457+0000 7f56017fa700 1 -- 172.21.15.102:0/833059499 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f5608001900 con 0x7f5614090990 2021-05-25T12:05:54.496 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.457+0000 7f56017fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:54.497 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.457+0000 7f56017fa700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:05:54.497 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.457+0000 7f56017fa700 10 monclient(hunting): dump: 2021-05-25T12:05:54.497 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:54.497 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:54.497 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:54.498 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:54.498 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:54.498 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:54.498 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:54.499 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:54.499 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:54.499 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:54.499 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.457+0000 7f56017fa700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:05:54.499 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.457+0000 7f56017fa700 1 -- 172.21.15.102:0/833059499 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1052862221 0 0) 0x7f5614118a20 con 0x7f561412e330 2021-05-25T12:05:54.500 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.457+0000 7f56017fa700 1 -- 172.21.15.102:0/833059499 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f55f4004c10 con 0x7f561412e330 2021-05-25T12:05:54.500 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.457+0000 7f56017fa700 1 -- 172.21.15.102:0/833059499 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f5610004090 con 0x7f561412e330 2021-05-25T12:05:54.500 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.457+0000 7f56017fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:54.500 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.457+0000 7f56017fa700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2021-05-25T12:05:54.501 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.457+0000 7f56017fa700 10 monclient(hunting): dump: 2021-05-25T12:05:54.501 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:54.501 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:54.501 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:54.501 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:54.502 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:54.502 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:54.502 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:54.502 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:54.502 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:54.503 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:54.503 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.457+0000 7f56017fa700 1 -- 172.21.15.102:0/833059499 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2174515470 0 0) 0x7f5614118780 con 0x7f5614090990 2021-05-25T12:05:54.503 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.457+0000 7f56017fa700 1 -- 172.21.15.102:0/833059499 >> v1:172.21.15.181:6789/0 conn(0x7f561412e330 legacy=0x7f5614143840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:54.503 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.457+0000 7f56017fa700 1 -- 172.21.15.102:0/833059499 >> v1:172.21.15.102:6789/0 conn(0x7f5614090d70 legacy=0x7f5614140110 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-05-25T12:05:54.504 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.457+0000 7f56017fa700 10 monclient: _finish_hunting 0 2021-05-25T12:05:54.504 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.457+0000 7f56017fa700 1 monclient: found mon.c 2021-05-25T12:05:54.504 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.457+0000 7f56017fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:05:54.504 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.457+0000 7f56017fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:05:54.504 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.457+0000 7f56017fa700 1 -- 172.21.15.102:0/833059499 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5614112f50 con 0x7f5614090990 2021-05-25T12:05:54.505 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.457+0000 7f56017fa700 10 monclient: _finish_auth 0 2021-05-25T12:05:54.505 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.457+0000 7f56017fa700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:05:54.505 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.457+0000 7f561b420700 5 monclient: authenticate success, global_id 184409 2021-05-25T12:05:54.505 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.457+0000 7f561b420700 10 monclient: _renew_subs 2021-05-25T12:05:54.505 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.457+0000 7f561b420700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:05:54.506 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.457+0000 7f561b420700 1 -- 172.21.15.102:0/833059499 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5614112470 con 0x7f5614090990 2021-05-25T12:05:54.506 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.457+0000 7f56017fa700 1 -- 172.21.15.102:0/833059499 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5608001ef0 con 0x7f5614090990 2021-05-25T12:05:54.506 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.457+0000 7f56017fa700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:05:54.506 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.457+0000 7f56017fa700 1 -- 172.21.15.102:0/833059499 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f5608004960 con 0x7f5614090990 2021-05-25T12:05:54.507 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.457+0000 7f56017fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:54.507 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.457+0000 7f56017fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:05:54.507 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.457+0000 7f56017fa700 10 monclient: dump: 2021-05-25T12:05:54.507 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:54.507 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:54.508 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:54.508 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:54.508 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:54.508 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:54.508 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:54.509 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:54.509 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:54.509 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:54.509 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.457+0000 7f56037fe700 4 set_mon_vals no callback set 2021-05-25T12:05:54.510 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.457+0000 7f561b420700 10 client.184409.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:05:54.510 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.457+0000 7f561b420700 10 monclient: _renew_subs 2021-05-25T12:05:54.510 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.457+0000 7f561b420700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:05:54.510 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.457+0000 7f561b420700 1 -- 172.21.15.102:0/833059499 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f561413bbd0 con 0x7f5614090990 2021-05-25T12:05:54.510 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.457+0000 7f561b420700 1 librados: init done 2021-05-25T12:05:54.511 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.458+0000 7f56017fa700 1 -- 172.21.15.102:0/833059499 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f560800dc80 con 0x7f5614090990 2021-05-25T12:05:54.511 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.458+0000 7f56017fa700 1 -- 172.21.15.102:0/833059499 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(172..172 src has 1..172) v4 ==== 5856+0+0 (unknown 973787454 0 0) 0x7f560803d610 con 0x7f5614090990 2021-05-25T12:05:54.511 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.458+0000 7f56017fa700 10 client.184409.objecter ms_dispatch 0x7f5614001550 osd_map(172..172 src has 1..172) v4 2021-05-25T12:05:54.511 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.458+0000 7f56017fa700 3 client.184409.objecter handle_osd_map got epochs [172,172] > 0 2021-05-25T12:05:54.512 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.458+0000 7f56017fa700 3 client.184409.objecter handle_osd_map decoding full epoch 172 2021-05-25T12:05:54.512 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.458+0000 7f561b420700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:05:54.512 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.458+0000 7f561b420700 1 -- 172.21.15.102:0/833059499 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f56141182f0 con 0x7f5614068910 2021-05-25T12:05:54.512 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.458+0000 7f56017fa700 20 client.184409.objecter dump_active .. 0 homeless 2021-05-25T12:05:54.512 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.458+0000 7f56189bb700 10 monclient: get_auth_request con 0x7f55f40332a0 auth_method 0 2021-05-25T12:05:54.513 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.458+0000 7f56017fa700 10 client.184409.objecter ms_handle_connect 0x7f5614068910 2021-05-25T12:05:54.513 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.458+0000 7f56017fa700 10 client.184409.objecter resend_mon_ops 2021-05-25T12:05:54.513 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.458+0000 7f56017fa700 10 client.184409.objecter ms_handle_connect 0x7f55f40332a0 2021-05-25T12:05:54.513 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.458+0000 7f56017fa700 1 -- 172.21.15.102:0/833059499 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2413277498 0 0) 0x7f56141182f0 con 0x7f5614068910 2021-05-25T12:05:54.513 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.458+0000 7f56017fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:54.514 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.458+0000 7f56017fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:54.514 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.458+0000 7f56017fa700 10 monclient(hunting): my global_id is 194215 2021-05-25T12:05:54.514 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.458+0000 7f56017fa700 1 -- 172.21.15.102:0/833059499 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5614118780 con 0x7f5614068910 2021-05-25T12:05:54.514 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.458+0000 7f56017fa700 1 -- 172.21.15.102:0/833059499 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2585447049 0 0) 0x7f5614118780 con 0x7f5614068910 2021-05-25T12:05:54.515 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.458+0000 7f56017fa700 1 -- 172.21.15.102:0/833059499 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f56141182f0 con 0x7f5614068910 2021-05-25T12:05:54.515 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.458+0000 7f56017fa700 1 -- 172.21.15.102:0/833059499 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f5610004790 con 0x7f5614068910 2021-05-25T12:05:54.515 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.458+0000 7f56017fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:54.515 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.458+0000 7f56017fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T12:05:54.515 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.458+0000 7f56017fa700 10 monclient: dump: 2021-05-25T12:05:54.516 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:54.516 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:54.516 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:54.516 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:54.517 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:54.517 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:54.517 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:54.517 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:54.517 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:54.518 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:54.518 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.459+0000 7f56017fa700 1 -- 172.21.15.102:0/833059499 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2448518487 0 0) 0x7f56141182f0 con 0x7f5614068910 2021-05-25T12:05:54.518 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.459+0000 7f56017fa700 1 -- 172.21.15.102:0/833059499 --> v1:172.21.15.181:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5614059730 con 0x7f5614068910 2021-05-25T12:05:54.518 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.459+0000 7f56017fa700 1 -- 172.21.15.102:0/833059499 <== mon.1 v1:172.21.15.181:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7f5610006ba0 con 0x7f5614068910 2021-05-25T12:05:54.518 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.459+0000 7f56017fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:05:54.519 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.459+0000 7f56017fa700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:05:54.519 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.459+0000 7f56017fa700 1 -- 172.21.15.102:0/833059499 >> v1:172.21.15.181:6789/0 conn(0x7f5614068910 legacy=0x7f5614064d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:54.519 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.459+0000 7f56017fa700 1 -- 172.21.15.102:0/833059499 >> v1:172.21.15.181:6789/0 conn(0x7f5614068910 legacy=0x7f5614064d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:05:54.519 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.465+0000 7f561b420700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:05:54.520 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.465+0000 7f561b420700 1 -- 172.21.15.102:0/833059499 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f561405d870 con 0x7f5614027ce0 2021-05-25T12:05:54.520 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.466+0000 7f56017fa700 10 client.184409.objecter ms_handle_connect 0x7f5614027ce0 2021-05-25T12:05:54.520 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.466+0000 7f56017fa700 10 client.184409.objecter resend_mon_ops 2021-05-25T12:05:54.520 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.466+0000 7f56017fa700 1 -- 172.21.15.102:0/833059499 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2652947449 0 0) 0x7f561405d870 con 0x7f5614027ce0 2021-05-25T12:05:54.520 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.466+0000 7f56017fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:05:54.521 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.466+0000 7f56017fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:05:54.521 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.466+0000 7f56017fa700 10 monclient(hunting): my global_id is 194218 2021-05-25T12:05:54.521 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.466+0000 7f56017fa700 1 -- 172.21.15.102:0/833059499 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56141182f0 con 0x7f5614027ce0 2021-05-25T12:05:54.521 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.466+0000 7f56017fa700 1 -- 172.21.15.102:0/833059499 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4206557424 0 0) 0x7f56141182f0 con 0x7f5614027ce0 2021-05-25T12:05:54.521 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.466+0000 7f56017fa700 1 -- 172.21.15.102:0/833059499 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f561405d870 con 0x7f5614027ce0 2021-05-25T12:05:54.522 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.466+0000 7f56017fa700 1 -- 172.21.15.102:0/833059499 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f5610005000 con 0x7f5614027ce0 2021-05-25T12:05:54.522 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.466+0000 7f56017fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:05:54.522 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.466+0000 7f56017fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T12:05:54.522 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.466+0000 7f56017fa700 10 monclient: dump: 2021-05-25T12:05:54.522 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:05:54.523 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:05:54.523 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:54.523 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:05:54.523 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:05:54.524 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:05:54.524 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:05:54.524 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:05:54.524 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:05:54.524 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:05:54.525 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.466+0000 7f56017fa700 1 -- 172.21.15.102:0/833059499 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2433755424 0 0) 0x7f561405d870 con 0x7f5614027ce0 2021-05-25T12:05:54.525 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.466+0000 7f56017fa700 1 -- 172.21.15.102:0/833059499 --> v1:172.21.15.181:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f5614065b30 con 0x7f5614027ce0 2021-05-25T12:05:54.525 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.466+0000 7f56017fa700 1 -- 172.21.15.102:0/833059499 <== mon.1 v1:172.21.15.181:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3929 (unknown 0 0 2493403068) 0x7f56100062e0 con 0x7f5614027ce0 2021-05-25T12:05:54.525 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.466+0000 7f56017fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:05:54.525 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.466+0000 7f56017fa700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:05:54.526 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.466+0000 7f56017fa700 1 -- 172.21.15.102:0/833059499 >> v1:172.21.15.181:6789/0 conn(0x7f5614027ce0 legacy=0x7f5614065420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:54.526 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.466+0000 7f56017fa700 1 -- 172.21.15.102:0/833059499 >> v1:172.21.15.181:6789/0 conn(0x7f5614027ce0 legacy=0x7f5614065420 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:05:54.526 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:05:54.527 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:05:54.527 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:05:54.527 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "peon", 2021-05-25T12:05:54.527 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 50, 2021-05-25T12:05:54.528 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:05:54.528 INFO:teuthology.orchestra.run.smithi102.stdout: 1, 2021-05-25T12:05:54.528 INFO:teuthology.orchestra.run.smithi102.stdout: 2 2021-05-25T12:05:54.528 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:05:54.529 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 0, 2021-05-25T12:05:54.529 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:05:54.529 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:05:54.529 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:05:54.529 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:05:54.530 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:05:54.530 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:05:54.530 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:05:54.530 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:05:54.530 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:05:54.531 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:05:54.531 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:05:54.531 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:05:54.531 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:05:54.532 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:05:54.532 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:05:54.532 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:05:54.532 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:05:54.532 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:05:54.533 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:05:54.533 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:05:54.533 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:05:54.533 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:05:54.533 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:05:54.534 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:05:54.534 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:05:54.534 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:05:54.534 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:05:54.535 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:05:54.535 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:05:54.535 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:05:54.535 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:05:54.535 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:05:54.536 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:05:54.536 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:05:54.536 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:05:54.536 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:05:54.536 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:05:54.537 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:05:54.537 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:05:54.537 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:05:54.537 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:05:54.538 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:05:54.538 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:05:54.538 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:05:54.538 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:05:54.538 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:05:54.539 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:05:54.539 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:05:54.539 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:05:54.539 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:05:54.539 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:05:54.540 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:05:54.540 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:05:54.540 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:05:54.540 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:05:54.541 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:05:54.541 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:05:54.541 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:05:54.541 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:05:54.541 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:05:54.542 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:05:54.542 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:05:54.542 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:05:54.542 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:05:54.542 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:05:54.543 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:05:54.543 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:05:54.543 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:05:54.543 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:05:54.543 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:05:54.544 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:05:54.544 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:05:54.544 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:05:54.544 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:05:54.544 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:05:54.545 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:05:54.545 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:05:54.545 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:05:54.545 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:05:54.546 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:05:54.546 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:05:54.546 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:05:54.546 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:05:54.546 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:05:54.547 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:05:54.547 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:05:54.547 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:05:54.547 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:05:54.547 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:05:54.548 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:05:54.548 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:05:54.548 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:05:54.548 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:05:54.548 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:05:54.549 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:05:54.549 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:05:54.549 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:05:54.549 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:05:54.549 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:05:54.550 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:05:54.550 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:05:54.550 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:05:54.550 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:05:54.551 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:05:54.551 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:05:54.551 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:05:54.551 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:05:54.551 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:05:54.552 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:05:54.552 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:05:54.552 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:05:54.552 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:05:54.552 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:05:54.553 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:05:54.553 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:05:54.553 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:05:54.553 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:05:54.553 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:05:54.554 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": [ 2021-05-25T12:05:54.554 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:05:54.554 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:05:54.554 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:05:54.555 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 4 2021-05-25T12:05:54.555 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:05:54.555 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:05:54.555 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:05:54.555 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:05:54.556 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:05:54.556 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:05:54.556 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 6 2021-05-25T12:05:54.556 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:05:54.556 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:05:54.557 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:05:54.557 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:05:54.557 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:05:54.559 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.467+0000 7f561b420700 10 librados: watch_flush enter 2021-05-25T12:05:54.559 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.467+0000 7f561b420700 10 librados: watch_flush exit 2021-05-25T12:05:54.560 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.467+0000 7f561b420700 20 client.184409.objecter shutdown clearing up homeless session... 2021-05-25T12:05:54.560 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.467+0000 7f561b420700 10 client.184409.objecter successfully canceled tick 2021-05-25T12:05:54.560 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.467+0000 7f561b420700 1 -- 172.21.15.102:0/833059499 >> v1:172.21.15.181:6812/29582 conn(0x7f55f40332a0 legacy=0x7f55f4035760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:54.560 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.467+0000 7f561b420700 10 monclient: shutdown 2021-05-25T12:05:54.561 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.467+0000 7f561b420700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:05:54.561 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.467+0000 7f561b420700 1 -- 172.21.15.102:0/833059499 >> v1:172.21.15.102:6790/0 conn(0x7f5614090990 legacy=0x7f561413b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:05:54.561 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.467+0000 7f56199bd700 1 -- 172.21.15.102:0/833059499 reap_dead start 2021-05-25T12:05:54.561 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.467+0000 7f561b420700 1 -- 172.21.15.102:0/833059499 shutdown_connections 2021-05-25T12:05:54.561 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.467+0000 7f561b420700 1 -- 172.21.15.102:0/833059499 >> 172.21.15.102:0/833059499 conn(0x7f561409b370 msgr2=0x7f56140939b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:05:54.562 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.467+0000 7f561b420700 1 -- 172.21.15.102:0/833059499 shutdown_connections 2021-05-25T12:05:54.562 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.467+0000 7f561b420700 1 -- 172.21.15.102:0/833059499 wait complete. 2021-05-25T12:05:54.562 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:05:54.467+0000 7f561b420700 1 librados: shutdown 2021-05-25T12:05:54.562 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2021-05-25T12:06:14.564 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2021-05-25T12:06:14.564 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2021-05-25T12:06:14.565 INFO:tasks.ceph.mon.a:Restarting daemon 2021-05-25T12:06:14.565 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2021-05-25T12:06:14.568 INFO:tasks.ceph.mon.a:Started 2021-05-25T12:06:14.569 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2021-05-25T12:06:14.569 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2021-05-25T12:06:14.709 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.707+0000 7f9deff0d700 10 monclient: get_monmap_and_config 2021-05-25T12:06:14.709 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.707+0000 7f9deff0d700 10 monclient: build_initial_monmap 2021-05-25T12:06:14.709 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.707+0000 7f9deff0d700 10 monclient: monmap: 2021-05-25T12:06:14.709 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:06:14.710 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:14.710 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:06:14.708018+0000 2021-05-25T12:06:14.710 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:06:14.708018+0000 2021-05-25T12:06:14.710 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:06:14.711 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:14.711 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:06:14.711 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:06:14.711 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:06:14.712 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:14.712 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.707+0000 7f9deff0d700 1 Processor -- start 2021-05-25T12:06:14.712 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.708+0000 7f9deff0d700 1 -- start start 2021-05-25T12:06:14.712 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.708+0000 7f9deff0d700 10 monclient: init 2021-05-25T12:06:14.714 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.708+0000 7f9deff0d700 10 monclient: _reopen_session rank -1 2021-05-25T12:06:14.714 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.708+0000 7f9deff0d700 10 monclient: _add_conns ranks=[1,2,0] 2021-05-25T12:06:14.714 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.708+0000 7f9deff0d700 10 monclient(hunting): picked mon.noname-c con 0x7f9de8090990 addr v1:172.21.15.181:6789/0 2021-05-25T12:06:14.714 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.708+0000 7f9deff0d700 10 monclient(hunting): picked mon.noname-b con 0x7f9de8090d70 addr v1:172.21.15.102:6790/0 2021-05-25T12:06:14.715 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.708+0000 7f9deff0d700 10 monclient(hunting): picked mon.noname-a con 0x7f9de812e330 addr v1:172.21.15.102:6789/0 2021-05-25T12:06:14.715 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.708+0000 7f9deff0d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9de8096fa0 con 0x7f9de812e330 2021-05-25T12:06:14.715 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.708+0000 7f9deff0d700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9de80926a0 con 0x7f9de8090990 2021-05-25T12:06:14.715 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.708+0000 7f9deff0d700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9de8092940 con 0x7f9de8090d70 2021-05-25T12:06:14.715 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.708+0000 7f9deff0d700 10 monclient(hunting): _renew_subs 2021-05-25T12:06:14.716 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.708+0000 7f9deff0d700 10 monclient(hunting): authenticate will time out at 1175.305669s 2021-05-25T12:06:14.716 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.708+0000 7f9dee4aa700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f9de812e330 0x7f9de8130720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33168/0 (socket says 172.21.15.102:33168) 2021-05-25T12:06:14.716 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.708+0000 7f9ded4a8700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f9de8090d70 0x7f9de8091200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:56276/0 (socket says 172.21.15.102:56276) 2021-05-25T12:06:14.716 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.708+0000 7f9dee4aa700 1 -- 172.21.15.102:0/3138735134 learned_addr learned my addr 172.21.15.102:0/3138735134 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:06:14.728 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.724+0000 7f9decca7700 1 -- 172.21.15.102:0/3138735134 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3541884758 0 0) 0x7f9de80926a0 con 0x7f9de8090990 2021-05-25T12:06:14.728 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.724+0000 7f9decca7700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:14.728 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.724+0000 7f9decca7700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:14.728 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.724+0000 7f9decca7700 10 monclient(hunting): my global_id is 194263 2021-05-25T12:06:14.729 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.724+0000 7f9decca7700 1 -- 172.21.15.102:0/3138735134 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9dd4004380 con 0x7f9de8090990 2021-05-25T12:06:14.729 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.724+0000 7f9decca7700 1 -- 172.21.15.102:0/3138735134 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1557247579 0 0) 0x7f9dd4004380 con 0x7f9de8090990 2021-05-25T12:06:14.729 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.724+0000 7f9decca7700 1 -- 172.21.15.102:0/3138735134 >> v1:172.21.15.102:6790/0 conn(0x7f9de8090d70 legacy=0x7f9de8091200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:14.729 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.724+0000 7f9decca7700 1 -- 172.21.15.102:0/3138735134 >> v1:172.21.15.102:6789/0 conn(0x7f9de812e330 legacy=0x7f9de8130720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:14.729 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.724+0000 7f9decca7700 10 monclient: _finish_hunting 0 2021-05-25T12:06:14.730 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.724+0000 7f9decca7700 1 monclient: found mon.noname-c 2021-05-25T12:06:14.730 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.724+0000 7f9decca7700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:06:14.730 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.724+0000 7f9decca7700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.181:6789/0 2021-05-25T12:06:14.730 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.724+0000 7f9decca7700 1 -- 172.21.15.102:0/3138735134 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9de8092be0 con 0x7f9de8090990 2021-05-25T12:06:14.731 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.724+0000 7f9decca7700 10 monclient: _finish_auth 0 2021-05-25T12:06:14.731 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.724+0000 7f9decca7700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:06:14.731 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.724+0000 7f9deff0d700 5 monclient: authenticate success, global_id 194263 2021-05-25T12:06:14.731 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.724+0000 7f9deff0d700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:06:14.731 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.724+0000 7f9decca7700 1 -- 172.21.15.102:0/3138735134 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f9dd0001610 con 0x7f9de8090990 2021-05-25T12:06:14.732 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.724+0000 7f9decca7700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:14.732 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.724+0000 7f9decca7700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:06:14.732 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.724+0000 7f9decca7700 10 monclient: dump: 2021-05-25T12:06:14.732 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:14.732 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:14.733 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:14.733 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:14.733 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:14.733 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:14.733 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:14.734 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:14.734 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:14.734 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:14.734 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.724+0000 7f9deff0d700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:06:14.735 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.724+0000 7f9decca7700 1 -- 172.21.15.102:0/3138735134 <== mon.1 v1:172.21.15.181:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9dd00018c0 con 0x7f9de8090990 2021-05-25T12:06:14.735 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.724+0000 7f9decca7700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:06:14.735 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.724+0000 7f9decca7700 1 -- 172.21.15.102:0/3138735134 <== mon.1 v1:172.21.15.181:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f9dd0001c10 con 0x7f9de8090990 2021-05-25T12:06:14.735 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.724+0000 7f9deff0d700 10 monclient: get_monmap_and_config success 2021-05-25T12:06:14.735 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.724+0000 7f9deff0d700 4 set_mon_vals no callback set 2021-05-25T12:06:14.736 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.724+0000 7f9decca7700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:14.736 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.724+0000 7f9decca7700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T12:06:14.736 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.724+0000 7f9decca7700 10 monclient: dump: 2021-05-25T12:06:14.736 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:14.737 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:14.737 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:14.737 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:14.737 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:14.737 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:14.738 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:14.738 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:14.738 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:14.738 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:14.738 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.725+0000 7f9deff0d700 10 monclient: shutdown 2021-05-25T12:06:14.739 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.725+0000 7f9deff0d700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:06:14.739 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.725+0000 7f9deff0d700 1 -- 172.21.15.102:0/3138735134 >> v1:172.21.15.181:6789/0 conn(0x7f9de8090990 legacy=0x7f9de8093240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:14.739 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.725+0000 7f9deff0d700 1 -- 172.21.15.102:0/3138735134 shutdown_connections 2021-05-25T12:06:14.739 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.725+0000 7f9deff0d700 1 -- 172.21.15.102:0/3138735134 >> 172.21.15.102:0/3138735134 conn(0x7f9de809b370 msgr2=0x7f9de8097430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:06:14.740 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.725+0000 7f9deff0d700 1 -- 172.21.15.102:0/3138735134 shutdown_connections 2021-05-25T12:06:14.740 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.725+0000 7f9deff0d700 1 -- 172.21.15.102:0/3138735134 wait complete. 2021-05-25T12:06:14.740 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.726+0000 7f9deff0d700 10 monclient: build_initial_monmap 2021-05-25T12:06:14.740 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.726+0000 7f9deff0d700 10 monclient: monmap: 2021-05-25T12:06:14.740 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:06:14.741 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:06:14.741 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:06:14.741 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:06:14.741 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:06:14.741 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:06:14.742 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:06:14.742 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:06:14.742 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:06:14.742 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:14.742 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.726+0000 7f9deff0d700 1 librados: starting msgr at 2021-05-25T12:06:14.743 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.726+0000 7f9deff0d700 1 librados: starting objecter 2021-05-25T12:06:14.743 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.726+0000 7f9deff0d700 1 Processor -- start 2021-05-25T12:06:14.743 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.726+0000 7f9deff0d700 1 -- start start 2021-05-25T12:06:14.743 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.726+0000 7f9deff0d700 1 librados: setting wanted keys 2021-05-25T12:06:14.744 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.726+0000 7f9deff0d700 1 librados: calling monclient init 2021-05-25T12:06:14.744 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.726+0000 7f9deff0d700 10 monclient: init 2021-05-25T12:06:14.745 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.727+0000 7f9deff0d700 10 monclient: _reopen_session rank -1 2021-05-25T12:06:14.746 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.727+0000 7f9deff0d700 10 monclient: _add_conns ranks=[0,1,2] 2021-05-25T12:06:14.746 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.727+0000 7f9deff0d700 10 monclient(hunting): picked mon.noname-a con 0x7f9de8090990 addr v1:172.21.15.102:6789/0 2021-05-25T12:06:14.746 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.727+0000 7f9deff0d700 10 monclient(hunting): picked mon.noname-b con 0x7f9de8090d70 addr v1:172.21.15.181:6789/0 2021-05-25T12:06:14.746 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.727+0000 7f9deff0d700 10 monclient(hunting): picked mon.noname-c con 0x7f9de812e330 addr v1:172.21.15.102:6790/0 2021-05-25T12:06:14.747 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.727+0000 7f9deff0d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9de8113c90 con 0x7f9de8090990 2021-05-25T12:06:14.747 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.727+0000 7f9deff0d700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9de8139280 con 0x7f9de8090d70 2021-05-25T12:06:14.747 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.727+0000 7f9deff0d700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9de8113ec0 con 0x7f9de812e330 2021-05-25T12:06:14.747 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.727+0000 7f9deff0d700 10 monclient(hunting): _renew_subs 2021-05-25T12:06:14.747 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.727+0000 7f9deff0d700 10 monclient(hunting): authenticate will time out at 1175.324609s 2021-05-25T12:06:14.748 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.727+0000 7f9dedca9700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f9de8090990 0x7f9de8138b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33170/0 (socket says 172.21.15.102:33170) 2021-05-25T12:06:14.748 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.727+0000 7f9dedca9700 1 -- 172.21.15.102:0/1883044569 learned_addr learned my addr 172.21.15.102:0/1883044569 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:06:14.748 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.727+0000 7f9dddffb700 10 client.?.objecter ms_handle_connect 0x7f9de8090990 2021-05-25T12:06:14.748 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.727+0000 7f9dddffb700 10 client.?.objecter resend_mon_ops 2021-05-25T12:06:14.749 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.728+0000 7f9dddffb700 10 client.?.objecter ms_handle_connect 0x7f9de8090d70 2021-05-25T12:06:14.749 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.728+0000 7f9dddffb700 10 client.?.objecter resend_mon_ops 2021-05-25T12:06:14.749 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.728+0000 7f9dddffb700 10 client.?.objecter ms_handle_connect 0x7f9de812e330 2021-05-25T12:06:14.749 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.728+0000 7f9dddffb700 10 client.?.objecter resend_mon_ops 2021-05-25T12:06:14.750 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.747+0000 7f9dddffb700 1 -- 172.21.15.102:0/1883044569 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3308548691 0 0) 0x7f9de8113ec0 con 0x7f9de812e330 2021-05-25T12:06:14.750 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.747+0000 7f9dddffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:14.750 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.747+0000 7f9dddffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:14.750 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.747+0000 7f9dddffb700 10 monclient(hunting): my global_id is 184442 2021-05-25T12:06:14.751 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.747+0000 7f9dddffb700 1 -- 172.21.15.102:0/1883044569 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9dc8004c10 con 0x7f9de812e330 2021-05-25T12:06:14.754 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.749+0000 7f9dddffb700 1 -- 172.21.15.102:0/1883044569 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3993538289 0 0) 0x7f9dc8004c10 con 0x7f9de812e330 2021-05-25T12:06:14.755 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.749+0000 7f9dddffb700 1 -- 172.21.15.102:0/1883044569 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9de8113ec0 con 0x7f9de812e330 2021-05-25T12:06:14.755 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.749+0000 7f9dddffb700 1 -- 172.21.15.102:0/1883044569 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f9de4001af0 con 0x7f9de812e330 2021-05-25T12:06:14.755 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.749+0000 7f9dddffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:14.755 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.749+0000 7f9dddffb700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:06:14.756 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.749+0000 7f9dddffb700 10 monclient(hunting): dump: 2021-05-25T12:06:14.756 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:14.756 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:14.756 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:14.756 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:14.757 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:14.757 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:14.757 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:14.757 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:14.757 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:14.758 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:14.758 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.749+0000 7f9dddffb700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:06:14.758 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.749+0000 7f9dddffb700 1 -- 172.21.15.102:0/1883044569 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3143797649 0 0) 0x7f9de8139280 con 0x7f9de8090d70 2021-05-25T12:06:14.758 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.749+0000 7f9dddffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:14.758 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.749+0000 7f9dddffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:14.759 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.749+0000 7f9dddffb700 10 monclient(hunting): my global_id is 194266 2021-05-25T12:06:14.759 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.749+0000 7f9dddffb700 1 -- 172.21.15.102:0/1883044569 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9dc8004c10 con 0x7f9de8090d70 2021-05-25T12:06:14.759 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.749+0000 7f9dddffb700 1 -- 172.21.15.102:0/1883044569 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1110995673 0 0) 0x7f9de8113ec0 con 0x7f9de812e330 2021-05-25T12:06:14.759 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.749+0000 7f9dddffb700 1 -- 172.21.15.102:0/1883044569 >> v1:172.21.15.181:6789/0 conn(0x7f9de8090d70 legacy=0x7f9de8127490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:14.760 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.749+0000 7f9dddffb700 1 -- 172.21.15.102:0/1883044569 >> v1:172.21.15.102:6789/0 conn(0x7f9de8090990 legacy=0x7f9de8138b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:14.760 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.749+0000 7f9dddffb700 10 monclient: _finish_hunting 0 2021-05-25T12:06:14.760 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.749+0000 7f9dddffb700 1 monclient: found mon.c 2021-05-25T12:06:14.760 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.749+0000 7f9dddffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:06:14.760 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.749+0000 7f9dddffb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:06:14.761 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.749+0000 7f9dddffb700 1 -- 172.21.15.102:0/1883044569 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9de8091fe0 con 0x7f9de812e330 2021-05-25T12:06:14.761 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.749+0000 7f9dddffb700 10 monclient: _finish_auth 0 2021-05-25T12:06:14.761 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.749+0000 7f9dddffb700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:06:14.761 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.749+0000 7f9deff0d700 5 monclient: authenticate success, global_id 184442 2021-05-25T12:06:14.761 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.749+0000 7f9deff0d700 10 monclient: _renew_subs 2021-05-25T12:06:14.762 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.749+0000 7f9deff0d700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:06:14.762 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.749+0000 7f9deff0d700 1 -- 172.21.15.102:0/1883044569 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9de8127ba0 con 0x7f9de812e330 2021-05-25T12:06:14.762 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.750+0000 7f9dddffb700 1 -- 172.21.15.102:0/1883044569 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9de40048b0 con 0x7f9de812e330 2021-05-25T12:06:14.762 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.750+0000 7f9dddffb700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:06:14.762 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.750+0000 7f9dddffb700 1 -- 172.21.15.102:0/1883044569 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f9de4004c40 con 0x7f9de812e330 2021-05-25T12:06:14.763 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.750+0000 7f9dddffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:14.763 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.750+0000 7f9dddffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:06:14.763 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.750+0000 7f9dddffb700 10 monclient: dump: 2021-05-25T12:06:14.763 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:14.764 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:14.764 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:14.764 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:14.764 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:14.764 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:14.765 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:14.765 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:14.765 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:14.765 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:14.765 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.750+0000 7f9ddffff700 4 set_mon_vals no callback set 2021-05-25T12:06:14.766 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.750+0000 7f9deff0d700 10 client.184442.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:06:14.766 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.750+0000 7f9deff0d700 10 monclient: _renew_subs 2021-05-25T12:06:14.766 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.750+0000 7f9deff0d700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:06:14.766 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.750+0000 7f9deff0d700 1 -- 172.21.15.102:0/1883044569 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9de8067110 con 0x7f9de812e330 2021-05-25T12:06:14.767 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.750+0000 7f9deff0d700 1 librados: init done 2021-05-25T12:06:14.767 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.750+0000 7f9dddffb700 1 -- 172.21.15.102:0/1883044569 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f9de4004e20 con 0x7f9de812e330 2021-05-25T12:06:14.767 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.751+0000 7f9deff0d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:06:14.767 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.751+0000 7f9deff0d700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:06:14.767 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.751+0000 7f9deff0d700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:06:14.768 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.751+0000 7f9deff0d700 1 -- 172.21.15.102:0/1883044569 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9de8059730 con 0x7f9de812e330 2021-05-25T12:06:14.768 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.751+0000 7f9dedca9700 10 monclient: get_auth_request con 0x7f9dc80331e0 auth_method 0 2021-05-25T12:06:14.768 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.751+0000 7f9dddffb700 10 client.184442.objecter ms_handle_connect 0x7f9dc80331e0 2021-05-25T12:06:14.768 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.753+0000 7f9dddffb700 1 -- 172.21.15.102:0/1883044569 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(192..192 src has 1..192) v4 ==== 5804+0+0 (unknown 636191040 0 0) 0x7f9de40132d0 con 0x7f9de812e330 2021-05-25T12:06:14.768 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.753+0000 7f9dddffb700 10 client.184442.objecter ms_dispatch 0x7f9de8001550 osd_map(192..192 src has 1..192) v4 2021-05-25T12:06:14.769 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.753+0000 7f9dddffb700 3 client.184442.objecter handle_osd_map got epochs [192,192] > 0 2021-05-25T12:06:14.769 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.753+0000 7f9dddffb700 3 client.184442.objecter handle_osd_map decoding full epoch 192 2021-05-25T12:06:14.769 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.753+0000 7f9dddffb700 20 client.184442.objecter dump_active .. 0 homeless 2021-05-25T12:06:14.770 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.757+0000 7f9dddffb700 1 -- 172.21.15.102:0/1883044569 <== mon.2 v1:172.21.15.102:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7f9de40055d0 con 0x7f9de812e330 2021-05-25T12:06:14.770 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.757+0000 7f9dddffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:06:14.770 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.757+0000 7f9dddffb700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:06:14.970 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.969+0000 7f9deff0d700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2021-05-25T12:06:14.971 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.969+0000 7f9deff0d700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2021-05-25T12:06:14.971 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.969+0000 7f9deff0d700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:06:14.971 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.969+0000 7f9deff0d700 1 -- 172.21.15.102:0/1883044569 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f9de8096500 con 0x7f9de812e330 2021-05-25T12:06:14.972 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.969+0000 7f9dddffb700 1 -- 172.21.15.102:0/1883044569 <== mon.2 v1:172.21.15.102:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1342 (unknown 254746657 0 3778802672) 0x7f9de4012e70 con 0x7f9de812e330 2021-05-25T12:06:14.972 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.969+0000 7f9dddffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2021-05-25T12:06:14.972 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.969+0000 7f9dddffb700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:06:14.973 INFO:teuthology.orchestra.run.smithi102.stdout:{"election_epoch":52,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138303579357183","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"20ee91b6-8935-4b98-b868-85c03789ccc4","modified":"2021-05-25T11:59:11.011522Z","created":"2021-05-25T11:59:11.011522Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6790","nonce":0}]},"addr":"172.21.15.102:6790/0","public_addr":"172.21.15.102:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2021-05-25T12:06:14.974 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.970+0000 7f9deff0d700 10 librados: watch_flush enter 2021-05-25T12:06:14.975 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.970+0000 7f9deff0d700 10 librados: watch_flush exit 2021-05-25T12:06:14.975 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.970+0000 7f9deff0d700 20 client.184442.objecter shutdown clearing up homeless session... 2021-05-25T12:06:14.975 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.970+0000 7f9deff0d700 10 client.184442.objecter successfully canceled tick 2021-05-25T12:06:14.975 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.971+0000 7f9deff0d700 1 -- 172.21.15.102:0/1883044569 >> v1:172.21.15.181:6812/29582 conn(0x7f9dc80331e0 legacy=0x7f9dc80356a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:14.976 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.971+0000 7f9deff0d700 10 monclient: shutdown 2021-05-25T12:06:14.976 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.971+0000 7f9deff0d700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:06:14.976 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.971+0000 7f9deff0d700 1 -- 172.21.15.102:0/1883044569 >> v1:172.21.15.102:6790/0 conn(0x7f9de812e330 legacy=0x7f9de8140f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:14.976 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.971+0000 7f9deff0d700 1 -- 172.21.15.102:0/1883044569 shutdown_connections 2021-05-25T12:06:14.976 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.971+0000 7f9deff0d700 1 -- 172.21.15.102:0/1883044569 >> 172.21.15.102:0/1883044569 conn(0x7f9de809b370 msgr2=0x7f9de809a040 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:06:14.977 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.971+0000 7f9deff0d700 1 -- 172.21.15.102:0/1883044569 shutdown_connections 2021-05-25T12:06:14.977 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.971+0000 7f9deff0d700 1 -- 172.21.15.102:0/1883044569 wait complete. 2021-05-25T12:06:14.977 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:14.971+0000 7f9deff0d700 1 librados: shutdown 2021-05-25T12:06:14.983 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2021-05-25T12:06:14.984 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2021-05-25T12:06:15.093 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.092+0000 7f0f14ef8700 10 monclient: get_monmap_and_config 2021-05-25T12:06:15.094 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.092+0000 7f0f14ef8700 10 monclient: build_initial_monmap 2021-05-25T12:06:15.094 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.092+0000 7f0f14ef8700 10 monclient: monmap: 2021-05-25T12:06:15.094 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:06:15.094 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:15.095 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:06:15.093022+0000 2021-05-25T12:06:15.095 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:06:15.093022+0000 2021-05-25T12:06:15.095 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:06:15.095 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:15.095 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:06:15.096 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:06:15.096 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:06:15.096 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:15.096 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.092+0000 7f0f14ef8700 1 Processor -- start 2021-05-25T12:06:15.097 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.092+0000 7f0f14ef8700 1 -- start start 2021-05-25T12:06:15.097 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.092+0000 7f0f14ef8700 10 monclient: init 2021-05-25T12:06:15.109 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.093+0000 7f0f14ef8700 10 monclient: _reopen_session rank -1 2021-05-25T12:06:15.110 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.093+0000 7f0f14ef8700 10 monclient: _add_conns ranks=[1,0,2] 2021-05-25T12:06:15.110 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.093+0000 7f0f14ef8700 10 monclient(hunting): picked mon.noname-c con 0x7f0f1009a5e0 addr v1:172.21.15.181:6789/0 2021-05-25T12:06:15.110 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.093+0000 7f0f14ef8700 10 monclient(hunting): picked mon.noname-a con 0x7f0f10063de0 addr v1:172.21.15.102:6789/0 2021-05-25T12:06:15.111 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.093+0000 7f0f14ef8700 10 monclient(hunting): picked mon.noname-b con 0x7f0f101325c0 addr v1:172.21.15.102:6790/0 2021-05-25T12:06:15.111 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.093+0000 7f0f14ef8700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f1009a3a0 con 0x7f0f10063de0 2021-05-25T12:06:15.111 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.093+0000 7f0f14ef8700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f1009b400 con 0x7f0f1009a5e0 2021-05-25T12:06:15.111 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.093+0000 7f0f14ef8700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f100963f0 con 0x7f0f101325c0 2021-05-25T12:06:15.111 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.093+0000 7f0f14ef8700 10 monclient(hunting): _renew_subs 2021-05-25T12:06:15.112 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.093+0000 7f0f14ef8700 10 monclient(hunting): authenticate will time out at 1175.690359s 2021-05-25T12:06:15.112 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.093+0000 7f0f0ed9d700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f0f101325c0 0x7f0f101349b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:56292/0 (socket says 172.21.15.102:56292) 2021-05-25T12:06:15.112 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.093+0000 7f0f0ed9d700 1 -- 172.21.15.102:0/3999766213 learned_addr learned my addr 172.21.15.102:0/3999766213 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:06:15.112 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.093+0000 7f0f0dd9b700 1 --1- 172.21.15.102:0/3999766213 >> v1:172.21.15.102:6789/0 conn(0x7f0f10063de0 0x7f0f10064260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33178/0 (socket says 172.21.15.102:33178) 2021-05-25T12:06:15.113 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.093+0000 7f0f0d59a700 1 -- 172.21.15.102:0/3999766213 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2787442456 0 0) 0x7f0f100963f0 con 0x7f0f101325c0 2021-05-25T12:06:15.113 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.093+0000 7f0f0d59a700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:15.113 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.093+0000 7f0f0d59a700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:15.113 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.093+0000 7f0f0d59a700 10 monclient(hunting): my global_id is 184445 2021-05-25T12:06:15.113 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.093+0000 7f0f0d59a700 1 -- 172.21.15.102:0/3999766213 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ef4003270 con 0x7f0f101325c0 2021-05-25T12:06:15.114 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.093+0000 7f0f0d59a700 1 -- 172.21.15.102:0/3999766213 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 933461554 0 0) 0x7f0f1009a3a0 con 0x7f0f10063de0 2021-05-25T12:06:15.114 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.093+0000 7f0f0d59a700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:15.114 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.093+0000 7f0f0d59a700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:15.114 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.093+0000 7f0f0d59a700 10 monclient(hunting): my global_id is 204099 2021-05-25T12:06:15.115 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.093+0000 7f0f0d59a700 1 -- 172.21.15.102:0/3999766213 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f100963f0 con 0x7f0f10063de0 2021-05-25T12:06:15.115 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.093+0000 7f0f0d59a700 1 -- 172.21.15.102:0/3999766213 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 967464556 0 0) 0x7f0f1009b400 con 0x7f0f1009a5e0 2021-05-25T12:06:15.115 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.093+0000 7f0f0d59a700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:15.115 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.093+0000 7f0f0d59a700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:15.115 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.093+0000 7f0f0d59a700 10 monclient(hunting): my global_id is 194269 2021-05-25T12:06:15.116 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.093+0000 7f0f0d59a700 1 -- 172.21.15.102:0/3999766213 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f1009a3a0 con 0x7f0f1009a5e0 2021-05-25T12:06:15.116 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.093+0000 7f0f0d59a700 1 -- 172.21.15.102:0/3999766213 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2855532283 0 0) 0x7f0f100963f0 con 0x7f0f10063de0 2021-05-25T12:06:15.116 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.093+0000 7f0f0d59a700 1 -- 172.21.15.102:0/3999766213 >> v1:172.21.15.102:6790/0 conn(0x7f0f101325c0 legacy=0x7f0f101349b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:15.116 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.093+0000 7f0f0d59a700 1 -- 172.21.15.102:0/3999766213 >> v1:172.21.15.181:6789/0 conn(0x7f0f1009a5e0 legacy=0x7f0f100636d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:15.117 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.093+0000 7f0f0d59a700 10 monclient: _finish_hunting 0 2021-05-25T12:06:15.117 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.093+0000 7f0f0d59a700 1 monclient: found mon.noname-a 2021-05-25T12:06:15.117 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.093+0000 7f0f0d59a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:06:15.117 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.094+0000 7f0f0d59a700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T12:06:15.117 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.094+0000 7f0f0d59a700 1 -- 172.21.15.102:0/3999766213 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f10096690 con 0x7f0f10063de0 2021-05-25T12:06:15.118 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.094+0000 7f0f0d59a700 10 monclient: _finish_auth 0 2021-05-25T12:06:15.118 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.094+0000 7f0f0d59a700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:06:15.118 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.094+0000 7f0f0d59a700 1 -- 172.21.15.102:0/3999766213 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f0efc0015f0 con 0x7f0f10063de0 2021-05-25T12:06:15.118 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.094+0000 7f0f14ef8700 5 monclient: authenticate success, global_id 204099 2021-05-25T12:06:15.118 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.094+0000 7f0f14ef8700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:06:15.119 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.094+0000 7f0f0d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:15.119 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.094+0000 7f0f0d59a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:06:15.119 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.094+0000 7f0f0d59a700 10 monclient: dump: 2021-05-25T12:06:15.119 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:15.120 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:15.120 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:15.120 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:15.120 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:15.120 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:15.121 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:15.121 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:15.121 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:15.121 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:15.121 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.094+0000 7f0f14ef8700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:06:15.122 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.094+0000 7f0f0d59a700 1 -- 172.21.15.102:0/3999766213 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0efc001820 con 0x7f0f10063de0 2021-05-25T12:06:15.122 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.094+0000 7f0f0d59a700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:06:15.122 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.094+0000 7f0f0d59a700 1 -- 172.21.15.102:0/3999766213 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f0efc001bb0 con 0x7f0f10063de0 2021-05-25T12:06:15.122 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.094+0000 7f0f14ef8700 10 monclient: get_monmap_and_config success 2021-05-25T12:06:15.122 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.094+0000 7f0f14ef8700 4 set_mon_vals no callback set 2021-05-25T12:06:15.123 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.094+0000 7f0f0d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:15.123 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.094+0000 7f0f0d59a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:06:15.123 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.094+0000 7f0f0d59a700 10 monclient: dump: 2021-05-25T12:06:15.123 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:15.123 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:15.124 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:15.124 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:15.124 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:15.124 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:15.124 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:15.125 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:15.125 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:15.125 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:15.125 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.094+0000 7f0f14ef8700 10 monclient: shutdown 2021-05-25T12:06:15.125 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.094+0000 7f0f14ef8700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:06:15.126 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.094+0000 7f0f14ef8700 1 -- 172.21.15.102:0/3999766213 >> v1:172.21.15.102:6789/0 conn(0x7f0f10063de0 legacy=0x7f0f10064260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:15.126 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.094+0000 7f0f14ef8700 1 -- 172.21.15.102:0/3999766213 shutdown_connections 2021-05-25T12:06:15.126 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.094+0000 7f0f14ef8700 1 -- 172.21.15.102:0/3999766213 >> 172.21.15.102:0/3999766213 conn(0x7f0f1009bc30 msgr2=0x7f0f1009c040 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:06:15.126 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.094+0000 7f0f14ef8700 1 -- 172.21.15.102:0/3999766213 shutdown_connections 2021-05-25T12:06:15.127 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.094+0000 7f0f14ef8700 1 -- 172.21.15.102:0/3999766213 wait complete. 2021-05-25T12:06:15.127 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.095+0000 7f0f14ef8700 10 monclient: build_initial_monmap 2021-05-25T12:06:15.127 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.095+0000 7f0f14ef8700 10 monclient: monmap: 2021-05-25T12:06:15.127 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:06:15.128 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:06:15.128 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:06:15.128 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:06:15.128 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:06:15.128 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:06:15.129 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:06:15.129 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:06:15.129 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:06:15.129 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:15.129 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.095+0000 7f0f14ef8700 1 librados: starting msgr at 2021-05-25T12:06:15.130 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.095+0000 7f0f14ef8700 1 librados: starting objecter 2021-05-25T12:06:15.130 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.095+0000 7f0f14ef8700 1 Processor -- start 2021-05-25T12:06:15.130 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.095+0000 7f0f14ef8700 1 -- start start 2021-05-25T12:06:15.130 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.095+0000 7f0f14ef8700 1 librados: setting wanted keys 2021-05-25T12:06:15.130 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.095+0000 7f0f14ef8700 1 librados: calling monclient init 2021-05-25T12:06:15.131 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.095+0000 7f0f14ef8700 10 monclient: init 2021-05-25T12:06:15.131 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.095+0000 7f0f14ef8700 10 monclient: _reopen_session rank -1 2021-05-25T12:06:15.131 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.095+0000 7f0f14ef8700 10 monclient: _add_conns ranks=[1,2,0] 2021-05-25T12:06:15.131 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.095+0000 7f0f14ef8700 10 monclient(hunting): picked mon.noname-b con 0x7f0f10063de0 addr v1:172.21.15.181:6789/0 2021-05-25T12:06:15.131 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.095+0000 7f0f14ef8700 10 monclient(hunting): picked mon.noname-c con 0x7f0f1009a5e0 addr v1:172.21.15.102:6790/0 2021-05-25T12:06:15.132 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.096+0000 7f0f14ef8700 10 monclient(hunting): picked mon.noname-a con 0x7f0f101325c0 addr v1:172.21.15.102:6789/0 2021-05-25T12:06:15.132 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.096+0000 7f0f14ef8700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f1012b220 con 0x7f0f101325c0 2021-05-25T12:06:15.132 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.096+0000 7f0f14ef8700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f10115520 con 0x7f0f10063de0 2021-05-25T12:06:15.132 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.096+0000 7f0f14ef8700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f1009ab10 con 0x7f0f1009a5e0 2021-05-25T12:06:15.132 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.096+0000 7f0f14ef8700 10 monclient(hunting): _renew_subs 2021-05-25T12:06:15.133 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.096+0000 7f0f14ef8700 10 monclient(hunting): authenticate will time out at 1175.693041s 2021-05-25T12:06:15.133 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.096+0000 7f0f0dd9b700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f0f1009a5e0 0x7f0f10140320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:56296/0 (socket says 172.21.15.102:56296) 2021-05-25T12:06:15.133 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.096+0000 7f0f0dd9b700 1 -- 172.21.15.102:0/72650063 learned_addr learned my addr 172.21.15.102:0/72650063 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:06:15.133 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.096+0000 7f0efa7fc700 10 client.?.objecter ms_handle_connect 0x7f0f1009a5e0 2021-05-25T12:06:15.133 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.096+0000 7f0efa7fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:06:15.134 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.096+0000 7f0efa7fc700 10 client.?.objecter ms_handle_connect 0x7f0f101325c0 2021-05-25T12:06:15.134 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.096+0000 7f0efa7fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:06:15.134 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.096+0000 7f0efa7fc700 1 -- 172.21.15.102:0/72650063 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 178718152 0 0) 0x7f0f1009ab10 con 0x7f0f1009a5e0 2021-05-25T12:06:15.134 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.096+0000 7f0efa7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:15.134 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.096+0000 7f0efa7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:15.135 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.096+0000 7f0efa7fc700 10 monclient(hunting): my global_id is 184448 2021-05-25T12:06:15.135 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.096+0000 7f0efa7fc700 1 -- 172.21.15.102:0/72650063 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0eec004c10 con 0x7f0f1009a5e0 2021-05-25T12:06:15.135 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.096+0000 7f0efa7fc700 10 client.?.objecter ms_handle_connect 0x7f0f10063de0 2021-05-25T12:06:15.135 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.096+0000 7f0efa7fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:06:15.136 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.096+0000 7f0efa7fc700 1 -- 172.21.15.102:0/72650063 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2724622082 0 0) 0x7f0f1012b220 con 0x7f0f101325c0 2021-05-25T12:06:15.136 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.096+0000 7f0efa7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:15.136 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.096+0000 7f0efa7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:15.136 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.096+0000 7f0efa7fc700 10 monclient(hunting): my global_id is 204102 2021-05-25T12:06:15.136 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.096+0000 7f0efa7fc700 1 -- 172.21.15.102:0/72650063 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f1009ab10 con 0x7f0f101325c0 2021-05-25T12:06:15.137 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.096+0000 7f0efa7fc700 1 -- 172.21.15.102:0/72650063 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2059290798 0 0) 0x7f0f10115520 con 0x7f0f10063de0 2021-05-25T12:06:15.137 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.096+0000 7f0efa7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:15.137 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.096+0000 7f0efa7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:15.137 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.096+0000 7f0efa7fc700 10 monclient(hunting): my global_id is 194272 2021-05-25T12:06:15.137 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.096+0000 7f0efa7fc700 1 -- 172.21.15.102:0/72650063 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f1012b220 con 0x7f0f10063de0 2021-05-25T12:06:15.138 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.096+0000 7f0efa7fc700 1 -- 172.21.15.102:0/72650063 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 557986201 0 0) 0x7f0f1009ab10 con 0x7f0f101325c0 2021-05-25T12:06:15.138 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.096+0000 7f0efa7fc700 1 -- 172.21.15.102:0/72650063 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0f10115520 con 0x7f0f101325c0 2021-05-25T12:06:15.138 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.096+0000 7f0efa7fc700 1 -- 172.21.15.102:0/72650063 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f0f00001db0 con 0x7f0f101325c0 2021-05-25T12:06:15.138 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.096+0000 7f0efa7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:15.138 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.096+0000 7f0efa7fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:06:15.139 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.096+0000 7f0efa7fc700 10 monclient(hunting): dump: 2021-05-25T12:06:15.139 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:15.139 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:15.139 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:15.140 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:15.140 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:15.140 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:15.140 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:15.140 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:15.141 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:15.141 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:15.141 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.096+0000 7f0efa7fc700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:06:15.141 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.096+0000 7f0efa7fc700 1 -- 172.21.15.102:0/72650063 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3291222664 0 0) 0x7f0eec004c10 con 0x7f0f1009a5e0 2021-05-25T12:06:15.141 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.096+0000 7f0efa7fc700 1 -- 172.21.15.102:0/72650063 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0f1009ab10 con 0x7f0f1009a5e0 2021-05-25T12:06:15.142 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.096+0000 7f0efa7fc700 1 -- 172.21.15.102:0/72650063 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f0efc001570 con 0x7f0f1009a5e0 2021-05-25T12:06:15.142 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.096+0000 7f0efa7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:15.142 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.096+0000 7f0efa7fc700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2021-05-25T12:06:15.142 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.096+0000 7f0efa7fc700 10 monclient(hunting): dump: 2021-05-25T12:06:15.142 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:15.143 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:15.143 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:15.143 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:15.143 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:15.143 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:15.144 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:15.144 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:15.144 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:15.144 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:15.144 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.096+0000 7f0efa7fc700 1 -- 172.21.15.102:0/72650063 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 374634917 0 0) 0x7f0f1012b220 con 0x7f0f10063de0 2021-05-25T12:06:15.145 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.096+0000 7f0efa7fc700 1 -- 172.21.15.102:0/72650063 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0eec004c10 con 0x7f0f10063de0 2021-05-25T12:06:15.145 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.096+0000 7f0efa7fc700 1 -- 172.21.15.102:0/72650063 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f0f040018c0 con 0x7f0f10063de0 2021-05-25T12:06:15.145 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.096+0000 7f0efa7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:15.145 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.096+0000 7f0efa7fc700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2021-05-25T12:06:15.146 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.096+0000 7f0efa7fc700 10 monclient(hunting): dump: 2021-05-25T12:06:15.146 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:15.146 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:15.146 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:15.146 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:15.147 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:15.147 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:15.147 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:15.147 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:15.147 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:15.148 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:15.148 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.096+0000 7f0efa7fc700 1 -- 172.21.15.102:0/72650063 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3441218271 0 0) 0x7f0f10115520 con 0x7f0f101325c0 2021-05-25T12:06:15.148 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.096+0000 7f0efa7fc700 1 -- 172.21.15.102:0/72650063 >> v1:172.21.15.102:6790/0 conn(0x7f0f1009a5e0 legacy=0x7f0f10140320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:15.148 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.096+0000 7f0efa7fc700 1 -- 172.21.15.102:0/72650063 >> v1:172.21.15.181:6789/0 conn(0x7f0f10063de0 legacy=0x7f0f1012ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:15.148 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.096+0000 7f0efa7fc700 10 monclient: _finish_hunting 0 2021-05-25T12:06:15.149 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.096+0000 7f0efa7fc700 1 monclient: found mon.a 2021-05-25T12:06:15.149 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.096+0000 7f0efa7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:06:15.149 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.096+0000 7f0efa7fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:06:15.149 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.096+0000 7f0efa7fc700 1 -- 172.21.15.102:0/72650063 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f1012e8a0 con 0x7f0f101325c0 2021-05-25T12:06:15.149 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.096+0000 7f0efa7fc700 10 monclient: _finish_auth 0 2021-05-25T12:06:15.150 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.096+0000 7f0efa7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:06:15.150 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.096+0000 7f0f14ef8700 5 monclient: authenticate success, global_id 204102 2021-05-25T12:06:15.150 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.096+0000 7f0f14ef8700 10 monclient: _renew_subs 2021-05-25T12:06:15.150 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.096+0000 7f0f14ef8700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:06:15.150 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.096+0000 7f0f14ef8700 1 -- 172.21.15.102:0/72650063 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f10094480 con 0x7f0f101325c0 2021-05-25T12:06:15.151 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.096+0000 7f0f14ef8700 10 client.204102.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:06:15.151 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.096+0000 7f0f14ef8700 10 monclient: _renew_subs 2021-05-25T12:06:15.151 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.096+0000 7f0f14ef8700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:06:15.151 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.096+0000 7f0f14ef8700 1 -- 172.21.15.102:0/72650063 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f100931c0 con 0x7f0f101325c0 2021-05-25T12:06:15.152 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.096+0000 7f0f14ef8700 1 librados: init done 2021-05-25T12:06:15.152 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.096+0000 7f0efa7fc700 1 -- 172.21.15.102:0/72650063 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0f00002060 con 0x7f0f101325c0 2021-05-25T12:06:15.152 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.096+0000 7f0efa7fc700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:06:15.152 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.096+0000 7f0efa7fc700 1 -- 172.21.15.102:0/72650063 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f0f00004d00 con 0x7f0f101325c0 2021-05-25T12:06:15.152 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.096+0000 7f0efa7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:15.153 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.096+0000 7f0efa7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:06:15.153 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.096+0000 7f0f0cd99700 4 set_mon_vals no callback set 2021-05-25T12:06:15.153 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.096+0000 7f0efa7fc700 10 monclient: dump: 2021-05-25T12:06:15.153 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:15.153 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:15.154 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:15.154 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:15.154 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:15.154 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:15.154 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:15.155 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:15.155 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:15.155 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:15.155 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.097+0000 7f0f14ef8700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:06:15.155 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.097+0000 7f0f14ef8700 1 -- 172.21.15.102:0/72650063 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f0ef00065a0 con 0x7f0ef0002840 2021-05-25T12:06:15.156 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.097+0000 7f0efa7fc700 1 -- 172.21.15.102:0/72650063 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f0f0000e020 con 0x7f0f101325c0 2021-05-25T12:06:15.156 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.097+0000 7f0efa7fc700 10 client.204102.objecter ms_handle_connect 0x7f0ef0002840 2021-05-25T12:06:15.156 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.097+0000 7f0efa7fc700 10 client.204102.objecter resend_mon_ops 2021-05-25T12:06:15.156 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.097+0000 7f0efa7fc700 1 -- 172.21.15.102:0/72650063 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(193..193 src has 1..193) v4 ==== 6511+0+0 (unknown 3415019523 0 0) 0x7f0f0003dc20 con 0x7f0f101325c0 2021-05-25T12:06:15.157 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.097+0000 7f0efa7fc700 10 client.204102.objecter ms_dispatch 0x7f0f10001550 osd_map(193..193 src has 1..193) v4 2021-05-25T12:06:15.157 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.097+0000 7f0efa7fc700 3 client.204102.objecter handle_osd_map got epochs [193,193] > 0 2021-05-25T12:06:15.157 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.097+0000 7f0efa7fc700 3 client.204102.objecter handle_osd_map decoding full epoch 193 2021-05-25T12:06:15.157 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.097+0000 7f0efa7fc700 20 client.204102.objecter dump_active .. 0 homeless 2021-05-25T12:06:15.157 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.097+0000 7f0efa7fc700 1 -- 172.21.15.102:0/72650063 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2779793999 0 0) 0x7f0ef00065a0 con 0x7f0ef0002840 2021-05-25T12:06:15.158 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.097+0000 7f0efa7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:15.158 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.097+0000 7f0efa7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:15.158 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.097+0000 7f0efa7fc700 10 monclient(hunting): my global_id is 204105 2021-05-25T12:06:15.158 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.097+0000 7f0efa7fc700 1 -- 172.21.15.102:0/72650063 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f10115520 con 0x7f0ef0002840 2021-05-25T12:06:15.159 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.097+0000 7f0f0dd9b700 10 monclient: get_auth_request con 0x7f0eec0334e0 auth_method 0 2021-05-25T12:06:15.159 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.097+0000 7f0efa7fc700 1 -- 172.21.15.102:0/72650063 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 337325200 0 0) 0x7f0f10115520 con 0x7f0ef0002840 2021-05-25T12:06:15.159 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.098+0000 7f0efa7fc700 1 -- 172.21.15.102:0/72650063 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0ef00065a0 con 0x7f0ef0002840 2021-05-25T12:06:15.159 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.098+0000 7f0efa7fc700 1 -- 172.21.15.102:0/72650063 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f0f04002040 con 0x7f0ef0002840 2021-05-25T12:06:15.159 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.098+0000 7f0efa7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:15.160 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.098+0000 7f0efa7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:06:15.160 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.098+0000 7f0efa7fc700 10 monclient: dump: 2021-05-25T12:06:15.160 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:15.160 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:15.160 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:15.161 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:15.161 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:15.161 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:15.161 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:15.161 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:15.162 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:15.162 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:15.162 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.098+0000 7f0efa7fc700 1 -- 172.21.15.102:0/72650063 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3753759156 0 0) 0x7f0ef00065a0 con 0x7f0ef0002840 2021-05-25T12:06:15.162 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.098+0000 7f0efa7fc700 1 -- 172.21.15.102:0/72650063 --> v1:172.21.15.102:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0ef0009e40 con 0x7f0ef0002840 2021-05-25T12:06:15.162 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.098+0000 7f0efa7fc700 10 client.204102.objecter ms_handle_connect 0x7f0eec0334e0 2021-05-25T12:06:15.163 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.098+0000 7f0efa7fc700 1 -- 172.21.15.102:0/72650063 <== mon.0 v1:172.21.15.102:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7f0f04002380 con 0x7f0ef0002840 2021-05-25T12:06:15.163 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.098+0000 7f0efa7fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:06:15.163 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.098+0000 7f0efa7fc700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:06:15.163 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.098+0000 7f0efa7fc700 1 -- 172.21.15.102:0/72650063 >> v1:172.21.15.102:6789/0 conn(0x7f0ef0002840 legacy=0x7f0ef0004ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:15.163 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.098+0000 7f0efa7fc700 1 -- 172.21.15.102:0/72650063 >> v1:172.21.15.102:6789/0 conn(0x7f0ef0002840 legacy=0x7f0ef0004ce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:06:15.164 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.104+0000 7f0f14ef8700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:06:15.164 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.104+0000 7f0f14ef8700 1 -- 172.21.15.102:0/72650063 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f0ef0009b00 con 0x7f0ef0008dc0 2021-05-25T12:06:15.164 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.105+0000 7f0efa7fc700 10 client.204102.objecter ms_handle_connect 0x7f0ef0008dc0 2021-05-25T12:06:15.164 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.105+0000 7f0efa7fc700 10 client.204102.objecter resend_mon_ops 2021-05-25T12:06:15.164 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.105+0000 7f0efa7fc700 1 -- 172.21.15.102:0/72650063 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 451976567 0 0) 0x7f0ef0009b00 con 0x7f0ef0008dc0 2021-05-25T12:06:15.165 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.105+0000 7f0efa7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:15.165 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.105+0000 7f0efa7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:15.165 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.105+0000 7f0efa7fc700 10 monclient(hunting): my global_id is 204108 2021-05-25T12:06:15.165 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.105+0000 7f0efa7fc700 1 -- 172.21.15.102:0/72650063 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ef00065a0 con 0x7f0ef0008dc0 2021-05-25T12:06:15.165 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.105+0000 7f0efa7fc700 1 -- 172.21.15.102:0/72650063 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2411943730 0 0) 0x7f0ef00065a0 con 0x7f0ef0008dc0 2021-05-25T12:06:15.166 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.105+0000 7f0efa7fc700 1 -- 172.21.15.102:0/72650063 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0ef0009b00 con 0x7f0ef0008dc0 2021-05-25T12:06:15.166 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.105+0000 7f0efa7fc700 1 -- 172.21.15.102:0/72650063 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f0f04002b00 con 0x7f0ef0008dc0 2021-05-25T12:06:15.166 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.105+0000 7f0efa7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:15.166 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.105+0000 7f0efa7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:06:15.166 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.105+0000 7f0efa7fc700 10 monclient: dump: 2021-05-25T12:06:15.167 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:15.167 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:15.167 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:15.167 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:15.167 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:15.168 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:15.168 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:15.168 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:15.168 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:15.169 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:15.169 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.105+0000 7f0efa7fc700 1 -- 172.21.15.102:0/72650063 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2549737320 0 0) 0x7f0ef0009b00 con 0x7f0ef0008dc0 2021-05-25T12:06:15.169 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.105+0000 7f0efa7fc700 1 -- 172.21.15.102:0/72650063 --> v1:172.21.15.102:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f0ef0008740 con 0x7f0ef0008dc0 2021-05-25T12:06:15.169 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.105+0000 7f0efa7fc700 1 -- 172.21.15.102:0/72650063 <== mon.0 v1:172.21.15.102:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3770 (unknown 0 0 2737540109) 0x7f0f04003ed0 con 0x7f0ef0008dc0 2021-05-25T12:06:15.169 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.105+0000 7f0efa7fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:06:15.170 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.105+0000 7f0efa7fc700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:06:15.170 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.105+0000 7f0efa7fc700 1 -- 172.21.15.102:0/72650063 >> v1:172.21.15.102:6789/0 conn(0x7f0ef0008dc0 legacy=0x7f0ef0009250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:15.170 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.105+0000 7f0efa7fc700 1 -- 172.21.15.102:0/72650063 >> v1:172.21.15.102:6789/0 conn(0x7f0ef0008dc0 legacy=0x7f0ef0009250 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:06:15.171 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:06:15.171 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:06:15.171 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:06:15.171 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "peon", 2021-05-25T12:06:15.172 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 52, 2021-05-25T12:06:15.172 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:06:15.172 INFO:teuthology.orchestra.run.smithi102.stdout: 0, 2021-05-25T12:06:15.172 INFO:teuthology.orchestra.run.smithi102.stdout: 1, 2021-05-25T12:06:15.173 INFO:teuthology.orchestra.run.smithi102.stdout: 2 2021-05-25T12:06:15.173 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:06:15.173 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 0, 2021-05-25T12:06:15.173 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:06:15.173 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:06:15.174 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:06:15.174 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:06:15.174 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:06:15.174 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:06:15.174 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:06:15.175 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:06:15.175 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:06:15.175 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:06:15.175 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:06:15.175 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:06:15.176 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:06:15.176 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:06:15.176 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:06:15.176 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:06:15.176 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:06:15.177 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:06:15.177 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:06:15.177 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:06:15.177 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:06:15.177 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:06:15.178 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:06:15.178 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:06:15.178 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:15.178 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:15.179 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:06:15.179 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:06:15.179 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:06:15.179 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:06:15.179 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:06:15.180 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:06:15.180 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:06:15.180 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:06:15.180 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:06:15.180 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:06:15.181 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:06:15.181 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:06:15.181 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:06:15.181 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:06:15.181 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:06:15.182 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:06:15.182 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:06:15.182 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:06:15.182 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:06:15.182 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:06:15.183 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:06:15.183 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:06:15.183 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:06:15.183 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:06:15.183 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:06:15.184 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:06:15.184 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:15.184 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:06:15.184 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:15.184 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:06:15.185 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:06:15.185 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:06:15.185 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:06:15.185 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:15.186 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:06:15.186 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:06:15.186 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:06:15.186 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:15.186 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:15.187 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:15.187 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:06:15.187 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:06:15.187 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:06:15.187 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:06:15.188 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:06:15.188 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:15.188 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:15.188 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:06:15.188 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:06:15.189 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:06:15.189 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:06:15.189 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:15.189 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:06:15.189 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:06:15.190 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:06:15.190 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:15.190 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:15.190 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:15.190 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:06:15.191 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:06:15.191 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:06:15.191 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:06:15.191 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:06:15.192 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:15.192 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:15.192 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:06:15.192 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:06:15.192 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:06:15.192 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:06:15.193 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:15.193 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:06:15.193 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:06:15.193 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:06:15.193 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:15.194 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:15.194 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:15.194 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:06:15.194 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:06:15.195 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:06:15.195 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:06:15.195 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:06:15.195 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:15.195 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:15.196 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:15.196 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:06:15.196 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:06:15.196 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:15.196 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:06:15.197 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:06:15.197 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:06:15.197 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:15.197 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:06:15.198 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:06:15.198 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:15.198 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:06:15.198 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:06:15.198 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 2 2021-05-25T12:06:15.199 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:15.199 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:15.199 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:15.199 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:06:15.199 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:06:15.201 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.105+0000 7f0f14ef8700 10 librados: watch_flush enter 2021-05-25T12:06:15.202 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.106+0000 7f0f14ef8700 10 librados: watch_flush exit 2021-05-25T12:06:15.202 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.106+0000 7f0f14ef8700 20 client.204102.objecter shutdown clearing up homeless session... 2021-05-25T12:06:15.202 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.106+0000 7f0f14ef8700 10 client.204102.objecter successfully canceled tick 2021-05-25T12:06:15.202 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.106+0000 7f0f14ef8700 1 -- 172.21.15.102:0/72650063 >> v1:172.21.15.181:6812/29582 conn(0x7f0eec0334e0 legacy=0x7f0eec0359a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:15.202 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.106+0000 7f0f14ef8700 10 monclient: shutdown 2021-05-25T12:06:15.203 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.106+0000 7f0f14ef8700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:06:15.203 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.106+0000 7f0f14ef8700 1 -- 172.21.15.102:0/72650063 >> v1:172.21.15.102:6789/0 conn(0x7f0f101325c0 legacy=0x7f0f10143a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:15.203 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.106+0000 7f0f0ed9d700 1 -- 172.21.15.102:0/72650063 reap_dead start 2021-05-25T12:06:15.203 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.106+0000 7f0f0ed9d700 1 -- 172.21.15.102:0/72650063 reap_dead start 2021-05-25T12:06:15.204 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.106+0000 7f0f14ef8700 1 -- 172.21.15.102:0/72650063 shutdown_connections 2021-05-25T12:06:15.204 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.106+0000 7f0f14ef8700 1 -- 172.21.15.102:0/72650063 >> 172.21.15.102:0/72650063 conn(0x7f0f1009bc30 msgr2=0x7f0f10131cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:06:15.204 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.106+0000 7f0f14ef8700 1 -- 172.21.15.102:0/72650063 shutdown_connections 2021-05-25T12:06:15.204 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.106+0000 7f0f14ef8700 1 -- 172.21.15.102:0/72650063 wait complete. 2021-05-25T12:06:15.204 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.106+0000 7f0f14ef8700 1 librados: shutdown 2021-05-25T12:06:15.205 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2021-05-25T12:06:15.315 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.313+0000 7f91cc17c700 10 monclient: get_monmap_and_config 2021-05-25T12:06:15.315 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.313+0000 7f91cc17c700 10 monclient: build_initial_monmap 2021-05-25T12:06:15.316 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.313+0000 7f91cc17c700 10 monclient: monmap: 2021-05-25T12:06:15.316 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:06:15.316 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:15.316 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:06:15.314617+0000 2021-05-25T12:06:15.316 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:06:15.314617+0000 2021-05-25T12:06:15.317 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:06:15.317 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:15.317 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:06:15.317 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:06:15.318 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:06:15.318 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:15.318 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.314+0000 7f91cc17c700 1 Processor -- start 2021-05-25T12:06:15.318 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.314+0000 7f91cc17c700 1 -- start start 2021-05-25T12:06:15.318 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.314+0000 7f91cc17c700 10 monclient: init 2021-05-25T12:06:15.332 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.314+0000 7f91cc17c700 10 monclient: _reopen_session rank -1 2021-05-25T12:06:15.332 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.314+0000 7f91cc17c700 10 monclient: _add_conns ranks=[2,1,0] 2021-05-25T12:06:15.332 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.314+0000 7f91cc17c700 10 monclient(hunting): picked mon.noname-b con 0x7f91c4090990 addr v1:172.21.15.102:6790/0 2021-05-25T12:06:15.333 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.314+0000 7f91cc17c700 10 monclient(hunting): picked mon.noname-c con 0x7f91c4090d70 addr v1:172.21.15.181:6789/0 2021-05-25T12:06:15.333 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.314+0000 7f91cc17c700 10 monclient(hunting): picked mon.noname-a con 0x7f91c412e330 addr v1:172.21.15.102:6789/0 2021-05-25T12:06:15.333 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.314+0000 7f91cc17c700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91c4096fa0 con 0x7f91c412e330 2021-05-25T12:06:15.333 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.314+0000 7f91cc17c700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91c40926a0 con 0x7f91c4090d70 2021-05-25T12:06:15.334 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.314+0000 7f91cc17c700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91c4092940 con 0x7f91c4090990 2021-05-25T12:06:15.334 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.314+0000 7f91cc17c700 10 monclient(hunting): _renew_subs 2021-05-25T12:06:15.334 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.314+0000 7f91cc17c700 10 monclient(hunting): authenticate will time out at 1175.911847s 2021-05-25T12:06:15.334 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.314+0000 7f91c9f18700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f91c4090990 0x7f91c4093240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:56306/0 (socket says 172.21.15.102:56306) 2021-05-25T12:06:15.334 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.314+0000 7f91c9f18700 1 -- 172.21.15.102:0/1424544401 learned_addr learned my addr 172.21.15.102:0/1424544401 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:06:15.335 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.315+0000 7f91c8f16700 1 -- 172.21.15.102:0/1424544401 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1257165704 0 0) 0x7f91c4096fa0 con 0x7f91c412e330 2021-05-25T12:06:15.335 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.315+0000 7f91c8f16700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:15.335 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.315+0000 7f91c8f16700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:15.335 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.315+0000 7f91c8f16700 10 monclient(hunting): my global_id is 204111 2021-05-25T12:06:15.336 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.315+0000 7f91c8f16700 1 -- 172.21.15.102:0/1424544401 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91ac003270 con 0x7f91c412e330 2021-05-25T12:06:15.336 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.315+0000 7f91c8f16700 1 -- 172.21.15.102:0/1424544401 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1715858513 0 0) 0x7f91c4092940 con 0x7f91c4090990 2021-05-25T12:06:15.336 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.315+0000 7f91c8f16700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:15.336 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.315+0000 7f91c8f16700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:15.337 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.315+0000 7f91c8f16700 10 monclient(hunting): my global_id is 184451 2021-05-25T12:06:15.337 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.315+0000 7f91c8f16700 1 -- 172.21.15.102:0/1424544401 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91c4096fa0 con 0x7f91c4090990 2021-05-25T12:06:15.337 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.315+0000 7f91c8f16700 1 -- 172.21.15.102:0/1424544401 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3695669532 0 0) 0x7f91c40926a0 con 0x7f91c4090d70 2021-05-25T12:06:15.337 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.315+0000 7f91c8f16700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:15.337 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.315+0000 7f91c8f16700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:15.338 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.315+0000 7f91c8f16700 10 monclient(hunting): my global_id is 194275 2021-05-25T12:06:15.338 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.315+0000 7f91c8f16700 1 -- 172.21.15.102:0/1424544401 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91c4092940 con 0x7f91c4090d70 2021-05-25T12:06:15.338 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.315+0000 7f91c8f16700 1 -- 172.21.15.102:0/1424544401 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 386035310 0 0) 0x7f91ac003270 con 0x7f91c412e330 2021-05-25T12:06:15.338 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.315+0000 7f91c8f16700 1 -- 172.21.15.102:0/1424544401 >> v1:172.21.15.102:6790/0 conn(0x7f91c4090990 legacy=0x7f91c4093240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:15.339 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.315+0000 7f91c8f16700 1 -- 172.21.15.102:0/1424544401 >> v1:172.21.15.181:6789/0 conn(0x7f91c4090d70 legacy=0x7f91c4091200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:15.339 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.315+0000 7f91c8f16700 10 monclient: _finish_hunting 0 2021-05-25T12:06:15.339 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.315+0000 7f91c8f16700 1 monclient: found mon.noname-a 2021-05-25T12:06:15.339 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.315+0000 7f91c8f16700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:06:15.339 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.315+0000 7f91c8f16700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T12:06:15.340 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.315+0000 7f91c8f16700 1 -- 172.21.15.102:0/1424544401 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91c4092be0 con 0x7f91c412e330 2021-05-25T12:06:15.340 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.315+0000 7f91c8f16700 10 monclient: _finish_auth 0 2021-05-25T12:06:15.340 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.315+0000 7f91c8f16700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:06:15.340 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.315+0000 7f91cc17c700 5 monclient: authenticate success, global_id 204111 2021-05-25T12:06:15.340 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.315+0000 7f91c8f16700 1 -- 172.21.15.102:0/1424544401 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f91c0001990 con 0x7f91c412e330 2021-05-25T12:06:15.341 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.315+0000 7f91c8f16700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:15.341 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.315+0000 7f91c8f16700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:06:15.341 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.315+0000 7f91c8f16700 10 monclient: dump: 2021-05-25T12:06:15.341 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:15.341 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:15.342 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:15.342 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:15.342 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:15.342 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:15.343 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:15.343 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:15.343 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:15.343 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:15.343 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.315+0000 7f91cc17c700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:06:15.344 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.315+0000 7f91c8f16700 1 -- 172.21.15.102:0/1424544401 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f91c0001c00 con 0x7f91c412e330 2021-05-25T12:06:15.344 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.315+0000 7f91c8f16700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:06:15.344 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.315+0000 7f91c8f16700 1 -- 172.21.15.102:0/1424544401 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f91c0001f90 con 0x7f91c412e330 2021-05-25T12:06:15.344 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.315+0000 7f91cc17c700 10 monclient: get_monmap_and_config success 2021-05-25T12:06:15.344 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.315+0000 7f91cc17c700 4 set_mon_vals no callback set 2021-05-25T12:06:15.345 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.315+0000 7f91c8f16700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:15.345 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.315+0000 7f91c8f16700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:06:15.345 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.315+0000 7f91c8f16700 10 monclient: dump: 2021-05-25T12:06:15.345 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:15.345 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:15.346 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:15.346 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:15.346 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:15.346 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:15.346 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:15.347 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:15.347 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:15.347 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:15.347 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.315+0000 7f91cc17c700 10 monclient: shutdown 2021-05-25T12:06:15.347 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.315+0000 7f91cc17c700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:06:15.348 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.315+0000 7f91cc17c700 1 -- 172.21.15.102:0/1424544401 >> v1:172.21.15.102:6789/0 conn(0x7f91c412e330 legacy=0x7f91c4130720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:15.348 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.316+0000 7f91cc17c700 1 -- 172.21.15.102:0/1424544401 shutdown_connections 2021-05-25T12:06:15.348 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.316+0000 7f91cc17c700 1 -- 172.21.15.102:0/1424544401 >> 172.21.15.102:0/1424544401 conn(0x7f91c409b370 msgr2=0x7f91c4097430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:06:15.348 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.316+0000 7f91cc17c700 1 -- 172.21.15.102:0/1424544401 shutdown_connections 2021-05-25T12:06:15.349 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.316+0000 7f91cc17c700 1 -- 172.21.15.102:0/1424544401 wait complete. 2021-05-25T12:06:15.349 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.316+0000 7f91cc17c700 10 monclient: build_initial_monmap 2021-05-25T12:06:15.349 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.316+0000 7f91cc17c700 10 monclient: monmap: 2021-05-25T12:06:15.349 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:06:15.349 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:06:15.350 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:06:15.350 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:06:15.350 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:06:15.350 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:06:15.351 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:06:15.351 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:06:15.351 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:06:15.351 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:15.351 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.316+0000 7f91cc17c700 1 librados: starting msgr at 2021-05-25T12:06:15.352 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.316+0000 7f91cc17c700 1 librados: starting objecter 2021-05-25T12:06:15.352 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.316+0000 7f91cc17c700 1 Processor -- start 2021-05-25T12:06:15.352 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.316+0000 7f91cc17c700 1 -- start start 2021-05-25T12:06:15.352 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.316+0000 7f91cc17c700 1 librados: setting wanted keys 2021-05-25T12:06:15.353 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.316+0000 7f91cc17c700 1 librados: calling monclient init 2021-05-25T12:06:15.353 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.316+0000 7f91cc17c700 10 monclient: init 2021-05-25T12:06:15.353 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.317+0000 7f91cc17c700 10 monclient: _reopen_session rank -1 2021-05-25T12:06:15.353 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.317+0000 7f91cc17c700 10 monclient: _add_conns ranks=[0,2,1] 2021-05-25T12:06:15.353 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.317+0000 7f91cc17c700 10 monclient(hunting): picked mon.noname-a con 0x7f91c4090990 addr v1:172.21.15.102:6789/0 2021-05-25T12:06:15.354 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.317+0000 7f91cc17c700 10 monclient(hunting): picked mon.noname-c con 0x7f91c4090d70 addr v1:172.21.15.102:6790/0 2021-05-25T12:06:15.354 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.317+0000 7f91cc17c700 10 monclient(hunting): picked mon.noname-b con 0x7f91c412e330 addr v1:172.21.15.181:6789/0 2021-05-25T12:06:15.354 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.317+0000 7f91cc17c700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91c4117a00 con 0x7f91c4090990 2021-05-25T12:06:15.354 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.317+0000 7f91cc17c700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91c4117ca0 con 0x7f91c412e330 2021-05-25T12:06:15.354 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.317+0000 7f91cc17c700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91c4117f40 con 0x7f91c4090d70 2021-05-25T12:06:15.355 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.317+0000 7f91cc17c700 10 monclient(hunting): _renew_subs 2021-05-25T12:06:15.355 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.317+0000 7f91cc17c700 10 monclient(hunting): authenticate will time out at 1175.914424s 2021-05-25T12:06:15.355 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.317+0000 7f91c9f18700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f91c4090990 0x7f91c413ad50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33202/0 (socket says 172.21.15.102:33202) 2021-05-25T12:06:15.355 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.317+0000 7f91c9f18700 1 -- 172.21.15.102:0/3361034572 learned_addr learned my addr 172.21.15.102:0/3361034572 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:06:15.355 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.317+0000 7f91b1ffb700 10 client.?.objecter ms_handle_connect 0x7f91c4090990 2021-05-25T12:06:15.356 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.317+0000 7f91b1ffb700 10 client.?.objecter resend_mon_ops 2021-05-25T12:06:15.356 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.317+0000 7f91b1ffb700 10 client.?.objecter ms_handle_connect 0x7f91c4090d70 2021-05-25T12:06:15.356 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.317+0000 7f91b1ffb700 10 client.?.objecter resend_mon_ops 2021-05-25T12:06:15.356 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.317+0000 7f91b1ffb700 1 -- 172.21.15.102:0/3361034572 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4290632472 0 0) 0x7f91c4117a00 con 0x7f91c4090990 2021-05-25T12:06:15.357 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.317+0000 7f91b1ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:15.357 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.317+0000 7f91b1ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:15.357 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.317+0000 7f91b1ffb700 10 monclient(hunting): my global_id is 204114 2021-05-25T12:06:15.357 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.317+0000 7f91b1ffb700 1 -- 172.21.15.102:0/3361034572 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91a4004c10 con 0x7f91c4090990 2021-05-25T12:06:15.357 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.317+0000 7f91b1ffb700 10 client.?.objecter ms_handle_connect 0x7f91c412e330 2021-05-25T12:06:15.358 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.317+0000 7f91b1ffb700 10 client.?.objecter resend_mon_ops 2021-05-25T12:06:15.358 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.317+0000 7f91b1ffb700 1 -- 172.21.15.102:0/3361034572 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2950417807 0 0) 0x7f91c4117f40 con 0x7f91c4090d70 2021-05-25T12:06:15.358 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.317+0000 7f91b1ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:15.358 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.317+0000 7f91b1ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:15.358 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.317+0000 7f91b1ffb700 10 monclient(hunting): my global_id is 184454 2021-05-25T12:06:15.359 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.317+0000 7f91b1ffb700 1 -- 172.21.15.102:0/3361034572 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91c4117a00 con 0x7f91c4090d70 2021-05-25T12:06:15.359 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.317+0000 7f91b1ffb700 1 -- 172.21.15.102:0/3361034572 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2286371749 0 0) 0x7f91c4117ca0 con 0x7f91c412e330 2021-05-25T12:06:15.359 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.317+0000 7f91b1ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:15.359 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.317+0000 7f91b1ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:15.359 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.317+0000 7f91b1ffb700 10 monclient(hunting): my global_id is 194278 2021-05-25T12:06:15.360 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.317+0000 7f91b1ffb700 1 -- 172.21.15.102:0/3361034572 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91c4117f40 con 0x7f91c412e330 2021-05-25T12:06:15.360 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.317+0000 7f91b1ffb700 1 -- 172.21.15.102:0/3361034572 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3187300121 0 0) 0x7f91a4004c10 con 0x7f91c4090990 2021-05-25T12:06:15.360 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.317+0000 7f91b1ffb700 1 -- 172.21.15.102:0/3361034572 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f91c4117ca0 con 0x7f91c4090990 2021-05-25T12:06:15.360 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.317+0000 7f91b1ffb700 1 -- 172.21.15.102:0/3361034572 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f91b40019d0 con 0x7f91c4090990 2021-05-25T12:06:15.361 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.317+0000 7f91b1ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:15.361 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.317+0000 7f91b1ffb700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:06:15.361 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.317+0000 7f91b1ffb700 10 monclient(hunting): dump: 2021-05-25T12:06:15.361 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:15.361 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:15.362 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:15.362 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:15.362 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:15.362 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:15.362 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:15.363 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:15.363 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:15.363 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:15.363 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.317+0000 7f91b1ffb700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:06:15.363 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.317+0000 7f91b1ffb700 1 -- 172.21.15.102:0/3361034572 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1067666092 0 0) 0x7f91c4117a00 con 0x7f91c4090d70 2021-05-25T12:06:15.364 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.318+0000 7f91b1ffb700 1 -- 172.21.15.102:0/3361034572 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f91a4004c10 con 0x7f91c4090d70 2021-05-25T12:06:15.364 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.318+0000 7f91b1ffb700 1 -- 172.21.15.102:0/3361034572 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f91b80018c0 con 0x7f91c4090d70 2021-05-25T12:06:15.364 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.318+0000 7f91b1ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:15.364 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.318+0000 7f91b1ffb700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2021-05-25T12:06:15.365 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.318+0000 7f91b1ffb700 10 monclient(hunting): dump: 2021-05-25T12:06:15.365 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:15.365 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:15.365 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:15.365 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:15.366 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:15.366 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:15.366 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:15.366 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:15.367 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:15.367 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:15.367 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.318+0000 7f91b1ffb700 1 -- 172.21.15.102:0/3361034572 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1018029076 0 0) 0x7f91c4117f40 con 0x7f91c412e330 2021-05-25T12:06:15.367 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.318+0000 7f91b1ffb700 1 -- 172.21.15.102:0/3361034572 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f91c4117a00 con 0x7f91c412e330 2021-05-25T12:06:15.367 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.318+0000 7f91b1ffb700 1 -- 172.21.15.102:0/3361034572 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2311838185 0 0) 0x7f91c4117ca0 con 0x7f91c4090990 2021-05-25T12:06:15.368 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.318+0000 7f91b1ffb700 1 -- 172.21.15.102:0/3361034572 >> v1:172.21.15.102:6790/0 conn(0x7f91c4090d70 legacy=0x7f91c41294f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:15.368 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.318+0000 7f91b1ffb700 1 -- 172.21.15.102:0/3361034572 >> v1:172.21.15.181:6789/0 conn(0x7f91c412e330 legacy=0x7f91c4142f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:15.368 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.318+0000 7f91b1ffb700 10 monclient: _finish_hunting 0 2021-05-25T12:06:15.368 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.318+0000 7f91b1ffb700 1 monclient: found mon.a 2021-05-25T12:06:15.369 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.318+0000 7f91b1ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:06:15.369 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.318+0000 7f91b1ffb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:06:15.369 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.318+0000 7f91b1ffb700 1 -- 172.21.15.102:0/3361034572 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91c413b460 con 0x7f91c4090990 2021-05-25T12:06:15.369 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.318+0000 7f91b1ffb700 10 monclient: _finish_auth 0 2021-05-25T12:06:15.369 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.318+0000 7f91b1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:06:15.370 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.318+0000 7f91cc17c700 5 monclient: authenticate success, global_id 204114 2021-05-25T12:06:15.370 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.318+0000 7f91cc17c700 10 monclient: _renew_subs 2021-05-25T12:06:15.370 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.318+0000 7f91cc17c700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:06:15.370 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.318+0000 7f91cc17c700 1 -- 172.21.15.102:0/3361034572 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91c4065960 con 0x7f91c4090990 2021-05-25T12:06:15.370 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.318+0000 7f91b1ffb700 1 -- 172.21.15.102:0/3361034572 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f91b40047e0 con 0x7f91c4090990 2021-05-25T12:06:15.371 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.318+0000 7f91b1ffb700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:06:15.371 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.318+0000 7f91b1ffb700 1 -- 172.21.15.102:0/3361034572 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f91b4004af0 con 0x7f91c4090990 2021-05-25T12:06:15.371 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.318+0000 7f91b1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:15.371 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.318+0000 7f91b1ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:06:15.372 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.318+0000 7f91b1ffb700 10 monclient: dump: 2021-05-25T12:06:15.372 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:15.372 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:15.372 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:15.372 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:15.373 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:15.373 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:15.373 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:15.373 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:15.373 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:15.374 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:15.374 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.318+0000 7f91cc17c700 10 client.204114.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:06:15.374 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.318+0000 7f91b3fff700 4 set_mon_vals no callback set 2021-05-25T12:06:15.374 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.318+0000 7f91cc17c700 10 monclient: _renew_subs 2021-05-25T12:06:15.374 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.318+0000 7f91cc17c700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:06:15.375 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.318+0000 7f91cc17c700 1 -- 172.21.15.102:0/3361034572 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f91c4129dc0 con 0x7f91c4090990 2021-05-25T12:06:15.375 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.318+0000 7f91cc17c700 1 librados: init done 2021-05-25T12:06:15.375 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.318+0000 7f91b1ffb700 1 -- 172.21.15.102:0/3361034572 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f91b400de10 con 0x7f91c4090990 2021-05-25T12:06:15.375 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.319+0000 7f91cc17c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:06:15.376 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.319+0000 7f91cc17c700 1 -- 172.21.15.102:0/3361034572 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f91c4059730 con 0x7f91c4065f10 2021-05-25T12:06:15.376 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.319+0000 7f91b1ffb700 1 -- 172.21.15.102:0/3361034572 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(193..193 src has 1..193) v4 ==== 6511+0+0 (unknown 3415019523 0 0) 0x7f91b403da80 con 0x7f91c4090990 2021-05-25T12:06:15.376 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.319+0000 7f91b1ffb700 10 client.204114.objecter ms_dispatch 0x7f91c4001550 osd_map(193..193 src has 1..193) v4 2021-05-25T12:06:15.376 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.319+0000 7f91b1ffb700 3 client.204114.objecter handle_osd_map got epochs [193,193] > 0 2021-05-25T12:06:15.376 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.319+0000 7f91b1ffb700 3 client.204114.objecter handle_osd_map decoding full epoch 193 2021-05-25T12:06:15.377 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.319+0000 7f91b1ffb700 20 client.204114.objecter dump_active .. 0 homeless 2021-05-25T12:06:15.377 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.319+0000 7f91b1ffb700 10 client.204114.objecter ms_handle_connect 0x7f91c4065f10 2021-05-25T12:06:15.377 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.319+0000 7f91b1ffb700 10 client.204114.objecter resend_mon_ops 2021-05-25T12:06:15.377 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.319+0000 7f91b1ffb700 1 -- 172.21.15.102:0/3361034572 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3284672040 0 0) 0x7f91c4059730 con 0x7f91c4065f10 2021-05-25T12:06:15.378 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.319+0000 7f91b1ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:15.378 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.319+0000 7f91b1ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:15.378 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.319+0000 7f91b1ffb700 10 monclient(hunting): my global_id is 184457 2021-05-25T12:06:15.378 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.319+0000 7f91b1ffb700 1 -- 172.21.15.102:0/3361034572 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91c4117ca0 con 0x7f91c4065f10 2021-05-25T12:06:15.378 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.319+0000 7f91ca719700 10 monclient: get_auth_request con 0x7f91a4033460 auth_method 0 2021-05-25T12:06:15.379 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.319+0000 7f91b1ffb700 1 -- 172.21.15.102:0/3361034572 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3697576895 0 0) 0x7f91c4117ca0 con 0x7f91c4065f10 2021-05-25T12:06:15.379 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.319+0000 7f91b1ffb700 1 -- 172.21.15.102:0/3361034572 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f91c4059730 con 0x7f91c4065f10 2021-05-25T12:06:15.379 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.319+0000 7f91b1ffb700 1 -- 172.21.15.102:0/3361034572 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f91b80021a0 con 0x7f91c4065f10 2021-05-25T12:06:15.379 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.319+0000 7f91b1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:15.379 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.319+0000 7f91b1ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:06:15.380 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.319+0000 7f91b1ffb700 10 monclient: dump: 2021-05-25T12:06:15.380 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:15.380 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:15.380 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:15.381 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:15.381 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:15.381 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:15.381 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:15.381 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:15.382 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:15.382 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:15.382 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.319+0000 7f91b1ffb700 10 client.204114.objecter ms_handle_connect 0x7f91a4033460 2021-05-25T12:06:15.382 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.319+0000 7f91b1ffb700 1 -- 172.21.15.102:0/3361034572 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1837405035 0 0) 0x7f91c4059730 con 0x7f91c4065f10 2021-05-25T12:06:15.382 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.319+0000 7f91b1ffb700 1 -- 172.21.15.102:0/3361034572 --> v1:172.21.15.102:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f91c40008d0 con 0x7f91c4065f10 2021-05-25T12:06:15.383 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.319+0000 7f91b1ffb700 1 -- 172.21.15.102:0/3361034572 <== mon.2 v1:172.21.15.102:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7f91b8002600 con 0x7f91c4065f10 2021-05-25T12:06:15.383 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.319+0000 7f91b1ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:06:15.383 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.319+0000 7f91b1ffb700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:06:15.383 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.319+0000 7f91b1ffb700 1 -- 172.21.15.102:0/3361034572 >> v1:172.21.15.102:6790/0 conn(0x7f91c4065f10 legacy=0x7f91c40662f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:15.384 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.319+0000 7f91b1ffb700 1 -- 172.21.15.102:0/3361034572 >> v1:172.21.15.102:6790/0 conn(0x7f91c4065f10 legacy=0x7f91c40662f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:06:15.384 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.326+0000 7f91cc17c700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:06:15.384 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.326+0000 7f91cc17c700 1 -- 172.21.15.102:0/3361034572 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f91c4066a00 con 0x7f91c4027ce0 2021-05-25T12:06:15.384 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.326+0000 7f91b1ffb700 10 client.204114.objecter ms_handle_connect 0x7f91c4027ce0 2021-05-25T12:06:15.384 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.326+0000 7f91b1ffb700 10 client.204114.objecter resend_mon_ops 2021-05-25T12:06:15.385 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.326+0000 7f91b1ffb700 1 -- 172.21.15.102:0/3361034572 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1825377283 0 0) 0x7f91c4066a00 con 0x7f91c4027ce0 2021-05-25T12:06:15.385 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.326+0000 7f91b1ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:15.385 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.326+0000 7f91b1ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:15.385 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.326+0000 7f91b1ffb700 10 monclient(hunting): my global_id is 184460 2021-05-25T12:06:15.386 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.326+0000 7f91b1ffb700 1 -- 172.21.15.102:0/3361034572 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91c4059730 con 0x7f91c4027ce0 2021-05-25T12:06:15.386 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.326+0000 7f91b1ffb700 1 -- 172.21.15.102:0/3361034572 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1542258213 0 0) 0x7f91c4059730 con 0x7f91c4027ce0 2021-05-25T12:06:15.386 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.326+0000 7f91b1ffb700 1 -- 172.21.15.102:0/3361034572 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f91c4066a00 con 0x7f91c4027ce0 2021-05-25T12:06:15.386 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.326+0000 7f91b1ffb700 1 -- 172.21.15.102:0/3361034572 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f91b8002ca0 con 0x7f91c4027ce0 2021-05-25T12:06:15.386 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.326+0000 7f91b1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:15.387 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.327+0000 7f91b1ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:06:15.387 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.327+0000 7f91b1ffb700 10 monclient: dump: 2021-05-25T12:06:15.387 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:15.387 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:15.387 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:15.388 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:15.388 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:15.388 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:15.388 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:15.389 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:15.389 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:15.389 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:15.389 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.327+0000 7f91b1ffb700 1 -- 172.21.15.102:0/3361034572 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 819125392 0 0) 0x7f91c4066a00 con 0x7f91c4027ce0 2021-05-25T12:06:15.389 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.327+0000 7f91b1ffb700 1 -- 172.21.15.102:0/3361034572 --> v1:172.21.15.102:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f91c4066b90 con 0x7f91c4027ce0 2021-05-25T12:06:15.390 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.327+0000 7f91b1ffb700 1 -- 172.21.15.102:0/3361034572 <== mon.2 v1:172.21.15.102:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4112 (unknown 0 0 2600402779) 0x7f91b80050e0 con 0x7f91c4027ce0 2021-05-25T12:06:15.390 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.327+0000 7f91b1ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:06:15.390 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.327+0000 7f91b1ffb700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:06:15.390 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.327+0000 7f91b1ffb700 1 -- 172.21.15.102:0/3361034572 >> v1:172.21.15.102:6790/0 conn(0x7f91c4027ce0 legacy=0x7f91c4148640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:15.390 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.327+0000 7f91b1ffb700 1 -- 172.21.15.102:0/3361034572 >> v1:172.21.15.102:6790/0 conn(0x7f91c4027ce0 legacy=0x7f91c4148640 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:06:15.391 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:06:15.391 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:06:15.391 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:06:15.392 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "leader", 2021-05-25T12:06:15.392 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 52, 2021-05-25T12:06:15.392 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:06:15.392 INFO:teuthology.orchestra.run.smithi102.stdout: 0, 2021-05-25T12:06:15.393 INFO:teuthology.orchestra.run.smithi102.stdout: 1, 2021-05-25T12:06:15.393 INFO:teuthology.orchestra.run.smithi102.stdout: 2 2021-05-25T12:06:15.393 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:06:15.393 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 0, 2021-05-25T12:06:15.393 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:06:15.394 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:06:15.394 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:06:15.394 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:06:15.394 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:06:15.395 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:06:15.395 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:06:15.395 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:06:15.395 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:06:15.395 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:06:15.396 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:06:15.396 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:06:15.396 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:06:15.396 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:06:15.396 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:06:15.397 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:06:15.397 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:06:15.397 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:06:15.397 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:06:15.398 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:06:15.398 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:06:15.398 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:06:15.398 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:06:15.398 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:06:15.399 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:15.399 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:15.399 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:06:15.399 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:06:15.400 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:06:15.400 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:06:15.400 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:06:15.400 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:06:15.400 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:06:15.401 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:06:15.401 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:06:15.401 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:06:15.401 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:06:15.401 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:06:15.402 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:06:15.402 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:06:15.402 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:06:15.402 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:06:15.402 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:06:15.403 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:06:15.403 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:06:15.403 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:06:15.403 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:06:15.404 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:06:15.404 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:06:15.404 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:06:15.404 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:06:15.404 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:06:15.405 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:15.405 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:06:15.405 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:15.405 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:06:15.405 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:06:15.406 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:06:15.406 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:06:15.406 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:15.406 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:06:15.407 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:06:15.407 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:06:15.407 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:15.407 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:15.407 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:15.408 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:06:15.408 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:06:15.408 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:06:15.408 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:06:15.408 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:06:15.409 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:15.409 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:15.409 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:06:15.409 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:06:15.409 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:06:15.410 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:06:15.410 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:15.410 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:06:15.410 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:06:15.411 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:06:15.411 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:15.411 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:15.411 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:15.411 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:06:15.412 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:06:15.412 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:06:15.412 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:06:15.412 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:06:15.413 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:15.413 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:15.413 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:06:15.413 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:06:15.413 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:06:15.414 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:06:15.414 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:15.414 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:06:15.414 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:06:15.415 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:06:15.415 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:15.415 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:15.415 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:15.415 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:06:15.416 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:06:15.416 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:06:15.416 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:06:15.416 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:06:15.416 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:15.417 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:15.417 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:15.417 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:06:15.417 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:06:15.417 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:15.418 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:06:15.418 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:06:15.418 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:06:15.418 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:15.419 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:06:15.419 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": [ 2021-05-25T12:06:15.419 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:15.419 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:06:15.419 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:06:15.420 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 2 2021-05-25T12:06:15.420 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:15.420 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:06:15.420 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:06:15.420 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:15.421 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:06:15.421 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:06:15.421 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 2 2021-05-25T12:06:15.421 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:15.421 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:06:15.422 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": [ 2021-05-25T12:06:15.422 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:15.422 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:06:15.422 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:06:15.423 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:06:15.423 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:15.423 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:15.423 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:15.423 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:06:15.424 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:06:15.426 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.327+0000 7f91cc17c700 10 librados: watch_flush enter 2021-05-25T12:06:15.426 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.327+0000 7f91cc17c700 10 librados: watch_flush exit 2021-05-25T12:06:15.426 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.327+0000 7f91cc17c700 20 client.204114.objecter shutdown clearing up homeless session... 2021-05-25T12:06:15.426 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.327+0000 7f91cc17c700 10 client.204114.objecter successfully canceled tick 2021-05-25T12:06:15.427 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.327+0000 7f91cc17c700 1 -- 172.21.15.102:0/3361034572 >> v1:172.21.15.181:6812/29582 conn(0x7f91a4033460 legacy=0x7f91a4035920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:15.427 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.327+0000 7f91cc17c700 10 monclient: shutdown 2021-05-25T12:06:15.427 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.327+0000 7f91cc17c700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:06:15.427 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.327+0000 7f91cc17c700 1 -- 172.21.15.102:0/3361034572 >> v1:172.21.15.102:6789/0 conn(0x7f91c4090990 legacy=0x7f91c413ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:15.427 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.327+0000 7f91ca719700 1 -- 172.21.15.102:0/3361034572 reap_dead start 2021-05-25T12:06:15.428 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.327+0000 7f91ca719700 1 -- 172.21.15.102:0/3361034572 reap_dead start 2021-05-25T12:06:15.428 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.327+0000 7f91cc17c700 1 -- 172.21.15.102:0/3361034572 shutdown_connections 2021-05-25T12:06:15.428 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.327+0000 7f91cc17c700 1 -- 172.21.15.102:0/3361034572 >> 172.21.15.102:0/3361034572 conn(0x7f91c409b370 msgr2=0x7f91c40973c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:06:15.428 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.327+0000 7f91cc17c700 1 -- 172.21.15.102:0/3361034572 shutdown_connections 2021-05-25T12:06:15.429 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.327+0000 7f91cc17c700 1 -- 172.21.15.102:0/3361034572 wait complete. 2021-05-25T12:06:15.429 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.327+0000 7f91cc17c700 1 librados: shutdown 2021-05-25T12:06:15.429 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2021-05-25T12:06:15.541 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.539+0000 7f061186b700 10 monclient: get_monmap_and_config 2021-05-25T12:06:15.541 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.539+0000 7f061186b700 10 monclient: build_initial_monmap 2021-05-25T12:06:15.541 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.539+0000 7f061186b700 10 monclient: monmap: 2021-05-25T12:06:15.541 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:06:15.542 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:15.542 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:06:15.540380+0000 2021-05-25T12:06:15.542 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:06:15.540380+0000 2021-05-25T12:06:15.542 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:06:15.543 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:15.543 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:06:15.543 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:06:15.543 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:06:15.543 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:15.544 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.539+0000 7f061186b700 1 Processor -- start 2021-05-25T12:06:15.544 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.540+0000 7f061186b700 1 -- start start 2021-05-25T12:06:15.544 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.540+0000 7f061186b700 10 monclient: init 2021-05-25T12:06:15.557 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.540+0000 7f061186b700 10 monclient: _reopen_session rank -1 2021-05-25T12:06:15.557 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.540+0000 7f061186b700 10 monclient: _add_conns ranks=[2,0,1] 2021-05-25T12:06:15.558 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.540+0000 7f061186b700 10 monclient(hunting): picked mon.noname-b con 0x7f060c093fc0 addr v1:172.21.15.102:6790/0 2021-05-25T12:06:15.558 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.540+0000 7f061186b700 10 monclient(hunting): picked mon.noname-a con 0x7f060c0692d0 addr v1:172.21.15.102:6789/0 2021-05-25T12:06:15.558 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.540+0000 7f061186b700 10 monclient(hunting): picked mon.noname-c con 0x7f060c12f130 addr v1:172.21.15.181:6789/0 2021-05-25T12:06:15.558 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.540+0000 7f061186b700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f060c097160 con 0x7f060c0692d0 2021-05-25T12:06:15.559 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.540+0000 7f061186b700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f060c092fe0 con 0x7f060c12f130 2021-05-25T12:06:15.559 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.540+0000 7f061186b700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f060c093280 con 0x7f060c093fc0 2021-05-25T12:06:15.559 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.540+0000 7f061186b700 10 monclient(hunting): _renew_subs 2021-05-25T12:06:15.559 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.540+0000 7f061186b700 10 monclient(hunting): authenticate will time out at 1176.137512s 2021-05-25T12:06:15.560 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.540+0000 7f060affd700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f060c093fc0 0x7f060c096340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:56324/0 (socket says 172.21.15.102:56324) 2021-05-25T12:06:15.560 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.540+0000 7f060a7fc700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f060c0692d0 0x7f060c096a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33216/0 (socket says 172.21.15.102:33216) 2021-05-25T12:06:15.560 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.540+0000 7f060affd700 1 -- 172.21.15.102:0/2026471557 learned_addr learned my addr 172.21.15.102:0/2026471557 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:06:15.560 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.540+0000 7f0609ffb700 1 -- 172.21.15.102:0/2026471557 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 333734663 0 0) 0x7f060c093280 con 0x7f060c093fc0 2021-05-25T12:06:15.560 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.540+0000 7f0609ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:15.561 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.540+0000 7f0609ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:15.561 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.540+0000 7f0609ffb700 10 monclient(hunting): my global_id is 184463 2021-05-25T12:06:15.561 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.540+0000 7f0609ffb700 1 -- 172.21.15.102:0/2026471557 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05f8003270 con 0x7f060c093fc0 2021-05-25T12:06:15.561 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.540+0000 7f0609ffb700 1 -- 172.21.15.102:0/2026471557 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1586823541 0 0) 0x7f060c097160 con 0x7f060c0692d0 2021-05-25T12:06:15.562 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.540+0000 7f0609ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:15.562 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.540+0000 7f0609ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:15.562 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.540+0000 7f0609ffb700 10 monclient(hunting): my global_id is 204117 2021-05-25T12:06:15.562 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.540+0000 7f0609ffb700 1 -- 172.21.15.102:0/2026471557 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f060c093280 con 0x7f060c0692d0 2021-05-25T12:06:15.562 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.541+0000 7f0609ffb700 1 -- 172.21.15.102:0/2026471557 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 44117683 0 0) 0x7f05f8003270 con 0x7f060c093fc0 2021-05-25T12:06:15.563 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.541+0000 7f0609ffb700 1 -- 172.21.15.102:0/2026471557 >> v1:172.21.15.181:6789/0 conn(0x7f060c12f130 legacy=0x7f060c131520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:15.563 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.541+0000 7f0609ffb700 1 -- 172.21.15.102:0/2026471557 >> v1:172.21.15.102:6789/0 conn(0x7f060c0692d0 legacy=0x7f060c096a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:15.563 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.541+0000 7f0609ffb700 10 monclient: _finish_hunting 0 2021-05-25T12:06:15.563 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.541+0000 7f0609ffb700 1 monclient: found mon.noname-b 2021-05-25T12:06:15.563 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.541+0000 7f0609ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:06:15.564 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.541+0000 7f0609ffb700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T12:06:15.564 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.541+0000 7f0609ffb700 1 -- 172.21.15.102:0/2026471557 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f060c093520 con 0x7f060c093fc0 2021-05-25T12:06:15.564 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.541+0000 7f0609ffb700 10 monclient: _finish_auth 0 2021-05-25T12:06:15.564 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.541+0000 7f0609ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:06:15.565 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.541+0000 7f061186b700 5 monclient: authenticate success, global_id 184463 2021-05-25T12:06:15.565 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.541+0000 7f0609ffb700 1 -- 172.21.15.102:0/2026471557 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f0600001610 con 0x7f060c093fc0 2021-05-25T12:06:15.565 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.541+0000 7f061186b700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:06:15.565 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.541+0000 7f0609ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:15.565 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.541+0000 7f0609ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:06:15.566 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.541+0000 7f0609ffb700 10 monclient: dump: 2021-05-25T12:06:15.566 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:15.566 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:15.566 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:15.566 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:15.567 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:15.567 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:15.567 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:15.567 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:15.568 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:15.568 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:15.568 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.541+0000 7f061186b700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:06:15.568 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.541+0000 7f0609ffb700 1 -- 172.21.15.102:0/2026471557 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f06000018b0 con 0x7f060c093fc0 2021-05-25T12:06:15.568 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.541+0000 7f0609ffb700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:06:15.569 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.541+0000 7f061186b700 10 monclient: get_monmap_and_config success 2021-05-25T12:06:15.569 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.541+0000 7f0609ffb700 1 -- 172.21.15.102:0/2026471557 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f0600001c40 con 0x7f060c093fc0 2021-05-25T12:06:15.569 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.541+0000 7f061186b700 4 set_mon_vals no callback set 2021-05-25T12:06:15.569 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.541+0000 7f0609ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:15.569 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.541+0000 7f0609ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:06:15.570 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.541+0000 7f0609ffb700 10 monclient: dump: 2021-05-25T12:06:15.570 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:15.570 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:15.570 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:15.570 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:15.571 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:15.571 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:15.571 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:15.571 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:15.572 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:15.572 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:15.572 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.541+0000 7f061186b700 10 monclient: shutdown 2021-05-25T12:06:15.572 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.541+0000 7f061186b700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:06:15.572 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.541+0000 7f061186b700 1 -- 172.21.15.102:0/2026471557 >> v1:172.21.15.102:6790/0 conn(0x7f060c093fc0 legacy=0x7f060c096340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:15.573 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.541+0000 7f061186b700 1 -- 172.21.15.102:0/2026471557 shutdown_connections 2021-05-25T12:06:15.573 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.541+0000 7f061186b700 1 -- 172.21.15.102:0/2026471557 >> 172.21.15.102:0/2026471557 conn(0x7f060c09b370 msgr2=0x7f060c097430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:06:15.573 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.541+0000 7f061186b700 1 -- 172.21.15.102:0/2026471557 shutdown_connections 2021-05-25T12:06:15.573 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.541+0000 7f061186b700 1 -- 172.21.15.102:0/2026471557 wait complete. 2021-05-25T12:06:15.574 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.542+0000 7f061186b700 10 monclient: build_initial_monmap 2021-05-25T12:06:15.574 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.542+0000 7f061186b700 10 monclient: monmap: 2021-05-25T12:06:15.574 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:06:15.574 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:06:15.574 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:06:15.575 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:06:15.575 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:06:15.575 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:06:15.575 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:06:15.575 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:06:15.576 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:06:15.576 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:15.576 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.542+0000 7f061186b700 1 librados: starting msgr at 2021-05-25T12:06:15.576 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.542+0000 7f061186b700 1 librados: starting objecter 2021-05-25T12:06:15.576 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.542+0000 7f061186b700 1 Processor -- start 2021-05-25T12:06:15.577 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.542+0000 7f061186b700 1 -- start start 2021-05-25T12:06:15.577 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.542+0000 7f061186b700 1 librados: setting wanted keys 2021-05-25T12:06:15.577 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.542+0000 7f061186b700 1 librados: calling monclient init 2021-05-25T12:06:15.577 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.542+0000 7f061186b700 10 monclient: init 2021-05-25T12:06:15.578 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.543+0000 7f061186b700 10 monclient: _reopen_session rank -1 2021-05-25T12:06:15.578 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.543+0000 7f061186b700 10 monclient: _add_conns ranks=[2,0,1] 2021-05-25T12:06:15.578 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.543+0000 7f061186b700 10 monclient(hunting): picked mon.noname-c con 0x7f060c0692d0 addr v1:172.21.15.102:6790/0 2021-05-25T12:06:15.578 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.543+0000 7f061186b700 10 monclient(hunting): picked mon.noname-a con 0x7f060c093fc0 addr v1:172.21.15.102:6789/0 2021-05-25T12:06:15.578 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.543+0000 7f061186b700 10 monclient(hunting): picked mon.noname-b con 0x7f060c12f130 addr v1:172.21.15.181:6789/0 2021-05-25T12:06:15.579 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.543+0000 7f061186b700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f060c112e30 con 0x7f060c093fc0 2021-05-25T12:06:15.579 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.543+0000 7f061186b700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f060c092350 con 0x7f060c12f130 2021-05-25T12:06:15.579 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.543+0000 7f061186b700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f060c12b2b0 con 0x7f060c0692d0 2021-05-25T12:06:15.579 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.543+0000 7f061186b700 10 monclient(hunting): _renew_subs 2021-05-25T12:06:15.579 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.543+0000 7f061186b700 10 monclient(hunting): authenticate will time out at 1176.140120s 2021-05-25T12:06:15.580 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.543+0000 7f060affd700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f060c0692d0 0x7f060c090990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:56330/0 (socket says 172.21.15.102:56330) 2021-05-25T12:06:15.580 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.543+0000 7f060affd700 1 -- 172.21.15.102:0/2507361401 learned_addr learned my addr 172.21.15.102:0/2507361401 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:06:15.580 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.543+0000 7f060a7fc700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f060c093fc0 0x7f060c0910a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33222/0 (socket says 172.21.15.102:33222) 2021-05-25T12:06:15.580 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.543+0000 7f05f2ffd700 10 client.?.objecter ms_handle_connect 0x7f060c0692d0 2021-05-25T12:06:15.581 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.543+0000 7f05f2ffd700 10 client.?.objecter resend_mon_ops 2021-05-25T12:06:15.581 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.543+0000 7f05f2ffd700 10 client.?.objecter ms_handle_connect 0x7f060c093fc0 2021-05-25T12:06:15.581 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.543+0000 7f05f2ffd700 10 client.?.objecter resend_mon_ops 2021-05-25T12:06:15.581 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.543+0000 7f05f2ffd700 1 -- 172.21.15.102:0/2507361401 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1222048706 0 0) 0x7f060c12b2b0 con 0x7f060c0692d0 2021-05-25T12:06:15.581 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.543+0000 7f05f2ffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:15.582 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.543+0000 7f05f2ffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:15.582 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.543+0000 7f05f2ffd700 10 monclient(hunting): my global_id is 184466 2021-05-25T12:06:15.582 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.543+0000 7f05f2ffd700 1 -- 172.21.15.102:0/2507361401 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05ec004c10 con 0x7f060c0692d0 2021-05-25T12:06:15.582 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.543+0000 7f05f2ffd700 10 client.?.objecter ms_handle_connect 0x7f060c12f130 2021-05-25T12:06:15.582 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.543+0000 7f05f2ffd700 10 client.?.objecter resend_mon_ops 2021-05-25T12:06:15.583 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.543+0000 7f05f2ffd700 1 -- 172.21.15.102:0/2507361401 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1037730835 0 0) 0x7f060c112e30 con 0x7f060c093fc0 2021-05-25T12:06:15.583 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.543+0000 7f05f2ffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:15.583 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.543+0000 7f05f2ffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:15.583 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.543+0000 7f05f2ffd700 10 monclient(hunting): my global_id is 204120 2021-05-25T12:06:15.584 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.543+0000 7f05f2ffd700 1 -- 172.21.15.102:0/2507361401 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f060c12b2b0 con 0x7f060c093fc0 2021-05-25T12:06:15.584 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.543+0000 7f05f2ffd700 1 -- 172.21.15.102:0/2507361401 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2055097066 0 0) 0x7f060c092350 con 0x7f060c12f130 2021-05-25T12:06:15.584 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.543+0000 7f05f2ffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:15.584 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.543+0000 7f05f2ffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:15.584 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.543+0000 7f05f2ffd700 10 monclient(hunting): my global_id is 194284 2021-05-25T12:06:15.585 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.543+0000 7f05f2ffd700 1 -- 172.21.15.102:0/2507361401 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f060c112e30 con 0x7f060c12f130 2021-05-25T12:06:15.585 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.543+0000 7f05f2ffd700 1 -- 172.21.15.102:0/2507361401 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2038377512 0 0) 0x7f060c12b2b0 con 0x7f060c093fc0 2021-05-25T12:06:15.585 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.543+0000 7f05f2ffd700 1 -- 172.21.15.102:0/2507361401 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f060c092350 con 0x7f060c093fc0 2021-05-25T12:06:15.585 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.543+0000 7f05f2ffd700 1 -- 172.21.15.102:0/2507361401 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f05f4001aa0 con 0x7f060c093fc0 2021-05-25T12:06:15.585 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.543+0000 7f05f2ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:15.586 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.543+0000 7f05f2ffd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:06:15.586 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.543+0000 7f05f2ffd700 10 monclient(hunting): dump: 2021-05-25T12:06:15.586 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:15.586 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:15.587 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:15.587 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:15.587 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:15.587 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:15.587 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:15.588 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:15.588 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:15.588 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:15.588 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.543+0000 7f05f2ffd700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:06:15.588 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.543+0000 7f05f2ffd700 1 -- 172.21.15.102:0/2507361401 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2732168584 0 0) 0x7f05ec004c10 con 0x7f060c0692d0 2021-05-25T12:06:15.589 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.543+0000 7f05f2ffd700 1 -- 172.21.15.102:0/2507361401 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f060c12b2b0 con 0x7f060c0692d0 2021-05-25T12:06:15.589 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.543+0000 7f05f2ffd700 1 -- 172.21.15.102:0/2507361401 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f0600001bc0 con 0x7f060c0692d0 2021-05-25T12:06:15.589 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.543+0000 7f05f2ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:15.589 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.543+0000 7f05f2ffd700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2021-05-25T12:06:15.589 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.543+0000 7f05f2ffd700 10 monclient(hunting): dump: 2021-05-25T12:06:15.590 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:15.590 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:15.590 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:15.590 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:15.591 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:15.591 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:15.591 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:15.591 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:15.591 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:15.592 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:15.592 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.543+0000 7f05f2ffd700 1 -- 172.21.15.102:0/2507361401 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3361469413 0 0) 0x7f060c112e30 con 0x7f060c12f130 2021-05-25T12:06:15.592 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.543+0000 7f05f2ffd700 1 -- 172.21.15.102:0/2507361401 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f05ec004c10 con 0x7f060c12f130 2021-05-25T12:06:15.592 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.543+0000 7f05f2ffd700 1 -- 172.21.15.102:0/2507361401 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3273580287 0 0) 0x7f060c092350 con 0x7f060c093fc0 2021-05-25T12:06:15.592 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.543+0000 7f05f2ffd700 1 -- 172.21.15.102:0/2507361401 >> v1:172.21.15.102:6790/0 conn(0x7f060c0692d0 legacy=0x7f060c090990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:15.593 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.543+0000 7f05f2ffd700 1 -- 172.21.15.102:0/2507361401 >> v1:172.21.15.181:6789/0 conn(0x7f060c12f130 legacy=0x7f060c143620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:15.593 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.543+0000 7f05f2ffd700 10 monclient: _finish_hunting 0 2021-05-25T12:06:15.593 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.543+0000 7f05f2ffd700 1 monclient: found mon.a 2021-05-25T12:06:15.593 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.543+0000 7f05f2ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:06:15.594 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.543+0000 7f05f2ffd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:06:15.594 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.543+0000 7f05f2ffd700 1 -- 172.21.15.102:0/2507361401 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f060c129f10 con 0x7f060c093fc0 2021-05-25T12:06:15.594 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.543+0000 7f05f2ffd700 10 monclient: _finish_auth 0 2021-05-25T12:06:15.594 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.543+0000 7f05f2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:06:15.594 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.543+0000 7f061186b700 5 monclient: authenticate success, global_id 204120 2021-05-25T12:06:15.595 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.543+0000 7f061186b700 10 monclient: _renew_subs 2021-05-25T12:06:15.595 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.543+0000 7f061186b700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:06:15.595 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.543+0000 7f061186b700 1 -- 172.21.15.102:0/2507361401 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f060c112790 con 0x7f060c093fc0 2021-05-25T12:06:15.595 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.543+0000 7f061186b700 10 client.204120.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:06:15.596 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.543+0000 7f061186b700 10 monclient: _renew_subs 2021-05-25T12:06:15.596 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.543+0000 7f061186b700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:06:15.596 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.543+0000 7f061186b700 1 -- 172.21.15.102:0/2507361401 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f060c117ae0 con 0x7f060c093fc0 2021-05-25T12:06:15.596 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.543+0000 7f061186b700 1 librados: init done 2021-05-25T12:06:15.596 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.543+0000 7f05f2ffd700 1 -- 172.21.15.102:0/2507361401 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f05f4001c80 con 0x7f060c093fc0 2021-05-25T12:06:15.597 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.544+0000 7f05f2ffd700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:06:15.597 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.544+0000 7f05f2ffd700 1 -- 172.21.15.102:0/2507361401 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f05f4004980 con 0x7f060c093fc0 2021-05-25T12:06:15.597 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.544+0000 7f05f2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:15.597 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.544+0000 7f05f2ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:06:15.597 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.544+0000 7f06097fa700 4 set_mon_vals no callback set 2021-05-25T12:06:15.598 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.544+0000 7f05f2ffd700 10 monclient: dump: 2021-05-25T12:06:15.598 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:15.598 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:15.598 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:15.599 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:15.599 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:15.599 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:15.599 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:15.599 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:15.600 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:15.600 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:15.600 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.544+0000 7f05f2ffd700 1 -- 172.21.15.102:0/2507361401 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f05f400dca0 con 0x7f060c093fc0 2021-05-25T12:06:15.600 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.544+0000 7f05f2ffd700 1 -- 172.21.15.102:0/2507361401 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(193..193 src has 1..193) v4 ==== 6511+0+0 (unknown 3415019523 0 0) 0x7f05f403d7c0 con 0x7f060c093fc0 2021-05-25T12:06:15.600 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.544+0000 7f061186b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:06:15.601 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.544+0000 7f05f2ffd700 10 client.204120.objecter ms_dispatch 0x7f060c001550 osd_map(193..193 src has 1..193) v4 2021-05-25T12:06:15.601 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.544+0000 7f05f2ffd700 3 client.204120.objecter handle_osd_map got epochs [193,193] > 0 2021-05-25T12:06:15.601 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.544+0000 7f05f2ffd700 3 client.204120.objecter handle_osd_map decoding full epoch 193 2021-05-25T12:06:15.601 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.544+0000 7f061186b700 1 -- 172.21.15.102:0/2507361401 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f05e40065a0 con 0x7f05e4002840 2021-05-25T12:06:15.601 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.544+0000 7f05f2ffd700 20 client.204120.objecter dump_active .. 0 homeless 2021-05-25T12:06:15.602 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.544+0000 7f060affd700 10 monclient: get_auth_request con 0x7f05ec0334e0 auth_method 0 2021-05-25T12:06:15.602 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.545+0000 7f05f2ffd700 10 client.204120.objecter ms_handle_connect 0x7f05e4002840 2021-05-25T12:06:15.602 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.545+0000 7f05f2ffd700 10 client.204120.objecter resend_mon_ops 2021-05-25T12:06:15.602 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.545+0000 7f05f2ffd700 10 client.204120.objecter ms_handle_connect 0x7f05ec0334e0 2021-05-25T12:06:15.603 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.545+0000 7f05f2ffd700 1 -- 172.21.15.102:0/2507361401 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1036426209 0 0) 0x7f05e40065a0 con 0x7f05e4002840 2021-05-25T12:06:15.603 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.545+0000 7f05f2ffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:15.603 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.545+0000 7f05f2ffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:15.603 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.545+0000 7f05f2ffd700 10 monclient(hunting): my global_id is 194287 2021-05-25T12:06:15.603 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.545+0000 7f05f2ffd700 1 -- 172.21.15.102:0/2507361401 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f060c092350 con 0x7f05e4002840 2021-05-25T12:06:15.604 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.545+0000 7f05f2ffd700 1 -- 172.21.15.102:0/2507361401 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3193900206 0 0) 0x7f060c092350 con 0x7f05e4002840 2021-05-25T12:06:15.604 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.545+0000 7f05f2ffd700 1 -- 172.21.15.102:0/2507361401 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f05e40065a0 con 0x7f05e4002840 2021-05-25T12:06:15.604 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.545+0000 7f05f2ffd700 1 -- 172.21.15.102:0/2507361401 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f05fc002200 con 0x7f05e4002840 2021-05-25T12:06:15.604 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.545+0000 7f05f2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:15.604 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.545+0000 7f05f2ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T12:06:15.605 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.545+0000 7f05f2ffd700 10 monclient: dump: 2021-05-25T12:06:15.605 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:15.605 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:15.605 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:15.606 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:15.606 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:15.606 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:15.606 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:15.606 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:15.607 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:15.607 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:15.607 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.545+0000 7f05f2ffd700 1 -- 172.21.15.102:0/2507361401 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2977110903 0 0) 0x7f05e40065a0 con 0x7f05e4002840 2021-05-25T12:06:15.607 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.545+0000 7f05f2ffd700 1 -- 172.21.15.102:0/2507361401 --> v1:172.21.15.181:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f05e4009e40 con 0x7f05e4002840 2021-05-25T12:06:15.607 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.546+0000 7f05f2ffd700 1 -- 172.21.15.102:0/2507361401 <== mon.1 v1:172.21.15.181:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7f05fc004550 con 0x7f05e4002840 2021-05-25T12:06:15.608 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.546+0000 7f05f2ffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:06:15.608 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.546+0000 7f05f2ffd700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:06:15.608 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.546+0000 7f05f2ffd700 1 -- 172.21.15.102:0/2507361401 >> v1:172.21.15.181:6789/0 conn(0x7f05e4002840 legacy=0x7f05e4004ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:15.608 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.546+0000 7f05f2ffd700 1 -- 172.21.15.102:0/2507361401 >> v1:172.21.15.181:6789/0 conn(0x7f05e4002840 legacy=0x7f05e4004ce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:06:15.609 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.552+0000 7f061186b700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:06:15.609 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.552+0000 7f061186b700 1 -- 172.21.15.102:0/2507361401 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f05e4007540 con 0x7f05e40068e0 2021-05-25T12:06:15.609 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.552+0000 7f05f2ffd700 10 client.204120.objecter ms_handle_connect 0x7f05e40068e0 2021-05-25T12:06:15.609 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.552+0000 7f05f2ffd700 10 client.204120.objecter resend_mon_ops 2021-05-25T12:06:15.609 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.553+0000 7f05f2ffd700 1 -- 172.21.15.102:0/2507361401 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2190604582 0 0) 0x7f05e4007540 con 0x7f05e40068e0 2021-05-25T12:06:15.610 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.553+0000 7f05f2ffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:15.610 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.553+0000 7f05f2ffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:15.610 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.553+0000 7f05f2ffd700 10 monclient(hunting): my global_id is 194290 2021-05-25T12:06:15.610 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.553+0000 7f05f2ffd700 1 -- 172.21.15.102:0/2507361401 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05e40065a0 con 0x7f05e40068e0 2021-05-25T12:06:15.611 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.553+0000 7f05f2ffd700 1 -- 172.21.15.102:0/2507361401 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 925466223 0 0) 0x7f05e40065a0 con 0x7f05e40068e0 2021-05-25T12:06:15.611 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.553+0000 7f05f2ffd700 1 -- 172.21.15.102:0/2507361401 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f05e4007540 con 0x7f05e40068e0 2021-05-25T12:06:15.611 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.553+0000 7f05f2ffd700 1 -- 172.21.15.102:0/2507361401 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f05fc002a30 con 0x7f05e40068e0 2021-05-25T12:06:15.611 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.553+0000 7f05f2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:15.611 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.553+0000 7f05f2ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T12:06:15.612 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.553+0000 7f05f2ffd700 10 monclient: dump: 2021-05-25T12:06:15.612 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:15.612 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:15.612 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:15.612 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:15.613 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:15.613 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:15.613 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:15.613 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:15.614 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:15.614 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:15.614 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.553+0000 7f05f2ffd700 1 -- 172.21.15.102:0/2507361401 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1161622473 0 0) 0x7f05e4007540 con 0x7f05e40068e0 2021-05-25T12:06:15.614 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.553+0000 7f05f2ffd700 1 -- 172.21.15.102:0/2507361401 --> v1:172.21.15.181:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f05e4008770 con 0x7f05e40068e0 2021-05-25T12:06:15.614 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.553+0000 7f05f2ffd700 1 -- 172.21.15.102:0/2507361401 <== mon.1 v1:172.21.15.181:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3940 (unknown 0 0 1217976465) 0x7f05fc003e30 con 0x7f05e40068e0 2021-05-25T12:06:15.615 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.553+0000 7f05f2ffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:06:15.615 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.553+0000 7f05f2ffd700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:06:15.615 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.553+0000 7f05f2ffd700 1 -- 172.21.15.102:0/2507361401 >> v1:172.21.15.181:6789/0 conn(0x7f05e40068e0 legacy=0x7f05e4006d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:15.615 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.553+0000 7f05f2ffd700 1 -- 172.21.15.102:0/2507361401 >> v1:172.21.15.181:6789/0 conn(0x7f05e40068e0 legacy=0x7f05e4006d70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:06:15.616 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:06:15.616 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:06:15.616 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:06:15.616 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "peon", 2021-05-25T12:06:15.617 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 52, 2021-05-25T12:06:15.617 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:06:15.617 INFO:teuthology.orchestra.run.smithi102.stdout: 0, 2021-05-25T12:06:15.617 INFO:teuthology.orchestra.run.smithi102.stdout: 1, 2021-05-25T12:06:15.618 INFO:teuthology.orchestra.run.smithi102.stdout: 2 2021-05-25T12:06:15.618 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:06:15.618 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 0, 2021-05-25T12:06:15.618 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:06:15.618 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:06:15.619 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:06:15.619 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:06:15.619 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:06:15.619 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:06:15.619 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:06:15.620 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:06:15.620 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:06:15.620 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:06:15.620 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:06:15.620 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:06:15.621 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:06:15.621 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:06:15.621 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:06:15.621 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:06:15.622 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:06:15.622 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:06:15.622 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:06:15.622 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:06:15.622 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:06:15.623 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:06:15.623 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:06:15.623 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:06:15.623 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:15.623 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:15.624 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:06:15.624 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:06:15.624 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:06:15.624 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:06:15.625 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:06:15.625 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:06:15.625 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:06:15.625 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:06:15.625 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:06:15.626 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:06:15.626 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:06:15.626 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:06:15.626 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:06:15.626 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:06:15.627 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:06:15.627 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:06:15.627 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:06:15.627 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:06:15.628 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:06:15.628 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:06:15.628 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:06:15.628 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:06:15.628 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:06:15.629 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:06:15.629 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:06:15.629 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:06:15.629 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:15.629 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:06:15.630 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:15.630 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:06:15.630 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:06:15.630 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:06:15.630 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:06:15.631 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:15.631 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:06:15.631 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:06:15.631 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:06:15.631 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:15.632 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:15.632 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:15.632 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:06:15.632 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:06:15.632 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:06:15.632 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:06:15.633 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:06:15.633 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:15.633 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:15.633 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:06:15.633 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:06:15.634 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:06:15.634 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:06:15.634 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:15.634 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:06:15.634 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:06:15.635 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:06:15.635 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:15.635 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:15.635 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:15.635 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:06:15.636 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:06:15.636 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:06:15.636 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:06:15.636 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:06:15.636 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:15.637 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:15.637 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:06:15.637 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:06:15.637 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:06:15.637 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:06:15.638 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:15.638 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:06:15.638 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:06:15.638 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:06:15.638 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:15.639 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:15.639 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:15.639 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:06:15.639 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:06:15.639 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:06:15.640 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:06:15.640 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:06:15.640 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:15.640 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:15.640 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:15.640 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:06:15.641 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:06:15.641 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:15.641 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:06:15.641 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:06:15.641 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:06:15.642 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:15.642 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:06:15.642 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": [ 2021-05-25T12:06:15.642 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:15.642 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:06:15.643 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:06:15.643 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 4 2021-05-25T12:06:15.643 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:15.643 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:06:15.643 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:06:15.644 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:15.644 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:06:15.644 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:06:15.644 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 6 2021-05-25T12:06:15.645 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:15.645 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:15.645 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:15.645 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:06:15.645 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:06:15.647 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.554+0000 7f061186b700 10 librados: watch_flush enter 2021-05-25T12:06:15.648 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.554+0000 7f061186b700 10 librados: watch_flush exit 2021-05-25T12:06:15.648 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.554+0000 7f061186b700 20 client.204120.objecter shutdown clearing up homeless session... 2021-05-25T12:06:15.648 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.554+0000 7f061186b700 10 client.204120.objecter successfully canceled tick 2021-05-25T12:06:15.648 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.554+0000 7f061186b700 1 -- 172.21.15.102:0/2507361401 >> v1:172.21.15.181:6812/29582 conn(0x7f05ec0334e0 legacy=0x7f05ec035980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:15.649 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.554+0000 7f061186b700 10 monclient: shutdown 2021-05-25T12:06:15.649 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.554+0000 7f061186b700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:06:15.649 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.554+0000 7f061186b700 1 -- 172.21.15.102:0/2507361401 >> v1:172.21.15.102:6789/0 conn(0x7f060c093fc0 legacy=0x7f060c0910a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:15.649 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.554+0000 7f060b7fe700 1 -- 172.21.15.102:0/2507361401 reap_dead start 2021-05-25T12:06:15.649 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.554+0000 7f061186b700 1 -- 172.21.15.102:0/2507361401 shutdown_connections 2021-05-25T12:06:15.650 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.554+0000 7f061186b700 1 -- 172.21.15.102:0/2507361401 >> 172.21.15.102:0/2507361401 conn(0x7f060c09b370 msgr2=0x7f060c068500 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:06:15.650 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.554+0000 7f061186b700 1 -- 172.21.15.102:0/2507361401 shutdown_connections 2021-05-25T12:06:15.650 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.554+0000 7f061186b700 1 -- 172.21.15.102:0/2507361401 wait complete. 2021-05-25T12:06:15.650 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.554+0000 7f061186b700 1 librados: shutdown 2021-05-25T12:06:15.651 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2021-05-25T12:06:15.651 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2021-05-25T12:06:15.760 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.758+0000 7f3ed7b15700 10 monclient: get_monmap_and_config 2021-05-25T12:06:15.760 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.758+0000 7f3ed7b15700 10 monclient: build_initial_monmap 2021-05-25T12:06:15.760 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.759+0000 7f3ed7b15700 10 monclient: monmap: 2021-05-25T12:06:15.761 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:06:15.761 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:15.761 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:06:15.759884+0000 2021-05-25T12:06:15.761 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:06:15.759884+0000 2021-05-25T12:06:15.761 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:06:15.762 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:15.762 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:06:15.762 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:06:15.762 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:06:15.763 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:15.772 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.759+0000 7f3ed7b15700 1 Processor -- start 2021-05-25T12:06:15.772 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.759+0000 7f3ed7b15700 1 -- start start 2021-05-25T12:06:15.773 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.759+0000 7f3ed7b15700 10 monclient: init 2021-05-25T12:06:15.773 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.760+0000 7f3ed7b15700 10 monclient: _reopen_session rank -1 2021-05-25T12:06:15.773 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.760+0000 7f3ed7b15700 10 monclient: _add_conns ranks=[1,0,2] 2021-05-25T12:06:15.773 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.760+0000 7f3ed7b15700 10 monclient(hunting): picked mon.noname-c con 0x7f3ed0063840 addr v1:172.21.15.181:6789/0 2021-05-25T12:06:15.773 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.760+0000 7f3ed7b15700 10 monclient(hunting): picked mon.noname-a con 0x7f3ed0094da0 addr v1:172.21.15.102:6789/0 2021-05-25T12:06:15.774 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.760+0000 7f3ed7b15700 10 monclient(hunting): picked mon.noname-b con 0x7f3ed01323c0 addr v1:172.21.15.102:6790/0 2021-05-25T12:06:15.774 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.760+0000 7f3ed7b15700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ed0063fb0 con 0x7f3ed0094da0 2021-05-25T12:06:15.774 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.760+0000 7f3ed7b15700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ed00982f0 con 0x7f3ed0063840 2021-05-25T12:06:15.774 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.760+0000 7f3ed7b15700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ed0064510 con 0x7f3ed01323c0 2021-05-25T12:06:15.775 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.760+0000 7f3ed7b15700 10 monclient(hunting): _renew_subs 2021-05-25T12:06:15.775 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.760+0000 7f3ed7b15700 10 monclient(hunting): authenticate will time out at 1176.357168s 2021-05-25T12:06:15.775 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.760+0000 7f3ed60b2700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f3ed01323c0 0x7f3ed01347b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:56346/0 (socket says 172.21.15.102:56346) 2021-05-25T12:06:15.775 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.760+0000 7f3ed50b0700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f3ed0094da0 0x7f3ed0097be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33232/0 (socket says 172.21.15.102:33232) 2021-05-25T12:06:15.775 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.760+0000 7f3ed60b2700 1 -- 172.21.15.102:0/3157694629 learned_addr learned my addr 172.21.15.102:0/3157694629 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:06:15.776 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.760+0000 7f3ed48af700 1 -- 172.21.15.102:0/3157694629 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 992393157 0 0) 0x7f3ed0064510 con 0x7f3ed01323c0 2021-05-25T12:06:15.776 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.760+0000 7f3ed48af700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:15.776 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.760+0000 7f3ed48af700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:15.776 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.760+0000 7f3ed48af700 10 monclient(hunting): my global_id is 184469 2021-05-25T12:06:15.777 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.760+0000 7f3ed48af700 1 -- 172.21.15.102:0/3157694629 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3eb8003270 con 0x7f3ed01323c0 2021-05-25T12:06:15.777 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.760+0000 7f3ed48af700 1 -- 172.21.15.102:0/3157694629 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3015221168 0 0) 0x7f3ed0063fb0 con 0x7f3ed0094da0 2021-05-25T12:06:15.777 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.760+0000 7f3ed48af700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:15.777 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.760+0000 7f3ed48af700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:15.777 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.760+0000 7f3ed48af700 10 monclient(hunting): my global_id is 204123 2021-05-25T12:06:15.778 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.760+0000 7f3ed48af700 1 -- 172.21.15.102:0/3157694629 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ed0064510 con 0x7f3ed0094da0 2021-05-25T12:06:15.778 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.760+0000 7f3ed48af700 1 -- 172.21.15.102:0/3157694629 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1068579440 0 0) 0x7f3ed00982f0 con 0x7f3ed0063840 2021-05-25T12:06:15.778 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.760+0000 7f3ed48af700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:15.778 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.760+0000 7f3ed48af700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:15.779 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.760+0000 7f3ed48af700 10 monclient(hunting): my global_id is 194293 2021-05-25T12:06:15.779 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.760+0000 7f3ed48af700 1 -- 172.21.15.102:0/3157694629 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ed0063fb0 con 0x7f3ed0063840 2021-05-25T12:06:15.779 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.760+0000 7f3ed48af700 1 -- 172.21.15.102:0/3157694629 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2064302100 0 0) 0x7f3eb8003270 con 0x7f3ed01323c0 2021-05-25T12:06:15.779 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.760+0000 7f3ed48af700 1 -- 172.21.15.102:0/3157694629 >> v1:172.21.15.181:6789/0 conn(0x7f3ed0063840 legacy=0x7f3ed00974d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:15.779 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.760+0000 7f3ed48af700 1 -- 172.21.15.102:0/3157694629 >> v1:172.21.15.102:6789/0 conn(0x7f3ed0094da0 legacy=0x7f3ed0097be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:15.780 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.760+0000 7f3ed48af700 10 monclient: _finish_hunting 0 2021-05-25T12:06:15.780 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.760+0000 7f3ed48af700 1 monclient: found mon.noname-b 2021-05-25T12:06:15.780 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.760+0000 7f3ed48af700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:06:15.780 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.760+0000 7f3ed48af700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T12:06:15.781 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.760+0000 7f3ed48af700 1 -- 172.21.15.102:0/3157694629 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ed00647b0 con 0x7f3ed01323c0 2021-05-25T12:06:15.781 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.760+0000 7f3ed48af700 10 monclient: _finish_auth 0 2021-05-25T12:06:15.781 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.760+0000 7f3ed48af700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:06:15.781 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.760+0000 7f3ed48af700 1 -- 172.21.15.102:0/3157694629 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f3ecc001990 con 0x7f3ed01323c0 2021-05-25T12:06:15.781 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.760+0000 7f3ed7b15700 5 monclient: authenticate success, global_id 184469 2021-05-25T12:06:15.782 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.760+0000 7f3ed7b15700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:06:15.782 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.760+0000 7f3ed48af700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:15.782 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.760+0000 7f3ed48af700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:06:15.782 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.760+0000 7f3ed48af700 10 monclient: dump: 2021-05-25T12:06:15.782 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:15.783 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:15.783 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:15.783 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:15.783 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:15.784 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:15.784 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:15.784 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:15.784 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:15.784 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:15.785 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.760+0000 7f3ed7b15700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:06:15.785 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.760+0000 7f3ed48af700 1 -- 172.21.15.102:0/3157694629 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3ecc001c00 con 0x7f3ed01323c0 2021-05-25T12:06:15.785 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.760+0000 7f3ed48af700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:06:15.785 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.760+0000 7f3ed7b15700 10 monclient: get_monmap_and_config success 2021-05-25T12:06:15.786 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.760+0000 7f3ed48af700 1 -- 172.21.15.102:0/3157694629 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f3ecc001f90 con 0x7f3ed01323c0 2021-05-25T12:06:15.786 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.760+0000 7f3ed7b15700 4 set_mon_vals no callback set 2021-05-25T12:06:15.786 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.760+0000 7f3ed48af700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:15.786 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.760+0000 7f3ed48af700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:06:15.786 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.760+0000 7f3ed48af700 10 monclient: dump: 2021-05-25T12:06:15.787 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:15.787 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:15.787 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:15.787 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:15.788 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:15.788 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:15.788 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:15.788 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:15.788 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:15.789 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:15.789 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.761+0000 7f3ed7b15700 10 monclient: shutdown 2021-05-25T12:06:15.789 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.761+0000 7f3ed7b15700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:06:15.789 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.761+0000 7f3ed7b15700 1 -- 172.21.15.102:0/3157694629 >> v1:172.21.15.102:6790/0 conn(0x7f3ed01323c0 legacy=0x7f3ed01347b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:15.789 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.761+0000 7f3ed7b15700 1 -- 172.21.15.102:0/3157694629 shutdown_connections 2021-05-25T12:06:15.790 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.761+0000 7f3ed7b15700 1 -- 172.21.15.102:0/3157694629 >> 172.21.15.102:0/3157694629 conn(0x7f3ed009cb30 msgr2=0x7f3ed009cf40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:06:15.790 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.761+0000 7f3ed7b15700 1 -- 172.21.15.102:0/3157694629 shutdown_connections 2021-05-25T12:06:15.790 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.761+0000 7f3ed7b15700 1 -- 172.21.15.102:0/3157694629 wait complete. 2021-05-25T12:06:15.790 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.761+0000 7f3ed7b15700 10 monclient: build_initial_monmap 2021-05-25T12:06:15.791 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.762+0000 7f3ed7b15700 10 monclient: monmap: 2021-05-25T12:06:15.791 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:06:15.791 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:06:15.791 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:06:15.791 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:06:15.792 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:06:15.792 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:06:15.792 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:06:15.792 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:06:15.793 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:06:15.793 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:15.793 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.762+0000 7f3ed7b15700 1 librados: starting msgr at 2021-05-25T12:06:15.793 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.762+0000 7f3ed7b15700 1 librados: starting objecter 2021-05-25T12:06:15.793 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.762+0000 7f3ed7b15700 1 Processor -- start 2021-05-25T12:06:15.794 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.762+0000 7f3ed7b15700 1 -- start start 2021-05-25T12:06:15.794 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.762+0000 7f3ed7b15700 1 librados: setting wanted keys 2021-05-25T12:06:15.794 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.762+0000 7f3ed7b15700 1 librados: calling monclient init 2021-05-25T12:06:15.794 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.762+0000 7f3ed7b15700 10 monclient: init 2021-05-25T12:06:15.795 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.762+0000 7f3ed7b15700 10 monclient: _reopen_session rank -1 2021-05-25T12:06:15.795 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.762+0000 7f3ed7b15700 10 monclient: _add_conns ranks=[1,0,2] 2021-05-25T12:06:15.795 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.762+0000 7f3ed7b15700 10 monclient(hunting): picked mon.noname-b con 0x7f3ed0063840 addr v1:172.21.15.181:6789/0 2021-05-25T12:06:15.795 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.762+0000 7f3ed7b15700 10 monclient(hunting): picked mon.noname-a con 0x7f3ed0094da0 addr v1:172.21.15.102:6789/0 2021-05-25T12:06:15.795 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.762+0000 7f3ed7b15700 10 monclient(hunting): picked mon.noname-c con 0x7f3ed01323c0 addr v1:172.21.15.102:6790/0 2021-05-25T12:06:15.796 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.762+0000 7f3ed7b15700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ed011c080 con 0x7f3ed0094da0 2021-05-25T12:06:15.796 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.762+0000 7f3ed7b15700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ed011c320 con 0x7f3ed0063840 2021-05-25T12:06:15.796 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.762+0000 7f3ed7b15700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ed011c5c0 con 0x7f3ed01323c0 2021-05-25T12:06:15.796 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.762+0000 7f3ed7b15700 10 monclient(hunting): _renew_subs 2021-05-25T12:06:15.796 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.762+0000 7f3ed7b15700 10 monclient(hunting): authenticate will time out at 1176.359841s 2021-05-25T12:06:15.797 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.762+0000 7f3ed50b0700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f3ed0094da0 0x7f3ed0119000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33240/0 (socket says 172.21.15.102:33240) 2021-05-25T12:06:15.797 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.762+0000 7f3ed50b0700 1 -- 172.21.15.102:0/2345193985 learned_addr learned my addr 172.21.15.102:0/2345193985 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:06:15.797 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.763+0000 7f3ebdffb700 10 client.?.objecter ms_handle_connect 0x7f3ed0094da0 2021-05-25T12:06:15.797 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.763+0000 7f3ebdffb700 10 client.?.objecter resend_mon_ops 2021-05-25T12:06:15.797 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.763+0000 7f3ebdffb700 10 client.?.objecter ms_handle_connect 0x7f3ed01323c0 2021-05-25T12:06:15.798 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.763+0000 7f3ebdffb700 10 client.?.objecter resend_mon_ops 2021-05-25T12:06:15.798 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.763+0000 7f3ebdffb700 1 -- 172.21.15.102:0/2345193985 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 863329692 0 0) 0x7f3ed011c080 con 0x7f3ed0094da0 2021-05-25T12:06:15.798 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.763+0000 7f3ebdffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:15.798 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.763+0000 7f3ebdffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:15.798 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.763+0000 7f3ebdffb700 10 monclient(hunting): my global_id is 204126 2021-05-25T12:06:15.799 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.763+0000 7f3ebdffb700 1 -- 172.21.15.102:0/2345193985 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3eb0004c10 con 0x7f3ed0094da0 2021-05-25T12:06:15.799 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.763+0000 7f3ebdffb700 1 -- 172.21.15.102:0/2345193985 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1716375543 0 0) 0x7f3ed011c5c0 con 0x7f3ed01323c0 2021-05-25T12:06:15.799 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.763+0000 7f3ebdffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:15.799 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.763+0000 7f3ebdffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:15.799 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.763+0000 7f3ebdffb700 10 monclient(hunting): my global_id is 184472 2021-05-25T12:06:15.800 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.763+0000 7f3ebdffb700 1 -- 172.21.15.102:0/2345193985 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ed011c080 con 0x7f3ed01323c0 2021-05-25T12:06:15.800 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.763+0000 7f3ebdffb700 10 client.?.objecter ms_handle_connect 0x7f3ed0063840 2021-05-25T12:06:15.800 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.763+0000 7f3ebdffb700 10 client.?.objecter resend_mon_ops 2021-05-25T12:06:15.800 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.763+0000 7f3ebdffb700 1 -- 172.21.15.102:0/2345193985 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3778253942 0 0) 0x7f3ed011c320 con 0x7f3ed0063840 2021-05-25T12:06:15.800 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.763+0000 7f3ebdffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:15.801 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.763+0000 7f3ebdffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:15.801 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.763+0000 7f3ebdffb700 10 monclient(hunting): my global_id is 194296 2021-05-25T12:06:15.801 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.763+0000 7f3ebdffb700 1 -- 172.21.15.102:0/2345193985 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ed011c5c0 con 0x7f3ed0063840 2021-05-25T12:06:15.801 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.763+0000 7f3ebdffb700 1 -- 172.21.15.102:0/2345193985 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1946263004 0 0) 0x7f3eb0004c10 con 0x7f3ed0094da0 2021-05-25T12:06:15.801 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.763+0000 7f3ebdffb700 1 -- 172.21.15.102:0/2345193985 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3ed011c320 con 0x7f3ed0094da0 2021-05-25T12:06:15.802 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.763+0000 7f3ebdffb700 1 -- 172.21.15.102:0/2345193985 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f3ec00019d0 con 0x7f3ed0094da0 2021-05-25T12:06:15.802 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.763+0000 7f3ebdffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:15.802 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.763+0000 7f3ebdffb700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:06:15.802 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.763+0000 7f3ebdffb700 10 monclient(hunting): dump: 2021-05-25T12:06:15.803 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:15.803 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:15.803 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:15.803 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:15.803 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:15.804 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:15.804 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:15.804 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:15.804 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:15.804 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:15.805 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.763+0000 7f3ebdffb700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:06:15.805 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.763+0000 7f3ebdffb700 1 -- 172.21.15.102:0/2345193985 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 27166545 0 0) 0x7f3ed011c080 con 0x7f3ed01323c0 2021-05-25T12:06:15.805 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.763+0000 7f3ebdffb700 1 -- 172.21.15.102:0/2345193985 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3eb0004c10 con 0x7f3ed01323c0 2021-05-25T12:06:15.805 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.763+0000 7f3ebdffb700 1 -- 172.21.15.102:0/2345193985 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f3ecc001950 con 0x7f3ed01323c0 2021-05-25T12:06:15.805 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.763+0000 7f3ebdffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:15.806 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.763+0000 7f3ebdffb700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2021-05-25T12:06:15.806 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.763+0000 7f3ebdffb700 10 monclient(hunting): dump: 2021-05-25T12:06:15.806 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:15.806 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:15.807 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:15.807 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:15.807 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:15.807 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:15.807 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:15.807 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:15.808 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:15.808 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:15.808 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.763+0000 7f3ebdffb700 1 -- 172.21.15.102:0/2345193985 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2393461923 0 0) 0x7f3ed011c320 con 0x7f3ed0094da0 2021-05-25T12:06:15.808 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.763+0000 7f3ebdffb700 1 -- 172.21.15.102:0/2345193985 >> v1:172.21.15.102:6790/0 conn(0x7f3ed01323c0 legacy=0x7f3ed0143880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:15.808 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.763+0000 7f3ebdffb700 1 -- 172.21.15.102:0/2345193985 >> v1:172.21.15.181:6789/0 conn(0x7f3ed0063840 legacy=0x7f3ed01188f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:15.809 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.763+0000 7f3ebdffb700 10 monclient: _finish_hunting 0 2021-05-25T12:06:15.809 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.763+0000 7f3ebdffb700 1 monclient: found mon.a 2021-05-25T12:06:15.809 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.763+0000 7f3ebdffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:06:15.809 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.763+0000 7f3ebdffb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:06:15.809 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.763+0000 7f3ebdffb700 1 -- 172.21.15.102:0/2345193985 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ed0131250 con 0x7f3ed0094da0 2021-05-25T12:06:15.810 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.763+0000 7f3ebdffb700 10 monclient: _finish_auth 0 2021-05-25T12:06:15.810 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.763+0000 7f3ebdffb700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:06:15.810 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.763+0000 7f3ed7b15700 5 monclient: authenticate success, global_id 204126 2021-05-25T12:06:15.810 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.763+0000 7f3ed7b15700 10 monclient: _renew_subs 2021-05-25T12:06:15.810 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.763+0000 7f3ed7b15700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:06:15.811 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.763+0000 7f3ed7b15700 1 -- 172.21.15.102:0/2345193985 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ed011c9e0 con 0x7f3ed0094da0 2021-05-25T12:06:15.811 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.763+0000 7f3ebdffb700 1 -- 172.21.15.102:0/2345193985 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3ec00047e0 con 0x7f3ed0094da0 2021-05-25T12:06:15.811 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.763+0000 7f3ebdffb700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:06:15.811 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.763+0000 7f3ebdffb700 1 -- 172.21.15.102:0/2345193985 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f3ec0004af0 con 0x7f3ed0094da0 2021-05-25T12:06:15.812 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.763+0000 7f3ebdffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:15.812 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.763+0000 7f3ebdffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:06:15.812 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.763+0000 7f3ebdffb700 10 monclient: dump: 2021-05-25T12:06:15.812 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:15.812 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:15.812 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:15.813 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:15.813 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:15.813 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:15.813 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:15.814 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:15.814 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:15.814 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:15.814 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.763+0000 7f3ed7b15700 10 client.204126.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:06:15.814 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.763+0000 7f3ebffff700 4 set_mon_vals no callback set 2021-05-25T12:06:15.815 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.763+0000 7f3ed7b15700 10 monclient: _renew_subs 2021-05-25T12:06:15.815 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.763+0000 7f3ed7b15700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:06:15.815 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.763+0000 7f3ed7b15700 1 -- 172.21.15.102:0/2345193985 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ed0131f90 con 0x7f3ed0094da0 2021-05-25T12:06:15.815 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.763+0000 7f3ed7b15700 1 librados: init done 2021-05-25T12:06:15.815 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.764+0000 7f3ebdffb700 1 -- 172.21.15.102:0/2345193985 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f3ec000de10 con 0x7f3ed0094da0 2021-05-25T12:06:15.816 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.764+0000 7f3ebdffb700 1 -- 172.21.15.102:0/2345193985 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(193..193 src has 1..193) v4 ==== 6511+0+0 (unknown 3415019523 0 0) 0x7f3ec003db20 con 0x7f3ed0094da0 2021-05-25T12:06:15.816 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.764+0000 7f3ebdffb700 10 client.204126.objecter ms_dispatch 0x7f3ed0001550 osd_map(193..193 src has 1..193) v4 2021-05-25T12:06:15.816 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.764+0000 7f3ebdffb700 3 client.204126.objecter handle_osd_map got epochs [193,193] > 0 2021-05-25T12:06:15.816 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.764+0000 7f3ebdffb700 3 client.204126.objecter handle_osd_map decoding full epoch 193 2021-05-25T12:06:15.816 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.764+0000 7f3ebdffb700 20 client.204126.objecter dump_active .. 0 homeless 2021-05-25T12:06:15.817 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.764+0000 7f3ed7b15700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:06:15.817 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.764+0000 7f3ed7b15700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:06:15.817 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.764+0000 7f3ed7b15700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:06:15.817 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.764+0000 7f3ed7b15700 1 -- 172.21.15.102:0/2345193985 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ed0059730 con 0x7f3ed0094da0 2021-05-25T12:06:15.817 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.764+0000 7f3ed58b1700 10 monclient: get_auth_request con 0x7f3eb00331b0 auth_method 0 2021-05-25T12:06:15.818 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.764+0000 7f3ebdffb700 10 client.204126.objecter ms_handle_connect 0x7f3eb00331b0 2021-05-25T12:06:15.818 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.768+0000 7f3ebdffb700 1 -- 172.21.15.102:0/2345193985 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7f3ec0012f30 con 0x7f3ed0094da0 2021-05-25T12:06:15.818 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.768+0000 7f3ebdffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:06:15.818 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.768+0000 7f3ebdffb700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:06:15.986 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.985+0000 7f3ed7b15700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2021-05-25T12:06:15.987 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.985+0000 7f3ed7b15700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2021-05-25T12:06:15.987 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.985+0000 7f3ed7b15700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:06:15.987 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.985+0000 7f3ed7b15700 1 -- 172.21.15.102:0/2345193985 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f3ed006c9a0 con 0x7f3ed0094da0 2021-05-25T12:06:15.988 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.986+0000 7f3ebdffb700 1 -- 172.21.15.102:0/2345193985 <== mon.0 v1:172.21.15.102:6789/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7f3ec0012b20 con 0x7f3ed0094da0 2021-05-25T12:06:15.988 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.986+0000 7f3ebdffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2021-05-25T12:06:15.988 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.986+0000 7f3ebdffb700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:06:15.989 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:15.987+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:06:15.990 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:15.987+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=32,config=2,health=12,kv=6,logm=48} crc {auth=972265555,config=3140514009,health=1387961980,kv=1184736763,logm=3602939887}) 2021-05-25T12:06:15.990 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:15.987+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=31,config=2,health=12,kv=6,logm=49} crc {auth=2198052783,config=3140514009,health=3607396046,kv=1184736763,logm=3447992972}) 2021-05-25T12:06:15.990 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:15.987+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:06:15.990 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:15.987+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=32,config=2,health=12,kv=6,logm=48} crc {auth=972265555,config=3140514009,health=1387961980,kv=1184736763,logm=3602939887}) 2021-05-25T12:06:15.991 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:15.987+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=29,config=2,health=10,kv=6,logm=53} crc {auth=1365704578,config=3140514009,health=1650836530,kv=1184736763,logm=2747892120}) 2021-05-25T12:06:15.991 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.988+0000 7f3ed7b15700 10 librados: watch_flush enter 2021-05-25T12:06:15.992 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:15.988+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:06:15.992 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:15.988+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3379688349}) 2021-05-25T12:06:15.992 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:15.988+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=765162814}) 2021-05-25T12:06:15.993 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.988+0000 7f3ed7b15700 10 librados: watch_flush exit 2021-05-25T12:06:15.993 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:15.988+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:06:15.994 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.988+0000 7f3ed7b15700 20 client.204126.objecter shutdown clearing up homeless session... 2021-05-25T12:06:15.994 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.988+0000 7f3ed7b15700 10 client.204126.objecter successfully canceled tick 2021-05-25T12:06:15.995 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:15.988+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3379688349}) 2021-05-25T12:06:15.995 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:15.988+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1071065715}) 2021-05-25T12:06:15.996 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.988+0000 7f3ed7b15700 1 -- 172.21.15.102:0/2345193985 >> v1:172.21.15.181:6812/29582 conn(0x7f3eb00331b0 legacy=0x7f3eb0035670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:15.996 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.988+0000 7f3ed7b15700 10 monclient: shutdown 2021-05-25T12:06:15.997 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.988+0000 7f3ed7b15700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:06:15.997 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.988+0000 7f3ed7b15700 1 -- 172.21.15.102:0/2345193985 >> v1:172.21.15.102:6789/0 conn(0x7f3ed0094da0 legacy=0x7f3ed0119000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:15.997 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.989+0000 7f3ed7b15700 1 -- 172.21.15.102:0/2345193985 shutdown_connections 2021-05-25T12:06:15.997 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.989+0000 7f3ed7b15700 1 -- 172.21.15.102:0/2345193985 >> 172.21.15.102:0/2345193985 conn(0x7f3ed009cb30 msgr2=0x7f3ed0068a30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:06:15.997 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.989+0000 7f3ed7b15700 1 -- 172.21.15.102:0/2345193985 shutdown_connections 2021-05-25T12:06:15.998 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.989+0000 7f3ed7b15700 1 -- 172.21.15.102:0/2345193985 wait complete. 2021-05-25T12:06:15.998 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:15.989+0000 7f3ed7b15700 1 librados: shutdown 2021-05-25T12:06:16.001 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:15.990+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:06:16.002 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:15.990+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=865191849}) 2021-05-25T12:06:16.002 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:15.990+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2005295756}) 2021-05-25T12:06:16.002 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:15.990+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:06:16.002 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:15.991+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=865191849}) 2021-05-25T12:06:16.003 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:15.991+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4106177761}) 2021-05-25T12:06:16.003 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:15.994+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:06:16.003 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:15.994+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1734663624}) 2021-05-25T12:06:16.003 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:15.994+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3711651926}) 2021-05-25T12:06:16.004 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:15.994+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:06:16.004 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:15.994+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1734663624}) 2021-05-25T12:06:16.004 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:15.994+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3375012782}) 2021-05-25T12:06:16.005 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:15.995+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:06:16.005 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:15.995+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=22,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=3,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=26,osdmap=18} crc {logm=3353874714,mdsmap=1531151339,mgr=2996579194,mgr_command_descs=3824783157,mgr_metadata=1267308364,mgrstat=1060138711,mon_config_key=1824720183,monmap=3809614708,osd_metadata=725478305,osd_pg_creating=4146971715,osd_snap=1895124561,osdmap=532519532}) 2021-05-25T12:06:16.005 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:15.995+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=22,mdsmap=3,mgr=5,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=4,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=25,osdmap=20} crc {logm=1294938432,mdsmap=1531151339,mgr=146218881,mgr_command_descs=3824783157,mgr_metadata=1267308364,mgrstat=4196642328,mon_config_key=4118394475,monmap=287184522,osd_metadata=725478305,osd_pg_creating=4146971715,osd_snap=1625517018,osdmap=36576226}) 2021-05-25T12:06:16.005 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:15.996+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:06:16.005 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:15.996+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=22,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=3,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=26,osdmap=18} crc {logm=3353874714,mdsmap=1531151339,mgr=2996579194,mgr_command_descs=3824783157,mgr_metadata=1267308364,mgrstat=1060138711,mon_config_key=1824720183,monmap=3809614708,osd_metadata=725478305,osd_pg_creating=4146971715,osd_snap=1895124561,osdmap=532519532}) 2021-05-25T12:06:16.006 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:15.996+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=23,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=4,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=25,osdmap=16} crc {logm=2713154728,mdsmap=1531151339,mgr=2996579194,mgr_command_descs=3824783157,mgr_metadata=1267308364,mgrstat=1215404072,mon_config_key=4118394475,monmap=3809614708,osd_metadata=725478305,osd_pg_creating=4146971715,osd_snap=42547571,osdmap=4264445155}) 2021-05-25T12:06:16.006 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:15.997+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:06:16.006 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:15.997+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=834421715}) 2021-05-25T12:06:16.007 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:15.997+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3251929014}) 2021-05-25T12:06:16.007 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:15.997+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:06:16.007 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:15.997+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=834421715}) 2021-05-25T12:06:16.007 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:15.997+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=34764303}) 2021-05-25T12:06:16.007 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:15.998+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:06:16.008 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:15.998+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2231425499}) 2021-05-25T12:06:16.008 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:15.998+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=659936720}) 2021-05-25T12:06:16.008 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:15.998+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:06:16.008 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:15.998+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2231425499}) 2021-05-25T12:06:16.008 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:15.998+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3024512275}) 2021-05-25T12:06:16.010 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2021-05-25T12:06:16.011 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:16.007+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:06:16.011 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:16.007+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=72} crc {osdmap=4083697940}) 2021-05-25T12:06:16.011 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:16.007+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=73} crc {osdmap=96173287}) 2021-05-25T12:06:16.011 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:16.007+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:06:16.012 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:16.007+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=72} crc {osdmap=4083697940}) 2021-05-25T12:06:16.012 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:16.007+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=71} crc {osdmap=158088934}) 2021-05-25T12:06:17.011 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2021-05-25T12:06:17.012 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2021-05-25T12:06:17.128 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.127+0000 7fa155d3e700 10 monclient: get_monmap_and_config 2021-05-25T12:06:17.129 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.127+0000 7fa155d3e700 10 monclient: build_initial_monmap 2021-05-25T12:06:17.129 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.127+0000 7fa155d3e700 10 monclient: monmap: 2021-05-25T12:06:17.129 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:06:17.129 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:17.130 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:06:17.128418+0000 2021-05-25T12:06:17.130 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:06:17.128418+0000 2021-05-25T12:06:17.130 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:06:17.130 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:17.131 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:06:17.131 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:06:17.131 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:06:17.131 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:17.142 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.128+0000 7fa155d3e700 1 Processor -- start 2021-05-25T12:06:17.142 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.128+0000 7fa155d3e700 1 -- start start 2021-05-25T12:06:17.142 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.128+0000 7fa155d3e700 10 monclient: init 2021-05-25T12:06:17.143 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.128+0000 7fa155d3e700 10 monclient: _reopen_session rank -1 2021-05-25T12:06:17.143 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.128+0000 7fa155d3e700 10 monclient: _add_conns ranks=[2,1,0] 2021-05-25T12:06:17.143 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.128+0000 7fa155d3e700 10 monclient(hunting): picked mon.noname-b con 0x7fa150092550 addr v1:172.21.15.102:6790/0 2021-05-25T12:06:17.143 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.128+0000 7fa155d3e700 10 monclient(hunting): picked mon.noname-c con 0x7fa15012b100 addr v1:172.21.15.181:6789/0 2021-05-25T12:06:17.143 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.128+0000 7fa155d3e700 10 monclient(hunting): picked mon.noname-a con 0x7fa15012ec10 addr v1:172.21.15.102:6789/0 2021-05-25T12:06:17.144 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.128+0000 7fa155d3e700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa150092930 con 0x7fa15012ec10 2021-05-25T12:06:17.144 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.128+0000 7fa155d3e700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1500931b0 con 0x7fa15012b100 2021-05-25T12:06:17.144 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.128+0000 7fa155d3e700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa150093450 con 0x7fa150092550 2021-05-25T12:06:17.144 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.128+0000 7fa155d3e700 10 monclient(hunting): _renew_subs 2021-05-25T12:06:17.145 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.128+0000 7fa155d3e700 10 monclient(hunting): authenticate will time out at 1177.725774s 2021-05-25T12:06:17.145 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.128+0000 7fa14f7fe700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fa150092550 0x7fa150093d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:56358/0 (socket says 172.21.15.102:56358) 2021-05-25T12:06:17.145 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.128+0000 7fa14f7fe700 1 -- 172.21.15.102:0/3304007715 learned_addr learned my addr 172.21.15.102:0/3304007715 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:06:17.145 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.128+0000 7fa14ffff700 1 --1- 172.21.15.102:0/3304007715 >> v1:172.21.15.102:6789/0 conn(0x7fa15012ec10 0x7fa150131000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33252/0 (socket says 172.21.15.102:33252) 2021-05-25T12:06:17.145 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.129+0000 7fa14e7fc700 1 -- 172.21.15.102:0/3304007715 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 228714437 0 0) 0x7fa150092930 con 0x7fa15012ec10 2021-05-25T12:06:17.146 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.129+0000 7fa14e7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:17.146 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.129+0000 7fa14e7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:17.146 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.129+0000 7fa14e7fc700 10 monclient(hunting): my global_id is 204129 2021-05-25T12:06:17.146 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.129+0000 7fa14e7fc700 1 -- 172.21.15.102:0/3304007715 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa138003270 con 0x7fa15012ec10 2021-05-25T12:06:17.147 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.129+0000 7fa14e7fc700 1 -- 172.21.15.102:0/3304007715 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3997676461 0 0) 0x7fa150093450 con 0x7fa150092550 2021-05-25T12:06:17.147 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.129+0000 7fa14e7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:17.147 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.129+0000 7fa14e7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:17.147 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.129+0000 7fa14e7fc700 10 monclient(hunting): my global_id is 184475 2021-05-25T12:06:17.147 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.129+0000 7fa14e7fc700 1 -- 172.21.15.102:0/3304007715 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa150092930 con 0x7fa150092550 2021-05-25T12:06:17.148 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.129+0000 7fa14e7fc700 1 -- 172.21.15.102:0/3304007715 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1420926613 0 0) 0x7fa138003270 con 0x7fa15012ec10 2021-05-25T12:06:17.148 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.129+0000 7fa14e7fc700 1 -- 172.21.15.102:0/3304007715 >> v1:172.21.15.102:6790/0 conn(0x7fa150092550 legacy=0x7fa150093d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:17.148 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.129+0000 7fa14e7fc700 1 -- 172.21.15.102:0/3304007715 >> v1:172.21.15.181:6789/0 conn(0x7fa15012b100 legacy=0x7fa15012d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:17.148 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.129+0000 7fa14e7fc700 10 monclient: _finish_hunting 0 2021-05-25T12:06:17.148 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.129+0000 7fa14e7fc700 1 monclient: found mon.noname-a 2021-05-25T12:06:17.149 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.129+0000 7fa14e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:06:17.149 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.129+0000 7fa14e7fc700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T12:06:17.149 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.129+0000 7fa14e7fc700 1 -- 172.21.15.102:0/3304007715 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1500936f0 con 0x7fa15012ec10 2021-05-25T12:06:17.149 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.129+0000 7fa14e7fc700 10 monclient: _finish_auth 0 2021-05-25T12:06:17.149 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.129+0000 7fa14e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:06:17.150 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.129+0000 7fa14e7fc700 1 -- 172.21.15.102:0/3304007715 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fa13c001610 con 0x7fa15012ec10 2021-05-25T12:06:17.150 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.129+0000 7fa155d3e700 5 monclient: authenticate success, global_id 204129 2021-05-25T12:06:17.150 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.129+0000 7fa155d3e700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:06:17.150 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.129+0000 7fa14e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:17.151 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.129+0000 7fa14e7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:06:17.151 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.129+0000 7fa14e7fc700 10 monclient: dump: 2021-05-25T12:06:17.151 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:17.151 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:17.151 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:17.152 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:17.152 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:17.152 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:17.152 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:17.153 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:17.153 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:17.153 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:17.153 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.129+0000 7fa155d3e700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:06:17.153 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.129+0000 7fa14e7fc700 1 -- 172.21.15.102:0/3304007715 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa13c001840 con 0x7fa15012ec10 2021-05-25T12:06:17.154 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.129+0000 7fa14e7fc700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:06:17.154 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.129+0000 7fa14e7fc700 1 -- 172.21.15.102:0/3304007715 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fa13c001bd0 con 0x7fa15012ec10 2021-05-25T12:06:17.154 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.129+0000 7fa155d3e700 10 monclient: get_monmap_and_config success 2021-05-25T12:06:17.154 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.129+0000 7fa155d3e700 4 set_mon_vals no callback set 2021-05-25T12:06:17.155 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.129+0000 7fa14e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:17.155 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.129+0000 7fa14e7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:06:17.155 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.129+0000 7fa14e7fc700 10 monclient: dump: 2021-05-25T12:06:17.155 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:17.155 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:17.156 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:17.156 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:17.156 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:17.156 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:17.157 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:17.157 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:17.157 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:17.157 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:17.157 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.129+0000 7fa155d3e700 10 monclient: shutdown 2021-05-25T12:06:17.158 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.129+0000 7fa155d3e700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:06:17.158 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.129+0000 7fa155d3e700 1 -- 172.21.15.102:0/3304007715 >> v1:172.21.15.102:6789/0 conn(0x7fa15012ec10 legacy=0x7fa150131000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:17.158 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.129+0000 7fa155d3e700 1 -- 172.21.15.102:0/3304007715 shutdown_connections 2021-05-25T12:06:17.158 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.129+0000 7fa155d3e700 1 -- 172.21.15.102:0/3304007715 >> 172.21.15.102:0/3304007715 conn(0x7fa15009b370 msgr2=0x7fa150097430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:06:17.158 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.130+0000 7fa155d3e700 1 -- 172.21.15.102:0/3304007715 shutdown_connections 2021-05-25T12:06:17.159 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.130+0000 7fa155d3e700 1 -- 172.21.15.102:0/3304007715 wait complete. 2021-05-25T12:06:17.159 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.130+0000 7fa155d3e700 10 monclient: build_initial_monmap 2021-05-25T12:06:17.159 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.130+0000 7fa155d3e700 10 monclient: monmap: 2021-05-25T12:06:17.159 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:06:17.160 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:06:17.160 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:06:17.160 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:06:17.160 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:06:17.160 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:06:17.161 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:06:17.161 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:06:17.161 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:06:17.161 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:17.162 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.130+0000 7fa155d3e700 1 librados: starting msgr at 2021-05-25T12:06:17.162 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.130+0000 7fa155d3e700 1 librados: starting objecter 2021-05-25T12:06:17.162 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.130+0000 7fa155d3e700 1 Processor -- start 2021-05-25T12:06:17.162 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.131+0000 7fa155d3e700 1 -- start start 2021-05-25T12:06:17.162 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.131+0000 7fa155d3e700 1 librados: setting wanted keys 2021-05-25T12:06:17.163 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.131+0000 7fa155d3e700 1 librados: calling monclient init 2021-05-25T12:06:17.163 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.131+0000 7fa155d3e700 10 monclient: init 2021-05-25T12:06:17.163 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.131+0000 7fa155d3e700 10 monclient: _reopen_session rank -1 2021-05-25T12:06:17.163 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.131+0000 7fa155d3e700 10 monclient: _add_conns ranks=[1,2,0] 2021-05-25T12:06:17.164 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.131+0000 7fa155d3e700 10 monclient(hunting): picked mon.noname-b con 0x7fa150092550 addr v1:172.21.15.181:6789/0 2021-05-25T12:06:17.164 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.131+0000 7fa155d3e700 10 monclient(hunting): picked mon.noname-c con 0x7fa15012b100 addr v1:172.21.15.102:6790/0 2021-05-25T12:06:17.164 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.131+0000 7fa155d3e700 10 monclient(hunting): picked mon.noname-a con 0x7fa15012ec10 addr v1:172.21.15.102:6789/0 2021-05-25T12:06:17.164 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.131+0000 7fa155d3e700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa150118270 con 0x7fa15012ec10 2021-05-25T12:06:17.164 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.131+0000 7fa155d3e700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa150118510 con 0x7fa150092550 2021-05-25T12:06:17.165 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.131+0000 7fa155d3e700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1501187b0 con 0x7fa15012b100 2021-05-25T12:06:17.165 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.131+0000 7fa155d3e700 10 monclient(hunting): _renew_subs 2021-05-25T12:06:17.165 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.131+0000 7fa155d3e700 10 monclient(hunting): authenticate will time out at 1177.728441s 2021-05-25T12:06:17.165 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.131+0000 7fa14effd700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fa15012b100 0x7fa1501403e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:56366/0 (socket says 172.21.15.102:56366) 2021-05-25T12:06:17.165 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.131+0000 7fa14effd700 1 -- 172.21.15.102:0/1489002217 learned_addr learned my addr 172.21.15.102:0/1489002217 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:06:17.166 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.131+0000 7fa1437fe700 10 client.?.objecter ms_handle_connect 0x7fa15012ec10 2021-05-25T12:06:17.166 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.131+0000 7fa1437fe700 10 client.?.objecter resend_mon_ops 2021-05-25T12:06:17.166 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.131+0000 7fa1437fe700 10 client.?.objecter ms_handle_connect 0x7fa15012b100 2021-05-25T12:06:17.166 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.131+0000 7fa1437fe700 10 client.?.objecter resend_mon_ops 2021-05-25T12:06:17.167 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.131+0000 7fa1437fe700 10 client.?.objecter ms_handle_connect 0x7fa150092550 2021-05-25T12:06:17.167 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.131+0000 7fa1437fe700 10 client.?.objecter resend_mon_ops 2021-05-25T12:06:17.167 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.131+0000 7fa1437fe700 1 -- 172.21.15.102:0/1489002217 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3258926442 0 0) 0x7fa150118270 con 0x7fa15012ec10 2021-05-25T12:06:17.167 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.131+0000 7fa1437fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:17.167 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.131+0000 7fa1437fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:17.168 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.131+0000 7fa1437fe700 10 monclient(hunting): my global_id is 204132 2021-05-25T12:06:17.168 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.131+0000 7fa1437fe700 1 -- 172.21.15.102:0/1489002217 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa12c004c10 con 0x7fa15012ec10 2021-05-25T12:06:17.168 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.131+0000 7fa1437fe700 1 -- 172.21.15.102:0/1489002217 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4084497648 0 0) 0x7fa1501187b0 con 0x7fa15012b100 2021-05-25T12:06:17.168 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.131+0000 7fa1437fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:17.168 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.131+0000 7fa1437fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:17.169 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.131+0000 7fa1437fe700 10 monclient(hunting): my global_id is 184478 2021-05-25T12:06:17.169 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.131+0000 7fa1437fe700 1 -- 172.21.15.102:0/1489002217 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa150118270 con 0x7fa15012b100 2021-05-25T12:06:17.169 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.131+0000 7fa1437fe700 1 -- 172.21.15.102:0/1489002217 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2972899401 0 0) 0x7fa150118510 con 0x7fa150092550 2021-05-25T12:06:17.169 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.131+0000 7fa1437fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:17.170 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.131+0000 7fa1437fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:17.170 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.131+0000 7fa1437fe700 10 monclient(hunting): my global_id is 194302 2021-05-25T12:06:17.170 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.131+0000 7fa1437fe700 1 -- 172.21.15.102:0/1489002217 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1501187b0 con 0x7fa150092550 2021-05-25T12:06:17.170 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.131+0000 7fa1437fe700 1 -- 172.21.15.102:0/1489002217 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4250043331 0 0) 0x7fa12c004c10 con 0x7fa15012ec10 2021-05-25T12:06:17.170 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.131+0000 7fa1437fe700 1 -- 172.21.15.102:0/1489002217 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa150118510 con 0x7fa15012ec10 2021-05-25T12:06:17.171 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.132+0000 7fa1437fe700 1 -- 172.21.15.102:0/1489002217 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fa13c001b50 con 0x7fa15012ec10 2021-05-25T12:06:17.171 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.132+0000 7fa1437fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:17.171 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.132+0000 7fa1437fe700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:06:17.171 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.132+0000 7fa1437fe700 10 monclient(hunting): dump: 2021-05-25T12:06:17.171 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:17.172 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:17.172 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:17.172 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:17.172 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:17.173 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:17.173 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:17.173 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:17.173 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:17.173 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:17.174 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.132+0000 7fa1437fe700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:06:17.174 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.132+0000 7fa1437fe700 1 -- 172.21.15.102:0/1489002217 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4139589664 0 0) 0x7fa150118270 con 0x7fa15012b100 2021-05-25T12:06:17.174 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.132+0000 7fa1437fe700 1 -- 172.21.15.102:0/1489002217 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa12c004c10 con 0x7fa15012b100 2021-05-25T12:06:17.174 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.132+0000 7fa1437fe700 1 -- 172.21.15.102:0/1489002217 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fa134001940 con 0x7fa15012b100 2021-05-25T12:06:17.174 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.132+0000 7fa1437fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:17.175 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.132+0000 7fa1437fe700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2021-05-25T12:06:17.175 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.132+0000 7fa1437fe700 10 monclient(hunting): dump: 2021-05-25T12:06:17.175 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:17.175 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:17.176 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:17.176 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:17.176 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:17.176 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:17.176 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:17.177 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:17.177 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:17.177 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:17.177 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.132+0000 7fa1437fe700 1 -- 172.21.15.102:0/1489002217 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1560831385 0 0) 0x7fa1501187b0 con 0x7fa150092550 2021-05-25T12:06:17.177 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.132+0000 7fa1437fe700 1 -- 172.21.15.102:0/1489002217 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa150118270 con 0x7fa150092550 2021-05-25T12:06:17.178 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.132+0000 7fa1437fe700 1 -- 172.21.15.102:0/1489002217 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fa144001db0 con 0x7fa150092550 2021-05-25T12:06:17.178 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.132+0000 7fa1437fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:17.178 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.132+0000 7fa1437fe700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2021-05-25T12:06:17.178 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.132+0000 7fa1437fe700 10 monclient(hunting): dump: 2021-05-25T12:06:17.179 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:17.179 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:17.179 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:17.179 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:17.180 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:17.180 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:17.180 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:17.180 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:17.180 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:17.181 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:17.181 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.132+0000 7fa1437fe700 1 -- 172.21.15.102:0/1489002217 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1829317390 0 0) 0x7fa150118510 con 0x7fa15012ec10 2021-05-25T12:06:17.181 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.132+0000 7fa1437fe700 1 -- 172.21.15.102:0/1489002217 >> v1:172.21.15.102:6790/0 conn(0x7fa15012b100 legacy=0x7fa1501403e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:17.181 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.132+0000 7fa1437fe700 1 -- 172.21.15.102:0/1489002217 >> v1:172.21.15.181:6789/0 conn(0x7fa150092550 legacy=0x7fa15012a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:17.181 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.132+0000 7fa1437fe700 10 monclient: _finish_hunting 0 2021-05-25T12:06:17.182 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.132+0000 7fa1437fe700 1 monclient: found mon.a 2021-05-25T12:06:17.182 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.132+0000 7fa1437fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:06:17.183 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.132+0000 7fa1437fe700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:06:17.183 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.132+0000 7fa1437fe700 1 -- 172.21.15.102:0/1489002217 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa150116950 con 0x7fa15012ec10 2021-05-25T12:06:17.183 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.132+0000 7fa1437fe700 10 monclient: _finish_auth 0 2021-05-25T12:06:17.183 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.132+0000 7fa1437fe700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:06:17.183 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.132+0000 7fa155d3e700 5 monclient: authenticate success, global_id 204132 2021-05-25T12:06:17.184 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.132+0000 7fa155d3e700 10 monclient: _renew_subs 2021-05-25T12:06:17.184 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.132+0000 7fa155d3e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:06:17.184 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.132+0000 7fa155d3e700 1 -- 172.21.15.102:0/1489002217 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa15013beb0 con 0x7fa15012ec10 2021-05-25T12:06:17.184 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.132+0000 7fa155d3e700 10 client.204132.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:06:17.185 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.132+0000 7fa155d3e700 10 monclient: _renew_subs 2021-05-25T12:06:17.185 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.132+0000 7fa155d3e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:06:17.185 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.132+0000 7fa155d3e700 1 -- 172.21.15.102:0/1489002217 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa15012f6c0 con 0x7fa15012ec10 2021-05-25T12:06:17.185 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.132+0000 7fa155d3e700 1 librados: init done 2021-05-25T12:06:17.185 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.132+0000 7fa1437fe700 1 -- 172.21.15.102:0/1489002217 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa13c001190 con 0x7fa15012ec10 2021-05-25T12:06:17.186 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.132+0000 7fa1437fe700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:06:17.186 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.132+0000 7fa1437fe700 1 -- 172.21.15.102:0/1489002217 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fa13c004a00 con 0x7fa15012ec10 2021-05-25T12:06:17.186 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.132+0000 7fa1437fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:17.186 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.132+0000 7fa1437fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:06:17.187 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.132+0000 7fa14dffb700 4 set_mon_vals no callback set 2021-05-25T12:06:17.187 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.132+0000 7fa1437fe700 10 monclient: dump: 2021-05-25T12:06:17.187 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:17.187 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:17.187 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:17.188 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:17.188 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:17.188 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:17.188 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:17.188 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:17.189 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:17.189 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:17.189 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.133+0000 7fa1437fe700 1 -- 172.21.15.102:0/1489002217 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7fa13c004be0 con 0x7fa15012ec10 2021-05-25T12:06:17.190 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.133+0000 7fa155d3e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:06:17.190 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.133+0000 7fa1437fe700 1 -- 172.21.15.102:0/1489002217 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(195..195 src has 1..195) v4 ==== 6620+0+0 (unknown 3265642060 0 0) 0x7fa13c03dae0 con 0x7fa15012ec10 2021-05-25T12:06:17.190 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.133+0000 7fa1437fe700 10 client.204132.objecter ms_dispatch 0x7fa150001550 osd_map(195..195 src has 1..195) v4 2021-05-25T12:06:17.190 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.133+0000 7fa1437fe700 3 client.204132.objecter handle_osd_map got epochs [195,195] > 0 2021-05-25T12:06:17.190 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.133+0000 7fa1437fe700 3 client.204132.objecter handle_osd_map decoding full epoch 195 2021-05-25T12:06:17.191 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.133+0000 7fa155d3e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:06:17.191 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.133+0000 7fa155d3e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:06:17.191 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.133+0000 7fa155d3e700 1 -- 172.21.15.102:0/1489002217 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa130004db0 con 0x7fa15012ec10 2021-05-25T12:06:17.191 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.133+0000 7fa1437fe700 20 client.204132.objecter dump_active .. 0 homeless 2021-05-25T12:06:17.192 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.133+0000 7fa14f7fe700 10 monclient: get_auth_request con 0x7fa12c0334f0 auth_method 0 2021-05-25T12:06:17.192 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.133+0000 7fa1437fe700 10 client.204132.objecter ms_handle_connect 0x7fa12c0334f0 2021-05-25T12:06:17.192 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.137+0000 7fa1437fe700 1 -- 172.21.15.102:0/1489002217 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7fa13c012c60 con 0x7fa15012ec10 2021-05-25T12:06:17.192 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.137+0000 7fa1437fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:06:17.192 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.137+0000 7fa1437fe700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:06:17.343 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.341+0000 7fa155d3e700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2021-05-25T12:06:17.343 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.341+0000 7fa155d3e700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2021-05-25T12:06:17.344 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.341+0000 7fa155d3e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:06:17.344 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.341+0000 7fa155d3e700 1 -- 172.21.15.102:0/1489002217 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fa130005060 con 0x7fa15012ec10 2021-05-25T12:06:17.344 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.341+0000 7fa1437fe700 1 -- 172.21.15.102:0/1489002217 <== mon.0 v1:172.21.15.102:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1342 (unknown 254746657 0 1437223952) 0x7fa13c0130c0 con 0x7fa15012ec10 2021-05-25T12:06:17.344 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.342+0000 7fa1437fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2021-05-25T12:06:17.344 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.342+0000 7fa1437fe700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:06:17.345 INFO:teuthology.orchestra.run.smithi102.stdout:{"election_epoch":52,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":2,"features":{"quorum_con":"4540138303579357183","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"20ee91b6-8935-4b98-b868-85c03789ccc4","modified":"2021-05-25T11:59:11.011522Z","created":"2021-05-25T11:59:11.011522Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6790","nonce":0}]},"addr":"172.21.15.102:6790/0","public_addr":"172.21.15.102:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2021-05-25T12:06:17.346 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.343+0000 7fa155d3e700 10 librados: watch_flush enter 2021-05-25T12:06:17.347 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.343+0000 7fa155d3e700 10 librados: watch_flush exit 2021-05-25T12:06:17.347 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.343+0000 7fa155d3e700 20 client.204132.objecter shutdown clearing up homeless session... 2021-05-25T12:06:17.347 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.343+0000 7fa155d3e700 10 client.204132.objecter successfully canceled tick 2021-05-25T12:06:17.347 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.343+0000 7fa155d3e700 1 -- 172.21.15.102:0/1489002217 >> v1:172.21.15.181:6812/29582 conn(0x7fa12c0334f0 legacy=0x7fa12c0359b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:17.348 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.343+0000 7fa155d3e700 10 monclient: shutdown 2021-05-25T12:06:17.348 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.343+0000 7fa155d3e700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:06:17.348 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.343+0000 7fa155d3e700 1 -- 172.21.15.102:0/1489002217 >> v1:172.21.15.102:6789/0 conn(0x7fa15012ec10 legacy=0x7fa150143b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:17.348 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.343+0000 7fa155d3e700 1 -- 172.21.15.102:0/1489002217 shutdown_connections 2021-05-25T12:06:17.348 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.343+0000 7fa155d3e700 1 -- 172.21.15.102:0/1489002217 >> 172.21.15.102:0/1489002217 conn(0x7fa15009b370 msgr2=0x7fa1500959a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:06:17.349 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.343+0000 7fa155d3e700 1 -- 172.21.15.102:0/1489002217 shutdown_connections 2021-05-25T12:06:17.349 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.343+0000 7fa155d3e700 1 -- 172.21.15.102:0/1489002217 wait complete. 2021-05-25T12:06:17.349 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.343+0000 7fa155d3e700 1 librados: shutdown 2021-05-25T12:06:17.355 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2021-05-25T12:06:17.356 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2021-05-25T12:06:17.356 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2021-05-25T12:06:17.465 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.464+0000 7fedb5018700 10 monclient: get_monmap_and_config 2021-05-25T12:06:17.466 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.464+0000 7fedb5018700 10 monclient: build_initial_monmap 2021-05-25T12:06:17.466 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.464+0000 7fedb5018700 10 monclient: monmap: 2021-05-25T12:06:17.466 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:06:17.466 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:17.467 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:06:17.465347+0000 2021-05-25T12:06:17.467 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:06:17.465347+0000 2021-05-25T12:06:17.467 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:06:17.467 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:17.468 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:06:17.468 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:06:17.468 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:06:17.468 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:17.481 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.465+0000 7fedb5018700 1 Processor -- start 2021-05-25T12:06:17.481 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.465+0000 7fedb5018700 1 -- start start 2021-05-25T12:06:17.481 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.465+0000 7fedb5018700 10 monclient: init 2021-05-25T12:06:17.482 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.465+0000 7fedb5018700 10 monclient: _reopen_session rank -1 2021-05-25T12:06:17.482 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.465+0000 7fedb5018700 10 monclient: _add_conns ranks=[1,2,0] 2021-05-25T12:06:17.482 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.465+0000 7fedb5018700 10 monclient(hunting): picked mon.noname-c con 0x7fedb0095c40 addr v1:172.21.15.181:6789/0 2021-05-25T12:06:17.482 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.465+0000 7fedb5018700 10 monclient(hunting): picked mon.noname-b con 0x7fedb0096020 addr v1:172.21.15.102:6790/0 2021-05-25T12:06:17.483 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.465+0000 7fedb5018700 10 monclient(hunting): picked mon.noname-a con 0x7fedb012e5c0 addr v1:172.21.15.102:6789/0 2021-05-25T12:06:17.483 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.465+0000 7fedb5018700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedb0092990 con 0x7fedb012e5c0 2021-05-25T12:06:17.483 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.465+0000 7fedb5018700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedb0092c30 con 0x7fedb0095c40 2021-05-25T12:06:17.483 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.465+0000 7fedb5018700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedb0092ed0 con 0x7fedb0096020 2021-05-25T12:06:17.484 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.465+0000 7fedb5018700 10 monclient(hunting): _renew_subs 2021-05-25T12:06:17.484 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.465+0000 7fedb5018700 10 monclient(hunting): authenticate will time out at 1178.062704s 2021-05-25T12:06:17.484 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.465+0000 7fedaf7fe700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fedb0096020 0x7fedb00964d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:56372/0 (socket says 172.21.15.102:56372) 2021-05-25T12:06:17.484 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.465+0000 7fedaf7fe700 1 -- 172.21.15.102:0/1001679376 learned_addr learned my addr 172.21.15.102:0/1001679376 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:06:17.484 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.465+0000 7fedb4817700 1 --1- 172.21.15.102:0/1001679376 >> v1:172.21.15.102:6789/0 conn(0x7fedb012e5c0 0x7fedb01309b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33266/0 (socket says 172.21.15.102:33266) 2021-05-25T12:06:17.485 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.466+0000 7fedaeffd700 1 -- 172.21.15.102:0/1001679376 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 518743728 0 0) 0x7fedb0092990 con 0x7fedb012e5c0 2021-05-25T12:06:17.485 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.466+0000 7fedaeffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:17.485 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.466+0000 7fedaeffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:17.485 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.466+0000 7fedaeffd700 10 monclient(hunting): my global_id is 204135 2021-05-25T12:06:17.485 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.466+0000 7fedaeffd700 1 -- 172.21.15.102:0/1001679376 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed94004380 con 0x7fedb012e5c0 2021-05-25T12:06:17.486 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.466+0000 7fedaeffd700 1 -- 172.21.15.102:0/1001679376 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4072111553 0 0) 0x7fedb0092ed0 con 0x7fedb0096020 2021-05-25T12:06:17.486 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.466+0000 7fedaeffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:17.486 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.466+0000 7fedaeffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:17.486 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.466+0000 7fedaeffd700 10 monclient(hunting): my global_id is 184481 2021-05-25T12:06:17.487 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.466+0000 7fedaeffd700 1 -- 172.21.15.102:0/1001679376 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedb0092990 con 0x7fedb0096020 2021-05-25T12:06:17.487 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.466+0000 7fedaeffd700 1 -- 172.21.15.102:0/1001679376 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2780782127 0 0) 0x7fed94004380 con 0x7fedb012e5c0 2021-05-25T12:06:17.487 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.466+0000 7fedaeffd700 1 -- 172.21.15.102:0/1001679376 >> v1:172.21.15.102:6790/0 conn(0x7fedb0096020 legacy=0x7fedb00964d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:17.487 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.466+0000 7fedaeffd700 1 -- 172.21.15.102:0/1001679376 >> v1:172.21.15.181:6789/0 conn(0x7fedb0095c40 legacy=0x7fedb0093290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:17.487 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.466+0000 7fedaeffd700 10 monclient: _finish_hunting 0 2021-05-25T12:06:17.488 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.466+0000 7fedaeffd700 1 monclient: found mon.noname-a 2021-05-25T12:06:17.488 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.466+0000 7fedaeffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:06:17.488 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.466+0000 7fedaeffd700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T12:06:17.488 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.466+0000 7fedaeffd700 1 -- 172.21.15.102:0/1001679376 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedb0092430 con 0x7fedb012e5c0 2021-05-25T12:06:17.489 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.466+0000 7fedaeffd700 10 monclient: _finish_auth 0 2021-05-25T12:06:17.489 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.466+0000 7fedaeffd700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:06:17.489 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.466+0000 7fedaeffd700 1 -- 172.21.15.102:0/1001679376 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7feda40019e0 con 0x7fedb012e5c0 2021-05-25T12:06:17.489 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.466+0000 7fedb5018700 5 monclient: authenticate success, global_id 204135 2021-05-25T12:06:17.489 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.466+0000 7fedb5018700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:06:17.490 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.466+0000 7fedaeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:17.490 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.466+0000 7fedaeffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:06:17.490 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.466+0000 7fedaeffd700 10 monclient: dump: 2021-05-25T12:06:17.490 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:17.491 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:17.491 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:17.491 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:17.491 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:17.491 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:17.492 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:17.492 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:17.492 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:17.492 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:17.492 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.466+0000 7fedb5018700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:06:17.493 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.466+0000 7fedaeffd700 1 -- 172.21.15.102:0/1001679376 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feda4001c10 con 0x7fedb012e5c0 2021-05-25T12:06:17.493 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.466+0000 7fedaeffd700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:06:17.493 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.466+0000 7fedb5018700 10 monclient: get_monmap_and_config success 2021-05-25T12:06:17.493 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.466+0000 7fedb5018700 4 set_mon_vals no callback set 2021-05-25T12:06:17.494 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.466+0000 7fedaeffd700 1 -- 172.21.15.102:0/1001679376 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7feda4001fa0 con 0x7fedb012e5c0 2021-05-25T12:06:17.494 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.466+0000 7fedaeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:17.494 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.466+0000 7fedaeffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:06:17.494 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.466+0000 7fedaeffd700 10 monclient: dump: 2021-05-25T12:06:17.494 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:17.495 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:17.495 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:17.495 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:17.495 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:17.495 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:17.496 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:17.496 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:17.496 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:17.496 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:17.497 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.466+0000 7fedb5018700 10 monclient: shutdown 2021-05-25T12:06:17.497 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.466+0000 7fedb5018700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:06:17.497 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.466+0000 7fedb5018700 1 -- 172.21.15.102:0/1001679376 >> v1:172.21.15.102:6789/0 conn(0x7fedb012e5c0 legacy=0x7fedb01309b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:17.497 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.466+0000 7fedb5018700 1 -- 172.21.15.102:0/1001679376 shutdown_connections 2021-05-25T12:06:17.497 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.466+0000 7fedb5018700 1 -- 172.21.15.102:0/1001679376 >> 172.21.15.102:0/1001679376 conn(0x7fedb00974b0 msgr2=0x7fedb00978c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:06:17.498 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.467+0000 7fedb5018700 1 -- 172.21.15.102:0/1001679376 shutdown_connections 2021-05-25T12:06:17.498 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.467+0000 7fedb5018700 1 -- 172.21.15.102:0/1001679376 wait complete. 2021-05-25T12:06:17.498 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.467+0000 7fedb5018700 10 monclient: build_initial_monmap 2021-05-25T12:06:17.498 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.467+0000 7fedb5018700 10 monclient: monmap: 2021-05-25T12:06:17.499 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:06:17.499 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:06:17.499 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:06:17.499 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:06:17.499 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:06:17.500 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:06:17.500 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:06:17.500 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:06:17.500 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:06:17.500 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:17.501 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.467+0000 7fedb5018700 1 librados: starting msgr at 2021-05-25T12:06:17.501 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.467+0000 7fedb5018700 1 librados: starting objecter 2021-05-25T12:06:17.501 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.467+0000 7fedb5018700 1 Processor -- start 2021-05-25T12:06:17.501 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.467+0000 7fedb5018700 1 -- start start 2021-05-25T12:06:17.502 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.467+0000 7fedb5018700 1 librados: setting wanted keys 2021-05-25T12:06:17.502 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.467+0000 7fedb5018700 1 librados: calling monclient init 2021-05-25T12:06:17.502 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.467+0000 7fedb5018700 10 monclient: init 2021-05-25T12:06:17.502 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.468+0000 7fedb5018700 10 monclient: _reopen_session rank -1 2021-05-25T12:06:17.502 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.468+0000 7fedb5018700 10 monclient: _add_conns ranks=[0,1,2] 2021-05-25T12:06:17.503 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.468+0000 7fedb5018700 10 monclient(hunting): picked mon.noname-a con 0x7fedb0095c40 addr v1:172.21.15.102:6789/0 2021-05-25T12:06:17.503 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.468+0000 7fedb5018700 10 monclient(hunting): picked mon.noname-b con 0x7fedb0096020 addr v1:172.21.15.181:6789/0 2021-05-25T12:06:17.503 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.468+0000 7fedb5018700 10 monclient(hunting): picked mon.noname-c con 0x7fedb012e5c0 addr v1:172.21.15.102:6790/0 2021-05-25T12:06:17.503 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.468+0000 7fedb5018700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedb0129380 con 0x7fedb0095c40 2021-05-25T12:06:17.503 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.468+0000 7fedb5018700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedb012d670 con 0x7fedb0096020 2021-05-25T12:06:17.504 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.468+0000 7fedb5018700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedb0112ce0 con 0x7fedb012e5c0 2021-05-25T12:06:17.504 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.468+0000 7fedb5018700 10 monclient(hunting): _renew_subs 2021-05-25T12:06:17.504 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.468+0000 7fedb5018700 10 monclient(hunting): authenticate will time out at 1178.065393s 2021-05-25T12:06:17.504 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.468+0000 7fedaffff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fedb0095c40 0x7fedb0124530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33268/0 (socket says 172.21.15.102:33268) 2021-05-25T12:06:17.505 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.468+0000 7fedaffff700 1 -- 172.21.15.102:0/4150563965 learned_addr learned my addr 172.21.15.102:0/4150563965 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:06:17.505 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.468+0000 7fed9bfff700 10 client.?.objecter ms_handle_connect 0x7fedb012e5c0 2021-05-25T12:06:17.505 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.468+0000 7fed9bfff700 10 client.?.objecter resend_mon_ops 2021-05-25T12:06:17.505 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.468+0000 7fed9bfff700 10 client.?.objecter ms_handle_connect 0x7fedb0095c40 2021-05-25T12:06:17.505 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.468+0000 7fed9bfff700 10 client.?.objecter resend_mon_ops 2021-05-25T12:06:17.506 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.468+0000 7fed9bfff700 1 -- 172.21.15.102:0/4150563965 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 929636969 0 0) 0x7fedb0112ce0 con 0x7fedb012e5c0 2021-05-25T12:06:17.506 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.468+0000 7fed9bfff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:17.506 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.468+0000 7fed9bfff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:17.506 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.468+0000 7fed9bfff700 10 monclient(hunting): my global_id is 184484 2021-05-25T12:06:17.507 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.468+0000 7fed9bfff700 1 -- 172.21.15.102:0/4150563965 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed90004c10 con 0x7fedb012e5c0 2021-05-25T12:06:17.507 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.468+0000 7fed9bfff700 1 -- 172.21.15.102:0/4150563965 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1941719746 0 0) 0x7fedb0129380 con 0x7fedb0095c40 2021-05-25T12:06:17.507 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.468+0000 7fed9bfff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:17.507 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.468+0000 7fed9bfff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:17.507 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.468+0000 7fed9bfff700 10 monclient(hunting): my global_id is 204138 2021-05-25T12:06:17.508 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.468+0000 7fed9bfff700 1 -- 172.21.15.102:0/4150563965 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedb0112ce0 con 0x7fedb0095c40 2021-05-25T12:06:17.508 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.468+0000 7fed9bfff700 10 client.?.objecter ms_handle_connect 0x7fedb0096020 2021-05-25T12:06:17.508 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.468+0000 7fed9bfff700 10 client.?.objecter resend_mon_ops 2021-05-25T12:06:17.508 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.468+0000 7fed9bfff700 1 -- 172.21.15.102:0/4150563965 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3817709184 0 0) 0x7fed90004c10 con 0x7fedb012e5c0 2021-05-25T12:06:17.509 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.468+0000 7fed9bfff700 1 -- 172.21.15.102:0/4150563965 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fedb0129380 con 0x7fedb012e5c0 2021-05-25T12:06:17.509 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.468+0000 7fed9bfff700 1 -- 172.21.15.102:0/4150563965 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7feda4001f20 con 0x7fedb012e5c0 2021-05-25T12:06:17.509 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.468+0000 7fed9bfff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:17.509 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.468+0000 7fed9bfff700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:06:17.509 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.468+0000 7fed9bfff700 10 monclient(hunting): dump: 2021-05-25T12:06:17.510 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:17.510 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:17.510 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:17.510 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:17.511 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:17.511 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:17.511 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:17.511 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:17.511 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:17.512 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:17.512 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.468+0000 7fed9bfff700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:06:17.512 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.468+0000 7fed9bfff700 1 -- 172.21.15.102:0/4150563965 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1539675595 0 0) 0x7fedb0112ce0 con 0x7fedb0095c40 2021-05-25T12:06:17.512 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.468+0000 7fed9bfff700 1 -- 172.21.15.102:0/4150563965 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fed90004c10 con 0x7fedb0095c40 2021-05-25T12:06:17.512 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.468+0000 7fed9bfff700 1 -- 172.21.15.102:0/4150563965 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7feda0001910 con 0x7fedb0095c40 2021-05-25T12:06:17.513 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.468+0000 7fed9bfff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:17.513 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.468+0000 7fed9bfff700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2021-05-25T12:06:17.513 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.468+0000 7fed9bfff700 10 monclient(hunting): dump: 2021-05-25T12:06:17.513 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:17.514 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:17.514 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:17.514 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:17.514 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:17.514 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:17.515 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:17.515 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:17.515 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:17.515 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:17.515 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.469+0000 7fed9bfff700 1 -- 172.21.15.102:0/4150563965 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 964556746 0 0) 0x7fedb012d670 con 0x7fedb0096020 2021-05-25T12:06:17.516 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.469+0000 7fed9bfff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:17.516 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.469+0000 7fed9bfff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:17.516 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.469+0000 7fed9bfff700 10 monclient(hunting): my global_id is 194308 2021-05-25T12:06:17.516 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.469+0000 7fed9bfff700 1 -- 172.21.15.102:0/4150563965 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedb0112ce0 con 0x7fedb0096020 2021-05-25T12:06:17.517 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.469+0000 7fed9bfff700 1 -- 172.21.15.102:0/4150563965 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2600281896 0 0) 0x7fedb0129380 con 0x7fedb012e5c0 2021-05-25T12:06:17.517 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.469+0000 7fed9bfff700 1 -- 172.21.15.102:0/4150563965 >> v1:172.21.15.181:6789/0 conn(0x7fedb0096020 legacy=0x7fedb0140150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:17.517 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.469+0000 7fed9bfff700 1 -- 172.21.15.102:0/4150563965 >> v1:172.21.15.102:6789/0 conn(0x7fedb0095c40 legacy=0x7fedb0124530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:17.517 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.469+0000 7fed9bfff700 10 monclient: _finish_hunting 0 2021-05-25T12:06:17.517 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.469+0000 7fed9bfff700 1 monclient: found mon.c 2021-05-25T12:06:17.518 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.469+0000 7fed9bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:06:17.518 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.469+0000 7fed9bfff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:06:17.518 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.469+0000 7fed9bfff700 1 -- 172.21.15.102:0/4150563965 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedb0112f80 con 0x7fedb012e5c0 2021-05-25T12:06:17.518 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.469+0000 7fed9bfff700 10 monclient: _finish_auth 0 2021-05-25T12:06:17.519 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.469+0000 7fed9bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:06:17.519 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.469+0000 7fedb5018700 5 monclient: authenticate success, global_id 184484 2021-05-25T12:06:17.519 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.469+0000 7fedb5018700 10 monclient: _renew_subs 2021-05-25T12:06:17.519 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.469+0000 7fedb5018700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:06:17.519 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.469+0000 7fedb5018700 1 -- 172.21.15.102:0/4150563965 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedb0111960 con 0x7fedb012e5c0 2021-05-25T12:06:17.520 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.469+0000 7fedb5018700 10 client.184484.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:06:17.520 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.469+0000 7fedb5018700 10 monclient: _renew_subs 2021-05-25T12:06:17.520 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.469+0000 7fedb5018700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:06:17.520 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.469+0000 7fedb5018700 1 -- 172.21.15.102:0/4150563965 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fedb0124c40 con 0x7fedb012e5c0 2021-05-25T12:06:17.520 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.469+0000 7fedb5018700 1 librados: init done 2021-05-25T12:06:17.521 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.469+0000 7fed9bfff700 1 -- 172.21.15.102:0/4150563965 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feda4004bb0 con 0x7fedb012e5c0 2021-05-25T12:06:17.521 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.469+0000 7fed9bfff700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:06:17.521 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.469+0000 7fed9bfff700 1 -- 172.21.15.102:0/4150563965 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7feda4004ec0 con 0x7fedb012e5c0 2021-05-25T12:06:17.521 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.469+0000 7fed9bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:17.522 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.469+0000 7fed9bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:06:17.522 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.469+0000 7fedae7fc700 4 set_mon_vals no callback set 2021-05-25T12:06:17.522 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.469+0000 7fed9bfff700 10 monclient: dump: 2021-05-25T12:06:17.522 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:17.522 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:17.523 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:17.523 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:17.523 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:17.523 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:17.524 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:17.524 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:17.524 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:17.524 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:17.524 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.469+0000 7fedb5018700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:06:17.525 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.469+0000 7fedb5018700 1 -- 172.21.15.102:0/4150563965 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fed880065a0 con 0x7fed88002840 2021-05-25T12:06:17.525 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.470+0000 7fed9bfff700 1 -- 172.21.15.102:0/4150563965 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7feda400e1e0 con 0x7fedb012e5c0 2021-05-25T12:06:17.525 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.470+0000 7fed9bfff700 10 client.184484.objecter ms_handle_connect 0x7fed88002840 2021-05-25T12:06:17.525 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.470+0000 7fed9bfff700 10 client.184484.objecter resend_mon_ops 2021-05-25T12:06:17.525 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.470+0000 7fed9bfff700 1 -- 172.21.15.102:0/4150563965 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(195..195 src has 1..195) v4 ==== 6620+0+0 (unknown 3265642060 0 0) 0x7feda403de60 con 0x7fedb012e5c0 2021-05-25T12:06:17.526 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.470+0000 7fed9bfff700 10 client.184484.objecter ms_dispatch 0x7fedb0001550 osd_map(195..195 src has 1..195) v4 2021-05-25T12:06:17.526 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.470+0000 7fed9bfff700 3 client.184484.objecter handle_osd_map got epochs [195,195] > 0 2021-05-25T12:06:17.526 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.470+0000 7fed9bfff700 3 client.184484.objecter handle_osd_map decoding full epoch 195 2021-05-25T12:06:17.526 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.470+0000 7fed9bfff700 20 client.184484.objecter dump_active .. 0 homeless 2021-05-25T12:06:17.526 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.470+0000 7fed9bfff700 1 -- 172.21.15.102:0/4150563965 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3949625752 0 0) 0x7fed880065a0 con 0x7fed88002840 2021-05-25T12:06:17.527 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.470+0000 7fed9bfff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:17.527 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.470+0000 7fed9bfff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:17.527 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.470+0000 7fed9bfff700 10 monclient(hunting): my global_id is 204141 2021-05-25T12:06:17.527 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.470+0000 7fed9bfff700 1 -- 172.21.15.102:0/4150563965 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedb0129380 con 0x7fed88002840 2021-05-25T12:06:17.528 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.470+0000 7fedaf7fe700 10 monclient: get_auth_request con 0x7fed900333e0 auth_method 0 2021-05-25T12:06:17.528 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.470+0000 7fed9bfff700 1 -- 172.21.15.102:0/4150563965 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1450262900 0 0) 0x7fedb0129380 con 0x7fed88002840 2021-05-25T12:06:17.528 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.470+0000 7fed9bfff700 1 -- 172.21.15.102:0/4150563965 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fed880065a0 con 0x7fed88002840 2021-05-25T12:06:17.528 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.470+0000 7fed9bfff700 1 -- 172.21.15.102:0/4150563965 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7feda00023d0 con 0x7fed88002840 2021-05-25T12:06:17.528 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.470+0000 7fed9bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:17.529 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.470+0000 7fed9bfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:06:17.529 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.470+0000 7fed9bfff700 10 monclient: dump: 2021-05-25T12:06:17.529 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:17.529 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:17.529 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:17.530 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:17.530 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:17.530 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:17.530 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:17.530 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:17.531 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:17.531 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:17.531 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.470+0000 7fed9bfff700 10 client.184484.objecter ms_handle_connect 0x7fed900333e0 2021-05-25T12:06:17.531 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.470+0000 7fed9bfff700 1 -- 172.21.15.102:0/4150563965 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2866375451 0 0) 0x7fed880065a0 con 0x7fed88002840 2021-05-25T12:06:17.532 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.470+0000 7fed9bfff700 1 -- 172.21.15.102:0/4150563965 --> v1:172.21.15.102:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fed88009e40 con 0x7fed88002840 2021-05-25T12:06:17.532 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.471+0000 7fed9bfff700 1 -- 172.21.15.102:0/4150563965 <== mon.0 v1:172.21.15.102:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7feda00025f0 con 0x7fed88002840 2021-05-25T12:06:17.532 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.471+0000 7fed9bfff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:06:17.532 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.471+0000 7fed9bfff700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:06:17.532 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.471+0000 7fed9bfff700 1 -- 172.21.15.102:0/4150563965 >> v1:172.21.15.102:6789/0 conn(0x7fed88002840 legacy=0x7fed88004ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:17.533 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.471+0000 7fed9bfff700 1 -- 172.21.15.102:0/4150563965 >> v1:172.21.15.102:6789/0 conn(0x7fed88002840 legacy=0x7fed88004ce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:06:17.533 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.477+0000 7fedb5018700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:06:17.533 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.477+0000 7fedb5018700 1 -- 172.21.15.102:0/4150563965 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fed88009b00 con 0x7fed88008dc0 2021-05-25T12:06:17.533 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.477+0000 7fed9bfff700 10 client.184484.objecter ms_handle_connect 0x7fed88008dc0 2021-05-25T12:06:17.533 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.477+0000 7fed9bfff700 10 client.184484.objecter resend_mon_ops 2021-05-25T12:06:17.534 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.477+0000 7fed9bfff700 1 -- 172.21.15.102:0/4150563965 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1694166308 0 0) 0x7fed88009b00 con 0x7fed88008dc0 2021-05-25T12:06:17.534 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.477+0000 7fed9bfff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:17.534 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.477+0000 7fed9bfff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:17.534 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.477+0000 7fed9bfff700 10 monclient(hunting): my global_id is 204144 2021-05-25T12:06:17.534 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.477+0000 7fed9bfff700 1 -- 172.21.15.102:0/4150563965 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed880065a0 con 0x7fed88008dc0 2021-05-25T12:06:17.535 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.478+0000 7fed9bfff700 1 -- 172.21.15.102:0/4150563965 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3044327112 0 0) 0x7fed880065a0 con 0x7fed88008dc0 2021-05-25T12:06:17.535 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.478+0000 7fed9bfff700 1 -- 172.21.15.102:0/4150563965 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fed88009b00 con 0x7fed88008dc0 2021-05-25T12:06:17.535 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.478+0000 7fed9bfff700 1 -- 172.21.15.102:0/4150563965 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7feda0002d30 con 0x7fed88008dc0 2021-05-25T12:06:17.535 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.478+0000 7fed9bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:17.536 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.478+0000 7fed9bfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:06:17.536 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.478+0000 7fed9bfff700 10 monclient: dump: 2021-05-25T12:06:17.536 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:17.536 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:17.536 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:17.537 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:17.537 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:17.537 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:17.537 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:17.537 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:17.538 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:17.538 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:17.538 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.478+0000 7fed9bfff700 1 -- 172.21.15.102:0/4150563965 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2152643221 0 0) 0x7fed88009b00 con 0x7fed88008dc0 2021-05-25T12:06:17.538 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.478+0000 7fed9bfff700 1 -- 172.21.15.102:0/4150563965 --> v1:172.21.15.102:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fed88008740 con 0x7fed88008dc0 2021-05-25T12:06:17.539 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.478+0000 7fed9bfff700 1 -- 172.21.15.102:0/4150563965 <== mon.0 v1:172.21.15.102:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3770 (unknown 0 0 1721313141) 0x7feda0006ed0 con 0x7fed88008dc0 2021-05-25T12:06:17.539 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.478+0000 7fed9bfff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:06:17.539 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.478+0000 7fed9bfff700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:06:17.539 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.478+0000 7fed9bfff700 1 -- 172.21.15.102:0/4150563965 >> v1:172.21.15.102:6789/0 conn(0x7fed88008dc0 legacy=0x7fed88009250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:17.539 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.478+0000 7fed9bfff700 1 -- 172.21.15.102:0/4150563965 >> v1:172.21.15.102:6789/0 conn(0x7fed88008dc0 legacy=0x7fed88009250 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:06:17.540 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:06:17.540 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:06:17.540 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:06:17.541 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "peon", 2021-05-25T12:06:17.541 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 52, 2021-05-25T12:06:17.541 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:06:17.541 INFO:teuthology.orchestra.run.smithi102.stdout: 0, 2021-05-25T12:06:17.541 INFO:teuthology.orchestra.run.smithi102.stdout: 1, 2021-05-25T12:06:17.542 INFO:teuthology.orchestra.run.smithi102.stdout: 2 2021-05-25T12:06:17.542 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:06:17.542 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 2, 2021-05-25T12:06:17.542 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:06:17.543 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:06:17.543 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:06:17.543 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:06:17.543 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:06:17.543 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:06:17.544 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:06:17.544 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:06:17.544 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:06:17.544 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:06:17.544 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:06:17.545 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:06:17.545 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:06:17.545 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:06:17.545 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:06:17.545 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:06:17.546 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:06:17.546 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:06:17.546 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:06:17.546 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:06:17.546 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:06:17.547 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:06:17.547 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:06:17.547 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:06:17.547 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:17.548 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:17.548 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:06:17.548 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:06:17.548 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:06:17.548 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:06:17.549 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:06:17.549 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:06:17.549 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:06:17.549 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:06:17.549 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:06:17.550 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:06:17.550 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:06:17.550 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:06:17.550 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:06:17.551 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:06:17.551 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:06:17.551 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:06:17.551 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:06:17.551 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:06:17.552 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:06:17.552 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:06:17.552 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:06:17.552 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:06:17.552 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:06:17.553 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:06:17.553 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:06:17.553 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:06:17.553 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:17.553 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:06:17.554 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:17.554 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:06:17.554 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:06:17.554 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:06:17.554 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:06:17.555 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:17.555 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:06:17.555 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:06:17.555 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:06:17.556 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:17.556 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:17.556 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:17.556 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:06:17.556 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:06:17.557 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:06:17.557 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:06:17.557 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:06:17.557 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:17.557 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:17.558 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:06:17.558 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:06:17.558 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:06:17.558 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:06:17.559 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:17.559 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:06:17.559 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:06:17.559 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:06:17.559 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:17.560 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:17.560 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:17.560 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:06:17.560 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:06:17.561 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:06:17.561 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:06:17.561 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:06:17.561 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:17.561 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:17.562 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:06:17.562 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:06:17.562 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:06:17.562 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:06:17.563 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:17.563 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:06:17.563 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:06:17.563 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:06:17.563 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:17.564 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:17.564 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:17.564 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:06:17.564 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:06:17.564 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:06:17.565 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:06:17.565 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:06:17.565 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:17.565 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:17.565 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:17.566 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:06:17.566 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:06:17.566 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:17.566 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:06:17.566 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:06:17.567 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:06:17.567 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:17.567 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:06:17.567 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:06:17.567 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:17.568 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:06:17.568 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:06:17.568 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:06:17.568 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:17.568 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:17.569 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:17.569 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:06:17.569 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:06:17.571 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.478+0000 7fedb5018700 10 librados: watch_flush enter 2021-05-25T12:06:17.572 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.478+0000 7fedb5018700 10 librados: watch_flush exit 2021-05-25T12:06:17.572 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.478+0000 7fedb5018700 20 client.184484.objecter shutdown clearing up homeless session... 2021-05-25T12:06:17.572 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.478+0000 7fedb5018700 10 client.184484.objecter successfully canceled tick 2021-05-25T12:06:17.572 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.478+0000 7fedb5018700 1 -- 172.21.15.102:0/4150563965 >> v1:172.21.15.181:6812/29582 conn(0x7fed900333e0 legacy=0x7fed900358a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:17.573 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.478+0000 7fedb5018700 10 monclient: shutdown 2021-05-25T12:06:17.573 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.478+0000 7fedb5018700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:06:17.573 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.478+0000 7fedb5018700 1 -- 172.21.15.102:0/4150563965 >> v1:172.21.15.102:6790/0 conn(0x7fedb012e5c0 legacy=0x7fedb0143880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:17.573 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.478+0000 7fedb4817700 1 -- 172.21.15.102:0/4150563965 reap_dead start 2021-05-25T12:06:17.573 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.478+0000 7fedb4817700 1 -- 172.21.15.102:0/4150563965 reap_dead start 2021-05-25T12:06:17.574 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.478+0000 7fedb5018700 1 -- 172.21.15.102:0/4150563965 shutdown_connections 2021-05-25T12:06:17.574 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.478+0000 7fedb5018700 1 -- 172.21.15.102:0/4150563965 >> 172.21.15.102:0/4150563965 conn(0x7fedb00974b0 msgr2=0x7fedb012da80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:06:17.574 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.478+0000 7fedb5018700 1 -- 172.21.15.102:0/4150563965 shutdown_connections 2021-05-25T12:06:17.574 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.478+0000 7fedb5018700 1 -- 172.21.15.102:0/4150563965 wait complete. 2021-05-25T12:06:17.574 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.478+0000 7fedb5018700 1 librados: shutdown 2021-05-25T12:06:17.575 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2021-05-25T12:06:17.685 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.684+0000 7f5b0b861700 10 monclient: get_monmap_and_config 2021-05-25T12:06:17.686 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.684+0000 7f5b0b861700 10 monclient: build_initial_monmap 2021-05-25T12:06:17.686 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.684+0000 7f5b0b861700 10 monclient: monmap: 2021-05-25T12:06:17.686 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:06:17.686 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:17.687 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:06:17.685413+0000 2021-05-25T12:06:17.687 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:06:17.685413+0000 2021-05-25T12:06:17.687 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:06:17.687 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:17.688 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:06:17.688 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:06:17.688 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:06:17.688 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:17.698 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.685+0000 7f5b0b861700 1 Processor -- start 2021-05-25T12:06:17.698 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.685+0000 7f5b0b861700 1 -- start start 2021-05-25T12:06:17.698 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.685+0000 7f5b0b861700 10 monclient: init 2021-05-25T12:06:17.699 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.685+0000 7f5b0b861700 10 monclient: _reopen_session rank -1 2021-05-25T12:06:17.699 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.685+0000 7f5b0b861700 10 monclient: _add_conns ranks=[1,2,0] 2021-05-25T12:06:17.699 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.685+0000 7f5b0b861700 10 monclient(hunting): picked mon.noname-c con 0x7f5b04090990 addr v1:172.21.15.181:6789/0 2021-05-25T12:06:17.699 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.685+0000 7f5b0b861700 10 monclient(hunting): picked mon.noname-b con 0x7f5b04090d70 addr v1:172.21.15.102:6790/0 2021-05-25T12:06:17.700 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.685+0000 7f5b0b861700 10 monclient(hunting): picked mon.noname-a con 0x7f5b0412e330 addr v1:172.21.15.102:6789/0 2021-05-25T12:06:17.700 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.685+0000 7f5b0b861700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b04096fa0 con 0x7f5b0412e330 2021-05-25T12:06:17.700 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.685+0000 7f5b0b861700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b040926a0 con 0x7f5b04090990 2021-05-25T12:06:17.700 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.685+0000 7f5b0b861700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b04092940 con 0x7f5b04090d70 2021-05-25T12:06:17.700 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.685+0000 7f5b0b861700 10 monclient(hunting): _renew_subs 2021-05-25T12:06:17.701 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.685+0000 7f5b0b861700 10 monclient(hunting): authenticate will time out at 1178.282787s 2021-05-25T12:06:17.701 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.685+0000 7f5b08dfc700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f5b04090d70 0x7f5b04091200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:56390/0 (socket says 172.21.15.102:56390) 2021-05-25T12:06:17.701 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.685+0000 7f5b09dfe700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f5b0412e330 0x7f5b04130720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33284/0 (socket says 172.21.15.102:33284) 2021-05-25T12:06:17.701 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.685+0000 7f5b08dfc700 1 -- 172.21.15.102:0/1887457542 learned_addr learned my addr 172.21.15.102:0/1887457542 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:06:17.701 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.686+0000 7f5afbfff700 1 -- 172.21.15.102:0/1887457542 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 462059359 0 0) 0x7f5b04096fa0 con 0x7f5b0412e330 2021-05-25T12:06:17.702 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.686+0000 7f5afbfff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:17.702 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.686+0000 7f5afbfff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:17.702 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.686+0000 7f5afbfff700 10 monclient(hunting): my global_id is 204147 2021-05-25T12:06:17.702 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.686+0000 7f5afbfff700 1 -- 172.21.15.102:0/1887457542 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5af0003270 con 0x7f5b0412e330 2021-05-25T12:06:17.702 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.686+0000 7f5afbfff700 1 -- 172.21.15.102:0/1887457542 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2919837478 0 0) 0x7f5b04092940 con 0x7f5b04090d70 2021-05-25T12:06:17.703 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.686+0000 7f5afbfff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:17.703 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.686+0000 7f5afbfff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:17.703 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.686+0000 7f5afbfff700 10 monclient(hunting): my global_id is 184487 2021-05-25T12:06:17.703 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.686+0000 7f5afbfff700 1 -- 172.21.15.102:0/1887457542 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b04096fa0 con 0x7f5b04090d70 2021-05-25T12:06:17.704 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.686+0000 7f5afbfff700 1 -- 172.21.15.102:0/1887457542 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3985925322 0 0) 0x7f5b040926a0 con 0x7f5b04090990 2021-05-25T12:06:17.704 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.686+0000 7f5afbfff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:17.704 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.686+0000 7f5afbfff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:17.704 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.686+0000 7f5afbfff700 10 monclient(hunting): my global_id is 194311 2021-05-25T12:06:17.704 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.686+0000 7f5afbfff700 1 -- 172.21.15.102:0/1887457542 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b04092940 con 0x7f5b04090990 2021-05-25T12:06:17.705 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.686+0000 7f5afbfff700 1 -- 172.21.15.102:0/1887457542 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2803243914 0 0) 0x7f5af0003270 con 0x7f5b0412e330 2021-05-25T12:06:17.705 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.686+0000 7f5afbfff700 1 -- 172.21.15.102:0/1887457542 >> v1:172.21.15.102:6790/0 conn(0x7f5b04090d70 legacy=0x7f5b04091200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:17.705 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.686+0000 7f5afbfff700 1 -- 172.21.15.102:0/1887457542 >> v1:172.21.15.181:6789/0 conn(0x7f5b04090990 legacy=0x7f5b04093240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:17.705 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.686+0000 7f5afbfff700 10 monclient: _finish_hunting 0 2021-05-25T12:06:17.705 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.686+0000 7f5afbfff700 1 monclient: found mon.noname-a 2021-05-25T12:06:17.706 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.686+0000 7f5afbfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:06:17.706 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.686+0000 7f5afbfff700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T12:06:17.706 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.686+0000 7f5afbfff700 1 -- 172.21.15.102:0/1887457542 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b04092be0 con 0x7f5b0412e330 2021-05-25T12:06:17.706 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.686+0000 7f5afbfff700 10 monclient: _finish_auth 0 2021-05-25T12:06:17.707 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.686+0000 7f5afbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:06:17.707 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.686+0000 7f5afbfff700 1 -- 172.21.15.102:0/1887457542 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f5b00001990 con 0x7f5b0412e330 2021-05-25T12:06:17.707 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.686+0000 7f5b0b861700 5 monclient: authenticate success, global_id 204147 2021-05-25T12:06:17.707 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.686+0000 7f5afbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:17.707 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.686+0000 7f5afbfff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:06:17.708 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.686+0000 7f5afbfff700 10 monclient: dump: 2021-05-25T12:06:17.708 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:17.708 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:17.708 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:17.708 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:17.709 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:17.709 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:17.709 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:17.709 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:17.709 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:17.710 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:17.710 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.686+0000 7f5b0b861700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:06:17.710 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.686+0000 7f5afbfff700 1 -- 172.21.15.102:0/1887457542 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5b00001b80 con 0x7f5b0412e330 2021-05-25T12:06:17.710 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.686+0000 7f5afbfff700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:06:17.710 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.686+0000 7f5afbfff700 1 -- 172.21.15.102:0/1887457542 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f5b00001f10 con 0x7f5b0412e330 2021-05-25T12:06:17.711 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.686+0000 7f5b0b861700 10 monclient: get_monmap_and_config success 2021-05-25T12:06:17.711 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.686+0000 7f5b0b861700 4 set_mon_vals no callback set 2021-05-25T12:06:17.711 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.686+0000 7f5afbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:17.711 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.686+0000 7f5afbfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:06:17.712 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.686+0000 7f5afbfff700 10 monclient: dump: 2021-05-25T12:06:17.712 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:17.712 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:17.712 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:17.712 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:17.713 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:17.713 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:17.713 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:17.713 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:17.713 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:17.714 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:17.714 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.687+0000 7f5b0b861700 10 monclient: shutdown 2021-05-25T12:06:17.714 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.687+0000 7f5b0b861700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:06:17.714 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.687+0000 7f5b0b861700 1 -- 172.21.15.102:0/1887457542 >> v1:172.21.15.102:6789/0 conn(0x7f5b0412e330 legacy=0x7f5b04130720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:17.714 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.687+0000 7f5b0b861700 1 -- 172.21.15.102:0/1887457542 shutdown_connections 2021-05-25T12:06:17.715 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.687+0000 7f5b0b861700 1 -- 172.21.15.102:0/1887457542 >> 172.21.15.102:0/1887457542 conn(0x7f5b0409b370 msgr2=0x7f5b04097430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:06:17.715 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.687+0000 7f5b0b861700 1 -- 172.21.15.102:0/1887457542 shutdown_connections 2021-05-25T12:06:17.715 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.687+0000 7f5b0b861700 1 -- 172.21.15.102:0/1887457542 wait complete. 2021-05-25T12:06:17.715 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.687+0000 7f5b0b861700 10 monclient: build_initial_monmap 2021-05-25T12:06:17.716 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.687+0000 7f5b0b861700 10 monclient: monmap: 2021-05-25T12:06:17.716 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:06:17.716 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:06:17.716 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:06:17.716 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:06:17.717 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:06:17.717 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:06:17.717 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:06:17.717 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:06:17.717 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:06:17.718 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:17.718 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.687+0000 7f5b0b861700 1 librados: starting msgr at 2021-05-25T12:06:17.718 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.687+0000 7f5b0b861700 1 librados: starting objecter 2021-05-25T12:06:17.718 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.687+0000 7f5b0b861700 1 Processor -- start 2021-05-25T12:06:17.718 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.687+0000 7f5b0b861700 1 -- start start 2021-05-25T12:06:17.719 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.687+0000 7f5b0b861700 1 librados: setting wanted keys 2021-05-25T12:06:17.719 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.687+0000 7f5b0b861700 1 librados: calling monclient init 2021-05-25T12:06:17.719 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.687+0000 7f5b0b861700 10 monclient: init 2021-05-25T12:06:17.719 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.688+0000 7f5b0b861700 10 monclient: _reopen_session rank -1 2021-05-25T12:06:17.720 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.688+0000 7f5b0b861700 10 monclient: _add_conns ranks=[0,2,1] 2021-05-25T12:06:17.720 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.688+0000 7f5b0b861700 10 monclient(hunting): picked mon.noname-a con 0x7f5b04090990 addr v1:172.21.15.102:6789/0 2021-05-25T12:06:17.720 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.688+0000 7f5b0b861700 10 monclient(hunting): picked mon.noname-c con 0x7f5b04090d70 addr v1:172.21.15.102:6790/0 2021-05-25T12:06:17.720 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.688+0000 7f5b0b861700 10 monclient(hunting): picked mon.noname-b con 0x7f5b0412e330 addr v1:172.21.15.181:6789/0 2021-05-25T12:06:17.720 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.688+0000 7f5b0b861700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b04111d90 con 0x7f5b04090990 2021-05-25T12:06:17.721 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.688+0000 7f5b0b861700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b04114510 con 0x7f5b0412e330 2021-05-25T12:06:17.721 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.688+0000 7f5b0b861700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b041147b0 con 0x7f5b04090d70 2021-05-25T12:06:17.721 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.688+0000 7f5b0b861700 10 monclient(hunting): _renew_subs 2021-05-25T12:06:17.721 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.688+0000 7f5b0b861700 10 monclient(hunting): authenticate will time out at 1178.285536s 2021-05-25T12:06:17.721 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.688+0000 7f5b095fd700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f5b04090990 0x7f5b04110f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33286/0 (socket says 172.21.15.102:33286) 2021-05-25T12:06:17.722 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.688+0000 7f5b095fd700 1 -- 172.21.15.102:0/1757233537 learned_addr learned my addr 172.21.15.102:0/1757233537 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:06:17.722 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.688+0000 7f5af97fa700 10 client.?.objecter ms_handle_connect 0x7f5b04090d70 2021-05-25T12:06:17.722 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.688+0000 7f5af97fa700 10 client.?.objecter resend_mon_ops 2021-05-25T12:06:17.722 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.688+0000 7f5af97fa700 10 client.?.objecter ms_handle_connect 0x7f5b04090990 2021-05-25T12:06:17.723 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.688+0000 7f5af97fa700 10 client.?.objecter resend_mon_ops 2021-05-25T12:06:17.723 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.688+0000 7f5af97fa700 1 -- 172.21.15.102:0/1757233537 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1919405423 0 0) 0x7f5b041147b0 con 0x7f5b04090d70 2021-05-25T12:06:17.723 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.688+0000 7f5af97fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:17.723 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.688+0000 7f5af97fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:17.723 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.688+0000 7f5af97fa700 10 monclient(hunting): my global_id is 184490 2021-05-25T12:06:17.724 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.688+0000 7f5af97fa700 1 -- 172.21.15.102:0/1757233537 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ae4004c10 con 0x7f5b04090d70 2021-05-25T12:06:17.724 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.688+0000 7f5af97fa700 1 -- 172.21.15.102:0/1757233537 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2773829896 0 0) 0x7f5b04111d90 con 0x7f5b04090990 2021-05-25T12:06:17.724 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.688+0000 7f5af97fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:17.724 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.688+0000 7f5af97fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:17.725 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.688+0000 7f5af97fa700 10 monclient(hunting): my global_id is 204150 2021-05-25T12:06:17.725 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.688+0000 7f5af97fa700 1 -- 172.21.15.102:0/1757233537 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b041147b0 con 0x7f5b04090990 2021-05-25T12:06:17.725 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.688+0000 7f5af97fa700 10 client.?.objecter ms_handle_connect 0x7f5b0412e330 2021-05-25T12:06:17.725 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.688+0000 7f5af97fa700 10 client.?.objecter resend_mon_ops 2021-05-25T12:06:17.725 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.688+0000 7f5af97fa700 1 -- 172.21.15.102:0/1757233537 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2079753965 0 0) 0x7f5b04114510 con 0x7f5b0412e330 2021-05-25T12:06:17.726 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.689+0000 7f5af97fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:17.726 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.689+0000 7f5af97fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:17.726 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.689+0000 7f5af97fa700 10 monclient(hunting): my global_id is 194314 2021-05-25T12:06:17.726 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.689+0000 7f5af97fa700 1 -- 172.21.15.102:0/1757233537 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b04111d90 con 0x7f5b0412e330 2021-05-25T12:06:17.727 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.689+0000 7f5af97fa700 1 -- 172.21.15.102:0/1757233537 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1160163403 0 0) 0x7f5ae4004c10 con 0x7f5b04090d70 2021-05-25T12:06:17.727 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.689+0000 7f5af97fa700 1 -- 172.21.15.102:0/1757233537 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5b04114510 con 0x7f5b04090d70 2021-05-25T12:06:17.727 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.689+0000 7f5af97fa700 1 -- 172.21.15.102:0/1757233537 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f5aec001900 con 0x7f5b04090d70 2021-05-25T12:06:17.727 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.689+0000 7f5af97fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:17.727 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.689+0000 7f5af97fa700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:06:17.728 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.689+0000 7f5af97fa700 10 monclient(hunting): dump: 2021-05-25T12:06:17.728 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:17.728 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:17.728 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:17.729 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:17.729 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:17.729 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:17.729 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:17.729 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:17.730 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:17.730 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:17.730 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.689+0000 7f5af97fa700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:06:17.730 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.689+0000 7f5af97fa700 1 -- 172.21.15.102:0/1757233537 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1753088107 0 0) 0x7f5b041147b0 con 0x7f5b04090990 2021-05-25T12:06:17.730 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.689+0000 7f5af97fa700 1 -- 172.21.15.102:0/1757233537 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ae4004c10 con 0x7f5b04090990 2021-05-25T12:06:17.731 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.689+0000 7f5af97fa700 1 -- 172.21.15.102:0/1757233537 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f5af4001980 con 0x7f5b04090990 2021-05-25T12:06:17.731 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.689+0000 7f5af97fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:17.731 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.689+0000 7f5af97fa700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2021-05-25T12:06:17.731 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.689+0000 7f5af97fa700 10 monclient(hunting): dump: 2021-05-25T12:06:17.732 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:17.732 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:17.732 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:17.732 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:17.733 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:17.733 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:17.733 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:17.733 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:17.734 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:17.734 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:17.734 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.689+0000 7f5af97fa700 1 -- 172.21.15.102:0/1757233537 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2583857532 0 0) 0x7f5b04111d90 con 0x7f5b0412e330 2021-05-25T12:06:17.734 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.689+0000 7f5af97fa700 1 -- 172.21.15.102:0/1757233537 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5b041147b0 con 0x7f5b0412e330 2021-05-25T12:06:17.735 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.689+0000 7f5af97fa700 1 -- 172.21.15.102:0/1757233537 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 484493396 0 0) 0x7f5b04114510 con 0x7f5b04090d70 2021-05-25T12:06:17.735 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.689+0000 7f5af97fa700 1 -- 172.21.15.102:0/1757233537 >> v1:172.21.15.181:6789/0 conn(0x7f5b0412e330 legacy=0x7f5b04140f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:17.735 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.689+0000 7f5af97fa700 1 -- 172.21.15.102:0/1757233537 >> v1:172.21.15.102:6789/0 conn(0x7f5b04090990 legacy=0x7f5b04110f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:17.735 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.689+0000 7f5af97fa700 10 monclient: _finish_hunting 0 2021-05-25T12:06:17.735 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.689+0000 7f5af97fa700 1 monclient: found mon.c 2021-05-25T12:06:17.736 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.689+0000 7f5af97fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:06:17.736 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.689+0000 7f5af97fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:06:17.736 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.689+0000 7f5af97fa700 1 -- 172.21.15.102:0/1757233537 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b04091fe0 con 0x7f5b04090d70 2021-05-25T12:06:17.736 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.689+0000 7f5af97fa700 10 monclient: _finish_auth 0 2021-05-25T12:06:17.737 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.689+0000 7f5af97fa700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:06:17.737 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.689+0000 7f5b0b861700 5 monclient: authenticate success, global_id 184490 2021-05-25T12:06:17.737 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.689+0000 7f5b0b861700 10 monclient: _renew_subs 2021-05-25T12:06:17.737 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.689+0000 7f5b0b861700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:06:17.738 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.689+0000 7f5b0b861700 1 -- 172.21.15.102:0/1757233537 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b04112660 con 0x7f5b04090d70 2021-05-25T12:06:17.738 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.689+0000 7f5af97fa700 1 -- 172.21.15.102:0/1757233537 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5aec001cf0 con 0x7f5b04090d70 2021-05-25T12:06:17.738 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.689+0000 7f5af97fa700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:06:17.738 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.689+0000 7f5b0b861700 10 client.184490.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:06:17.738 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.689+0000 7f5b0b861700 10 monclient: _renew_subs 2021-05-25T12:06:17.739 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.689+0000 7f5af97fa700 1 -- 172.21.15.102:0/1757233537 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f5aec004980 con 0x7f5b04090d70 2021-05-25T12:06:17.739 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.689+0000 7f5b0b861700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:06:17.739 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.689+0000 7f5b0b861700 1 -- 172.21.15.102:0/1757233537 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b04114b70 con 0x7f5b04090d70 2021-05-25T12:06:17.739 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.689+0000 7f5b0b861700 1 librados: init done 2021-05-25T12:06:17.740 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.689+0000 7f5af97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:17.740 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.689+0000 7f5afb7fe700 4 set_mon_vals no callback set 2021-05-25T12:06:17.740 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.689+0000 7f5af97fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:06:17.740 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.689+0000 7f5af97fa700 10 monclient: dump: 2021-05-25T12:06:17.740 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:17.741 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:17.741 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:17.741 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:17.741 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:17.741 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:17.742 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:17.742 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:17.742 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:17.742 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:17.743 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.690+0000 7f5af97fa700 1 -- 172.21.15.102:0/1757233537 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f5aec00dca0 con 0x7f5b04090d70 2021-05-25T12:06:17.743 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.690+0000 7f5b0b861700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:06:17.743 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.690+0000 7f5b0b861700 1 -- 172.21.15.102:0/1757233537 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f5b04114370 con 0x7f5b04066d20 2021-05-25T12:06:17.743 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.690+0000 7f5af97fa700 1 -- 172.21.15.102:0/1757233537 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(195..195 src has 1..195) v4 ==== 6620+0+0 (unknown 3265642060 0 0) 0x7f5aec03d970 con 0x7f5b04090d70 2021-05-25T12:06:17.743 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.690+0000 7f5af97fa700 10 client.184490.objecter ms_dispatch 0x7f5b04001550 osd_map(195..195 src has 1..195) v4 2021-05-25T12:06:17.744 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.690+0000 7f5af97fa700 3 client.184490.objecter handle_osd_map got epochs [195,195] > 0 2021-05-25T12:06:17.744 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.690+0000 7f5af97fa700 3 client.184490.objecter handle_osd_map decoding full epoch 195 2021-05-25T12:06:17.744 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.690+0000 7f5af97fa700 20 client.184490.objecter dump_active .. 0 homeless 2021-05-25T12:06:17.744 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.690+0000 7f5af97fa700 10 client.184490.objecter ms_handle_connect 0x7f5b04066d20 2021-05-25T12:06:17.744 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.690+0000 7f5af97fa700 10 client.184490.objecter resend_mon_ops 2021-05-25T12:06:17.745 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.690+0000 7f5af97fa700 1 -- 172.21.15.102:0/1757233537 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 206092190 0 0) 0x7f5b04092940 con 0x7f5b04066d20 2021-05-25T12:06:17.745 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.690+0000 7f5af97fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:17.745 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.690+0000 7f5af97fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:17.745 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.690+0000 7f5af97fa700 10 monclient(hunting): my global_id is 184493 2021-05-25T12:06:17.745 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.690+0000 7f5af97fa700 1 -- 172.21.15.102:0/1757233537 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b04114510 con 0x7f5b04066d20 2021-05-25T12:06:17.746 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.690+0000 7f5b09dfe700 10 monclient: get_auth_request con 0x7f5ae4037900 auth_method 0 2021-05-25T12:06:17.746 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.690+0000 7f5af97fa700 1 -- 172.21.15.102:0/1757233537 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 612466812 0 0) 0x7f5b04114510 con 0x7f5b04066d20 2021-05-25T12:06:17.746 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.690+0000 7f5af97fa700 1 -- 172.21.15.102:0/1757233537 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5b04092940 con 0x7f5b04066d20 2021-05-25T12:06:17.746 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.690+0000 7f5af97fa700 1 -- 172.21.15.102:0/1757233537 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f5af4001f10 con 0x7f5b04066d20 2021-05-25T12:06:17.747 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.690+0000 7f5af97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:17.747 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.690+0000 7f5af97fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:06:17.747 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.690+0000 7f5af97fa700 10 monclient: dump: 2021-05-25T12:06:17.747 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:17.747 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:17.748 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:17.748 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:17.748 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:17.748 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:17.748 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:17.749 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:17.749 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:17.749 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:17.749 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.690+0000 7f5af97fa700 10 client.184490.objecter ms_handle_connect 0x7f5ae4037900 2021-05-25T12:06:17.749 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.690+0000 7f5af97fa700 1 -- 172.21.15.102:0/1757233537 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3868994648 0 0) 0x7f5b04092940 con 0x7f5b04066d20 2021-05-25T12:06:17.750 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.690+0000 7f5af97fa700 1 -- 172.21.15.102:0/1757233537 --> v1:172.21.15.102:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5b04133060 con 0x7f5b04066d20 2021-05-25T12:06:17.750 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.691+0000 7f5af97fa700 1 -- 172.21.15.102:0/1757233537 <== mon.2 v1:172.21.15.102:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7f5af4002500 con 0x7f5b04066d20 2021-05-25T12:06:17.750 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.691+0000 7f5af97fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:06:17.750 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.691+0000 7f5af97fa700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:06:17.750 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.691+0000 7f5af97fa700 1 -- 172.21.15.102:0/1757233537 >> v1:172.21.15.102:6790/0 conn(0x7f5b04066d20 legacy=0x7f5b04067100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:17.751 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.691+0000 7f5af97fa700 1 -- 172.21.15.102:0/1757233537 >> v1:172.21.15.102:6790/0 conn(0x7f5b04066d20 legacy=0x7f5b04067100 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:06:17.753 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.697+0000 7f5b0b861700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:06:17.753 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.697+0000 7f5b0b861700 1 -- 172.21.15.102:0/1757233537 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f5b04114370 con 0x7f5b04027ce0 2021-05-25T12:06:17.753 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.697+0000 7f5af97fa700 10 client.184490.objecter ms_handle_connect 0x7f5b04027ce0 2021-05-25T12:06:17.754 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.697+0000 7f5af97fa700 10 client.184490.objecter resend_mon_ops 2021-05-25T12:06:17.754 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.698+0000 7f5af97fa700 1 -- 172.21.15.102:0/1757233537 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 196994137 0 0) 0x7f5af4002790 con 0x7f5b04027ce0 2021-05-25T12:06:17.754 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.698+0000 7f5af97fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:17.754 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.698+0000 7f5af97fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:17.754 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.698+0000 7f5af97fa700 10 monclient(hunting): my global_id is 184496 2021-05-25T12:06:17.755 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.698+0000 7f5af97fa700 1 -- 172.21.15.102:0/1757233537 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b04092940 con 0x7f5b04027ce0 2021-05-25T12:06:17.755 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.698+0000 7f5af97fa700 1 -- 172.21.15.102:0/1757233537 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3846716316 0 0) 0x7f5b04092940 con 0x7f5b04027ce0 2021-05-25T12:06:17.755 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.698+0000 7f5af97fa700 1 -- 172.21.15.102:0/1757233537 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5af4002790 con 0x7f5b04027ce0 2021-05-25T12:06:17.755 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.698+0000 7f5af97fa700 1 -- 172.21.15.102:0/1757233537 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f5af4005510 con 0x7f5b04027ce0 2021-05-25T12:06:17.756 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.698+0000 7f5af97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:17.756 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.698+0000 7f5af97fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:06:17.756 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.698+0000 7f5af97fa700 10 monclient: dump: 2021-05-25T12:06:17.756 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:17.756 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:17.757 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:17.757 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:17.757 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:17.757 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:17.757 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:17.758 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:17.758 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:17.758 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:17.758 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.698+0000 7f5af97fa700 1 -- 172.21.15.102:0/1757233537 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1365825444 0 0) 0x7f5af4002790 con 0x7f5b04027ce0 2021-05-25T12:06:17.759 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.698+0000 7f5af97fa700 1 -- 172.21.15.102:0/1757233537 --> v1:172.21.15.102:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f5b04059730 con 0x7f5b04027ce0 2021-05-25T12:06:17.759 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.698+0000 7f5af97fa700 1 -- 172.21.15.102:0/1757233537 <== mon.2 v1:172.21.15.102:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4112 (unknown 0 0 322949165) 0x7f5af40077e0 con 0x7f5b04027ce0 2021-05-25T12:06:17.759 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.698+0000 7f5af97fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:06:17.759 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.698+0000 7f5af97fa700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:06:17.760 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.698+0000 7f5af97fa700 1 -- 172.21.15.102:0/1757233537 >> v1:172.21.15.102:6790/0 conn(0x7f5b04027ce0 legacy=0x7f5b04144630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:17.760 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.698+0000 7f5af97fa700 1 -- 172.21.15.102:0/1757233537 >> v1:172.21.15.102:6790/0 conn(0x7f5b04027ce0 legacy=0x7f5b04144630 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:06:17.760 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:06:17.760 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:06:17.761 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:06:17.761 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "leader", 2021-05-25T12:06:17.761 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 52, 2021-05-25T12:06:17.761 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:06:17.762 INFO:teuthology.orchestra.run.smithi102.stdout: 0, 2021-05-25T12:06:17.762 INFO:teuthology.orchestra.run.smithi102.stdout: 1, 2021-05-25T12:06:17.762 INFO:teuthology.orchestra.run.smithi102.stdout: 2 2021-05-25T12:06:17.762 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:06:17.762 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 2, 2021-05-25T12:06:17.763 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:06:17.763 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:06:17.763 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:06:17.763 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:06:17.763 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:06:17.763 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:06:17.764 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:06:17.764 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:06:17.764 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:06:17.764 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:06:17.765 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:06:17.765 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:06:17.765 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:06:17.765 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:06:17.765 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:06:17.766 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:06:17.766 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:06:17.766 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:06:17.766 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:06:17.767 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:06:17.767 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:06:17.767 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:06:17.767 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:06:17.767 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:06:17.768 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:17.768 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:17.768 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:06:17.768 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:06:17.768 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:06:17.769 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:06:17.769 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:06:17.769 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:06:17.769 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:06:17.769 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:06:17.770 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:06:17.770 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:06:17.770 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:06:17.770 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:06:17.771 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:06:17.771 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:06:17.771 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:06:17.771 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:06:17.771 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:06:17.772 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:06:17.772 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:06:17.772 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:06:17.772 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:06:17.772 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:06:17.773 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:06:17.773 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:06:17.773 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:06:17.773 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:06:17.773 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:17.774 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:06:17.774 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:17.774 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:06:17.774 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:06:17.774 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:06:17.775 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:06:17.775 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:17.775 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:06:17.775 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:06:17.776 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:06:17.776 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:17.776 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:17.776 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:17.776 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:06:17.777 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:06:17.777 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:06:17.777 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:06:17.777 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:06:17.777 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:17.777 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:17.778 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:06:17.778 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:06:17.778 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:06:17.778 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:06:17.778 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:17.779 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:06:17.779 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:06:17.779 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:06:17.779 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:17.779 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:17.780 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:17.780 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:06:17.780 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:06:17.780 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:06:17.780 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:06:17.781 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:06:17.781 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:17.781 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:17.781 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:06:17.781 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:06:17.781 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:06:17.782 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:06:17.782 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:17.782 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:06:17.782 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:06:17.782 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:06:17.783 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:17.783 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:17.783 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:17.783 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:06:17.783 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:06:17.784 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:06:17.784 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:06:17.784 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:06:17.784 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:17.784 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:17.785 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:17.785 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:06:17.785 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:06:17.785 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:17.785 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:06:17.786 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:06:17.786 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:06:17.786 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:17.786 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:06:17.787 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": [ 2021-05-25T12:06:17.787 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:17.787 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:06:17.787 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:06:17.787 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 2 2021-05-25T12:06:17.788 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:17.788 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:06:17.788 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:06:17.788 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:17.788 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:06:17.789 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:06:17.789 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 3 2021-05-25T12:06:17.789 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:17.789 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:06:17.790 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": [ 2021-05-25T12:06:17.790 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:17.790 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:06:17.790 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:06:17.790 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:06:17.791 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:17.791 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:17.791 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:17.791 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:06:17.791 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:06:17.793 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.698+0000 7f5b0b861700 10 librados: watch_flush enter 2021-05-25T12:06:17.794 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.698+0000 7f5b0b861700 10 librados: watch_flush exit 2021-05-25T12:06:17.794 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.698+0000 7f5b0b861700 20 client.184490.objecter shutdown clearing up homeless session... 2021-05-25T12:06:17.794 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.698+0000 7f5b0b861700 10 client.184490.objecter successfully canceled tick 2021-05-25T12:06:17.794 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.698+0000 7f5b0b861700 1 -- 172.21.15.102:0/1757233537 >> v1:172.21.15.181:6812/29582 conn(0x7f5ae4037900 legacy=0x7f5ae4039dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:17.795 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.698+0000 7f5b0b861700 10 monclient: shutdown 2021-05-25T12:06:17.795 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.698+0000 7f5b0b861700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:06:17.795 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.698+0000 7f5b0b861700 1 -- 172.21.15.102:0/1757233537 >> v1:172.21.15.102:6790/0 conn(0x7f5b04090d70 legacy=0x7f5b04111680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:17.795 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.698+0000 7f5b09dfe700 1 -- 172.21.15.102:0/1757233537 reap_dead start 2021-05-25T12:06:17.796 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.698+0000 7f5b09dfe700 1 -- 172.21.15.102:0/1757233537 reap_dead start 2021-05-25T12:06:17.796 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.698+0000 7f5b0b861700 1 -- 172.21.15.102:0/1757233537 shutdown_connections 2021-05-25T12:06:17.796 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.698+0000 7f5b0b861700 1 -- 172.21.15.102:0/1757233537 >> 172.21.15.102:0/1757233537 conn(0x7f5b0409b370 msgr2=0x7f5b04093a60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:06:17.796 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.699+0000 7f5b0b861700 1 -- 172.21.15.102:0/1757233537 shutdown_connections 2021-05-25T12:06:17.797 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.699+0000 7f5b0b861700 1 -- 172.21.15.102:0/1757233537 wait complete. 2021-05-25T12:06:17.797 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.699+0000 7f5b0b861700 1 librados: shutdown 2021-05-25T12:06:17.797 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2021-05-25T12:06:17.908 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.907+0000 7f9517787700 10 monclient: get_monmap_and_config 2021-05-25T12:06:17.908 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.907+0000 7f9517787700 10 monclient: build_initial_monmap 2021-05-25T12:06:17.909 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.907+0000 7f9517787700 10 monclient: monmap: 2021-05-25T12:06:17.909 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:06:17.909 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:17.909 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:06:17.908025+0000 2021-05-25T12:06:17.909 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:06:17.908025+0000 2021-05-25T12:06:17.910 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:06:17.910 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:17.910 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:06:17.910 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:06:17.911 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:06:17.911 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:17.925 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.907+0000 7f9517787700 1 Processor -- start 2021-05-25T12:06:17.925 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.907+0000 7f9517787700 1 -- start start 2021-05-25T12:06:17.926 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.907+0000 7f9517787700 10 monclient: init 2021-05-25T12:06:17.926 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.908+0000 7f9517787700 10 monclient: _reopen_session rank -1 2021-05-25T12:06:17.926 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.908+0000 7f9517787700 10 monclient: _add_conns ranks=[1,0,2] 2021-05-25T12:06:17.926 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.908+0000 7f9517787700 10 monclient(hunting): picked mon.noname-c con 0x7f951009a5e0 addr v1:172.21.15.181:6789/0 2021-05-25T12:06:17.927 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.908+0000 7f9517787700 10 monclient(hunting): picked mon.noname-a con 0x7f9510063dc0 addr v1:172.21.15.102:6789/0 2021-05-25T12:06:17.927 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.908+0000 7f9517787700 10 monclient(hunting): picked mon.noname-b con 0x7f95101325c0 addr v1:172.21.15.102:6790/0 2021-05-25T12:06:17.927 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.908+0000 7f9517787700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9510099fe0 con 0x7f9510063dc0 2021-05-25T12:06:17.927 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.908+0000 7f9517787700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f951009a1e0 con 0x7f951009a5e0 2021-05-25T12:06:17.928 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.908+0000 7f9517787700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f951009b400 con 0x7f95101325c0 2021-05-25T12:06:17.928 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.908+0000 7f9517787700 10 monclient(hunting): _renew_subs 2021-05-25T12:06:17.928 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.908+0000 7f9517787700 10 monclient(hunting): authenticate will time out at 1178.505322s 2021-05-25T12:06:17.928 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.908+0000 7f9515d24700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f95101325c0 0x7f95101349b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:56410/0 (socket says 172.21.15.102:56410) 2021-05-25T12:06:17.928 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.908+0000 7f9514d22700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f9510063dc0 0x7f9510064240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33300/0 (socket says 172.21.15.102:33300) 2021-05-25T12:06:17.929 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.908+0000 7f9515d24700 1 -- 172.21.15.102:0/917470837 learned_addr learned my addr 172.21.15.102:0/917470837 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:06:17.929 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.908+0000 7f9507fff700 1 -- 172.21.15.102:0/917470837 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1257172876 0 0) 0x7f951009b400 con 0x7f95101325c0 2021-05-25T12:06:17.929 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.908+0000 7f9507fff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:17.929 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.908+0000 7f9507fff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:17.929 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.908+0000 7f9507fff700 10 monclient(hunting): my global_id is 184499 2021-05-25T12:06:17.930 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.908+0000 7f9507fff700 1 -- 172.21.15.102:0/917470837 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94fc003270 con 0x7f95101325c0 2021-05-25T12:06:17.930 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.908+0000 7f9507fff700 1 -- 172.21.15.102:0/917470837 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3670253753 0 0) 0x7f9510099fe0 con 0x7f9510063dc0 2021-05-25T12:06:17.930 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.908+0000 7f9507fff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:17.930 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.908+0000 7f9507fff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:17.931 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.908+0000 7f9507fff700 10 monclient(hunting): my global_id is 204153 2021-05-25T12:06:17.931 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.908+0000 7f9507fff700 1 -- 172.21.15.102:0/917470837 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f951009b400 con 0x7f9510063dc0 2021-05-25T12:06:17.931 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.908+0000 7f9507fff700 1 -- 172.21.15.102:0/917470837 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3688511704 0 0) 0x7f951009a1e0 con 0x7f951009a5e0 2021-05-25T12:06:17.931 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.908+0000 7f9507fff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:17.931 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.908+0000 7f9507fff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:17.932 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.908+0000 7f9507fff700 10 monclient(hunting): my global_id is 194317 2021-05-25T12:06:17.932 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.908+0000 7f9507fff700 1 -- 172.21.15.102:0/917470837 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9510099fe0 con 0x7f951009a5e0 2021-05-25T12:06:17.932 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.908+0000 7f9507fff700 1 -- 172.21.15.102:0/917470837 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3320251708 0 0) 0x7f94fc003270 con 0x7f95101325c0 2021-05-25T12:06:17.932 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.908+0000 7f9507fff700 1 -- 172.21.15.102:0/917470837 >> v1:172.21.15.181:6789/0 conn(0x7f951009a5e0 legacy=0x7f95100636b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:17.933 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.908+0000 7f9507fff700 1 -- 172.21.15.102:0/917470837 >> v1:172.21.15.102:6789/0 conn(0x7f9510063dc0 legacy=0x7f9510064240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:17.933 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.908+0000 7f9507fff700 10 monclient: _finish_hunting 0 2021-05-25T12:06:17.933 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.908+0000 7f9507fff700 1 monclient: found mon.noname-b 2021-05-25T12:06:17.933 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.908+0000 7f9507fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:06:17.933 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.908+0000 7f9507fff700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T12:06:17.934 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.908+0000 7f9507fff700 1 -- 172.21.15.102:0/917470837 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95100963f0 con 0x7f95101325c0 2021-05-25T12:06:17.934 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.908+0000 7f9507fff700 10 monclient: _finish_auth 0 2021-05-25T12:06:17.934 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.908+0000 7f9507fff700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:06:17.934 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.908+0000 7f9507fff700 1 -- 172.21.15.102:0/917470837 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f950c0019d0 con 0x7f95101325c0 2021-05-25T12:06:17.935 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.908+0000 7f9517787700 5 monclient: authenticate success, global_id 184499 2021-05-25T12:06:17.935 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.908+0000 7f9517787700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:06:17.935 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.908+0000 7f9507fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:17.935 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.908+0000 7f9507fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:06:17.935 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.908+0000 7f9507fff700 10 monclient: dump: 2021-05-25T12:06:17.936 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:17.936 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:17.936 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:17.936 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:17.937 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:17.937 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:17.937 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:17.937 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:17.937 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:17.938 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:17.938 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.909+0000 7f9517787700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:06:17.938 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.909+0000 7f9507fff700 1 -- 172.21.15.102:0/917470837 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f950c001bf0 con 0x7f95101325c0 2021-05-25T12:06:17.938 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.909+0000 7f9507fff700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:06:17.938 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.909+0000 7f9507fff700 1 -- 172.21.15.102:0/917470837 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f950c001f80 con 0x7f95101325c0 2021-05-25T12:06:17.939 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.909+0000 7f9517787700 10 monclient: get_monmap_and_config success 2021-05-25T12:06:17.939 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.909+0000 7f9517787700 4 set_mon_vals no callback set 2021-05-25T12:06:17.939 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.909+0000 7f9507fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:17.939 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.909+0000 7f9507fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:06:17.940 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.909+0000 7f9507fff700 10 monclient: dump: 2021-05-25T12:06:17.940 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:17.940 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:17.940 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:17.940 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:17.941 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:17.941 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:17.941 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:17.941 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:17.941 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:17.942 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:17.942 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.909+0000 7f9517787700 10 monclient: shutdown 2021-05-25T12:06:17.942 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.909+0000 7f9517787700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:06:17.942 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.909+0000 7f9517787700 1 -- 172.21.15.102:0/917470837 >> v1:172.21.15.102:6790/0 conn(0x7f95101325c0 legacy=0x7f95101349b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:17.943 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.909+0000 7f9517787700 1 -- 172.21.15.102:0/917470837 shutdown_connections 2021-05-25T12:06:17.943 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.909+0000 7f9517787700 1 -- 172.21.15.102:0/917470837 >> 172.21.15.102:0/917470837 conn(0x7f951009bc30 msgr2=0x7f951009c040 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:06:17.943 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.909+0000 7f9517787700 1 -- 172.21.15.102:0/917470837 shutdown_connections 2021-05-25T12:06:17.943 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.909+0000 7f9517787700 1 -- 172.21.15.102:0/917470837 wait complete. 2021-05-25T12:06:17.943 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.910+0000 7f9517787700 10 monclient: build_initial_monmap 2021-05-25T12:06:17.944 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.910+0000 7f9517787700 10 monclient: monmap: 2021-05-25T12:06:17.944 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:06:17.944 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:06:17.944 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:06:17.945 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:06:17.945 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:06:17.945 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:06:17.945 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:06:17.945 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:06:17.946 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:06:17.946 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:17.946 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.910+0000 7f9517787700 1 librados: starting msgr at 2021-05-25T12:06:17.946 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.910+0000 7f9517787700 1 librados: starting objecter 2021-05-25T12:06:17.947 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.910+0000 7f9517787700 1 Processor -- start 2021-05-25T12:06:17.947 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.910+0000 7f9517787700 1 -- start start 2021-05-25T12:06:17.947 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.910+0000 7f9517787700 1 librados: setting wanted keys 2021-05-25T12:06:17.947 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.910+0000 7f9517787700 1 librados: calling monclient init 2021-05-25T12:06:17.947 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.910+0000 7f9517787700 10 monclient: init 2021-05-25T12:06:17.948 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.910+0000 7f9517787700 10 monclient: _reopen_session rank -1 2021-05-25T12:06:17.948 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.910+0000 7f9517787700 10 monclient: _add_conns ranks=[0,2,1] 2021-05-25T12:06:17.948 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.910+0000 7f9517787700 10 monclient(hunting): picked mon.noname-a con 0x7f9510063dc0 addr v1:172.21.15.102:6789/0 2021-05-25T12:06:17.948 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.910+0000 7f9517787700 10 monclient(hunting): picked mon.noname-c con 0x7f951009a5e0 addr v1:172.21.15.102:6790/0 2021-05-25T12:06:17.948 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.910+0000 7f9517787700 10 monclient(hunting): picked mon.noname-b con 0x7f95101325c0 addr v1:172.21.15.181:6789/0 2021-05-25T12:06:17.949 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.911+0000 7f9517787700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9510094fe0 con 0x7f9510063dc0 2021-05-25T12:06:17.949 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.911+0000 7f9517787700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95101154a0 con 0x7f95101325c0 2021-05-25T12:06:17.949 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.911+0000 7f9517787700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9510116420 con 0x7f951009a5e0 2021-05-25T12:06:17.949 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.911+0000 7f9517787700 10 monclient(hunting): _renew_subs 2021-05-25T12:06:17.950 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.911+0000 7f9517787700 10 monclient(hunting): authenticate will time out at 1178.508031s 2021-05-25T12:06:17.950 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.911+0000 7f9514d22700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f951009a5e0 0x7f9510140470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:56416/0 (socket says 172.21.15.102:56416) 2021-05-25T12:06:17.950 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.911+0000 7f9514d22700 1 -- 172.21.15.102:0/2510808106 learned_addr learned my addr 172.21.15.102:0/2510808106 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:06:17.950 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.911+0000 7f95057fa700 10 client.?.objecter ms_handle_connect 0x7f951009a5e0 2021-05-25T12:06:17.950 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.911+0000 7f95057fa700 10 client.?.objecter resend_mon_ops 2021-05-25T12:06:17.951 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.911+0000 7f95057fa700 10 client.?.objecter ms_handle_connect 0x7f9510063dc0 2021-05-25T12:06:17.951 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.911+0000 7f95057fa700 10 client.?.objecter resend_mon_ops 2021-05-25T12:06:17.951 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.911+0000 7f95057fa700 1 -- 172.21.15.102:0/2510808106 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2071711558 0 0) 0x7f9510116420 con 0x7f951009a5e0 2021-05-25T12:06:17.951 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.911+0000 7f95057fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:17.952 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.911+0000 7f95057fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:17.952 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.911+0000 7f95057fa700 10 monclient(hunting): my global_id is 184502 2021-05-25T12:06:17.952 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.911+0000 7f95057fa700 1 -- 172.21.15.102:0/2510808106 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94f0004c10 con 0x7f951009a5e0 2021-05-25T12:06:17.952 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.911+0000 7f95057fa700 10 client.?.objecter ms_handle_connect 0x7f95101325c0 2021-05-25T12:06:17.952 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.911+0000 7f95057fa700 10 client.?.objecter resend_mon_ops 2021-05-25T12:06:17.953 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.911+0000 7f95057fa700 1 -- 172.21.15.102:0/2510808106 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 245674037 0 0) 0x7f9510094fe0 con 0x7f9510063dc0 2021-05-25T12:06:17.953 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.911+0000 7f95057fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:17.953 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.911+0000 7f95057fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:17.953 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.911+0000 7f95057fa700 10 monclient(hunting): my global_id is 204156 2021-05-25T12:06:17.953 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.911+0000 7f95057fa700 1 -- 172.21.15.102:0/2510808106 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9510116420 con 0x7f9510063dc0 2021-05-25T12:06:17.954 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.911+0000 7f95057fa700 1 -- 172.21.15.102:0/2510808106 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1631215998 0 0) 0x7f95101154a0 con 0x7f95101325c0 2021-05-25T12:06:17.954 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.911+0000 7f95057fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:17.954 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.911+0000 7f95057fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:17.954 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.911+0000 7f95057fa700 10 monclient(hunting): my global_id is 194320 2021-05-25T12:06:17.955 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.911+0000 7f95057fa700 1 -- 172.21.15.102:0/2510808106 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9510094fe0 con 0x7f95101325c0 2021-05-25T12:06:17.955 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.911+0000 7f95057fa700 1 -- 172.21.15.102:0/2510808106 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2957092624 0 0) 0x7f94f0004c10 con 0x7f951009a5e0 2021-05-25T12:06:17.955 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.911+0000 7f95057fa700 1 -- 172.21.15.102:0/2510808106 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f95101154a0 con 0x7f951009a5e0 2021-05-25T12:06:17.955 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.911+0000 7f95057fa700 1 -- 172.21.15.102:0/2510808106 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f94f80019d0 con 0x7f951009a5e0 2021-05-25T12:06:17.956 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.911+0000 7f95057fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:17.956 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.911+0000 7f95057fa700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:06:17.956 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.911+0000 7f95057fa700 10 monclient(hunting): dump: 2021-05-25T12:06:17.956 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:17.956 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:17.957 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:17.957 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:17.957 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:17.957 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:17.958 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:17.958 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:17.958 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:17.958 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:17.958 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.911+0000 7f95057fa700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:06:17.959 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.911+0000 7f95057fa700 1 -- 172.21.15.102:0/2510808106 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 606360693 0 0) 0x7f9510116420 con 0x7f9510063dc0 2021-05-25T12:06:17.959 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.911+0000 7f95057fa700 1 -- 172.21.15.102:0/2510808106 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f94f0004c10 con 0x7f9510063dc0 2021-05-25T12:06:17.959 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.911+0000 7f95057fa700 1 -- 172.21.15.102:0/2510808106 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f9500001880 con 0x7f9510063dc0 2021-05-25T12:06:17.959 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.911+0000 7f95057fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:17.959 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.911+0000 7f95057fa700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2021-05-25T12:06:17.960 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.911+0000 7f95057fa700 10 monclient(hunting): dump: 2021-05-25T12:06:17.960 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:17.960 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:17.960 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:17.961 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:17.961 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:17.961 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:17.961 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:17.961 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:17.962 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:17.962 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:17.962 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.911+0000 7f95057fa700 1 -- 172.21.15.102:0/2510808106 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3222396686 0 0) 0x7f9510094fe0 con 0x7f95101325c0 2021-05-25T12:06:17.962 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.911+0000 7f95057fa700 1 -- 172.21.15.102:0/2510808106 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9510116420 con 0x7f95101325c0 2021-05-25T12:06:17.963 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.911+0000 7f95057fa700 1 -- 172.21.15.102:0/2510808106 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f950c001990 con 0x7f95101325c0 2021-05-25T12:06:17.963 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.911+0000 7f95057fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:17.963 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.911+0000 7f95057fa700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2021-05-25T12:06:17.963 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.911+0000 7f95057fa700 10 monclient(hunting): dump: 2021-05-25T12:06:17.963 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:17.964 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:17.964 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:17.964 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:17.964 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:17.964 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:17.965 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:17.965 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:17.965 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:17.965 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:17.966 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.911+0000 7f95057fa700 1 -- 172.21.15.102:0/2510808106 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 156632699 0 0) 0x7f95101154a0 con 0x7f951009a5e0 2021-05-25T12:06:17.966 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.911+0000 7f95057fa700 1 -- 172.21.15.102:0/2510808106 >> v1:172.21.15.181:6789/0 conn(0x7f95101325c0 legacy=0x7f9510143ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:17.966 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.911+0000 7f95057fa700 1 -- 172.21.15.102:0/2510808106 >> v1:172.21.15.102:6789/0 conn(0x7f9510063dc0 legacy=0x7f951012bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:17.966 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.911+0000 7f95057fa700 10 monclient: _finish_hunting 0 2021-05-25T12:06:17.966 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.911+0000 7f95057fa700 1 monclient: found mon.c 2021-05-25T12:06:17.967 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.911+0000 7f95057fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:06:17.967 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.911+0000 7f95057fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:06:17.967 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.911+0000 7f95057fa700 1 -- 172.21.15.102:0/2510808106 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95101166c0 con 0x7f951009a5e0 2021-05-25T12:06:17.967 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.911+0000 7f95057fa700 10 monclient: _finish_auth 0 2021-05-25T12:06:17.968 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.911+0000 7f95057fa700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:06:17.968 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.911+0000 7f9517787700 5 monclient: authenticate success, global_id 184502 2021-05-25T12:06:17.968 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.911+0000 7f9517787700 10 monclient: _renew_subs 2021-05-25T12:06:17.968 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.911+0000 7f9517787700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:06:17.968 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.911+0000 7f9517787700 1 -- 172.21.15.102:0/2510808106 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f951012c480 con 0x7f951009a5e0 2021-05-25T12:06:17.969 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.911+0000 7f9517787700 10 client.184502.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:06:17.969 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.911+0000 7f9517787700 10 monclient: _renew_subs 2021-05-25T12:06:17.969 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.911+0000 7f9517787700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:06:17.969 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.911+0000 7f95057fa700 1 -- 172.21.15.102:0/2510808106 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f94f80047e0 con 0x7f951009a5e0 2021-05-25T12:06:17.969 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.911+0000 7f9517787700 1 -- 172.21.15.102:0/2510808106 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f951013bbf0 con 0x7f951009a5e0 2021-05-25T12:06:17.970 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.911+0000 7f9517787700 1 librados: init done 2021-05-25T12:06:17.970 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.911+0000 7f95057fa700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:06:17.970 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.911+0000 7f95057fa700 1 -- 172.21.15.102:0/2510808106 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f94f8004af0 con 0x7f951009a5e0 2021-05-25T12:06:17.970 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.911+0000 7f95057fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:17.971 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.911+0000 7f95057fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:06:17.971 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.911+0000 7f95057fa700 10 monclient: dump: 2021-05-25T12:06:17.971 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:17.971 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:17.971 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:17.972 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:17.972 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:17.972 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:17.972 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:17.972 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:17.973 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:17.973 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:17.973 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.911+0000 7f95077fe700 4 set_mon_vals no callback set 2021-05-25T12:06:17.973 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.912+0000 7f95057fa700 1 -- 172.21.15.102:0/2510808106 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f94f800de10 con 0x7f951009a5e0 2021-05-25T12:06:17.974 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.912+0000 7f95057fa700 1 -- 172.21.15.102:0/2510808106 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(196..196 src has 1..196) v4 ==== 6330+0+0 (unknown 1059698009 0 0) 0x7f94f803da70 con 0x7f951009a5e0 2021-05-25T12:06:17.974 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.912+0000 7f95057fa700 10 client.184502.objecter ms_dispatch 0x7f9510001550 osd_map(196..196 src has 1..196) v4 2021-05-25T12:06:17.974 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.912+0000 7f95057fa700 3 client.184502.objecter handle_osd_map got epochs [196,196] > 0 2021-05-25T12:06:17.974 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.912+0000 7f95057fa700 3 client.184502.objecter handle_osd_map decoding full epoch 196 2021-05-25T12:06:17.974 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.912+0000 7f95057fa700 20 client.184502.objecter dump_active .. 0 homeless 2021-05-25T12:06:17.975 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.912+0000 7f9517787700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:06:17.975 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.912+0000 7f9517787700 1 -- 172.21.15.102:0/2510808106 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f951012f9d0 con 0x7f951013b2a0 2021-05-25T12:06:17.975 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.912+0000 7f9515523700 10 monclient: get_auth_request con 0x7f94f0033510 auth_method 0 2021-05-25T12:06:17.975 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.913+0000 7f95057fa700 10 client.184502.objecter ms_handle_connect 0x7f951013b2a0 2021-05-25T12:06:17.975 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.913+0000 7f95057fa700 10 client.184502.objecter resend_mon_ops 2021-05-25T12:06:17.976 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.913+0000 7f95057fa700 10 client.184502.objecter ms_handle_connect 0x7f94f0033510 2021-05-25T12:06:17.976 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.913+0000 7f95057fa700 1 -- 172.21.15.102:0/2510808106 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 419105102 0 0) 0x7f951012f9d0 con 0x7f951013b2a0 2021-05-25T12:06:17.976 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.913+0000 7f95057fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:17.976 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.913+0000 7f95057fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:17.977 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.913+0000 7f95057fa700 10 monclient(hunting): my global_id is 194323 2021-05-25T12:06:17.977 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.913+0000 7f95057fa700 1 -- 172.21.15.102:0/2510808106 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95101154a0 con 0x7f951013b2a0 2021-05-25T12:06:17.977 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.913+0000 7f95057fa700 1 -- 172.21.15.102:0/2510808106 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3419060919 0 0) 0x7f95101154a0 con 0x7f951013b2a0 2021-05-25T12:06:17.977 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.913+0000 7f95057fa700 1 -- 172.21.15.102:0/2510808106 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f951012f9d0 con 0x7f951013b2a0 2021-05-25T12:06:17.977 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.913+0000 7f95057fa700 1 -- 172.21.15.102:0/2510808106 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f950c004c40 con 0x7f951013b2a0 2021-05-25T12:06:17.978 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.913+0000 7f95057fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:17.978 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.913+0000 7f95057fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T12:06:17.978 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.913+0000 7f95057fa700 10 monclient: dump: 2021-05-25T12:06:17.978 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:17.979 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:17.979 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:17.979 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:17.979 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:17.979 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:17.980 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:17.980 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:17.980 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:17.980 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:17.980 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.913+0000 7f95057fa700 1 -- 172.21.15.102:0/2510808106 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1122687403 0 0) 0x7f951012f9d0 con 0x7f951013b2a0 2021-05-25T12:06:17.981 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.913+0000 7f95057fa700 1 -- 172.21.15.102:0/2510808106 --> v1:172.21.15.181:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9510099c90 con 0x7f951013b2a0 2021-05-25T12:06:17.981 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.913+0000 7f95057fa700 1 -- 172.21.15.102:0/2510808106 <== mon.1 v1:172.21.15.181:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7f950c004f30 con 0x7f951013b2a0 2021-05-25T12:06:17.981 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.914+0000 7f95057fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:06:17.981 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.914+0000 7f95057fa700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:06:17.982 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.914+0000 7f95057fa700 1 -- 172.21.15.102:0/2510808106 >> v1:172.21.15.181:6789/0 conn(0x7f951013b2a0 legacy=0x7f951012fe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:17.982 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.914+0000 7f95057fa700 1 -- 172.21.15.102:0/2510808106 >> v1:172.21.15.181:6789/0 conn(0x7f951013b2a0 legacy=0x7f951012fe40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:06:17.982 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.920+0000 7f9517787700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:06:17.982 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.920+0000 7f9517787700 1 -- 172.21.15.102:0/2510808106 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f9510059730 con 0x7f9510027ce0 2021-05-25T12:06:17.982 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.920+0000 7f95057fa700 10 client.184502.objecter ms_handle_connect 0x7f9510027ce0 2021-05-25T12:06:17.983 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.920+0000 7f95057fa700 10 client.184502.objecter resend_mon_ops 2021-05-25T12:06:17.983 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.920+0000 7f95057fa700 1 -- 172.21.15.102:0/2510808106 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 427534525 0 0) 0x7f9510059730 con 0x7f9510027ce0 2021-05-25T12:06:17.983 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.920+0000 7f95057fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:17.983 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.920+0000 7f95057fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:17.984 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.920+0000 7f95057fa700 10 monclient(hunting): my global_id is 194326 2021-05-25T12:06:17.984 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.920+0000 7f95057fa700 1 -- 172.21.15.102:0/2510808106 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f951012f9d0 con 0x7f9510027ce0 2021-05-25T12:06:17.984 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.921+0000 7f95057fa700 1 -- 172.21.15.102:0/2510808106 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4292205449 0 0) 0x7f951012f9d0 con 0x7f9510027ce0 2021-05-25T12:06:17.984 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.921+0000 7f95057fa700 1 -- 172.21.15.102:0/2510808106 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9510059730 con 0x7f9510027ce0 2021-05-25T12:06:17.984 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.921+0000 7f95057fa700 1 -- 172.21.15.102:0/2510808106 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f950c005580 con 0x7f9510027ce0 2021-05-25T12:06:17.985 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.921+0000 7f95057fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:17.985 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.921+0000 7f95057fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T12:06:17.985 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.921+0000 7f95057fa700 10 monclient: dump: 2021-05-25T12:06:17.985 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:17.985 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:17.986 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:17.986 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:17.986 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:17.986 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:17.987 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:17.987 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:17.987 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:17.987 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:17.987 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.921+0000 7f95057fa700 1 -- 172.21.15.102:0/2510808106 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2838913788 0 0) 0x7f9510059730 con 0x7f9510027ce0 2021-05-25T12:06:17.988 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.921+0000 7f95057fa700 1 -- 172.21.15.102:0/2510808106 --> v1:172.21.15.181:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f9510098170 con 0x7f9510027ce0 2021-05-25T12:06:17.988 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.921+0000 7f95057fa700 1 -- 172.21.15.102:0/2510808106 <== mon.1 v1:172.21.15.181:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3940 (unknown 0 0 3615116401) 0x7f950c006850 con 0x7f9510027ce0 2021-05-25T12:06:17.988 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.921+0000 7f95057fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:06:17.988 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.921+0000 7f95057fa700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:06:17.989 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.921+0000 7f95057fa700 1 -- 172.21.15.102:0/2510808106 >> v1:172.21.15.181:6789/0 conn(0x7f9510027ce0 legacy=0x7f9510130550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:17.989 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.921+0000 7f95057fa700 1 -- 172.21.15.102:0/2510808106 >> v1:172.21.15.181:6789/0 conn(0x7f9510027ce0 legacy=0x7f9510130550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:06:17.989 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:06:17.990 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:06:17.990 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:06:17.990 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "peon", 2021-05-25T12:06:17.990 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 52, 2021-05-25T12:06:17.991 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:06:17.991 INFO:teuthology.orchestra.run.smithi102.stdout: 0, 2021-05-25T12:06:17.991 INFO:teuthology.orchestra.run.smithi102.stdout: 1, 2021-05-25T12:06:17.991 INFO:teuthology.orchestra.run.smithi102.stdout: 2 2021-05-25T12:06:17.991 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:06:17.992 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 3, 2021-05-25T12:06:17.992 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:06:17.992 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:06:17.992 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:06:17.992 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:06:17.993 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:06:17.993 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:06:17.993 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:06:17.993 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:06:17.994 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:06:17.994 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:06:17.994 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:06:17.994 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:06:17.994 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:06:17.995 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:06:17.995 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:06:17.995 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:06:17.995 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:06:17.996 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:06:17.996 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:06:17.996 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:06:17.996 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:06:17.996 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:06:17.997 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:06:17.997 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:06:17.997 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:17.997 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:17.997 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:06:17.998 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:06:17.998 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:06:17.998 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:06:17.998 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:06:17.999 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:06:17.999 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:06:17.999 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:06:17.999 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:06:17.999 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:06:18.000 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:06:18.000 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:06:18.000 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:06:18.000 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:06:18.001 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:06:18.001 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:06:18.001 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:06:18.001 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:06:18.001 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:06:18.002 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:06:18.002 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:06:18.002 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:06:18.002 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:06:18.002 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:06:18.003 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:06:18.003 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:06:18.003 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:18.003 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:06:18.004 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:18.004 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:06:18.004 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:06:18.004 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:06:18.004 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:06:18.005 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:18.005 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:06:18.005 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:06:18.005 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:06:18.006 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:18.006 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:18.006 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:18.006 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:06:18.006 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:06:18.007 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:06:18.007 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:06:18.007 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:06:18.007 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:18.007 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:18.008 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:06:18.008 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:06:18.008 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:06:18.008 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:06:18.009 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:18.009 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:06:18.009 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:06:18.009 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:06:18.009 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:18.010 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:18.010 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:18.010 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:06:18.010 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:06:18.010 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:06:18.011 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:06:18.011 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:06:18.011 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:18.011 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:18.012 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:06:18.012 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:06:18.012 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:06:18.012 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:06:18.012 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:18.013 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:06:18.013 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:06:18.013 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:06:18.013 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:18.013 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:18.014 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:18.014 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:06:18.014 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:06:18.014 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:06:18.015 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:06:18.015 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:06:18.015 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:18.015 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:18.015 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:18.016 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:06:18.016 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:06:18.016 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:18.016 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:06:18.016 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:06:18.017 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:06:18.017 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:18.017 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:06:18.017 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": [ 2021-05-25T12:06:18.017 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:18.018 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:06:18.018 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:06:18.018 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 4 2021-05-25T12:06:18.018 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:18.019 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:06:18.019 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:06:18.019 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:18.019 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:06:18.019 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:06:18.020 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 6 2021-05-25T12:06:18.020 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:18.020 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:18.020 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:18.020 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:06:18.021 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:06:18.023 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.921+0000 7f9517787700 10 librados: watch_flush enter 2021-05-25T12:06:18.023 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.921+0000 7f9517787700 10 librados: watch_flush exit 2021-05-25T12:06:18.023 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.921+0000 7f9517787700 20 client.184502.objecter shutdown clearing up homeless session... 2021-05-25T12:06:18.024 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.921+0000 7f9517787700 10 client.184502.objecter successfully canceled tick 2021-05-25T12:06:18.024 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.921+0000 7f9517787700 1 -- 172.21.15.102:0/2510808106 >> v1:172.21.15.181:6812/29582 conn(0x7f94f0033510 legacy=0x7f94f00359d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:18.024 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.921+0000 7f9517787700 10 monclient: shutdown 2021-05-25T12:06:18.024 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.921+0000 7f9517787700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:06:18.024 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.921+0000 7f9517787700 1 -- 172.21.15.102:0/2510808106 >> v1:172.21.15.102:6790/0 conn(0x7f951009a5e0 legacy=0x7f9510140470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:18.025 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.921+0000 7f9515d24700 1 -- 172.21.15.102:0/2510808106 reap_dead start 2021-05-25T12:06:18.025 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.921+0000 7f9515d24700 1 -- 172.21.15.102:0/2510808106 reap_dead start 2021-05-25T12:06:18.025 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.922+0000 7f9517787700 1 -- 172.21.15.102:0/2510808106 shutdown_connections 2021-05-25T12:06:18.025 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.922+0000 7f9517787700 1 -- 172.21.15.102:0/2510808106 >> 172.21.15.102:0/2510808106 conn(0x7f951009bc30 msgr2=0x7f951012eaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:06:18.025 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.922+0000 7f9517787700 1 -- 172.21.15.102:0/2510808106 shutdown_connections 2021-05-25T12:06:18.026 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.922+0000 7f9517787700 1 -- 172.21.15.102:0/2510808106 wait complete. 2021-05-25T12:06:18.026 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:17.922+0000 7f9517787700 1 librados: shutdown 2021-05-25T12:06:18.026 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2021-05-25T12:06:18.027 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2021-05-25T12:06:18.027 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2021-05-25T12:06:18.027 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2021-05-25T12:06:18.028 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2021-05-25T12:06:18.028 INFO:teuthology.orchestra.run:waiting for 300 2021-05-25T12:06:18.041 INFO:tasks.ceph.mon.b:Stopped 2021-05-25T12:06:18.042 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2021-05-25T12:06:18.042 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2021-05-25T12:06:18.151 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.150+0000 7f95b8596700 10 monclient: get_monmap_and_config 2021-05-25T12:06:18.151 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.150+0000 7f95b8596700 10 monclient: build_initial_monmap 2021-05-25T12:06:18.151 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.150+0000 7f95b8596700 10 monclient: monmap: 2021-05-25T12:06:18.152 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:06:18.152 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:18.152 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:06:18.150948+0000 2021-05-25T12:06:18.152 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:06:18.150948+0000 2021-05-25T12:06:18.153 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:06:18.153 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:18.153 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:06:18.153 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:06:18.153 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:06:18.154 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:18.163 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.150+0000 7f95b8596700 1 Processor -- start 2021-05-25T12:06:18.163 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.150+0000 7f95b8596700 1 -- start start 2021-05-25T12:06:18.164 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.150+0000 7f95b8596700 10 monclient: init 2021-05-25T12:06:18.164 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.151+0000 7f95b8596700 10 monclient: _reopen_session rank -1 2021-05-25T12:06:18.164 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.151+0000 7f95b8596700 10 monclient: _add_conns ranks=[1,2,0] 2021-05-25T12:06:18.164 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.151+0000 7f95b8596700 10 monclient(hunting): picked mon.noname-c con 0x7f95b009a640 addr v1:172.21.15.181:6789/0 2021-05-25T12:06:18.165 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.151+0000 7f95b8596700 10 monclient(hunting): picked mon.noname-b con 0x7f95b0063bb0 addr v1:172.21.15.102:6790/0 2021-05-25T12:06:18.165 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.151+0000 7f95b8596700 10 monclient(hunting): picked mon.noname-a con 0x7f95b01325c0 addr v1:172.21.15.102:6789/0 2021-05-25T12:06:18.165 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.151+0000 7f95b8596700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95b0096b20 con 0x7f95b01325c0 2021-05-25T12:06:18.165 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.151+0000 7f95b8596700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95b009a210 con 0x7f95b009a640 2021-05-25T12:06:18.166 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.151+0000 7f95b8596700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95b009b460 con 0x7f95b0063bb0 2021-05-25T12:06:18.166 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.151+0000 7f95b8596700 10 monclient(hunting): _renew_subs 2021-05-25T12:06:18.166 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.151+0000 7f95b8596700 10 monclient(hunting): authenticate will time out at 1178.748354s 2021-05-25T12:06:18.166 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.151+0000 7f95b5b31700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f95b0063bb0 0x7f95b0064030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:56432/0 (socket says 172.21.15.102:56432) 2021-05-25T12:06:18.166 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.151+0000 7f95b6b33700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f95b01325c0 0x7f95b01349b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33324/0 (socket says 172.21.15.102:33324) 2021-05-25T12:06:18.167 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.151+0000 7f95b5b31700 1 -- 172.21.15.102:0/368555420 learned_addr learned my addr 172.21.15.102:0/368555420 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:06:18.167 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.151+0000 7f95b6332700 1 -- 172.21.15.102:0/368555420 >> v1:172.21.15.181:6789/0 conn(0x7f95b009a640 legacy=0x7f95b00634a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2021-05-25T12:06:18.167 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.151+0000 7f95b5330700 1 -- 172.21.15.102:0/368555420 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3886256796 0 0) 0x7f95b009b460 con 0x7f95b0063bb0 2021-05-25T12:06:18.167 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.151+0000 7f95b5330700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:18.168 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.151+0000 7f95b5330700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:18.168 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.151+0000 7f95b5330700 10 monclient(hunting): my global_id is 184505 2021-05-25T12:06:18.168 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.151+0000 7f95b5330700 1 -- 172.21.15.102:0/368555420 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f959c003270 con 0x7f95b0063bb0 2021-05-25T12:06:18.168 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.151+0000 7f95b5330700 1 -- 172.21.15.102:0/368555420 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1092657518 0 0) 0x7f95b0096b20 con 0x7f95b01325c0 2021-05-25T12:06:18.168 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.151+0000 7f95b5330700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:18.169 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.151+0000 7f95b5330700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:18.169 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.151+0000 7f95b5330700 10 monclient(hunting): my global_id is 204159 2021-05-25T12:06:18.169 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.151+0000 7f95b5330700 1 -- 172.21.15.102:0/368555420 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95b009b460 con 0x7f95b01325c0 2021-05-25T12:06:18.169 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.151+0000 7f95b5330700 1 -- 172.21.15.102:0/368555420 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 171161174 0 0) 0x7f95b009b460 con 0x7f95b01325c0 2021-05-25T12:06:18.170 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.151+0000 7f95b5330700 1 -- 172.21.15.102:0/368555420 >> v1:172.21.15.102:6790/0 conn(0x7f95b0063bb0 legacy=0x7f95b0064030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:18.170 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.151+0000 7f95b5330700 1 -- 172.21.15.102:0/368555420 >> v1:172.21.15.181:6789/0 conn(0x7f95b009a640 legacy=0x7f95b00634a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-05-25T12:06:18.170 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.151+0000 7f95b5330700 10 monclient: _finish_hunting 0 2021-05-25T12:06:18.170 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.151+0000 7f95b5330700 1 monclient: found mon.noname-a 2021-05-25T12:06:18.170 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.151+0000 7f95b5330700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:06:18.171 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.151+0000 7f95b5330700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T12:06:18.171 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.152+0000 7f95b5330700 1 -- 172.21.15.102:0/368555420 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95b00964c0 con 0x7f95b01325c0 2021-05-25T12:06:18.171 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.152+0000 7f95b5330700 10 monclient: _finish_auth 0 2021-05-25T12:06:18.171 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.152+0000 7f95b5330700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:06:18.172 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.152+0000 7f95b8596700 5 monclient: authenticate success, global_id 204159 2021-05-25T12:06:18.172 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.152+0000 7f95b5330700 1 -- 172.21.15.102:0/368555420 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f95ac001970 con 0x7f95b01325c0 2021-05-25T12:06:18.172 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.152+0000 7f95b8596700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:06:18.172 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.152+0000 7f95b5330700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:18.172 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.152+0000 7f95b5330700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:06:18.173 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.152+0000 7f95b5330700 10 monclient: dump: 2021-05-25T12:06:18.173 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:18.173 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:18.173 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:18.174 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:18.174 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:18.174 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:18.174 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:18.174 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:18.175 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:18.175 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:18.175 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.152+0000 7f95b8596700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:06:18.175 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.152+0000 7f95b5330700 1 -- 172.21.15.102:0/368555420 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f95ac001c10 con 0x7f95b01325c0 2021-05-25T12:06:18.176 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.152+0000 7f95b5330700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:06:18.176 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.152+0000 7f95b5330700 1 -- 172.21.15.102:0/368555420 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f95ac001fa0 con 0x7f95b01325c0 2021-05-25T12:06:18.176 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.152+0000 7f95b8596700 10 monclient: get_monmap_and_config success 2021-05-25T12:06:18.176 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.152+0000 7f95b8596700 4 set_mon_vals no callback set 2021-05-25T12:06:18.176 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.152+0000 7f95b5330700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:18.177 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.152+0000 7f95b5330700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:06:18.177 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.152+0000 7f95b5330700 10 monclient: dump: 2021-05-25T12:06:18.177 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:18.177 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:18.177 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:18.178 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:18.178 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:18.178 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:18.178 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:18.179 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:18.179 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:18.179 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:18.179 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.152+0000 7f95b8596700 10 monclient: shutdown 2021-05-25T12:06:18.179 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.152+0000 7f95b8596700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:06:18.180 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.152+0000 7f95b8596700 1 -- 172.21.15.102:0/368555420 >> v1:172.21.15.102:6789/0 conn(0x7f95b01325c0 legacy=0x7f95b01349b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:18.180 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.152+0000 7f95b8596700 1 -- 172.21.15.102:0/368555420 shutdown_connections 2021-05-25T12:06:18.180 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.152+0000 7f95b8596700 1 -- 172.21.15.102:0/368555420 >> 172.21.15.102:0/368555420 conn(0x7f95b009ef50 msgr2=0x7f95b009f360 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:06:18.180 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.152+0000 7f95b8596700 1 -- 172.21.15.102:0/368555420 shutdown_connections 2021-05-25T12:06:18.181 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.152+0000 7f95b8596700 1 -- 172.21.15.102:0/368555420 wait complete. 2021-05-25T12:06:18.181 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.153+0000 7f95b8596700 10 monclient: build_initial_monmap 2021-05-25T12:06:18.181 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.153+0000 7f95b8596700 10 monclient: monmap: 2021-05-25T12:06:18.181 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:06:18.181 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:06:18.182 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:06:18.182 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:06:18.182 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:06:18.182 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:06:18.183 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:06:18.183 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:06:18.183 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:06:18.183 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:18.183 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.153+0000 7f95b8596700 1 librados: starting msgr at 2021-05-25T12:06:18.184 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.153+0000 7f95b8596700 1 librados: starting objecter 2021-05-25T12:06:18.184 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.153+0000 7f95b8596700 1 Processor -- start 2021-05-25T12:06:18.184 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.153+0000 7f95b8596700 1 -- start start 2021-05-25T12:06:18.184 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.153+0000 7f95b8596700 1 librados: setting wanted keys 2021-05-25T12:06:18.184 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.153+0000 7f95b8596700 1 librados: calling monclient init 2021-05-25T12:06:18.185 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.153+0000 7f95b8596700 10 monclient: init 2021-05-25T12:06:18.185 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.153+0000 7f95b8596700 10 monclient: _reopen_session rank -1 2021-05-25T12:06:18.185 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.153+0000 7f95b8596700 10 monclient: _add_conns ranks=[1,0,2] 2021-05-25T12:06:18.185 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.153+0000 7f95b8596700 10 monclient(hunting): picked mon.noname-b con 0x7f95b0063bb0 addr v1:172.21.15.181:6789/0 2021-05-25T12:06:18.186 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.153+0000 7f95b8596700 10 monclient(hunting): picked mon.noname-a con 0x7f95b009a640 addr v1:172.21.15.102:6789/0 2021-05-25T12:06:18.186 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.153+0000 7f95b8596700 10 monclient(hunting): picked mon.noname-c con 0x7f95b01325c0 addr v1:172.21.15.102:6790/0 2021-05-25T12:06:18.186 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.153+0000 7f95b8596700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95b0065b60 con 0x7f95b009a640 2021-05-25T12:06:18.186 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.153+0000 7f95b8596700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95b0065e00 con 0x7f95b0063bb0 2021-05-25T12:06:18.186 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.153+0000 7f95b8596700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95b00660a0 con 0x7f95b01325c0 2021-05-25T12:06:18.187 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.153+0000 7f95b8596700 10 monclient(hunting): _renew_subs 2021-05-25T12:06:18.187 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.153+0000 7f95b8596700 10 monclient(hunting): authenticate will time out at 1178.750901s 2021-05-25T12:06:18.187 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.153+0000 7f95b5b31700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f95b009a640 0x7f95b0092c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33330/0 (socket says 172.21.15.102:33330) 2021-05-25T12:06:18.187 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.154+0000 7f95b5b31700 1 -- 172.21.15.102:0/2357505798 learned_addr learned my addr 172.21.15.102:0/2357505798 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:06:18.188 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.154+0000 7f95b6332700 1 -- 172.21.15.102:0/2357505798 >> v1:172.21.15.181:6789/0 conn(0x7f95b0063bb0 legacy=0x7f95b0092520 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.181:6789/0 2021-05-25T12:06:18.188 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.154+0000 7f95a67fc700 10 client.?.objecter ms_handle_connect 0x7f95b009a640 2021-05-25T12:06:18.188 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.154+0000 7f95a67fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:06:18.188 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.154+0000 7f95a67fc700 10 client.?.objecter ms_handle_connect 0x7f95b01325c0 2021-05-25T12:06:18.188 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.154+0000 7f95a67fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:06:18.189 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.154+0000 7f95a67fc700 1 -- 172.21.15.102:0/2357505798 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3651826983 0 0) 0x7f95b0065b60 con 0x7f95b009a640 2021-05-25T12:06:18.189 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.154+0000 7f95a67fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:18.189 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.154+0000 7f95a67fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:18.189 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.154+0000 7f95a67fc700 10 monclient(hunting): my global_id is 204162 2021-05-25T12:06:18.190 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.154+0000 7f95a67fc700 1 -- 172.21.15.102:0/2357505798 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9590004c10 con 0x7f95b009a640 2021-05-25T12:06:18.190 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.154+0000 7f95a67fc700 1 -- 172.21.15.102:0/2357505798 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3214396324 0 0) 0x7f95b00660a0 con 0x7f95b01325c0 2021-05-25T12:06:18.190 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.154+0000 7f95a67fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:18.190 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.154+0000 7f95a67fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:18.190 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.154+0000 7f95a67fc700 10 monclient(hunting): my global_id is 184508 2021-05-25T12:06:18.191 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.154+0000 7f95a67fc700 1 -- 172.21.15.102:0/2357505798 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95b0065b60 con 0x7f95b01325c0 2021-05-25T12:06:18.191 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.154+0000 7f95a67fc700 1 -- 172.21.15.102:0/2357505798 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3360016120 0 0) 0x7f9590004c10 con 0x7f95b009a640 2021-05-25T12:06:18.191 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.154+0000 7f95a67fc700 1 -- 172.21.15.102:0/2357505798 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f95b00660a0 con 0x7f95b009a640 2021-05-25T12:06:18.191 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.154+0000 7f95a67fc700 1 -- 172.21.15.102:0/2357505798 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f95a00019f0 con 0x7f95b009a640 2021-05-25T12:06:18.191 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.154+0000 7f95a67fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:18.192 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.154+0000 7f95a67fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:06:18.192 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.154+0000 7f95a67fc700 10 monclient(hunting): dump: 2021-05-25T12:06:18.192 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:18.192 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:18.193 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:18.193 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:18.193 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:18.193 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:18.193 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:18.194 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:18.194 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:18.194 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:18.194 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.154+0000 7f95a67fc700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:06:18.195 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.154+0000 7f95a67fc700 1 -- 172.21.15.102:0/2357505798 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1796660093 0 0) 0x7f95b0065b60 con 0x7f95b01325c0 2021-05-25T12:06:18.195 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.154+0000 7f95a67fc700 1 -- 172.21.15.102:0/2357505798 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9590004c10 con 0x7f95b01325c0 2021-05-25T12:06:18.195 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.154+0000 7f95a67fc700 1 -- 172.21.15.102:0/2357505798 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f95ac001870 con 0x7f95b01325c0 2021-05-25T12:06:18.195 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.154+0000 7f95a67fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:18.195 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.154+0000 7f95a67fc700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2021-05-25T12:06:18.196 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.154+0000 7f95a67fc700 10 monclient(hunting): dump: 2021-05-25T12:06:18.196 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:18.196 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:18.196 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:18.197 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:18.197 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:18.197 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:18.197 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:18.197 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:18.198 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:18.198 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:18.198 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.154+0000 7f95a67fc700 1 -- 172.21.15.102:0/2357505798 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1419419686 0 0) 0x7f95b00660a0 con 0x7f95b009a640 2021-05-25T12:06:18.198 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.154+0000 7f95a67fc700 1 -- 172.21.15.102:0/2357505798 >> v1:172.21.15.102:6790/0 conn(0x7f95b01325c0 legacy=0x7f95b0147ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:18.199 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.154+0000 7f95a67fc700 1 -- 172.21.15.102:0/2357505798 >> v1:172.21.15.181:6789/0 conn(0x7f95b0063bb0 legacy=0x7f95b0092520 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-05-25T12:06:18.199 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.154+0000 7f95a67fc700 10 monclient: _finish_hunting 0 2021-05-25T12:06:18.199 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.154+0000 7f95a67fc700 1 monclient: found mon.a 2021-05-25T12:06:18.199 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.154+0000 7f95a67fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:06:18.199 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.154+0000 7f95a67fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:06:18.200 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.154+0000 7f95a67fc700 1 -- 172.21.15.102:0/2357505798 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95b0093f10 con 0x7f95b009a640 2021-05-25T12:06:18.200 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.154+0000 7f95a67fc700 10 monclient: _finish_auth 0 2021-05-25T12:06:18.200 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.154+0000 7f95a67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:06:18.200 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.154+0000 7f95b8596700 5 monclient: authenticate success, global_id 204162 2021-05-25T12:06:18.201 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.154+0000 7f95b8596700 10 monclient: _renew_subs 2021-05-25T12:06:18.201 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.154+0000 7f95b8596700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:06:18.201 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.154+0000 7f95b8596700 1 -- 172.21.15.102:0/2357505798 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95b0094b30 con 0x7f95b009a640 2021-05-25T12:06:18.201 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.154+0000 7f95b8596700 10 client.204162.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:06:18.201 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.154+0000 7f95b8596700 10 monclient: _renew_subs 2021-05-25T12:06:18.202 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.154+0000 7f95b8596700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:06:18.202 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.154+0000 7f95b8596700 1 -- 172.21.15.102:0/2357505798 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f95b0066ba0 con 0x7f95b009a640 2021-05-25T12:06:18.202 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.154+0000 7f95b8596700 1 librados: init done 2021-05-25T12:06:18.202 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.154+0000 7f95a67fc700 1 -- 172.21.15.102:0/2357505798 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f95a0001d30 con 0x7f95b009a640 2021-05-25T12:06:18.203 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.154+0000 7f95a67fc700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:06:18.203 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.154+0000 7f95a67fc700 1 -- 172.21.15.102:0/2357505798 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f95a0004a00 con 0x7f95b009a640 2021-05-25T12:06:18.203 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.154+0000 7f95a67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:18.203 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.154+0000 7f95b4b2f700 4 set_mon_vals no callback set 2021-05-25T12:06:18.203 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.154+0000 7f95a67fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:06:18.204 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.154+0000 7f95a67fc700 10 monclient: dump: 2021-05-25T12:06:18.204 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:18.204 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:18.204 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:18.205 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:18.205 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:18.205 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:18.205 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:18.205 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:18.206 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:18.206 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:18.206 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.155+0000 7f95a67fc700 1 -- 172.21.15.102:0/2357505798 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f95a000dda0 con 0x7f95b009a640 2021-05-25T12:06:18.206 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.155+0000 7f95b8596700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:06:18.207 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.155+0000 7f95b8596700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:06:18.207 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.155+0000 7f95b8596700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:06:18.207 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.155+0000 7f95b8596700 1 -- 172.21.15.102:0/2357505798 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9594004db0 con 0x7f95b009a640 2021-05-25T12:06:18.207 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.155+0000 7f95a67fc700 1 -- 172.21.15.102:0/2357505798 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(196..196 src has 1..196) v4 ==== 6330+0+0 (unknown 1059698009 0 0) 0x7f95a003da70 con 0x7f95b009a640 2021-05-25T12:06:18.207 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.155+0000 7f95a67fc700 10 client.204162.objecter ms_dispatch 0x7f95b0001550 osd_map(196..196 src has 1..196) v4 2021-05-25T12:06:18.208 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.155+0000 7f95a67fc700 3 client.204162.objecter handle_osd_map got epochs [196,196] > 0 2021-05-25T12:06:18.208 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.155+0000 7f95a67fc700 3 client.204162.objecter handle_osd_map decoding full epoch 196 2021-05-25T12:06:18.208 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.155+0000 7f95a67fc700 20 client.204162.objecter dump_active .. 0 homeless 2021-05-25T12:06:18.208 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.155+0000 7f95b6332700 10 monclient: get_auth_request con 0x7f95900332a0 auth_method 0 2021-05-25T12:06:18.208 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.156+0000 7f95a67fc700 10 client.204162.objecter ms_handle_connect 0x7f95900332a0 2021-05-25T12:06:18.209 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.159+0000 7f95a67fc700 1 -- 172.21.15.102:0/2357505798 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7f95a0012ee0 con 0x7f95b009a640 2021-05-25T12:06:18.209 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.159+0000 7f95a67fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:06:18.209 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.159+0000 7f95a67fc700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:06:18.367 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.365+0000 7f95b8596700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2021-05-25T12:06:18.367 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.365+0000 7f95b8596700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2021-05-25T12:06:18.368 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.365+0000 7f95b8596700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:06:18.368 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.365+0000 7f95b8596700 1 -- 172.21.15.102:0/2357505798 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f9594005060 con 0x7f95b009a640 2021-05-25T12:06:18.368 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.366+0000 7f95a67fc700 1 -- 172.21.15.102:0/2357505798 <== mon.0 v1:172.21.15.102:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1342 (unknown 254746657 0 266507232) 0x7f95a0012ad0 con 0x7f95b009a640 2021-05-25T12:06:18.368 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.366+0000 7f95a67fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2021-05-25T12:06:18.369 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.366+0000 7f95a67fc700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:06:18.369 INFO:teuthology.orchestra.run.smithi102.stdout:{"election_epoch":52,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":3,"features":{"quorum_con":"4540138303579357183","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"20ee91b6-8935-4b98-b868-85c03789ccc4","modified":"2021-05-25T11:59:11.011522Z","created":"2021-05-25T11:59:11.011522Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6790","nonce":0}]},"addr":"172.21.15.102:6790/0","public_addr":"172.21.15.102:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2021-05-25T12:06:18.370 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.367+0000 7f95b8596700 10 librados: watch_flush enter 2021-05-25T12:06:18.371 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.367+0000 7f95b8596700 10 librados: watch_flush exit 2021-05-25T12:06:18.371 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.367+0000 7f95b8596700 20 client.204162.objecter shutdown clearing up homeless session... 2021-05-25T12:06:18.371 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.367+0000 7f95b8596700 10 client.204162.objecter successfully canceled tick 2021-05-25T12:06:18.371 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.367+0000 7f95b8596700 1 -- 172.21.15.102:0/2357505798 >> v1:172.21.15.181:6812/29582 conn(0x7f95900332a0 legacy=0x7f9590035760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:18.372 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.367+0000 7f95b8596700 10 monclient: shutdown 2021-05-25T12:06:18.372 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.367+0000 7f95b8596700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:06:18.372 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.367+0000 7f95b8596700 1 -- 172.21.15.102:0/2357505798 >> v1:172.21.15.102:6789/0 conn(0x7f95b009a640 legacy=0x7f95b0092c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:18.372 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.367+0000 7f95b8596700 1 -- 172.21.15.102:0/2357505798 shutdown_connections 2021-05-25T12:06:18.372 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.367+0000 7f95b8596700 1 -- 172.21.15.102:0/2357505798 >> 172.21.15.102:0/2357505798 conn(0x7f95b009ef50 msgr2=0x7f95b0096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:06:18.373 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.367+0000 7f95b8596700 1 -- 172.21.15.102:0/2357505798 shutdown_connections 2021-05-25T12:06:18.373 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.367+0000 7f95b8596700 1 -- 172.21.15.102:0/2357505798 wait complete. 2021-05-25T12:06:18.373 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:18.367+0000 7f95b8596700 1 librados: shutdown 2021-05-25T12:06:21.381 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2021-05-25T12:06:21.504 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.502+0000 7fe946a0b700 10 monclient: get_monmap_and_config 2021-05-25T12:06:21.504 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.502+0000 7fe946a0b700 10 monclient: build_initial_monmap 2021-05-25T12:06:21.505 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.502+0000 7fe946a0b700 10 monclient: monmap: 2021-05-25T12:06:21.505 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:06:21.505 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:21.505 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:06:21.503451+0000 2021-05-25T12:06:21.505 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:06:21.503451+0000 2021-05-25T12:06:21.506 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:06:21.506 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:21.506 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:06:21.506 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:06:21.507 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:06:21.507 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:21.507 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.503+0000 7fe946a0b700 1 Processor -- start 2021-05-25T12:06:21.507 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.503+0000 7fe946a0b700 1 -- start start 2021-05-25T12:06:21.507 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.503+0000 7fe946a0b700 10 monclient: init 2021-05-25T12:06:21.516 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.503+0000 7fe946a0b700 10 monclient: _reopen_session rank -1 2021-05-25T12:06:21.517 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.503+0000 7fe946a0b700 10 monclient: _add_conns ranks=[1,0,2] 2021-05-25T12:06:21.517 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.503+0000 7fe946a0b700 10 monclient(hunting): picked mon.noname-c con 0x7fe940090990 addr v1:172.21.15.181:6789/0 2021-05-25T12:06:21.517 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.503+0000 7fe946a0b700 10 monclient(hunting): picked mon.noname-a con 0x7fe940090d70 addr v1:172.21.15.102:6789/0 2021-05-25T12:06:21.517 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.503+0000 7fe946a0b700 10 monclient(hunting): picked mon.noname-b con 0x7fe94012e330 addr v1:172.21.15.102:6790/0 2021-05-25T12:06:21.517 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.503+0000 7fe946a0b700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe940096fa0 con 0x7fe940090d70 2021-05-25T12:06:21.518 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.503+0000 7fe946a0b700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9400926a0 con 0x7fe940090990 2021-05-25T12:06:21.518 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.503+0000 7fe946a0b700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe940092940 con 0x7fe94012e330 2021-05-25T12:06:21.518 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.503+0000 7fe946a0b700 10 monclient(hunting): _renew_subs 2021-05-25T12:06:21.518 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.503+0000 7fe946a0b700 10 monclient(hunting): authenticate will time out at 1182.100695s 2021-05-25T12:06:21.519 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.503+0000 7fe93ffff700 1 -- >> v1:172.21.15.181:6789/0 conn(0x7fe940090990 legacy=0x7fe940093240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2021-05-25T12:06:21.519 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.503+0000 7fe93f7fe700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fe940090d70 0x7fe940091200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33348/0 (socket says 172.21.15.102:33348) 2021-05-25T12:06:21.519 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.503+0000 7fe944fa8700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fe94012e330 0x7fe940130720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:56460/0 (socket says 172.21.15.102:56460) 2021-05-25T12:06:21.519 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.503+0000 7fe93f7fe700 1 -- 172.21.15.102:0/577992465 learned_addr learned my addr 172.21.15.102:0/577992465 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:06:21.519 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.504+0000 7fe93effd700 1 -- 172.21.15.102:0/577992465 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2619347575 0 0) 0x7fe940092940 con 0x7fe94012e330 2021-05-25T12:06:21.520 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.504+0000 7fe93effd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:21.520 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.504+0000 7fe93effd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:21.520 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.504+0000 7fe93effd700 10 monclient(hunting): my global_id is 184511 2021-05-25T12:06:21.520 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.504+0000 7fe93effd700 1 -- 172.21.15.102:0/577992465 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe924003270 con 0x7fe94012e330 2021-05-25T12:06:21.520 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.504+0000 7fe93effd700 1 -- 172.21.15.102:0/577992465 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1575158659 0 0) 0x7fe924003270 con 0x7fe94012e330 2021-05-25T12:06:21.521 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.504+0000 7fe93effd700 1 -- 172.21.15.102:0/577992465 >> v1:172.21.15.181:6789/0 conn(0x7fe940090990 legacy=0x7fe940093240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-05-25T12:06:21.521 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.504+0000 7fe93effd700 1 -- 172.21.15.102:0/577992465 >> v1:172.21.15.102:6789/0 conn(0x7fe940090d70 legacy=0x7fe940091200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:21.521 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.504+0000 7fe93effd700 10 monclient: _finish_hunting 0 2021-05-25T12:06:21.521 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.504+0000 7fe93effd700 1 monclient: found mon.noname-b 2021-05-25T12:06:21.522 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.504+0000 7fe93effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:06:21.522 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.504+0000 7fe93effd700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T12:06:21.522 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.504+0000 7fe93effd700 1 -- 172.21.15.102:0/577992465 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe940092be0 con 0x7fe94012e330 2021-05-25T12:06:21.522 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.504+0000 7fe93effd700 10 monclient: _finish_auth 0 2021-05-25T12:06:21.523 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.504+0000 7fe93effd700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:06:21.523 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.504+0000 7fe93effd700 1 -- 172.21.15.102:0/577992465 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fe934001990 con 0x7fe94012e330 2021-05-25T12:06:21.523 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.504+0000 7fe946a0b700 5 monclient: authenticate success, global_id 184511 2021-05-25T12:06:21.523 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.504+0000 7fe946a0b700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:06:21.524 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.504+0000 7fe93effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:21.524 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.504+0000 7fe93effd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:06:21.524 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.504+0000 7fe93effd700 10 monclient: dump: 2021-05-25T12:06:21.524 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:21.524 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:21.525 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:21.525 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:21.525 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:21.525 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:21.526 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:21.526 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:21.526 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:21.526 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:21.526 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.504+0000 7fe946a0b700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:06:21.527 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.504+0000 7fe93effd700 1 -- 172.21.15.102:0/577992465 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe934001c40 con 0x7fe94012e330 2021-05-25T12:06:21.527 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.504+0000 7fe93effd700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:06:21.527 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.504+0000 7fe946a0b700 10 monclient: get_monmap_and_config success 2021-05-25T12:06:21.527 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.504+0000 7fe93effd700 1 -- 172.21.15.102:0/577992465 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fe934002010 con 0x7fe94012e330 2021-05-25T12:06:21.528 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.504+0000 7fe946a0b700 4 set_mon_vals no callback set 2021-05-25T12:06:21.528 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.504+0000 7fe93effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:21.528 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.504+0000 7fe93effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:06:21.528 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.504+0000 7fe93effd700 10 monclient: dump: 2021-05-25T12:06:21.528 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:21.529 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:21.529 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:21.529 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:21.530 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:21.530 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:21.530 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:21.530 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:21.531 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:21.531 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:21.531 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.504+0000 7fe946a0b700 10 monclient: shutdown 2021-05-25T12:06:21.531 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.504+0000 7fe946a0b700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:06:21.532 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.504+0000 7fe946a0b700 1 -- 172.21.15.102:0/577992465 >> v1:172.21.15.102:6790/0 conn(0x7fe94012e330 legacy=0x7fe940130720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:21.532 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.504+0000 7fe946a0b700 1 -- 172.21.15.102:0/577992465 shutdown_connections 2021-05-25T12:06:21.532 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.504+0000 7fe946a0b700 1 -- 172.21.15.102:0/577992465 >> 172.21.15.102:0/577992465 conn(0x7fe94009b370 msgr2=0x7fe940097430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:06:21.532 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.504+0000 7fe946a0b700 1 -- 172.21.15.102:0/577992465 shutdown_connections 2021-05-25T12:06:21.532 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.505+0000 7fe946a0b700 1 -- 172.21.15.102:0/577992465 wait complete. 2021-05-25T12:06:21.533 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.505+0000 7fe946a0b700 10 monclient: build_initial_monmap 2021-05-25T12:06:21.533 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.505+0000 7fe946a0b700 10 monclient: monmap: 2021-05-25T12:06:21.533 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:06:21.533 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:06:21.533 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:06:21.534 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:06:21.534 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:06:21.534 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:06:21.534 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:06:21.534 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:06:21.535 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:06:21.535 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:21.535 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.505+0000 7fe946a0b700 1 librados: starting msgr at 2021-05-25T12:06:21.535 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.505+0000 7fe946a0b700 1 librados: starting objecter 2021-05-25T12:06:21.536 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.505+0000 7fe946a0b700 1 Processor -- start 2021-05-25T12:06:21.536 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.506+0000 7fe946a0b700 1 -- start start 2021-05-25T12:06:21.536 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.506+0000 7fe946a0b700 1 librados: setting wanted keys 2021-05-25T12:06:21.536 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.506+0000 7fe946a0b700 1 librados: calling monclient init 2021-05-25T12:06:21.536 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.506+0000 7fe946a0b700 10 monclient: init 2021-05-25T12:06:21.537 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.506+0000 7fe946a0b700 10 monclient: _reopen_session rank -1 2021-05-25T12:06:21.537 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.506+0000 7fe946a0b700 10 monclient: _add_conns ranks=[1,2,0] 2021-05-25T12:06:21.537 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.506+0000 7fe946a0b700 10 monclient(hunting): picked mon.noname-b con 0x7fe940090990 addr v1:172.21.15.181:6789/0 2021-05-25T12:06:21.537 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.506+0000 7fe946a0b700 10 monclient(hunting): picked mon.noname-c con 0x7fe940090d70 addr v1:172.21.15.102:6790/0 2021-05-25T12:06:21.537 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.506+0000 7fe946a0b700 10 monclient(hunting): picked mon.noname-a con 0x7fe94012e330 addr v1:172.21.15.102:6789/0 2021-05-25T12:06:21.538 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.506+0000 7fe946a0b700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe940117960 con 0x7fe94012e330 2021-05-25T12:06:21.538 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.506+0000 7fe946a0b700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe940117c00 con 0x7fe940090990 2021-05-25T12:06:21.538 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.506+0000 7fe946a0b700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe940117ea0 con 0x7fe940090d70 2021-05-25T12:06:21.538 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.506+0000 7fe946a0b700 10 monclient(hunting): _renew_subs 2021-05-25T12:06:21.539 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.506+0000 7fe946a0b700 10 monclient(hunting): authenticate will time out at 1182.103536s 2021-05-25T12:06:21.539 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.506+0000 7fe93f7fe700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fe940090d70 0x7fe940129450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:56464/0 (socket says 172.21.15.102:56464) 2021-05-25T12:06:21.539 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.506+0000 7fe944fa8700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fe94012e330 0x7fe940142ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33356/0 (socket says 172.21.15.102:33356) 2021-05-25T12:06:21.539 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.506+0000 7fe944fa8700 1 -- 172.21.15.102:0/352513649 learned_addr learned my addr 172.21.15.102:0/352513649 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:06:21.539 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.506+0000 7fe93ffff700 1 -- 172.21.15.102:0/352513649 >> v1:172.21.15.181:6789/0 conn(0x7fe940090990 legacy=0x7fe94013ab20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.181:6789/0 2021-05-25T12:06:21.540 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.506+0000 7fe92bfff700 10 client.?.objecter ms_handle_connect 0x7fe94012e330 2021-05-25T12:06:21.540 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.506+0000 7fe92bfff700 10 client.?.objecter resend_mon_ops 2021-05-25T12:06:21.540 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.506+0000 7fe92bfff700 10 client.?.objecter ms_handle_connect 0x7fe940090d70 2021-05-25T12:06:21.540 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.506+0000 7fe92bfff700 10 client.?.objecter resend_mon_ops 2021-05-25T12:06:21.541 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.506+0000 7fe92bfff700 1 -- 172.21.15.102:0/352513649 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4277204920 0 0) 0x7fe940117ea0 con 0x7fe940090d70 2021-05-25T12:06:21.541 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.506+0000 7fe92bfff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:21.541 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.506+0000 7fe92bfff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:21.541 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.506+0000 7fe92bfff700 10 monclient(hunting): my global_id is 184514 2021-05-25T12:06:21.541 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.506+0000 7fe92bfff700 1 -- 172.21.15.102:0/352513649 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe920004c10 con 0x7fe940090d70 2021-05-25T12:06:21.542 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.507+0000 7fe92bfff700 1 -- 172.21.15.102:0/352513649 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 239551842 0 0) 0x7fe920004c10 con 0x7fe940090d70 2021-05-25T12:06:21.542 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.507+0000 7fe92bfff700 1 -- 172.21.15.102:0/352513649 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe940117ea0 con 0x7fe940090d70 2021-05-25T12:06:21.542 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.507+0000 7fe92bfff700 1 -- 172.21.15.102:0/352513649 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fe92c001770 con 0x7fe940090d70 2021-05-25T12:06:21.542 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.507+0000 7fe92bfff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:21.543 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.507+0000 7fe92bfff700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:06:21.543 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.507+0000 7fe92bfff700 10 monclient(hunting): dump: 2021-05-25T12:06:21.543 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:21.543 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:21.543 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:21.544 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:21.544 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:21.544 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:21.544 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:21.545 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:21.545 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:21.545 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:21.545 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.507+0000 7fe92bfff700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:06:21.545 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.507+0000 7fe92bfff700 1 -- 172.21.15.102:0/352513649 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2430598854 0 0) 0x7fe940117ea0 con 0x7fe940090d70 2021-05-25T12:06:21.546 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.507+0000 7fe92bfff700 1 -- 172.21.15.102:0/352513649 >> v1:172.21.15.181:6789/0 conn(0x7fe940090990 legacy=0x7fe94013ab20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-05-25T12:06:21.546 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.507+0000 7fe92bfff700 1 -- 172.21.15.102:0/352513649 >> v1:172.21.15.102:6789/0 conn(0x7fe94012e330 legacy=0x7fe940142ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:21.546 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.507+0000 7fe92bfff700 10 monclient: _finish_hunting 0 2021-05-25T12:06:21.546 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.507+0000 7fe92bfff700 1 monclient: found mon.c 2021-05-25T12:06:21.547 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.507+0000 7fe92bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:06:21.547 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.507+0000 7fe92bfff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:06:21.547 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.507+0000 7fe92bfff700 1 -- 172.21.15.102:0/352513649 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe940115d30 con 0x7fe940090d70 2021-05-25T12:06:21.547 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.507+0000 7fe92bfff700 10 monclient: _finish_auth 0 2021-05-25T12:06:21.547 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.507+0000 7fe92bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:06:21.548 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.507+0000 7fe946a0b700 5 monclient: authenticate success, global_id 184514 2021-05-25T12:06:21.548 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.507+0000 7fe946a0b700 10 monclient: _renew_subs 2021-05-25T12:06:21.548 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.507+0000 7fe946a0b700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:06:21.548 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.507+0000 7fe946a0b700 1 -- 172.21.15.102:0/352513649 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe94013b230 con 0x7fe940090d70 2021-05-25T12:06:21.548 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.507+0000 7fe946a0b700 10 client.184514.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:06:21.549 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.507+0000 7fe946a0b700 10 monclient: _renew_subs 2021-05-25T12:06:21.549 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.507+0000 7fe946a0b700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:06:21.549 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.507+0000 7fe946a0b700 1 -- 172.21.15.102:0/352513649 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe94013b4e0 con 0x7fe940090d70 2021-05-25T12:06:21.549 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.507+0000 7fe946a0b700 1 librados: init done 2021-05-25T12:06:21.550 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.507+0000 7fe92bfff700 1 -- 172.21.15.102:0/352513649 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe92c004530 con 0x7fe940090d70 2021-05-25T12:06:21.550 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.507+0000 7fe92bfff700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:06:21.550 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.507+0000 7fe92bfff700 1 -- 172.21.15.102:0/352513649 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fe92c0048c0 con 0x7fe940090d70 2021-05-25T12:06:21.550 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.507+0000 7fe92bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:21.550 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.507+0000 7fe92bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:06:21.551 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.507+0000 7fe93e7fc700 4 set_mon_vals no callback set 2021-05-25T12:06:21.551 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.507+0000 7fe92bfff700 10 monclient: dump: 2021-05-25T12:06:21.551 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:21.551 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:21.552 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:21.552 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:21.552 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:21.552 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:21.552 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:21.553 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:21.553 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:21.553 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:21.553 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.508+0000 7fe92bfff700 1 -- 172.21.15.102:0/352513649 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7fe92c004a20 con 0x7fe940090d70 2021-05-25T12:06:21.554 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.508+0000 7fe946a0b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:06:21.554 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.508+0000 7fe946a0b700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:06:21.554 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.508+0000 7fe946a0b700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:06:21.554 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.508+0000 7fe946a0b700 1 -- 172.21.15.102:0/352513649 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe918004db0 con 0x7fe940090d70 2021-05-25T12:06:21.554 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.508+0000 7fe92bfff700 1 -- 172.21.15.102:0/352513649 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(196..196 src has 1..196) v4 ==== 6330+0+0 (unknown 1059698009 0 0) 0x7fe92c03d710 con 0x7fe940090d70 2021-05-25T12:06:21.555 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.508+0000 7fe92bfff700 10 client.184514.objecter ms_dispatch 0x7fe940001550 osd_map(196..196 src has 1..196) v4 2021-05-25T12:06:21.555 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.508+0000 7fe92bfff700 3 client.184514.objecter handle_osd_map got epochs [196,196] > 0 2021-05-25T12:06:21.555 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.508+0000 7fe92bfff700 3 client.184514.objecter handle_osd_map decoding full epoch 196 2021-05-25T12:06:21.555 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.508+0000 7fe92bfff700 20 client.184514.objecter dump_active .. 0 homeless 2021-05-25T12:06:21.555 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.508+0000 7fe93ffff700 10 monclient: get_auth_request con 0x7fe9200331d0 auth_method 0 2021-05-25T12:06:21.556 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.509+0000 7fe92bfff700 10 client.184514.objecter ms_handle_connect 0x7fe9200331d0 2021-05-25T12:06:21.556 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.512+0000 7fe92bfff700 1 -- 172.21.15.102:0/352513649 <== mon.2 v1:172.21.15.102:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7fe92c005130 con 0x7fe940090d70 2021-05-25T12:06:21.556 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.512+0000 7fe92bfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:06:21.556 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.512+0000 7fe92bfff700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:06:21.722 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.720+0000 7fe946a0b700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2021-05-25T12:06:21.722 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.720+0000 7fe946a0b700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2021-05-25T12:06:21.723 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.720+0000 7fe946a0b700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:06:21.723 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.720+0000 7fe946a0b700 1 -- 172.21.15.102:0/352513649 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fe918005060 con 0x7fe940090d70 2021-05-25T12:06:21.723 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.721+0000 7fe92bfff700 1 -- 172.21.15.102:0/352513649 <== mon.2 v1:172.21.15.102:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1342 (unknown 254746657 0 1663933137) 0x7fe92c012b80 con 0x7fe940090d70 2021-05-25T12:06:21.723 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.721+0000 7fe92bfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2021-05-25T12:06:21.724 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.721+0000 7fe92bfff700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:06:21.724 INFO:teuthology.orchestra.run.smithi102.stdout:{"election_epoch":52,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":7,"features":{"quorum_con":"4540138303579357183","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"20ee91b6-8935-4b98-b868-85c03789ccc4","modified":"2021-05-25T11:59:11.011522Z","created":"2021-05-25T11:59:11.011522Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6790","nonce":0}]},"addr":"172.21.15.102:6790/0","public_addr":"172.21.15.102:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2021-05-25T12:06:21.726 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.722+0000 7fe946a0b700 10 librados: watch_flush enter 2021-05-25T12:06:21.726 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.722+0000 7fe946a0b700 10 librados: watch_flush exit 2021-05-25T12:06:21.726 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.722+0000 7fe946a0b700 20 client.184514.objecter shutdown clearing up homeless session... 2021-05-25T12:06:21.726 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.722+0000 7fe946a0b700 10 client.184514.objecter successfully canceled tick 2021-05-25T12:06:21.727 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.722+0000 7fe946a0b700 1 -- 172.21.15.102:0/352513649 >> v1:172.21.15.181:6812/29582 conn(0x7fe9200331d0 legacy=0x7fe920035690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:21.727 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.722+0000 7fe946a0b700 10 monclient: shutdown 2021-05-25T12:06:21.727 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.722+0000 7fe946a0b700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:06:21.727 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.722+0000 7fe946a0b700 1 -- 172.21.15.102:0/352513649 >> v1:172.21.15.102:6790/0 conn(0x7fe940090d70 legacy=0x7fe940129450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:21.727 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.722+0000 7fe946a0b700 1 -- 172.21.15.102:0/352513649 shutdown_connections 2021-05-25T12:06:21.728 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.722+0000 7fe946a0b700 1 -- 172.21.15.102:0/352513649 >> 172.21.15.102:0/352513649 conn(0x7fe94009b370 msgr2=0x7fe940068180 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:06:21.728 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.722+0000 7fe946a0b700 1 -- 172.21.15.102:0/352513649 shutdown_connections 2021-05-25T12:06:21.728 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.722+0000 7fe946a0b700 1 -- 172.21.15.102:0/352513649 wait complete. 2021-05-25T12:06:21.728 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:21.722+0000 7fe946a0b700 1 librados: shutdown 2021-05-25T12:06:24.735 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2021-05-25T12:06:24.857 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:24.855+0000 7fd90a033700 10 monclient: get_monmap_and_config 2021-05-25T12:06:24.858 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:24.855+0000 7fd90a033700 10 monclient: build_initial_monmap 2021-05-25T12:06:24.858 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:24.855+0000 7fd90a033700 10 monclient: monmap: 2021-05-25T12:06:24.858 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:06:24.858 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:24.859 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:06:24.857370+0000 2021-05-25T12:06:24.859 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:06:24.857370+0000 2021-05-25T12:06:24.859 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:06:24.859 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:24.860 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:06:24.860 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:06:24.860 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:06:24.860 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:24.862 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:24.856+0000 7fd90a033700 1 Processor -- start 2021-05-25T12:06:24.862 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:24.856+0000 7fd90a033700 1 -- start start 2021-05-25T12:06:24.862 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:24.856+0000 7fd90a033700 10 monclient: init 2021-05-25T12:06:24.862 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:24.856+0000 7fd90a033700 10 monclient: _reopen_session rank -1 2021-05-25T12:06:24.863 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:24.856+0000 7fd90a033700 10 monclient: _add_conns ranks=[0,1,2] 2021-05-25T12:06:24.863 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:24.856+0000 7fd90a033700 10 monclient(hunting): picked mon.noname-a con 0x7fd904097fa0 addr v1:172.21.15.102:6789/0 2021-05-25T12:06:24.863 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:24.856+0000 7fd90a033700 10 monclient(hunting): picked mon.noname-c con 0x7fd904063b70 addr v1:172.21.15.181:6789/0 2021-05-25T12:06:24.863 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:24.856+0000 7fd90a033700 10 monclient(hunting): picked mon.noname-b con 0x7fd9041325c0 addr v1:172.21.15.102:6790/0 2021-05-25T12:06:24.864 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:24.856+0000 7fd90a033700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd904096b20 con 0x7fd904097fa0 2021-05-25T12:06:24.864 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:24.856+0000 7fd90a033700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd90409b460 con 0x7fd904063b70 2021-05-25T12:06:24.864 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:24.856+0000 7fd90a033700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9040964c0 con 0x7fd9041325c0 2021-05-25T12:06:24.864 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:24.856+0000 7fd90a033700 10 monclient(hunting): _renew_subs 2021-05-25T12:06:24.865 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:24.856+0000 7fd90a033700 10 monclient(hunting): authenticate will time out at 1185.454673s 2021-05-25T12:06:24.865 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:24.856+0000 7fd903fff700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fd9041325c0 0x7fd9041349b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:56474/0 (socket says 172.21.15.102:56474) 2021-05-25T12:06:24.865 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:24.856+0000 7fd903fff700 1 -- 172.21.15.102:0/298688112 learned_addr learned my addr 172.21.15.102:0/298688112 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:06:24.865 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:24.856+0000 7fd9037fe700 1 --1- 172.21.15.102:0/298688112 >> v1:172.21.15.102:6789/0 conn(0x7fd904097fa0 0x7fd904063460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33364/0 (socket says 172.21.15.102:33364) 2021-05-25T12:06:24.865 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:24.856+0000 7fd902ffd700 1 -- 172.21.15.102:0/298688112 >> v1:172.21.15.181:6789/0 conn(0x7fd904063b70 legacy=0x7fd904064020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2021-05-25T12:06:25.059 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:25.057+0000 7fd902ffd700 1 -- 172.21.15.102:0/298688112 >> v1:172.21.15.181:6789/0 conn(0x7fd904063b70 legacy=0x7fd904064020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2021-05-25T12:06:25.460 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:25.458+0000 7fd902ffd700 1 -- 172.21.15.102:0/298688112 >> v1:172.21.15.181:6789/0 conn(0x7fd904063b70 legacy=0x7fd904064020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2021-05-25T12:06:26.261 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:26.259+0000 7fd902ffd700 1 -- 172.21.15.102:0/298688112 >> v1:172.21.15.181:6789/0 conn(0x7fd904063b70 legacy=0x7fd904064020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2021-05-25T12:06:27.859 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:27.857+0000 7fd9017fa700 10 monclient(hunting): tick 2021-05-25T12:06:27.860 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:27.857+0000 7fd9017fa700 1 monclient(hunting): continuing hunt 2021-05-25T12:06:27.860 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:27.857+0000 7fd9017fa700 10 monclient(hunting): _reopen_session rank -1 2021-05-25T12:06:27.860 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:27.857+0000 7fd9017fa700 1 -- 172.21.15.102:0/298688112 >> v1:172.21.15.102:6790/0 conn(0x7fd9041325c0 legacy=0x7fd9041349b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:27.860 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:27.857+0000 7fd9017fa700 1 -- 172.21.15.102:0/298688112 >> v1:172.21.15.181:6789/0 conn(0x7fd904063b70 legacy=0x7fd904064020 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-05-25T12:06:27.861 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:27.857+0000 7fd9017fa700 1 -- 172.21.15.102:0/298688112 >> v1:172.21.15.102:6789/0 conn(0x7fd904097fa0 legacy=0x7fd904063460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:27.861 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:27.857+0000 7fd9017fa700 10 monclient: _add_conns ranks=[2,1,0] 2021-05-25T12:06:27.861 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:27.857+0000 7fd9017fa700 10 monclient(hunting): picked mon.noname-b con 0x7fd8ec0033e0 addr v1:172.21.15.102:6790/0 2021-05-25T12:06:27.861 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:27.857+0000 7fd9017fa700 10 monclient(hunting): picked mon.noname-c con 0x7fd904063b70 addr v1:172.21.15.181:6789/0 2021-05-25T12:06:27.861 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:27.857+0000 7fd9017fa700 10 monclient(hunting): picked mon.noname-a con 0x7fd8ec00a880 addr v1:172.21.15.102:6789/0 2021-05-25T12:06:27.862 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:27.857+0000 7fd9017fa700 1 -- 172.21.15.102:0/298688112 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd904096760 con 0x7fd8ec00a880 2021-05-25T12:06:27.862 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:27.857+0000 7fd9017fa700 1 -- 172.21.15.102:0/298688112 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd90409b460 con 0x7fd904063b70 2021-05-25T12:06:27.862 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:27.857+0000 7fd9017fa700 1 -- 172.21.15.102:0/298688112 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8ec0105c0 con 0x7fd8ec0033e0 2021-05-25T12:06:27.862 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:27.857+0000 7fd902ffd700 1 -- 172.21.15.102:0/298688112 >> v1:172.21.15.181:6789/0 conn(0x7fd904063b70 legacy=0x7fd8ec0090e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2021-05-25T12:06:27.863 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:27.857+0000 7fd9017fa700 10 monclient(hunting): _renew_subs 2021-05-25T12:06:28.060 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:28.058+0000 7fd902ffd700 1 -- 172.21.15.102:0/298688112 >> v1:172.21.15.181:6789/0 conn(0x7fd904063b70 legacy=0x7fd8ec0090e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2021-05-25T12:06:28.461 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:28.459+0000 7fd902ffd700 1 -- 172.21.15.102:0/298688112 >> v1:172.21.15.181:6789/0 conn(0x7fd904063b70 legacy=0x7fd8ec0090e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2021-05-25T12:06:29.262 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:29.261+0000 7fd902ffd700 1 -- 172.21.15.102:0/298688112 >> v1:172.21.15.181:6789/0 conn(0x7fd904063b70 legacy=0x7fd8ec0090e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2021-05-25T12:06:30.859 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:30.858+0000 7fd9017fa700 10 monclient(hunting): tick 2021-05-25T12:06:30.860 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:30.858+0000 7fd9017fa700 1 monclient(hunting): continuing hunt 2021-05-25T12:06:30.860 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:30.858+0000 7fd9017fa700 10 monclient(hunting): _reopen_session rank -1 2021-05-25T12:06:30.860 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:30.858+0000 7fd9017fa700 1 -- 172.21.15.102:0/298688112 >> v1:172.21.15.102:6790/0 conn(0x7fd8ec0033e0 legacy=0x7fd8ec005830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:30.861 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:30.858+0000 7fd9017fa700 1 -- 172.21.15.102:0/298688112 >> v1:172.21.15.181:6789/0 conn(0x7fd904063b70 legacy=0x7fd8ec0090e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-05-25T12:06:30.861 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:30.858+0000 7fd9017fa700 1 -- 172.21.15.102:0/298688112 >> v1:172.21.15.102:6789/0 conn(0x7fd8ec00a880 legacy=0x7fd8ec00cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:30.861 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:30.858+0000 7fd9017fa700 10 monclient: _add_conns ranks=[1,2,0] 2021-05-25T12:06:30.861 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:30.858+0000 7fd9017fa700 10 monclient(hunting): picked mon.noname-c con 0x7fd904063b70 addr v1:172.21.15.181:6789/0 2021-05-25T12:06:30.861 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:30.858+0000 7fd9017fa700 10 monclient(hunting): picked mon.noname-b con 0x7fd8ec009830 addr v1:172.21.15.102:6790/0 2021-05-25T12:06:30.862 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:30.858+0000 7fd9017fa700 10 monclient(hunting): picked mon.noname-a con 0x7fd8ec00f630 addr v1:172.21.15.102:6789/0 2021-05-25T12:06:30.862 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:30.858+0000 7fd9017fa700 1 -- 172.21.15.102:0/298688112 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8ec011890 con 0x7fd8ec00f630 2021-05-25T12:06:30.862 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:30.858+0000 7fd9017fa700 1 -- 172.21.15.102:0/298688112 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd90409b460 con 0x7fd904063b70 2021-05-25T12:06:30.862 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:30.858+0000 7fd9017fa700 1 -- 172.21.15.102:0/298688112 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8ec00fa30 con 0x7fd8ec009830 2021-05-25T12:06:30.863 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:30.858+0000 7fd9017fa700 10 monclient(hunting): _renew_subs 2021-05-25T12:06:30.863 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:30.858+0000 7fd9037fe700 1 -- 172.21.15.102:0/298688112 >> v1:172.21.15.181:6789/0 conn(0x7fd904063b70 legacy=0x7fd8ec0090e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2021-05-25T12:06:31.060 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:31.059+0000 7fd9037fe700 1 -- 172.21.15.102:0/298688112 >> v1:172.21.15.181:6789/0 conn(0x7fd904063b70 legacy=0x7fd8ec0090e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2021-05-25T12:06:31.461 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:31.460+0000 7fd9037fe700 1 -- 172.21.15.102:0/298688112 >> v1:172.21.15.181:6789/0 conn(0x7fd904063b70 legacy=0x7fd8ec0090e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2021-05-25T12:06:32.262 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.261+0000 7fd9037fe700 1 -- 172.21.15.102:0/298688112 >> v1:172.21.15.181:6789/0 conn(0x7fd904063b70 legacy=0x7fd8ec0090e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2021-05-25T12:06:32.827 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.825+0000 7fd9027fc700 1 -- 172.21.15.102:0/298688112 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3300063689 0 0) 0x7fd8ec011890 con 0x7fd8ec00f630 2021-05-25T12:06:32.827 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.825+0000 7fd9027fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:32.828 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.825+0000 7fd9027fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:32.828 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.825+0000 7fd9027fc700 10 monclient(hunting): my global_id is 204165 2021-05-25T12:06:32.828 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.825+0000 7fd9027fc700 1 -- 172.21.15.102:0/298688112 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8e8003270 con 0x7fd8ec00f630 2021-05-25T12:06:32.828 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.826+0000 7fd9027fc700 1 -- 172.21.15.102:0/298688112 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 722499604 0 0) 0x7fd8e8003270 con 0x7fd8ec00f630 2021-05-25T12:06:32.829 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.826+0000 7fd9027fc700 1 -- 172.21.15.102:0/298688112 >> v1:172.21.15.102:6790/0 conn(0x7fd8ec009830 legacy=0x7fd8ec009cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:32.829 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.826+0000 7fd9027fc700 1 -- 172.21.15.102:0/298688112 >> v1:172.21.15.181:6789/0 conn(0x7fd904063b70 legacy=0x7fd8ec0090e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-05-25T12:06:32.829 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.826+0000 7fd9027fc700 10 monclient: _finish_hunting 0 2021-05-25T12:06:32.829 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.826+0000 7fd9027fc700 1 monclient: found mon.noname-a 2021-05-25T12:06:32.830 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.826+0000 7fd9027fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:06:32.830 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.826+0000 7fd9027fc700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T12:06:32.830 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.826+0000 7fd9027fc700 1 -- 172.21.15.102:0/298688112 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8ec00fbc0 con 0x7fd8ec00f630 2021-05-25T12:06:32.830 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.826+0000 7fd9027fc700 10 monclient: _finish_auth 0 2021-05-25T12:06:32.830 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.826+0000 7fd9027fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:06:32.831 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.826+0000 7fd9027fc700 1 -- 172.21.15.102:0/298688112 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fd8f8001910 con 0x7fd8ec00f630 2021-05-25T12:06:32.831 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.826+0000 7fd9027fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:32.831 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.826+0000 7fd9027fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:06:32.831 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.826+0000 7fd9027fc700 10 monclient: dump: 2021-05-25T12:06:32.831 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:32.832 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:32.832 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:32.832 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:32.832 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:32.832 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:32.833 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:32.833 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:32.833 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:32.833 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:32.834 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.826+0000 7fd9027fc700 1 -- 172.21.15.102:0/298688112 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd8f8001b30 con 0x7fd8ec00f630 2021-05-25T12:06:32.834 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.826+0000 7fd9027fc700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:06:32.834 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.826+0000 7fd9027fc700 1 -- 172.21.15.102:0/298688112 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fd8f8001e80 con 0x7fd8ec00f630 2021-05-25T12:06:32.834 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.826+0000 7fd9027fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:32.834 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.826+0000 7fd9027fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:06:32.835 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.826+0000 7fd9027fc700 10 monclient: dump: 2021-05-25T12:06:32.835 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:32.835 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:32.835 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:32.835 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:32.836 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:32.836 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:32.836 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:32.836 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:32.836 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:32.837 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:32.837 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.826+0000 7fd90a033700 5 monclient: authenticate success, global_id 204165 2021-05-25T12:06:32.837 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.826+0000 7fd90a033700 10 monclient: get_monmap_and_config success 2021-05-25T12:06:32.837 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.826+0000 7fd90a033700 4 set_mon_vals no callback set 2021-05-25T12:06:32.843 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.827+0000 7fd90a033700 10 monclient: shutdown 2021-05-25T12:06:32.844 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.827+0000 7fd90a033700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:06:32.844 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.827+0000 7fd90a033700 1 -- 172.21.15.102:0/298688112 >> v1:172.21.15.102:6789/0 conn(0x7fd8ec00f630 legacy=0x7fd8ec005830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:32.844 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.827+0000 7fd90a033700 1 -- 172.21.15.102:0/298688112 shutdown_connections 2021-05-25T12:06:32.844 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.827+0000 7fd90a033700 1 -- 172.21.15.102:0/298688112 >> 172.21.15.102:0/298688112 conn(0x7fd90409ef50 msgr2=0x7fd90409f360 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:06:32.845 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.827+0000 7fd90a033700 1 -- 172.21.15.102:0/298688112 shutdown_connections 2021-05-25T12:06:32.845 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.828+0000 7fd90a033700 1 -- 172.21.15.102:0/298688112 wait complete. 2021-05-25T12:06:32.845 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.828+0000 7fd90a033700 10 monclient: build_initial_monmap 2021-05-25T12:06:32.845 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.828+0000 7fd90a033700 10 monclient: monmap: 2021-05-25T12:06:32.845 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:06:32.846 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:06:32.846 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:06:32.846 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:06:32.846 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:06:32.847 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:06:32.847 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:06:32.847 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:06:32.847 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:06:32.847 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:32.848 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.829+0000 7fd90a033700 1 librados: starting msgr at 2021-05-25T12:06:32.848 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.829+0000 7fd90a033700 1 librados: starting objecter 2021-05-25T12:06:32.848 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.829+0000 7fd90a033700 1 Processor -- start 2021-05-25T12:06:32.848 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.829+0000 7fd90a033700 1 -- start start 2021-05-25T12:06:32.848 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.829+0000 7fd90a033700 1 librados: setting wanted keys 2021-05-25T12:06:32.849 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.829+0000 7fd90a033700 1 librados: calling monclient init 2021-05-25T12:06:32.849 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.829+0000 7fd90a033700 10 monclient: init 2021-05-25T12:06:32.849 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.830+0000 7fd90a033700 10 monclient: _reopen_session rank -1 2021-05-25T12:06:32.849 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.830+0000 7fd90a033700 10 monclient: _add_conns ranks=[0,1,2] 2021-05-25T12:06:32.849 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.830+0000 7fd90a033700 10 monclient(hunting): picked mon.noname-a con 0x7fd8ec009830 addr v1:172.21.15.102:6789/0 2021-05-25T12:06:32.850 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.830+0000 7fd90a033700 10 monclient(hunting): picked mon.noname-b con 0x7fd8ec00f630 addr v1:172.21.15.181:6789/0 2021-05-25T12:06:32.850 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.830+0000 7fd90a033700 10 monclient(hunting): picked mon.noname-c con 0x7fd904063b70 addr v1:172.21.15.102:6790/0 2021-05-25T12:06:32.850 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.830+0000 7fd90a033700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9041323c0 con 0x7fd8ec009830 2021-05-25T12:06:32.850 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.830+0000 7fd90a033700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd90409ab70 con 0x7fd8ec00f630 2021-05-25T12:06:32.851 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.830+0000 7fd90a033700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd90411c420 con 0x7fd904063b70 2021-05-25T12:06:32.851 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.830+0000 7fd90a033700 10 monclient(hunting): _renew_subs 2021-05-25T12:06:32.851 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.830+0000 7fd90a033700 10 monclient(hunting): authenticate will time out at 1193.427215s 2021-05-25T12:06:32.851 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.830+0000 7fd902ffd700 1 -- >> v1:172.21.15.181:6789/0 conn(0x7fd8ec00f630 legacy=0x7fd90413b4b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.181:6789/0 2021-05-25T12:06:32.851 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.830+0000 7fd903fff700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fd904063b70 0x7fd9041434e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:56516/0 (socket says 172.21.15.102:56516) 2021-05-25T12:06:32.852 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.830+0000 7fd9037fe700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fd8ec009830 0x7fd904131cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33402/0 (socket says 172.21.15.102:33402) 2021-05-25T12:06:32.852 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.830+0000 7fd903fff700 1 -- 172.21.15.102:0/2894766839 learned_addr learned my addr 172.21.15.102:0/2894766839 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:06:32.852 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.830+0000 7fd8e77fe700 10 client.?.objecter ms_handle_connect 0x7fd904063b70 2021-05-25T12:06:32.852 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.830+0000 7fd8e77fe700 10 client.?.objecter resend_mon_ops 2021-05-25T12:06:32.852 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.830+0000 7fd8e77fe700 10 client.?.objecter ms_handle_connect 0x7fd8ec009830 2021-05-25T12:06:32.853 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.830+0000 7fd8e77fe700 10 client.?.objecter resend_mon_ops 2021-05-25T12:06:32.853 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.833+0000 7fd8e77fe700 1 -- 172.21.15.102:0/2894766839 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3814971015 0 0) 0x7fd90411c420 con 0x7fd904063b70 2021-05-25T12:06:32.853 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.833+0000 7fd8e77fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:32.853 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.833+0000 7fd8e77fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:32.854 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.833+0000 7fd8e77fe700 10 monclient(hunting): my global_id is 184517 2021-05-25T12:06:32.854 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.833+0000 7fd8e77fe700 1 -- 172.21.15.102:0/2894766839 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8ec00a6f0 con 0x7fd904063b70 2021-05-25T12:06:32.854 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.833+0000 7fd8e77fe700 1 -- 172.21.15.102:0/2894766839 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3339559306 0 0) 0x7fd9041323c0 con 0x7fd8ec009830 2021-05-25T12:06:32.854 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.833+0000 7fd8e77fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:32.854 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.833+0000 7fd8e77fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:32.855 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.833+0000 7fd8e77fe700 10 monclient(hunting): my global_id is 204168 2021-05-25T12:06:32.855 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.833+0000 7fd8e77fe700 1 -- 172.21.15.102:0/2894766839 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd90411c420 con 0x7fd8ec009830 2021-05-25T12:06:32.855 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.834+0000 7fd8e77fe700 1 -- 172.21.15.102:0/2894766839 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2165863484 0 0) 0x7fd90411c420 con 0x7fd8ec009830 2021-05-25T12:06:32.855 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.834+0000 7fd8e77fe700 1 -- 172.21.15.102:0/2894766839 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd9041323c0 con 0x7fd8ec009830 2021-05-25T12:06:32.855 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.834+0000 7fd8e77fe700 1 -- 172.21.15.102:0/2894766839 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fd8f0001830 con 0x7fd8ec009830 2021-05-25T12:06:32.856 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.834+0000 7fd8e77fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:32.856 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.834+0000 7fd8e77fe700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:06:32.856 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.834+0000 7fd8e77fe700 10 monclient(hunting): dump: 2021-05-25T12:06:32.856 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:32.857 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:32.857 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:32.857 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:32.857 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:32.857 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:32.858 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:32.858 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:32.858 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:32.858 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:32.858 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.834+0000 7fd8e77fe700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:06:32.859 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.834+0000 7fd8e77fe700 1 -- 172.21.15.102:0/2894766839 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 745466924 0 0) 0x7fd8ec00a6f0 con 0x7fd904063b70 2021-05-25T12:06:32.859 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.834+0000 7fd8e77fe700 1 -- 172.21.15.102:0/2894766839 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd90411c420 con 0x7fd904063b70 2021-05-25T12:06:32.859 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.834+0000 7fd8e77fe700 1 -- 172.21.15.102:0/2894766839 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fd8f8001810 con 0x7fd904063b70 2021-05-25T12:06:32.859 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.834+0000 7fd8e77fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:32.860 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.834+0000 7fd8e77fe700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2021-05-25T12:06:32.860 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.834+0000 7fd8e77fe700 10 monclient(hunting): dump: 2021-05-25T12:06:32.860 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:32.860 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:32.860 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:32.861 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:32.861 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:32.861 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:32.861 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:32.861 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:32.862 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:32.862 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:32.862 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.834+0000 7fd8e77fe700 1 -- 172.21.15.102:0/2894766839 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 384623152 0 0) 0x7fd9041323c0 con 0x7fd8ec009830 2021-05-25T12:06:32.863 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.834+0000 7fd8e77fe700 1 -- 172.21.15.102:0/2894766839 >> v1:172.21.15.102:6790/0 conn(0x7fd904063b70 legacy=0x7fd9041434e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:32.863 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.834+0000 7fd8e77fe700 1 -- 172.21.15.102:0/2894766839 >> v1:172.21.15.181:6789/0 conn(0x7fd8ec00f630 legacy=0x7fd90413b4b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-05-25T12:06:32.863 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.834+0000 7fd8e77fe700 10 monclient: _finish_hunting 0 2021-05-25T12:06:32.863 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.834+0000 7fd8e77fe700 1 monclient: found mon.a 2021-05-25T12:06:32.863 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.834+0000 7fd8e77fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:06:32.864 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.834+0000 7fd8e77fe700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:06:32.864 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.834+0000 7fd8e77fe700 1 -- 172.21.15.102:0/2894766839 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd90411c6c0 con 0x7fd8ec009830 2021-05-25T12:06:32.864 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.834+0000 7fd8e77fe700 10 monclient: _finish_auth 0 2021-05-25T12:06:32.864 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.834+0000 7fd8e77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:06:32.865 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.834+0000 7fd90a033700 5 monclient: authenticate success, global_id 204168 2021-05-25T12:06:32.865 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.834+0000 7fd90a033700 10 monclient: _renew_subs 2021-05-25T12:06:32.865 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.834+0000 7fd90a033700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:06:32.865 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.834+0000 7fd90a033700 1 -- 172.21.15.102:0/2894766839 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd90413bbc0 con 0x7fd8ec009830 2021-05-25T12:06:32.865 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.834+0000 7fd90a033700 10 client.204168.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:06:32.866 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.834+0000 7fd90a033700 10 monclient: _renew_subs 2021-05-25T12:06:32.866 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.834+0000 7fd90a033700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:06:32.866 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.834+0000 7fd90a033700 1 -- 172.21.15.102:0/2894766839 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd90411b1f0 con 0x7fd8ec009830 2021-05-25T12:06:32.866 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.834+0000 7fd90a033700 1 librados: init done 2021-05-25T12:06:32.866 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.834+0000 7fd8e77fe700 1 -- 172.21.15.102:0/2894766839 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd8f0004530 con 0x7fd8ec009830 2021-05-25T12:06:32.867 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.834+0000 7fd8e77fe700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:06:32.867 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.834+0000 7fd8e77fe700 1 -- 172.21.15.102:0/2894766839 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fd8f0004840 con 0x7fd8ec009830 2021-05-25T12:06:32.867 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.834+0000 7fd8e77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:32.867 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.834+0000 7fd8e77fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:06:32.867 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.834+0000 7fd901ffb700 4 set_mon_vals no callback set 2021-05-25T12:06:32.868 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.834+0000 7fd8e77fe700 10 monclient: dump: 2021-05-25T12:06:32.868 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:32.868 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:32.868 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:32.869 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:32.869 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:32.869 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:32.869 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:32.869 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:32.870 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:32.870 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:32.870 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.835+0000 7fd8e77fe700 1 -- 172.21.15.102:0/2894766839 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7fd8f0004a20 con 0x7fd8ec009830 2021-05-25T12:06:32.870 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.835+0000 7fd8e77fe700 1 -- 172.21.15.102:0/2894766839 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(197..197 src has 1..197) v4 ==== 5804+0+0 (unknown 2269415058 0 0) 0x7fd8f003d480 con 0x7fd8ec009830 2021-05-25T12:06:32.870 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.835+0000 7fd90a033700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:06:32.871 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.835+0000 7fd8e77fe700 10 client.204168.objecter ms_dispatch 0x7fd904001550 osd_map(197..197 src has 1..197) v4 2021-05-25T12:06:32.871 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.835+0000 7fd8e77fe700 3 client.204168.objecter handle_osd_map got epochs [197,197] > 0 2021-05-25T12:06:32.871 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.835+0000 7fd8e77fe700 3 client.204168.objecter handle_osd_map decoding full epoch 197 2021-05-25T12:06:32.871 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.835+0000 7fd90a033700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:06:32.872 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.835+0000 7fd90a033700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:06:32.872 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.835+0000 7fd90a033700 1 -- 172.21.15.102:0/2894766839 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8e0004db0 con 0x7fd8ec009830 2021-05-25T12:06:32.872 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.835+0000 7fd8e77fe700 20 client.204168.objecter dump_active .. 0 homeless 2021-05-25T12:06:32.872 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.836+0000 7fd902ffd700 10 monclient: get_auth_request con 0x7fd8ec034cb0 auth_method 0 2021-05-25T12:06:32.872 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.836+0000 7fd8e77fe700 10 client.204168.objecter ms_handle_connect 0x7fd8ec034cb0 2021-05-25T12:06:32.873 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.840+0000 7fd8e77fe700 1 -- 172.21.15.102:0/2894766839 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7fd8f0012870 con 0x7fd8ec009830 2021-05-25T12:06:32.873 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.840+0000 7fd8e77fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:06:32.873 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:32.840+0000 7fd8e77fe700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:06:33.055 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.053+0000 7fd90a033700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2021-05-25T12:06:33.056 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.054+0000 7fd90a033700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2021-05-25T12:06:33.056 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.054+0000 7fd90a033700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:06:33.056 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.054+0000 7fd90a033700 1 -- 172.21.15.102:0/2894766839 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fd8e0005060 con 0x7fd8ec009830 2021-05-25T12:06:33.056 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.054+0000 7fd8e77fe700 1 -- 172.21.15.102:0/2894766839 <== mon.0 v1:172.21.15.102:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1336 (unknown 254746657 0 389016801) 0x7fd8f0012690 con 0x7fd8ec009830 2021-05-25T12:06:33.056 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.054+0000 7fd8e77fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2021-05-25T12:06:33.057 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.054+0000 7fd8e77fe700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:06:33.057 INFO:teuthology.orchestra.run.smithi102.stdout:{"election_epoch":56,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138303579357183","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"20ee91b6-8935-4b98-b868-85c03789ccc4","modified":"2021-05-25T11:59:11.011522Z","created":"2021-05-25T11:59:11.011522Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6790","nonce":0}]},"addr":"172.21.15.102:6790/0","public_addr":"172.21.15.102:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2021-05-25T12:06:33.059 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.055+0000 7fd90a033700 10 librados: watch_flush enter 2021-05-25T12:06:33.059 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.055+0000 7fd90a033700 10 librados: watch_flush exit 2021-05-25T12:06:33.059 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.055+0000 7fd90a033700 20 client.204168.objecter shutdown clearing up homeless session... 2021-05-25T12:06:33.059 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.055+0000 7fd90a033700 10 client.204168.objecter successfully canceled tick 2021-05-25T12:06:33.060 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.055+0000 7fd90a033700 1 -- 172.21.15.102:0/2894766839 >> v1:172.21.15.181:6812/29582 conn(0x7fd8ec034cb0 legacy=0x7fd8ec037170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:33.060 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.055+0000 7fd90a033700 10 monclient: shutdown 2021-05-25T12:06:33.060 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.055+0000 7fd90a033700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:06:33.061 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.055+0000 7fd90a033700 1 -- 172.21.15.102:0/2894766839 >> v1:172.21.15.102:6789/0 conn(0x7fd8ec009830 legacy=0x7fd904131cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:33.061 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.055+0000 7fd90a033700 1 -- 172.21.15.102:0/2894766839 shutdown_connections 2021-05-25T12:06:33.061 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.056+0000 7fd90a033700 1 -- 172.21.15.102:0/2894766839 >> 172.21.15.102:0/2894766839 conn(0x7fd90409ef50 msgr2=0x7fd904094850 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:06:33.061 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.056+0000 7fd90a033700 1 -- 172.21.15.102:0/2894766839 shutdown_connections 2021-05-25T12:06:33.062 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.056+0000 7fd90a033700 1 -- 172.21.15.102:0/2894766839 wait complete. 2021-05-25T12:06:33.062 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.056+0000 7fd90a033700 1 librados: shutdown 2021-05-25T12:06:33.068 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2021-05-25T12:06:33.068 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2021-05-25T12:06:33.178 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.176+0000 7f987dd6e700 10 monclient: get_monmap_and_config 2021-05-25T12:06:33.178 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.176+0000 7f987dd6e700 10 monclient: build_initial_monmap 2021-05-25T12:06:33.179 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.176+0000 7f987dd6e700 10 monclient: monmap: 2021-05-25T12:06:33.179 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:06:33.179 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:33.179 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:06:33.177794+0000 2021-05-25T12:06:33.180 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:06:33.177794+0000 2021-05-25T12:06:33.180 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:06:33.180 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:33.180 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:06:33.180 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:06:33.181 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:06:33.181 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:33.194 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.177+0000 7f987dd6e700 1 Processor -- start 2021-05-25T12:06:33.194 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.177+0000 7f987dd6e700 1 -- start start 2021-05-25T12:06:33.195 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.177+0000 7f987dd6e700 10 monclient: init 2021-05-25T12:06:33.195 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.178+0000 7f987dd6e700 10 monclient: _reopen_session rank -1 2021-05-25T12:06:33.195 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.178+0000 7f987dd6e700 10 monclient: _add_conns ranks=[2,0,1] 2021-05-25T12:06:33.195 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.178+0000 7f987dd6e700 10 monclient(hunting): picked mon.noname-b con 0x7f9878092550 addr v1:172.21.15.102:6790/0 2021-05-25T12:06:33.196 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.178+0000 7f987dd6e700 10 monclient(hunting): picked mon.noname-a con 0x7f987812b100 addr v1:172.21.15.102:6789/0 2021-05-25T12:06:33.196 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.178+0000 7f987dd6e700 10 monclient(hunting): picked mon.noname-c con 0x7f987812ec10 addr v1:172.21.15.181:6789/0 2021-05-25T12:06:33.196 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.178+0000 7f987dd6e700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9878092930 con 0x7f987812b100 2021-05-25T12:06:33.196 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.178+0000 7f987dd6e700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98780931b0 con 0x7f987812ec10 2021-05-25T12:06:33.196 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.178+0000 7f987dd6e700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9878093450 con 0x7f9878092550 2021-05-25T12:06:33.197 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.178+0000 7f987dd6e700 10 monclient(hunting): _renew_subs 2021-05-25T12:06:33.197 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.178+0000 7f987dd6e700 10 monclient(hunting): authenticate will time out at 1193.775123s 2021-05-25T12:06:33.197 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.178+0000 7f98777fe700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f9878092550 0x7f9878093d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:56520/0 (socket says 172.21.15.102:56520) 2021-05-25T12:06:33.197 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.178+0000 7f98777fe700 1 -- 172.21.15.102:0/1737839580 learned_addr learned my addr 172.21.15.102:0/1737839580 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:06:33.198 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.178+0000 7f9876ffd700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f987812b100 0x7f987812d4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33410/0 (socket says 172.21.15.102:33410) 2021-05-25T12:06:33.198 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.178+0000 7f9877fff700 1 -- 172.21.15.102:0/1737839580 >> v1:172.21.15.181:6789/0 conn(0x7f987812ec10 legacy=0x7f9878131000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2021-05-25T12:06:33.198 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.178+0000 7f98767fc700 1 -- 172.21.15.102:0/1737839580 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1726009640 0 0) 0x7f9878093450 con 0x7f9878092550 2021-05-25T12:06:33.198 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.178+0000 7f98767fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:33.199 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.178+0000 7f98767fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:33.199 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.178+0000 7f98767fc700 10 monclient(hunting): my global_id is 184526 2021-05-25T12:06:33.199 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.178+0000 7f98767fc700 1 -- 172.21.15.102:0/1737839580 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f985c003270 con 0x7f9878092550 2021-05-25T12:06:33.199 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.178+0000 7f98767fc700 1 -- 172.21.15.102:0/1737839580 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 358529008 0 0) 0x7f9878092930 con 0x7f987812b100 2021-05-25T12:06:33.200 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.178+0000 7f98767fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:33.200 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.178+0000 7f98767fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:33.200 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.178+0000 7f98767fc700 10 monclient(hunting): my global_id is 204177 2021-05-25T12:06:33.200 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.178+0000 7f98767fc700 1 -- 172.21.15.102:0/1737839580 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9878093450 con 0x7f987812b100 2021-05-25T12:06:33.200 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.178+0000 7f98767fc700 1 -- 172.21.15.102:0/1737839580 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 430609180 0 0) 0x7f985c003270 con 0x7f9878092550 2021-05-25T12:06:33.201 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.178+0000 7f98767fc700 1 -- 172.21.15.102:0/1737839580 >> v1:172.21.15.181:6789/0 conn(0x7f987812ec10 legacy=0x7f9878131000 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-05-25T12:06:33.201 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.178+0000 7f98767fc700 1 -- 172.21.15.102:0/1737839580 >> v1:172.21.15.102:6789/0 conn(0x7f987812b100 legacy=0x7f987812d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:33.201 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.178+0000 7f98767fc700 10 monclient: _finish_hunting 0 2021-05-25T12:06:33.202 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.178+0000 7f98767fc700 1 monclient: found mon.noname-b 2021-05-25T12:06:33.202 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.178+0000 7f98767fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:06:33.202 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.178+0000 7f98767fc700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T12:06:33.202 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.178+0000 7f98767fc700 1 -- 172.21.15.102:0/1737839580 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98780936f0 con 0x7f9878092550 2021-05-25T12:06:33.203 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.178+0000 7f98767fc700 10 monclient: _finish_auth 0 2021-05-25T12:06:33.203 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.178+0000 7f98767fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:06:33.203 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.178+0000 7f987dd6e700 5 monclient: authenticate success, global_id 184526 2021-05-25T12:06:33.203 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.178+0000 7f98767fc700 1 -- 172.21.15.102:0/1737839580 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f9864001650 con 0x7f9878092550 2021-05-25T12:06:33.204 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.178+0000 7f987dd6e700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:06:33.204 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.178+0000 7f98767fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:33.204 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.178+0000 7f98767fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:06:33.204 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.178+0000 7f98767fc700 10 monclient: dump: 2021-05-25T12:06:33.204 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:33.205 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:33.205 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:33.205 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:33.205 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:33.205 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:33.206 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:33.206 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:33.206 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:33.206 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:33.206 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.178+0000 7f987dd6e700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:06:33.207 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.178+0000 7f98767fc700 1 -- 172.21.15.102:0/1737839580 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9864001870 con 0x7f9878092550 2021-05-25T12:06:33.207 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.178+0000 7f98767fc700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:06:33.207 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.178+0000 7f987dd6e700 10 monclient: get_monmap_and_config success 2021-05-25T12:06:33.207 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.178+0000 7f98767fc700 1 -- 172.21.15.102:0/1737839580 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f9864001c00 con 0x7f9878092550 2021-05-25T12:06:33.207 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.178+0000 7f98767fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:33.208 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.178+0000 7f987dd6e700 4 set_mon_vals no callback set 2021-05-25T12:06:33.208 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.178+0000 7f98767fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:06:33.208 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.178+0000 7f98767fc700 10 monclient: dump: 2021-05-25T12:06:33.208 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:33.208 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:33.209 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:33.209 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:33.209 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:33.209 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:33.209 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:33.210 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:33.210 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:33.210 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:33.210 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.179+0000 7f987dd6e700 10 monclient: shutdown 2021-05-25T12:06:33.210 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.179+0000 7f987dd6e700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:06:33.211 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.179+0000 7f987dd6e700 1 -- 172.21.15.102:0/1737839580 >> v1:172.21.15.102:6790/0 conn(0x7f9878092550 legacy=0x7f9878093d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:33.211 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.179+0000 7f987dd6e700 1 -- 172.21.15.102:0/1737839580 shutdown_connections 2021-05-25T12:06:33.211 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.179+0000 7f987dd6e700 1 -- 172.21.15.102:0/1737839580 >> 172.21.15.102:0/1737839580 conn(0x7f987809b370 msgr2=0x7f9878097430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:06:33.211 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.179+0000 7f987dd6e700 1 -- 172.21.15.102:0/1737839580 shutdown_connections 2021-05-25T12:06:33.211 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.179+0000 7f987dd6e700 1 -- 172.21.15.102:0/1737839580 wait complete. 2021-05-25T12:06:33.212 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.179+0000 7f987dd6e700 10 monclient: build_initial_monmap 2021-05-25T12:06:33.212 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.179+0000 7f987dd6e700 10 monclient: monmap: 2021-05-25T12:06:33.212 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:06:33.212 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:06:33.212 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:06:33.213 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:06:33.213 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:06:33.213 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:06:33.213 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:06:33.214 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:06:33.214 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:06:33.214 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:33.214 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.180+0000 7f987dd6e700 1 librados: starting msgr at 2021-05-25T12:06:33.214 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.180+0000 7f987dd6e700 1 librados: starting objecter 2021-05-25T12:06:33.215 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.180+0000 7f987dd6e700 1 Processor -- start 2021-05-25T12:06:33.215 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.180+0000 7f987dd6e700 1 -- start start 2021-05-25T12:06:33.215 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.180+0000 7f987dd6e700 1 librados: setting wanted keys 2021-05-25T12:06:33.215 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.180+0000 7f987dd6e700 1 librados: calling monclient init 2021-05-25T12:06:33.216 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.180+0000 7f987dd6e700 10 monclient: init 2021-05-25T12:06:33.216 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.180+0000 7f987dd6e700 10 monclient: _reopen_session rank -1 2021-05-25T12:06:33.216 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.180+0000 7f987dd6e700 10 monclient: _add_conns ranks=[1,2,0] 2021-05-25T12:06:33.216 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.180+0000 7f987dd6e700 10 monclient(hunting): picked mon.noname-b con 0x7f9878092550 addr v1:172.21.15.181:6789/0 2021-05-25T12:06:33.217 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.180+0000 7f987dd6e700 10 monclient(hunting): picked mon.noname-c con 0x7f987812b100 addr v1:172.21.15.102:6790/0 2021-05-25T12:06:33.217 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.180+0000 7f987dd6e700 10 monclient(hunting): picked mon.noname-a con 0x7f987812ec10 addr v1:172.21.15.102:6789/0 2021-05-25T12:06:33.217 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.180+0000 7f987dd6e700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9878113ff0 con 0x7f987812ec10 2021-05-25T12:06:33.217 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.180+0000 7f987dd6e700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9878114290 con 0x7f9878092550 2021-05-25T12:06:33.218 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.180+0000 7f987dd6e700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9878112190 con 0x7f987812b100 2021-05-25T12:06:33.218 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.180+0000 7f987dd6e700 10 monclient(hunting): _renew_subs 2021-05-25T12:06:33.218 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.180+0000 7f987dd6e700 10 monclient(hunting): authenticate will time out at 1193.777863s 2021-05-25T12:06:33.218 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.180+0000 7f9876ffd700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f987812b100 0x7f9878111680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:56528/0 (socket says 172.21.15.102:56528) 2021-05-25T12:06:33.218 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.180+0000 7f9876ffd700 1 -- 172.21.15.102:0/2080338982 learned_addr learned my addr 172.21.15.102:0/2080338982 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:06:33.219 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.180+0000 7f98777fe700 1 -- 172.21.15.102:0/2080338982 >> v1:172.21.15.181:6789/0 conn(0x7f9878092550 legacy=0x7f9878110f70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.181:6789/0 2021-05-25T12:06:33.219 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.181+0000 7f98637fe700 10 client.?.objecter ms_handle_connect 0x7f987812ec10 2021-05-25T12:06:33.219 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.181+0000 7f98637fe700 10 client.?.objecter resend_mon_ops 2021-05-25T12:06:33.219 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.181+0000 7f98637fe700 10 client.?.objecter ms_handle_connect 0x7f987812b100 2021-05-25T12:06:33.220 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.181+0000 7f98637fe700 10 client.?.objecter resend_mon_ops 2021-05-25T12:06:33.220 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.181+0000 7f98637fe700 1 -- 172.21.15.102:0/2080338982 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1120930317 0 0) 0x7f9878113ff0 con 0x7f987812ec10 2021-05-25T12:06:33.220 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.181+0000 7f98637fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:33.220 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.181+0000 7f98637fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:33.220 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.181+0000 7f98637fe700 10 monclient(hunting): my global_id is 204180 2021-05-25T12:06:33.221 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.181+0000 7f98637fe700 1 -- 172.21.15.102:0/2080338982 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9858004c10 con 0x7f987812ec10 2021-05-25T12:06:33.221 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.181+0000 7f98637fe700 1 -- 172.21.15.102:0/2080338982 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 60356655 0 0) 0x7f9878112190 con 0x7f987812b100 2021-05-25T12:06:33.221 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.181+0000 7f98637fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:33.221 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.181+0000 7f98637fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:33.221 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.181+0000 7f98637fe700 10 monclient(hunting): my global_id is 184529 2021-05-25T12:06:33.222 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.181+0000 7f98637fe700 1 -- 172.21.15.102:0/2080338982 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9878113ff0 con 0x7f987812b100 2021-05-25T12:06:33.222 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.181+0000 7f98637fe700 1 -- 172.21.15.102:0/2080338982 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3908935162 0 0) 0x7f9878113ff0 con 0x7f987812b100 2021-05-25T12:06:33.222 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.181+0000 7f98637fe700 1 -- 172.21.15.102:0/2080338982 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9878112190 con 0x7f987812b100 2021-05-25T12:06:33.222 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.181+0000 7f98637fe700 1 -- 172.21.15.102:0/2080338982 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f98680019d0 con 0x7f987812b100 2021-05-25T12:06:33.222 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.181+0000 7f98637fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:33.223 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.181+0000 7f98637fe700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:06:33.223 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.181+0000 7f98637fe700 10 monclient(hunting): dump: 2021-05-25T12:06:33.223 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:33.223 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:33.223 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:33.224 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:33.224 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:33.224 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:33.224 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:33.225 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:33.225 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:33.225 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:33.225 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.181+0000 7f98637fe700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:06:33.225 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.181+0000 7f98637fe700 1 -- 172.21.15.102:0/2080338982 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 549816151 0 0) 0x7f9858004c10 con 0x7f987812ec10 2021-05-25T12:06:33.226 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.181+0000 7f98637fe700 1 -- 172.21.15.102:0/2080338982 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9878113ff0 con 0x7f987812ec10 2021-05-25T12:06:33.226 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.181+0000 7f98637fe700 1 -- 172.21.15.102:0/2080338982 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f986c004090 con 0x7f987812ec10 2021-05-25T12:06:33.226 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.181+0000 7f98637fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:33.226 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.181+0000 7f98637fe700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2021-05-25T12:06:33.226 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.181+0000 7f98637fe700 10 monclient(hunting): dump: 2021-05-25T12:06:33.227 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:33.227 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:33.227 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:33.227 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:33.227 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:33.228 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:33.228 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:33.228 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:33.228 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:33.228 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:33.229 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.181+0000 7f98637fe700 1 -- 172.21.15.102:0/2080338982 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2156326982 0 0) 0x7f9878112190 con 0x7f987812b100 2021-05-25T12:06:33.229 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.181+0000 7f98637fe700 1 -- 172.21.15.102:0/2080338982 >> v1:172.21.15.181:6789/0 conn(0x7f9878092550 legacy=0x7f9878110f70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-05-25T12:06:33.229 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.181+0000 7f98637fe700 1 -- 172.21.15.102:0/2080338982 >> v1:172.21.15.102:6789/0 conn(0x7f987812ec10 legacy=0x7f987813f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:33.229 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.181+0000 7f98637fe700 10 monclient: _finish_hunting 0 2021-05-25T12:06:33.229 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.181+0000 7f98637fe700 1 monclient: found mon.c 2021-05-25T12:06:33.229 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.181+0000 7f98637fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:06:33.230 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.181+0000 7f98637fe700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:06:33.230 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.181+0000 7f98637fe700 1 -- 172.21.15.102:0/2080338982 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9878129ee0 con 0x7f987812b100 2021-05-25T12:06:33.230 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.181+0000 7f98637fe700 10 monclient: _finish_auth 0 2021-05-25T12:06:33.230 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.181+0000 7f98637fe700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:06:33.231 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.181+0000 7f987dd6e700 5 monclient: authenticate success, global_id 184529 2021-05-25T12:06:33.231 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.181+0000 7f987dd6e700 10 monclient: _renew_subs 2021-05-25T12:06:33.231 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.181+0000 7f987dd6e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:06:33.231 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.181+0000 7f987dd6e700 1 -- 172.21.15.102:0/2080338982 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9878114650 con 0x7f987812b100 2021-05-25T12:06:33.231 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.181+0000 7f987dd6e700 10 client.184529.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:06:33.232 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.181+0000 7f987dd6e700 10 monclient: _renew_subs 2021-05-25T12:06:33.232 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.181+0000 7f987dd6e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:06:33.232 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.181+0000 7f987dd6e700 1 -- 172.21.15.102:0/2080338982 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f987812ae30 con 0x7f987812b100 2021-05-25T12:06:33.232 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.181+0000 7f987dd6e700 1 librados: init done 2021-05-25T12:06:33.232 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.181+0000 7f98637fe700 1 -- 172.21.15.102:0/2080338982 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9868001cc0 con 0x7f987812b100 2021-05-25T12:06:33.233 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.181+0000 7f98637fe700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:06:33.233 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.181+0000 7f98637fe700 1 -- 172.21.15.102:0/2080338982 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f9868004980 con 0x7f987812b100 2021-05-25T12:06:33.233 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.181+0000 7f98637fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:33.233 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.181+0000 7f98637fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:06:33.233 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.181+0000 7f9875ffb700 4 set_mon_vals no callback set 2021-05-25T12:06:33.234 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.181+0000 7f98637fe700 10 monclient: dump: 2021-05-25T12:06:33.234 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:33.234 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:33.234 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:33.235 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:33.235 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:33.235 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:33.235 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:33.235 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:33.236 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:33.236 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:33.236 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.182+0000 7f987dd6e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:06:33.236 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.182+0000 7f987dd6e700 1 -- 172.21.15.102:0/2080338982 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f98500065a0 con 0x7f9850002840 2021-05-25T12:06:33.236 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.182+0000 7f98637fe700 1 -- 172.21.15.102:0/2080338982 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f986800dd20 con 0x7f987812b100 2021-05-25T12:06:33.237 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.182+0000 7f98637fe700 10 client.184529.objecter ms_handle_connect 0x7f9850002840 2021-05-25T12:06:33.237 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.182+0000 7f98637fe700 10 client.184529.objecter resend_mon_ops 2021-05-25T12:06:33.237 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.182+0000 7f98637fe700 1 -- 172.21.15.102:0/2080338982 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(197..197 src has 1..197) v4 ==== 5804+0+0 (unknown 2269415058 0 0) 0x7f986803d770 con 0x7f987812b100 2021-05-25T12:06:33.237 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.182+0000 7f98637fe700 10 client.184529.objecter ms_dispatch 0x7f9878001550 osd_map(197..197 src has 1..197) v4 2021-05-25T12:06:33.237 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.182+0000 7f98637fe700 3 client.184529.objecter handle_osd_map got epochs [197,197] > 0 2021-05-25T12:06:33.238 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.182+0000 7f98637fe700 3 client.184529.objecter handle_osd_map decoding full epoch 197 2021-05-25T12:06:33.238 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.182+0000 7f98637fe700 20 client.184529.objecter dump_active .. 0 homeless 2021-05-25T12:06:33.238 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.182+0000 7f98637fe700 1 -- 172.21.15.102:0/2080338982 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4214288267 0 0) 0x7f98500065a0 con 0x7f9850002840 2021-05-25T12:06:33.238 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.182+0000 7f98637fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:33.238 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.182+0000 7f98637fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:33.239 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.182+0000 7f98637fe700 10 monclient(hunting): my global_id is 204183 2021-05-25T12:06:33.239 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.182+0000 7f98637fe700 1 -- 172.21.15.102:0/2080338982 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9878112190 con 0x7f9850002840 2021-05-25T12:06:33.239 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.182+0000 7f9877fff700 10 monclient: get_auth_request con 0x7f98580332a0 auth_method 0 2021-05-25T12:06:33.239 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.182+0000 7f98637fe700 1 -- 172.21.15.102:0/2080338982 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2229359079 0 0) 0x7f9878112190 con 0x7f9850002840 2021-05-25T12:06:33.239 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.182+0000 7f98637fe700 1 -- 172.21.15.102:0/2080338982 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f98500065a0 con 0x7f9850002840 2021-05-25T12:06:33.240 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.182+0000 7f98637fe700 1 -- 172.21.15.102:0/2080338982 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f9864001250 con 0x7f9850002840 2021-05-25T12:06:33.240 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.182+0000 7f98637fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:33.240 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.182+0000 7f98637fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:06:33.240 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.182+0000 7f98637fe700 10 monclient: dump: 2021-05-25T12:06:33.240 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:33.240 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:33.241 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:33.241 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:33.241 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:33.241 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:33.241 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:33.242 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:33.242 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:33.242 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:33.242 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.183+0000 7f98637fe700 1 -- 172.21.15.102:0/2080338982 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 201967902 0 0) 0x7f98500065a0 con 0x7f9850002840 2021-05-25T12:06:33.242 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.183+0000 7f98637fe700 1 -- 172.21.15.102:0/2080338982 --> v1:172.21.15.102:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9850009e40 con 0x7f9850002840 2021-05-25T12:06:33.243 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.183+0000 7f98637fe700 10 client.184529.objecter ms_handle_connect 0x7f98580332a0 2021-05-25T12:06:33.243 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.183+0000 7f98637fe700 1 -- 172.21.15.102:0/2080338982 <== mon.0 v1:172.21.15.102:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7f9864001c50 con 0x7f9850002840 2021-05-25T12:06:33.243 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.183+0000 7f98637fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:06:33.243 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.183+0000 7f98637fe700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:06:33.243 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.183+0000 7f98637fe700 1 -- 172.21.15.102:0/2080338982 >> v1:172.21.15.102:6789/0 conn(0x7f9850002840 legacy=0x7f9850004ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:33.244 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.183+0000 7f98637fe700 1 -- 172.21.15.102:0/2080338982 >> v1:172.21.15.102:6789/0 conn(0x7f9850002840 legacy=0x7f9850004ce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:06:33.244 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.189+0000 7f987dd6e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:06:33.244 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.189+0000 7f987dd6e700 1 -- 172.21.15.102:0/2080338982 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f9850007540 con 0x7f98500068e0 2021-05-25T12:06:33.244 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.190+0000 7f98637fe700 10 client.184529.objecter ms_handle_connect 0x7f98500068e0 2021-05-25T12:06:33.244 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.190+0000 7f98637fe700 10 client.184529.objecter resend_mon_ops 2021-05-25T12:06:33.245 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.190+0000 7f98637fe700 1 -- 172.21.15.102:0/2080338982 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3867975860 0 0) 0x7f9850007540 con 0x7f98500068e0 2021-05-25T12:06:33.245 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.190+0000 7f98637fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:33.245 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.190+0000 7f98637fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:33.245 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.190+0000 7f98637fe700 10 monclient(hunting): my global_id is 204186 2021-05-25T12:06:33.245 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.190+0000 7f98637fe700 1 -- 172.21.15.102:0/2080338982 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98500065a0 con 0x7f98500068e0 2021-05-25T12:06:33.246 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.190+0000 7f98637fe700 1 -- 172.21.15.102:0/2080338982 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 974859496 0 0) 0x7f98500065a0 con 0x7f98500068e0 2021-05-25T12:06:33.246 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.190+0000 7f98637fe700 1 -- 172.21.15.102:0/2080338982 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9850007540 con 0x7f98500068e0 2021-05-25T12:06:33.246 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.190+0000 7f98637fe700 1 -- 172.21.15.102:0/2080338982 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f9864004a60 con 0x7f98500068e0 2021-05-25T12:06:33.246 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.190+0000 7f98637fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:33.247 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.190+0000 7f98637fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:06:33.247 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.190+0000 7f98637fe700 10 monclient: dump: 2021-05-25T12:06:33.247 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:33.247 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:33.247 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:33.248 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:33.248 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:33.248 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:33.248 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:33.248 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:33.249 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:33.249 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:33.249 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.190+0000 7f98637fe700 1 -- 172.21.15.102:0/2080338982 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3310070634 0 0) 0x7f9850007540 con 0x7f98500068e0 2021-05-25T12:06:33.249 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.190+0000 7f98637fe700 1 -- 172.21.15.102:0/2080338982 --> v1:172.21.15.102:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f9850008770 con 0x7f98500068e0 2021-05-25T12:06:33.249 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.190+0000 7f98637fe700 1 -- 172.21.15.102:0/2080338982 <== mon.0 v1:172.21.15.102:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3929 (unknown 0 0 2185482176) 0x7f9864005f70 con 0x7f98500068e0 2021-05-25T12:06:33.250 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.190+0000 7f98637fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:06:33.250 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.190+0000 7f98637fe700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:06:33.250 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.190+0000 7f98637fe700 1 -- 172.21.15.102:0/2080338982 >> v1:172.21.15.102:6789/0 conn(0x7f98500068e0 legacy=0x7f9850006d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:33.250 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.190+0000 7f98637fe700 1 -- 172.21.15.102:0/2080338982 >> v1:172.21.15.102:6789/0 conn(0x7f98500068e0 legacy=0x7f9850006d70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:06:33.251 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:06:33.251 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:06:33.251 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:06:33.252 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "peon", 2021-05-25T12:06:33.252 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 56, 2021-05-25T12:06:33.252 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:06:33.252 INFO:teuthology.orchestra.run.smithi102.stdout: 0, 2021-05-25T12:06:33.252 INFO:teuthology.orchestra.run.smithi102.stdout: 2 2021-05-25T12:06:33.253 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:06:33.253 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 0, 2021-05-25T12:06:33.253 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:06:33.253 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:06:33.253 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:06:33.254 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:06:33.254 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:06:33.254 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:06:33.254 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:06:33.255 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:06:33.255 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:06:33.255 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:06:33.255 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:06:33.255 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:06:33.256 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:06:33.256 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:06:33.256 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:06:33.256 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:06:33.256 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:06:33.257 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:06:33.257 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:06:33.257 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:06:33.257 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:06:33.257 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:06:33.258 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:06:33.258 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:06:33.258 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:33.258 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:33.259 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:06:33.259 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:06:33.259 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:06:33.259 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:06:33.259 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:06:33.260 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:06:33.260 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:06:33.260 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:06:33.260 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:06:33.261 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:06:33.261 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:06:33.261 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:06:33.261 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:06:33.261 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:06:33.262 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:06:33.262 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:06:33.262 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:06:33.262 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:06:33.262 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:06:33.263 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:06:33.263 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:06:33.263 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:06:33.263 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:06:33.263 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:06:33.264 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:06:33.264 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:06:33.264 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:33.264 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:06:33.264 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:33.265 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:06:33.265 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:06:33.265 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:06:33.265 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:06:33.265 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:33.266 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:06:33.266 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:06:33.266 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:06:33.266 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:33.266 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:33.267 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:33.267 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:06:33.267 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:06:33.267 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:06:33.268 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:06:33.268 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:06:33.268 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:33.268 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:33.268 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:06:33.269 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:06:33.269 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:06:33.269 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:06:33.269 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:33.269 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:06:33.270 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:06:33.270 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:06:33.270 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:33.270 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:33.270 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:33.271 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:06:33.271 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:06:33.271 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:06:33.271 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:06:33.271 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:06:33.272 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:33.272 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:33.272 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:06:33.272 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:06:33.272 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:06:33.273 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:06:33.273 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:33.273 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:06:33.273 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:06:33.273 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:06:33.274 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:33.274 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:33.274 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:33.274 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:06:33.274 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:06:33.275 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:06:33.275 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:06:33.275 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:06:33.275 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:33.275 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:33.276 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:33.276 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:06:33.276 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:06:33.276 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:33.276 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:06:33.277 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:06:33.277 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:06:33.277 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:33.277 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:06:33.277 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": [ 2021-05-25T12:06:33.278 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:33.278 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:06:33.278 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:06:33.278 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 2 2021-05-25T12:06:33.279 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:33.279 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:06:33.279 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:06:33.279 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:33.279 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:06:33.280 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:06:33.280 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 4 2021-05-25T12:06:33.280 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:33.280 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:33.280 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:33.281 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:06:33.281 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:06:33.283 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.190+0000 7f987dd6e700 10 librados: watch_flush enter 2021-05-25T12:06:33.283 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.191+0000 7f987dd6e700 10 librados: watch_flush exit 2021-05-25T12:06:33.283 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.191+0000 7f987dd6e700 20 client.184529.objecter shutdown clearing up homeless session... 2021-05-25T12:06:33.284 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.191+0000 7f987dd6e700 10 client.184529.objecter successfully canceled tick 2021-05-25T12:06:33.284 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.191+0000 7f987dd6e700 1 -- 172.21.15.102:0/2080338982 >> v1:172.21.15.181:6812/29582 conn(0x7f98580332a0 legacy=0x7f9858035760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:33.284 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.191+0000 7f987dd6e700 10 monclient: shutdown 2021-05-25T12:06:33.284 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.191+0000 7f987dd6e700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:06:33.284 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.191+0000 7f987dd6e700 1 -- 172.21.15.102:0/2080338982 >> v1:172.21.15.102:6790/0 conn(0x7f987812b100 legacy=0x7f9878111680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:33.285 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.191+0000 7f9877fff700 1 -- 172.21.15.102:0/2080338982 reap_dead start 2021-05-25T12:06:33.285 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.191+0000 7f9877fff700 1 -- 172.21.15.102:0/2080338982 reap_dead start 2021-05-25T12:06:33.285 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.191+0000 7f987dd6e700 1 -- 172.21.15.102:0/2080338982 shutdown_connections 2021-05-25T12:06:33.285 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.191+0000 7f987dd6e700 1 -- 172.21.15.102:0/2080338982 >> 172.21.15.102:0/2080338982 conn(0x7f987809b370 msgr2=0x7f987812e470 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:06:33.286 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.191+0000 7f987dd6e700 1 -- 172.21.15.102:0/2080338982 shutdown_connections 2021-05-25T12:06:33.286 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.191+0000 7f987dd6e700 1 -- 172.21.15.102:0/2080338982 wait complete. 2021-05-25T12:06:33.286 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.191+0000 7f987dd6e700 1 librados: shutdown 2021-05-25T12:06:33.287 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2021-05-25T12:06:33.396 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.394+0000 7efca859b700 10 monclient: get_monmap_and_config 2021-05-25T12:06:33.396 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.394+0000 7efca859b700 10 monclient: build_initial_monmap 2021-05-25T12:06:33.396 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.394+0000 7efca859b700 10 monclient: monmap: 2021-05-25T12:06:33.397 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:06:33.397 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:33.397 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:06:33.395857+0000 2021-05-25T12:06:33.397 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:06:33.395857+0000 2021-05-25T12:06:33.398 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:06:33.398 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:33.398 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:06:33.398 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:06:33.399 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:06:33.399 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:33.412 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.395+0000 7efca859b700 1 Processor -- start 2021-05-25T12:06:33.413 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.395+0000 7efca859b700 1 -- start start 2021-05-25T12:06:33.413 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.395+0000 7efca859b700 10 monclient: init 2021-05-25T12:06:33.413 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.396+0000 7efca859b700 10 monclient: _reopen_session rank -1 2021-05-25T12:06:33.414 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.396+0000 7efca859b700 10 monclient: _add_conns ranks=[1,2,0] 2021-05-25T12:06:33.414 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.396+0000 7efca859b700 10 monclient(hunting): picked mon.noname-c con 0x7efca0093d90 addr v1:172.21.15.181:6789/0 2021-05-25T12:06:33.414 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.396+0000 7efca859b700 10 monclient(hunting): picked mon.noname-b con 0x7efca0132900 addr v1:172.21.15.102:6790/0 2021-05-25T12:06:33.414 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.396+0000 7efca859b700 10 monclient(hunting): picked mon.noname-a con 0x7efca0136410 addr v1:172.21.15.102:6789/0 2021-05-25T12:06:33.414 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.396+0000 7efca859b700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efca0091ed0 con 0x7efca0136410 2021-05-25T12:06:33.415 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.396+0000 7efca859b700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efca0092170 con 0x7efca0093d90 2021-05-25T12:06:33.415 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.396+0000 7efca859b700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efca0092410 con 0x7efca0132900 2021-05-25T12:06:33.415 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.396+0000 7efca859b700 10 monclient(hunting): _renew_subs 2021-05-25T12:06:33.415 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.396+0000 7efca859b700 10 monclient(hunting): authenticate will time out at 1193.993138s 2021-05-25T12:06:33.416 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.396+0000 7efca6337700 1 -- >> v1:172.21.15.181:6789/0 conn(0x7efca0093d90 legacy=0x7efca00691c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2021-05-25T12:06:33.416 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.396+0000 7efca5b36700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7efca0132900 0x7efca0134cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:56538/0 (socket says 172.21.15.102:56538) 2021-05-25T12:06:33.416 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.396+0000 7efca5b36700 1 -- 172.21.15.102:0/154105846 learned_addr learned my addr 172.21.15.102:0/154105846 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:06:33.416 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.396+0000 7efca6b38700 1 --1- 172.21.15.102:0/154105846 >> v1:172.21.15.102:6789/0 conn(0x7efca0136410 0x7efca0138800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33432/0 (socket says 172.21.15.102:33432) 2021-05-25T12:06:33.416 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.396+0000 7efca5335700 1 -- 172.21.15.102:0/154105846 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2496495292 0 0) 0x7efca0091ed0 con 0x7efca0136410 2021-05-25T12:06:33.417 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.396+0000 7efca5335700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:33.417 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.396+0000 7efca5335700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:33.417 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.396+0000 7efca5335700 10 monclient(hunting): my global_id is 204189 2021-05-25T12:06:33.417 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.396+0000 7efca5335700 1 -- 172.21.15.102:0/154105846 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc88003270 con 0x7efca0136410 2021-05-25T12:06:33.418 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.396+0000 7efca5335700 1 -- 172.21.15.102:0/154105846 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3981568768 0 0) 0x7efca0092410 con 0x7efca0132900 2021-05-25T12:06:33.418 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.396+0000 7efca5335700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:33.418 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.396+0000 7efca5335700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:33.418 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.396+0000 7efca5335700 10 monclient(hunting): my global_id is 184532 2021-05-25T12:06:33.419 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.396+0000 7efca5335700 1 -- 172.21.15.102:0/154105846 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efca0091ed0 con 0x7efca0132900 2021-05-25T12:06:33.419 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.396+0000 7efca5335700 1 -- 172.21.15.102:0/154105846 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 367284445 0 0) 0x7efca0091ed0 con 0x7efca0132900 2021-05-25T12:06:33.419 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.396+0000 7efca5335700 1 -- 172.21.15.102:0/154105846 >> v1:172.21.15.181:6789/0 conn(0x7efca0093d90 legacy=0x7efca00691c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-05-25T12:06:33.419 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.396+0000 7efca5335700 1 -- 172.21.15.102:0/154105846 >> v1:172.21.15.102:6789/0 conn(0x7efca0136410 legacy=0x7efca0138800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:33.420 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.396+0000 7efca5335700 10 monclient: _finish_hunting 0 2021-05-25T12:06:33.420 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.396+0000 7efca5335700 1 monclient: found mon.noname-b 2021-05-25T12:06:33.420 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.396+0000 7efca5335700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:06:33.420 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.396+0000 7efca5335700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T12:06:33.421 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.396+0000 7efca5335700 1 -- 172.21.15.102:0/154105846 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efca0094170 con 0x7efca0132900 2021-05-25T12:06:33.421 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.396+0000 7efca5335700 10 monclient: _finish_auth 0 2021-05-25T12:06:33.421 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.396+0000 7efca5335700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:06:33.421 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.396+0000 7efca5335700 1 -- 172.21.15.102:0/154105846 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7efc940015f0 con 0x7efca0132900 2021-05-25T12:06:33.422 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.396+0000 7efca859b700 5 monclient: authenticate success, global_id 184532 2021-05-25T12:06:33.422 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.396+0000 7efca859b700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:06:33.422 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.396+0000 7efca5335700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:33.422 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.396+0000 7efca5335700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:06:33.422 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.396+0000 7efca5335700 10 monclient: dump: 2021-05-25T12:06:33.423 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:33.423 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:33.423 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:33.423 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:33.424 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:33.424 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:33.424 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:33.424 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:33.425 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:33.425 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:33.425 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.396+0000 7efca859b700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:06:33.425 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.396+0000 7efca5335700 1 -- 172.21.15.102:0/154105846 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efc94001910 con 0x7efca0132900 2021-05-25T12:06:33.425 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.396+0000 7efca5335700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:06:33.426 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.396+0000 7efca5335700 1 -- 172.21.15.102:0/154105846 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7efc94001ca0 con 0x7efca0132900 2021-05-25T12:06:33.426 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.396+0000 7efca859b700 10 monclient: get_monmap_and_config success 2021-05-25T12:06:33.426 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.396+0000 7efca859b700 4 set_mon_vals no callback set 2021-05-25T12:06:33.426 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.396+0000 7efca5335700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:33.427 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.396+0000 7efca5335700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:06:33.427 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.396+0000 7efca5335700 10 monclient: dump: 2021-05-25T12:06:33.427 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:33.427 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:33.427 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:33.428 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:33.428 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:33.428 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:33.428 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:33.429 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:33.429 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:33.429 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:33.429 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.397+0000 7efca859b700 10 monclient: shutdown 2021-05-25T12:06:33.429 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.397+0000 7efca859b700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:06:33.430 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.397+0000 7efca859b700 1 -- 172.21.15.102:0/154105846 >> v1:172.21.15.102:6790/0 conn(0x7efca0132900 legacy=0x7efca0134cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:33.430 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.397+0000 7efca859b700 1 -- 172.21.15.102:0/154105846 shutdown_connections 2021-05-25T12:06:33.430 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.397+0000 7efca859b700 1 -- 172.21.15.102:0/154105846 >> 172.21.15.102:0/154105846 conn(0x7efca009b370 msgr2=0x7efca0097430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:06:33.430 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.397+0000 7efca859b700 1 -- 172.21.15.102:0/154105846 shutdown_connections 2021-05-25T12:06:33.431 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.397+0000 7efca859b700 1 -- 172.21.15.102:0/154105846 wait complete. 2021-05-25T12:06:33.431 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.397+0000 7efca859b700 10 monclient: build_initial_monmap 2021-05-25T12:06:33.431 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.397+0000 7efca859b700 10 monclient: monmap: 2021-05-25T12:06:33.431 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:06:33.431 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:06:33.432 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:06:33.432 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:06:33.432 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:06:33.432 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:06:33.433 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:06:33.433 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:06:33.433 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:06:33.433 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:33.433 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.398+0000 7efca859b700 1 librados: starting msgr at 2021-05-25T12:06:33.434 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.398+0000 7efca859b700 1 librados: starting objecter 2021-05-25T12:06:33.434 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.398+0000 7efca859b700 1 Processor -- start 2021-05-25T12:06:33.434 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.398+0000 7efca859b700 1 -- start start 2021-05-25T12:06:33.434 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.398+0000 7efca859b700 1 librados: setting wanted keys 2021-05-25T12:06:33.435 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.398+0000 7efca859b700 1 librados: calling monclient init 2021-05-25T12:06:33.435 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.398+0000 7efca859b700 10 monclient: init 2021-05-25T12:06:33.435 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.398+0000 7efca859b700 10 monclient: _reopen_session rank -1 2021-05-25T12:06:33.435 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.398+0000 7efca859b700 10 monclient: _add_conns ranks=[2,0,1] 2021-05-25T12:06:33.435 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.398+0000 7efca859b700 10 monclient(hunting): picked mon.noname-c con 0x7efca0093d90 addr v1:172.21.15.102:6790/0 2021-05-25T12:06:33.436 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.398+0000 7efca859b700 10 monclient(hunting): picked mon.noname-a con 0x7efca0132900 addr v1:172.21.15.102:6789/0 2021-05-25T12:06:33.436 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.398+0000 7efca859b700 10 monclient(hunting): picked mon.noname-b con 0x7efca0136410 addr v1:172.21.15.181:6789/0 2021-05-25T12:06:33.436 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.398+0000 7efca859b700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efca013a4c0 con 0x7efca0132900 2021-05-25T12:06:33.436 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.398+0000 7efca859b700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efca00694c0 con 0x7efca0136410 2021-05-25T12:06:33.436 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.398+0000 7efca859b700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efca0094420 con 0x7efca0093d90 2021-05-25T12:06:33.437 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.398+0000 7efca859b700 10 monclient(hunting): _renew_subs 2021-05-25T12:06:33.437 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.398+0000 7efca859b700 10 monclient(hunting): authenticate will time out at 1193.995659s 2021-05-25T12:06:33.437 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.398+0000 7efca6b38700 1 -- >> v1:172.21.15.181:6789/0 conn(0x7efca0136410 legacy=0x7efca0147a60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.181:6789/0 2021-05-25T12:06:33.437 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.398+0000 7efca6337700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7efca0093d90 0x7efca013f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:56544/0 (socket says 172.21.15.102:56544) 2021-05-25T12:06:33.437 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.398+0000 7efca6337700 1 -- 172.21.15.102:0/1424560884 learned_addr learned my addr 172.21.15.102:0/1424560884 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:06:33.438 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.398+0000 7efc8e7fc700 10 client.?.objecter ms_handle_connect 0x7efca0132900 2021-05-25T12:06:33.438 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.398+0000 7efc8e7fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:06:33.438 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.398+0000 7efc8e7fc700 10 client.?.objecter ms_handle_connect 0x7efca0093d90 2021-05-25T12:06:33.438 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.398+0000 7efc8e7fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:06:33.439 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.398+0000 7efc8e7fc700 1 -- 172.21.15.102:0/1424560884 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1586706296 0 0) 0x7efca0094420 con 0x7efca0093d90 2021-05-25T12:06:33.439 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.398+0000 7efc8e7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:33.439 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.398+0000 7efc8e7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:33.439 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.398+0000 7efc8e7fc700 10 monclient(hunting): my global_id is 184535 2021-05-25T12:06:33.439 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.398+0000 7efc8e7fc700 1 -- 172.21.15.102:0/1424560884 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc80004c10 con 0x7efca0093d90 2021-05-25T12:06:33.440 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.398+0000 7efc8e7fc700 1 -- 172.21.15.102:0/1424560884 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1549900260 0 0) 0x7efca013a4c0 con 0x7efca0132900 2021-05-25T12:06:33.440 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.398+0000 7efc8e7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:33.440 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.398+0000 7efc8e7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:33.440 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.398+0000 7efc8e7fc700 10 monclient(hunting): my global_id is 204192 2021-05-25T12:06:33.440 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.398+0000 7efc8e7fc700 1 -- 172.21.15.102:0/1424560884 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efca0094420 con 0x7efca0132900 2021-05-25T12:06:33.441 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.399+0000 7efc8e7fc700 1 -- 172.21.15.102:0/1424560884 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 76865303 0 0) 0x7efc80004c10 con 0x7efca0093d90 2021-05-25T12:06:33.441 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.399+0000 7efc8e7fc700 1 -- 172.21.15.102:0/1424560884 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efca013a4c0 con 0x7efca0093d90 2021-05-25T12:06:33.441 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.399+0000 7efc8e7fc700 1 -- 172.21.15.102:0/1424560884 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7efc90003de0 con 0x7efca0093d90 2021-05-25T12:06:33.441 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.399+0000 7efc8e7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:33.442 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.399+0000 7efc8e7fc700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:06:33.442 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.399+0000 7efc8e7fc700 10 monclient(hunting): dump: 2021-05-25T12:06:33.442 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:33.442 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:33.442 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:33.443 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:33.443 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:33.443 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:33.443 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:33.443 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:33.444 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:33.444 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:33.444 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.399+0000 7efc8e7fc700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:06:33.444 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.399+0000 7efc8e7fc700 1 -- 172.21.15.102:0/1424560884 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2378249958 0 0) 0x7efca0094420 con 0x7efca0132900 2021-05-25T12:06:33.445 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.399+0000 7efc8e7fc700 1 -- 172.21.15.102:0/1424560884 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efc80004c10 con 0x7efca0132900 2021-05-25T12:06:33.445 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.399+0000 7efc8e7fc700 1 -- 172.21.15.102:0/1424560884 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7efc940015b0 con 0x7efca0132900 2021-05-25T12:06:33.445 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.399+0000 7efc8e7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:33.445 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.399+0000 7efc8e7fc700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2021-05-25T12:06:33.445 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.399+0000 7efc8e7fc700 10 monclient(hunting): dump: 2021-05-25T12:06:33.446 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:33.446 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:33.446 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:33.446 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:33.446 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:33.447 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:33.447 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:33.447 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:33.447 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:33.448 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:33.448 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.399+0000 7efc8e7fc700 1 -- 172.21.15.102:0/1424560884 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2302590088 0 0) 0x7efca013a4c0 con 0x7efca0093d90 2021-05-25T12:06:33.448 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.399+0000 7efc8e7fc700 1 -- 172.21.15.102:0/1424560884 >> v1:172.21.15.181:6789/0 conn(0x7efca0136410 legacy=0x7efca0147a60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-05-25T12:06:33.448 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.399+0000 7efc8e7fc700 1 -- 172.21.15.102:0/1424560884 >> v1:172.21.15.102:6789/0 conn(0x7efca0132900 legacy=0x7efca0144330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:33.448 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.399+0000 7efc8e7fc700 10 monclient: _finish_hunting 0 2021-05-25T12:06:33.449 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.399+0000 7efc8e7fc700 1 monclient: found mon.c 2021-05-25T12:06:33.449 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.399+0000 7efc8e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:06:33.449 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.399+0000 7efc8e7fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:06:33.449 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.399+0000 7efc8e7fc700 1 -- 172.21.15.102:0/1424560884 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efca0096460 con 0x7efca0093d90 2021-05-25T12:06:33.449 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.399+0000 7efc8e7fc700 10 monclient: _finish_auth 0 2021-05-25T12:06:33.450 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.399+0000 7efc8e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:06:33.450 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.399+0000 7efca859b700 5 monclient: authenticate success, global_id 184535 2021-05-25T12:06:33.450 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.399+0000 7efca859b700 10 monclient: _renew_subs 2021-05-25T12:06:33.450 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.399+0000 7efca859b700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:06:33.450 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.399+0000 7efca859b700 1 -- 172.21.15.102:0/1424560884 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efca013fe20 con 0x7efca0093d90 2021-05-25T12:06:33.451 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.399+0000 7efca859b700 10 client.184535.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:06:33.451 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.399+0000 7efca859b700 10 monclient: _renew_subs 2021-05-25T12:06:33.451 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.399+0000 7efca859b700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:06:33.451 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.399+0000 7efca859b700 1 -- 172.21.15.102:0/1424560884 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efca0067e30 con 0x7efca0093d90 2021-05-25T12:06:33.451 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.399+0000 7efca859b700 1 librados: init done 2021-05-25T12:06:33.452 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.399+0000 7efc8e7fc700 1 -- 172.21.15.102:0/1424560884 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efc90003fb0 con 0x7efca0093d90 2021-05-25T12:06:33.452 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.399+0000 7efc8e7fc700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:06:33.452 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.399+0000 7efc8e7fc700 1 -- 172.21.15.102:0/1424560884 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7efc90004340 con 0x7efca0093d90 2021-05-25T12:06:33.452 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.399+0000 7efc8e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:33.453 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.399+0000 7efc8e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:06:33.453 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.399+0000 7efc8e7fc700 10 monclient: dump: 2021-05-25T12:06:33.453 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:33.453 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:33.453 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:33.454 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:33.454 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:33.454 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:33.454 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:33.454 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:33.455 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:33.455 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:33.455 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.399+0000 7efca4b34700 4 set_mon_vals no callback set 2021-05-25T12:06:33.455 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.400+0000 7efc8e7fc700 1 -- 172.21.15.102:0/1424560884 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7efc90004520 con 0x7efca0093d90 2021-05-25T12:06:33.456 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.400+0000 7efca859b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:06:33.456 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.400+0000 7efca859b700 1 -- 172.21.15.102:0/1424560884 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7efc840065a0 con 0x7efc84002840 2021-05-25T12:06:33.456 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.400+0000 7efc8e7fc700 1 -- 172.21.15.102:0/1424560884 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(197..197 src has 1..197) v4 ==== 5804+0+0 (unknown 2269415058 0 0) 0x7efc9003d570 con 0x7efca0093d90 2021-05-25T12:06:33.456 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.400+0000 7efc8e7fc700 10 client.184535.objecter ms_dispatch 0x7efca0001550 osd_map(197..197 src has 1..197) v4 2021-05-25T12:06:33.457 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.400+0000 7efc8e7fc700 3 client.184535.objecter handle_osd_map got epochs [197,197] > 0 2021-05-25T12:06:33.457 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.400+0000 7efc8e7fc700 3 client.184535.objecter handle_osd_map decoding full epoch 197 2021-05-25T12:06:33.457 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.400+0000 7efc8e7fc700 20 client.184535.objecter dump_active .. 0 homeless 2021-05-25T12:06:33.457 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.400+0000 7efc8e7fc700 10 client.184535.objecter ms_handle_connect 0x7efc84002840 2021-05-25T12:06:33.457 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.400+0000 7efc8e7fc700 10 client.184535.objecter resend_mon_ops 2021-05-25T12:06:33.458 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.400+0000 7efc8e7fc700 1 -- 172.21.15.102:0/1424560884 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2112707980 0 0) 0x7efc840065a0 con 0x7efc84002840 2021-05-25T12:06:33.458 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.400+0000 7efc8e7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:33.458 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.400+0000 7efc8e7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:33.458 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.400+0000 7efc8e7fc700 10 monclient(hunting): my global_id is 184538 2021-05-25T12:06:33.459 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.400+0000 7efc8e7fc700 1 -- 172.21.15.102:0/1424560884 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efca013a4c0 con 0x7efc84002840 2021-05-25T12:06:33.459 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.400+0000 7efca6b38700 10 monclient: get_auth_request con 0x7efc80033330 auth_method 0 2021-05-25T12:06:33.459 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.400+0000 7efc8e7fc700 1 -- 172.21.15.102:0/1424560884 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4020750639 0 0) 0x7efca013a4c0 con 0x7efc84002840 2021-05-25T12:06:33.459 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.400+0000 7efc8e7fc700 1 -- 172.21.15.102:0/1424560884 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efc840065a0 con 0x7efc84002840 2021-05-25T12:06:33.459 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.400+0000 7efc8e7fc700 1 -- 172.21.15.102:0/1424560884 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7efc94004cd0 con 0x7efc84002840 2021-05-25T12:06:33.460 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.400+0000 7efc8e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:33.460 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.400+0000 7efc8e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:06:33.460 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.400+0000 7efc8e7fc700 10 monclient: dump: 2021-05-25T12:06:33.460 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:33.461 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:33.461 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:33.461 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:33.461 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:33.461 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:33.462 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:33.462 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:33.462 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:33.462 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:33.462 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.400+0000 7efc8e7fc700 1 -- 172.21.15.102:0/1424560884 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3427990599 0 0) 0x7efc840065a0 con 0x7efc84002840 2021-05-25T12:06:33.463 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.400+0000 7efc8e7fc700 1 -- 172.21.15.102:0/1424560884 --> v1:172.21.15.102:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7efc84009e40 con 0x7efc84002840 2021-05-25T12:06:33.463 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.400+0000 7efc8e7fc700 10 client.184535.objecter ms_handle_connect 0x7efc80033330 2021-05-25T12:06:33.463 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.401+0000 7efc8e7fc700 1 -- 172.21.15.102:0/1424560884 <== mon.2 v1:172.21.15.102:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7efc94004fc0 con 0x7efc84002840 2021-05-25T12:06:33.463 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.401+0000 7efc8e7fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:06:33.463 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.401+0000 7efc8e7fc700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:06:33.464 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.401+0000 7efc8e7fc700 1 -- 172.21.15.102:0/1424560884 >> v1:172.21.15.102:6790/0 conn(0x7efc84002840 legacy=0x7efc84004ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:33.464 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.401+0000 7efc8e7fc700 1 -- 172.21.15.102:0/1424560884 >> v1:172.21.15.102:6790/0 conn(0x7efc84002840 legacy=0x7efc84004ce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:06:33.464 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.407+0000 7efca859b700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:06:33.464 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.407+0000 7efca859b700 1 -- 172.21.15.102:0/1424560884 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7efc84009ad0 con 0x7efc84008d90 2021-05-25T12:06:33.465 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.407+0000 7efc8e7fc700 10 client.184535.objecter ms_handle_connect 0x7efc84008d90 2021-05-25T12:06:33.465 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.407+0000 7efc8e7fc700 10 client.184535.objecter resend_mon_ops 2021-05-25T12:06:33.465 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.407+0000 7efc8e7fc700 1 -- 172.21.15.102:0/1424560884 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1308361473 0 0) 0x7efc84009ad0 con 0x7efc84008d90 2021-05-25T12:06:33.465 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.407+0000 7efc8e7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:33.465 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.407+0000 7efc8e7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:33.466 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.407+0000 7efc8e7fc700 10 monclient(hunting): my global_id is 184541 2021-05-25T12:06:33.466 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.407+0000 7efc8e7fc700 1 -- 172.21.15.102:0/1424560884 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc840065a0 con 0x7efc84008d90 2021-05-25T12:06:33.466 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.408+0000 7efc8e7fc700 1 -- 172.21.15.102:0/1424560884 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2786212910 0 0) 0x7efc840065a0 con 0x7efc84008d90 2021-05-25T12:06:33.466 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.408+0000 7efc8e7fc700 1 -- 172.21.15.102:0/1424560884 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efc84009ad0 con 0x7efc84008d90 2021-05-25T12:06:33.467 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.408+0000 7efc8e7fc700 1 -- 172.21.15.102:0/1424560884 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7efc94005590 con 0x7efc84008d90 2021-05-25T12:06:33.467 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.408+0000 7efc8e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:33.467 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.408+0000 7efc8e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:06:33.467 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.408+0000 7efc8e7fc700 10 monclient: dump: 2021-05-25T12:06:33.467 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:33.468 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:33.468 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:33.468 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:33.468 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:33.468 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:33.469 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:33.469 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:33.469 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:33.469 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:33.470 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.408+0000 7efc8e7fc700 1 -- 172.21.15.102:0/1424560884 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2340694883 0 0) 0x7efc84009ad0 con 0x7efc84008d90 2021-05-25T12:06:33.470 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.408+0000 7efc8e7fc700 1 -- 172.21.15.102:0/1424560884 --> v1:172.21.15.102:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7efc84008740 con 0x7efc84008d90 2021-05-25T12:06:33.470 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.408+0000 7efc8e7fc700 1 -- 172.21.15.102:0/1424560884 <== mon.2 v1:172.21.15.102:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4101 (unknown 0 0 1828375890) 0x7efc940068b0 con 0x7efc84008d90 2021-05-25T12:06:33.470 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.408+0000 7efc8e7fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:06:33.470 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.408+0000 7efc8e7fc700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:06:33.471 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.408+0000 7efc8e7fc700 1 -- 172.21.15.102:0/1424560884 >> v1:172.21.15.102:6790/0 conn(0x7efc84008d90 legacy=0x7efc84009220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:33.471 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.408+0000 7efc8e7fc700 1 -- 172.21.15.102:0/1424560884 >> v1:172.21.15.102:6790/0 conn(0x7efc84008d90 legacy=0x7efc84009220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:06:33.471 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:06:33.472 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:06:33.472 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:06:33.472 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "leader", 2021-05-25T12:06:33.472 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 56, 2021-05-25T12:06:33.473 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:06:33.473 INFO:teuthology.orchestra.run.smithi102.stdout: 0, 2021-05-25T12:06:33.473 INFO:teuthology.orchestra.run.smithi102.stdout: 2 2021-05-25T12:06:33.473 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:06:33.473 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 0, 2021-05-25T12:06:33.474 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:06:33.474 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:06:33.474 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:06:33.474 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:06:33.475 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:06:33.475 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:06:33.475 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:06:33.475 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:06:33.475 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:06:33.476 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:06:33.476 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:06:33.476 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:06:33.476 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:06:33.476 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:06:33.477 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:06:33.477 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:06:33.477 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:06:33.477 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:06:33.478 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:06:33.478 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:06:33.478 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:06:33.478 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:06:33.478 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:06:33.479 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:06:33.479 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:33.479 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:33.479 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:06:33.479 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:06:33.480 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:06:33.480 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:06:33.480 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:06:33.480 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:06:33.480 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:06:33.481 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:06:33.481 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:06:33.481 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:06:33.481 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:06:33.481 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:06:33.482 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:06:33.482 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:06:33.482 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:06:33.482 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:06:33.482 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:06:33.483 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:06:33.483 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:06:33.483 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:06:33.483 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:06:33.483 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:06:33.484 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:06:33.484 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:06:33.484 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:06:33.484 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:06:33.485 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:33.485 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:06:33.485 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:33.485 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:06:33.485 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:06:33.486 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:06:33.486 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:06:33.486 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:33.486 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:06:33.486 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:06:33.487 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:06:33.487 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:33.487 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:33.487 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:33.488 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:06:33.488 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:06:33.488 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:06:33.488 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:06:33.488 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:06:33.489 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:33.489 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:33.489 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:06:33.489 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:06:33.489 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:06:33.490 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:06:33.490 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:33.490 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:06:33.490 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:06:33.491 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:06:33.491 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:33.491 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:33.491 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:33.491 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:06:33.492 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:06:33.492 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:06:33.492 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:06:33.492 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:06:33.492 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:33.493 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:33.493 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:06:33.493 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:06:33.493 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:06:33.493 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:06:33.494 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:33.494 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:06:33.494 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:06:33.494 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:06:33.494 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:33.495 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:33.495 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:33.495 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:06:33.495 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:06:33.496 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:06:33.496 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:06:33.496 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:06:33.496 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:33.496 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:33.497 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:33.497 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:06:33.497 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:06:33.497 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:33.497 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:06:33.498 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:06:33.498 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:06:33.498 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:33.498 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:06:33.498 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": [ 2021-05-25T12:06:33.499 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:33.499 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:06:33.499 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:06:33.499 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 4 2021-05-25T12:06:33.499 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:33.500 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:06:33.500 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:06:33.500 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:33.500 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:06:33.501 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:06:33.501 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 5 2021-05-25T12:06:33.501 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:33.501 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:06:33.501 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": [ 2021-05-25T12:06:33.502 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:33.502 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:06:33.502 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:06:33.502 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:06:33.503 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:33.503 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:33.503 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:33.503 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:06:33.503 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:06:33.505 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.408+0000 7efca859b700 10 librados: watch_flush enter 2021-05-25T12:06:33.506 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.408+0000 7efca859b700 10 librados: watch_flush exit 2021-05-25T12:06:33.506 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.408+0000 7efca859b700 20 client.184535.objecter shutdown clearing up homeless session... 2021-05-25T12:06:33.506 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.408+0000 7efca859b700 10 client.184535.objecter successfully canceled tick 2021-05-25T12:06:33.506 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.408+0000 7efca859b700 1 -- 172.21.15.102:0/1424560884 >> v1:172.21.15.181:6812/29582 conn(0x7efc80033330 legacy=0x7efc800357f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:33.507 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.408+0000 7efca859b700 10 monclient: shutdown 2021-05-25T12:06:33.507 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.408+0000 7efca859b700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:06:33.507 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.408+0000 7efca859b700 1 -- 172.21.15.102:0/1424560884 >> v1:172.21.15.102:6790/0 conn(0x7efca0093d90 legacy=0x7efca013f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:33.507 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.408+0000 7efca6b38700 1 -- 172.21.15.102:0/1424560884 reap_dead start 2021-05-25T12:06:33.508 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.408+0000 7efca6b38700 1 -- 172.21.15.102:0/1424560884 reap_dead start 2021-05-25T12:06:33.508 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.408+0000 7efca859b700 1 -- 172.21.15.102:0/1424560884 shutdown_connections 2021-05-25T12:06:33.508 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.408+0000 7efca859b700 1 -- 172.21.15.102:0/1424560884 >> 172.21.15.102:0/1424560884 conn(0x7efca009b370 msgr2=0x7efca00990a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:06:33.508 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.408+0000 7efca859b700 1 -- 172.21.15.102:0/1424560884 shutdown_connections 2021-05-25T12:06:33.509 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.408+0000 7efca859b700 1 -- 172.21.15.102:0/1424560884 wait complete. 2021-05-25T12:06:33.509 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:33.408+0000 7efca859b700 1 librados: shutdown 2021-05-25T12:06:33.509 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2021-05-25T12:06:52.738 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ api_tier_pp: [==========] Running 76 tests from 4 test suites. 2021-05-25T12:06:52.739 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: [----------] Global test environment set-up. 2021-05-25T12:06:52.739 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP 2021-05-25T12:06:52.739 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: seed 33015 2021-05-25T12:06:52.740 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.Dirty 2021-05-25T12:06:52.740 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: [ OK ] LibRadosTierPP.Dirty (26 ms) 2021-05-25T12:06:52.740 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.FlushWriteRaces 2021-05-25T12:06:52.741 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: [ OK ] LibRadosTierPP.FlushWriteRaces (10511 ms) 2021-05-25T12:06:52.741 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.HitSetNone 2021-05-25T12:06:52.741 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: [ OK ] LibRadosTierPP.HitSetNone (12 ms) 2021-05-25T12:06:52.741 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP (10549 ms total) 2021-05-25T12:06:52.741 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: 2021-05-25T12:06:52.742 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: [----------] 47 tests from LibRadosTwoPoolsPP 2021-05-25T12:06:52.742 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Overlay 2021-05-25T12:06:52.742 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Overlay (7057 ms) 2021-05-25T12:06:52.742 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Promote 2021-05-25T12:06:52.742 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Promote (32787 ms) 2021-05-25T12:06:52.743 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnap 2021-05-25T12:06:52.743 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnap (9974 ms) 2021-05-25T12:06:52.743 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapScrub 2021-05-25T12:06:52.743 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: my_snaps [4] 2021-05-25T12:06:52.744 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: my_snaps [5,4] 2021-05-25T12:06:52.744 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: my_snaps [6,5,4] 2021-05-25T12:06:52.744 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: my_snaps [7,6,5,4] 2021-05-25T12:06:52.744 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: promoting some heads 2021-05-25T12:06:52.744 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: promoting from clones for snap 7 2021-05-25T12:06:52.745 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: promoting from clones for snap 6 2021-05-25T12:06:52.745 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: promoting from clones for snap 5 2021-05-25T12:06:52.745 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: promoting from clones for snap 4 2021-05-25T12:06:52.745 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: waiting for scrubs... 2021-05-25T12:06:52.745 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: done waiting 2021-05-25T12:06:52.746 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapScrub (78287 ms) 2021-05-25T12:06:52.746 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapTrimRace 2021-05-25T12:06:52.746 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapTrimRace (9964 ms) 2021-05-25T12:06:52.746 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Whiteout 2021-05-25T12:06:52.747 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Whiteout (22074 ms) 2021-05-25T12:06:52.747 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate 2021-05-25T12:06:52.747 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate (7555 ms) 2021-05-25T12:06:52.747 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Evict 2021-05-25T12:06:52.747 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Evict (14185 ms) 2021-05-25T12:06:52.748 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap 2021-05-25T12:06:52.748 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap (35129 ms) 2021-05-25T12:06:52.748 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap2 2021-05-25T12:06:52.748 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap2 (8991 ms) 2021-05-25T12:06:52.748 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ListSnap 2021-05-25T12:06:52.749 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ListSnap (23699 ms) 2021-05-25T12:06:52.749 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace 2021-05-25T12:06:52.749 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace (35104 ms) 2021-05-25T12:06:52.749 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlush 2021-05-25T12:06:52.749 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlush (7919 ms) 2021-05-25T12:06:52.750 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Flush 2021-05-25T12:06:52.750 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Flush (7986 ms) 2021-05-25T12:06:52.750 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushSnap 2021-05-25T12:06:52.750 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushSnap (27107 ms) 2021-05-25T12:06:52.751 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushTryFlushRaces 2021-05-25T12:06:52.751 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushTryFlushRaces (7877 ms) 2021-05-25T12:06:52.751 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlushReadRace 2021-05-25T12:06:52.751 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlushReadRace (8000 ms) 2021-05-25T12:06:53.510 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2021-05-25T12:06:53.510 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2021-05-25T12:06:53.511 INFO:tasks.ceph.mon.b:Restarting daemon 2021-05-25T12:06:53.511 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2021-05-25T12:06:53.514 INFO:tasks.ceph.mon.b:Started 2021-05-25T12:06:53.515 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2021-05-25T12:06:53.515 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2021-05-25T12:06:53.639 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.637+0000 7f9effb53700 10 monclient: get_monmap_and_config 2021-05-25T12:06:53.639 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.637+0000 7f9effb53700 10 monclient: build_initial_monmap 2021-05-25T12:06:53.640 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.637+0000 7f9effb53700 10 monclient: monmap: 2021-05-25T12:06:53.640 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:06:53.640 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:53.640 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:06:53.638680+0000 2021-05-25T12:06:53.641 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:06:53.638680+0000 2021-05-25T12:06:53.641 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:06:53.641 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:53.641 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:06:53.642 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:06:53.642 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:06:53.642 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:53.642 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.638+0000 7f9effb53700 1 Processor -- start 2021-05-25T12:06:53.642 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.638+0000 7f9effb53700 1 -- start start 2021-05-25T12:06:53.643 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.638+0000 7f9effb53700 10 monclient: init 2021-05-25T12:06:53.644 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.638+0000 7f9effb53700 10 monclient: _reopen_session rank -1 2021-05-25T12:06:53.645 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.638+0000 7f9effb53700 10 monclient: _add_conns ranks=[2,0,1] 2021-05-25T12:06:53.645 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.638+0000 7f9effb53700 10 monclient(hunting): picked mon.noname-b con 0x7f9ef8090990 addr v1:172.21.15.102:6790/0 2021-05-25T12:06:53.645 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.638+0000 7f9effb53700 10 monclient(hunting): picked mon.noname-a con 0x7f9ef8090d70 addr v1:172.21.15.102:6789/0 2021-05-25T12:06:53.645 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.638+0000 7f9effb53700 10 monclient(hunting): picked mon.noname-c con 0x7f9ef812e330 addr v1:172.21.15.181:6789/0 2021-05-25T12:06:53.646 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.638+0000 7f9effb53700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ef8096fa0 con 0x7f9ef8090d70 2021-05-25T12:06:53.646 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.639+0000 7f9effb53700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ef80926a0 con 0x7f9ef812e330 2021-05-25T12:06:53.646 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.639+0000 7f9effb53700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ef8092940 con 0x7f9ef8090990 2021-05-25T12:06:53.646 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.639+0000 7f9effb53700 10 monclient(hunting): _renew_subs 2021-05-25T12:06:53.647 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.639+0000 7f9effb53700 10 monclient(hunting): authenticate will time out at 1214.236062s 2021-05-25T12:06:53.647 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.639+0000 7f9efd0ee700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f9ef8090d70 0x7f9ef8091200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33460/0 (socket says 172.21.15.102:33460) 2021-05-25T12:06:53.647 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.639+0000 7f9efd0ee700 1 -- 172.21.15.102:0/1938196027 learned_addr learned my addr 172.21.15.102:0/1938196027 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:06:53.665 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.664+0000 7f9efc8ed700 1 -- 172.21.15.102:0/1938196027 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3409716316 0 0) 0x7f9ef8092940 con 0x7f9ef8090990 2021-05-25T12:06:53.665 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.664+0000 7f9efc8ed700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:53.665 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.664+0000 7f9efc8ed700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:53.666 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.664+0000 7f9efc8ed700 10 monclient(hunting): my global_id is 184568 2021-05-25T12:06:53.666 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.664+0000 7f9efc8ed700 1 -- 172.21.15.102:0/1938196027 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ee0004380 con 0x7f9ef8090990 2021-05-25T12:06:53.678 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.666+0000 7f9efc8ed700 1 -- 172.21.15.102:0/1938196027 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3420281082 0 0) 0x7f9ef8096fa0 con 0x7f9ef8090d70 2021-05-25T12:06:53.679 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.666+0000 7f9efc8ed700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:53.679 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.666+0000 7f9efc8ed700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:53.679 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.666+0000 7f9efc8ed700 10 monclient(hunting): my global_id is 204228 2021-05-25T12:06:53.680 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.666+0000 7f9efc8ed700 1 -- 172.21.15.102:0/1938196027 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ef8092940 con 0x7f9ef8090d70 2021-05-25T12:06:53.680 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.666+0000 7f9efc8ed700 1 -- 172.21.15.102:0/1938196027 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2989238428 0 0) 0x7f9ef8092940 con 0x7f9ef8090d70 2021-05-25T12:06:53.681 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.666+0000 7f9efc8ed700 1 -- 172.21.15.102:0/1938196027 >> v1:172.21.15.102:6790/0 conn(0x7f9ef8090990 legacy=0x7f9ef8093240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:53.681 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.666+0000 7f9efc8ed700 1 -- 172.21.15.102:0/1938196027 >> v1:172.21.15.181:6789/0 conn(0x7f9ef812e330 legacy=0x7f9ef8130720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:53.681 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.666+0000 7f9efc8ed700 10 monclient: _finish_hunting 0 2021-05-25T12:06:53.682 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.666+0000 7f9efc8ed700 1 monclient: found mon.noname-a 2021-05-25T12:06:53.682 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.666+0000 7f9efc8ed700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:06:53.682 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.666+0000 7f9efc8ed700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T12:06:53.682 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.666+0000 7f9efc8ed700 1 -- 172.21.15.102:0/1938196027 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ef8092be0 con 0x7f9ef8090d70 2021-05-25T12:06:53.683 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.666+0000 7f9efc8ed700 10 monclient: _finish_auth 0 2021-05-25T12:06:53.683 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.666+0000 7f9efc8ed700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:06:53.683 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.666+0000 7f9effb53700 5 monclient: authenticate success, global_id 204228 2021-05-25T12:06:53.683 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.666+0000 7f9efc8ed700 1 -- 172.21.15.102:0/1938196027 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f9eec0015f0 con 0x7f9ef8090d70 2021-05-25T12:06:53.683 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.666+0000 7f9effb53700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:06:53.684 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.666+0000 7f9efc8ed700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:53.684 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.666+0000 7f9efc8ed700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:06:53.684 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.666+0000 7f9efc8ed700 10 monclient: dump: 2021-05-25T12:06:53.684 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:53.685 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:53.685 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:53.685 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:53.685 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:53.686 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:53.686 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:53.686 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:53.686 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:53.687 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:53.687 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.666+0000 7f9effb53700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:06:53.687 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.666+0000 7f9efc8ed700 1 -- 172.21.15.102:0/1938196027 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9eec0018a0 con 0x7f9ef8090d70 2021-05-25T12:06:53.687 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.666+0000 7f9efc8ed700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:06:53.688 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.666+0000 7f9efc8ed700 1 -- 172.21.15.102:0/1938196027 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f9eec001c30 con 0x7f9ef8090d70 2021-05-25T12:06:53.688 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.666+0000 7f9effb53700 10 monclient: get_monmap_and_config success 2021-05-25T12:06:53.688 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.666+0000 7f9effb53700 4 set_mon_vals no callback set 2021-05-25T12:06:53.688 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.666+0000 7f9efc8ed700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:53.689 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.666+0000 7f9efc8ed700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:06:53.689 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.666+0000 7f9efc8ed700 10 monclient: dump: 2021-05-25T12:06:53.689 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:53.689 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:53.690 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:53.690 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:53.691 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:53.691 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:53.691 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:53.692 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:53.692 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:53.693 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:53.693 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.667+0000 7f9effb53700 10 monclient: shutdown 2021-05-25T12:06:53.694 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.667+0000 7f9effb53700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:06:53.694 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.667+0000 7f9effb53700 1 -- 172.21.15.102:0/1938196027 >> v1:172.21.15.102:6789/0 conn(0x7f9ef8090d70 legacy=0x7f9ef8091200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:53.694 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.667+0000 7f9effb53700 1 -- 172.21.15.102:0/1938196027 shutdown_connections 2021-05-25T12:06:53.695 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.667+0000 7f9effb53700 1 -- 172.21.15.102:0/1938196027 >> 172.21.15.102:0/1938196027 conn(0x7f9ef809b370 msgr2=0x7f9ef8097430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:06:53.695 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.667+0000 7f9effb53700 1 -- 172.21.15.102:0/1938196027 shutdown_connections 2021-05-25T12:06:53.695 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.667+0000 7f9effb53700 1 -- 172.21.15.102:0/1938196027 wait complete. 2021-05-25T12:06:53.696 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.667+0000 7f9effb53700 10 monclient: build_initial_monmap 2021-05-25T12:06:53.696 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.667+0000 7f9effb53700 10 monclient: monmap: 2021-05-25T12:06:53.696 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:06:53.696 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:06:53.696 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:06:53.697 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:06:53.697 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:06:53.697 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:06:53.697 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:06:53.698 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:06:53.698 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:06:53.698 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:53.698 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.668+0000 7f9effb53700 1 librados: starting msgr at 2021-05-25T12:06:53.698 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.668+0000 7f9effb53700 1 librados: starting objecter 2021-05-25T12:06:53.699 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.668+0000 7f9effb53700 1 Processor -- start 2021-05-25T12:06:53.699 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.668+0000 7f9effb53700 1 -- start start 2021-05-25T12:06:53.699 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.668+0000 7f9effb53700 1 librados: setting wanted keys 2021-05-25T12:06:53.699 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.668+0000 7f9effb53700 1 librados: calling monclient init 2021-05-25T12:06:53.700 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.668+0000 7f9effb53700 10 monclient: init 2021-05-25T12:06:53.700 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.668+0000 7f9effb53700 10 monclient: _reopen_session rank -1 2021-05-25T12:06:53.700 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.668+0000 7f9effb53700 10 monclient: _add_conns ranks=[2,0,1] 2021-05-25T12:06:53.700 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.668+0000 7f9effb53700 10 monclient(hunting): picked mon.noname-c con 0x7f9ef8090990 addr v1:172.21.15.102:6790/0 2021-05-25T12:06:53.701 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.668+0000 7f9effb53700 10 monclient(hunting): picked mon.noname-a con 0x7f9ef8090d70 addr v1:172.21.15.102:6789/0 2021-05-25T12:06:53.701 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.668+0000 7f9effb53700 10 monclient(hunting): picked mon.noname-b con 0x7f9ef812e330 addr v1:172.21.15.181:6789/0 2021-05-25T12:06:53.701 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.668+0000 7f9effb53700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ef8091fe0 con 0x7f9ef8090d70 2021-05-25T12:06:53.701 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.668+0000 7f9effb53700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ef812a1a0 con 0x7f9ef812e330 2021-05-25T12:06:53.702 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.668+0000 7f9effb53700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ef80687f0 con 0x7f9ef8090990 2021-05-25T12:06:53.702 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.668+0000 7f9effb53700 10 monclient(hunting): _renew_subs 2021-05-25T12:06:53.702 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.668+0000 7f9effb53700 10 monclient(hunting): authenticate will time out at 1214.265796s 2021-05-25T12:06:53.702 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.668+0000 7f9efd0ee700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f9ef8090d70 0x7f9ef8126620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33466/0 (socket says 172.21.15.102:33466) 2021-05-25T12:06:53.702 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.668+0000 7f9efd0ee700 1 -- 172.21.15.102:0/363420951 learned_addr learned my addr 172.21.15.102:0/363420951 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:06:53.703 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.668+0000 7f9efd8ef700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f9ef8090990 0x7f9ef813ad00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:56574/0 (socket says 172.21.15.102:56574) 2021-05-25T12:06:53.703 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.668+0000 7f9ee5ffb700 10 client.?.objecter ms_handle_connect 0x7f9ef8090d70 2021-05-25T12:06:53.703 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.668+0000 7f9ee5ffb700 10 client.?.objecter resend_mon_ops 2021-05-25T12:06:53.703 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.668+0000 7f9ee5ffb700 10 client.?.objecter ms_handle_connect 0x7f9ef8090990 2021-05-25T12:06:53.704 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.668+0000 7f9ee5ffb700 10 client.?.objecter resend_mon_ops 2021-05-25T12:06:53.704 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.669+0000 7f9ee5ffb700 10 client.?.objecter ms_handle_connect 0x7f9ef812e330 2021-05-25T12:06:53.704 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.669+0000 7f9ee5ffb700 10 client.?.objecter resend_mon_ops 2021-05-25T12:06:53.704 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.669+0000 7f9ee5ffb700 1 -- 172.21.15.102:0/363420951 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 135727691 0 0) 0x7f9ef80687f0 con 0x7f9ef8090990 2021-05-25T12:06:53.704 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.669+0000 7f9ee5ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:53.705 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.669+0000 7f9ee5ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:53.705 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.669+0000 7f9ee5ffb700 10 monclient(hunting): my global_id is 184571 2021-05-25T12:06:53.705 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.669+0000 7f9ee5ffb700 1 -- 172.21.15.102:0/363420951 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ed8004c10 con 0x7f9ef8090990 2021-05-25T12:06:53.705 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.669+0000 7f9ee5ffb700 1 -- 172.21.15.102:0/363420951 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3855170972 0 0) 0x7f9ef812a1a0 con 0x7f9ef812e330 2021-05-25T12:06:53.705 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.669+0000 7f9ee5ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:53.706 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.669+0000 7f9ee5ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:53.706 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.669+0000 7f9ee5ffb700 10 monclient(hunting): my global_id is 224098 2021-05-25T12:06:53.706 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.669+0000 7f9ee5ffb700 1 -- 172.21.15.102:0/363420951 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ef80687f0 con 0x7f9ef812e330 2021-05-25T12:06:53.707 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.669+0000 7f9ee5ffb700 1 -- 172.21.15.102:0/363420951 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2285104073 0 0) 0x7f9ef8091fe0 con 0x7f9ef8090d70 2021-05-25T12:06:53.707 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.669+0000 7f9ee5ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:53.707 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.669+0000 7f9ee5ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:53.707 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.669+0000 7f9ee5ffb700 10 monclient(hunting): my global_id is 204231 2021-05-25T12:06:53.708 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.669+0000 7f9ee5ffb700 1 -- 172.21.15.102:0/363420951 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ef812a1a0 con 0x7f9ef8090d70 2021-05-25T12:06:53.708 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.669+0000 7f9ee5ffb700 1 -- 172.21.15.102:0/363420951 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2597556214 0 0) 0x7f9ed8004c10 con 0x7f9ef8090990 2021-05-25T12:06:53.708 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.669+0000 7f9ee5ffb700 1 -- 172.21.15.102:0/363420951 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9ef8091fe0 con 0x7f9ef8090990 2021-05-25T12:06:53.708 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.669+0000 7f9ee5ffb700 1 -- 172.21.15.102:0/363420951 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2873453177 0 0) 0x7f9ef80687f0 con 0x7f9ef812e330 2021-05-25T12:06:53.708 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.669+0000 7f9ee5ffb700 1 -- 172.21.15.102:0/363420951 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9ed8004c10 con 0x7f9ef812e330 2021-05-25T12:06:53.709 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.669+0000 7f9ee5ffb700 1 -- 172.21.15.102:0/363420951 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f9ee8001a80 con 0x7f9ef8090990 2021-05-25T12:06:53.709 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.669+0000 7f9ee5ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:53.709 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.669+0000 7f9ee5ffb700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:06:53.709 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.669+0000 7f9ee5ffb700 10 monclient(hunting): dump: 2021-05-25T12:06:53.709 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:53.710 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:53.710 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:53.710 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:53.710 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:53.711 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:53.711 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:53.712 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:53.712 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:53.712 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:53.713 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.669+0000 7f9ee5ffb700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:06:53.713 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.669+0000 7f9ee5ffb700 1 -- 172.21.15.102:0/363420951 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f9ef4001ab0 con 0x7f9ef812e330 2021-05-25T12:06:53.713 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.669+0000 7f9ee5ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:53.714 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.669+0000 7f9ee5ffb700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2021-05-25T12:06:53.714 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.669+0000 7f9ee5ffb700 10 monclient(hunting): dump: 2021-05-25T12:06:53.714 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:53.714 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:53.714 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:53.715 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:53.715 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:53.715 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:53.715 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:53.716 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:53.716 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:53.716 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:53.716 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.669+0000 7f9ee5ffb700 1 -- 172.21.15.102:0/363420951 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1316621851 0 0) 0x7f9ef812a1a0 con 0x7f9ef8090d70 2021-05-25T12:06:53.717 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.669+0000 7f9ee5ffb700 1 -- 172.21.15.102:0/363420951 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9ef80687f0 con 0x7f9ef8090d70 2021-05-25T12:06:53.717 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.669+0000 7f9ee5ffb700 1 -- 172.21.15.102:0/363420951 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f9eec001670 con 0x7f9ef8090d70 2021-05-25T12:06:53.717 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.669+0000 7f9ee5ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:53.717 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.669+0000 7f9ee5ffb700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2021-05-25T12:06:53.718 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.669+0000 7f9ee5ffb700 10 monclient(hunting): dump: 2021-05-25T12:06:53.718 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:53.718 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:53.718 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:53.718 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:53.719 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:53.719 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:53.719 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:53.719 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:53.720 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:53.720 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:53.720 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.669+0000 7f9ee5ffb700 1 -- 172.21.15.102:0/363420951 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4185349679 0 0) 0x7f9ed8004c10 con 0x7f9ef812e330 2021-05-25T12:06:53.720 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.669+0000 7f9ee5ffb700 1 -- 172.21.15.102:0/363420951 >> v1:172.21.15.102:6790/0 conn(0x7f9ef8090990 legacy=0x7f9ef813ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:53.721 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.669+0000 7f9ee5ffb700 1 -- 172.21.15.102:0/363420951 >> v1:172.21.15.102:6789/0 conn(0x7f9ef8090d70 legacy=0x7f9ef8126620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:53.721 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.669+0000 7f9ee5ffb700 10 monclient: _finish_hunting 0 2021-05-25T12:06:53.721 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.669+0000 7f9ee5ffb700 1 monclient: found mon.b 2021-05-25T12:06:53.721 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.669+0000 7f9ee5ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:06:53.722 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.669+0000 7f9ee5ffb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T12:06:53.722 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.669+0000 7f9ee5ffb700 1 -- 172.21.15.102:0/363420951 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ef8068a90 con 0x7f9ef812e330 2021-05-25T12:06:53.722 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.669+0000 7f9ee5ffb700 10 monclient: _finish_auth 0 2021-05-25T12:06:53.722 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.669+0000 7f9ee5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:06:53.723 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.669+0000 7f9effb53700 5 monclient: authenticate success, global_id 224098 2021-05-25T12:06:53.723 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.669+0000 7f9effb53700 10 monclient: _renew_subs 2021-05-25T12:06:53.723 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.669+0000 7f9effb53700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T12:06:53.723 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.669+0000 7f9effb53700 1 -- 172.21.15.102:0/363420951 --> v1:172.21.15.181:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ef80669e0 con 0x7f9ef812e330 2021-05-25T12:06:53.723 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.669+0000 7f9ee5ffb700 1 -- 172.21.15.102:0/363420951 <== mon.1 v1:172.21.15.181:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9ef4001c50 con 0x7f9ef812e330 2021-05-25T12:06:53.724 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.669+0000 7f9ee5ffb700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:06:53.724 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.670+0000 7f9ee5ffb700 1 -- 172.21.15.102:0/363420951 <== mon.1 v1:172.21.15.181:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f9ef4001fe0 con 0x7f9ef812e330 2021-05-25T12:06:53.724 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.670+0000 7f9ee5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:53.724 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.670+0000 7f9ee5ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T12:06:53.725 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.670+0000 7f9ee5ffb700 10 monclient: dump: 2021-05-25T12:06:53.725 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:53.725 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:53.725 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:53.725 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:53.726 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:53.726 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:53.726 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:53.726 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:53.727 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:53.727 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:53.727 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.670+0000 7f9effb53700 10 client.224098.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:06:53.727 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.670+0000 7f9ee7fff700 4 set_mon_vals no callback set 2021-05-25T12:06:53.728 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.670+0000 7f9effb53700 10 monclient: _renew_subs 2021-05-25T12:06:53.728 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.670+0000 7f9effb53700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T12:06:53.728 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.670+0000 7f9effb53700 1 -- 172.21.15.102:0/363420951 --> v1:172.21.15.181:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ef8126d30 con 0x7f9ef812e330 2021-05-25T12:06:53.728 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.670+0000 7f9effb53700 1 librados: init done 2021-05-25T12:06:53.728 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.670+0000 7f9ee5ffb700 1 -- 172.21.15.102:0/363420951 <== mon.1 v1:172.21.15.181:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f9ef4002290 con 0x7f9ef812e330 2021-05-25T12:06:53.729 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.670+0000 7f9ee5ffb700 1 -- 172.21.15.102:0/363420951 <== mon.1 v1:172.21.15.181:6789/0 8 ==== osd_map(217..217 src has 1..217) v4 ==== 5439+0+0 (unknown 3413788257 0 0) 0x7f9ef403b0b0 con 0x7f9ef812e330 2021-05-25T12:06:53.729 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.670+0000 7f9ee5ffb700 10 client.224098.objecter ms_dispatch 0x7f9ef8001550 osd_map(217..217 src has 1..217) v4 2021-05-25T12:06:53.729 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.670+0000 7f9ee5ffb700 3 client.224098.objecter handle_osd_map got epochs [217,217] > 0 2021-05-25T12:06:53.729 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.670+0000 7f9ee5ffb700 3 client.224098.objecter handle_osd_map decoding full epoch 217 2021-05-25T12:06:53.730 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.670+0000 7f9ee5ffb700 20 client.224098.objecter dump_active .. 0 homeless 2021-05-25T12:06:53.730 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.670+0000 7f9efd8ef700 10 monclient: get_auth_request con 0x7f9ed8033520 auth_method 0 2021-05-25T12:06:53.730 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.670+0000 7f9effb53700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:06:53.730 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.670+0000 7f9effb53700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:06:53.731 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.670+0000 7f9effb53700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T12:06:53.731 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.670+0000 7f9effb53700 1 -- 172.21.15.102:0/363420951 --> v1:172.21.15.181:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ef8094b50 con 0x7f9ef812e330 2021-05-25T12:06:53.731 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.671+0000 7f9ee5ffb700 10 client.224098.objecter ms_handle_connect 0x7f9ed8033520 2021-05-25T12:06:53.731 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.675+0000 7f9ee5ffb700 1 -- 172.21.15.102:0/363420951 <== mon.1 v1:172.21.15.181:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7f9ef4002a40 con 0x7f9ef812e330 2021-05-25T12:06:53.731 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.675+0000 7f9ee5ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:06:53.732 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.675+0000 7f9ee5ffb700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:06:53.893 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.891+0000 7f9effb53700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2021-05-25T12:06:53.893 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.891+0000 7f9effb53700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2021-05-25T12:06:53.893 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.891+0000 7f9effb53700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T12:06:53.894 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.891+0000 7f9effb53700 1 -- 172.21.15.102:0/363420951 --> v1:172.21.15.181:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f9ef8059730 con 0x7f9ef812e330 2021-05-25T12:06:53.894 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.892+0000 7f9ee5ffb700 1 -- 172.21.15.102:0/363420951 <== mon.1 v1:172.21.15.181:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1342 (unknown 254746657 0 4041394906) 0x7f9ef40103d0 con 0x7f9ef812e330 2021-05-25T12:06:53.894 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.892+0000 7f9ee5ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2021-05-25T12:06:53.894 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.892+0000 7f9ee5ffb700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:06:53.895 INFO:teuthology.orchestra.run.smithi102.stdout:{"election_epoch":58,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138303579357183","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"20ee91b6-8935-4b98-b868-85c03789ccc4","modified":"2021-05-25T11:59:11.011522Z","created":"2021-05-25T11:59:11.011522Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6790","nonce":0}]},"addr":"172.21.15.102:6790/0","public_addr":"172.21.15.102:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2021-05-25T12:06:53.897 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.893+0000 7f9effb53700 10 librados: watch_flush enter 2021-05-25T12:06:53.897 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.893+0000 7f9effb53700 10 librados: watch_flush exit 2021-05-25T12:06:53.897 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.893+0000 7f9effb53700 20 client.224098.objecter shutdown clearing up homeless session... 2021-05-25T12:06:53.897 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.893+0000 7f9effb53700 10 client.224098.objecter successfully canceled tick 2021-05-25T12:06:53.897 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.893+0000 7f9effb53700 1 -- 172.21.15.102:0/363420951 >> v1:172.21.15.181:6812/29582 conn(0x7f9ed8033520 legacy=0x7f9ed80359e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:53.898 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.893+0000 7f9effb53700 10 monclient: shutdown 2021-05-25T12:06:53.898 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.893+0000 7f9effb53700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:06:53.898 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.893+0000 7f9effb53700 1 -- 172.21.15.102:0/363420951 >> v1:172.21.15.181:6789/0 conn(0x7f9ef812e330 legacy=0x7f9ef81430b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:53.898 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.893+0000 7f9effb53700 1 -- 172.21.15.102:0/363420951 shutdown_connections 2021-05-25T12:06:53.899 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.893+0000 7f9effb53700 1 -- 172.21.15.102:0/363420951 >> 172.21.15.102:0/363420951 conn(0x7f9ef809b370 msgr2=0x7f9ef8099090 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:06:53.899 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.893+0000 7f9effb53700 1 -- 172.21.15.102:0/363420951 shutdown_connections 2021-05-25T12:06:53.899 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.893+0000 7f9effb53700 1 -- 172.21.15.102:0/363420951 wait complete. 2021-05-25T12:06:53.899 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:53.893+0000 7f9effb53700 1 librados: shutdown 2021-05-25T12:06:53.906 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2021-05-25T12:06:53.906 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2021-05-25T12:06:54.018 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.017+0000 7f0aa1331700 10 monclient: get_monmap_and_config 2021-05-25T12:06:54.019 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.017+0000 7f0aa1331700 10 monclient: build_initial_monmap 2021-05-25T12:06:54.019 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.017+0000 7f0aa1331700 10 monclient: monmap: 2021-05-25T12:06:54.019 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:06:54.019 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:54.019 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:06:54.017915+0000 2021-05-25T12:06:54.020 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:06:54.017915+0000 2021-05-25T12:06:54.020 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:06:54.020 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:54.020 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:06:54.021 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:06:54.021 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:06:54.021 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:54.021 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.017+0000 7f0aa1331700 1 Processor -- start 2021-05-25T12:06:54.021 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.017+0000 7f0aa1331700 1 -- start start 2021-05-25T12:06:54.022 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.017+0000 7f0aa1331700 10 monclient: init 2021-05-25T12:06:54.036 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.017+0000 7f0aa1331700 10 monclient: _reopen_session rank -1 2021-05-25T12:06:54.036 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.017+0000 7f0aa1331700 10 monclient: _add_conns ranks=[0,2,1] 2021-05-25T12:06:54.037 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.017+0000 7f0aa1331700 10 monclient(hunting): picked mon.noname-a con 0x7f0a9c090990 addr v1:172.21.15.102:6789/0 2021-05-25T12:06:54.037 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.017+0000 7f0aa1331700 10 monclient(hunting): picked mon.noname-b con 0x7f0a9c090d70 addr v1:172.21.15.102:6790/0 2021-05-25T12:06:54.038 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.017+0000 7f0aa1331700 10 monclient(hunting): picked mon.noname-c con 0x7f0a9c12e330 addr v1:172.21.15.181:6789/0 2021-05-25T12:06:54.038 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.018+0000 7f0aa1331700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a9c096fa0 con 0x7f0a9c090990 2021-05-25T12:06:54.038 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.018+0000 7f0aa1331700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a9c0926a0 con 0x7f0a9c12e330 2021-05-25T12:06:54.039 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.018+0000 7f0aa1331700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a9c092940 con 0x7f0a9c090d70 2021-05-25T12:06:54.039 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.018+0000 7f0aa1331700 10 monclient(hunting): _renew_subs 2021-05-25T12:06:54.039 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.018+0000 7f0aa1331700 10 monclient(hunting): authenticate will time out at 1214.615037s 2021-05-25T12:06:54.040 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.018+0000 7f0a9affd700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f0a9c090990 0x7f0a9c093240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33472/0 (socket says 172.21.15.102:33472) 2021-05-25T12:06:54.040 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.018+0000 7f0a9affd700 1 -- 172.21.15.102:0/1861837479 learned_addr learned my addr 172.21.15.102:0/1861837479 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:06:54.041 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.018+0000 7f0a9a7fc700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f0a9c090d70 0x7f0a9c091200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:56584/0 (socket says 172.21.15.102:56584) 2021-05-25T12:06:54.041 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.018+0000 7f0a99ffb700 1 -- 172.21.15.102:0/1861837479 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1924661754 0 0) 0x7f0a9c092940 con 0x7f0a9c090d70 2021-05-25T12:06:54.042 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.018+0000 7f0a99ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:54.042 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.018+0000 7f0a99ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:54.042 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.018+0000 7f0a99ffb700 10 monclient(hunting): my global_id is 184580 2021-05-25T12:06:54.043 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.018+0000 7f0a99ffb700 1 -- 172.21.15.102:0/1861837479 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a88003270 con 0x7f0a9c090d70 2021-05-25T12:06:54.043 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.018+0000 7f0a99ffb700 1 -- 172.21.15.102:0/1861837479 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2294745912 0 0) 0x7f0a9c096fa0 con 0x7f0a9c090990 2021-05-25T12:06:54.043 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.018+0000 7f0a99ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:54.043 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.018+0000 7f0a99ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:54.044 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.018+0000 7f0a99ffb700 10 monclient(hunting): my global_id is 204240 2021-05-25T12:06:54.044 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.018+0000 7f0a99ffb700 1 -- 172.21.15.102:0/1861837479 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a9c092940 con 0x7f0a9c090990 2021-05-25T12:06:54.044 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.018+0000 7f0a99ffb700 1 -- 172.21.15.102:0/1861837479 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3024448682 0 0) 0x7f0a88003270 con 0x7f0a9c090d70 2021-05-25T12:06:54.044 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.018+0000 7f0a99ffb700 1 -- 172.21.15.102:0/1861837479 >> v1:172.21.15.181:6789/0 conn(0x7f0a9c12e330 legacy=0x7f0a9c130720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:54.045 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.018+0000 7f0a99ffb700 1 -- 172.21.15.102:0/1861837479 >> v1:172.21.15.102:6789/0 conn(0x7f0a9c090990 legacy=0x7f0a9c093240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:54.045 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.018+0000 7f0a99ffb700 10 monclient: _finish_hunting 0 2021-05-25T12:06:54.045 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.018+0000 7f0a99ffb700 1 monclient: found mon.noname-b 2021-05-25T12:06:54.045 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.018+0000 7f0a99ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:06:54.046 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.018+0000 7f0a99ffb700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T12:06:54.046 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.018+0000 7f0a99ffb700 1 -- 172.21.15.102:0/1861837479 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a9c092be0 con 0x7f0a9c090d70 2021-05-25T12:06:54.046 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.018+0000 7f0a99ffb700 10 monclient: _finish_auth 0 2021-05-25T12:06:54.046 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.018+0000 7f0a99ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:06:54.046 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.018+0000 7f0aa1331700 5 monclient: authenticate success, global_id 184580 2021-05-25T12:06:54.047 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.018+0000 7f0a99ffb700 1 -- 172.21.15.102:0/1861837479 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f0a84001610 con 0x7f0a9c090d70 2021-05-25T12:06:54.047 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.018+0000 7f0a99ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:54.047 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.018+0000 7f0a99ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:06:54.047 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.018+0000 7f0a99ffb700 10 monclient: dump: 2021-05-25T12:06:54.048 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:54.048 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:54.048 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:54.048 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:54.048 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:54.049 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:54.049 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:54.049 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:54.049 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:54.050 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:54.050 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.018+0000 7f0aa1331700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:06:54.050 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.018+0000 7f0a99ffb700 1 -- 172.21.15.102:0/1861837479 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0a840017c0 con 0x7f0a9c090d70 2021-05-25T12:06:54.050 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.018+0000 7f0a99ffb700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:06:54.050 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.018+0000 7f0a99ffb700 1 -- 172.21.15.102:0/1861837479 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f0a84001b50 con 0x7f0a9c090d70 2021-05-25T12:06:54.051 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.018+0000 7f0aa1331700 10 monclient: get_monmap_and_config success 2021-05-25T12:06:54.051 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.018+0000 7f0aa1331700 4 set_mon_vals no callback set 2021-05-25T12:06:54.051 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.018+0000 7f0a99ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:54.051 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.018+0000 7f0a99ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:06:54.051 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.018+0000 7f0a99ffb700 10 monclient: dump: 2021-05-25T12:06:54.052 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:54.052 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:54.052 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:54.052 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:54.053 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:54.053 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:54.053 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:54.053 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:54.053 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:54.054 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:54.054 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.019+0000 7f0aa1331700 10 monclient: shutdown 2021-05-25T12:06:54.054 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.019+0000 7f0aa1331700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:06:54.054 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.019+0000 7f0aa1331700 1 -- 172.21.15.102:0/1861837479 >> v1:172.21.15.102:6790/0 conn(0x7f0a9c090d70 legacy=0x7f0a9c091200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:54.054 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.019+0000 7f0aa1331700 1 -- 172.21.15.102:0/1861837479 shutdown_connections 2021-05-25T12:06:54.055 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.019+0000 7f0aa1331700 1 -- 172.21.15.102:0/1861837479 >> 172.21.15.102:0/1861837479 conn(0x7f0a9c09b370 msgr2=0x7f0a9c097430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:06:54.055 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.019+0000 7f0aa1331700 1 -- 172.21.15.102:0/1861837479 shutdown_connections 2021-05-25T12:06:54.055 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.019+0000 7f0aa1331700 1 -- 172.21.15.102:0/1861837479 wait complete. 2021-05-25T12:06:54.055 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.019+0000 7f0aa1331700 10 monclient: build_initial_monmap 2021-05-25T12:06:54.056 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.019+0000 7f0aa1331700 10 monclient: monmap: 2021-05-25T12:06:54.056 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:06:54.056 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:06:54.056 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:06:54.056 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:06:54.057 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:06:54.057 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:06:54.057 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:06:54.058 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:06:54.058 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:06:54.058 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:54.058 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.020+0000 7f0aa1331700 1 librados: starting msgr at 2021-05-25T12:06:54.058 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.020+0000 7f0aa1331700 1 librados: starting objecter 2021-05-25T12:06:54.059 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.020+0000 7f0aa1331700 1 Processor -- start 2021-05-25T12:06:54.059 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.020+0000 7f0aa1331700 1 -- start start 2021-05-25T12:06:54.059 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.020+0000 7f0aa1331700 1 librados: setting wanted keys 2021-05-25T12:06:54.059 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.020+0000 7f0aa1331700 1 librados: calling monclient init 2021-05-25T12:06:54.060 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.020+0000 7f0aa1331700 10 monclient: init 2021-05-25T12:06:54.060 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.020+0000 7f0aa1331700 10 monclient: _reopen_session rank -1 2021-05-25T12:06:54.060 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.020+0000 7f0aa1331700 10 monclient: _add_conns ranks=[2,1,0] 2021-05-25T12:06:54.060 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.020+0000 7f0aa1331700 10 monclient(hunting): picked mon.noname-c con 0x7f0a9c090990 addr v1:172.21.15.102:6790/0 2021-05-25T12:06:54.060 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.020+0000 7f0aa1331700 10 monclient(hunting): picked mon.noname-b con 0x7f0a9c090d70 addr v1:172.21.15.181:6789/0 2021-05-25T12:06:54.061 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.020+0000 7f0aa1331700 10 monclient(hunting): picked mon.noname-a con 0x7f0a9c12e330 addr v1:172.21.15.102:6789/0 2021-05-25T12:06:54.061 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.020+0000 7f0aa1331700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a9c12e0a0 con 0x7f0a9c12e330 2021-05-25T12:06:54.061 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.020+0000 7f0aa1331700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a9c111d00 con 0x7f0a9c090d70 2021-05-25T12:06:54.061 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.020+0000 7f0aa1331700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a9c111fa0 con 0x7f0a9c090990 2021-05-25T12:06:54.061 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.020+0000 7f0aa1331700 10 monclient(hunting): _renew_subs 2021-05-25T12:06:54.062 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.020+0000 7f0aa1331700 10 monclient(hunting): authenticate will time out at 1214.617720s 2021-05-25T12:06:54.062 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.020+0000 7f0a9affd700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f0a9c090990 0x7f0a9c12c410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:56588/0 (socket says 172.21.15.102:56588) 2021-05-25T12:06:54.062 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.020+0000 7f0a9b7fe700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f0a9c12e330 0x7f0a9c143830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33482/0 (socket says 172.21.15.102:33482) 2021-05-25T12:06:54.062 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.020+0000 7f0a9affd700 1 -- 172.21.15.102:0/1286269510 learned_addr learned my addr 172.21.15.102:0/1286269510 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:06:54.063 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.020+0000 7f0a82ffd700 10 client.?.objecter ms_handle_connect 0x7f0a9c090990 2021-05-25T12:06:54.063 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.020+0000 7f0a82ffd700 10 client.?.objecter resend_mon_ops 2021-05-25T12:06:54.063 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.020+0000 7f0a82ffd700 10 client.?.objecter ms_handle_connect 0x7f0a9c12e330 2021-05-25T12:06:54.063 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.020+0000 7f0a82ffd700 10 client.?.objecter resend_mon_ops 2021-05-25T12:06:54.063 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.020+0000 7f0a82ffd700 10 client.?.objecter ms_handle_connect 0x7f0a9c090d70 2021-05-25T12:06:54.064 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.020+0000 7f0a82ffd700 10 client.?.objecter resend_mon_ops 2021-05-25T12:06:54.064 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.020+0000 7f0a82ffd700 1 -- 172.21.15.102:0/1286269510 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1251274937 0 0) 0x7f0a9c111fa0 con 0x7f0a9c090990 2021-05-25T12:06:54.064 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.021+0000 7f0a82ffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:54.064 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.021+0000 7f0a82ffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:54.064 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.021+0000 7f0a82ffd700 10 monclient(hunting): my global_id is 184583 2021-05-25T12:06:54.065 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.021+0000 7f0a82ffd700 1 -- 172.21.15.102:0/1286269510 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a7c004c10 con 0x7f0a9c090990 2021-05-25T12:06:54.065 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.021+0000 7f0a82ffd700 1 -- 172.21.15.102:0/1286269510 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1679783243 0 0) 0x7f0a9c12e0a0 con 0x7f0a9c12e330 2021-05-25T12:06:54.065 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.021+0000 7f0a82ffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:54.065 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.021+0000 7f0a82ffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:54.066 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.021+0000 7f0a82ffd700 10 monclient(hunting): my global_id is 204243 2021-05-25T12:06:54.066 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.021+0000 7f0a82ffd700 1 -- 172.21.15.102:0/1286269510 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a9c111fa0 con 0x7f0a9c12e330 2021-05-25T12:06:54.066 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.021+0000 7f0a82ffd700 1 -- 172.21.15.102:0/1286269510 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2610048342 0 0) 0x7f0a9c111d00 con 0x7f0a9c090d70 2021-05-25T12:06:54.066 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.021+0000 7f0a82ffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:54.066 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.021+0000 7f0a82ffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:54.067 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.021+0000 7f0a82ffd700 10 monclient(hunting): my global_id is 224110 2021-05-25T12:06:54.067 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.021+0000 7f0a82ffd700 1 -- 172.21.15.102:0/1286269510 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a9c12e0a0 con 0x7f0a9c090d70 2021-05-25T12:06:54.067 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.021+0000 7f0a82ffd700 1 -- 172.21.15.102:0/1286269510 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 45003394 0 0) 0x7f0a7c004c10 con 0x7f0a9c090990 2021-05-25T12:06:54.067 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.021+0000 7f0a82ffd700 1 -- 172.21.15.102:0/1286269510 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0a9c111d00 con 0x7f0a9c090990 2021-05-25T12:06:54.067 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.021+0000 7f0a82ffd700 1 -- 172.21.15.102:0/1286269510 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3191857282 0 0) 0x7f0a9c111fa0 con 0x7f0a9c12e330 2021-05-25T12:06:54.068 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.021+0000 7f0a82ffd700 1 -- 172.21.15.102:0/1286269510 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0a7c004c10 con 0x7f0a9c12e330 2021-05-25T12:06:54.068 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.021+0000 7f0a82ffd700 1 -- 172.21.15.102:0/1286269510 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f0a8c001df0 con 0x7f0a9c090990 2021-05-25T12:06:54.068 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.021+0000 7f0a82ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:54.068 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.021+0000 7f0a82ffd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:06:54.069 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.021+0000 7f0a82ffd700 10 monclient(hunting): dump: 2021-05-25T12:06:54.069 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:54.069 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:54.069 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:54.069 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:54.070 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:54.070 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:54.070 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:54.070 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:54.070 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:54.071 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:54.071 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.021+0000 7f0a82ffd700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:06:54.071 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.021+0000 7f0a82ffd700 1 -- 172.21.15.102:0/1286269510 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4194543469 0 0) 0x7f0a9c12e0a0 con 0x7f0a9c090d70 2021-05-25T12:06:54.071 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.021+0000 7f0a82ffd700 1 -- 172.21.15.102:0/1286269510 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0a9c111fa0 con 0x7f0a9c090d70 2021-05-25T12:06:54.072 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.021+0000 7f0a82ffd700 1 -- 172.21.15.102:0/1286269510 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f0a90001780 con 0x7f0a9c12e330 2021-05-25T12:06:54.072 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.021+0000 7f0a82ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:54.072 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.021+0000 7f0a82ffd700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2021-05-25T12:06:54.072 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.021+0000 7f0a82ffd700 10 monclient(hunting): dump: 2021-05-25T12:06:54.072 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:54.073 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:54.073 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:54.073 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:54.073 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:54.073 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:54.074 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:54.074 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:54.074 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:54.074 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:54.075 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.021+0000 7f0a82ffd700 1 -- 172.21.15.102:0/1286269510 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f0a84001430 con 0x7f0a9c090d70 2021-05-25T12:06:54.075 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.021+0000 7f0a82ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:54.075 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.021+0000 7f0a82ffd700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2021-05-25T12:06:54.075 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.021+0000 7f0a82ffd700 10 monclient(hunting): dump: 2021-05-25T12:06:54.075 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:54.076 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:54.076 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:54.076 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:54.076 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:54.077 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:54.077 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:54.077 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:54.077 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:54.077 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:54.078 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.021+0000 7f0a82ffd700 1 -- 172.21.15.102:0/1286269510 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1446453289 0 0) 0x7f0a9c111d00 con 0x7f0a9c090990 2021-05-25T12:06:54.078 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.021+0000 7f0a82ffd700 1 -- 172.21.15.102:0/1286269510 >> v1:172.21.15.181:6789/0 conn(0x7f0a9c090d70 legacy=0x7f0a9c12cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:54.078 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.021+0000 7f0a82ffd700 1 -- 172.21.15.102:0/1286269510 >> v1:172.21.15.102:6789/0 conn(0x7f0a9c12e330 legacy=0x7f0a9c143830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:54.078 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.021+0000 7f0a82ffd700 10 monclient: _finish_hunting 0 2021-05-25T12:06:54.078 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.021+0000 7f0a82ffd700 1 monclient: found mon.c 2021-05-25T12:06:54.079 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.021+0000 7f0a82ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:06:54.079 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.021+0000 7f0a82ffd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:06:54.079 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.021+0000 7f0a82ffd700 1 -- 172.21.15.102:0/1286269510 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a9c112240 con 0x7f0a9c090990 2021-05-25T12:06:54.079 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.021+0000 7f0a82ffd700 10 monclient: _finish_auth 0 2021-05-25T12:06:54.080 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.021+0000 7f0a82ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:06:54.080 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.021+0000 7f0aa1331700 5 monclient: authenticate success, global_id 184583 2021-05-25T12:06:54.080 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.021+0000 7f0aa1331700 10 monclient: _renew_subs 2021-05-25T12:06:54.080 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.021+0000 7f0aa1331700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:06:54.080 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.021+0000 7f0aa1331700 1 -- 172.21.15.102:0/1286269510 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a9c110f70 con 0x7f0a9c090990 2021-05-25T12:06:54.080 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.021+0000 7f0aa1331700 10 client.184583.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:06:54.081 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.021+0000 7f0aa1331700 10 monclient: _renew_subs 2021-05-25T12:06:54.081 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.021+0000 7f0aa1331700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:06:54.081 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.021+0000 7f0aa1331700 1 -- 172.21.15.102:0/1286269510 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a9c12d230 con 0x7f0a9c090990 2021-05-25T12:06:54.081 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.021+0000 7f0aa1331700 1 librados: init done 2021-05-25T12:06:54.082 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.021+0000 7f0a82ffd700 1 -- 172.21.15.102:0/1286269510 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0a8c0020d0 con 0x7f0a9c090990 2021-05-25T12:06:54.082 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.021+0000 7f0a82ffd700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:06:54.082 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.021+0000 7f0a82ffd700 1 -- 172.21.15.102:0/1286269510 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f0a8c004d00 con 0x7f0a9c090990 2021-05-25T12:06:54.082 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.021+0000 7f0a82ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:54.082 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.021+0000 7f0a997fa700 4 set_mon_vals no callback set 2021-05-25T12:06:54.083 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.021+0000 7f0a82ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:06:54.083 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.021+0000 7f0a82ffd700 10 monclient: dump: 2021-05-25T12:06:54.083 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:54.083 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:54.084 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:54.084 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:54.084 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:54.085 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:54.085 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:54.085 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:54.085 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:54.086 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:54.086 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.022+0000 7f0aa1331700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:06:54.086 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.022+0000 7f0aa1331700 1 -- 172.21.15.102:0/1286269510 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f0a740065a0 con 0x7f0a74002840 2021-05-25T12:06:54.086 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.022+0000 7f0a82ffd700 1 -- 172.21.15.102:0/1286269510 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f0a8c00e0a0 con 0x7f0a9c090990 2021-05-25T12:06:54.087 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.022+0000 7f0a82ffd700 10 client.184583.objecter ms_handle_connect 0x7f0a74002840 2021-05-25T12:06:54.087 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.022+0000 7f0a82ffd700 10 client.184583.objecter resend_mon_ops 2021-05-25T12:06:54.087 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.022+0000 7f0a82ffd700 1 -- 172.21.15.102:0/1286269510 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(218..218 src has 1..218) v4 ==== 5811+0+0 (unknown 452711690 0 0) 0x7f0a8c03dad0 con 0x7f0a9c090990 2021-05-25T12:06:54.087 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.022+0000 7f0a82ffd700 10 client.184583.objecter ms_dispatch 0x7f0a9c001550 osd_map(218..218 src has 1..218) v4 2021-05-25T12:06:54.087 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.022+0000 7f0a82ffd700 3 client.184583.objecter handle_osd_map got epochs [218,218] > 0 2021-05-25T12:06:54.088 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.022+0000 7f0a82ffd700 3 client.184583.objecter handle_osd_map decoding full epoch 218 2021-05-25T12:06:54.088 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.022+0000 7f0a82ffd700 20 client.184583.objecter dump_active .. 0 homeless 2021-05-25T12:06:54.088 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.022+0000 7f0a82ffd700 1 -- 172.21.15.102:0/1286269510 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1501671608 0 0) 0x7f0a740065a0 con 0x7f0a74002840 2021-05-25T12:06:54.088 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.022+0000 7f0a82ffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:54.089 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.022+0000 7f0a82ffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:54.089 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.022+0000 7f0a82ffd700 10 monclient(hunting): my global_id is 204246 2021-05-25T12:06:54.089 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.022+0000 7f0a82ffd700 1 -- 172.21.15.102:0/1286269510 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a9c111d00 con 0x7f0a74002840 2021-05-25T12:06:54.089 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.022+0000 7f0a9b7fe700 10 monclient: get_auth_request con 0x7f0a7c033470 auth_method 0 2021-05-25T12:06:54.089 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.022+0000 7f0a82ffd700 1 -- 172.21.15.102:0/1286269510 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2509201075 0 0) 0x7f0a9c111d00 con 0x7f0a74002840 2021-05-25T12:06:54.090 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.022+0000 7f0a82ffd700 1 -- 172.21.15.102:0/1286269510 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0a740065a0 con 0x7f0a74002840 2021-05-25T12:06:54.090 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.022+0000 7f0a82ffd700 1 -- 172.21.15.102:0/1286269510 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f0a840048c0 con 0x7f0a74002840 2021-05-25T12:06:54.090 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.022+0000 7f0a82ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:54.090 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.022+0000 7f0a82ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:06:54.091 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.022+0000 7f0a82ffd700 10 monclient: dump: 2021-05-25T12:06:54.091 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:54.091 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:54.091 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:54.091 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:54.092 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:54.092 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:54.092 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:54.092 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:54.093 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:54.093 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:54.093 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.022+0000 7f0a82ffd700 10 client.184583.objecter ms_handle_connect 0x7f0a7c033470 2021-05-25T12:06:54.093 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.022+0000 7f0a82ffd700 1 -- 172.21.15.102:0/1286269510 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 228052568 0 0) 0x7f0a740065a0 con 0x7f0a74002840 2021-05-25T12:06:54.093 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.022+0000 7f0a82ffd700 1 -- 172.21.15.102:0/1286269510 --> v1:172.21.15.102:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0a74009e40 con 0x7f0a74002840 2021-05-25T12:06:54.094 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.023+0000 7f0a82ffd700 1 -- 172.21.15.102:0/1286269510 <== mon.0 v1:172.21.15.102:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7f0a84004a90 con 0x7f0a74002840 2021-05-25T12:06:54.094 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.023+0000 7f0a82ffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:06:54.094 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.023+0000 7f0a82ffd700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:06:54.094 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.023+0000 7f0a82ffd700 1 -- 172.21.15.102:0/1286269510 >> v1:172.21.15.102:6789/0 conn(0x7f0a74002840 legacy=0x7f0a74004ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:54.095 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.023+0000 7f0a82ffd700 1 -- 172.21.15.102:0/1286269510 >> v1:172.21.15.102:6789/0 conn(0x7f0a74002840 legacy=0x7f0a74004ce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:06:54.095 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.029+0000 7f0aa1331700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:06:54.095 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.029+0000 7f0aa1331700 1 -- 172.21.15.102:0/1286269510 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f0a74007520 con 0x7f0a740068c0 2021-05-25T12:06:54.095 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.029+0000 7f0a82ffd700 10 client.184583.objecter ms_handle_connect 0x7f0a740068c0 2021-05-25T12:06:54.095 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.029+0000 7f0a82ffd700 10 client.184583.objecter resend_mon_ops 2021-05-25T12:06:54.096 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.030+0000 7f0a82ffd700 1 -- 172.21.15.102:0/1286269510 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3612272344 0 0) 0x7f0a74007520 con 0x7f0a740068c0 2021-05-25T12:06:54.096 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.030+0000 7f0a82ffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:54.096 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.030+0000 7f0a82ffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:54.096 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.030+0000 7f0a82ffd700 10 monclient(hunting): my global_id is 204249 2021-05-25T12:06:54.097 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.030+0000 7f0a82ffd700 1 -- 172.21.15.102:0/1286269510 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a740065a0 con 0x7f0a740068c0 2021-05-25T12:06:54.097 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.030+0000 7f0a82ffd700 1 -- 172.21.15.102:0/1286269510 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2618741805 0 0) 0x7f0a740065a0 con 0x7f0a740068c0 2021-05-25T12:06:54.097 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.030+0000 7f0a82ffd700 1 -- 172.21.15.102:0/1286269510 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0a74007520 con 0x7f0a740068c0 2021-05-25T12:06:54.097 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.030+0000 7f0a82ffd700 1 -- 172.21.15.102:0/1286269510 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f0a84005130 con 0x7f0a740068c0 2021-05-25T12:06:54.097 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.030+0000 7f0a82ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:54.098 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.030+0000 7f0a82ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:06:54.098 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.030+0000 7f0a82ffd700 10 monclient: dump: 2021-05-25T12:06:54.098 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:54.098 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:54.099 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:54.099 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:54.099 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:54.099 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:54.099 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:54.100 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:54.100 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:54.100 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:54.100 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.030+0000 7f0a82ffd700 1 -- 172.21.15.102:0/1286269510 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 546608810 0 0) 0x7f0a74007520 con 0x7f0a740068c0 2021-05-25T12:06:54.101 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.030+0000 7f0a82ffd700 1 -- 172.21.15.102:0/1286269510 --> v1:172.21.15.102:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f0a74008770 con 0x7f0a740068c0 2021-05-25T12:06:54.101 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.030+0000 7f0a82ffd700 1 -- 172.21.15.102:0/1286269510 <== mon.0 v1:172.21.15.102:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3940 (unknown 0 0 2539278332) 0x7f0a840053d0 con 0x7f0a740068c0 2021-05-25T12:06:54.101 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.030+0000 7f0a82ffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:06:54.101 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.030+0000 7f0a82ffd700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:06:54.101 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.030+0000 7f0a82ffd700 1 -- 172.21.15.102:0/1286269510 >> v1:172.21.15.102:6789/0 conn(0x7f0a740068c0 legacy=0x7f0a74006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:54.102 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.030+0000 7f0a82ffd700 1 -- 172.21.15.102:0/1286269510 >> v1:172.21.15.102:6789/0 conn(0x7f0a740068c0 legacy=0x7f0a74006d50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:06:54.102 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:06:54.102 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:06:54.103 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:06:54.103 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "peon", 2021-05-25T12:06:54.103 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 58, 2021-05-25T12:06:54.103 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:06:54.104 INFO:teuthology.orchestra.run.smithi102.stdout: 0, 2021-05-25T12:06:54.104 INFO:teuthology.orchestra.run.smithi102.stdout: 1, 2021-05-25T12:06:54.104 INFO:teuthology.orchestra.run.smithi102.stdout: 2 2021-05-25T12:06:54.104 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:06:54.104 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 0, 2021-05-25T12:06:54.105 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:06:54.105 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:06:54.105 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:06:54.105 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:06:54.106 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:06:54.106 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:06:54.106 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:06:54.106 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:06:54.107 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:06:54.107 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:06:54.107 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:06:54.107 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:06:54.108 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:06:54.108 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:06:54.108 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:06:54.108 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:06:54.108 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:06:54.109 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:06:54.109 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:06:54.109 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:06:54.109 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:06:54.109 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:06:54.110 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:06:54.110 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:06:54.110 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:54.110 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:54.110 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:06:54.111 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:06:54.111 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:06:54.111 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:06:54.111 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:06:54.111 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:06:54.112 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:06:54.112 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:06:54.112 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:06:54.112 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:06:54.113 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:06:54.113 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:06:54.113 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:06:54.113 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:06:54.113 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:06:54.114 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:06:54.114 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:06:54.114 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:06:54.114 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:06:54.114 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:06:54.115 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:06:54.115 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:06:54.115 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:06:54.115 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:06:54.115 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:06:54.116 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:06:54.116 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:54.116 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:06:54.116 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:54.117 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:06:54.117 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:06:54.117 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:06:54.117 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:06:54.117 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:54.118 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:06:54.118 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:06:54.118 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:06:54.119 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:54.119 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:54.119 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:54.119 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:06:54.120 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:06:54.120 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:06:54.120 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:06:54.120 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:06:54.120 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:54.121 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:54.121 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:06:54.121 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:06:54.121 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:06:54.121 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:06:54.122 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:54.122 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:06:54.122 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:06:54.122 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:06:54.123 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:54.123 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:54.123 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:54.123 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:06:54.124 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:06:54.124 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:06:54.124 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:06:54.124 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:06:54.125 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:54.125 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:54.125 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:06:54.125 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:06:54.126 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:06:54.126 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:06:54.126 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:54.126 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:06:54.126 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:06:54.127 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:06:54.127 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:54.127 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:54.127 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:54.128 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:06:54.128 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:06:54.128 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:06:54.128 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:06:54.129 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:06:54.129 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:54.129 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:54.129 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:54.129 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:06:54.130 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:06:54.130 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:54.130 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:06:54.130 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:06:54.130 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:06:54.131 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:54.131 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:06:54.131 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": [ 2021-05-25T12:06:54.131 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:54.131 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:06:54.132 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:06:54.132 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 2 2021-05-25T12:06:54.132 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:54.132 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:06:54.133 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:06:54.133 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:54.133 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:06:54.133 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:06:54.133 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 2 2021-05-25T12:06:54.134 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:54.134 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:54.134 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:54.135 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:06:54.135 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:06:54.137 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.030+0000 7f0aa1331700 10 librados: watch_flush enter 2021-05-25T12:06:54.137 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.030+0000 7f0aa1331700 10 librados: watch_flush exit 2021-05-25T12:06:54.138 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.030+0000 7f0aa1331700 20 client.184583.objecter shutdown clearing up homeless session... 2021-05-25T12:06:54.138 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.030+0000 7f0aa1331700 10 client.184583.objecter successfully canceled tick 2021-05-25T12:06:54.138 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.031+0000 7f0aa1331700 1 -- 172.21.15.102:0/1286269510 >> v1:172.21.15.181:6812/29582 conn(0x7f0a7c033470 legacy=0x7f0a7c035930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:54.138 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.031+0000 7f0aa1331700 10 monclient: shutdown 2021-05-25T12:06:54.138 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.031+0000 7f0aa1331700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:06:54.139 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.031+0000 7f0aa1331700 1 -- 172.21.15.102:0/1286269510 >> v1:172.21.15.102:6790/0 conn(0x7f0a9c090990 legacy=0x7f0a9c12c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:54.139 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.031+0000 7f0a9b7fe700 1 -- 172.21.15.102:0/1286269510 reap_dead start 2021-05-25T12:06:54.139 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.031+0000 7f0a9b7fe700 1 -- 172.21.15.102:0/1286269510 reap_dead start 2021-05-25T12:06:54.139 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.031+0000 7f0aa1331700 1 -- 172.21.15.102:0/1286269510 shutdown_connections 2021-05-25T12:06:54.140 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.031+0000 7f0aa1331700 1 -- 172.21.15.102:0/1286269510 >> 172.21.15.102:0/1286269510 conn(0x7f0a9c09b370 msgr2=0x7f0a9c129610 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:06:54.140 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.031+0000 7f0aa1331700 1 -- 172.21.15.102:0/1286269510 shutdown_connections 2021-05-25T12:06:54.140 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.031+0000 7f0aa1331700 1 -- 172.21.15.102:0/1286269510 wait complete. 2021-05-25T12:06:54.141 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.031+0000 7f0aa1331700 1 librados: shutdown 2021-05-25T12:06:54.141 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2021-05-25T12:06:54.253 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.251+0000 7ff71b901700 10 monclient: get_monmap_and_config 2021-05-25T12:06:54.253 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.251+0000 7ff71b901700 10 monclient: build_initial_monmap 2021-05-25T12:06:54.253 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.251+0000 7ff71b901700 10 monclient: monmap: 2021-05-25T12:06:54.254 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:06:54.254 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:54.254 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:06:54.252438+0000 2021-05-25T12:06:54.254 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:06:54.252438+0000 2021-05-25T12:06:54.255 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:06:54.255 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:54.255 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:06:54.256 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:06:54.256 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:06:54.256 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:54.256 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.251+0000 7ff71b901700 1 Processor -- start 2021-05-25T12:06:54.257 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.252+0000 7ff71b901700 1 -- start start 2021-05-25T12:06:54.257 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.252+0000 7ff71b901700 10 monclient: init 2021-05-25T12:06:54.270 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.252+0000 7ff71b901700 10 monclient: _reopen_session rank -1 2021-05-25T12:06:54.271 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.252+0000 7ff71b901700 10 monclient: _add_conns ranks=[2,0,1] 2021-05-25T12:06:54.271 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.252+0000 7ff71b901700 10 monclient(hunting): picked mon.noname-b con 0x7ff714127d30 addr v1:172.21.15.102:6790/0 2021-05-25T12:06:54.271 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.252+0000 7ff71b901700 10 monclient(hunting): picked mon.noname-a con 0x7ff714128110 addr v1:172.21.15.102:6789/0 2021-05-25T12:06:54.271 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.252+0000 7ff71b901700 10 monclient(hunting): picked mon.noname-c con 0x7ff7140965d0 addr v1:172.21.15.181:6789/0 2021-05-25T12:06:54.272 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.252+0000 7ff71b901700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff714094e60 con 0x7ff714128110 2021-05-25T12:06:54.272 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.252+0000 7ff71b901700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff714097020 con 0x7ff7140965d0 2021-05-25T12:06:54.272 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.252+0000 7ff71b901700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7140972c0 con 0x7ff714127d30 2021-05-25T12:06:54.272 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.252+0000 7ff71b901700 10 monclient(hunting): _renew_subs 2021-05-25T12:06:54.273 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.252+0000 7ff71b901700 10 monclient(hunting): authenticate will time out at 1214.849668s 2021-05-25T12:06:54.273 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.252+0000 7ff718e9c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7ff714128110 0x7ff71412d0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33492/0 (socket says 172.21.15.102:33492) 2021-05-25T12:06:54.273 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.252+0000 7ff718e9c700 1 -- 172.21.15.102:0/475603303 learned_addr learned my addr 172.21.15.102:0/475603303 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:06:54.273 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.252+0000 7ff71969d700 1 --1- 172.21.15.102:0/475603303 >> v1:172.21.15.102:6790/0 conn(0x7ff714127d30 0x7ff714094750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:56600/0 (socket says 172.21.15.102:56600) 2021-05-25T12:06:54.274 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.253+0000 7ff703fff700 1 -- 172.21.15.102:0/475603303 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1654420786 0 0) 0x7ff7140972c0 con 0x7ff714127d30 2021-05-25T12:06:54.274 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.253+0000 7ff703fff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:54.274 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.253+0000 7ff703fff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:54.274 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.253+0000 7ff703fff700 10 monclient(hunting): my global_id is 184586 2021-05-25T12:06:54.274 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.253+0000 7ff703fff700 1 -- 172.21.15.102:0/475603303 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6fc003270 con 0x7ff714127d30 2021-05-25T12:06:54.275 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.253+0000 7ff703fff700 1 -- 172.21.15.102:0/475603303 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2714254200 0 0) 0x7ff714094e60 con 0x7ff714128110 2021-05-25T12:06:54.275 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.253+0000 7ff703fff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:54.275 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.253+0000 7ff703fff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:54.275 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.253+0000 7ff703fff700 10 monclient(hunting): my global_id is 204252 2021-05-25T12:06:54.276 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.253+0000 7ff703fff700 1 -- 172.21.15.102:0/475603303 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7140972c0 con 0x7ff714128110 2021-05-25T12:06:54.276 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.253+0000 7ff703fff700 1 -- 172.21.15.102:0/475603303 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3510876599 0 0) 0x7ff7140972c0 con 0x7ff714128110 2021-05-25T12:06:54.276 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.253+0000 7ff703fff700 1 -- 172.21.15.102:0/475603303 >> v1:172.21.15.102:6790/0 conn(0x7ff714127d30 legacy=0x7ff714094750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:54.276 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.253+0000 7ff703fff700 1 -- 172.21.15.102:0/475603303 >> v1:172.21.15.181:6789/0 conn(0x7ff7140965d0 legacy=0x7ff7141307f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:54.276 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.253+0000 7ff703fff700 10 monclient: _finish_hunting 0 2021-05-25T12:06:54.277 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.253+0000 7ff703fff700 1 monclient: found mon.noname-a 2021-05-25T12:06:54.277 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.253+0000 7ff703fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:06:54.277 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.253+0000 7ff703fff700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T12:06:54.277 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.253+0000 7ff703fff700 1 -- 172.21.15.102:0/475603303 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff714097560 con 0x7ff714128110 2021-05-25T12:06:54.278 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.253+0000 7ff703fff700 10 monclient: _finish_auth 0 2021-05-25T12:06:54.278 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.253+0000 7ff703fff700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:06:54.278 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.253+0000 7ff703fff700 1 -- 172.21.15.102:0/475603303 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7ff708001610 con 0x7ff714128110 2021-05-25T12:06:54.278 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.253+0000 7ff71b901700 5 monclient: authenticate success, global_id 204252 2021-05-25T12:06:54.278 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.253+0000 7ff71b901700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:06:54.279 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.253+0000 7ff703fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:54.279 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.253+0000 7ff703fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:06:54.279 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.253+0000 7ff703fff700 10 monclient: dump: 2021-05-25T12:06:54.279 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:54.280 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:54.280 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:54.280 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:54.280 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:54.280 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:54.281 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:54.281 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:54.281 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:54.281 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:54.281 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.253+0000 7ff71b901700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:06:54.282 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.253+0000 7ff703fff700 1 -- 172.21.15.102:0/475603303 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff708001830 con 0x7ff714128110 2021-05-25T12:06:54.282 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.253+0000 7ff703fff700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:06:54.282 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.253+0000 7ff703fff700 1 -- 172.21.15.102:0/475603303 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7ff708001bc0 con 0x7ff714128110 2021-05-25T12:06:54.282 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.253+0000 7ff71b901700 10 monclient: get_monmap_and_config success 2021-05-25T12:06:54.283 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.253+0000 7ff71b901700 4 set_mon_vals no callback set 2021-05-25T12:06:54.283 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.253+0000 7ff703fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:54.283 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.253+0000 7ff703fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:06:54.283 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.253+0000 7ff703fff700 10 monclient: dump: 2021-05-25T12:06:54.284 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:54.284 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:54.284 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:54.284 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:54.284 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:54.285 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:54.285 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:54.285 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:54.285 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:54.286 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:54.286 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.253+0000 7ff71b901700 10 monclient: shutdown 2021-05-25T12:06:54.286 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.253+0000 7ff71b901700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:06:54.286 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.253+0000 7ff71b901700 1 -- 172.21.15.102:0/475603303 >> v1:172.21.15.102:6789/0 conn(0x7ff714128110 legacy=0x7ff71412d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:54.286 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.253+0000 7ff71b901700 1 -- 172.21.15.102:0/475603303 shutdown_connections 2021-05-25T12:06:54.287 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.253+0000 7ff71b901700 1 -- 172.21.15.102:0/475603303 >> 172.21.15.102:0/475603303 conn(0x7ff714062810 msgr2=0x7ff71409aeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:06:54.287 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.253+0000 7ff71b901700 1 -- 172.21.15.102:0/475603303 shutdown_connections 2021-05-25T12:06:54.287 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.253+0000 7ff71b901700 1 -- 172.21.15.102:0/475603303 wait complete. 2021-05-25T12:06:54.287 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.254+0000 7ff71b901700 10 monclient: build_initial_monmap 2021-05-25T12:06:54.287 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.254+0000 7ff71b901700 10 monclient: monmap: 2021-05-25T12:06:54.288 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:06:54.288 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:06:54.288 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:06:54.288 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:06:54.289 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:06:54.289 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:06:54.289 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:06:54.289 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:06:54.289 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:06:54.290 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:54.290 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.254+0000 7ff71b901700 1 librados: starting msgr at 2021-05-25T12:06:54.290 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.254+0000 7ff71b901700 1 librados: starting objecter 2021-05-25T12:06:54.290 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.254+0000 7ff71b901700 1 Processor -- start 2021-05-25T12:06:54.291 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.254+0000 7ff71b901700 1 -- start start 2021-05-25T12:06:54.291 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.254+0000 7ff71b901700 1 librados: setting wanted keys 2021-05-25T12:06:54.291 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.254+0000 7ff71b901700 1 librados: calling monclient init 2021-05-25T12:06:54.291 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.254+0000 7ff71b901700 10 monclient: init 2021-05-25T12:06:54.291 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.255+0000 7ff71b901700 10 monclient: _reopen_session rank -1 2021-05-25T12:06:54.292 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.255+0000 7ff71b901700 10 monclient: _add_conns ranks=[2,1,0] 2021-05-25T12:06:54.292 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.255+0000 7ff71b901700 10 monclient(hunting): picked mon.noname-c con 0x7ff7140965d0 addr v1:172.21.15.102:6790/0 2021-05-25T12:06:54.292 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.255+0000 7ff71b901700 10 monclient(hunting): picked mon.noname-b con 0x7ff714127d30 addr v1:172.21.15.181:6789/0 2021-05-25T12:06:54.292 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.255+0000 7ff71b901700 10 monclient(hunting): picked mon.noname-a con 0x7ff714128110 addr v1:172.21.15.102:6789/0 2021-05-25T12:06:54.292 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.255+0000 7ff71b901700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff714096ab0 con 0x7ff714128110 2021-05-25T12:06:54.293 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.255+0000 7ff71b901700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff714115270 con 0x7ff714127d30 2021-05-25T12:06:54.293 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.255+0000 7ff71b901700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff714115510 con 0x7ff7140965d0 2021-05-25T12:06:54.293 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.255+0000 7ff71b901700 10 monclient(hunting): _renew_subs 2021-05-25T12:06:54.294 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.255+0000 7ff71b901700 10 monclient(hunting): authenticate will time out at 1214.852386s 2021-05-25T12:06:54.294 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.255+0000 7ff71969d700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7ff7140965d0 0x7ff714124aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:56606/0 (socket says 172.21.15.102:56606) 2021-05-25T12:06:54.294 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.255+0000 7ff719e9e700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7ff714128110 0x7ff71413f910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33500/0 (socket says 172.21.15.102:33500) 2021-05-25T12:06:54.294 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.255+0000 7ff71969d700 1 -- 172.21.15.102:0/905942828 learned_addr learned my addr 172.21.15.102:0/905942828 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:06:54.294 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.255+0000 7ff7017fa700 10 client.?.objecter ms_handle_connect 0x7ff7140965d0 2021-05-25T12:06:54.295 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.255+0000 7ff7017fa700 10 client.?.objecter resend_mon_ops 2021-05-25T12:06:54.295 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.255+0000 7ff7017fa700 10 client.?.objecter ms_handle_connect 0x7ff714128110 2021-05-25T12:06:54.295 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.255+0000 7ff7017fa700 10 client.?.objecter resend_mon_ops 2021-05-25T12:06:54.295 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.255+0000 7ff7017fa700 10 client.?.objecter ms_handle_connect 0x7ff714127d30 2021-05-25T12:06:54.296 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.255+0000 7ff7017fa700 10 client.?.objecter resend_mon_ops 2021-05-25T12:06:54.296 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.255+0000 7ff7017fa700 1 -- 172.21.15.102:0/905942828 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 437114560 0 0) 0x7ff714115510 con 0x7ff7140965d0 2021-05-25T12:06:54.296 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.255+0000 7ff7017fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:54.296 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.255+0000 7ff7017fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:54.296 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.255+0000 7ff7017fa700 10 monclient(hunting): my global_id is 184589 2021-05-25T12:06:54.297 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.255+0000 7ff7017fa700 1 -- 172.21.15.102:0/905942828 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6f4004c10 con 0x7ff7140965d0 2021-05-25T12:06:54.297 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.255+0000 7ff7017fa700 1 -- 172.21.15.102:0/905942828 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1626063060 0 0) 0x7ff714096ab0 con 0x7ff714128110 2021-05-25T12:06:54.297 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.255+0000 7ff7017fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:54.297 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.255+0000 7ff7017fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:54.297 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.255+0000 7ff7017fa700 10 monclient(hunting): my global_id is 204255 2021-05-25T12:06:54.298 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.255+0000 7ff7017fa700 1 -- 172.21.15.102:0/905942828 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff714115510 con 0x7ff714128110 2021-05-25T12:06:54.298 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.255+0000 7ff7017fa700 1 -- 172.21.15.102:0/905942828 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3737499148 0 0) 0x7ff714115270 con 0x7ff714127d30 2021-05-25T12:06:54.298 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.255+0000 7ff7017fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:54.298 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.255+0000 7ff7017fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:54.299 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.255+0000 7ff7017fa700 10 monclient(hunting): my global_id is 224116 2021-05-25T12:06:54.299 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.255+0000 7ff7017fa700 1 -- 172.21.15.102:0/905942828 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff714096ab0 con 0x7ff714127d30 2021-05-25T12:06:54.299 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.255+0000 7ff7017fa700 1 -- 172.21.15.102:0/905942828 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3552229004 0 0) 0x7ff714115510 con 0x7ff714128110 2021-05-25T12:06:54.299 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.255+0000 7ff7017fa700 1 -- 172.21.15.102:0/905942828 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff714115270 con 0x7ff714128110 2021-05-25T12:06:54.299 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.255+0000 7ff7017fa700 1 -- 172.21.15.102:0/905942828 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2657049518 0 0) 0x7ff6f4004c10 con 0x7ff7140965d0 2021-05-25T12:06:54.300 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.255+0000 7ff7017fa700 1 -- 172.21.15.102:0/905942828 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff714115510 con 0x7ff7140965d0 2021-05-25T12:06:54.300 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.255+0000 7ff7017fa700 1 -- 172.21.15.102:0/905942828 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7ff710001bc0 con 0x7ff714128110 2021-05-25T12:06:54.300 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.256+0000 7ff7017fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:54.300 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.256+0000 7ff7017fa700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:06:54.300 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.256+0000 7ff7017fa700 10 monclient(hunting): dump: 2021-05-25T12:06:54.301 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:54.301 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:54.301 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:54.301 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:54.302 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:54.302 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:54.302 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:54.302 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:54.302 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:54.303 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:54.303 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.256+0000 7ff7017fa700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:06:54.303 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.256+0000 7ff7017fa700 1 -- 172.21.15.102:0/905942828 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7ff7040018d0 con 0x7ff7140965d0 2021-05-25T12:06:54.303 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.256+0000 7ff7017fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:54.304 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.256+0000 7ff7017fa700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2021-05-25T12:06:54.304 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.256+0000 7ff7017fa700 10 monclient(hunting): dump: 2021-05-25T12:06:54.304 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:54.304 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:54.304 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:54.305 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:54.305 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:54.305 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:54.305 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:54.305 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:54.306 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:54.306 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:54.306 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.256+0000 7ff7017fa700 1 -- 172.21.15.102:0/905942828 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2193201312 0 0) 0x7ff714115270 con 0x7ff714128110 2021-05-25T12:06:54.306 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.256+0000 7ff7017fa700 1 -- 172.21.15.102:0/905942828 >> v1:172.21.15.102:6790/0 conn(0x7ff7140965d0 legacy=0x7ff714124aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:54.307 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.256+0000 7ff7017fa700 1 -- 172.21.15.102:0/905942828 >> v1:172.21.15.181:6789/0 conn(0x7ff714127d30 legacy=0x7ff71413c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:54.307 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.256+0000 7ff7017fa700 10 monclient: _finish_hunting 0 2021-05-25T12:06:54.307 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.256+0000 7ff7017fa700 1 monclient: found mon.a 2021-05-25T12:06:54.307 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.256+0000 7ff7017fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:06:54.307 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.256+0000 7ff7017fa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:06:54.308 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.256+0000 7ff7017fa700 1 -- 172.21.15.102:0/905942828 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7141133a0 con 0x7ff714128110 2021-05-25T12:06:54.308 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.256+0000 7ff7017fa700 10 monclient: _finish_auth 0 2021-05-25T12:06:54.308 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.256+0000 7ff7017fa700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:06:54.308 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.256+0000 7ff71b901700 5 monclient: authenticate success, global_id 204255 2021-05-25T12:06:54.309 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.256+0000 7ff71b901700 10 monclient: _renew_subs 2021-05-25T12:06:54.309 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.256+0000 7ff71b901700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:06:54.309 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.256+0000 7ff71b901700 1 -- 172.21.15.102:0/905942828 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7141251b0 con 0x7ff714128110 2021-05-25T12:06:54.309 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.256+0000 7ff71b901700 10 client.204255.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:06:54.309 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.256+0000 7ff71b901700 10 monclient: _renew_subs 2021-05-25T12:06:54.310 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.256+0000 7ff71b901700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:06:54.310 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.256+0000 7ff71b901700 1 -- 172.21.15.102:0/905942828 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff714112840 con 0x7ff714128110 2021-05-25T12:06:54.310 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.256+0000 7ff71b901700 1 librados: init done 2021-05-25T12:06:54.310 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.256+0000 7ff7017fa700 1 -- 172.21.15.102:0/905942828 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff710002300 con 0x7ff714128110 2021-05-25T12:06:54.310 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.256+0000 7ff7017fa700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:06:54.311 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.256+0000 7ff7017fa700 1 -- 172.21.15.102:0/905942828 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7ff710002610 con 0x7ff714128110 2021-05-25T12:06:54.311 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.256+0000 7ff7017fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:54.311 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.256+0000 7ff7017fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:06:54.311 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.256+0000 7ff7037fe700 4 set_mon_vals no callback set 2021-05-25T12:06:54.312 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.256+0000 7ff7017fa700 10 monclient: dump: 2021-05-25T12:06:54.312 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:54.312 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:54.312 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:54.312 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:54.313 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:54.313 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:54.313 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:54.313 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:54.313 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:54.314 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:54.314 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.256+0000 7ff71b901700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:06:54.314 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.256+0000 7ff71b901700 1 -- 172.21.15.102:0/905942828 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff6ec0065a0 con 0x7ff6ec002840 2021-05-25T12:06:54.314 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.256+0000 7ff7017fa700 1 -- 172.21.15.102:0/905942828 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7ff7100027f0 con 0x7ff714128110 2021-05-25T12:06:54.315 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.257+0000 7ff7017fa700 10 client.204255.objecter ms_handle_connect 0x7ff6ec002840 2021-05-25T12:06:54.315 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.257+0000 7ff7017fa700 10 client.204255.objecter resend_mon_ops 2021-05-25T12:06:54.315 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.257+0000 7ff7017fa700 1 -- 172.21.15.102:0/905942828 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(218..218 src has 1..218) v4 ==== 5811+0+0 (unknown 452711690 0 0) 0x7ff71003d920 con 0x7ff714128110 2021-05-25T12:06:54.315 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.257+0000 7ff7017fa700 10 client.204255.objecter ms_dispatch 0x7ff714001550 osd_map(218..218 src has 1..218) v4 2021-05-25T12:06:54.315 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.257+0000 7ff7017fa700 3 client.204255.objecter handle_osd_map got epochs [218,218] > 0 2021-05-25T12:06:54.316 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.257+0000 7ff7017fa700 3 client.204255.objecter handle_osd_map decoding full epoch 218 2021-05-25T12:06:54.316 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.257+0000 7ff7017fa700 20 client.204255.objecter dump_active .. 0 homeless 2021-05-25T12:06:54.316 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.257+0000 7ff7017fa700 1 -- 172.21.15.102:0/905942828 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 836505002 0 0) 0x7ff6ec0065a0 con 0x7ff6ec002840 2021-05-25T12:06:54.316 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.257+0000 7ff7017fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:54.317 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.257+0000 7ff7017fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:54.317 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.257+0000 7ff7017fa700 10 monclient(hunting): my global_id is 184592 2021-05-25T12:06:54.317 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.257+0000 7ff7017fa700 1 -- 172.21.15.102:0/905942828 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff714115270 con 0x7ff6ec002840 2021-05-25T12:06:54.317 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.257+0000 7ff718e9c700 10 monclient: get_auth_request con 0x7ff6f4033230 auth_method 0 2021-05-25T12:06:54.317 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.257+0000 7ff7017fa700 1 -- 172.21.15.102:0/905942828 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1373183579 0 0) 0x7ff714115270 con 0x7ff6ec002840 2021-05-25T12:06:54.318 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.257+0000 7ff7017fa700 1 -- 172.21.15.102:0/905942828 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff6ec0065a0 con 0x7ff6ec002840 2021-05-25T12:06:54.318 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.257+0000 7ff7017fa700 1 -- 172.21.15.102:0/905942828 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7ff7040048c0 con 0x7ff6ec002840 2021-05-25T12:06:54.318 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.257+0000 7ff7017fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:54.318 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.257+0000 7ff7017fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:06:54.319 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.257+0000 7ff7017fa700 10 monclient: dump: 2021-05-25T12:06:54.319 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:54.319 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:54.319 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:54.319 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:54.320 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:54.320 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:54.320 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:54.320 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:54.320 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:54.321 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:54.321 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.257+0000 7ff7017fa700 10 client.204255.objecter ms_handle_connect 0x7ff6f4033230 2021-05-25T12:06:54.321 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.257+0000 7ff7017fa700 1 -- 172.21.15.102:0/905942828 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 371588341 0 0) 0x7ff6ec0065a0 con 0x7ff6ec002840 2021-05-25T12:06:54.321 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.257+0000 7ff7017fa700 1 -- 172.21.15.102:0/905942828 --> v1:172.21.15.102:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff6ec009e40 con 0x7ff6ec002840 2021-05-25T12:06:54.322 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.257+0000 7ff7017fa700 1 -- 172.21.15.102:0/905942828 <== mon.2 v1:172.21.15.102:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7ff704006bb0 con 0x7ff6ec002840 2021-05-25T12:06:54.322 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.257+0000 7ff7017fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:06:54.322 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.257+0000 7ff7017fa700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:06:54.322 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.257+0000 7ff7017fa700 1 -- 172.21.15.102:0/905942828 >> v1:172.21.15.102:6790/0 conn(0x7ff6ec002840 legacy=0x7ff6ec004ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:54.322 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.258+0000 7ff7017fa700 1 -- 172.21.15.102:0/905942828 >> v1:172.21.15.102:6790/0 conn(0x7ff6ec002840 legacy=0x7ff6ec004ce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:06:54.323 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.264+0000 7ff71b901700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:06:54.323 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.264+0000 7ff71b901700 1 -- 172.21.15.102:0/905942828 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff6ec007540 con 0x7ff6ec0068e0 2021-05-25T12:06:54.323 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.264+0000 7ff7017fa700 10 client.204255.objecter ms_handle_connect 0x7ff6ec0068e0 2021-05-25T12:06:54.323 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.264+0000 7ff7017fa700 10 client.204255.objecter resend_mon_ops 2021-05-25T12:06:54.324 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.264+0000 7ff7017fa700 1 -- 172.21.15.102:0/905942828 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2333394867 0 0) 0x7ff6ec007540 con 0x7ff6ec0068e0 2021-05-25T12:06:54.324 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.264+0000 7ff7017fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:54.324 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.264+0000 7ff7017fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:54.324 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.264+0000 7ff7017fa700 10 monclient(hunting): my global_id is 184595 2021-05-25T12:06:54.324 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.264+0000 7ff7017fa700 1 -- 172.21.15.102:0/905942828 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6ec0065a0 con 0x7ff6ec0068e0 2021-05-25T12:06:54.325 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.264+0000 7ff7017fa700 1 -- 172.21.15.102:0/905942828 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4233101523 0 0) 0x7ff6ec0065a0 con 0x7ff6ec0068e0 2021-05-25T12:06:54.325 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.265+0000 7ff7017fa700 1 -- 172.21.15.102:0/905942828 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff6ec007540 con 0x7ff6ec0068e0 2021-05-25T12:06:54.325 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.265+0000 7ff7017fa700 1 -- 172.21.15.102:0/905942828 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7ff7040050f0 con 0x7ff6ec0068e0 2021-05-25T12:06:54.325 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.265+0000 7ff7017fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:54.325 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.265+0000 7ff7017fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:06:54.326 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.265+0000 7ff7017fa700 10 monclient: dump: 2021-05-25T12:06:54.326 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:54.326 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:54.326 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:54.327 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:54.327 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:54.327 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:54.327 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:54.327 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:54.328 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:54.328 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:54.328 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.265+0000 7ff7017fa700 1 -- 172.21.15.102:0/905942828 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 583692567 0 0) 0x7ff6ec007540 con 0x7ff6ec0068e0 2021-05-25T12:06:54.328 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.265+0000 7ff7017fa700 1 -- 172.21.15.102:0/905942828 --> v1:172.21.15.102:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ff6ec008770 con 0x7ff6ec0068e0 2021-05-25T12:06:54.329 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.265+0000 7ff7017fa700 1 -- 172.21.15.102:0/905942828 <== mon.2 v1:172.21.15.102:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4112 (unknown 0 0 1248657511) 0x7ff704006510 con 0x7ff6ec0068e0 2021-05-25T12:06:54.329 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.265+0000 7ff7017fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:06:54.329 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.265+0000 7ff7017fa700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:06:54.329 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.265+0000 7ff7017fa700 1 -- 172.21.15.102:0/905942828 >> v1:172.21.15.102:6790/0 conn(0x7ff6ec0068e0 legacy=0x7ff6ec006d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:54.329 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.265+0000 7ff7017fa700 1 -- 172.21.15.102:0/905942828 >> v1:172.21.15.102:6790/0 conn(0x7ff6ec0068e0 legacy=0x7ff6ec006d70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:06:54.330 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:06:54.330 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:06:54.330 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:06:54.331 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "leader", 2021-05-25T12:06:54.331 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 58, 2021-05-25T12:06:54.331 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:06:54.332 INFO:teuthology.orchestra.run.smithi102.stdout: 0, 2021-05-25T12:06:54.332 INFO:teuthology.orchestra.run.smithi102.stdout: 1, 2021-05-25T12:06:54.332 INFO:teuthology.orchestra.run.smithi102.stdout: 2 2021-05-25T12:06:54.332 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:06:54.333 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 0, 2021-05-25T12:06:54.333 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:06:54.333 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:06:54.333 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:06:54.333 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:06:54.334 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:06:54.334 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:06:54.334 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:06:54.334 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:06:54.335 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:06:54.335 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:06:54.335 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:06:54.335 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:06:54.335 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:06:54.336 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:06:54.336 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:06:54.336 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:06:54.336 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:06:54.336 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:06:54.337 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:06:54.337 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:06:54.337 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:06:54.338 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:06:54.338 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:06:54.338 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:06:54.338 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:54.338 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:54.339 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:06:54.339 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:06:54.339 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:06:54.339 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:06:54.339 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:06:54.340 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:06:54.340 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:06:54.340 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:06:54.340 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:06:54.341 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:06:54.341 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:06:54.341 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:06:54.341 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:06:54.341 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:06:54.342 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:06:54.342 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:06:54.342 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:06:54.342 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:06:54.342 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:06:54.343 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:06:54.343 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:06:54.343 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:06:54.343 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:06:54.344 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:06:54.344 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:06:54.344 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:06:54.344 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:54.344 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:06:54.345 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:54.345 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:06:54.345 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:06:54.345 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:06:54.345 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:06:54.346 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:54.346 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:06:54.346 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:06:54.346 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:06:54.346 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:54.347 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:54.347 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:54.347 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:06:54.347 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:06:54.348 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:06:54.348 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:06:54.348 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:06:54.348 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:54.348 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:54.349 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:06:54.349 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:06:54.349 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:06:54.349 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:06:54.349 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:54.350 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:06:54.350 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:06:54.350 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:06:54.350 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:54.351 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:54.351 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:54.351 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:06:54.351 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:06:54.351 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:06:54.352 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:06:54.352 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:06:54.352 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:54.352 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:54.353 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:06:54.353 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:06:54.353 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:06:54.353 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:06:54.353 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:54.354 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:06:54.354 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:06:54.354 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:06:54.354 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:54.355 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:54.355 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:54.355 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:06:54.355 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:06:54.355 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:06:54.356 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:06:54.356 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:06:54.356 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:54.356 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:54.356 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:54.357 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:06:54.357 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:06:54.357 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:54.357 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:06:54.357 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:06:54.358 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:06:54.358 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:54.358 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:06:54.358 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": [ 2021-05-25T12:06:54.359 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:54.359 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:06:54.359 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:06:54.359 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 4 2021-05-25T12:06:54.359 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:54.360 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:06:54.360 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:06:54.360 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:54.360 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:06:54.361 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:06:54.361 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 5 2021-05-25T12:06:54.361 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:54.361 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:06:54.361 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": [ 2021-05-25T12:06:54.362 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:54.362 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:06:54.362 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:06:54.362 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:06:54.363 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:54.363 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:54.363 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:54.363 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:06:54.363 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:06:54.366 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.265+0000 7ff71b901700 10 librados: watch_flush enter 2021-05-25T12:06:54.366 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.265+0000 7ff71b901700 10 librados: watch_flush exit 2021-05-25T12:06:54.366 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.265+0000 7ff71b901700 20 client.204255.objecter shutdown clearing up homeless session... 2021-05-25T12:06:54.367 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.265+0000 7ff71b901700 10 client.204255.objecter successfully canceled tick 2021-05-25T12:06:54.367 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.265+0000 7ff71b901700 1 -- 172.21.15.102:0/905942828 >> v1:172.21.15.181:6812/29582 conn(0x7ff6f4033230 legacy=0x7ff6f40356f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:54.367 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.265+0000 7ff71b901700 10 monclient: shutdown 2021-05-25T12:06:54.367 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.265+0000 7ff71b901700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:06:54.368 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.265+0000 7ff71b901700 1 -- 172.21.15.102:0/905942828 >> v1:172.21.15.102:6789/0 conn(0x7ff714128110 legacy=0x7ff71413f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:54.368 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.265+0000 7ff719e9e700 1 -- 172.21.15.102:0/905942828 reap_dead start 2021-05-25T12:06:54.368 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.265+0000 7ff719e9e700 1 -- 172.21.15.102:0/905942828 reap_dead start 2021-05-25T12:06:54.368 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.265+0000 7ff71b901700 1 -- 172.21.15.102:0/905942828 shutdown_connections 2021-05-25T12:06:54.369 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.265+0000 7ff71b901700 1 -- 172.21.15.102:0/905942828 >> 172.21.15.102:0/905942828 conn(0x7ff714062810 msgr2=0x7ff714126da0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:06:54.369 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.265+0000 7ff71b901700 1 -- 172.21.15.102:0/905942828 shutdown_connections 2021-05-25T12:06:54.369 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.265+0000 7ff71b901700 1 -- 172.21.15.102:0/905942828 wait complete. 2021-05-25T12:06:54.369 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.265+0000 7ff71b901700 1 librados: shutdown 2021-05-25T12:06:54.370 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2021-05-25T12:06:54.480 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.478+0000 7f5f6b45c700 10 monclient: get_monmap_and_config 2021-05-25T12:06:54.480 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.478+0000 7f5f6b45c700 10 monclient: build_initial_monmap 2021-05-25T12:06:54.480 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.478+0000 7f5f6b45c700 10 monclient: monmap: 2021-05-25T12:06:54.481 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:06:54.481 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:54.481 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:06:54.479147+0000 2021-05-25T12:06:54.481 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:06:54.479147+0000 2021-05-25T12:06:54.482 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:06:54.482 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:54.482 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:06:54.482 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:06:54.483 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:06:54.483 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:54.483 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.478+0000 7f5f6b45c700 1 Processor -- start 2021-05-25T12:06:54.483 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.478+0000 7f5f6b45c700 1 -- start start 2021-05-25T12:06:54.484 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.478+0000 7f5f6b45c700 10 monclient: init 2021-05-25T12:06:54.495 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.479+0000 7f5f6b45c700 10 monclient: _reopen_session rank -1 2021-05-25T12:06:54.496 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.479+0000 7f5f6b45c700 10 monclient: _add_conns ranks=[1,0,2] 2021-05-25T12:06:54.496 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.479+0000 7f5f6b45c700 10 monclient(hunting): picked mon.noname-c con 0x7f5f64090990 addr v1:172.21.15.181:6789/0 2021-05-25T12:06:54.496 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.479+0000 7f5f6b45c700 10 monclient(hunting): picked mon.noname-a con 0x7f5f64090d70 addr v1:172.21.15.102:6789/0 2021-05-25T12:06:54.496 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.479+0000 7f5f6b45c700 10 monclient(hunting): picked mon.noname-b con 0x7f5f6412e330 addr v1:172.21.15.102:6790/0 2021-05-25T12:06:54.497 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.479+0000 7f5f6b45c700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f64096fa0 con 0x7f5f64090d70 2021-05-25T12:06:54.497 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.479+0000 7f5f6b45c700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f640926a0 con 0x7f5f64090990 2021-05-25T12:06:54.497 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.479+0000 7f5f6b45c700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f64092940 con 0x7f5f6412e330 2021-05-25T12:06:54.497 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.479+0000 7f5f6b45c700 10 monclient(hunting): _renew_subs 2021-05-25T12:06:54.497 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.479+0000 7f5f6b45c700 10 monclient(hunting): authenticate will time out at 1215.076490s 2021-05-25T12:06:54.498 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.479+0000 7f5f699f9700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f5f6412e330 0x7f5f64130720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:56622/0 (socket says 172.21.15.102:56622) 2021-05-25T12:06:54.498 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.479+0000 7f5f689f7700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f5f64090d70 0x7f5f64091200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33510/0 (socket says 172.21.15.102:33510) 2021-05-25T12:06:54.498 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.479+0000 7f5f699f9700 1 -- 172.21.15.102:0/3507548423 learned_addr learned my addr 172.21.15.102:0/3507548423 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:06:54.498 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.479+0000 7f5f53fff700 1 -- 172.21.15.102:0/3507548423 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2635261134 0 0) 0x7f5f64092940 con 0x7f5f6412e330 2021-05-25T12:06:54.499 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.479+0000 7f5f53fff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:54.499 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.479+0000 7f5f53fff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:54.499 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.479+0000 7f5f53fff700 10 monclient(hunting): my global_id is 184598 2021-05-25T12:06:54.499 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.479+0000 7f5f53fff700 1 -- 172.21.15.102:0/3507548423 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f4c003270 con 0x7f5f6412e330 2021-05-25T12:06:54.499 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.479+0000 7f5f53fff700 1 -- 172.21.15.102:0/3507548423 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 865560465 0 0) 0x7f5f64096fa0 con 0x7f5f64090d70 2021-05-25T12:06:54.500 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.479+0000 7f5f53fff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:54.500 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.479+0000 7f5f53fff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:54.500 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.479+0000 7f5f53fff700 10 monclient(hunting): my global_id is 204258 2021-05-25T12:06:54.500 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.479+0000 7f5f53fff700 1 -- 172.21.15.102:0/3507548423 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f64092940 con 0x7f5f64090d70 2021-05-25T12:06:54.500 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.480+0000 7f5f53fff700 1 -- 172.21.15.102:0/3507548423 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3348106934 0 0) 0x7f5f64092940 con 0x7f5f64090d70 2021-05-25T12:06:54.501 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.480+0000 7f5f53fff700 1 -- 172.21.15.102:0/3507548423 >> v1:172.21.15.102:6790/0 conn(0x7f5f6412e330 legacy=0x7f5f64130720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:54.501 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.480+0000 7f5f53fff700 1 -- 172.21.15.102:0/3507548423 >> v1:172.21.15.181:6789/0 conn(0x7f5f64090990 legacy=0x7f5f64093240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:54.501 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.480+0000 7f5f53fff700 10 monclient: _finish_hunting 0 2021-05-25T12:06:54.501 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.480+0000 7f5f53fff700 1 monclient: found mon.noname-a 2021-05-25T12:06:54.502 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.480+0000 7f5f53fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:06:54.502 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.480+0000 7f5f53fff700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T12:06:54.502 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.480+0000 7f5f53fff700 1 -- 172.21.15.102:0/3507548423 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f64092be0 con 0x7f5f64090d70 2021-05-25T12:06:54.502 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.480+0000 7f5f53fff700 10 monclient: _finish_auth 0 2021-05-25T12:06:54.502 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.480+0000 7f5f53fff700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:06:54.503 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.480+0000 7f5f6b45c700 5 monclient: authenticate success, global_id 204258 2021-05-25T12:06:54.503 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.480+0000 7f5f53fff700 1 -- 172.21.15.102:0/3507548423 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f5f54001590 con 0x7f5f64090d70 2021-05-25T12:06:54.503 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.480+0000 7f5f53fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:54.503 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.480+0000 7f5f53fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:06:54.504 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.480+0000 7f5f53fff700 10 monclient: dump: 2021-05-25T12:06:54.504 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:54.504 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:54.504 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:54.504 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:54.505 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:54.505 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:54.505 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:54.505 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:54.505 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:54.506 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:54.506 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.480+0000 7f5f6b45c700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:06:54.506 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.480+0000 7f5f53fff700 1 -- 172.21.15.102:0/3507548423 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5f54001740 con 0x7f5f64090d70 2021-05-25T12:06:54.506 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.480+0000 7f5f53fff700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:06:54.506 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.480+0000 7f5f53fff700 1 -- 172.21.15.102:0/3507548423 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f5f54001ad0 con 0x7f5f64090d70 2021-05-25T12:06:54.507 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.480+0000 7f5f6b45c700 10 monclient: get_monmap_and_config success 2021-05-25T12:06:54.507 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.480+0000 7f5f6b45c700 4 set_mon_vals no callback set 2021-05-25T12:06:54.507 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.480+0000 7f5f53fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:54.507 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.480+0000 7f5f53fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:06:54.507 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.480+0000 7f5f53fff700 10 monclient: dump: 2021-05-25T12:06:54.508 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:54.508 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:54.508 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:54.508 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:54.508 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:54.509 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:54.509 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:54.509 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:54.509 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:54.509 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:54.509 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.480+0000 7f5f6b45c700 10 monclient: shutdown 2021-05-25T12:06:54.510 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.480+0000 7f5f6b45c700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:06:54.510 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.480+0000 7f5f6b45c700 1 -- 172.21.15.102:0/3507548423 >> v1:172.21.15.102:6789/0 conn(0x7f5f64090d70 legacy=0x7f5f64091200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:54.510 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.480+0000 7f5f6b45c700 1 -- 172.21.15.102:0/3507548423 shutdown_connections 2021-05-25T12:06:54.510 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.480+0000 7f5f6b45c700 1 -- 172.21.15.102:0/3507548423 >> 172.21.15.102:0/3507548423 conn(0x7f5f6409b370 msgr2=0x7f5f64097430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:06:54.511 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.480+0000 7f5f6b45c700 1 -- 172.21.15.102:0/3507548423 shutdown_connections 2021-05-25T12:06:54.511 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.480+0000 7f5f6b45c700 1 -- 172.21.15.102:0/3507548423 wait complete. 2021-05-25T12:06:54.511 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.481+0000 7f5f6b45c700 10 monclient: build_initial_monmap 2021-05-25T12:06:54.511 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.481+0000 7f5f6b45c700 10 monclient: monmap: 2021-05-25T12:06:54.511 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:06:54.512 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:06:54.512 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:06:54.512 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:06:54.512 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:06:54.512 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:06:54.512 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:06:54.513 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:06:54.513 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:06:54.513 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:54.513 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.481+0000 7f5f6b45c700 1 librados: starting msgr at 2021-05-25T12:06:54.513 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.481+0000 7f5f6b45c700 1 librados: starting objecter 2021-05-25T12:06:54.514 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.481+0000 7f5f6b45c700 1 Processor -- start 2021-05-25T12:06:54.514 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.481+0000 7f5f6b45c700 1 -- start start 2021-05-25T12:06:54.514 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.481+0000 7f5f6b45c700 1 librados: setting wanted keys 2021-05-25T12:06:54.514 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.481+0000 7f5f6b45c700 1 librados: calling monclient init 2021-05-25T12:06:54.514 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.481+0000 7f5f6b45c700 10 monclient: init 2021-05-25T12:06:54.515 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.482+0000 7f5f6b45c700 10 monclient: _reopen_session rank -1 2021-05-25T12:06:54.515 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.482+0000 7f5f6b45c700 10 monclient: _add_conns ranks=[0,1,2] 2021-05-25T12:06:54.515 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.482+0000 7f5f6b45c700 10 monclient(hunting): picked mon.noname-a con 0x7f5f64090990 addr v1:172.21.15.102:6789/0 2021-05-25T12:06:54.515 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.482+0000 7f5f6b45c700 10 monclient(hunting): picked mon.noname-b con 0x7f5f64090d70 addr v1:172.21.15.181:6789/0 2021-05-25T12:06:54.516 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.482+0000 7f5f6b45c700 10 monclient(hunting): picked mon.noname-c con 0x7f5f6412e330 addr v1:172.21.15.102:6790/0 2021-05-25T12:06:54.516 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.482+0000 7f5f6b45c700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f64091fe0 con 0x7f5f64090990 2021-05-25T12:06:54.516 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.482+0000 7f5f6b45c700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f64115060 con 0x7f5f64090d70 2021-05-25T12:06:54.516 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.482+0000 7f5f6b45c700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f6412a070 con 0x7f5f6412e330 2021-05-25T12:06:54.516 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.482+0000 7f5f6b45c700 10 monclient(hunting): _renew_subs 2021-05-25T12:06:54.517 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.482+0000 7f5f6b45c700 10 monclient(hunting): authenticate will time out at 1215.079280s 2021-05-25T12:06:54.517 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.482+0000 7f5f691f8700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f5f64090990 0x7f5f641290d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33514/0 (socket says 172.21.15.102:33514) 2021-05-25T12:06:54.517 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.482+0000 7f5f699f9700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f5f6412e330 0x7f5f64141250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:56628/0 (socket says 172.21.15.102:56628) 2021-05-25T12:06:54.517 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.482+0000 7f5f691f8700 1 -- 172.21.15.102:0/1593355057 learned_addr learned my addr 172.21.15.102:0/1593355057 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:06:54.517 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.482+0000 7f5f517fa700 10 client.?.objecter ms_handle_connect 0x7f5f6412e330 2021-05-25T12:06:54.518 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.482+0000 7f5f517fa700 10 client.?.objecter resend_mon_ops 2021-05-25T12:06:54.518 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.482+0000 7f5f517fa700 10 client.?.objecter ms_handle_connect 0x7f5f64090990 2021-05-25T12:06:54.518 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.482+0000 7f5f517fa700 10 client.?.objecter resend_mon_ops 2021-05-25T12:06:54.518 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.482+0000 7f5f517fa700 1 -- 172.21.15.102:0/1593355057 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1655080935 0 0) 0x7f5f6412a070 con 0x7f5f6412e330 2021-05-25T12:06:54.518 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.482+0000 7f5f517fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:54.519 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.482+0000 7f5f517fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:54.519 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.482+0000 7f5f517fa700 10 monclient(hunting): my global_id is 184601 2021-05-25T12:06:54.519 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.482+0000 7f5f517fa700 1 -- 172.21.15.102:0/1593355057 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f44004c10 con 0x7f5f6412e330 2021-05-25T12:06:54.519 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.482+0000 7f5f517fa700 10 client.?.objecter ms_handle_connect 0x7f5f64090d70 2021-05-25T12:06:54.519 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.482+0000 7f5f517fa700 10 client.?.objecter resend_mon_ops 2021-05-25T12:06:54.520 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.482+0000 7f5f517fa700 1 -- 172.21.15.102:0/1593355057 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 902986952 0 0) 0x7f5f64091fe0 con 0x7f5f64090990 2021-05-25T12:06:54.520 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.482+0000 7f5f517fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:54.520 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.482+0000 7f5f517fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:54.520 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.482+0000 7f5f517fa700 10 monclient(hunting): my global_id is 204261 2021-05-25T12:06:54.521 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.482+0000 7f5f517fa700 1 -- 172.21.15.102:0/1593355057 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f6412a070 con 0x7f5f64090990 2021-05-25T12:06:54.521 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.482+0000 7f5f517fa700 1 -- 172.21.15.102:0/1593355057 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4191871420 0 0) 0x7f5f64115060 con 0x7f5f64090d70 2021-05-25T12:06:54.521 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.482+0000 7f5f517fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:54.521 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.482+0000 7f5f517fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:54.521 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.482+0000 7f5f517fa700 10 monclient(hunting): my global_id is 224122 2021-05-25T12:06:54.522 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.482+0000 7f5f517fa700 1 -- 172.21.15.102:0/1593355057 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f64091fe0 con 0x7f5f64090d70 2021-05-25T12:06:54.522 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.482+0000 7f5f517fa700 1 -- 172.21.15.102:0/1593355057 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 992586896 0 0) 0x7f5f44004c10 con 0x7f5f6412e330 2021-05-25T12:06:54.522 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.482+0000 7f5f517fa700 1 -- 172.21.15.102:0/1593355057 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5f64115060 con 0x7f5f6412e330 2021-05-25T12:06:54.522 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.482+0000 7f5f517fa700 1 -- 172.21.15.102:0/1593355057 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f5f60001d70 con 0x7f5f6412e330 2021-05-25T12:06:54.523 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.482+0000 7f5f517fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:54.523 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.482+0000 7f5f517fa700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:06:54.523 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.482+0000 7f5f517fa700 10 monclient(hunting): dump: 2021-05-25T12:06:54.523 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:54.523 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:54.524 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:54.524 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:54.524 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:54.524 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:54.524 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:54.525 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:54.525 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:54.525 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:54.525 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.482+0000 7f5f517fa700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:06:54.525 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.482+0000 7f5f517fa700 1 -- 172.21.15.102:0/1593355057 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1618147538 0 0) 0x7f5f6412a070 con 0x7f5f64090990 2021-05-25T12:06:54.526 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.482+0000 7f5f517fa700 1 -- 172.21.15.102:0/1593355057 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5f44004c10 con 0x7f5f64090990 2021-05-25T12:06:54.526 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.482+0000 7f5f517fa700 1 -- 172.21.15.102:0/1593355057 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f5f58001950 con 0x7f5f64090990 2021-05-25T12:06:54.526 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.482+0000 7f5f517fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:54.526 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.482+0000 7f5f517fa700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2021-05-25T12:06:54.527 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.482+0000 7f5f517fa700 10 monclient(hunting): dump: 2021-05-25T12:06:54.527 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:54.527 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:54.527 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:54.528 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:54.528 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:54.528 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:54.528 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:54.529 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:54.529 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:54.529 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:54.529 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.482+0000 7f5f517fa700 1 -- 172.21.15.102:0/1593355057 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 891136084 0 0) 0x7f5f64115060 con 0x7f5f6412e330 2021-05-25T12:06:54.529 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.482+0000 7f5f517fa700 1 -- 172.21.15.102:0/1593355057 >> v1:172.21.15.181:6789/0 conn(0x7f5f64090d70 legacy=0x7f5f6413db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:54.530 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.482+0000 7f5f517fa700 1 -- 172.21.15.102:0/1593355057 >> v1:172.21.15.102:6789/0 conn(0x7f5f64090990 legacy=0x7f5f641290d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:54.530 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.482+0000 7f5f517fa700 10 monclient: _finish_hunting 0 2021-05-25T12:06:54.530 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.482+0000 7f5f517fa700 1 monclient: found mon.c 2021-05-25T12:06:54.530 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.482+0000 7f5f517fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:06:54.531 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.482+0000 7f5f517fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:06:54.531 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.482+0000 7f5f517fa700 1 -- 172.21.15.102:0/1593355057 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f640685e0 con 0x7f5f6412e330 2021-05-25T12:06:54.531 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.483+0000 7f5f517fa700 10 monclient: _finish_auth 0 2021-05-25T12:06:54.531 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.483+0000 7f5f517fa700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:06:54.531 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.483+0000 7f5f6b45c700 5 monclient: authenticate success, global_id 184601 2021-05-25T12:06:54.532 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.483+0000 7f5f6b45c700 10 monclient: _renew_subs 2021-05-25T12:06:54.532 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.483+0000 7f5f6b45c700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:06:54.532 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.483+0000 7f5f6b45c700 1 -- 172.21.15.102:0/1593355057 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f64114470 con 0x7f5f6412e330 2021-05-25T12:06:54.532 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.483+0000 7f5f6b45c700 10 client.184601.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:06:54.533 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.483+0000 7f5f6b45c700 10 monclient: _renew_subs 2021-05-25T12:06:54.533 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.483+0000 7f5f6b45c700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:06:54.533 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.483+0000 7f5f6b45c700 1 -- 172.21.15.102:0/1593355057 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f6412ec20 con 0x7f5f6412e330 2021-05-25T12:06:54.533 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.483+0000 7f5f6b45c700 1 librados: init done 2021-05-25T12:06:54.533 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.483+0000 7f5f517fa700 1 -- 172.21.15.102:0/1593355057 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5f600048b0 con 0x7f5f6412e330 2021-05-25T12:06:54.533 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.483+0000 7f5f517fa700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:06:54.534 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.483+0000 7f5f517fa700 1 -- 172.21.15.102:0/1593355057 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f5f60004bc0 con 0x7f5f6412e330 2021-05-25T12:06:54.534 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.483+0000 7f5f517fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:54.534 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.483+0000 7f5f537fe700 4 set_mon_vals no callback set 2021-05-25T12:06:54.534 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.483+0000 7f5f517fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:06:54.534 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.483+0000 7f5f517fa700 10 monclient: dump: 2021-05-25T12:06:54.535 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:54.535 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:54.535 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:54.535 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:54.535 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:54.536 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:54.536 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:54.536 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:54.536 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:54.536 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:54.537 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.483+0000 7f5f6b45c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:06:54.537 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.483+0000 7f5f6b45c700 1 -- 172.21.15.102:0/1593355057 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f5f3c0065a0 con 0x7f5f3c002840 2021-05-25T12:06:54.537 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.483+0000 7f5f517fa700 1 -- 172.21.15.102:0/1593355057 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f5f6000dee0 con 0x7f5f6412e330 2021-05-25T12:06:54.537 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.484+0000 7f5f517fa700 1 -- 172.21.15.102:0/1593355057 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(218..218 src has 1..218) v4 ==== 5811+0+0 (unknown 452711690 0 0) 0x7f5f6003d910 con 0x7f5f6412e330 2021-05-25T12:06:54.537 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.484+0000 7f5f517fa700 10 client.184601.objecter ms_dispatch 0x7f5f64001550 osd_map(218..218 src has 1..218) v4 2021-05-25T12:06:54.538 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.484+0000 7f5f517fa700 3 client.184601.objecter handle_osd_map got epochs [218,218] > 0 2021-05-25T12:06:54.538 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.484+0000 7f5f517fa700 3 client.184601.objecter handle_osd_map decoding full epoch 218 2021-05-25T12:06:54.538 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.484+0000 7f5f517fa700 20 client.184601.objecter dump_active .. 0 homeless 2021-05-25T12:06:54.538 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.484+0000 7f5f517fa700 10 client.184601.objecter ms_handle_connect 0x7f5f3c002840 2021-05-25T12:06:54.539 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.484+0000 7f5f517fa700 10 client.184601.objecter resend_mon_ops 2021-05-25T12:06:54.539 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.484+0000 7f5f517fa700 1 -- 172.21.15.102:0/1593355057 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 25585054 0 0) 0x7f5f3c0065a0 con 0x7f5f3c002840 2021-05-25T12:06:54.539 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.484+0000 7f5f517fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:54.539 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.484+0000 7f5f517fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:54.539 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.484+0000 7f5f517fa700 10 monclient(hunting): my global_id is 224125 2021-05-25T12:06:54.540 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.484+0000 7f5f517fa700 1 -- 172.21.15.102:0/1593355057 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f64115060 con 0x7f5f3c002840 2021-05-25T12:06:54.540 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.484+0000 7f5f689f7700 10 monclient: get_auth_request con 0x7f5f44033460 auth_method 0 2021-05-25T12:06:54.540 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.484+0000 7f5f517fa700 1 -- 172.21.15.102:0/1593355057 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1967013460 0 0) 0x7f5f64115060 con 0x7f5f3c002840 2021-05-25T12:06:54.540 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.484+0000 7f5f517fa700 1 -- 172.21.15.102:0/1593355057 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5f3c0065a0 con 0x7f5f3c002840 2021-05-25T12:06:54.540 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.484+0000 7f5f517fa700 10 client.184601.objecter ms_handle_connect 0x7f5f44033460 2021-05-25T12:06:54.541 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.484+0000 7f5f517fa700 1 -- 172.21.15.102:0/1593355057 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f5f580020a0 con 0x7f5f3c002840 2021-05-25T12:06:54.541 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.484+0000 7f5f517fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:54.541 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.484+0000 7f5f517fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T12:06:54.541 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.484+0000 7f5f517fa700 10 monclient: dump: 2021-05-25T12:06:54.541 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:54.542 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:54.542 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:54.542 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:54.542 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:54.542 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:54.543 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:54.543 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:54.543 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:54.543 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:54.543 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.484+0000 7f5f517fa700 1 -- 172.21.15.102:0/1593355057 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 300502802 0 0) 0x7f5f3c0065a0 con 0x7f5f3c002840 2021-05-25T12:06:54.544 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.484+0000 7f5f517fa700 1 -- 172.21.15.102:0/1593355057 --> v1:172.21.15.181:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5f3c009e40 con 0x7f5f3c002840 2021-05-25T12:06:54.544 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.485+0000 7f5f517fa700 1 -- 172.21.15.102:0/1593355057 <== mon.1 v1:172.21.15.181:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7f5f580024f0 con 0x7f5f3c002840 2021-05-25T12:06:54.544 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.485+0000 7f5f517fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:06:54.544 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.485+0000 7f5f517fa700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:06:54.545 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.485+0000 7f5f517fa700 1 -- 172.21.15.102:0/1593355057 >> v1:172.21.15.181:6789/0 conn(0x7f5f3c002840 legacy=0x7f5f3c004ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:54.545 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.485+0000 7f5f517fa700 1 -- 172.21.15.102:0/1593355057 >> v1:172.21.15.181:6789/0 conn(0x7f5f3c002840 legacy=0x7f5f3c004ce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:06:54.545 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.491+0000 7f5f6b45c700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:06:54.545 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.491+0000 7f5f6b45c700 1 -- 172.21.15.102:0/1593355057 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f5f3c009ad0 con 0x7f5f3c008d90 2021-05-25T12:06:54.545 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.492+0000 7f5f517fa700 10 client.184601.objecter ms_handle_connect 0x7f5f3c008d90 2021-05-25T12:06:54.546 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.492+0000 7f5f517fa700 10 client.184601.objecter resend_mon_ops 2021-05-25T12:06:54.546 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.492+0000 7f5f517fa700 1 -- 172.21.15.102:0/1593355057 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 552337013 0 0) 0x7f5f3c009ad0 con 0x7f5f3c008d90 2021-05-25T12:06:54.546 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.492+0000 7f5f517fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:54.546 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.492+0000 7f5f517fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:54.546 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.492+0000 7f5f517fa700 10 monclient(hunting): my global_id is 224128 2021-05-25T12:06:54.547 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.492+0000 7f5f517fa700 1 -- 172.21.15.102:0/1593355057 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f3c0065a0 con 0x7f5f3c008d90 2021-05-25T12:06:54.547 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.492+0000 7f5f517fa700 1 -- 172.21.15.102:0/1593355057 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3897173999 0 0) 0x7f5f3c0065a0 con 0x7f5f3c008d90 2021-05-25T12:06:54.547 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.492+0000 7f5f517fa700 1 -- 172.21.15.102:0/1593355057 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5f3c009ad0 con 0x7f5f3c008d90 2021-05-25T12:06:54.547 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.492+0000 7f5f517fa700 1 -- 172.21.15.102:0/1593355057 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f5f58002b10 con 0x7f5f3c008d90 2021-05-25T12:06:54.547 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.492+0000 7f5f517fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:54.548 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.492+0000 7f5f517fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T12:06:54.548 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.492+0000 7f5f517fa700 10 monclient: dump: 2021-05-25T12:06:54.548 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:54.548 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:54.548 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:54.549 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:54.549 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:54.549 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:54.549 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:54.549 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:54.550 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:54.550 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:54.550 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.492+0000 7f5f517fa700 1 -- 172.21.15.102:0/1593355057 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2435565433 0 0) 0x7f5f3c009ad0 con 0x7f5f3c008d90 2021-05-25T12:06:54.550 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.492+0000 7f5f517fa700 1 -- 172.21.15.102:0/1593355057 --> v1:172.21.15.181:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f5f3c008740 con 0x7f5f3c008d90 2021-05-25T12:06:54.551 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.493+0000 7f5f517fa700 1 -- 172.21.15.102:0/1593355057 <== mon.1 v1:172.21.15.181:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3770 (unknown 0 0 2173602530) 0x7f5f58005510 con 0x7f5f3c008d90 2021-05-25T12:06:54.551 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.493+0000 7f5f517fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:06:54.551 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.493+0000 7f5f517fa700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:06:54.551 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.493+0000 7f5f517fa700 1 -- 172.21.15.102:0/1593355057 >> v1:172.21.15.181:6789/0 conn(0x7f5f3c008d90 legacy=0x7f5f3c009220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:54.551 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.493+0000 7f5f517fa700 1 -- 172.21.15.102:0/1593355057 >> v1:172.21.15.181:6789/0 conn(0x7f5f3c008d90 legacy=0x7f5f3c009220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:06:54.552 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:06:54.552 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:06:54.552 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:06:54.552 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "peon", 2021-05-25T12:06:54.553 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 58, 2021-05-25T12:06:54.553 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:06:54.553 INFO:teuthology.orchestra.run.smithi102.stdout: 0, 2021-05-25T12:06:54.553 INFO:teuthology.orchestra.run.smithi102.stdout: 1, 2021-05-25T12:06:54.554 INFO:teuthology.orchestra.run.smithi102.stdout: 2 2021-05-25T12:06:54.554 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:06:54.554 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 0, 2021-05-25T12:06:54.554 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:06:54.554 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:06:54.555 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:06:54.555 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:06:54.555 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:06:54.555 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:06:54.555 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:06:54.556 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:06:54.556 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:06:54.556 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:06:54.556 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:06:54.556 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:06:54.557 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:06:54.557 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:06:54.557 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:06:54.557 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:06:54.557 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:06:54.558 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:06:54.558 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:06:54.558 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:06:54.558 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:06:54.559 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:06:54.559 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:06:54.559 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:06:54.559 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:54.559 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:54.560 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:06:54.560 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:06:54.560 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:06:54.560 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:06:54.560 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:06:54.561 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:06:54.561 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:06:54.561 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:06:54.561 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:06:54.561 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:06:54.562 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:06:54.562 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:06:54.562 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:06:54.562 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:06:54.563 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:06:54.563 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:06:54.563 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:06:54.563 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:06:54.563 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:06:54.564 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:06:54.564 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:06:54.564 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:06:54.564 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:06:54.564 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:06:54.565 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:06:54.565 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:06:54.565 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:54.565 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:06:54.565 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:54.566 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:06:54.566 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:06:54.566 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:06:54.566 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:06:54.567 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:54.567 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:06:54.567 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:06:54.567 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:06:54.567 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:54.568 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:54.568 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:54.568 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:06:54.568 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:06:54.568 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:06:54.569 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:06:54.569 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:06:54.569 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:54.569 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:54.570 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:06:54.570 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:06:54.570 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:06:54.570 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:06:54.570 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:54.571 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:06:54.571 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:06:54.571 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:06:54.571 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:54.571 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:54.572 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:54.572 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:06:54.572 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:06:54.572 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:06:54.572 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:06:54.573 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:06:54.573 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:54.573 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:54.573 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:06:54.573 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:06:54.574 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:06:54.574 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:06:54.574 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:54.574 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:06:54.575 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:06:54.575 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:06:54.575 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:54.575 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:54.575 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:54.576 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:06:54.576 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:06:54.576 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:06:54.576 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:06:54.576 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:06:54.577 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:54.577 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:54.577 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:54.577 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:06:54.577 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:06:54.578 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:54.578 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:06:54.578 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:06:54.578 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:06:54.579 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:54.579 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:06:54.579 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:06:54.579 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:54.579 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:06:54.580 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:06:54.580 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 2 2021-05-25T12:06:54.580 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:54.580 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:54.580 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:54.580 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:06:54.581 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:06:54.583 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.493+0000 7f5f6b45c700 10 librados: watch_flush enter 2021-05-25T12:06:54.583 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.493+0000 7f5f6b45c700 10 librados: watch_flush exit 2021-05-25T12:06:54.583 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.493+0000 7f5f6b45c700 20 client.184601.objecter shutdown clearing up homeless session... 2021-05-25T12:06:54.584 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.493+0000 7f5f6b45c700 10 client.184601.objecter successfully canceled tick 2021-05-25T12:06:54.584 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.493+0000 7f5f6b45c700 1 -- 172.21.15.102:0/1593355057 >> v1:172.21.15.181:6812/29582 conn(0x7f5f44033460 legacy=0x7f5f44035920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:54.584 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.493+0000 7f5f6b45c700 10 monclient: shutdown 2021-05-25T12:06:54.584 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.493+0000 7f5f6b45c700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:06:54.584 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.493+0000 7f5f6b45c700 1 -- 172.21.15.102:0/1593355057 >> v1:172.21.15.102:6790/0 conn(0x7f5f6412e330 legacy=0x7f5f64141250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:54.585 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.493+0000 7f5f699f9700 1 -- 172.21.15.102:0/1593355057 reap_dead start 2021-05-25T12:06:54.585 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.493+0000 7f5f699f9700 1 -- 172.21.15.102:0/1593355057 reap_dead start 2021-05-25T12:06:54.585 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.493+0000 7f5f6b45c700 1 -- 172.21.15.102:0/1593355057 shutdown_connections 2021-05-25T12:06:54.585 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.493+0000 7f5f6b45c700 1 -- 172.21.15.102:0/1593355057 >> 172.21.15.102:0/1593355057 conn(0x7f5f6409b370 msgr2=0x7f5f64093a60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:06:54.586 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.493+0000 7f5f6b45c700 1 -- 172.21.15.102:0/1593355057 shutdown_connections 2021-05-25T12:06:54.586 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.493+0000 7f5f6b45c700 1 -- 172.21.15.102:0/1593355057 wait complete. 2021-05-25T12:06:54.586 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.493+0000 7f5f6b45c700 1 librados: shutdown 2021-05-25T12:06:54.586 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2021-05-25T12:06:54.587 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2021-05-25T12:06:54.697 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.696+0000 7efd563c6700 10 monclient: get_monmap_and_config 2021-05-25T12:06:54.697 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.696+0000 7efd563c6700 10 monclient: build_initial_monmap 2021-05-25T12:06:54.698 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.696+0000 7efd563c6700 10 monclient: monmap: 2021-05-25T12:06:54.698 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:06:54.698 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:54.698 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:06:54.697101+0000 2021-05-25T12:06:54.699 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:06:54.697101+0000 2021-05-25T12:06:54.699 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:06:54.699 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:54.699 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:06:54.699 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:06:54.700 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:06:54.700 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:54.710 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.696+0000 7efd563c6700 1 Processor -- start 2021-05-25T12:06:54.710 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.696+0000 7efd563c6700 1 -- start start 2021-05-25T12:06:54.710 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.696+0000 7efd563c6700 10 monclient: init 2021-05-25T12:06:54.710 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.697+0000 7efd563c6700 10 monclient: _reopen_session rank -1 2021-05-25T12:06:54.711 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.697+0000 7efd563c6700 10 monclient: _add_conns ranks=[0,2,1] 2021-05-25T12:06:54.711 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.697+0000 7efd563c6700 10 monclient(hunting): picked mon.noname-a con 0x7efd50090990 addr v1:172.21.15.102:6789/0 2021-05-25T12:06:54.711 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.697+0000 7efd563c6700 10 monclient(hunting): picked mon.noname-b con 0x7efd50090d70 addr v1:172.21.15.102:6790/0 2021-05-25T12:06:54.711 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.697+0000 7efd563c6700 10 monclient(hunting): picked mon.noname-c con 0x7efd5012e330 addr v1:172.21.15.181:6789/0 2021-05-25T12:06:54.711 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.697+0000 7efd563c6700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd50096fa0 con 0x7efd50090990 2021-05-25T12:06:54.712 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.697+0000 7efd563c6700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd500926a0 con 0x7efd5012e330 2021-05-25T12:06:54.712 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.697+0000 7efd563c6700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd50092940 con 0x7efd50090d70 2021-05-25T12:06:54.712 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.697+0000 7efd563c6700 10 monclient(hunting): _renew_subs 2021-05-25T12:06:54.712 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.697+0000 7efd563c6700 10 monclient(hunting): authenticate will time out at 1215.294461s 2021-05-25T12:06:54.713 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.697+0000 7efd4ffff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7efd50090990 0x7efd50093240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33526/0 (socket says 172.21.15.102:33526) 2021-05-25T12:06:54.713 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.697+0000 7efd4ffff700 1 -- 172.21.15.102:0/3009585209 learned_addr learned my addr 172.21.15.102:0/3009585209 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:06:54.713 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.697+0000 7efd4f7fe700 1 --1- 172.21.15.102:0/3009585209 >> v1:172.21.15.102:6790/0 conn(0x7efd50090d70 0x7efd50091200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:56638/0 (socket says 172.21.15.102:56638) 2021-05-25T12:06:54.713 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.697+0000 7efd4effd700 1 -- 172.21.15.102:0/3009585209 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2343749191 0 0) 0x7efd50092940 con 0x7efd50090d70 2021-05-25T12:06:54.713 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.697+0000 7efd4effd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:54.714 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.697+0000 7efd4effd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:54.714 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.697+0000 7efd4effd700 10 monclient(hunting): my global_id is 184604 2021-05-25T12:06:54.714 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.697+0000 7efd4effd700 1 -- 172.21.15.102:0/3009585209 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd38003270 con 0x7efd50090d70 2021-05-25T12:06:54.714 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.697+0000 7efd4effd700 1 -- 172.21.15.102:0/3009585209 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 668698941 0 0) 0x7efd50096fa0 con 0x7efd50090990 2021-05-25T12:06:54.715 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.697+0000 7efd4effd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:54.715 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.697+0000 7efd4effd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:54.715 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.697+0000 7efd4effd700 10 monclient(hunting): my global_id is 204264 2021-05-25T12:06:54.715 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.697+0000 7efd4effd700 1 -- 172.21.15.102:0/3009585209 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd50092940 con 0x7efd50090990 2021-05-25T12:06:54.715 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.697+0000 7efd4effd700 1 -- 172.21.15.102:0/3009585209 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1233784140 0 0) 0x7efd38003270 con 0x7efd50090d70 2021-05-25T12:06:54.716 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.698+0000 7efd4effd700 1 -- 172.21.15.102:0/3009585209 >> v1:172.21.15.181:6789/0 conn(0x7efd5012e330 legacy=0x7efd50130720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:54.716 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.698+0000 7efd4effd700 1 -- 172.21.15.102:0/3009585209 >> v1:172.21.15.102:6789/0 conn(0x7efd50090990 legacy=0x7efd50093240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:54.716 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.698+0000 7efd4effd700 10 monclient: _finish_hunting 0 2021-05-25T12:06:54.716 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.698+0000 7efd4effd700 1 monclient: found mon.noname-b 2021-05-25T12:06:54.716 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.698+0000 7efd4effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:06:54.717 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.698+0000 7efd4effd700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T12:06:54.717 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.698+0000 7efd4effd700 1 -- 172.21.15.102:0/3009585209 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd50092be0 con 0x7efd50090d70 2021-05-25T12:06:54.717 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.698+0000 7efd4effd700 10 monclient: _finish_auth 0 2021-05-25T12:06:54.717 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.698+0000 7efd4effd700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:06:54.718 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.698+0000 7efd4effd700 1 -- 172.21.15.102:0/3009585209 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7efd3c001610 con 0x7efd50090d70 2021-05-25T12:06:54.718 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.698+0000 7efd563c6700 5 monclient: authenticate success, global_id 184604 2021-05-25T12:06:54.718 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.698+0000 7efd563c6700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:06:54.718 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.698+0000 7efd4effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:54.718 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.698+0000 7efd4effd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:06:54.719 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.698+0000 7efd4effd700 10 monclient: dump: 2021-05-25T12:06:54.719 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:54.719 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:54.719 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:54.720 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:54.720 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:54.720 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:54.720 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:54.720 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:54.721 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:54.721 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:54.721 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.698+0000 7efd563c6700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:06:54.721 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.698+0000 7efd4effd700 1 -- 172.21.15.102:0/3009585209 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efd3c001840 con 0x7efd50090d70 2021-05-25T12:06:54.722 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.698+0000 7efd4effd700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:06:54.722 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.698+0000 7efd4effd700 1 -- 172.21.15.102:0/3009585209 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7efd3c001bd0 con 0x7efd50090d70 2021-05-25T12:06:54.722 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.698+0000 7efd563c6700 10 monclient: get_monmap_and_config success 2021-05-25T12:06:54.722 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.698+0000 7efd563c6700 4 set_mon_vals no callback set 2021-05-25T12:06:54.722 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.698+0000 7efd4effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:54.723 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.698+0000 7efd4effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:06:54.723 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.698+0000 7efd4effd700 10 monclient: dump: 2021-05-25T12:06:54.723 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:54.723 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:54.724 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:54.724 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:54.724 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:54.724 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:54.724 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:54.725 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:54.725 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:54.725 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:54.725 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.698+0000 7efd563c6700 10 monclient: shutdown 2021-05-25T12:06:54.725 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.698+0000 7efd563c6700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:06:54.726 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.698+0000 7efd563c6700 1 -- 172.21.15.102:0/3009585209 >> v1:172.21.15.102:6790/0 conn(0x7efd50090d70 legacy=0x7efd50091200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:54.726 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.698+0000 7efd563c6700 1 -- 172.21.15.102:0/3009585209 shutdown_connections 2021-05-25T12:06:54.726 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.698+0000 7efd563c6700 1 -- 172.21.15.102:0/3009585209 >> 172.21.15.102:0/3009585209 conn(0x7efd5009b370 msgr2=0x7efd50097430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:06:54.726 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.698+0000 7efd563c6700 1 -- 172.21.15.102:0/3009585209 shutdown_connections 2021-05-25T12:06:54.727 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.698+0000 7efd563c6700 1 -- 172.21.15.102:0/3009585209 wait complete. 2021-05-25T12:06:54.727 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.699+0000 7efd563c6700 10 monclient: build_initial_monmap 2021-05-25T12:06:54.727 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.699+0000 7efd563c6700 10 monclient: monmap: 2021-05-25T12:06:54.727 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:06:54.727 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:06:54.728 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:06:54.728 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:06:54.728 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:06:54.728 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:06:54.728 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:06:54.729 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:06:54.729 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:06:54.729 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:54.729 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.699+0000 7efd563c6700 1 librados: starting msgr at 2021-05-25T12:06:54.730 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.699+0000 7efd563c6700 1 librados: starting objecter 2021-05-25T12:06:54.730 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.699+0000 7efd563c6700 1 Processor -- start 2021-05-25T12:06:54.730 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.699+0000 7efd563c6700 1 -- start start 2021-05-25T12:06:54.730 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.699+0000 7efd563c6700 1 librados: setting wanted keys 2021-05-25T12:06:54.730 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.699+0000 7efd563c6700 1 librados: calling monclient init 2021-05-25T12:06:54.731 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.699+0000 7efd563c6700 10 monclient: init 2021-05-25T12:06:54.731 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.700+0000 7efd563c6700 10 monclient: _reopen_session rank -1 2021-05-25T12:06:54.731 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.700+0000 7efd563c6700 10 monclient: _add_conns ranks=[2,0,1] 2021-05-25T12:06:54.731 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.700+0000 7efd563c6700 10 monclient(hunting): picked mon.noname-c con 0x7efd50090990 addr v1:172.21.15.102:6790/0 2021-05-25T12:06:54.731 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.700+0000 7efd563c6700 10 monclient(hunting): picked mon.noname-a con 0x7efd50090d70 addr v1:172.21.15.102:6789/0 2021-05-25T12:06:54.732 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.700+0000 7efd563c6700 10 monclient(hunting): picked mon.noname-b con 0x7efd5012e330 addr v1:172.21.15.181:6789/0 2021-05-25T12:06:54.732 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.700+0000 7efd563c6700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd50130e30 con 0x7efd50090d70 2021-05-25T12:06:54.732 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.700+0000 7efd563c6700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd500680c0 con 0x7efd5012e330 2021-05-25T12:06:54.732 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.700+0000 7efd563c6700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd50068360 con 0x7efd50090990 2021-05-25T12:06:54.733 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.700+0000 7efd563c6700 10 monclient(hunting): _renew_subs 2021-05-25T12:06:54.733 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.700+0000 7efd563c6700 10 monclient(hunting): authenticate will time out at 1215.297198s 2021-05-25T12:06:54.733 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.700+0000 7efd4ffff700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7efd50090990 0x7efd50139040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:56642/0 (socket says 172.21.15.102:56642) 2021-05-25T12:06:54.733 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.700+0000 7efd4ffff700 1 -- 172.21.15.102:0/495240149 learned_addr learned my addr 172.21.15.102:0/495240149 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:06:54.733 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.700+0000 7efd43fff700 10 client.?.objecter ms_handle_connect 0x7efd50090990 2021-05-25T12:06:54.734 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.700+0000 7efd43fff700 10 client.?.objecter resend_mon_ops 2021-05-25T12:06:54.734 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.700+0000 7efd43fff700 10 client.?.objecter ms_handle_connect 0x7efd50090d70 2021-05-25T12:06:54.734 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.700+0000 7efd43fff700 10 client.?.objecter resend_mon_ops 2021-05-25T12:06:54.734 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.700+0000 7efd43fff700 1 -- 172.21.15.102:0/495240149 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2835258627 0 0) 0x7efd50068360 con 0x7efd50090990 2021-05-25T12:06:54.735 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.700+0000 7efd43fff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:54.735 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.700+0000 7efd43fff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:54.735 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.700+0000 7efd43fff700 10 monclient(hunting): my global_id is 184607 2021-05-25T12:06:54.735 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.700+0000 7efd43fff700 1 -- 172.21.15.102:0/495240149 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd2c004c10 con 0x7efd50090990 2021-05-25T12:06:54.735 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.700+0000 7efd43fff700 1 -- 172.21.15.102:0/495240149 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 683225665 0 0) 0x7efd50130e30 con 0x7efd50090d70 2021-05-25T12:06:54.736 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.700+0000 7efd43fff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:54.736 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.700+0000 7efd43fff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:54.736 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.700+0000 7efd43fff700 10 monclient(hunting): my global_id is 204267 2021-05-25T12:06:54.736 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.700+0000 7efd43fff700 1 -- 172.21.15.102:0/495240149 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd50068360 con 0x7efd50090d70 2021-05-25T12:06:54.737 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.700+0000 7efd43fff700 1 -- 172.21.15.102:0/495240149 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1840466744 0 0) 0x7efd2c004c10 con 0x7efd50090990 2021-05-25T12:06:54.737 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.700+0000 7efd43fff700 1 -- 172.21.15.102:0/495240149 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efd50130e30 con 0x7efd50090990 2021-05-25T12:06:54.737 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.700+0000 7efd43fff700 10 client.?.objecter ms_handle_connect 0x7efd5012e330 2021-05-25T12:06:54.737 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.700+0000 7efd43fff700 10 client.?.objecter resend_mon_ops 2021-05-25T12:06:54.737 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.700+0000 7efd43fff700 1 -- 172.21.15.102:0/495240149 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7efd340019e0 con 0x7efd50090990 2021-05-25T12:06:54.738 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.700+0000 7efd43fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:54.738 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.700+0000 7efd43fff700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:06:54.738 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.700+0000 7efd43fff700 10 monclient(hunting): dump: 2021-05-25T12:06:54.738 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:54.738 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:54.739 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:54.739 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:54.739 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:54.739 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:54.740 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:54.740 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:54.740 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:54.740 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:54.740 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.700+0000 7efd43fff700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:06:54.741 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.700+0000 7efd43fff700 1 -- 172.21.15.102:0/495240149 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 418560865 0 0) 0x7efd50068360 con 0x7efd50090d70 2021-05-25T12:06:54.741 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.700+0000 7efd43fff700 1 -- 172.21.15.102:0/495240149 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efd2c004c10 con 0x7efd50090d70 2021-05-25T12:06:54.741 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.700+0000 7efd43fff700 1 -- 172.21.15.102:0/495240149 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7efd3c001b50 con 0x7efd50090d70 2021-05-25T12:06:54.741 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.700+0000 7efd43fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:54.742 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.700+0000 7efd43fff700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2021-05-25T12:06:54.742 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.700+0000 7efd43fff700 10 monclient(hunting): dump: 2021-05-25T12:06:54.742 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:54.742 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:54.742 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:54.743 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:54.743 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:54.743 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:54.743 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:54.744 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:54.744 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:54.744 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:54.744 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.700+0000 7efd43fff700 1 -- 172.21.15.102:0/495240149 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3140240646 0 0) 0x7efd500680c0 con 0x7efd5012e330 2021-05-25T12:06:54.744 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.700+0000 7efd43fff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:54.745 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.700+0000 7efd43fff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:54.745 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.700+0000 7efd43fff700 10 monclient(hunting): my global_id is 224134 2021-05-25T12:06:54.745 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.700+0000 7efd43fff700 1 -- 172.21.15.102:0/495240149 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd50068360 con 0x7efd5012e330 2021-05-25T12:06:54.745 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.700+0000 7efd43fff700 1 -- 172.21.15.102:0/495240149 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3698299309 0 0) 0x7efd50130e30 con 0x7efd50090990 2021-05-25T12:06:54.746 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.700+0000 7efd43fff700 1 -- 172.21.15.102:0/495240149 >> v1:172.21.15.181:6789/0 conn(0x7efd5012e330 legacy=0x7efd50141400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:54.746 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.700+0000 7efd43fff700 1 -- 172.21.15.102:0/495240149 >> v1:172.21.15.102:6789/0 conn(0x7efd50090d70 legacy=0x7efd5013dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:54.746 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.700+0000 7efd43fff700 10 monclient: _finish_hunting 0 2021-05-25T12:06:54.746 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.700+0000 7efd43fff700 1 monclient: found mon.c 2021-05-25T12:06:54.746 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.700+0000 7efd43fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:06:54.747 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.700+0000 7efd43fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:06:54.747 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.700+0000 7efd43fff700 1 -- 172.21.15.102:0/495240149 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd50068600 con 0x7efd50090990 2021-05-25T12:06:54.747 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.700+0000 7efd43fff700 10 monclient: _finish_auth 0 2021-05-25T12:06:54.747 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.700+0000 7efd43fff700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:06:54.748 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.701+0000 7efd563c6700 5 monclient: authenticate success, global_id 184607 2021-05-25T12:06:54.748 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.701+0000 7efd563c6700 10 monclient: _renew_subs 2021-05-25T12:06:54.748 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.701+0000 7efd563c6700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:06:54.748 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.701+0000 7efd563c6700 1 -- 172.21.15.102:0/495240149 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd50066730 con 0x7efd50090990 2021-05-25T12:06:54.749 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.701+0000 7efd563c6700 10 client.184607.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:06:54.749 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.701+0000 7efd563c6700 10 monclient: _renew_subs 2021-05-25T12:06:54.749 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.701+0000 7efd563c6700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:06:54.749 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.701+0000 7efd563c6700 1 -- 172.21.15.102:0/495240149 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efd500673e0 con 0x7efd50090990 2021-05-25T12:06:54.749 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.701+0000 7efd563c6700 1 librados: init done 2021-05-25T12:06:54.750 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.701+0000 7efd43fff700 1 -- 172.21.15.102:0/495240149 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efd34001d60 con 0x7efd50090990 2021-05-25T12:06:54.750 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.701+0000 7efd43fff700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:06:54.750 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.701+0000 7efd43fff700 1 -- 172.21.15.102:0/495240149 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7efd34004980 con 0x7efd50090990 2021-05-25T12:06:54.750 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.701+0000 7efd43fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:54.750 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.701+0000 7efd43fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:06:54.751 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.701+0000 7efd43fff700 10 monclient: dump: 2021-05-25T12:06:54.751 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:54.751 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:54.751 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:54.752 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:54.752 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:54.752 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:54.752 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:54.752 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:54.753 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:54.753 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:54.753 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.701+0000 7efd4e7fc700 4 set_mon_vals no callback set 2021-05-25T12:06:54.753 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.701+0000 7efd43fff700 1 -- 172.21.15.102:0/495240149 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7efd3400dca0 con 0x7efd50090990 2021-05-25T12:06:54.754 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.701+0000 7efd43fff700 1 -- 172.21.15.102:0/495240149 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(218..218 src has 1..218) v4 ==== 5811+0+0 (unknown 452711690 0 0) 0x7efd3403d6a0 con 0x7efd50090990 2021-05-25T12:06:54.754 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.701+0000 7efd563c6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:06:54.754 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.701+0000 7efd43fff700 10 client.184607.objecter ms_dispatch 0x7efd50001550 osd_map(218..218 src has 1..218) v4 2021-05-25T12:06:54.754 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.701+0000 7efd43fff700 3 client.184607.objecter handle_osd_map got epochs [218,218] > 0 2021-05-25T12:06:54.754 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.701+0000 7efd43fff700 3 client.184607.objecter handle_osd_map decoding full epoch 218 2021-05-25T12:06:54.755 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.701+0000 7efd563c6700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:06:54.755 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.701+0000 7efd563c6700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:06:54.755 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.701+0000 7efd563c6700 1 -- 172.21.15.102:0/495240149 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd30004db0 con 0x7efd50090990 2021-05-25T12:06:54.755 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.701+0000 7efd43fff700 20 client.184607.objecter dump_active .. 0 homeless 2021-05-25T12:06:54.756 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.702+0000 7efd4f7fe700 10 monclient: get_auth_request con 0x7efd2c033430 auth_method 0 2021-05-25T12:06:54.756 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.702+0000 7efd43fff700 10 client.184607.objecter ms_handle_connect 0x7efd2c033430 2021-05-25T12:06:54.756 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.706+0000 7efd43fff700 1 -- 172.21.15.102:0/495240149 <== mon.2 v1:172.21.15.102:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7efd34012aa0 con 0x7efd50090990 2021-05-25T12:06:54.756 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.706+0000 7efd43fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:06:54.756 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.706+0000 7efd43fff700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:06:54.926 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.925+0000 7efd563c6700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2021-05-25T12:06:54.927 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.925+0000 7efd563c6700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2021-05-25T12:06:54.927 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.925+0000 7efd563c6700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:06:54.927 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.925+0000 7efd563c6700 1 -- 172.21.15.102:0/495240149 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7efd30005870 con 0x7efd50090990 2021-05-25T12:06:54.928 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.926+0000 7efd43fff700 1 -- 172.21.15.102:0/495240149 <== mon.2 v1:172.21.15.102:6790/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7efd34012f00 con 0x7efd50090990 2021-05-25T12:06:54.928 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.926+0000 7efd43fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2021-05-25T12:06:54.928 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.926+0000 7efd43fff700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:06:54.930 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:54.926+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:06:54.930 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:54.926+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=24,config=2,health=11,kv=6,logm=57} crc {auth=2064833997,config=3140514009,health=218881760,kv=1184736763,logm=4157451922}) 2021-05-25T12:06:54.930 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:54.926+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=2,health=11,kv=6,logm=55} crc {auth=4011382090,config=3140514009,health=4112810995,kv=1184736763,logm=832888600}) 2021-05-25T12:06:54.931 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:54.926+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:06:54.931 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:54.926+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=24,config=2,health=11,kv=6,logm=57} crc {auth=2064833997,config=3140514009,health=218881760,kv=1184736763,logm=4157451922}) 2021-05-25T12:06:54.931 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:54.926+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=26,config=2,health=10,kv=6,logm=56} crc {auth=1334932478,config=3140514009,health=2726134039,kv=1184736763,logm=3846184840}) 2021-05-25T12:06:54.931 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:54.927+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:06:54.931 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:54.927+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1261403301}) 2021-05-25T12:06:54.932 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:54.927+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3900662451}) 2021-05-25T12:06:54.932 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:54.927+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:06:54.932 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:54.927+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1261403301}) 2021-05-25T12:06:54.932 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:54.927+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=33262977}) 2021-05-25T12:06:54.934 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.928+0000 7efd563c6700 10 librados: watch_flush enter 2021-05-25T12:06:54.934 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.928+0000 7efd563c6700 10 librados: watch_flush exit 2021-05-25T12:06:54.934 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.928+0000 7efd563c6700 20 client.184607.objecter shutdown clearing up homeless session... 2021-05-25T12:06:54.935 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.928+0000 7efd563c6700 10 client.184607.objecter successfully canceled tick 2021-05-25T12:06:54.935 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.928+0000 7efd563c6700 1 -- 172.21.15.102:0/495240149 >> v1:172.21.15.181:6812/29582 conn(0x7efd2c033430 legacy=0x7efd2c0358f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:54.935 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.928+0000 7efd563c6700 10 monclient: shutdown 2021-05-25T12:06:54.935 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.928+0000 7efd563c6700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:06:54.935 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.928+0000 7efd563c6700 1 -- 172.21.15.102:0/495240149 >> v1:172.21.15.102:6790/0 conn(0x7efd50090990 legacy=0x7efd50139040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:54.936 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.928+0000 7efd563c6700 1 -- 172.21.15.102:0/495240149 shutdown_connections 2021-05-25T12:06:54.936 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.928+0000 7efd563c6700 1 -- 172.21.15.102:0/495240149 >> 172.21.15.102:0/495240149 conn(0x7efd5009b370 msgr2=0x7efd501292f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:06:54.936 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.928+0000 7efd563c6700 1 -- 172.21.15.102:0/495240149 shutdown_connections 2021-05-25T12:06:54.936 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.928+0000 7efd563c6700 1 -- 172.21.15.102:0/495240149 wait complete. 2021-05-25T12:06:54.937 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:54.928+0000 7efd563c6700 1 librados: shutdown 2021-05-25T12:06:54.940 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:54.930+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:06:54.941 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:54.930+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1564375719}) 2021-05-25T12:06:54.941 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:54.930+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3583628032}) 2021-05-25T12:06:54.941 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:54.930+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:06:54.941 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:54.930+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1564375719}) 2021-05-25T12:06:54.942 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:54.930+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3838382053}) 2021-05-25T12:06:54.942 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:54.933+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:06:54.942 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:54.933+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1953897609}) 2021-05-25T12:06:54.942 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:54.933+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3933031656}) 2021-05-25T12:06:54.943 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:54.933+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:06:54.943 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:54.933+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1953897609}) 2021-05-25T12:06:54.943 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:54.933+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2201202362}) 2021-05-25T12:06:54.943 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:54.935+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:06:54.943 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:54.935+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=49,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=4,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=16} crc {logm=12371834,mdsmap=1531151339,mgr=2996579194,mgr_command_descs=3824783157,mgr_metadata=1267308364,mgrstat=1779860287,mon_config_key=4118394475,monmap=3809614708,osd_metadata=4154602222,osd_pg_creating=4156578598,osd_snap=3284500666}) 2021-05-25T12:06:54.944 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:54.935+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=47,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=3,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=17} crc {logm=4254588608,mdsmap=1531151339,mgr=3925590689,mgr_command_descs=3824783157,mgr_metadata=1267308364,mgrstat=1019300002,mon_config_key=1824720183,monmap=3809614708,osd_metadata=725478305,osd_pg_creating=4156578598,osd_snap=2342895583}) 2021-05-25T12:06:54.944 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:54.935+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:06:54.944 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:54.935+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=49,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=4,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=16} crc {logm=12371834,mdsmap=1531151339,mgr=2996579194,mgr_command_descs=3824783157,mgr_metadata=1267308364,mgrstat=1779860287,mon_config_key=4118394475,monmap=3809614708,osd_metadata=4154602222,osd_pg_creating=4156578598,osd_snap=3284500666}) 2021-05-25T12:06:54.944 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:54.935+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=47,mdsmap=3,mgr=7,mgrstat=12,mon_config_key=4,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=18} crc {logm=2788904226,mdsmap=1531151339,mgr=2996579194,mgrstat=2124556668,mon_config_key=4118394475,monmap=3809614708,osd_metadata=252357741,osd_pg_creating=4156578598,osd_snap=3768177512}) 2021-05-25T12:06:54.945 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:54.936+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:06:54.945 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:54.936+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=11,osdmap=89} crc {osd_snap=670108138,osdmap=561778695}) 2021-05-25T12:06:54.945 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:54.936+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=9,osdmap=91} crc {osd_snap=2967366734,osdmap=2571273150}) 2021-05-25T12:06:54.945 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:54.936+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:06:54.946 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:54.936+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=11,osdmap=89} crc {osd_snap=670108138,osdmap=561778695}) 2021-05-25T12:06:54.946 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:54.936+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=10,osdmap=90} crc {osd_snap=3602735115,osdmap=1152796417}) 2021-05-25T12:06:54.946 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:54.937+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:06:54.946 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:54.937+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3174755030}) 2021-05-25T12:06:54.946 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:54.937+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=290089251}) 2021-05-25T12:06:54.947 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:54.937+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:06:54.947 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:54.937+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3174755030}) 2021-05-25T12:06:54.947 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:54.937+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3949784538}) 2021-05-25T12:06:54.947 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:54.939+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:06:54.948 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:54.939+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=91724203}) 2021-05-25T12:06:54.948 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:54.939+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1351724768}) 2021-05-25T12:06:54.948 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:54.939+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:06:54.948 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:54.939+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=91724203}) 2021-05-25T12:06:54.948 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:54.939+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2584635514}) 2021-05-25T12:06:54.950 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2021-05-25T12:06:54.951 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:54.947+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:06:54.951 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:54.947+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=33} crc {osdmap=242459171}) 2021-05-25T12:06:54.951 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:54.947+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=33} crc {osdmap=4156437582}) 2021-05-25T12:06:54.951 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:54.947+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:06:54.952 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:54.947+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=33} crc {osdmap=242459171}) 2021-05-25T12:06:54.952 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:06:54.947+0000 7f5d0dd46700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=33} crc {osdmap=1056858798}) 2021-05-25T12:06:55.951 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2021-05-25T12:06:55.952 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2021-05-25T12:06:56.082 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.067+0000 7f6d5d63d700 10 monclient: get_monmap_and_config 2021-05-25T12:06:56.083 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.067+0000 7f6d5d63d700 10 monclient: build_initial_monmap 2021-05-25T12:06:56.083 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.067+0000 7f6d5d63d700 10 monclient: monmap: 2021-05-25T12:06:56.083 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:06:56.083 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:56.084 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:06:56.068130+0000 2021-05-25T12:06:56.084 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:06:56.068130+0000 2021-05-25T12:06:56.084 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:06:56.084 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:56.085 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:06:56.085 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:06:56.085 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:06:56.085 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:56.086 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.067+0000 7f6d5d63d700 1 Processor -- start 2021-05-25T12:06:56.086 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.067+0000 7f6d5d63d700 1 -- start start 2021-05-25T12:06:56.086 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.067+0000 7f6d5d63d700 10 monclient: init 2021-05-25T12:06:56.086 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.068+0000 7f6d5d63d700 10 monclient: _reopen_session rank -1 2021-05-25T12:06:56.087 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.068+0000 7f6d5d63d700 10 monclient: _add_conns ranks=[1,2,0] 2021-05-25T12:06:56.087 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.068+0000 7f6d5d63d700 10 monclient(hunting): picked mon.noname-c con 0x7f6d58090990 addr v1:172.21.15.181:6789/0 2021-05-25T12:06:56.087 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.068+0000 7f6d5d63d700 10 monclient(hunting): picked mon.noname-b con 0x7f6d58090d70 addr v1:172.21.15.102:6790/0 2021-05-25T12:06:56.087 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.068+0000 7f6d5d63d700 10 monclient(hunting): picked mon.noname-a con 0x7f6d5812e330 addr v1:172.21.15.102:6789/0 2021-05-25T12:06:56.088 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.068+0000 7f6d5d63d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d58096fa0 con 0x7f6d5812e330 2021-05-25T12:06:56.088 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.068+0000 7f6d5d63d700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d580926a0 con 0x7f6d58090990 2021-05-25T12:06:56.088 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.068+0000 7f6d5d63d700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d58092940 con 0x7f6d58090d70 2021-05-25T12:06:56.089 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.068+0000 7f6d5d63d700 10 monclient(hunting): _renew_subs 2021-05-25T12:06:56.089 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.068+0000 7f6d5d63d700 10 monclient(hunting): authenticate will time out at 1216.665248s 2021-05-25T12:06:56.089 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.068+0000 7f6d577fe700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f6d5812e330 0x7f6d58130720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33548/0 (socket says 172.21.15.102:33548) 2021-05-25T12:06:56.089 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.068+0000 7f6d567fc700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f6d58090d70 0x7f6d58091200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:56654/0 (socket says 172.21.15.102:56654) 2021-05-25T12:06:56.090 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.068+0000 7f6d577fe700 1 -- 172.21.15.102:0/3076244072 learned_addr learned my addr 172.21.15.102:0/3076244072 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:06:56.090 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.068+0000 7f6d55ffb700 1 -- 172.21.15.102:0/3076244072 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1744311381 0 0) 0x7f6d58096fa0 con 0x7f6d5812e330 2021-05-25T12:06:56.090 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.068+0000 7f6d55ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:56.090 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.068+0000 7f6d55ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:56.091 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.068+0000 7f6d55ffb700 10 monclient(hunting): my global_id is 204270 2021-05-25T12:06:56.091 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.068+0000 7f6d55ffb700 1 -- 172.21.15.102:0/3076244072 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d44003270 con 0x7f6d5812e330 2021-05-25T12:06:56.091 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.068+0000 7f6d55ffb700 1 -- 172.21.15.102:0/3076244072 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 839377718 0 0) 0x7f6d58092940 con 0x7f6d58090d70 2021-05-25T12:06:56.091 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.068+0000 7f6d55ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:56.091 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.068+0000 7f6d55ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:56.092 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.068+0000 7f6d55ffb700 10 monclient(hunting): my global_id is 184610 2021-05-25T12:06:56.092 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.068+0000 7f6d55ffb700 1 -- 172.21.15.102:0/3076244072 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d58096fa0 con 0x7f6d58090d70 2021-05-25T12:06:56.093 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.068+0000 7f6d55ffb700 1 -- 172.21.15.102:0/3076244072 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3464860648 0 0) 0x7f6d44003270 con 0x7f6d5812e330 2021-05-25T12:06:56.093 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.068+0000 7f6d55ffb700 1 -- 172.21.15.102:0/3076244072 >> v1:172.21.15.102:6790/0 conn(0x7f6d58090d70 legacy=0x7f6d58091200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:56.093 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.068+0000 7f6d55ffb700 1 -- 172.21.15.102:0/3076244072 >> v1:172.21.15.181:6789/0 conn(0x7f6d58090990 legacy=0x7f6d58093240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:56.093 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.068+0000 7f6d55ffb700 10 monclient: _finish_hunting 0 2021-05-25T12:06:56.093 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.068+0000 7f6d55ffb700 1 monclient: found mon.noname-a 2021-05-25T12:06:56.094 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.068+0000 7f6d55ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:06:56.094 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.068+0000 7f6d55ffb700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T12:06:56.095 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.068+0000 7f6d55ffb700 1 -- 172.21.15.102:0/3076244072 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d58092be0 con 0x7f6d5812e330 2021-05-25T12:06:56.095 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.068+0000 7f6d55ffb700 10 monclient: _finish_auth 0 2021-05-25T12:06:56.095 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.068+0000 7f6d55ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:06:56.095 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.068+0000 7f6d55ffb700 1 -- 172.21.15.102:0/3076244072 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f6d48001990 con 0x7f6d5812e330 2021-05-25T12:06:56.096 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.068+0000 7f6d5d63d700 5 monclient: authenticate success, global_id 204270 2021-05-25T12:06:56.096 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.068+0000 7f6d55ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:56.096 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.068+0000 7f6d55ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:06:56.096 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.068+0000 7f6d55ffb700 10 monclient: dump: 2021-05-25T12:06:56.097 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:56.097 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:56.097 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:56.097 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:56.097 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:56.098 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:56.098 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:56.098 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:56.099 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:56.099 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:56.099 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.068+0000 7f6d5d63d700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:06:56.100 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.069+0000 7f6d55ffb700 1 -- 172.21.15.102:0/3076244072 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6d48001c00 con 0x7f6d5812e330 2021-05-25T12:06:56.100 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.069+0000 7f6d55ffb700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:06:56.100 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.069+0000 7f6d5d63d700 10 monclient: get_monmap_and_config success 2021-05-25T12:06:56.100 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.069+0000 7f6d55ffb700 1 -- 172.21.15.102:0/3076244072 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f6d48001f90 con 0x7f6d5812e330 2021-05-25T12:06:56.100 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.069+0000 7f6d55ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:56.101 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.069+0000 7f6d5d63d700 4 set_mon_vals no callback set 2021-05-25T12:06:56.101 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.069+0000 7f6d55ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:06:56.101 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.069+0000 7f6d55ffb700 10 monclient: dump: 2021-05-25T12:06:56.101 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:56.102 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:56.102 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:56.102 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:56.102 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:56.102 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:56.103 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:56.103 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:56.103 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:56.104 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:56.104 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.069+0000 7f6d5d63d700 10 monclient: shutdown 2021-05-25T12:06:56.104 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.069+0000 7f6d5d63d700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:06:56.104 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.069+0000 7f6d5d63d700 1 -- 172.21.15.102:0/3076244072 >> v1:172.21.15.102:6789/0 conn(0x7f6d5812e330 legacy=0x7f6d58130720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:56.104 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.069+0000 7f6d5d63d700 1 -- 172.21.15.102:0/3076244072 shutdown_connections 2021-05-25T12:06:56.105 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.069+0000 7f6d5d63d700 1 -- 172.21.15.102:0/3076244072 >> 172.21.15.102:0/3076244072 conn(0x7f6d5809b370 msgr2=0x7f6d58097430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:06:56.105 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.069+0000 7f6d5d63d700 1 -- 172.21.15.102:0/3076244072 shutdown_connections 2021-05-25T12:06:56.105 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.069+0000 7f6d5d63d700 1 -- 172.21.15.102:0/3076244072 wait complete. 2021-05-25T12:06:56.105 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.070+0000 7f6d5d63d700 10 monclient: build_initial_monmap 2021-05-25T12:06:56.105 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.070+0000 7f6d5d63d700 10 monclient: monmap: 2021-05-25T12:06:56.106 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:06:56.106 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:06:56.106 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:06:56.107 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:06:56.107 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:06:56.107 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:06:56.107 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:06:56.107 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:06:56.108 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:06:56.108 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:56.108 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.070+0000 7f6d5d63d700 1 librados: starting msgr at 2021-05-25T12:06:56.109 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.070+0000 7f6d5d63d700 1 librados: starting objecter 2021-05-25T12:06:56.109 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.070+0000 7f6d5d63d700 1 Processor -- start 2021-05-25T12:06:56.110 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.070+0000 7f6d5d63d700 1 -- start start 2021-05-25T12:06:56.110 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.070+0000 7f6d5d63d700 1 librados: setting wanted keys 2021-05-25T12:06:56.110 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.070+0000 7f6d5d63d700 1 librados: calling monclient init 2021-05-25T12:06:56.111 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.070+0000 7f6d5d63d700 10 monclient: init 2021-05-25T12:06:56.111 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.070+0000 7f6d5d63d700 10 monclient: _reopen_session rank -1 2021-05-25T12:06:56.111 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.070+0000 7f6d5d63d700 10 monclient: _add_conns ranks=[2,0,1] 2021-05-25T12:06:56.112 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.070+0000 7f6d5d63d700 10 monclient(hunting): picked mon.noname-c con 0x7f6d58090990 addr v1:172.21.15.102:6790/0 2021-05-25T12:06:56.112 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.070+0000 7f6d5d63d700 10 monclient(hunting): picked mon.noname-a con 0x7f6d58090d70 addr v1:172.21.15.102:6789/0 2021-05-25T12:06:56.113 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.071+0000 7f6d5d63d700 10 monclient(hunting): picked mon.noname-b con 0x7f6d5812e330 addr v1:172.21.15.181:6789/0 2021-05-25T12:06:56.113 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.071+0000 7f6d5d63d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d58099b40 con 0x7f6d58090d70 2021-05-25T12:06:56.114 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.071+0000 7f6d5d63d700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d58119d20 con 0x7f6d5812e330 2021-05-25T12:06:56.114 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.071+0000 7f6d5d63d700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d58119fc0 con 0x7f6d58090990 2021-05-25T12:06:56.114 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.071+0000 7f6d5d63d700 10 monclient(hunting): _renew_subs 2021-05-25T12:06:56.115 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.071+0000 7f6d5d63d700 10 monclient(hunting): authenticate will time out at 1216.668057s 2021-05-25T12:06:56.115 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.071+0000 7f6d56ffd700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f6d58090990 0x7f6d5813ac80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:56660/0 (socket says 172.21.15.102:56660) 2021-05-25T12:06:56.116 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.071+0000 7f6d567fc700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f6d58090d70 0x7f6d5812b570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33552/0 (socket says 172.21.15.102:33552) 2021-05-25T12:06:56.116 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.071+0000 7f6d56ffd700 1 -- 172.21.15.102:0/3980347252 learned_addr learned my addr 172.21.15.102:0/3980347252 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:06:56.117 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.071+0000 7f6d3effd700 10 client.?.objecter ms_handle_connect 0x7f6d58090d70 2021-05-25T12:06:56.117 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.071+0000 7f6d3effd700 10 client.?.objecter resend_mon_ops 2021-05-25T12:06:56.117 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.071+0000 7f6d3effd700 10 client.?.objecter ms_handle_connect 0x7f6d58090990 2021-05-25T12:06:56.118 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.071+0000 7f6d3effd700 10 client.?.objecter resend_mon_ops 2021-05-25T12:06:56.118 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.071+0000 7f6d3effd700 10 client.?.objecter ms_handle_connect 0x7f6d5812e330 2021-05-25T12:06:56.119 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.071+0000 7f6d3effd700 10 client.?.objecter resend_mon_ops 2021-05-25T12:06:56.119 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.071+0000 7f6d3effd700 1 -- 172.21.15.102:0/3980347252 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 472450002 0 0) 0x7f6d58099b40 con 0x7f6d58090d70 2021-05-25T12:06:56.119 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.071+0000 7f6d3effd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:56.120 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.071+0000 7f6d3effd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:56.120 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.071+0000 7f6d3effd700 10 monclient(hunting): my global_id is 204273 2021-05-25T12:06:56.121 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.071+0000 7f6d3effd700 1 -- 172.21.15.102:0/3980347252 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d38004c10 con 0x7f6d58090d70 2021-05-25T12:06:56.121 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.071+0000 7f6d3effd700 1 -- 172.21.15.102:0/3980347252 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1342521963 0 0) 0x7f6d58119fc0 con 0x7f6d58090990 2021-05-25T12:06:56.121 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.071+0000 7f6d3effd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:56.122 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.071+0000 7f6d3effd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:56.122 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.071+0000 7f6d3effd700 10 monclient(hunting): my global_id is 184613 2021-05-25T12:06:56.123 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.071+0000 7f6d3effd700 1 -- 172.21.15.102:0/3980347252 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d58099b40 con 0x7f6d58090990 2021-05-25T12:06:56.123 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.071+0000 7f6d3effd700 1 -- 172.21.15.102:0/3980347252 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4070949119 0 0) 0x7f6d58119d20 con 0x7f6d5812e330 2021-05-25T12:06:56.123 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.071+0000 7f6d3effd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:56.123 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.071+0000 7f6d3effd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:56.124 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.071+0000 7f6d3effd700 10 monclient(hunting): my global_id is 224140 2021-05-25T12:06:56.124 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.071+0000 7f6d3effd700 1 -- 172.21.15.102:0/3980347252 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d58119fc0 con 0x7f6d5812e330 2021-05-25T12:06:56.124 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.071+0000 7f6d3effd700 1 -- 172.21.15.102:0/3980347252 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 585472758 0 0) 0x7f6d58099b40 con 0x7f6d58090990 2021-05-25T12:06:56.124 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.071+0000 7f6d3effd700 1 -- 172.21.15.102:0/3980347252 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6d58119d20 con 0x7f6d58090990 2021-05-25T12:06:56.124 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.071+0000 7f6d3effd700 1 -- 172.21.15.102:0/3980347252 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f6d4c001950 con 0x7f6d58090990 2021-05-25T12:06:56.125 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.071+0000 7f6d3effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:56.125 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.071+0000 7f6d3effd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:06:56.125 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.071+0000 7f6d3effd700 10 monclient(hunting): dump: 2021-05-25T12:06:56.125 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:56.125 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:56.126 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:56.126 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:56.126 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:56.126 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:56.127 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:56.127 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:56.127 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:56.127 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:56.127 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.071+0000 7f6d3effd700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:06:56.128 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.071+0000 7f6d3effd700 1 -- 172.21.15.102:0/3980347252 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3740169676 0 0) 0x7f6d38004c10 con 0x7f6d58090d70 2021-05-25T12:06:56.128 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.071+0000 7f6d3effd700 1 -- 172.21.15.102:0/3980347252 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6d58099b40 con 0x7f6d58090d70 2021-05-25T12:06:56.128 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.071+0000 7f6d3effd700 1 -- 172.21.15.102:0/3980347252 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f6d40001840 con 0x7f6d58090d70 2021-05-25T12:06:56.128 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.071+0000 7f6d3effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:56.129 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.071+0000 7f6d3effd700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2021-05-25T12:06:56.129 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.071+0000 7f6d3effd700 10 monclient(hunting): dump: 2021-05-25T12:06:56.129 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:56.130 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:56.130 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:56.131 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:56.131 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:56.132 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:56.132 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:56.133 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:56.133 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:56.133 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:56.134 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.071+0000 7f6d3effd700 1 -- 172.21.15.102:0/3980347252 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3808480884 0 0) 0x7f6d58119fc0 con 0x7f6d5812e330 2021-05-25T12:06:56.134 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.071+0000 7f6d3effd700 1 -- 172.21.15.102:0/3980347252 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6d38004c10 con 0x7f6d5812e330 2021-05-25T12:06:56.135 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.071+0000 7f6d3effd700 1 -- 172.21.15.102:0/3980347252 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f6d48001950 con 0x7f6d5812e330 2021-05-25T12:06:56.135 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.071+0000 7f6d3effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:56.136 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.071+0000 7f6d3effd700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2021-05-25T12:06:56.136 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.071+0000 7f6d3effd700 10 monclient(hunting): dump: 2021-05-25T12:06:56.137 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:56.137 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:56.137 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:56.138 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:56.138 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:56.138 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:56.139 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:56.139 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:56.139 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:56.139 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:56.139 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.071+0000 7f6d3effd700 1 -- 172.21.15.102:0/3980347252 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1625600660 0 0) 0x7f6d58119d20 con 0x7f6d58090990 2021-05-25T12:06:56.140 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.071+0000 7f6d3effd700 1 -- 172.21.15.102:0/3980347252 >> v1:172.21.15.181:6789/0 conn(0x7f6d5812e330 legacy=0x7f6d58142ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:56.140 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.071+0000 7f6d3effd700 1 -- 172.21.15.102:0/3980347252 >> v1:172.21.15.102:6789/0 conn(0x7f6d58090d70 legacy=0x7f6d5812b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:56.140 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.071+0000 7f6d3effd700 10 monclient: _finish_hunting 0 2021-05-25T12:06:56.140 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.071+0000 7f6d3effd700 1 monclient: found mon.c 2021-05-25T12:06:56.141 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.071+0000 7f6d3effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:06:56.141 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.071+0000 7f6d3effd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:06:56.141 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.071+0000 7f6d3effd700 1 -- 172.21.15.102:0/3980347252 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d58117e50 con 0x7f6d58090990 2021-05-25T12:06:56.141 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.071+0000 7f6d3effd700 10 monclient: _finish_auth 0 2021-05-25T12:06:56.141 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.071+0000 7f6d3effd700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:06:56.142 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.071+0000 7f6d5d63d700 5 monclient: authenticate success, global_id 184613 2021-05-25T12:06:56.142 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.071+0000 7f6d5d63d700 10 monclient: _renew_subs 2021-05-25T12:06:56.142 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.071+0000 7f6d5d63d700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:06:56.142 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.071+0000 7f6d5d63d700 1 -- 172.21.15.102:0/3980347252 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d5813b390 con 0x7f6d58090990 2021-05-25T12:06:56.143 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.071+0000 7f6d5d63d700 10 client.184613.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:06:56.143 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.071+0000 7f6d5d63d700 10 monclient: _renew_subs 2021-05-25T12:06:56.143 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.071+0000 7f6d5d63d700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:06:56.143 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.071+0000 7f6d5d63d700 1 -- 172.21.15.102:0/3980347252 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d5813b640 con 0x7f6d58090990 2021-05-25T12:06:56.143 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.071+0000 7f6d5d63d700 1 librados: init done 2021-05-25T12:06:56.144 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.072+0000 7f6d3effd700 1 -- 172.21.15.102:0/3980347252 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6d4c001d20 con 0x7f6d58090990 2021-05-25T12:06:56.144 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.072+0000 7f6d3effd700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:06:56.144 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.072+0000 7f6d3effd700 1 -- 172.21.15.102:0/3980347252 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f6d4c002350 con 0x7f6d58090990 2021-05-25T12:06:56.144 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.072+0000 7f6d3effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:56.144 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.072+0000 7f6d3effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:06:56.145 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.072+0000 7f6d557fa700 4 set_mon_vals no callback set 2021-05-25T12:06:56.145 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.072+0000 7f6d3effd700 10 monclient: dump: 2021-05-25T12:06:56.145 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:56.145 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:56.146 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:56.146 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:56.146 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:56.146 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:56.146 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:56.147 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:56.147 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:56.147 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:56.147 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.072+0000 7f6d3effd700 1 -- 172.21.15.102:0/3980347252 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f6d4c002530 con 0x7f6d58090990 2021-05-25T12:06:56.147 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.072+0000 7f6d5d63d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:06:56.148 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.072+0000 7f6d5d63d700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:06:56.148 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.072+0000 7f6d5d63d700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:06:56.148 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.072+0000 7f6d5d63d700 1 -- 172.21.15.102:0/3980347252 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d30004db0 con 0x7f6d58090990 2021-05-25T12:06:56.148 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.072+0000 7f6d3effd700 1 -- 172.21.15.102:0/3980347252 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(220..220 src has 1..220) v4 ==== 6648+0+0 (unknown 208579584 0 0) 0x7f6d4c03da60 con 0x7f6d58090990 2021-05-25T12:06:56.148 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.072+0000 7f6d3effd700 10 client.184613.objecter ms_dispatch 0x7f6d58001550 osd_map(220..220 src has 1..220) v4 2021-05-25T12:06:56.149 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.072+0000 7f6d3effd700 3 client.184613.objecter handle_osd_map got epochs [220,220] > 0 2021-05-25T12:06:56.149 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.072+0000 7f6d3effd700 3 client.184613.objecter handle_osd_map decoding full epoch 220 2021-05-25T12:06:56.149 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.073+0000 7f6d3effd700 20 client.184613.objecter dump_active .. 0 homeless 2021-05-25T12:06:56.149 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.073+0000 7f6d567fc700 10 monclient: get_auth_request con 0x7f6d38033500 auth_method 0 2021-05-25T12:06:56.149 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.073+0000 7f6d3effd700 10 client.184613.objecter ms_handle_connect 0x7f6d38033500 2021-05-25T12:06:56.150 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.077+0000 7f6d3effd700 1 -- 172.21.15.102:0/3980347252 <== mon.2 v1:172.21.15.102:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7f6d4c012df0 con 0x7f6d58090990 2021-05-25T12:06:56.150 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.077+0000 7f6d3effd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:06:56.150 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.077+0000 7f6d3effd700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:06:56.283 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.282+0000 7f6d5d63d700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2021-05-25T12:06:56.284 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.282+0000 7f6d5d63d700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2021-05-25T12:06:56.284 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.282+0000 7f6d5d63d700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:06:56.284 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.282+0000 7f6d5d63d700 1 -- 172.21.15.102:0/3980347252 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f6d30005060 con 0x7f6d58090990 2021-05-25T12:06:56.285 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.282+0000 7f6d3effd700 1 -- 172.21.15.102:0/3980347252 <== mon.2 v1:172.21.15.102:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1342 (unknown 254746657 0 1148417338) 0x7f6d4c012bc0 con 0x7f6d58090990 2021-05-25T12:06:56.285 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.282+0000 7f6d3effd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2021-05-25T12:06:56.285 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.282+0000 7f6d3effd700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:06:56.285 INFO:teuthology.orchestra.run.smithi102.stdout:{"election_epoch":58,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":2,"features":{"quorum_con":"4540138303579357183","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"20ee91b6-8935-4b98-b868-85c03789ccc4","modified":"2021-05-25T11:59:11.011522Z","created":"2021-05-25T11:59:11.011522Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6790","nonce":0}]},"addr":"172.21.15.102:6790/0","public_addr":"172.21.15.102:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2021-05-25T12:06:56.287 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.284+0000 7f6d5d63d700 10 librados: watch_flush enter 2021-05-25T12:06:56.287 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.284+0000 7f6d5d63d700 10 librados: watch_flush exit 2021-05-25T12:06:56.287 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.284+0000 7f6d5d63d700 20 client.184613.objecter shutdown clearing up homeless session... 2021-05-25T12:06:56.288 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.284+0000 7f6d5d63d700 10 client.184613.objecter successfully canceled tick 2021-05-25T12:06:56.288 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.284+0000 7f6d5d63d700 1 -- 172.21.15.102:0/3980347252 >> v1:172.21.15.181:6812/29582 conn(0x7f6d38033500 legacy=0x7f6d380359c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:56.288 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.284+0000 7f6d5d63d700 10 monclient: shutdown 2021-05-25T12:06:56.288 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.284+0000 7f6d5d63d700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:06:56.289 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.284+0000 7f6d5d63d700 1 -- 172.21.15.102:0/3980347252 >> v1:172.21.15.102:6790/0 conn(0x7f6d58090990 legacy=0x7f6d5813ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:56.289 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.284+0000 7f6d5d63d700 1 -- 172.21.15.102:0/3980347252 shutdown_connections 2021-05-25T12:06:56.289 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.284+0000 7f6d5d63d700 1 -- 172.21.15.102:0/3980347252 >> 172.21.15.102:0/3980347252 conn(0x7f6d5809b370 msgr2=0x7f6d5812d910 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:06:56.289 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.284+0000 7f6d5d63d700 1 -- 172.21.15.102:0/3980347252 shutdown_connections 2021-05-25T12:06:56.289 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.284+0000 7f6d5d63d700 1 -- 172.21.15.102:0/3980347252 wait complete. 2021-05-25T12:06:56.290 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.284+0000 7f6d5d63d700 1 librados: shutdown 2021-05-25T12:06:56.296 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2021-05-25T12:06:56.296 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2021-05-25T12:06:56.297 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2021-05-25T12:06:56.406 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.404+0000 7fa151b84700 10 monclient: get_monmap_and_config 2021-05-25T12:06:56.406 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.404+0000 7fa151b84700 10 monclient: build_initial_monmap 2021-05-25T12:06:56.407 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.404+0000 7fa151b84700 10 monclient: monmap: 2021-05-25T12:06:56.407 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:06:56.407 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:56.408 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:06:56.405812+0000 2021-05-25T12:06:56.408 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:06:56.405812+0000 2021-05-25T12:06:56.408 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:06:56.408 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:56.409 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:06:56.409 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:06:56.409 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:06:56.409 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:56.423 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.405+0000 7fa151b84700 1 Processor -- start 2021-05-25T12:06:56.423 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.405+0000 7fa151b84700 1 -- start start 2021-05-25T12:06:56.423 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.405+0000 7fa151b84700 10 monclient: init 2021-05-25T12:06:56.423 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.406+0000 7fa151b84700 10 monclient: _reopen_session rank -1 2021-05-25T12:06:56.424 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.406+0000 7fa151b84700 10 monclient: _add_conns ranks=[2,1,0] 2021-05-25T12:06:56.424 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.406+0000 7fa151b84700 10 monclient(hunting): picked mon.noname-b con 0x7fa14c096630 addr v1:172.21.15.102:6790/0 2021-05-25T12:06:56.424 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.406+0000 7fa151b84700 10 monclient(hunting): picked mon.noname-c con 0x7fa14c093bd0 addr v1:172.21.15.181:6789/0 2021-05-25T12:06:56.424 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.406+0000 7fa151b84700 10 monclient(hunting): picked mon.noname-a con 0x7fa14c096a30 addr v1:172.21.15.102:6789/0 2021-05-25T12:06:56.425 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.406+0000 7fa151b84700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa14c0630c0 con 0x7fa14c096a30 2021-05-25T12:06:56.425 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.406+0000 7fa151b84700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa14c064320 con 0x7fa14c093bd0 2021-05-25T12:06:56.425 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.406+0000 7fa151b84700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa14c0645c0 con 0x7fa14c096630 2021-05-25T12:06:56.425 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.406+0000 7fa151b84700 10 monclient(hunting): _renew_subs 2021-05-25T12:06:56.426 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.406+0000 7fa151b84700 10 monclient(hunting): authenticate will time out at 1217.003209s 2021-05-25T12:06:56.426 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.406+0000 7fa14b7fe700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fa14c096630 0x7fa14c0697d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:56668/0 (socket says 172.21.15.102:56668) 2021-05-25T12:06:56.426 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.406+0000 7fa14b7fe700 1 -- 172.21.15.102:0/3651900999 learned_addr learned my addr 172.21.15.102:0/3651900999 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:06:56.426 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.406+0000 7fa14bfff700 1 --1- 172.21.15.102:0/3651900999 >> v1:172.21.15.102:6789/0 conn(0x7fa14c096a30 0x7fa14c134020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33562/0 (socket says 172.21.15.102:33562) 2021-05-25T12:06:56.427 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.406+0000 7fa14a7fc700 1 -- 172.21.15.102:0/3651900999 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 961325655 0 0) 0x7fa14c0645c0 con 0x7fa14c096630 2021-05-25T12:06:56.427 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.406+0000 7fa14a7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:56.427 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.406+0000 7fa14a7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:56.427 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.406+0000 7fa14a7fc700 10 monclient(hunting): my global_id is 184616 2021-05-25T12:06:56.427 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.406+0000 7fa14a7fc700 1 -- 172.21.15.102:0/3651900999 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa130003270 con 0x7fa14c096630 2021-05-25T12:06:56.428 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.406+0000 7fa14a7fc700 1 -- 172.21.15.102:0/3651900999 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3449299586 0 0) 0x7fa14c0630c0 con 0x7fa14c096a30 2021-05-25T12:06:56.428 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.406+0000 7fa14a7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:56.428 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.406+0000 7fa14a7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:56.428 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.406+0000 7fa14a7fc700 10 monclient(hunting): my global_id is 204276 2021-05-25T12:06:56.429 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.406+0000 7fa14a7fc700 1 -- 172.21.15.102:0/3651900999 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa14c0645c0 con 0x7fa14c096a30 2021-05-25T12:06:56.429 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.406+0000 7fa14a7fc700 1 -- 172.21.15.102:0/3651900999 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 888708381 0 0) 0x7fa130003270 con 0x7fa14c096630 2021-05-25T12:06:56.429 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.406+0000 7fa14a7fc700 1 -- 172.21.15.102:0/3651900999 >> v1:172.21.15.181:6789/0 conn(0x7fa14c093bd0 legacy=0x7fa14c1308f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:56.429 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.406+0000 7fa14a7fc700 1 -- 172.21.15.102:0/3651900999 >> v1:172.21.15.102:6789/0 conn(0x7fa14c096a30 legacy=0x7fa14c134020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:56.429 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.406+0000 7fa14a7fc700 10 monclient: _finish_hunting 0 2021-05-25T12:06:56.430 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.406+0000 7fa14a7fc700 1 monclient: found mon.noname-b 2021-05-25T12:06:56.430 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.406+0000 7fa14a7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:06:56.430 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.406+0000 7fa14a7fc700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T12:06:56.430 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.406+0000 7fa14a7fc700 1 -- 172.21.15.102:0/3651900999 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa14c064860 con 0x7fa14c096630 2021-05-25T12:06:56.431 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.406+0000 7fa14a7fc700 10 monclient: _finish_auth 0 2021-05-25T12:06:56.431 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.406+0000 7fa14a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:06:56.431 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.406+0000 7fa14a7fc700 1 -- 172.21.15.102:0/3651900999 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fa138001610 con 0x7fa14c096630 2021-05-25T12:06:56.431 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.406+0000 7fa151b84700 5 monclient: authenticate success, global_id 184616 2021-05-25T12:06:56.432 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.406+0000 7fa151b84700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:06:56.432 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.406+0000 7fa14a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:56.432 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.406+0000 7fa14a7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:06:56.433 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.406+0000 7fa14a7fc700 10 monclient: dump: 2021-05-25T12:06:56.433 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:56.433 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:56.433 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:56.434 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:56.434 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:56.434 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:56.434 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:56.434 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:56.435 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:56.435 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:56.435 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.406+0000 7fa151b84700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:06:56.435 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.406+0000 7fa14a7fc700 1 -- 172.21.15.102:0/3651900999 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa138001840 con 0x7fa14c096630 2021-05-25T12:06:56.435 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.406+0000 7fa14a7fc700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:06:56.436 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.406+0000 7fa151b84700 10 monclient: get_monmap_and_config success 2021-05-25T12:06:56.436 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.406+0000 7fa14a7fc700 1 -- 172.21.15.102:0/3651900999 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fa138001bd0 con 0x7fa14c096630 2021-05-25T12:06:56.436 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.406+0000 7fa14a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:56.436 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.406+0000 7fa151b84700 4 set_mon_vals no callback set 2021-05-25T12:06:56.436 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.406+0000 7fa14a7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:06:56.437 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.406+0000 7fa14a7fc700 10 monclient: dump: 2021-05-25T12:06:56.437 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:56.437 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:56.437 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:56.438 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:56.438 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:56.438 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:56.438 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:56.438 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:56.439 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:56.439 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:56.439 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.407+0000 7fa151b84700 10 monclient: shutdown 2021-05-25T12:06:56.439 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.407+0000 7fa151b84700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:06:56.440 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.407+0000 7fa151b84700 1 -- 172.21.15.102:0/3651900999 >> v1:172.21.15.102:6790/0 conn(0x7fa14c096630 legacy=0x7fa14c0697d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:56.440 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.407+0000 7fa151b84700 1 -- 172.21.15.102:0/3651900999 shutdown_connections 2021-05-25T12:06:56.440 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.407+0000 7fa151b84700 1 -- 172.21.15.102:0/3651900999 >> 172.21.15.102:0/3651900999 conn(0x7fa14c09cd10 msgr2=0x7fa14c09d120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:06:56.440 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.407+0000 7fa151b84700 1 -- 172.21.15.102:0/3651900999 shutdown_connections 2021-05-25T12:06:56.441 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.407+0000 7fa151b84700 1 -- 172.21.15.102:0/3651900999 wait complete. 2021-05-25T12:06:56.441 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.407+0000 7fa151b84700 10 monclient: build_initial_monmap 2021-05-25T12:06:56.441 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.407+0000 7fa151b84700 10 monclient: monmap: 2021-05-25T12:06:56.441 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:06:56.441 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:06:56.442 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:06:56.442 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:06:56.442 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:06:56.443 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:06:56.443 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:06:56.443 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:06:56.443 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:06:56.444 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:56.444 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.408+0000 7fa151b84700 1 librados: starting msgr at 2021-05-25T12:06:56.444 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.408+0000 7fa151b84700 1 librados: starting objecter 2021-05-25T12:06:56.445 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.408+0000 7fa151b84700 1 Processor -- start 2021-05-25T12:06:56.445 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.408+0000 7fa151b84700 1 -- start start 2021-05-25T12:06:56.445 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.408+0000 7fa151b84700 1 librados: setting wanted keys 2021-05-25T12:06:56.445 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.408+0000 7fa151b84700 1 librados: calling monclient init 2021-05-25T12:06:56.445 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.408+0000 7fa151b84700 10 monclient: init 2021-05-25T12:06:56.446 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.408+0000 7fa151b84700 10 monclient: _reopen_session rank -1 2021-05-25T12:06:56.446 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.408+0000 7fa151b84700 10 monclient: _add_conns ranks=[0,2,1] 2021-05-25T12:06:56.446 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.408+0000 7fa151b84700 10 monclient(hunting): picked mon.noname-a con 0x7fa14c093bd0 addr v1:172.21.15.102:6789/0 2021-05-25T12:06:56.446 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.408+0000 7fa151b84700 10 monclient(hunting): picked mon.noname-c con 0x7fa14c096630 addr v1:172.21.15.102:6790/0 2021-05-25T12:06:56.447 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.408+0000 7fa151b84700 10 monclient(hunting): picked mon.noname-b con 0x7fa14c096a30 addr v1:172.21.15.181:6789/0 2021-05-25T12:06:56.447 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.408+0000 7fa151b84700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa14c11bb60 con 0x7fa14c093bd0 2021-05-25T12:06:56.447 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.408+0000 7fa151b84700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa14c11be00 con 0x7fa14c096a30 2021-05-25T12:06:56.447 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.408+0000 7fa151b84700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa14c11c0a0 con 0x7fa14c096630 2021-05-25T12:06:56.447 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.408+0000 7fa151b84700 10 monclient(hunting): _renew_subs 2021-05-25T12:06:56.448 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.408+0000 7fa151b84700 10 monclient(hunting): authenticate will time out at 1217.005797s 2021-05-25T12:06:56.448 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.408+0000 7fa14b7fe700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fa14c093bd0 0x7fa14c13acb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33564/0 (socket says 172.21.15.102:33564) 2021-05-25T12:06:56.448 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.408+0000 7fa14b7fe700 1 -- 172.21.15.102:0/3521183598 learned_addr learned my addr 172.21.15.102:0/3521183598 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:06:56.448 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.408+0000 7fa1377fe700 10 client.?.objecter ms_handle_connect 0x7fa14c096630 2021-05-25T12:06:56.449 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.408+0000 7fa1377fe700 10 client.?.objecter resend_mon_ops 2021-05-25T12:06:56.449 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.408+0000 7fa1377fe700 10 client.?.objecter ms_handle_connect 0x7fa14c093bd0 2021-05-25T12:06:56.449 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.408+0000 7fa1377fe700 10 client.?.objecter resend_mon_ops 2021-05-25T12:06:56.449 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.409+0000 7fa1377fe700 1 -- 172.21.15.102:0/3521183598 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1760542104 0 0) 0x7fa14c11c0a0 con 0x7fa14c096630 2021-05-25T12:06:56.450 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.409+0000 7fa1377fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:56.450 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.409+0000 7fa1377fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:56.450 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.409+0000 7fa1377fe700 10 monclient(hunting): my global_id is 184619 2021-05-25T12:06:56.450 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.409+0000 7fa1377fe700 1 -- 172.21.15.102:0/3521183598 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa12c004c10 con 0x7fa14c096630 2021-05-25T12:06:56.450 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.409+0000 7fa1377fe700 10 client.?.objecter ms_handle_connect 0x7fa14c096a30 2021-05-25T12:06:56.451 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.409+0000 7fa1377fe700 10 client.?.objecter resend_mon_ops 2021-05-25T12:06:56.451 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.409+0000 7fa1377fe700 1 -- 172.21.15.102:0/3521183598 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 54566939 0 0) 0x7fa14c11bb60 con 0x7fa14c093bd0 2021-05-25T12:06:56.451 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.409+0000 7fa1377fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:56.451 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.409+0000 7fa1377fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:56.452 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.409+0000 7fa1377fe700 10 monclient(hunting): my global_id is 204279 2021-05-25T12:06:56.452 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.409+0000 7fa1377fe700 1 -- 172.21.15.102:0/3521183598 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa14c11c0a0 con 0x7fa14c093bd0 2021-05-25T12:06:56.452 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.409+0000 7fa1377fe700 1 -- 172.21.15.102:0/3521183598 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 737792387 0 0) 0x7fa14c11be00 con 0x7fa14c096a30 2021-05-25T12:06:56.452 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.409+0000 7fa1377fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:56.452 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.409+0000 7fa1377fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:56.453 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.409+0000 7fa1377fe700 10 monclient(hunting): my global_id is 224146 2021-05-25T12:06:56.453 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.409+0000 7fa1377fe700 1 -- 172.21.15.102:0/3521183598 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa14c11bb60 con 0x7fa14c096a30 2021-05-25T12:06:56.453 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.409+0000 7fa1377fe700 1 -- 172.21.15.102:0/3521183598 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1170650543 0 0) 0x7fa12c004c10 con 0x7fa14c096630 2021-05-25T12:06:56.453 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.409+0000 7fa1377fe700 1 -- 172.21.15.102:0/3521183598 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa14c11be00 con 0x7fa14c096630 2021-05-25T12:06:56.454 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.409+0000 7fa1377fe700 1 -- 172.21.15.102:0/3521183598 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fa13c001880 con 0x7fa14c096630 2021-05-25T12:06:56.454 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.409+0000 7fa1377fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:56.454 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.409+0000 7fa1377fe700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:06:56.454 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.409+0000 7fa1377fe700 10 monclient(hunting): dump: 2021-05-25T12:06:56.454 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:56.455 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:56.455 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:56.455 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:56.455 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:56.455 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:56.456 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:56.456 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:56.456 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:56.456 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:56.456 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.409+0000 7fa1377fe700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:06:56.457 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.409+0000 7fa1377fe700 1 -- 172.21.15.102:0/3521183598 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2455711424 0 0) 0x7fa14c11c0a0 con 0x7fa14c093bd0 2021-05-25T12:06:56.457 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.409+0000 7fa1377fe700 1 -- 172.21.15.102:0/3521183598 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa12c004c10 con 0x7fa14c093bd0 2021-05-25T12:06:56.457 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.409+0000 7fa1377fe700 1 -- 172.21.15.102:0/3521183598 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fa138001b50 con 0x7fa14c093bd0 2021-05-25T12:06:56.457 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.409+0000 7fa1377fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:56.458 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.409+0000 7fa1377fe700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2021-05-25T12:06:56.458 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.409+0000 7fa1377fe700 10 monclient(hunting): dump: 2021-05-25T12:06:56.458 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:56.458 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:56.458 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:56.459 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:56.459 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:56.459 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:56.459 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:56.459 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:56.460 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:56.460 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:56.460 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.409+0000 7fa1377fe700 1 -- 172.21.15.102:0/3521183598 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2882262108 0 0) 0x7fa14c11be00 con 0x7fa14c096630 2021-05-25T12:06:56.460 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.409+0000 7fa1377fe700 1 -- 172.21.15.102:0/3521183598 >> v1:172.21.15.181:6789/0 conn(0x7fa14c096a30 legacy=0x7fa14c143000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:56.460 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.409+0000 7fa1377fe700 1 -- 172.21.15.102:0/3521183598 >> v1:172.21.15.102:6789/0 conn(0x7fa14c093bd0 legacy=0x7fa14c13acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:56.461 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.409+0000 7fa1377fe700 10 monclient: _finish_hunting 0 2021-05-25T12:06:56.461 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.409+0000 7fa1377fe700 1 monclient: found mon.c 2021-05-25T12:06:56.461 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.409+0000 7fa1377fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:06:56.461 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.409+0000 7fa1377fe700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:06:56.462 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.409+0000 7fa1377fe700 1 -- 172.21.15.102:0/3521183598 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa14c119fa0 con 0x7fa14c096630 2021-05-25T12:06:56.462 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.409+0000 7fa1377fe700 10 monclient: _finish_auth 0 2021-05-25T12:06:56.462 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.409+0000 7fa1377fe700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:06:56.462 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.409+0000 7fa151b84700 5 monclient: authenticate success, global_id 184619 2021-05-25T12:06:56.462 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.409+0000 7fa151b84700 10 monclient: _renew_subs 2021-05-25T12:06:56.463 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.409+0000 7fa151b84700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:06:56.463 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.409+0000 7fa151b84700 1 -- 172.21.15.102:0/3521183598 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa14c11abd0 con 0x7fa14c096630 2021-05-25T12:06:56.463 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.409+0000 7fa151b84700 10 client.184619.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:06:56.463 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.409+0000 7fa151b84700 10 monclient: _renew_subs 2021-05-25T12:06:56.464 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.409+0000 7fa151b84700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:06:56.464 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.409+0000 7fa151b84700 1 -- 172.21.15.102:0/3521183598 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa14c134be0 con 0x7fa14c096630 2021-05-25T12:06:56.464 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.409+0000 7fa151b84700 1 librados: init done 2021-05-25T12:06:56.464 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.410+0000 7fa1377fe700 1 -- 172.21.15.102:0/3521183598 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa13c001aa0 con 0x7fa14c096630 2021-05-25T12:06:56.464 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.410+0000 7fa1377fe700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:06:56.465 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.410+0000 7fa1377fe700 1 -- 172.21.15.102:0/3521183598 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fa13c001db0 con 0x7fa14c096630 2021-05-25T12:06:56.465 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.410+0000 7fa1377fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:56.465 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.410+0000 7fa151b84700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:06:56.465 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.410+0000 7fa1377fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:06:56.465 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.410+0000 7fa1377fe700 10 monclient: dump: 2021-05-25T12:06:56.466 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:56.466 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:56.466 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:56.466 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:56.467 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:56.467 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:56.467 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:56.467 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:56.467 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:56.468 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:56.468 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.410+0000 7fa149ffb700 4 set_mon_vals no callback set 2021-05-25T12:06:56.468 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.410+0000 7fa1377fe700 1 -- 172.21.15.102:0/3521183598 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7fa13c001f10 con 0x7fa14c096630 2021-05-25T12:06:56.468 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.410+0000 7fa1377fe700 1 -- 172.21.15.102:0/3521183598 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(220..220 src has 1..220) v4 ==== 6648+0+0 (unknown 208579584 0 0) 0x7fa13c03b2e0 con 0x7fa14c096630 2021-05-25T12:06:56.468 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.410+0000 7fa1377fe700 10 client.184619.objecter ms_dispatch 0x7fa14c001550 osd_map(220..220 src has 1..220) v4 2021-05-25T12:06:56.469 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.410+0000 7fa151b84700 1 -- 172.21.15.102:0/3521183598 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fa14c059730 con 0x7fa14c090990 2021-05-25T12:06:56.469 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.410+0000 7fa1377fe700 3 client.184619.objecter handle_osd_map got epochs [220,220] > 0 2021-05-25T12:06:56.469 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.410+0000 7fa1377fe700 3 client.184619.objecter handle_osd_map decoding full epoch 220 2021-05-25T12:06:56.469 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.410+0000 7fa1377fe700 20 client.184619.objecter dump_active .. 0 homeless 2021-05-25T12:06:56.469 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.410+0000 7fa1377fe700 10 client.184619.objecter ms_handle_connect 0x7fa14c090990 2021-05-25T12:06:56.470 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.410+0000 7fa1377fe700 10 client.184619.objecter resend_mon_ops 2021-05-25T12:06:56.470 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.410+0000 7fa14b7fe700 10 monclient: get_auth_request con 0x7fa12c0334d0 auth_method 0 2021-05-25T12:06:56.470 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.410+0000 7fa1377fe700 1 -- 172.21.15.102:0/3521183598 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1732940885 0 0) 0x7fa14c059730 con 0x7fa14c090990 2021-05-25T12:06:56.470 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.410+0000 7fa1377fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:56.471 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.410+0000 7fa1377fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:56.471 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.410+0000 7fa1377fe700 10 monclient(hunting): my global_id is 204282 2021-05-25T12:06:56.471 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.410+0000 7fa1377fe700 1 -- 172.21.15.102:0/3521183598 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa14c11be00 con 0x7fa14c090990 2021-05-25T12:06:56.471 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.411+0000 7fa1377fe700 1 -- 172.21.15.102:0/3521183598 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1703509043 0 0) 0x7fa14c11be00 con 0x7fa14c090990 2021-05-25T12:06:56.471 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.411+0000 7fa1377fe700 1 -- 172.21.15.102:0/3521183598 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa14c059730 con 0x7fa14c090990 2021-05-25T12:06:56.472 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.411+0000 7fa1377fe700 1 -- 172.21.15.102:0/3521183598 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fa1400019d0 con 0x7fa14c090990 2021-05-25T12:06:56.472 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.411+0000 7fa1377fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:56.472 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.411+0000 7fa1377fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:06:56.472 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.411+0000 7fa1377fe700 10 monclient: dump: 2021-05-25T12:06:56.472 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:56.473 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:56.473 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:56.473 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:56.473 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:56.474 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:56.474 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:56.474 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:56.474 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:56.474 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:56.475 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.411+0000 7fa1377fe700 10 client.184619.objecter ms_handle_connect 0x7fa12c0334d0 2021-05-25T12:06:56.475 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.411+0000 7fa1377fe700 1 -- 172.21.15.102:0/3521183598 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3173980584 0 0) 0x7fa14c059730 con 0x7fa14c090990 2021-05-25T12:06:56.475 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.411+0000 7fa1377fe700 1 -- 172.21.15.102:0/3521183598 --> v1:172.21.15.102:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa14c11adc0 con 0x7fa14c090990 2021-05-25T12:06:56.475 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.411+0000 7fa1377fe700 1 -- 172.21.15.102:0/3521183598 <== mon.0 v1:172.21.15.102:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7fa140002010 con 0x7fa14c090990 2021-05-25T12:06:56.476 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.411+0000 7fa1377fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:06:56.476 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.411+0000 7fa1377fe700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:06:56.476 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.411+0000 7fa1377fe700 1 -- 172.21.15.102:0/3521183598 >> v1:172.21.15.102:6789/0 conn(0x7fa14c090990 legacy=0x7fa14c090d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:56.476 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.411+0000 7fa1377fe700 1 -- 172.21.15.102:0/3521183598 >> v1:172.21.15.102:6789/0 conn(0x7fa14c090990 legacy=0x7fa14c090d70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:06:56.476 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.418+0000 7fa151b84700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:06:56.477 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.418+0000 7fa151b84700 1 -- 172.21.15.102:0/3521183598 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fa14c135740 con 0x7fa14c027ce0 2021-05-25T12:06:56.477 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.419+0000 7fa1377fe700 10 client.184619.objecter ms_handle_connect 0x7fa14c027ce0 2021-05-25T12:06:56.477 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.419+0000 7fa1377fe700 10 client.184619.objecter resend_mon_ops 2021-05-25T12:06:56.477 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.419+0000 7fa1377fe700 1 -- 172.21.15.102:0/3521183598 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2080971089 0 0) 0x7fa14c135740 con 0x7fa14c027ce0 2021-05-25T12:06:56.477 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.419+0000 7fa1377fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:56.478 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.419+0000 7fa1377fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:56.478 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.419+0000 7fa1377fe700 10 monclient(hunting): my global_id is 204285 2021-05-25T12:06:56.478 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.419+0000 7fa1377fe700 1 -- 172.21.15.102:0/3521183598 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa14c059730 con 0x7fa14c027ce0 2021-05-25T12:06:56.478 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.419+0000 7fa1377fe700 1 -- 172.21.15.102:0/3521183598 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1712527787 0 0) 0x7fa14c059730 con 0x7fa14c027ce0 2021-05-25T12:06:56.479 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.419+0000 7fa1377fe700 1 -- 172.21.15.102:0/3521183598 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa14c135740 con 0x7fa14c027ce0 2021-05-25T12:06:56.479 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.419+0000 7fa1377fe700 1 -- 172.21.15.102:0/3521183598 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fa140004ce0 con 0x7fa14c027ce0 2021-05-25T12:06:56.479 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.419+0000 7fa1377fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:56.479 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.419+0000 7fa1377fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:06:56.479 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.419+0000 7fa1377fe700 10 monclient: dump: 2021-05-25T12:06:56.480 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:56.480 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:56.480 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:56.480 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:56.481 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:56.481 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:56.481 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:56.481 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:56.481 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:56.482 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:56.482 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.419+0000 7fa1377fe700 1 -- 172.21.15.102:0/3521183598 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3585814307 0 0) 0x7fa14c135740 con 0x7fa14c027ce0 2021-05-25T12:06:56.482 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.419+0000 7fa1377fe700 1 -- 172.21.15.102:0/3521183598 --> v1:172.21.15.102:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa14c06c9a0 con 0x7fa14c027ce0 2021-05-25T12:06:56.482 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.419+0000 7fa1377fe700 1 -- 172.21.15.102:0/3521183598 <== mon.0 v1:172.21.15.102:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3940 (unknown 0 0 2112268476) 0x7fa140005010 con 0x7fa14c027ce0 2021-05-25T12:06:56.483 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.419+0000 7fa1377fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:06:56.483 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.419+0000 7fa1377fe700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:06:56.483 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.419+0000 7fa1377fe700 1 -- 172.21.15.102:0/3521183598 >> v1:172.21.15.102:6789/0 conn(0x7fa14c027ce0 legacy=0x7fa14c148740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:56.483 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.419+0000 7fa1377fe700 1 -- 172.21.15.102:0/3521183598 >> v1:172.21.15.102:6789/0 conn(0x7fa14c027ce0 legacy=0x7fa14c148740 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:06:56.484 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:06:56.484 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:06:56.484 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:06:56.484 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "peon", 2021-05-25T12:06:56.485 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 58, 2021-05-25T12:06:56.485 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:06:56.485 INFO:teuthology.orchestra.run.smithi102.stdout: 0, 2021-05-25T12:06:56.485 INFO:teuthology.orchestra.run.smithi102.stdout: 1, 2021-05-25T12:06:56.486 INFO:teuthology.orchestra.run.smithi102.stdout: 2 2021-05-25T12:06:56.486 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:06:56.486 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 2, 2021-05-25T12:06:56.486 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:06:56.486 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:06:56.487 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:06:56.487 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:06:56.487 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:06:56.487 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:06:56.487 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:06:56.488 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:06:56.488 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:06:56.488 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:06:56.488 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:06:56.489 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:06:56.489 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:06:56.489 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:06:56.489 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:06:56.490 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:06:56.490 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:06:56.490 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:06:56.490 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:06:56.490 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:06:56.491 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:06:56.491 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:06:56.491 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:06:56.491 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:06:56.492 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:56.492 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:56.492 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:06:56.492 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:06:56.492 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:06:56.493 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:06:56.493 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:06:56.493 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:06:56.493 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:06:56.493 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:06:56.494 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:06:56.494 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:06:56.494 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:06:56.494 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:06:56.495 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:06:56.495 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:06:56.495 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:06:56.495 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:06:56.495 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:06:56.496 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:06:56.496 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:06:56.496 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:06:56.496 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:06:56.496 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:06:56.497 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:06:56.497 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:06:56.497 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:06:56.497 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:06:56.498 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:56.498 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:06:56.498 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:56.498 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:06:56.499 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:06:56.499 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:06:56.499 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:06:56.499 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:56.499 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:06:56.500 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:06:56.500 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:06:56.500 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:56.500 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:56.501 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:56.501 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:06:56.501 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:06:56.501 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:06:56.501 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:06:56.502 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:06:56.502 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:56.502 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:56.502 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:06:56.503 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:06:56.503 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:06:56.503 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:06:56.503 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:56.503 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:06:56.504 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:06:56.504 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:06:56.504 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:56.504 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:56.504 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:56.505 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:06:56.505 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:06:56.505 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:06:56.505 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:06:56.505 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:06:56.506 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:56.506 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:56.506 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:06:56.506 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:06:56.507 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:06:56.507 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:06:56.507 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:56.507 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:06:56.507 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:06:56.508 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:06:56.508 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:56.508 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:56.508 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:56.508 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:06:56.509 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:06:56.509 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:06:56.509 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:06:56.509 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:06:56.509 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:56.510 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:56.510 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:56.510 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:06:56.510 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:06:56.510 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:56.511 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:06:56.511 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:06:56.511 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:06:56.511 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:56.511 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:06:56.512 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": [ 2021-05-25T12:06:56.512 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:56.512 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:06:56.512 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:06:56.512 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 2 2021-05-25T12:06:56.513 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:56.513 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:06:56.513 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:06:56.513 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:56.513 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:06:56.514 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:06:56.514 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 2 2021-05-25T12:06:56.514 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:56.514 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:56.515 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:56.515 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:06:56.515 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:06:56.518 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.420+0000 7fa151b84700 10 librados: watch_flush enter 2021-05-25T12:06:56.518 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.420+0000 7fa151b84700 10 librados: watch_flush exit 2021-05-25T12:06:56.519 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.420+0000 7fa151b84700 20 client.184619.objecter shutdown clearing up homeless session... 2021-05-25T12:06:56.519 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.420+0000 7fa151b84700 10 client.184619.objecter successfully canceled tick 2021-05-25T12:06:56.519 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.420+0000 7fa151b84700 1 -- 172.21.15.102:0/3521183598 >> v1:172.21.15.181:6812/29582 conn(0x7fa12c0334d0 legacy=0x7fa12c035990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:56.519 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.420+0000 7fa151b84700 10 monclient: shutdown 2021-05-25T12:06:56.519 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.420+0000 7fa151b84700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:06:56.520 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.420+0000 7fa151b84700 1 -- 172.21.15.102:0/3521183598 >> v1:172.21.15.102:6790/0 conn(0x7fa14c096630 legacy=0x7fa14c13b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:56.520 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.420+0000 7fa14bfff700 1 -- 172.21.15.102:0/3521183598 reap_dead start 2021-05-25T12:06:56.520 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.420+0000 7fa151b84700 1 -- 172.21.15.102:0/3521183598 shutdown_connections 2021-05-25T12:06:56.520 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.420+0000 7fa151b84700 1 -- 172.21.15.102:0/3521183598 >> 172.21.15.102:0/3521183598 conn(0x7fa14c09cd10 msgr2=0x7fa14c068140 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:06:56.520 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.420+0000 7fa151b84700 1 -- 172.21.15.102:0/3521183598 shutdown_connections 2021-05-25T12:06:56.521 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.420+0000 7fa151b84700 1 -- 172.21.15.102:0/3521183598 wait complete. 2021-05-25T12:06:56.521 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.420+0000 7fa151b84700 1 librados: shutdown 2021-05-25T12:06:56.521 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2021-05-25T12:06:56.632 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.631+0000 7fbb800be700 10 monclient: get_monmap_and_config 2021-05-25T12:06:56.633 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.631+0000 7fbb800be700 10 monclient: build_initial_monmap 2021-05-25T12:06:56.633 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.631+0000 7fbb800be700 10 monclient: monmap: 2021-05-25T12:06:56.633 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:06:56.633 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:56.634 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:06:56.632039+0000 2021-05-25T12:06:56.634 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:06:56.632039+0000 2021-05-25T12:06:56.634 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:06:56.634 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:56.635 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:06:56.635 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:06:56.635 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:06:56.635 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:56.635 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.631+0000 7fbb800be700 1 Processor -- start 2021-05-25T12:06:56.636 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.631+0000 7fbb800be700 1 -- start start 2021-05-25T12:06:56.636 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.631+0000 7fbb800be700 10 monclient: init 2021-05-25T12:06:56.650 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.632+0000 7fbb800be700 10 monclient: _reopen_session rank -1 2021-05-25T12:06:56.650 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.632+0000 7fbb800be700 10 monclient: _add_conns ranks=[1,0,2] 2021-05-25T12:06:56.650 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.632+0000 7fbb800be700 10 monclient(hunting): picked mon.noname-c con 0x7fbb78090990 addr v1:172.21.15.181:6789/0 2021-05-25T12:06:56.651 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.632+0000 7fbb800be700 10 monclient(hunting): picked mon.noname-a con 0x7fbb78090d70 addr v1:172.21.15.102:6789/0 2021-05-25T12:06:56.651 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.632+0000 7fbb800be700 10 monclient(hunting): picked mon.noname-b con 0x7fbb7812e330 addr v1:172.21.15.102:6790/0 2021-05-25T12:06:56.651 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.632+0000 7fbb800be700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb78096fa0 con 0x7fbb78090d70 2021-05-25T12:06:56.651 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.632+0000 7fbb800be700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb780926a0 con 0x7fbb78090990 2021-05-25T12:06:56.652 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.632+0000 7fbb800be700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb78092940 con 0x7fbb7812e330 2021-05-25T12:06:56.652 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.632+0000 7fbb800be700 10 monclient(hunting): _renew_subs 2021-05-25T12:06:56.652 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.632+0000 7fbb800be700 10 monclient(hunting): authenticate will time out at 1217.229267s 2021-05-25T12:06:56.652 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.632+0000 7fbb7e65b700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fbb7812e330 0x7fbb78130720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:56688/0 (socket says 172.21.15.102:56688) 2021-05-25T12:06:56.653 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.632+0000 7fbb7e65b700 1 -- 172.21.15.102:0/49575189 learned_addr learned my addr 172.21.15.102:0/49575189 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:06:56.653 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.632+0000 7fbb7d659700 1 --1- 172.21.15.102:0/49575189 >> v1:172.21.15.102:6789/0 conn(0x7fbb78090d70 0x7fbb78091200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33576/0 (socket says 172.21.15.102:33576) 2021-05-25T12:06:56.653 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.632+0000 7fbb7ce58700 1 -- 172.21.15.102:0/49575189 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3909554729 0 0) 0x7fbb78096fa0 con 0x7fbb78090d70 2021-05-25T12:06:56.653 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.632+0000 7fbb7ce58700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:56.654 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.632+0000 7fbb7ce58700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:56.654 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.632+0000 7fbb7ce58700 10 monclient(hunting): my global_id is 204288 2021-05-25T12:06:56.654 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.632+0000 7fbb7ce58700 1 -- 172.21.15.102:0/49575189 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb60003270 con 0x7fbb78090d70 2021-05-25T12:06:56.654 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.632+0000 7fbb7ce58700 1 -- 172.21.15.102:0/49575189 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4206404897 0 0) 0x7fbb78092940 con 0x7fbb7812e330 2021-05-25T12:06:56.654 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.632+0000 7fbb7ce58700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:56.655 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.632+0000 7fbb7ce58700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:56.655 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.632+0000 7fbb7ce58700 10 monclient(hunting): my global_id is 184622 2021-05-25T12:06:56.655 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.632+0000 7fbb7ce58700 1 -- 172.21.15.102:0/49575189 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb78096fa0 con 0x7fbb7812e330 2021-05-25T12:06:56.655 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.632+0000 7fbb7ce58700 1 -- 172.21.15.102:0/49575189 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3223372361 0 0) 0x7fbb78096fa0 con 0x7fbb7812e330 2021-05-25T12:06:56.656 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.632+0000 7fbb7ce58700 1 -- 172.21.15.102:0/49575189 >> v1:172.21.15.181:6789/0 conn(0x7fbb78090990 legacy=0x7fbb78093240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:56.656 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.632+0000 7fbb7ce58700 1 -- 172.21.15.102:0/49575189 >> v1:172.21.15.102:6789/0 conn(0x7fbb78090d70 legacy=0x7fbb78091200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:56.656 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.632+0000 7fbb7ce58700 10 monclient: _finish_hunting 0 2021-05-25T12:06:56.656 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.632+0000 7fbb7ce58700 1 monclient: found mon.noname-b 2021-05-25T12:06:56.657 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.632+0000 7fbb7ce58700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:06:56.657 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.632+0000 7fbb7ce58700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T12:06:56.657 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.632+0000 7fbb7ce58700 1 -- 172.21.15.102:0/49575189 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb78092be0 con 0x7fbb7812e330 2021-05-25T12:06:56.657 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.632+0000 7fbb7ce58700 10 monclient: _finish_auth 0 2021-05-25T12:06:56.657 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.632+0000 7fbb7ce58700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:06:56.658 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.632+0000 7fbb7ce58700 1 -- 172.21.15.102:0/49575189 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fbb74001990 con 0x7fbb7812e330 2021-05-25T12:06:56.658 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.632+0000 7fbb800be700 5 monclient: authenticate success, global_id 184622 2021-05-25T12:06:56.658 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.632+0000 7fbb800be700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:06:56.658 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.632+0000 7fbb7ce58700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:56.659 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.632+0000 7fbb7ce58700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:06:56.659 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.632+0000 7fbb7ce58700 10 monclient: dump: 2021-05-25T12:06:56.659 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:56.659 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:56.659 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:56.660 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:56.660 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:56.660 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:56.660 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:56.661 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:56.661 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:56.661 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:56.661 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.632+0000 7fbb800be700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:06:56.661 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.633+0000 7fbb7ce58700 1 -- 172.21.15.102:0/49575189 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbb74001bc0 con 0x7fbb7812e330 2021-05-25T12:06:56.662 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.633+0000 7fbb7ce58700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:06:56.662 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.633+0000 7fbb7ce58700 1 -- 172.21.15.102:0/49575189 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fbb74001f90 con 0x7fbb7812e330 2021-05-25T12:06:56.662 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.633+0000 7fbb800be700 10 monclient: get_monmap_and_config success 2021-05-25T12:06:56.662 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.633+0000 7fbb800be700 4 set_mon_vals no callback set 2021-05-25T12:06:56.663 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.633+0000 7fbb7ce58700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:56.663 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.633+0000 7fbb7ce58700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:06:56.663 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.633+0000 7fbb7ce58700 10 monclient: dump: 2021-05-25T12:06:56.663 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:56.663 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:56.664 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:56.664 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:56.664 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:56.664 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:56.665 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:56.665 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:56.665 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:56.665 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:56.665 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.633+0000 7fbb800be700 10 monclient: shutdown 2021-05-25T12:06:56.666 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.633+0000 7fbb800be700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:06:56.666 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.633+0000 7fbb800be700 1 -- 172.21.15.102:0/49575189 >> v1:172.21.15.102:6790/0 conn(0x7fbb7812e330 legacy=0x7fbb78130720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:56.666 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.633+0000 7fbb800be700 1 -- 172.21.15.102:0/49575189 shutdown_connections 2021-05-25T12:06:56.666 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.633+0000 7fbb800be700 1 -- 172.21.15.102:0/49575189 >> 172.21.15.102:0/49575189 conn(0x7fbb7809b370 msgr2=0x7fbb78097430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:06:56.667 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.633+0000 7fbb800be700 1 -- 172.21.15.102:0/49575189 shutdown_connections 2021-05-25T12:06:56.667 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.633+0000 7fbb800be700 1 -- 172.21.15.102:0/49575189 wait complete. 2021-05-25T12:06:56.667 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.633+0000 7fbb800be700 10 monclient: build_initial_monmap 2021-05-25T12:06:56.667 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.633+0000 7fbb800be700 10 monclient: monmap: 2021-05-25T12:06:56.667 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:06:56.668 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:06:56.668 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:06:56.668 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:06:56.668 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:06:56.668 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:06:56.669 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:06:56.669 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:06:56.669 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:06:56.669 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:56.669 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.634+0000 7fbb800be700 1 librados: starting msgr at 2021-05-25T12:06:56.670 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.634+0000 7fbb800be700 1 librados: starting objecter 2021-05-25T12:06:56.670 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.634+0000 7fbb800be700 1 Processor -- start 2021-05-25T12:06:56.670 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.634+0000 7fbb800be700 1 -- start start 2021-05-25T12:06:56.670 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.634+0000 7fbb800be700 1 librados: setting wanted keys 2021-05-25T12:06:56.670 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.634+0000 7fbb800be700 1 librados: calling monclient init 2021-05-25T12:06:56.671 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.634+0000 7fbb800be700 10 monclient: init 2021-05-25T12:06:56.671 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.634+0000 7fbb800be700 10 monclient: _reopen_session rank -1 2021-05-25T12:06:56.671 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.634+0000 7fbb800be700 10 monclient: _add_conns ranks=[0,2,1] 2021-05-25T12:06:56.671 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.634+0000 7fbb800be700 10 monclient(hunting): picked mon.noname-a con 0x7fbb78090990 addr v1:172.21.15.102:6789/0 2021-05-25T12:06:56.671 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.634+0000 7fbb800be700 10 monclient(hunting): picked mon.noname-c con 0x7fbb78090d70 addr v1:172.21.15.102:6790/0 2021-05-25T12:06:56.672 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.634+0000 7fbb800be700 10 monclient(hunting): picked mon.noname-b con 0x7fbb7812e330 addr v1:172.21.15.181:6789/0 2021-05-25T12:06:56.672 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.634+0000 7fbb800be700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb78091fe0 con 0x7fbb78090990 2021-05-25T12:06:56.672 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.634+0000 7fbb800be700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb78068790 con 0x7fbb7812e330 2021-05-25T12:06:56.672 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.634+0000 7fbb800be700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb78068a30 con 0x7fbb78090d70 2021-05-25T12:06:56.672 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.634+0000 7fbb800be700 10 monclient(hunting): _renew_subs 2021-05-25T12:06:56.673 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.634+0000 7fbb800be700 10 monclient(hunting): authenticate will time out at 1217.231779s 2021-05-25T12:06:56.673 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.634+0000 7fbb7d659700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fbb78090d70 0x7fbb780654c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:56692/0 (socket says 172.21.15.102:56692) 2021-05-25T12:06:56.673 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.634+0000 7fbb7d659700 1 -- 172.21.15.102:0/2364747686 learned_addr learned my addr 172.21.15.102:0/2364747686 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:06:56.673 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.634+0000 7fbb7de5a700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fbb78090990 0x7fbb78064db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33584/0 (socket says 172.21.15.102:33584) 2021-05-25T12:06:56.673 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.634+0000 7fbb65ffb700 10 client.?.objecter ms_handle_connect 0x7fbb78090d70 2021-05-25T12:06:56.674 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.634+0000 7fbb65ffb700 10 client.?.objecter resend_mon_ops 2021-05-25T12:06:56.674 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.634+0000 7fbb65ffb700 10 client.?.objecter ms_handle_connect 0x7fbb78090990 2021-05-25T12:06:56.674 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.634+0000 7fbb65ffb700 10 client.?.objecter resend_mon_ops 2021-05-25T12:06:56.674 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.635+0000 7fbb65ffb700 10 client.?.objecter ms_handle_connect 0x7fbb7812e330 2021-05-25T12:06:56.675 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.635+0000 7fbb65ffb700 10 client.?.objecter resend_mon_ops 2021-05-25T12:06:56.675 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.635+0000 7fbb65ffb700 1 -- 172.21.15.102:0/2364747686 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1999484630 0 0) 0x7fbb78091fe0 con 0x7fbb78090990 2021-05-25T12:06:56.675 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.635+0000 7fbb65ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:56.675 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.635+0000 7fbb65ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:56.675 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.635+0000 7fbb65ffb700 10 monclient(hunting): my global_id is 204291 2021-05-25T12:06:56.676 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.635+0000 7fbb65ffb700 1 -- 172.21.15.102:0/2364747686 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb58004c10 con 0x7fbb78090990 2021-05-25T12:06:56.676 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.635+0000 7fbb65ffb700 1 -- 172.21.15.102:0/2364747686 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1349042526 0 0) 0x7fbb78068a30 con 0x7fbb78090d70 2021-05-25T12:06:56.676 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.635+0000 7fbb65ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:56.676 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.635+0000 7fbb65ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:56.676 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.635+0000 7fbb65ffb700 10 monclient(hunting): my global_id is 184625 2021-05-25T12:06:56.677 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.635+0000 7fbb65ffb700 1 -- 172.21.15.102:0/2364747686 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb78091fe0 con 0x7fbb78090d70 2021-05-25T12:06:56.677 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.635+0000 7fbb65ffb700 1 -- 172.21.15.102:0/2364747686 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 450830975 0 0) 0x7fbb78068790 con 0x7fbb7812e330 2021-05-25T12:06:56.677 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.635+0000 7fbb65ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:56.677 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.635+0000 7fbb65ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:56.677 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.635+0000 7fbb65ffb700 10 monclient(hunting): my global_id is 224152 2021-05-25T12:06:56.678 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.635+0000 7fbb65ffb700 1 -- 172.21.15.102:0/2364747686 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb78068a30 con 0x7fbb7812e330 2021-05-25T12:06:56.678 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.635+0000 7fbb65ffb700 1 -- 172.21.15.102:0/2364747686 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 542606505 0 0) 0x7fbb58004c10 con 0x7fbb78090990 2021-05-25T12:06:56.678 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.635+0000 7fbb65ffb700 1 -- 172.21.15.102:0/2364747686 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbb78068790 con 0x7fbb78090990 2021-05-25T12:06:56.678 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.635+0000 7fbb65ffb700 1 -- 172.21.15.102:0/2364747686 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fbb680018c0 con 0x7fbb78090990 2021-05-25T12:06:56.678 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.635+0000 7fbb65ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:56.679 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.635+0000 7fbb65ffb700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:06:56.679 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.635+0000 7fbb65ffb700 10 monclient(hunting): dump: 2021-05-25T12:06:56.679 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:56.679 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:56.679 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:56.680 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:56.680 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:56.680 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:56.680 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:56.681 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:56.681 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:56.681 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:56.681 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.635+0000 7fbb65ffb700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:06:56.681 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.635+0000 7fbb65ffb700 1 -- 172.21.15.102:0/2364747686 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2189052398 0 0) 0x7fbb78091fe0 con 0x7fbb78090d70 2021-05-25T12:06:56.682 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.635+0000 7fbb65ffb700 1 -- 172.21.15.102:0/2364747686 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbb58004c10 con 0x7fbb78090d70 2021-05-25T12:06:56.682 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.635+0000 7fbb65ffb700 1 -- 172.21.15.102:0/2364747686 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fbb6c0019f0 con 0x7fbb78090d70 2021-05-25T12:06:56.682 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.635+0000 7fbb65ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:56.682 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.635+0000 7fbb65ffb700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2021-05-25T12:06:56.682 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.635+0000 7fbb65ffb700 10 monclient(hunting): dump: 2021-05-25T12:06:56.683 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:56.683 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:56.683 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:56.683 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:56.683 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:56.684 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:56.684 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:56.684 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:56.684 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:56.684 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:56.685 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.635+0000 7fbb65ffb700 1 -- 172.21.15.102:0/2364747686 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2771600666 0 0) 0x7fbb78068a30 con 0x7fbb7812e330 2021-05-25T12:06:56.685 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.635+0000 7fbb65ffb700 1 -- 172.21.15.102:0/2364747686 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbb78091fe0 con 0x7fbb7812e330 2021-05-25T12:06:56.685 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.635+0000 7fbb65ffb700 1 -- 172.21.15.102:0/2364747686 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fbb74001ed0 con 0x7fbb7812e330 2021-05-25T12:06:56.685 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.635+0000 7fbb65ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:56.686 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.635+0000 7fbb65ffb700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2021-05-25T12:06:56.686 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.635+0000 7fbb65ffb700 10 monclient(hunting): dump: 2021-05-25T12:06:56.686 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:56.686 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:56.687 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:56.687 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:56.687 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:56.687 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:56.688 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:56.688 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:56.688 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:56.688 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:56.688 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.635+0000 7fbb65ffb700 1 -- 172.21.15.102:0/2364747686 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2452566694 0 0) 0x7fbb78068790 con 0x7fbb78090990 2021-05-25T12:06:56.689 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.635+0000 7fbb65ffb700 1 -- 172.21.15.102:0/2364747686 >> v1:172.21.15.102:6790/0 conn(0x7fbb78090d70 legacy=0x7fbb780654c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:56.689 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.635+0000 7fbb65ffb700 1 -- 172.21.15.102:0/2364747686 >> v1:172.21.15.181:6789/0 conn(0x7fbb7812e330 legacy=0x7fbb78065bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:56.689 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.635+0000 7fbb65ffb700 10 monclient: _finish_hunting 0 2021-05-25T12:06:56.689 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.635+0000 7fbb65ffb700 1 monclient: found mon.a 2021-05-25T12:06:56.689 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.635+0000 7fbb65ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:06:56.690 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.635+0000 7fbb65ffb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:06:56.690 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.635+0000 7fbb65ffb700 1 -- 172.21.15.102:0/2364747686 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb78067f20 con 0x7fbb78090990 2021-05-25T12:06:56.690 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.635+0000 7fbb65ffb700 10 monclient: _finish_auth 0 2021-05-25T12:06:56.690 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.635+0000 7fbb65ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:06:56.691 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.635+0000 7fbb800be700 5 monclient: authenticate success, global_id 204291 2021-05-25T12:06:56.691 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.635+0000 7fbb800be700 10 monclient: _renew_subs 2021-05-25T12:06:56.691 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.635+0000 7fbb800be700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:06:56.691 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.635+0000 7fbb800be700 1 -- 172.21.15.102:0/2364747686 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb781147a0 con 0x7fbb78090990 2021-05-25T12:06:56.691 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.635+0000 7fbb65ffb700 1 -- 172.21.15.102:0/2364747686 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbb68001eb0 con 0x7fbb78090990 2021-05-25T12:06:56.692 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.635+0000 7fbb65ffb700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:06:56.692 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.635+0000 7fbb65ffb700 1 -- 172.21.15.102:0/2364747686 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fbb680020a0 con 0x7fbb78090990 2021-05-25T12:06:56.692 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.635+0000 7fbb65ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:56.692 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.635+0000 7fbb65ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:06:56.693 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.635+0000 7fbb65ffb700 10 monclient: dump: 2021-05-25T12:06:56.693 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:56.693 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:56.693 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:56.694 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:56.694 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:56.694 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:56.694 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:56.694 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:56.695 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:56.695 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:56.695 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.635+0000 7fbb800be700 10 client.204291.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:06:56.695 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.635+0000 7fbb67fff700 4 set_mon_vals no callback set 2021-05-25T12:06:56.695 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.635+0000 7fbb800be700 10 monclient: _renew_subs 2021-05-25T12:06:56.696 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.635+0000 7fbb800be700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:06:56.696 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.635+0000 7fbb800be700 1 -- 172.21.15.102:0/2364747686 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb78148880 con 0x7fbb78090990 2021-05-25T12:06:56.696 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.635+0000 7fbb800be700 1 librados: init done 2021-05-25T12:06:56.696 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.636+0000 7fbb800be700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:06:56.697 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.636+0000 7fbb800be700 1 -- 172.21.15.102:0/2364747686 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fbb78059730 con 0x7fbb7812a640 2021-05-25T12:06:56.697 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.636+0000 7fbb65ffb700 1 -- 172.21.15.102:0/2364747686 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7fbb68002280 con 0x7fbb78090990 2021-05-25T12:06:56.697 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.636+0000 7fbb65ffb700 10 client.204291.objecter ms_handle_connect 0x7fbb7812a640 2021-05-25T12:06:56.697 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.636+0000 7fbb65ffb700 10 client.204291.objecter resend_mon_ops 2021-05-25T12:06:56.697 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.636+0000 7fbb65ffb700 1 -- 172.21.15.102:0/2364747686 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(220..220 src has 1..220) v4 ==== 6648+0+0 (unknown 208579584 0 0) 0x7fbb6803d750 con 0x7fbb78090990 2021-05-25T12:06:56.698 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.636+0000 7fbb65ffb700 10 client.204291.objecter ms_dispatch 0x7fbb78001550 osd_map(220..220 src has 1..220) v4 2021-05-25T12:06:56.698 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.636+0000 7fbb65ffb700 3 client.204291.objecter handle_osd_map got epochs [220,220] > 0 2021-05-25T12:06:56.698 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.636+0000 7fbb65ffb700 3 client.204291.objecter handle_osd_map decoding full epoch 220 2021-05-25T12:06:56.698 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.636+0000 7fbb65ffb700 20 client.204291.objecter dump_active .. 0 homeless 2021-05-25T12:06:56.698 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.636+0000 7fbb65ffb700 1 -- 172.21.15.102:0/2364747686 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 241894743 0 0) 0x7fbb78059730 con 0x7fbb7812a640 2021-05-25T12:06:56.699 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.636+0000 7fbb65ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:56.699 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.636+0000 7fbb65ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:56.699 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.636+0000 7fbb65ffb700 10 monclient(hunting): my global_id is 184628 2021-05-25T12:06:56.699 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.636+0000 7fbb65ffb700 1 -- 172.21.15.102:0/2364747686 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb78068790 con 0x7fbb7812a640 2021-05-25T12:06:56.699 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.636+0000 7fbb7e65b700 10 monclient: get_auth_request con 0x7fbb58033540 auth_method 0 2021-05-25T12:06:56.700 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.636+0000 7fbb65ffb700 1 -- 172.21.15.102:0/2364747686 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3415097200 0 0) 0x7fbb78068790 con 0x7fbb7812a640 2021-05-25T12:06:56.700 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.636+0000 7fbb65ffb700 1 -- 172.21.15.102:0/2364747686 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbb78059730 con 0x7fbb7812a640 2021-05-25T12:06:56.700 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.636+0000 7fbb65ffb700 1 -- 172.21.15.102:0/2364747686 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fbb6c004940 con 0x7fbb7812a640 2021-05-25T12:06:56.700 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.637+0000 7fbb65ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:56.700 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.637+0000 7fbb65ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:06:56.701 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.637+0000 7fbb65ffb700 10 monclient: dump: 2021-05-25T12:06:56.701 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:56.701 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:56.701 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:56.701 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:56.702 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:56.702 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:56.702 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:56.702 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:56.703 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:56.703 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:56.703 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.637+0000 7fbb65ffb700 10 client.204291.objecter ms_handle_connect 0x7fbb58033540 2021-05-25T12:06:56.703 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.637+0000 7fbb65ffb700 1 -- 172.21.15.102:0/2364747686 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 626414181 0 0) 0x7fbb78059730 con 0x7fbb7812a640 2021-05-25T12:06:56.703 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.637+0000 7fbb65ffb700 1 -- 172.21.15.102:0/2364747686 --> v1:172.21.15.102:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbb78096500 con 0x7fbb7812a640 2021-05-25T12:06:56.704 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.637+0000 7fbb65ffb700 1 -- 172.21.15.102:0/2364747686 <== mon.2 v1:172.21.15.102:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7fbb6c004de0 con 0x7fbb7812a640 2021-05-25T12:06:56.704 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.637+0000 7fbb65ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:06:56.704 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.637+0000 7fbb65ffb700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:06:56.704 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.637+0000 7fbb65ffb700 1 -- 172.21.15.102:0/2364747686 >> v1:172.21.15.102:6790/0 conn(0x7fbb7812a640 legacy=0x7fbb7812aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:56.704 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.637+0000 7fbb65ffb700 1 -- 172.21.15.102:0/2364747686 >> v1:172.21.15.102:6790/0 conn(0x7fbb7812a640 legacy=0x7fbb7812aa20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:06:56.705 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.644+0000 7fbb800be700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:06:56.705 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.644+0000 7fbb800be700 1 -- 172.21.15.102:0/2364747686 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fbb780668b0 con 0x7fbb78027ce0 2021-05-25T12:06:56.705 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.644+0000 7fbb65ffb700 10 client.204291.objecter ms_handle_connect 0x7fbb78027ce0 2021-05-25T12:06:56.705 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.644+0000 7fbb65ffb700 10 client.204291.objecter resend_mon_ops 2021-05-25T12:06:56.706 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.644+0000 7fbb65ffb700 1 -- 172.21.15.102:0/2364747686 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2531776484 0 0) 0x7fbb780668b0 con 0x7fbb78027ce0 2021-05-25T12:06:56.706 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.644+0000 7fbb65ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:56.706 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.644+0000 7fbb65ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:56.706 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.644+0000 7fbb65ffb700 10 monclient(hunting): my global_id is 184631 2021-05-25T12:06:56.707 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.644+0000 7fbb65ffb700 1 -- 172.21.15.102:0/2364747686 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb78059730 con 0x7fbb78027ce0 2021-05-25T12:06:56.707 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.644+0000 7fbb65ffb700 1 -- 172.21.15.102:0/2364747686 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 143590586 0 0) 0x7fbb78059730 con 0x7fbb78027ce0 2021-05-25T12:06:56.707 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.644+0000 7fbb65ffb700 1 -- 172.21.15.102:0/2364747686 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbb780668b0 con 0x7fbb78027ce0 2021-05-25T12:06:56.707 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.644+0000 7fbb65ffb700 1 -- 172.21.15.102:0/2364747686 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fbb6c0054d0 con 0x7fbb78027ce0 2021-05-25T12:06:56.707 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.644+0000 7fbb65ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:56.708 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.644+0000 7fbb65ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:06:56.708 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.644+0000 7fbb65ffb700 10 monclient: dump: 2021-05-25T12:06:56.708 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:56.708 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:56.709 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:56.709 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:56.709 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:56.709 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:56.709 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:56.710 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:56.710 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:56.710 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:56.710 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.644+0000 7fbb65ffb700 1 -- 172.21.15.102:0/2364747686 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3165352234 0 0) 0x7fbb780668b0 con 0x7fbb78027ce0 2021-05-25T12:06:56.710 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.644+0000 7fbb65ffb700 1 -- 172.21.15.102:0/2364747686 --> v1:172.21.15.102:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fbb7805d870 con 0x7fbb78027ce0 2021-05-25T12:06:56.711 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.645+0000 7fbb65ffb700 1 -- 172.21.15.102:0/2364747686 <== mon.2 v1:172.21.15.102:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4112 (unknown 0 0 224341159) 0x7fbb6c0068a0 con 0x7fbb78027ce0 2021-05-25T12:06:56.711 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.645+0000 7fbb65ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:06:56.711 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.645+0000 7fbb65ffb700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:06:56.711 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.645+0000 7fbb65ffb700 1 -- 172.21.15.102:0/2364747686 >> v1:172.21.15.102:6790/0 conn(0x7fbb78027ce0 legacy=0x7fbb7814ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:56.712 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.645+0000 7fbb65ffb700 1 -- 172.21.15.102:0/2364747686 >> v1:172.21.15.102:6790/0 conn(0x7fbb78027ce0 legacy=0x7fbb7814ca30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:06:56.712 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:06:56.712 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:06:56.713 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:06:56.713 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "leader", 2021-05-25T12:06:56.713 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 58, 2021-05-25T12:06:56.713 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:06:56.713 INFO:teuthology.orchestra.run.smithi102.stdout: 0, 2021-05-25T12:06:56.714 INFO:teuthology.orchestra.run.smithi102.stdout: 1, 2021-05-25T12:06:56.714 INFO:teuthology.orchestra.run.smithi102.stdout: 2 2021-05-25T12:06:56.714 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:06:56.714 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 3, 2021-05-25T12:06:56.715 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:06:56.715 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:06:56.715 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:06:56.715 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:06:56.716 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:06:56.716 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:06:56.716 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:06:56.716 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:06:56.716 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:06:56.717 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:06:56.717 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:06:56.717 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:06:56.717 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:06:56.717 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:06:56.718 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:06:56.718 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:06:56.718 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:06:56.718 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:06:56.719 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:06:56.719 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:06:56.719 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:06:56.719 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:06:56.720 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:06:56.720 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:06:56.720 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:56.720 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:56.720 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:06:56.721 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:06:56.721 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:06:56.721 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:06:56.721 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:06:56.722 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:06:56.722 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:06:56.722 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:06:56.722 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:06:56.723 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:06:56.723 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:06:56.723 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:06:56.723 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:06:56.723 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:06:56.724 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:06:56.724 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:06:56.724 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:06:56.724 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:06:56.724 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:06:56.724 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:06:56.725 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:06:56.725 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:06:56.725 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:06:56.725 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:06:56.725 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:06:56.726 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:06:56.726 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:56.726 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:06:56.726 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:56.727 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:06:56.727 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:06:56.727 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:06:56.727 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:06:56.727 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:56.728 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:06:56.728 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:06:56.728 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:06:56.728 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:56.728 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:56.729 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:56.729 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:06:56.729 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:06:56.729 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:06:56.729 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:06:56.730 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:06:56.730 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:56.730 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:56.730 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:06:56.730 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:06:56.731 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:06:56.731 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:06:56.731 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:56.731 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:06:56.731 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:06:56.732 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:06:56.732 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:56.732 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:56.732 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:56.732 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:06:56.733 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:06:56.733 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:06:56.733 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:06:56.733 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:06:56.733 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:56.734 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:56.734 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:06:56.734 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:06:56.734 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:06:56.735 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:06:56.735 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:56.735 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:06:56.735 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:06:56.735 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:06:56.736 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:56.736 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:56.736 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:56.736 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:06:56.736 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:06:56.737 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:06:56.737 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:06:56.737 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:06:56.737 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:56.737 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:56.738 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:56.738 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:06:56.738 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:06:56.738 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:56.739 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:06:56.739 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:06:56.739 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:06:56.739 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:56.739 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:06:56.740 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": [ 2021-05-25T12:06:56.740 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:56.740 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:06:56.740 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:06:56.740 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 4 2021-05-25T12:06:56.741 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:56.741 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:06:56.741 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:06:56.741 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:56.741 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:06:56.742 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:06:56.742 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 5 2021-05-25T12:06:56.742 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:56.742 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:06:56.742 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": [ 2021-05-25T12:06:56.743 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:56.743 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:06:56.743 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:06:56.743 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:06:56.743 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:56.744 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:56.744 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:56.744 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:06:56.744 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:06:56.746 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.645+0000 7fbb800be700 10 librados: watch_flush enter 2021-05-25T12:06:56.746 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.645+0000 7fbb800be700 10 librados: watch_flush exit 2021-05-25T12:06:56.747 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.645+0000 7fbb800be700 20 client.204291.objecter shutdown clearing up homeless session... 2021-05-25T12:06:56.747 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.645+0000 7fbb800be700 10 client.204291.objecter successfully canceled tick 2021-05-25T12:06:56.747 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.645+0000 7fbb800be700 1 -- 172.21.15.102:0/2364747686 >> v1:172.21.15.181:6812/29582 conn(0x7fbb58033540 legacy=0x7fbb58035a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:56.747 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.645+0000 7fbb800be700 10 monclient: shutdown 2021-05-25T12:06:56.747 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.645+0000 7fbb800be700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:06:56.748 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.645+0000 7fbb800be700 1 -- 172.21.15.102:0/2364747686 >> v1:172.21.15.102:6789/0 conn(0x7fbb78090990 legacy=0x7fbb78064db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:56.748 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.645+0000 7fbb7e65b700 1 -- 172.21.15.102:0/2364747686 reap_dead start 2021-05-25T12:06:56.748 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.645+0000 7fbb7e65b700 1 -- 172.21.15.102:0/2364747686 reap_dead start 2021-05-25T12:06:56.748 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.645+0000 7fbb800be700 1 -- 172.21.15.102:0/2364747686 shutdown_connections 2021-05-25T12:06:56.748 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.645+0000 7fbb800be700 1 -- 172.21.15.102:0/2364747686 >> 172.21.15.102:0/2364747686 conn(0x7fbb7809b370 msgr2=0x7fbb781292f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:06:56.749 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.645+0000 7fbb800be700 1 -- 172.21.15.102:0/2364747686 shutdown_connections 2021-05-25T12:06:56.749 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.645+0000 7fbb800be700 1 -- 172.21.15.102:0/2364747686 wait complete. 2021-05-25T12:06:56.749 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.645+0000 7fbb800be700 1 librados: shutdown 2021-05-25T12:06:56.750 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2021-05-25T12:06:56.865 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.864+0000 7f1fd0eeb700 10 monclient: get_monmap_and_config 2021-05-25T12:06:56.866 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.864+0000 7f1fd0eeb700 10 monclient: build_initial_monmap 2021-05-25T12:06:56.866 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.864+0000 7f1fd0eeb700 10 monclient: monmap: 2021-05-25T12:06:56.866 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:06:56.866 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:56.867 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:06:56.865527+0000 2021-05-25T12:06:56.867 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:06:56.865527+0000 2021-05-25T12:06:56.867 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:06:56.868 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:56.868 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:06:56.868 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:06:56.868 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:06:56.869 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:56.882 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.865+0000 7f1fd0eeb700 1 Processor -- start 2021-05-25T12:06:56.882 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.865+0000 7f1fd0eeb700 1 -- start start 2021-05-25T12:06:56.883 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.865+0000 7f1fd0eeb700 10 monclient: init 2021-05-25T12:06:56.883 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.865+0000 7f1fd0eeb700 10 monclient: _reopen_session rank -1 2021-05-25T12:06:56.883 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.865+0000 7f1fd0eeb700 10 monclient: _add_conns ranks=[0,2,1] 2021-05-25T12:06:56.883 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.865+0000 7f1fd0eeb700 10 monclient(hunting): picked mon.noname-a con 0x7f1fcc098930 addr v1:172.21.15.102:6789/0 2021-05-25T12:06:56.883 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.865+0000 7f1fd0eeb700 10 monclient(hunting): picked mon.noname-b con 0x7f1fcc12bca0 addr v1:172.21.15.102:6790/0 2021-05-25T12:06:56.884 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.865+0000 7f1fd0eeb700 10 monclient(hunting): picked mon.noname-c con 0x7f1fcc12c150 addr v1:172.21.15.181:6789/0 2021-05-25T12:06:56.884 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.865+0000 7f1fd0eeb700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fcc099b20 con 0x7f1fcc098930 2021-05-25T12:06:56.884 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.865+0000 7f1fd0eeb700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fcc09a3a0 con 0x7f1fcc12c150 2021-05-25T12:06:56.884 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.865+0000 7f1fd0eeb700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fcc09ac60 con 0x7f1fcc12bca0 2021-05-25T12:06:56.885 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.865+0000 7f1fd0eeb700 10 monclient(hunting): _renew_subs 2021-05-25T12:06:56.885 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.865+0000 7f1fd0eeb700 10 monclient(hunting): authenticate will time out at 1217.462819s 2021-05-25T12:06:56.885 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.865+0000 7f1fcb7fe700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f1fcc098930 0x7f1fcc12b590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33596/0 (socket says 172.21.15.102:33596) 2021-05-25T12:06:56.885 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.865+0000 7f1fcb7fe700 1 -- 172.21.15.102:0/2606811219 learned_addr learned my addr 172.21.15.102:0/2606811219 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:06:56.885 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.865+0000 7f1fcaffd700 1 --1- 172.21.15.102:0/2606811219 >> v1:172.21.15.102:6790/0 conn(0x7f1fcc12bca0 0x7f1fcc1308c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:56704/0 (socket says 172.21.15.102:56704) 2021-05-25T12:06:56.886 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.866+0000 7f1fca7fc700 1 -- 172.21.15.102:0/2606811219 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1790702483 0 0) 0x7f1fcc09ac60 con 0x7f1fcc12bca0 2021-05-25T12:06:56.886 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.866+0000 7f1fca7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:56.886 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.866+0000 7f1fca7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:56.886 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.866+0000 7f1fca7fc700 10 monclient(hunting): my global_id is 184634 2021-05-25T12:06:56.886 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.866+0000 7f1fca7fc700 1 -- 172.21.15.102:0/2606811219 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fb4003270 con 0x7f1fcc12bca0 2021-05-25T12:06:56.887 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.866+0000 7f1fca7fc700 1 -- 172.21.15.102:0/2606811219 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 398002209 0 0) 0x7f1fcc099b20 con 0x7f1fcc098930 2021-05-25T12:06:56.887 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.866+0000 7f1fca7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:56.887 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.866+0000 7f1fca7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:56.887 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.866+0000 7f1fca7fc700 10 monclient(hunting): my global_id is 204294 2021-05-25T12:06:56.888 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.866+0000 7f1fca7fc700 1 -- 172.21.15.102:0/2606811219 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fcc09ac60 con 0x7f1fcc098930 2021-05-25T12:06:56.888 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.866+0000 7f1fca7fc700 1 -- 172.21.15.102:0/2606811219 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3489347823 0 0) 0x7f1fcc09a3a0 con 0x7f1fcc12c150 2021-05-25T12:06:56.888 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.866+0000 7f1fca7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:56.888 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.866+0000 7f1fca7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:56.888 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.866+0000 7f1fca7fc700 10 monclient(hunting): my global_id is 224155 2021-05-25T12:06:56.889 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.866+0000 7f1fca7fc700 1 -- 172.21.15.102:0/2606811219 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fcc099b20 con 0x7f1fcc12c150 2021-05-25T12:06:56.889 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.866+0000 7f1fca7fc700 1 -- 172.21.15.102:0/2606811219 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2519446377 0 0) 0x7f1fcc09ac60 con 0x7f1fcc098930 2021-05-25T12:06:56.889 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.866+0000 7f1fca7fc700 1 -- 172.21.15.102:0/2606811219 >> v1:172.21.15.102:6790/0 conn(0x7f1fcc12bca0 legacy=0x7f1fcc1308c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:56.889 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.866+0000 7f1fca7fc700 1 -- 172.21.15.102:0/2606811219 >> v1:172.21.15.181:6789/0 conn(0x7f1fcc12c150 legacy=0x7f1fcc133ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:56.890 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.866+0000 7f1fca7fc700 10 monclient: _finish_hunting 0 2021-05-25T12:06:56.890 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.866+0000 7f1fca7fc700 1 monclient: found mon.noname-a 2021-05-25T12:06:56.890 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.866+0000 7f1fca7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:06:56.890 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.866+0000 7f1fca7fc700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T12:06:56.891 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.866+0000 7f1fca7fc700 1 -- 172.21.15.102:0/2606811219 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fcc09af00 con 0x7f1fcc098930 2021-05-25T12:06:56.891 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.866+0000 7f1fca7fc700 10 monclient: _finish_auth 0 2021-05-25T12:06:56.891 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.866+0000 7f1fca7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:06:56.892 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.866+0000 7f1fca7fc700 1 -- 172.21.15.102:0/2606811219 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f1fb8001570 con 0x7f1fcc098930 2021-05-25T12:06:56.892 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.866+0000 7f1fd0eeb700 5 monclient: authenticate success, global_id 204294 2021-05-25T12:06:56.892 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.866+0000 7f1fd0eeb700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:06:56.892 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.866+0000 7f1fca7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:56.893 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.866+0000 7f1fca7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:06:56.893 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.866+0000 7f1fca7fc700 10 monclient: dump: 2021-05-25T12:06:56.893 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:56.893 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:56.894 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:56.894 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:56.894 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:56.894 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:56.894 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:56.895 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:56.895 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:56.895 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:56.895 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.866+0000 7f1fd0eeb700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:06:56.896 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.866+0000 7f1fca7fc700 1 -- 172.21.15.102:0/2606811219 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1fb80017a0 con 0x7f1fcc098930 2021-05-25T12:06:56.896 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.866+0000 7f1fca7fc700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:06:56.896 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.866+0000 7f1fca7fc700 1 -- 172.21.15.102:0/2606811219 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f1fb8001b30 con 0x7f1fcc098930 2021-05-25T12:06:56.896 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.866+0000 7f1fd0eeb700 10 monclient: get_monmap_and_config success 2021-05-25T12:06:56.896 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.866+0000 7f1fd0eeb700 4 set_mon_vals no callback set 2021-05-25T12:06:56.897 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.866+0000 7f1fca7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:56.897 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.866+0000 7f1fca7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:06:56.897 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.866+0000 7f1fca7fc700 10 monclient: dump: 2021-05-25T12:06:56.897 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:56.898 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:56.898 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:56.898 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:56.898 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:56.898 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:56.899 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:56.899 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:56.899 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:56.899 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:56.899 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.866+0000 7f1fd0eeb700 10 monclient: shutdown 2021-05-25T12:06:56.900 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.866+0000 7f1fd0eeb700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:06:56.900 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.866+0000 7f1fd0eeb700 1 -- 172.21.15.102:0/2606811219 >> v1:172.21.15.102:6789/0 conn(0x7f1fcc098930 legacy=0x7f1fcc12b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:56.900 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.867+0000 7f1fd0eeb700 1 -- 172.21.15.102:0/2606811219 shutdown_connections 2021-05-25T12:06:56.900 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.867+0000 7f1fd0eeb700 1 -- 172.21.15.102:0/2606811219 >> 172.21.15.102:0/2606811219 conn(0x7f1fcc09e9d0 msgr2=0x7f1fcc09ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:06:56.901 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.867+0000 7f1fd0eeb700 1 -- 172.21.15.102:0/2606811219 shutdown_connections 2021-05-25T12:06:56.901 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.867+0000 7f1fd0eeb700 1 -- 172.21.15.102:0/2606811219 wait complete. 2021-05-25T12:06:56.901 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.867+0000 7f1fd0eeb700 10 monclient: build_initial_monmap 2021-05-25T12:06:56.901 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.867+0000 7f1fd0eeb700 10 monclient: monmap: 2021-05-25T12:06:56.902 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:06:56.902 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:06:56.902 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:06:56.902 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:06:56.902 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:06:56.903 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:06:56.903 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:06:56.903 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:06:56.903 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:06:56.903 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:56.904 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.867+0000 7f1fd0eeb700 1 librados: starting msgr at 2021-05-25T12:06:56.904 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.867+0000 7f1fd0eeb700 1 librados: starting objecter 2021-05-25T12:06:56.904 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.867+0000 7f1fd0eeb700 1 Processor -- start 2021-05-25T12:06:56.904 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.868+0000 7f1fd0eeb700 1 -- start start 2021-05-25T12:06:56.905 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.868+0000 7f1fd0eeb700 1 librados: setting wanted keys 2021-05-25T12:06:56.905 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.868+0000 7f1fd0eeb700 1 librados: calling monclient init 2021-05-25T12:06:56.905 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.868+0000 7f1fd0eeb700 10 monclient: init 2021-05-25T12:06:56.905 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.868+0000 7f1fd0eeb700 10 monclient: _reopen_session rank -1 2021-05-25T12:06:56.905 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.868+0000 7f1fd0eeb700 10 monclient: _add_conns ranks=[1,0,2] 2021-05-25T12:06:56.906 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.868+0000 7f1fd0eeb700 10 monclient(hunting): picked mon.noname-b con 0x7f1fcc098930 addr v1:172.21.15.181:6789/0 2021-05-25T12:06:56.906 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.868+0000 7f1fd0eeb700 10 monclient(hunting): picked mon.noname-a con 0x7f1fcc12bca0 addr v1:172.21.15.102:6789/0 2021-05-25T12:06:56.906 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.868+0000 7f1fd0eeb700 10 monclient(hunting): picked mon.noname-c con 0x7f1fcc12c150 addr v1:172.21.15.102:6790/0 2021-05-25T12:06:56.906 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.868+0000 7f1fd0eeb700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fcc09a190 con 0x7f1fcc12bca0 2021-05-25T12:06:56.907 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.868+0000 7f1fd0eeb700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fcc094840 con 0x7f1fcc098930 2021-05-25T12:06:56.907 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.868+0000 7f1fd0eeb700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fcc094ac0 con 0x7f1fcc12c150 2021-05-25T12:06:56.907 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.868+0000 7f1fd0eeb700 10 monclient(hunting): _renew_subs 2021-05-25T12:06:56.907 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.868+0000 7f1fd0eeb700 10 monclient(hunting): authenticate will time out at 1217.465442s 2021-05-25T12:06:56.907 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.868+0000 7f1fcaffd700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f1fcc12bca0 0x7f1fcc096e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33600/0 (socket says 172.21.15.102:33600) 2021-05-25T12:06:56.908 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.868+0000 7f1fcaffd700 1 -- 172.21.15.102:0/1599628515 learned_addr learned my addr 172.21.15.102:0/1599628515 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:06:56.908 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.868+0000 7f1fcbfff700 1 --1- 172.21.15.102:0/1599628515 >> v1:172.21.15.102:6790/0 conn(0x7f1fcc12c150 0x7f1fcc142c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:56712/0 (socket says 172.21.15.102:56712) 2021-05-25T12:06:56.908 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.868+0000 7f1fbf7fe700 10 client.?.objecter ms_handle_connect 0x7f1fcc12bca0 2021-05-25T12:06:56.908 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.868+0000 7f1fbf7fe700 10 client.?.objecter resend_mon_ops 2021-05-25T12:06:56.909 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.868+0000 7f1fbf7fe700 10 client.?.objecter ms_handle_connect 0x7f1fcc12c150 2021-05-25T12:06:56.909 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.868+0000 7f1fbf7fe700 10 client.?.objecter resend_mon_ops 2021-05-25T12:06:56.909 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.868+0000 7f1fbf7fe700 1 -- 172.21.15.102:0/1599628515 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2180111958 0 0) 0x7f1fcc094ac0 con 0x7f1fcc12c150 2021-05-25T12:06:56.909 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.868+0000 7f1fbf7fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:56.909 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.868+0000 7f1fbf7fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:56.910 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.868+0000 7f1fbf7fe700 10 monclient(hunting): my global_id is 184637 2021-05-25T12:06:56.910 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.868+0000 7f1fbf7fe700 1 -- 172.21.15.102:0/1599628515 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fa8004c10 con 0x7f1fcc12c150 2021-05-25T12:06:56.910 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.868+0000 7f1fbf7fe700 1 -- 172.21.15.102:0/1599628515 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2344098883 0 0) 0x7f1fcc09a190 con 0x7f1fcc12bca0 2021-05-25T12:06:56.910 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.868+0000 7f1fbf7fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:56.911 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.868+0000 7f1fbf7fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:56.911 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.868+0000 7f1fbf7fe700 10 monclient(hunting): my global_id is 204297 2021-05-25T12:06:56.911 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.868+0000 7f1fbf7fe700 1 -- 172.21.15.102:0/1599628515 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fcc094ac0 con 0x7f1fcc12bca0 2021-05-25T12:06:56.911 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.868+0000 7f1fbf7fe700 1 -- 172.21.15.102:0/1599628515 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2301313788 0 0) 0x7f1fa8004c10 con 0x7f1fcc12c150 2021-05-25T12:06:56.912 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.868+0000 7f1fbf7fe700 1 -- 172.21.15.102:0/1599628515 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1fcc09a190 con 0x7f1fcc12c150 2021-05-25T12:06:56.912 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.868+0000 7f1fbf7fe700 10 client.?.objecter ms_handle_connect 0x7f1fcc098930 2021-05-25T12:06:56.912 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.868+0000 7f1fbf7fe700 10 client.?.objecter resend_mon_ops 2021-05-25T12:06:56.912 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.868+0000 7f1fbf7fe700 1 -- 172.21.15.102:0/1599628515 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f1fc0001c00 con 0x7f1fcc12c150 2021-05-25T12:06:56.912 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.868+0000 7f1fbf7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:56.913 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.868+0000 7f1fbf7fe700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:06:56.913 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.868+0000 7f1fbf7fe700 10 monclient(hunting): dump: 2021-05-25T12:06:56.913 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:56.913 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:56.914 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:56.914 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:56.914 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:56.914 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:56.914 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:56.915 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:56.915 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:56.915 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:56.915 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.868+0000 7f1fbf7fe700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:06:56.915 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.868+0000 7f1fbf7fe700 1 -- 172.21.15.102:0/1599628515 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 557464768 0 0) 0x7f1fcc094ac0 con 0x7f1fcc12bca0 2021-05-25T12:06:56.916 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.869+0000 7f1fbf7fe700 1 -- 172.21.15.102:0/1599628515 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1fa8004c10 con 0x7f1fcc12bca0 2021-05-25T12:06:56.916 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.869+0000 7f1fbf7fe700 1 -- 172.21.15.102:0/1599628515 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f1fb0001a70 con 0x7f1fcc12bca0 2021-05-25T12:06:56.916 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.869+0000 7f1fbf7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:56.916 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.869+0000 7f1fbf7fe700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2021-05-25T12:06:56.916 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.869+0000 7f1fbf7fe700 10 monclient(hunting): dump: 2021-05-25T12:06:56.917 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:56.917 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:56.917 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:56.917 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:56.917 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:56.918 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:56.918 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:56.918 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:56.918 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:56.919 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:56.919 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.869+0000 7f1fbf7fe700 1 -- 172.21.15.102:0/1599628515 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3154093886 0 0) 0x7f1fcc094840 con 0x7f1fcc098930 2021-05-25T12:06:56.919 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.869+0000 7f1fbf7fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:56.919 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.869+0000 7f1fbf7fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:56.920 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.869+0000 7f1fbf7fe700 10 monclient(hunting): my global_id is 224158 2021-05-25T12:06:56.920 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.869+0000 7f1fbf7fe700 1 -- 172.21.15.102:0/1599628515 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fcc094ac0 con 0x7f1fcc098930 2021-05-25T12:06:56.920 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.869+0000 7f1fbf7fe700 1 -- 172.21.15.102:0/1599628515 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3006255961 0 0) 0x7f1fcc09a190 con 0x7f1fcc12c150 2021-05-25T12:06:56.920 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.869+0000 7f1fbf7fe700 1 -- 172.21.15.102:0/1599628515 >> v1:172.21.15.181:6789/0 conn(0x7f1fcc098930 legacy=0x7f1fcc095430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:56.921 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.869+0000 7f1fbf7fe700 1 -- 172.21.15.102:0/1599628515 >> v1:172.21.15.102:6789/0 conn(0x7f1fcc12bca0 legacy=0x7f1fcc096e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:56.921 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.869+0000 7f1fbf7fe700 10 monclient: _finish_hunting 0 2021-05-25T12:06:56.921 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.869+0000 7f1fbf7fe700 1 monclient: found mon.c 2021-05-25T12:06:56.921 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.869+0000 7f1fbf7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:06:56.921 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.869+0000 7f1fbf7fe700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:06:56.922 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.869+0000 7f1fbf7fe700 1 -- 172.21.15.102:0/1599628515 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fcc092930 con 0x7f1fcc12c150 2021-05-25T12:06:56.922 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.869+0000 7f1fbf7fe700 10 monclient: _finish_auth 0 2021-05-25T12:06:56.922 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.869+0000 7f1fbf7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:06:56.922 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.869+0000 7f1fd0eeb700 5 monclient: authenticate success, global_id 184637 2021-05-25T12:06:56.922 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.869+0000 7f1fd0eeb700 10 monclient: _renew_subs 2021-05-25T12:06:56.923 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.869+0000 7f1fd0eeb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:06:56.923 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.869+0000 7f1fd0eeb700 1 -- 172.21.15.102:0/1599628515 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fcc091e60 con 0x7f1fcc12c150 2021-05-25T12:06:56.923 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.869+0000 7f1fd0eeb700 10 client.184637.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:06:56.923 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.869+0000 7f1fd0eeb700 10 monclient: _renew_subs 2021-05-25T12:06:56.924 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.869+0000 7f1fd0eeb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:06:56.924 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.869+0000 7f1fd0eeb700 1 -- 172.21.15.102:0/1599628515 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fcc097570 con 0x7f1fcc12c150 2021-05-25T12:06:56.924 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.869+0000 7f1fd0eeb700 1 librados: init done 2021-05-25T12:06:56.924 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.869+0000 7f1fbf7fe700 1 -- 172.21.15.102:0/1599628515 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1fc0001de0 con 0x7f1fcc12c150 2021-05-25T12:06:56.925 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.869+0000 7f1fbf7fe700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:06:56.925 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.869+0000 7f1fbf7fe700 1 -- 172.21.15.102:0/1599628515 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f1fc0002170 con 0x7f1fcc12c150 2021-05-25T12:06:56.925 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.869+0000 7f1fbf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:56.925 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.869+0000 7f1fbf7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:06:56.925 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.869+0000 7f1fc9ffb700 4 set_mon_vals no callback set 2021-05-25T12:06:56.926 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.869+0000 7f1fbf7fe700 10 monclient: dump: 2021-05-25T12:06:56.926 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:56.926 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:56.926 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:56.927 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:56.927 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:56.927 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:56.927 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:56.928 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:56.928 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:56.928 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:56.928 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.869+0000 7f1fd0eeb700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:06:56.928 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.869+0000 7f1fbf7fe700 1 -- 172.21.15.102:0/1599628515 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f1fc0002310 con 0x7f1fcc12c150 2021-05-25T12:06:56.929 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.870+0000 7f1fbf7fe700 1 -- 172.21.15.102:0/1599628515 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(221..221 src has 1..221) v4 ==== 6648+0+0 (unknown 2376337231 0 0) 0x7f1fc003dba0 con 0x7f1fcc12c150 2021-05-25T12:06:56.929 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.870+0000 7f1fbf7fe700 10 client.184637.objecter ms_dispatch 0x7f1fcc001550 osd_map(221..221 src has 1..221) v4 2021-05-25T12:06:56.929 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.870+0000 7f1fbf7fe700 3 client.184637.objecter handle_osd_map got epochs [221,221] > 0 2021-05-25T12:06:56.929 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.870+0000 7f1fbf7fe700 3 client.184637.objecter handle_osd_map decoding full epoch 221 2021-05-25T12:06:56.930 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.870+0000 7f1fbf7fe700 20 client.184637.objecter dump_active .. 0 homeless 2021-05-25T12:06:56.930 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.870+0000 7f1fd0eeb700 1 -- 172.21.15.102:0/1599628515 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1fac0065a0 con 0x7f1fac002840 2021-05-25T12:06:56.930 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.870+0000 7f1fcb7fe700 10 monclient: get_auth_request con 0x7f1fa80334f0 auth_method 0 2021-05-25T12:06:56.930 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.870+0000 7f1fbf7fe700 10 client.184637.objecter ms_handle_connect 0x7f1fac002840 2021-05-25T12:06:56.930 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.870+0000 7f1fbf7fe700 10 client.184637.objecter resend_mon_ops 2021-05-25T12:06:56.931 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.870+0000 7f1fbf7fe700 10 client.184637.objecter ms_handle_connect 0x7f1fa80334f0 2021-05-25T12:06:56.931 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.870+0000 7f1fbf7fe700 1 -- 172.21.15.102:0/1599628515 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3309612923 0 0) 0x7f1fac0065a0 con 0x7f1fac002840 2021-05-25T12:06:56.931 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.870+0000 7f1fbf7fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:56.931 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.870+0000 7f1fbf7fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:56.932 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.870+0000 7f1fbf7fe700 10 monclient(hunting): my global_id is 224161 2021-05-25T12:06:56.932 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.870+0000 7f1fbf7fe700 1 -- 172.21.15.102:0/1599628515 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fcc09a190 con 0x7f1fac002840 2021-05-25T12:06:56.932 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.871+0000 7f1fbf7fe700 1 -- 172.21.15.102:0/1599628515 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4016141018 0 0) 0x7f1fcc09a190 con 0x7f1fac002840 2021-05-25T12:06:56.932 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.871+0000 7f1fbf7fe700 1 -- 172.21.15.102:0/1599628515 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1fac0065a0 con 0x7f1fac002840 2021-05-25T12:06:56.932 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.871+0000 7f1fbf7fe700 1 -- 172.21.15.102:0/1599628515 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f1fb00046d0 con 0x7f1fac002840 2021-05-25T12:06:56.933 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.871+0000 7f1fbf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:56.933 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.871+0000 7f1fbf7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T12:06:56.933 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.871+0000 7f1fbf7fe700 10 monclient: dump: 2021-05-25T12:06:56.933 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:56.933 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:56.934 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:56.934 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:56.934 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:56.934 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:56.934 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:56.935 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:56.935 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:56.935 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:56.935 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.871+0000 7f1fbf7fe700 1 -- 172.21.15.102:0/1599628515 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2495682454 0 0) 0x7f1fac0065a0 con 0x7f1fac002840 2021-05-25T12:06:56.935 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.871+0000 7f1fbf7fe700 1 -- 172.21.15.102:0/1599628515 --> v1:172.21.15.181:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1fac009e40 con 0x7f1fac002840 2021-05-25T12:06:56.936 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.871+0000 7f1fbf7fe700 1 -- 172.21.15.102:0/1599628515 <== mon.1 v1:172.21.15.181:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7f1fb00069e0 con 0x7f1fac002840 2021-05-25T12:06:56.936 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.871+0000 7f1fbf7fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:06:56.936 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.871+0000 7f1fbf7fe700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:06:56.936 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.871+0000 7f1fbf7fe700 1 -- 172.21.15.102:0/1599628515 >> v1:172.21.15.181:6789/0 conn(0x7f1fac002840 legacy=0x7f1fac004ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:56.936 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.871+0000 7f1fbf7fe700 1 -- 172.21.15.102:0/1599628515 >> v1:172.21.15.181:6789/0 conn(0x7f1fac002840 legacy=0x7f1fac004ce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:06:56.937 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.878+0000 7f1fd0eeb700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:06:56.937 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.878+0000 7f1fd0eeb700 1 -- 172.21.15.102:0/1599628515 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1fac007540 con 0x7f1fac0068e0 2021-05-25T12:06:56.937 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.878+0000 7f1fbf7fe700 10 client.184637.objecter ms_handle_connect 0x7f1fac0068e0 2021-05-25T12:06:56.937 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.878+0000 7f1fbf7fe700 10 client.184637.objecter resend_mon_ops 2021-05-25T12:06:56.937 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.878+0000 7f1fbf7fe700 1 -- 172.21.15.102:0/1599628515 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2290583720 0 0) 0x7f1fac007540 con 0x7f1fac0068e0 2021-05-25T12:06:56.938 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.878+0000 7f1fbf7fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:56.938 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.878+0000 7f1fbf7fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:56.938 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.878+0000 7f1fbf7fe700 10 monclient(hunting): my global_id is 224164 2021-05-25T12:06:56.938 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.878+0000 7f1fbf7fe700 1 -- 172.21.15.102:0/1599628515 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fac0065a0 con 0x7f1fac0068e0 2021-05-25T12:06:56.938 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.879+0000 7f1fbf7fe700 1 -- 172.21.15.102:0/1599628515 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2775836048 0 0) 0x7f1fac0065a0 con 0x7f1fac0068e0 2021-05-25T12:06:56.938 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.879+0000 7f1fbf7fe700 1 -- 172.21.15.102:0/1599628515 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1fac007540 con 0x7f1fac0068e0 2021-05-25T12:06:56.939 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.879+0000 7f1fbf7fe700 1 -- 172.21.15.102:0/1599628515 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f1fb0004e80 con 0x7f1fac0068e0 2021-05-25T12:06:56.939 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.879+0000 7f1fbf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:56.939 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.879+0000 7f1fbf7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T12:06:56.939 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.879+0000 7f1fbf7fe700 10 monclient: dump: 2021-05-25T12:06:56.939 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:56.940 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:56.940 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:56.940 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:56.940 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:56.940 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:56.941 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:56.941 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:56.941 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:56.941 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:56.941 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.879+0000 7f1fbf7fe700 1 -- 172.21.15.102:0/1599628515 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3012872662 0 0) 0x7f1fac007540 con 0x7f1fac0068e0 2021-05-25T12:06:56.942 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.879+0000 7f1fbf7fe700 1 -- 172.21.15.102:0/1599628515 --> v1:172.21.15.181:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f1fac008770 con 0x7f1fac0068e0 2021-05-25T12:06:56.942 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.879+0000 7f1fbf7fe700 1 -- 172.21.15.102:0/1599628515 <== mon.1 v1:172.21.15.181:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3770 (unknown 0 0 3604378473) 0x7f1fb0006240 con 0x7f1fac0068e0 2021-05-25T12:06:56.942 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.879+0000 7f1fbf7fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:06:56.942 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.879+0000 7f1fbf7fe700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:06:56.942 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.879+0000 7f1fbf7fe700 1 -- 172.21.15.102:0/1599628515 >> v1:172.21.15.181:6789/0 conn(0x7f1fac0068e0 legacy=0x7f1fac006d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:56.943 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.879+0000 7f1fbf7fe700 1 -- 172.21.15.102:0/1599628515 >> v1:172.21.15.181:6789/0 conn(0x7f1fac0068e0 legacy=0x7f1fac006d70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:06:56.943 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:06:56.943 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:06:56.944 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:06:56.944 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "peon", 2021-05-25T12:06:56.944 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 58, 2021-05-25T12:06:56.944 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:06:56.944 INFO:teuthology.orchestra.run.smithi102.stdout: 0, 2021-05-25T12:06:56.945 INFO:teuthology.orchestra.run.smithi102.stdout: 1, 2021-05-25T12:06:56.945 INFO:teuthology.orchestra.run.smithi102.stdout: 2 2021-05-25T12:06:56.945 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:06:56.945 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 3, 2021-05-25T12:06:56.945 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:06:56.946 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:06:56.946 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:06:56.946 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:06:56.946 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:06:56.946 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:06:56.947 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:06:56.947 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:06:56.947 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:06:56.947 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:06:56.947 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:06:56.948 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:06:56.948 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:06:56.948 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:06:56.948 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:06:56.948 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:06:56.949 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:06:56.949 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:06:56.949 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:06:56.949 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:06:56.950 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:06:56.950 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:06:56.950 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:06:56.950 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:06:56.950 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:56.951 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:56.951 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:06:56.951 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:06:56.951 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:06:56.951 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:06:56.952 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:06:56.952 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:06:56.952 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:06:56.952 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:06:56.952 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:06:56.953 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:06:56.953 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:06:56.953 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:06:56.953 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:06:56.954 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:06:56.954 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:06:56.954 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:06:56.954 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:06:56.954 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:06:56.954 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:06:56.955 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:06:56.955 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:06:56.955 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:06:56.955 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:06:56.955 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:06:56.956 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:06:56.956 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:06:56.956 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:56.956 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:06:56.956 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:56.957 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:06:56.957 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:06:56.957 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:06:56.957 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:06:56.957 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:56.958 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:06:56.958 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:06:56.958 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:06:56.958 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:56.958 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:56.959 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:56.959 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:06:56.959 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:06:56.959 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:06:56.959 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:06:56.959 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:06:56.960 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:56.960 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:56.960 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:06:56.960 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:06:56.960 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:06:56.961 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:06:56.961 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:56.961 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:06:56.961 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:06:56.961 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:06:56.962 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:56.962 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:56.962 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:56.962 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:06:56.962 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:06:56.963 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:06:56.963 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:06:56.963 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:06:56.963 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:56.963 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:56.963 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:06:56.964 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:06:56.964 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:06:56.964 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:06:56.964 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:56.964 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:06:56.965 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:06:56.965 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:06:56.965 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:56.965 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:56.965 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:56.966 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:06:56.966 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:06:56.966 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:06:56.966 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:06:56.966 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:06:56.967 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:56.967 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:56.967 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:56.967 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:06:56.967 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:06:56.967 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:56.968 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:06:56.968 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:06:56.968 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:06:56.968 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:56.968 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:06:56.969 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:06:56.969 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:06:56.969 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:06:56.969 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:06:56.969 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 2 2021-05-25T12:06:56.970 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:06:56.970 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:06:56.970 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:06:56.970 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:06:56.970 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:06:56.972 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.879+0000 7f1fd0eeb700 10 librados: watch_flush enter 2021-05-25T12:06:56.972 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.879+0000 7f1fd0eeb700 10 librados: watch_flush exit 2021-05-25T12:06:56.973 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.879+0000 7f1fd0eeb700 20 client.184637.objecter shutdown clearing up homeless session... 2021-05-25T12:06:56.973 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.879+0000 7f1fd0eeb700 10 client.184637.objecter successfully canceled tick 2021-05-25T12:06:56.973 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.879+0000 7f1fd0eeb700 1 -- 172.21.15.102:0/1599628515 >> v1:172.21.15.181:6812/29582 conn(0x7f1fa80334f0 legacy=0x7f1fa80359b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:56.973 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.879+0000 7f1fd0eeb700 10 monclient: shutdown 2021-05-25T12:06:56.973 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.879+0000 7f1fd0eeb700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:06:56.974 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.879+0000 7f1fd0eeb700 1 -- 172.21.15.102:0/1599628515 >> v1:172.21.15.102:6790/0 conn(0x7f1fcc12c150 legacy=0x7f1fcc142c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:56.974 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.879+0000 7f1fcbfff700 1 -- 172.21.15.102:0/1599628515 reap_dead start 2021-05-25T12:06:56.974 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.879+0000 7f1fcbfff700 1 -- 172.21.15.102:0/1599628515 reap_dead start 2021-05-25T12:06:56.974 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.879+0000 7f1fd0eeb700 1 -- 172.21.15.102:0/1599628515 shutdown_connections 2021-05-25T12:06:56.974 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.880+0000 7f1fd0eeb700 1 -- 172.21.15.102:0/1599628515 >> 172.21.15.102:0/1599628515 conn(0x7f1fcc09e9d0 msgr2=0x7f1fcc12a7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:06:56.975 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.880+0000 7f1fd0eeb700 1 -- 172.21.15.102:0/1599628515 shutdown_connections 2021-05-25T12:06:56.975 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.880+0000 7f1fd0eeb700 1 -- 172.21.15.102:0/1599628515 wait complete. 2021-05-25T12:06:56.975 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:56.880+0000 7f1fd0eeb700 1 librados: shutdown 2021-05-25T12:06:56.975 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2021-05-25T12:06:56.976 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2021-05-25T12:06:56.976 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2021-05-25T12:06:56.976 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2021-05-25T12:06:56.977 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2021-05-25T12:06:56.977 INFO:teuthology.orchestra.run:waiting for 300 2021-05-25T12:06:56.988 INFO:tasks.ceph.mon.c:Stopped 2021-05-25T12:06:56.989 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2021-05-25T12:06:56.989 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2021-05-25T12:06:57.145 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.135+0000 7f5e9d45a700 10 monclient: get_monmap_and_config 2021-05-25T12:06:57.146 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.135+0000 7f5e9d45a700 10 monclient: build_initial_monmap 2021-05-25T12:06:57.146 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.135+0000 7f5e9d45a700 10 monclient: monmap: 2021-05-25T12:06:57.146 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:06:57.146 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:57.147 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:06:57.136515+0000 2021-05-25T12:06:57.147 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:06:57.136515+0000 2021-05-25T12:06:57.147 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:06:57.147 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:57.147 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:06:57.148 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:06:57.148 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:06:57.148 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:57.148 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.136+0000 7f5e9d45a700 1 Processor -- start 2021-05-25T12:06:57.148 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.136+0000 7f5e9d45a700 1 -- start start 2021-05-25T12:06:57.149 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.136+0000 7f5e9d45a700 10 monclient: init 2021-05-25T12:06:57.149 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.136+0000 7f5e9d45a700 10 monclient: _reopen_session rank -1 2021-05-25T12:06:57.149 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.136+0000 7f5e9d45a700 10 monclient: _add_conns ranks=[0,1,2] 2021-05-25T12:06:57.149 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.136+0000 7f5e9d45a700 10 monclient(hunting): picked mon.noname-a con 0x7f5e98068d50 addr v1:172.21.15.102:6789/0 2021-05-25T12:06:57.149 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.136+0000 7f5e9d45a700 10 monclient(hunting): picked mon.noname-c con 0x7f5e98069130 addr v1:172.21.15.181:6789/0 2021-05-25T12:06:57.150 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.136+0000 7f5e9d45a700 10 monclient(hunting): picked mon.noname-b con 0x7f5e980695e0 addr v1:172.21.15.102:6790/0 2021-05-25T12:06:57.150 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.136+0000 7f5e9d45a700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e98094380 con 0x7f5e98068d50 2021-05-25T12:06:57.150 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.136+0000 7f5e9d45a700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e98094620 con 0x7f5e98069130 2021-05-25T12:06:57.150 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.136+0000 7f5e9d45a700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e980948c0 con 0x7f5e980695e0 2021-05-25T12:06:57.151 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.136+0000 7f5e9d45a700 10 monclient(hunting): _renew_subs 2021-05-25T12:06:57.151 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.136+0000 7f5e9d45a700 10 monclient(hunting): authenticate will time out at 1217.733684s 2021-05-25T12:06:57.151 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.136+0000 7f5e977fe700 1 -- >> v1:172.21.15.102:6790/0 conn(0x7f5e980695e0 legacy=0x7f5e98132160 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6790/0 2021-05-25T12:06:57.152 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.136+0000 7f5e96ffd700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f5e98068d50 0x7f5e98129c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33624/0 (socket says 172.21.15.102:33624) 2021-05-25T12:06:57.152 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.136+0000 7f5e96ffd700 1 -- 172.21.15.102:0/4111926481 learned_addr learned my addr 172.21.15.102:0/4111926481 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:06:57.153 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.136+0000 7f5e95ffb700 1 -- 172.21.15.102:0/4111926481 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2145030930 0 0) 0x7f5e98094380 con 0x7f5e98068d50 2021-05-25T12:06:57.153 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.137+0000 7f5e95ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:57.153 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.137+0000 7f5e95ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:57.153 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.137+0000 7f5e95ffb700 10 monclient(hunting): my global_id is 204300 2021-05-25T12:06:57.154 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.137+0000 7f5e95ffb700 1 -- 172.21.15.102:0/4111926481 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e84003270 con 0x7f5e98068d50 2021-05-25T12:06:57.154 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.137+0000 7f5e95ffb700 1 -- 172.21.15.102:0/4111926481 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2362249145 0 0) 0x7f5e84003270 con 0x7f5e98068d50 2021-05-25T12:06:57.154 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.137+0000 7f5e95ffb700 1 -- 172.21.15.102:0/4111926481 >> v1:172.21.15.102:6790/0 conn(0x7f5e980695e0 legacy=0x7f5e98132160 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-05-25T12:06:57.154 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.137+0000 7f5e95ffb700 1 -- 172.21.15.102:0/4111926481 >> v1:172.21.15.181:6789/0 conn(0x7f5e98069130 legacy=0x7f5e9812ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:57.155 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.137+0000 7f5e95ffb700 10 monclient: _finish_hunting 0 2021-05-25T12:06:57.155 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.137+0000 7f5e95ffb700 1 monclient: found mon.noname-a 2021-05-25T12:06:57.155 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.137+0000 7f5e95ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:06:57.155 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.137+0000 7f5e95ffb700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T12:06:57.155 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.137+0000 7f5e95ffb700 1 -- 172.21.15.102:0/4111926481 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e98094c80 con 0x7f5e98068d50 2021-05-25T12:06:57.156 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.137+0000 7f5e95ffb700 10 monclient: _finish_auth 0 2021-05-25T12:06:57.156 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.137+0000 7f5e95ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:06:57.156 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.137+0000 7f5e9d45a700 5 monclient: authenticate success, global_id 204300 2021-05-25T12:06:57.156 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.137+0000 7f5e95ffb700 1 -- 172.21.15.102:0/4111926481 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f5e8c001610 con 0x7f5e98068d50 2021-05-25T12:06:57.156 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.137+0000 7f5e9d45a700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:06:57.157 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.137+0000 7f5e95ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:57.157 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.137+0000 7f5e95ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:06:57.157 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.137+0000 7f5e95ffb700 10 monclient: dump: 2021-05-25T12:06:57.157 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:57.158 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:57.158 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:57.158 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:57.158 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:57.158 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:57.159 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:57.159 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:57.159 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:57.159 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:57.159 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.137+0000 7f5e9d45a700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:06:57.160 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.137+0000 7f5e95ffb700 1 -- 172.21.15.102:0/4111926481 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5e8c001930 con 0x7f5e98068d50 2021-05-25T12:06:57.160 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.137+0000 7f5e95ffb700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:06:57.160 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.137+0000 7f5e9d45a700 10 monclient: get_monmap_and_config success 2021-05-25T12:06:57.160 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.137+0000 7f5e95ffb700 1 -- 172.21.15.102:0/4111926481 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f5e8c001d00 con 0x7f5e98068d50 2021-05-25T12:06:57.161 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.137+0000 7f5e95ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:57.161 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.137+0000 7f5e9d45a700 4 set_mon_vals no callback set 2021-05-25T12:06:57.161 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.137+0000 7f5e95ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:06:57.161 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.137+0000 7f5e95ffb700 10 monclient: dump: 2021-05-25T12:06:57.161 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:57.162 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:57.162 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:57.162 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:57.162 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:57.163 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:57.163 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:57.163 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:57.163 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:57.163 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:57.164 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.137+0000 7f5e9d45a700 10 monclient: shutdown 2021-05-25T12:06:57.164 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.137+0000 7f5e9d45a700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:06:57.164 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.137+0000 7f5e9d45a700 1 -- 172.21.15.102:0/4111926481 >> v1:172.21.15.102:6789/0 conn(0x7f5e98068d50 legacy=0x7f5e98129c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:57.164 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.137+0000 7f5e9d45a700 1 -- 172.21.15.102:0/4111926481 shutdown_connections 2021-05-25T12:06:57.164 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.137+0000 7f5e9d45a700 1 -- 172.21.15.102:0/4111926481 >> 172.21.15.102:0/4111926481 conn(0x7f5e9809b370 msgr2=0x7f5e98097430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:06:57.165 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.137+0000 7f5e9d45a700 1 -- 172.21.15.102:0/4111926481 shutdown_connections 2021-05-25T12:06:57.165 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.137+0000 7f5e9d45a700 1 -- 172.21.15.102:0/4111926481 wait complete. 2021-05-25T12:06:57.165 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.138+0000 7f5e9d45a700 10 monclient: build_initial_monmap 2021-05-25T12:06:57.165 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.138+0000 7f5e9d45a700 10 monclient: monmap: 2021-05-25T12:06:57.165 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:06:57.166 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:06:57.166 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:06:57.166 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:06:57.166 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:06:57.167 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:06:57.167 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:06:57.167 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:06:57.167 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:06:57.167 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:57.168 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.138+0000 7f5e9d45a700 1 librados: starting msgr at 2021-05-25T12:06:57.168 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.138+0000 7f5e9d45a700 1 librados: starting objecter 2021-05-25T12:06:57.168 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.138+0000 7f5e9d45a700 1 Processor -- start 2021-05-25T12:06:57.168 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.138+0000 7f5e9d45a700 1 -- start start 2021-05-25T12:06:57.168 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.138+0000 7f5e9d45a700 1 librados: setting wanted keys 2021-05-25T12:06:57.169 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.138+0000 7f5e9d45a700 1 librados: calling monclient init 2021-05-25T12:06:57.169 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.138+0000 7f5e9d45a700 10 monclient: init 2021-05-25T12:06:57.169 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.139+0000 7f5e9d45a700 10 monclient: _reopen_session rank -1 2021-05-25T12:06:57.169 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.139+0000 7f5e9d45a700 10 monclient: _add_conns ranks=[0,2,1] 2021-05-25T12:06:57.170 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.139+0000 7f5e9d45a700 10 monclient(hunting): picked mon.noname-a con 0x7f5e98068d50 addr v1:172.21.15.102:6789/0 2021-05-25T12:06:57.170 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.139+0000 7f5e9d45a700 10 monclient(hunting): picked mon.noname-c con 0x7f5e98069130 addr v1:172.21.15.102:6790/0 2021-05-25T12:06:57.170 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.139+0000 7f5e9d45a700 10 monclient(hunting): picked mon.noname-b con 0x7f5e980695e0 addr v1:172.21.15.181:6789/0 2021-05-25T12:06:57.170 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.139+0000 7f5e9d45a700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e98124230 con 0x7f5e98068d50 2021-05-25T12:06:57.170 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.139+0000 7f5e9d45a700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e98125d80 con 0x7f5e980695e0 2021-05-25T12:06:57.171 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.139+0000 7f5e9d45a700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e98128aa0 con 0x7f5e98069130 2021-05-25T12:06:57.171 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.139+0000 7f5e9d45a700 10 monclient(hunting): _renew_subs 2021-05-25T12:06:57.171 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.139+0000 7f5e9d45a700 10 monclient(hunting): authenticate will time out at 1217.736424s 2021-05-25T12:06:57.171 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.139+0000 7f5e967fc700 1 -- >> v1:172.21.15.102:6790/0 conn(0x7f5e98069130 legacy=0x7f5e98138f60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.102:6790/0 2021-05-25T12:06:57.172 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.139+0000 7f5e96ffd700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f5e98068d50 0x7f5e98125670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33630/0 (socket says 172.21.15.102:33630) 2021-05-25T12:06:57.172 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.139+0000 7f5e96ffd700 1 -- 172.21.15.102:0/920430872 learned_addr learned my addr 172.21.15.102:0/920430872 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:06:57.172 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.139+0000 7f5e7effd700 10 client.?.objecter ms_handle_connect 0x7f5e98068d50 2021-05-25T12:06:57.172 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.139+0000 7f5e7effd700 10 client.?.objecter resend_mon_ops 2021-05-25T12:06:57.172 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.139+0000 7f5e7effd700 1 -- 172.21.15.102:0/920430872 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1883274476 0 0) 0x7f5e98124230 con 0x7f5e98068d50 2021-05-25T12:06:57.173 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.139+0000 7f5e7effd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:57.173 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.139+0000 7f5e7effd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:57.173 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.139+0000 7f5e7effd700 10 monclient(hunting): my global_id is 204303 2021-05-25T12:06:57.173 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.139+0000 7f5e7effd700 1 -- 172.21.15.102:0/920430872 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e78004c10 con 0x7f5e98068d50 2021-05-25T12:06:57.173 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.139+0000 7f5e7effd700 10 client.?.objecter ms_handle_connect 0x7f5e980695e0 2021-05-25T12:06:57.174 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.139+0000 7f5e7effd700 10 client.?.objecter resend_mon_ops 2021-05-25T12:06:57.174 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.139+0000 7f5e7effd700 1 -- 172.21.15.102:0/920430872 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2461359937 0 0) 0x7f5e78004c10 con 0x7f5e98068d50 2021-05-25T12:06:57.174 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.139+0000 7f5e7effd700 1 -- 172.21.15.102:0/920430872 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5e98124230 con 0x7f5e98068d50 2021-05-25T12:06:57.174 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.139+0000 7f5e7effd700 1 -- 172.21.15.102:0/920430872 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f5e8c001c80 con 0x7f5e98068d50 2021-05-25T12:06:57.175 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.139+0000 7f5e7effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:57.175 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.139+0000 7f5e7effd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:06:57.175 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.139+0000 7f5e7effd700 10 monclient(hunting): dump: 2021-05-25T12:06:57.175 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:57.175 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:57.176 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:57.176 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:57.176 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:57.176 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:57.176 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:57.177 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:57.177 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:57.177 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:57.177 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.139+0000 7f5e7effd700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:06:57.178 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.139+0000 7f5e7effd700 1 -- 172.21.15.102:0/920430872 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 719308395 0 0) 0x7f5e98125d80 con 0x7f5e980695e0 2021-05-25T12:06:57.178 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.139+0000 7f5e7effd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:06:57.178 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.139+0000 7f5e7effd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:06:57.178 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.139+0000 7f5e7effd700 10 monclient(hunting): my global_id is 224170 2021-05-25T12:06:57.178 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.139+0000 7f5e7effd700 1 -- 172.21.15.102:0/920430872 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e78004c10 con 0x7f5e980695e0 2021-05-25T12:06:57.179 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.140+0000 7f5e7effd700 1 -- 172.21.15.102:0/920430872 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1139829125 0 0) 0x7f5e98124230 con 0x7f5e98068d50 2021-05-25T12:06:57.179 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.140+0000 7f5e7effd700 1 -- 172.21.15.102:0/920430872 >> v1:172.21.15.102:6790/0 conn(0x7f5e98069130 legacy=0x7f5e98138f60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-05-25T12:06:57.179 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.140+0000 7f5e7effd700 1 -- 172.21.15.102:0/920430872 >> v1:172.21.15.181:6789/0 conn(0x7f5e980695e0 legacy=0x7f5e981412b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:57.179 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.140+0000 7f5e7effd700 10 monclient: _finish_hunting 0 2021-05-25T12:06:57.179 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.140+0000 7f5e7effd700 1 monclient: found mon.a 2021-05-25T12:06:57.180 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.140+0000 7f5e7effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:06:57.180 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.140+0000 7f5e7effd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:06:57.180 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.140+0000 7f5e7effd700 1 -- 172.21.15.102:0/920430872 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e981236a0 con 0x7f5e98068d50 2021-05-25T12:06:57.180 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.140+0000 7f5e7effd700 10 monclient: _finish_auth 0 2021-05-25T12:06:57.181 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.140+0000 7f5e7effd700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:06:57.181 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.140+0000 7f5e9d45a700 5 monclient: authenticate success, global_id 204303 2021-05-25T12:06:57.181 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.140+0000 7f5e9d45a700 10 monclient: _renew_subs 2021-05-25T12:06:57.181 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.140+0000 7f5e9d45a700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:06:57.181 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.140+0000 7f5e9d45a700 1 -- 172.21.15.102:0/920430872 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e98122d00 con 0x7f5e98068d50 2021-05-25T12:06:57.182 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.140+0000 7f5e9d45a700 10 client.204303.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:06:57.182 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.140+0000 7f5e9d45a700 10 monclient: _renew_subs 2021-05-25T12:06:57.182 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.140+0000 7f5e9d45a700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:06:57.182 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.140+0000 7f5e9d45a700 1 -- 172.21.15.102:0/920430872 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e98139670 con 0x7f5e98068d50 2021-05-25T12:06:57.182 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.140+0000 7f5e9d45a700 1 librados: init done 2021-05-25T12:06:57.183 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.140+0000 7f5e7effd700 1 -- 172.21.15.102:0/920430872 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5e8c004ad0 con 0x7f5e98068d50 2021-05-25T12:06:57.183 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.140+0000 7f5e7effd700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:06:57.183 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.140+0000 7f5e7effd700 1 -- 172.21.15.102:0/920430872 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f5e8c004e60 con 0x7f5e98068d50 2021-05-25T12:06:57.183 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.140+0000 7f5e7effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:06:57.184 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.140+0000 7f5e957fa700 4 set_mon_vals no callback set 2021-05-25T12:06:57.184 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.140+0000 7f5e7effd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:06:57.184 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.140+0000 7f5e7effd700 10 monclient: dump: 2021-05-25T12:06:57.184 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:06:57.184 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:06:57.185 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:57.185 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:06:57.185 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:06:57.185 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:06:57.186 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:06:57.186 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:06:57.186 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:06:57.186 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:06:57.186 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.140+0000 7f5e7effd700 1 -- 172.21.15.102:0/920430872 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f5e8c005040 con 0x7f5e98068d50 2021-05-25T12:06:57.187 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.140+0000 7f5e9d45a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:06:57.187 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.140+0000 7f5e9d45a700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:06:57.188 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.140+0000 7f5e9d45a700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:06:57.188 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.140+0000 7f5e9d45a700 1 -- 172.21.15.102:0/920430872 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e70004db0 con 0x7f5e98068d50 2021-05-25T12:06:57.188 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.141+0000 7f5e7effd700 1 -- 172.21.15.102:0/920430872 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(221..221 src has 1..221) v4 ==== 6648+0+0 (unknown 2376337231 0 0) 0x7f5e8c03dd80 con 0x7f5e98068d50 2021-05-25T12:06:57.188 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.141+0000 7f5e7effd700 10 client.204303.objecter ms_dispatch 0x7f5e98001550 osd_map(221..221 src has 1..221) v4 2021-05-25T12:06:57.189 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.141+0000 7f5e7effd700 3 client.204303.objecter handle_osd_map got epochs [221,221] > 0 2021-05-25T12:06:57.189 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.141+0000 7f5e7effd700 3 client.204303.objecter handle_osd_map decoding full epoch 221 2021-05-25T12:06:57.189 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.141+0000 7f5e7effd700 20 client.204303.objecter dump_active .. 0 homeless 2021-05-25T12:06:57.189 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.141+0000 7f5e967fc700 10 monclient: get_auth_request con 0x7f5e78033150 auth_method 0 2021-05-25T12:06:57.189 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.141+0000 7f5e7effd700 10 client.204303.objecter ms_handle_connect 0x7f5e78033150 2021-05-25T12:06:57.190 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.145+0000 7f5e7effd700 1 -- 172.21.15.102:0/920430872 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7f5e8c006680 con 0x7f5e98068d50 2021-05-25T12:06:57.190 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.145+0000 7f5e7effd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:06:57.191 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.145+0000 7f5e7effd700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:06:57.353 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.351+0000 7f5e9d45a700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2021-05-25T12:06:57.353 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.351+0000 7f5e9d45a700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2021-05-25T12:06:57.354 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.351+0000 7f5e9d45a700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:06:57.354 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.351+0000 7f5e9d45a700 1 -- 172.21.15.102:0/920430872 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f5e70005060 con 0x7f5e98068d50 2021-05-25T12:06:57.354 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.352+0000 7f5e7effd700 1 -- 172.21.15.102:0/920430872 <== mon.0 v1:172.21.15.102:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1342 (unknown 254746657 0 507226826) 0x7f5e8c012e20 con 0x7f5e98068d50 2021-05-25T12:06:57.355 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.352+0000 7f5e7effd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2021-05-25T12:06:57.355 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.352+0000 7f5e7effd700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:06:57.355 INFO:teuthology.orchestra.run.smithi102.stdout:{"election_epoch":58,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":3,"features":{"quorum_con":"4540138303579357183","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"20ee91b6-8935-4b98-b868-85c03789ccc4","modified":"2021-05-25T11:59:11.011522Z","created":"2021-05-25T11:59:11.011522Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6790","nonce":0}]},"addr":"172.21.15.102:6790/0","public_addr":"172.21.15.102:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2021-05-25T12:06:57.357 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.353+0000 7f5e9d45a700 10 librados: watch_flush enter 2021-05-25T12:06:57.357 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.353+0000 7f5e9d45a700 10 librados: watch_flush exit 2021-05-25T12:06:57.357 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.353+0000 7f5e9d45a700 20 client.204303.objecter shutdown clearing up homeless session... 2021-05-25T12:06:57.358 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.353+0000 7f5e9d45a700 10 client.204303.objecter successfully canceled tick 2021-05-25T12:06:57.358 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.353+0000 7f5e9d45a700 1 -- 172.21.15.102:0/920430872 >> v1:172.21.15.181:6812/29582 conn(0x7f5e78033150 legacy=0x7f5e78035610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:57.358 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.353+0000 7f5e9d45a700 10 monclient: shutdown 2021-05-25T12:06:57.358 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.353+0000 7f5e9d45a700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:06:57.359 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.353+0000 7f5e9d45a700 1 -- 172.21.15.102:0/920430872 >> v1:172.21.15.102:6789/0 conn(0x7f5e98068d50 legacy=0x7f5e98125670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:06:57.359 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.353+0000 7f5e9d45a700 1 -- 172.21.15.102:0/920430872 shutdown_connections 2021-05-25T12:06:57.359 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.353+0000 7f5e9d45a700 1 -- 172.21.15.102:0/920430872 >> 172.21.15.102:0/920430872 conn(0x7f5e9809b370 msgr2=0x7f5e981274d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:06:57.359 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.353+0000 7f5e9d45a700 1 -- 172.21.15.102:0/920430872 shutdown_connections 2021-05-25T12:06:57.359 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.353+0000 7f5e9d45a700 1 -- 172.21.15.102:0/920430872 wait complete. 2021-05-25T12:06:57.360 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:06:57.353+0000 7f5e9d45a700 1 librados: shutdown 2021-05-25T12:07:00.367 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2021-05-25T12:07:00.492 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.490+0000 7ffac0f1d700 10 monclient: get_monmap_and_config 2021-05-25T12:07:00.492 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.490+0000 7ffac0f1d700 10 monclient: build_initial_monmap 2021-05-25T12:07:00.492 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.490+0000 7ffac0f1d700 10 monclient: monmap: 2021-05-25T12:07:00.493 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:07:00.493 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:00.493 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:07:00.491805+0000 2021-05-25T12:07:00.493 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:07:00.491805+0000 2021-05-25T12:07:00.493 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:07:00.494 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:00.494 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:07:00.494 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:07:00.494 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:07:00.495 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:00.504 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.491+0000 7ffac0f1d700 1 Processor -- start 2021-05-25T12:07:00.504 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.491+0000 7ffac0f1d700 1 -- start start 2021-05-25T12:07:00.504 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.491+0000 7ffac0f1d700 10 monclient: init 2021-05-25T12:07:00.505 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.492+0000 7ffac0f1d700 10 monclient: _reopen_session rank -1 2021-05-25T12:07:00.505 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.492+0000 7ffac0f1d700 10 monclient: _add_conns ranks=[0,2,1] 2021-05-25T12:07:00.505 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.492+0000 7ffac0f1d700 10 monclient(hunting): picked mon.noname-a con 0x7ffabc093fc0 addr v1:172.21.15.102:6789/0 2021-05-25T12:07:00.505 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.492+0000 7ffac0f1d700 10 monclient(hunting): picked mon.noname-b con 0x7ffabc0692d0 addr v1:172.21.15.102:6790/0 2021-05-25T12:07:00.506 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.492+0000 7ffac0f1d700 10 monclient(hunting): picked mon.noname-c con 0x7ffabc12f130 addr v1:172.21.15.181:6789/0 2021-05-25T12:07:00.506 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.492+0000 7ffac0f1d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffabc097160 con 0x7ffabc093fc0 2021-05-25T12:07:00.506 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.492+0000 7ffac0f1d700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffabc092fe0 con 0x7ffabc12f130 2021-05-25T12:07:00.506 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.492+0000 7ffac0f1d700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffabc093280 con 0x7ffabc0692d0 2021-05-25T12:07:00.506 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.492+0000 7ffac0f1d700 10 monclient(hunting): _renew_subs 2021-05-25T12:07:00.507 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.492+0000 7ffac0f1d700 10 monclient(hunting): authenticate will time out at 1221.089233s 2021-05-25T12:07:00.507 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.492+0000 7ffab9d9b700 1 -- >> v1:172.21.15.102:6790/0 conn(0x7ffabc0692d0 legacy=0x7ffabc096a50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6790/0 2021-05-25T12:07:00.507 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.492+0000 7ffaba59c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7ffabc093fc0 0x7ffabc096340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33646/0 (socket says 172.21.15.102:33646) 2021-05-25T12:07:00.507 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.492+0000 7ffaba59c700 1 -- 172.21.15.102:0/1793381408 learned_addr learned my addr 172.21.15.102:0/1793381408 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:07:00.508 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.492+0000 7ffab959a700 1 -- 172.21.15.102:0/1793381408 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1943243059 0 0) 0x7ffabc097160 con 0x7ffabc093fc0 2021-05-25T12:07:00.508 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.492+0000 7ffab959a700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:00.508 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.492+0000 7ffab959a700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:00.508 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.492+0000 7ffab959a700 10 monclient(hunting): my global_id is 204306 2021-05-25T12:07:00.508 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.492+0000 7ffab959a700 1 -- 172.21.15.102:0/1793381408 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffaa0003270 con 0x7ffabc093fc0 2021-05-25T12:07:00.509 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.492+0000 7ffab959a700 1 -- 172.21.15.102:0/1793381408 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1149035317 0 0) 0x7ffaa0003270 con 0x7ffabc093fc0 2021-05-25T12:07:00.509 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.492+0000 7ffab959a700 1 -- 172.21.15.102:0/1793381408 >> v1:172.21.15.102:6790/0 conn(0x7ffabc0692d0 legacy=0x7ffabc096a50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-05-25T12:07:00.509 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.492+0000 7ffab959a700 1 -- 172.21.15.102:0/1793381408 >> v1:172.21.15.181:6789/0 conn(0x7ffabc12f130 legacy=0x7ffabc131520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:00.509 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.492+0000 7ffab959a700 10 monclient: _finish_hunting 0 2021-05-25T12:07:00.509 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.492+0000 7ffab959a700 1 monclient: found mon.noname-a 2021-05-25T12:07:00.510 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.492+0000 7ffab959a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:07:00.510 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.492+0000 7ffab959a700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T12:07:00.510 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.492+0000 7ffab959a700 1 -- 172.21.15.102:0/1793381408 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffabc093520 con 0x7ffabc093fc0 2021-05-25T12:07:00.510 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.492+0000 7ffab959a700 10 monclient: _finish_auth 0 2021-05-25T12:07:00.511 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.492+0000 7ffab959a700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:07:00.511 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.492+0000 7ffac0f1d700 5 monclient: authenticate success, global_id 204306 2021-05-25T12:07:00.511 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.492+0000 7ffab959a700 1 -- 172.21.15.102:0/1793381408 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7ffaa4001610 con 0x7ffabc093fc0 2021-05-25T12:07:00.511 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.492+0000 7ffac0f1d700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:07:00.511 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.492+0000 7ffab959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:00.512 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.492+0000 7ffab959a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:07:00.512 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.492+0000 7ffab959a700 10 monclient: dump: 2021-05-25T12:07:00.512 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:00.512 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:00.512 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:00.513 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:00.513 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:00.513 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:00.513 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:00.513 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:00.514 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:00.514 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:00.514 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.492+0000 7ffac0f1d700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:07:00.514 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.493+0000 7ffab959a700 1 -- 172.21.15.102:0/1793381408 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffaa4001930 con 0x7ffabc093fc0 2021-05-25T12:07:00.514 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.493+0000 7ffab959a700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:07:00.515 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.493+0000 7ffab959a700 1 -- 172.21.15.102:0/1793381408 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7ffaa4001cc0 con 0x7ffabc093fc0 2021-05-25T12:07:00.515 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.493+0000 7ffac0f1d700 10 monclient: get_monmap_and_config success 2021-05-25T12:07:00.515 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.493+0000 7ffac0f1d700 4 set_mon_vals no callback set 2021-05-25T12:07:00.515 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.493+0000 7ffab959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:00.515 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.493+0000 7ffab959a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:07:00.516 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.493+0000 7ffab959a700 10 monclient: dump: 2021-05-25T12:07:00.516 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:00.516 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:00.516 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:00.517 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:00.517 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:00.517 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:00.517 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:00.518 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:00.518 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:00.518 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:00.518 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.493+0000 7ffac0f1d700 10 monclient: shutdown 2021-05-25T12:07:00.518 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.493+0000 7ffac0f1d700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:07:00.519 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.493+0000 7ffac0f1d700 1 -- 172.21.15.102:0/1793381408 >> v1:172.21.15.102:6789/0 conn(0x7ffabc093fc0 legacy=0x7ffabc096340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:00.519 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.493+0000 7ffac0f1d700 1 -- 172.21.15.102:0/1793381408 shutdown_connections 2021-05-25T12:07:00.519 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.493+0000 7ffac0f1d700 1 -- 172.21.15.102:0/1793381408 >> 172.21.15.102:0/1793381408 conn(0x7ffabc09b370 msgr2=0x7ffabc097430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:07:00.519 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.493+0000 7ffac0f1d700 1 -- 172.21.15.102:0/1793381408 shutdown_connections 2021-05-25T12:07:00.519 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.493+0000 7ffac0f1d700 1 -- 172.21.15.102:0/1793381408 wait complete. 2021-05-25T12:07:00.520 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.494+0000 7ffac0f1d700 10 monclient: build_initial_monmap 2021-05-25T12:07:00.520 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.494+0000 7ffac0f1d700 10 monclient: monmap: 2021-05-25T12:07:00.520 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:07:00.520 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:07:00.521 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:07:00.521 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:07:00.521 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:07:00.521 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:07:00.521 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:07:00.522 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:07:00.522 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:07:00.522 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:00.522 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.494+0000 7ffac0f1d700 1 librados: starting msgr at 2021-05-25T12:07:00.522 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.494+0000 7ffac0f1d700 1 librados: starting objecter 2021-05-25T12:07:00.523 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.494+0000 7ffac0f1d700 1 Processor -- start 2021-05-25T12:07:00.523 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.494+0000 7ffac0f1d700 1 -- start start 2021-05-25T12:07:00.523 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.494+0000 7ffac0f1d700 1 librados: setting wanted keys 2021-05-25T12:07:00.523 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.494+0000 7ffac0f1d700 1 librados: calling monclient init 2021-05-25T12:07:00.524 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.494+0000 7ffac0f1d700 10 monclient: init 2021-05-25T12:07:00.524 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.494+0000 7ffac0f1d700 10 monclient: _reopen_session rank -1 2021-05-25T12:07:00.524 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.494+0000 7ffac0f1d700 10 monclient: _add_conns ranks=[2,1,0] 2021-05-25T12:07:00.524 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.494+0000 7ffac0f1d700 10 monclient(hunting): picked mon.noname-c con 0x7ffabc0692d0 addr v1:172.21.15.102:6790/0 2021-05-25T12:07:00.525 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.494+0000 7ffac0f1d700 10 monclient(hunting): picked mon.noname-b con 0x7ffabc093fc0 addr v1:172.21.15.181:6789/0 2021-05-25T12:07:00.525 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.494+0000 7ffac0f1d700 10 monclient(hunting): picked mon.noname-a con 0x7ffabc12f130 addr v1:172.21.15.102:6789/0 2021-05-25T12:07:00.525 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.494+0000 7ffac0f1d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffabc12aec0 con 0x7ffabc12f130 2021-05-25T12:07:00.525 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.494+0000 7ffac0f1d700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffabc1161d0 con 0x7ffabc093fc0 2021-05-25T12:07:00.525 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.494+0000 7ffac0f1d700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffabc092350 con 0x7ffabc0692d0 2021-05-25T12:07:00.526 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.494+0000 7ffac0f1d700 10 monclient(hunting): _renew_subs 2021-05-25T12:07:00.526 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.494+0000 7ffac0f1d700 10 monclient(hunting): authenticate will time out at 1221.091923s 2021-05-25T12:07:00.526 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.494+0000 7ffaba59c700 1 -- >> v1:172.21.15.102:6790/0 conn(0x7ffabc0692d0 legacy=0x7ffabc12a7b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.102:6790/0 2021-05-25T12:07:00.526 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.495+0000 7ffabad9d700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7ffabc12f130 0x7ffabc1415d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33656/0 (socket says 172.21.15.102:33656) 2021-05-25T12:07:00.526 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.495+0000 7ffabad9d700 1 -- 172.21.15.102:0/2230878910 learned_addr learned my addr 172.21.15.102:0/2230878910 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:07:00.527 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.495+0000 7ffaaa7fc700 10 client.?.objecter ms_handle_connect 0x7ffabc12f130 2021-05-25T12:07:00.527 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.495+0000 7ffaaa7fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:07:00.527 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.495+0000 7ffaaa7fc700 10 client.?.objecter ms_handle_connect 0x7ffabc093fc0 2021-05-25T12:07:00.528 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.495+0000 7ffaaa7fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:07:00.528 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.495+0000 7ffaaa7fc700 1 -- 172.21.15.102:0/2230878910 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1539162998 0 0) 0x7ffabc12aec0 con 0x7ffabc12f130 2021-05-25T12:07:00.528 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.495+0000 7ffaaa7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:00.528 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.495+0000 7ffaaa7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:00.528 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.495+0000 7ffaaa7fc700 10 monclient(hunting): my global_id is 204309 2021-05-25T12:07:00.529 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.495+0000 7ffaaa7fc700 1 -- 172.21.15.102:0/2230878910 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa98004c10 con 0x7ffabc12f130 2021-05-25T12:07:00.529 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.495+0000 7ffaaa7fc700 1 -- 172.21.15.102:0/2230878910 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4292949563 0 0) 0x7ffabc1161d0 con 0x7ffabc093fc0 2021-05-25T12:07:00.529 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.495+0000 7ffaaa7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:00.529 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.495+0000 7ffaaa7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:00.529 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.495+0000 7ffaaa7fc700 10 monclient(hunting): my global_id is 224176 2021-05-25T12:07:00.530 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.495+0000 7ffaaa7fc700 1 -- 172.21.15.102:0/2230878910 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffabc12aec0 con 0x7ffabc093fc0 2021-05-25T12:07:00.530 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.495+0000 7ffaaa7fc700 1 -- 172.21.15.102:0/2230878910 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2908992202 0 0) 0x7ffa98004c10 con 0x7ffabc12f130 2021-05-25T12:07:00.530 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.495+0000 7ffaaa7fc700 1 -- 172.21.15.102:0/2230878910 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffabc1161d0 con 0x7ffabc12f130 2021-05-25T12:07:00.530 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.495+0000 7ffaaa7fc700 1 -- 172.21.15.102:0/2230878910 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7ffaac001cd0 con 0x7ffabc12f130 2021-05-25T12:07:00.531 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.495+0000 7ffaaa7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:00.531 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.495+0000 7ffaaa7fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:07:00.531 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.495+0000 7ffaaa7fc700 10 monclient(hunting): dump: 2021-05-25T12:07:00.531 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:00.531 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:00.532 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:00.532 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:00.532 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:00.532 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:00.532 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:00.533 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:00.533 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:00.533 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:00.533 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.495+0000 7ffaaa7fc700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:07:00.533 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.495+0000 7ffaaa7fc700 1 -- 172.21.15.102:0/2230878910 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2095777565 0 0) 0x7ffabc12aec0 con 0x7ffabc093fc0 2021-05-25T12:07:00.534 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.495+0000 7ffaaa7fc700 1 -- 172.21.15.102:0/2230878910 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffa98004c10 con 0x7ffabc093fc0 2021-05-25T12:07:00.534 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.495+0000 7ffaaa7fc700 1 -- 172.21.15.102:0/2230878910 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 715199803 0 0) 0x7ffabc1161d0 con 0x7ffabc12f130 2021-05-25T12:07:00.534 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.495+0000 7ffaaa7fc700 1 -- 172.21.15.102:0/2230878910 >> v1:172.21.15.102:6790/0 conn(0x7ffabc0692d0 legacy=0x7ffabc12a7b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-05-25T12:07:00.534 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.495+0000 7ffaaa7fc700 1 -- 172.21.15.102:0/2230878910 >> v1:172.21.15.181:6789/0 conn(0x7ffabc093fc0 legacy=0x7ffabc13dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:00.534 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.495+0000 7ffaaa7fc700 10 monclient: _finish_hunting 0 2021-05-25T12:07:00.535 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.495+0000 7ffaaa7fc700 1 monclient: found mon.a 2021-05-25T12:07:00.535 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.495+0000 7ffaaa7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:07:00.535 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.495+0000 7ffaaa7fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:07:00.535 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.495+0000 7ffaaa7fc700 1 -- 172.21.15.102:0/2230878910 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffabc091570 con 0x7ffabc12f130 2021-05-25T12:07:00.536 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.495+0000 7ffaaa7fc700 10 monclient: _finish_auth 0 2021-05-25T12:07:00.536 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.495+0000 7ffaaa7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:07:00.536 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.495+0000 7ffac0f1d700 5 monclient: authenticate success, global_id 204309 2021-05-25T12:07:00.536 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.495+0000 7ffac0f1d700 10 monclient: _renew_subs 2021-05-25T12:07:00.536 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.495+0000 7ffac0f1d700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:07:00.537 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.495+0000 7ffac0f1d700 1 -- 172.21.15.102:0/2230878910 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffabc136380 con 0x7ffabc12f130 2021-05-25T12:07:00.537 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.495+0000 7ffac0f1d700 10 client.204309.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:07:00.537 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.495+0000 7ffac0f1d700 10 monclient: _renew_subs 2021-05-25T12:07:00.537 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.495+0000 7ffac0f1d700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:07:00.537 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.495+0000 7ffac0f1d700 1 -- 172.21.15.102:0/2230878910 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffabc114d00 con 0x7ffabc12f130 2021-05-25T12:07:00.538 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.495+0000 7ffac0f1d700 1 librados: init done 2021-05-25T12:07:00.538 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.495+0000 7ffaaa7fc700 1 -- 172.21.15.102:0/2230878910 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffaac002060 con 0x7ffabc12f130 2021-05-25T12:07:00.538 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.495+0000 7ffaaa7fc700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:07:00.538 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.495+0000 7ffaaa7fc700 1 -- 172.21.15.102:0/2230878910 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7ffaac0027d0 con 0x7ffabc12f130 2021-05-25T12:07:00.538 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.495+0000 7ffaaa7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:00.539 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.495+0000 7ffaaa7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:07:00.539 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.495+0000 7ffab8d99700 4 set_mon_vals no callback set 2021-05-25T12:07:00.539 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.495+0000 7ffaaa7fc700 10 monclient: dump: 2021-05-25T12:07:00.539 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:00.540 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:00.540 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:00.540 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:00.540 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:00.540 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:00.541 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:00.541 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:00.541 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:00.541 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:00.541 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.496+0000 7ffaaa7fc700 1 -- 172.21.15.102:0/2230878910 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7ffaac0029b0 con 0x7ffabc12f130 2021-05-25T12:07:00.542 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.496+0000 7ffac0f1d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:07:00.542 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.496+0000 7ffac0f1d700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:07:00.542 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.496+0000 7ffac0f1d700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:07:00.542 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.496+0000 7ffac0f1d700 1 -- 172.21.15.102:0/2230878910 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa9c004db0 con 0x7ffabc12f130 2021-05-25T12:07:00.543 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.496+0000 7ffaaa7fc700 1 -- 172.21.15.102:0/2230878910 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(221..221 src has 1..221) v4 ==== 6648+0+0 (unknown 2376337231 0 0) 0x7ffaac03de80 con 0x7ffabc12f130 2021-05-25T12:07:00.543 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.496+0000 7ffaaa7fc700 10 client.204309.objecter ms_dispatch 0x7ffabc001550 osd_map(221..221 src has 1..221) v4 2021-05-25T12:07:00.543 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.496+0000 7ffaaa7fc700 3 client.204309.objecter handle_osd_map got epochs [221,221] > 0 2021-05-25T12:07:00.543 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.496+0000 7ffaaa7fc700 3 client.204309.objecter handle_osd_map decoding full epoch 221 2021-05-25T12:07:00.543 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.496+0000 7ffaaa7fc700 20 client.204309.objecter dump_active .. 0 homeless 2021-05-25T12:07:00.544 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.497+0000 7ffaba59c700 10 monclient: get_auth_request con 0x7ffa98033260 auth_method 0 2021-05-25T12:07:00.544 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.497+0000 7ffaaa7fc700 10 client.204309.objecter ms_handle_connect 0x7ffa98033260 2021-05-25T12:07:00.544 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.501+0000 7ffaaa7fc700 1 -- 172.21.15.102:0/2230878910 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7ffaac012fe0 con 0x7ffabc12f130 2021-05-25T12:07:00.544 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.501+0000 7ffaaa7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:07:00.545 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.501+0000 7ffaaa7fc700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:07:00.710 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.708+0000 7ffac0f1d700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2021-05-25T12:07:00.710 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.708+0000 7ffac0f1d700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2021-05-25T12:07:00.711 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.708+0000 7ffac0f1d700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:07:00.711 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.708+0000 7ffac0f1d700 1 -- 172.21.15.102:0/2230878910 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7ffa9c005060 con 0x7ffabc12f130 2021-05-25T12:07:00.711 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.708+0000 7ffaaa7fc700 1 -- 172.21.15.102:0/2230878910 <== mon.0 v1:172.21.15.102:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1342 (unknown 254746657 0 1928635387) 0x7ffaac012e00 con 0x7ffabc12f130 2021-05-25T12:07:00.711 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.708+0000 7ffaaa7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2021-05-25T12:07:00.712 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.708+0000 7ffaaa7fc700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:07:00.712 INFO:teuthology.orchestra.run.smithi102.stdout:{"election_epoch":58,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":7,"features":{"quorum_con":"4540138303579357183","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"20ee91b6-8935-4b98-b868-85c03789ccc4","modified":"2021-05-25T11:59:11.011522Z","created":"2021-05-25T11:59:11.011522Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6790","nonce":0}]},"addr":"172.21.15.102:6790/0","public_addr":"172.21.15.102:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2021-05-25T12:07:00.713 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.710+0000 7ffac0f1d700 10 librados: watch_flush enter 2021-05-25T12:07:00.714 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.710+0000 7ffac0f1d700 10 librados: watch_flush exit 2021-05-25T12:07:00.714 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.710+0000 7ffac0f1d700 20 client.204309.objecter shutdown clearing up homeless session... 2021-05-25T12:07:00.714 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.710+0000 7ffac0f1d700 10 client.204309.objecter successfully canceled tick 2021-05-25T12:07:00.714 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.710+0000 7ffac0f1d700 1 -- 172.21.15.102:0/2230878910 >> v1:172.21.15.181:6812/29582 conn(0x7ffa98033260 legacy=0x7ffa98035720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:00.715 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.710+0000 7ffac0f1d700 10 monclient: shutdown 2021-05-25T12:07:00.715 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.710+0000 7ffac0f1d700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:07:00.715 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.710+0000 7ffac0f1d700 1 -- 172.21.15.102:0/2230878910 >> v1:172.21.15.102:6789/0 conn(0x7ffabc12f130 legacy=0x7ffabc1415d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:00.715 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.710+0000 7ffac0f1d700 1 -- 172.21.15.102:0/2230878910 shutdown_connections 2021-05-25T12:07:00.716 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.710+0000 7ffac0f1d700 1 -- 172.21.15.102:0/2230878910 >> 172.21.15.102:0/2230878910 conn(0x7ffabc09b370 msgr2=0x7ffabc12d440 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:07:00.716 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.710+0000 7ffac0f1d700 1 -- 172.21.15.102:0/2230878910 shutdown_connections 2021-05-25T12:07:00.716 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.710+0000 7ffac0f1d700 1 -- 172.21.15.102:0/2230878910 wait complete. 2021-05-25T12:07:00.716 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:00.710+0000 7ffac0f1d700 1 librados: shutdown 2021-05-25T12:07:03.724 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2021-05-25T12:07:03.843 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:03.841+0000 7ff396964700 10 monclient: get_monmap_and_config 2021-05-25T12:07:03.843 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:03.841+0000 7ff396964700 10 monclient: build_initial_monmap 2021-05-25T12:07:03.844 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:03.841+0000 7ff396964700 10 monclient: monmap: 2021-05-25T12:07:03.844 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:07:03.844 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:03.844 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:07:03.842699+0000 2021-05-25T12:07:03.845 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:07:03.842699+0000 2021-05-25T12:07:03.845 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:07:03.845 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:03.845 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:07:03.846 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:07:03.846 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:07:03.846 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:03.846 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:03.842+0000 7ff396964700 1 Processor -- start 2021-05-25T12:07:03.846 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:03.842+0000 7ff396964700 1 -- start start 2021-05-25T12:07:03.847 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:03.842+0000 7ff396964700 10 monclient: init 2021-05-25T12:07:03.848 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:03.842+0000 7ff396964700 10 monclient: _reopen_session rank -1 2021-05-25T12:07:03.848 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:03.842+0000 7ff396964700 10 monclient: _add_conns ranks=[1,0,2] 2021-05-25T12:07:03.849 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:03.842+0000 7ff396964700 10 monclient(hunting): picked mon.noname-c con 0x7ff39012b7a0 addr v1:172.21.15.181:6789/0 2021-05-25T12:07:03.849 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:03.842+0000 7ff396964700 10 monclient(hunting): picked mon.noname-a con 0x7ff390098700 addr v1:172.21.15.102:6789/0 2021-05-25T12:07:03.849 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:03.842+0000 7ff396964700 10 monclient(hunting): picked mon.noname-b con 0x7ff390098b70 addr v1:172.21.15.102:6790/0 2021-05-25T12:07:03.849 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:03.842+0000 7ff396964700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff390063fa0 con 0x7ff390098700 2021-05-25T12:07:03.850 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:03.843+0000 7ff396964700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff39012c570 con 0x7ff39012b7a0 2021-05-25T12:07:03.850 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:03.843+0000 7ff396964700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff39009af40 con 0x7ff390098b70 2021-05-25T12:07:03.850 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:03.843+0000 7ff396964700 10 monclient(hunting): _renew_subs 2021-05-25T12:07:03.850 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:03.843+0000 7ff396964700 10 monclient(hunting): authenticate will time out at 1224.440025s 2021-05-25T12:07:03.850 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:03.843+0000 7ff396163700 1 -- >> v1:172.21.15.102:6790/0 conn(0x7ff390098b70 legacy=0x7ff390134260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6790/0 2021-05-25T12:07:03.851 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:03.843+0000 7ff395161700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7ff390098700 0x7ff390130b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33662/0 (socket says 172.21.15.102:33662) 2021-05-25T12:07:03.851 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:03.843+0000 7ff395161700 1 -- 172.21.15.102:0/1603863509 learned_addr learned my addr 172.21.15.102:0/1603863509 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:07:04.044 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:04.044+0000 7ff396163700 1 -- 172.21.15.102:0/1603863509 >> v1:172.21.15.102:6790/0 conn(0x7ff390098b70 legacy=0x7ff390134260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6790/0 2021-05-25T12:07:04.446 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:04.445+0000 7ff396163700 1 -- 172.21.15.102:0/1603863509 >> v1:172.21.15.102:6790/0 conn(0x7ff390098b70 legacy=0x7ff390134260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6790/0 2021-05-25T12:07:05.247 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:05.246+0000 7ff396163700 1 -- 172.21.15.102:0/1603863509 >> v1:172.21.15.102:6790/0 conn(0x7ff390098b70 legacy=0x7ff390134260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6790/0 2021-05-25T12:07:06.845 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:06.843+0000 7ff37f7fe700 10 monclient(hunting): tick 2021-05-25T12:07:06.845 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:06.843+0000 7ff37f7fe700 1 monclient(hunting): continuing hunt 2021-05-25T12:07:06.845 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:06.843+0000 7ff37f7fe700 10 monclient(hunting): _reopen_session rank -1 2021-05-25T12:07:06.846 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:06.843+0000 7ff37f7fe700 1 -- 172.21.15.102:0/1603863509 >> v1:172.21.15.102:6790/0 conn(0x7ff390098b70 legacy=0x7ff390134260 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-05-25T12:07:06.846 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:06.843+0000 7ff37f7fe700 1 -- 172.21.15.102:0/1603863509 >> v1:172.21.15.181:6789/0 conn(0x7ff39012b7a0 legacy=0x7ff39012be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:06.846 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:06.843+0000 7ff37f7fe700 1 -- 172.21.15.102:0/1603863509 >> v1:172.21.15.102:6789/0 conn(0x7ff390098700 legacy=0x7ff390130b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:06.847 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:06.843+0000 7ff37f7fe700 10 monclient: _add_conns ranks=[2,1,0] 2021-05-25T12:07:06.847 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:06.843+0000 7ff37f7fe700 10 monclient(hunting): picked mon.noname-b con 0x7ff390098b70 addr v1:172.21.15.102:6790/0 2021-05-25T12:07:06.847 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:06.843+0000 7ff37f7fe700 10 monclient(hunting): picked mon.noname-c con 0x7ff39012b7a0 addr v1:172.21.15.181:6789/0 2021-05-25T12:07:06.847 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:06.843+0000 7ff37f7fe700 10 monclient(hunting): picked mon.noname-a con 0x7ff390098700 addr v1:172.21.15.102:6789/0 2021-05-25T12:07:06.848 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:06.843+0000 7ff37f7fe700 1 -- 172.21.15.102:0/1603863509 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff39009b1e0 con 0x7ff390098700 2021-05-25T12:07:06.848 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:06.843+0000 7ff37f7fe700 1 -- 172.21.15.102:0/1603863509 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff39009af40 con 0x7ff39012b7a0 2021-05-25T12:07:06.848 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:06.843+0000 7ff37f7fe700 1 -- 172.21.15.102:0/1603863509 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff37000fb90 con 0x7ff390098b70 2021-05-25T12:07:06.848 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:06.843+0000 7ff37f7fe700 10 monclient(hunting): _renew_subs 2021-05-25T12:07:06.848 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:06.843+0000 7ff395962700 1 -- 172.21.15.102:0/1603863509 >> v1:172.21.15.102:6790/0 conn(0x7ff390098b70 legacy=0x7ff370005430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6790/0 2021-05-25T12:07:07.045 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:07.044+0000 7ff395962700 1 -- 172.21.15.102:0/1603863509 >> v1:172.21.15.102:6790/0 conn(0x7ff390098b70 legacy=0x7ff370005430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6790/0 2021-05-25T12:07:07.446 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:07.445+0000 7ff395962700 1 -- 172.21.15.102:0/1603863509 >> v1:172.21.15.102:6790/0 conn(0x7ff390098b70 legacy=0x7ff370005430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6790/0 2021-05-25T12:07:08.246 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:08.246+0000 7ff395962700 1 -- 172.21.15.102:0/1603863509 >> v1:172.21.15.102:6790/0 conn(0x7ff390098b70 legacy=0x7ff370005430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6790/0 2021-05-25T12:07:09.846 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:09.843+0000 7ff37f7fe700 10 monclient(hunting): tick 2021-05-25T12:07:09.846 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:09.843+0000 7ff37f7fe700 1 monclient(hunting): continuing hunt 2021-05-25T12:07:09.847 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:09.843+0000 7ff37f7fe700 10 monclient(hunting): _reopen_session rank -1 2021-05-25T12:07:09.847 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:09.843+0000 7ff37f7fe700 1 -- 172.21.15.102:0/1603863509 >> v1:172.21.15.102:6790/0 conn(0x7ff390098b70 legacy=0x7ff370005430 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-05-25T12:07:09.847 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:09.843+0000 7ff37f7fe700 1 -- 172.21.15.102:0/1603863509 >> v1:172.21.15.181:6789/0 conn(0x7ff39012b7a0 legacy=0x7ff370008bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:09.847 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:09.843+0000 7ff37f7fe700 1 -- 172.21.15.102:0/1603863509 >> v1:172.21.15.102:6789/0 conn(0x7ff390098700 legacy=0x7ff37000c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:09.848 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:09.843+0000 7ff37f7fe700 10 monclient: _add_conns ranks=[1,2,0] 2021-05-25T12:07:09.848 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:09.843+0000 7ff37f7fe700 10 monclient(hunting): picked mon.noname-c con 0x7ff37000fe20 addr v1:172.21.15.181:6789/0 2021-05-25T12:07:09.848 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:09.843+0000 7ff37f7fe700 10 monclient(hunting): picked mon.noname-b con 0x7ff390098b70 addr v1:172.21.15.102:6790/0 2021-05-25T12:07:09.848 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:09.843+0000 7ff37f7fe700 10 monclient(hunting): picked mon.noname-a con 0x7ff3700092f0 addr v1:172.21.15.102:6789/0 2021-05-25T12:07:09.849 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:09.843+0000 7ff37f7fe700 1 -- 172.21.15.102:0/1603863509 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff370010e00 con 0x7ff3700092f0 2021-05-25T12:07:09.849 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:09.843+0000 7ff37f7fe700 1 -- 172.21.15.102:0/1603863509 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff37000fb90 con 0x7ff37000fe20 2021-05-25T12:07:09.849 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:09.843+0000 7ff37f7fe700 1 -- 172.21.15.102:0/1603863509 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff370009770 con 0x7ff390098b70 2021-05-25T12:07:09.849 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:09.843+0000 7ff37f7fe700 10 monclient(hunting): _renew_subs 2021-05-25T12:07:09.849 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:09.843+0000 7ff395161700 1 -- 172.21.15.102:0/1603863509 >> v1:172.21.15.102:6790/0 conn(0x7ff390098b70 legacy=0x7ff370005430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6790/0 2021-05-25T12:07:10.045 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:10.044+0000 7ff395161700 1 -- 172.21.15.102:0/1603863509 >> v1:172.21.15.102:6790/0 conn(0x7ff390098b70 legacy=0x7ff370005430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6790/0 2021-05-25T12:07:10.446 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:10.445+0000 7ff395161700 1 -- 172.21.15.102:0/1603863509 >> v1:172.21.15.102:6790/0 conn(0x7ff390098b70 legacy=0x7ff370005430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6790/0 2021-05-25T12:07:11.247 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.246+0000 7ff395161700 1 -- 172.21.15.102:0/1603863509 >> v1:172.21.15.102:6790/0 conn(0x7ff390098b70 legacy=0x7ff370005430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6790/0 2021-05-25T12:07:11.842 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.837+0000 7ff394960700 1 -- 172.21.15.102:0/1603863509 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1467805331 0 0) 0x7ff370010e00 con 0x7ff3700092f0 2021-05-25T12:07:11.843 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.837+0000 7ff394960700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:11.843 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.837+0000 7ff394960700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:11.843 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.837+0000 7ff394960700 10 monclient(hunting): my global_id is 204312 2021-05-25T12:07:11.843 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.837+0000 7ff394960700 1 -- 172.21.15.102:0/1603863509 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff378003270 con 0x7ff3700092f0 2021-05-25T12:07:11.844 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.837+0000 7ff394960700 1 -- 172.21.15.102:0/1603863509 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1517530823 0 0) 0x7ff378003270 con 0x7ff3700092f0 2021-05-25T12:07:11.844 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.837+0000 7ff394960700 1 -- 172.21.15.102:0/1603863509 >> v1:172.21.15.102:6790/0 conn(0x7ff390098b70 legacy=0x7ff370005430 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-05-25T12:07:11.844 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.837+0000 7ff394960700 1 -- 172.21.15.102:0/1603863509 >> v1:172.21.15.181:6789/0 conn(0x7ff37000fe20 legacy=0x7ff370010200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:11.844 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.837+0000 7ff394960700 10 monclient: _finish_hunting 0 2021-05-25T12:07:11.845 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.837+0000 7ff394960700 1 monclient: found mon.noname-a 2021-05-25T12:07:11.845 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.837+0000 7ff394960700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:07:11.845 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.837+0000 7ff394960700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T12:07:11.845 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.837+0000 7ff394960700 1 -- 172.21.15.102:0/1603863509 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff370009900 con 0x7ff3700092f0 2021-05-25T12:07:11.845 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.837+0000 7ff394960700 10 monclient: _finish_auth 0 2021-05-25T12:07:11.846 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.837+0000 7ff394960700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:07:11.846 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.837+0000 7ff394960700 1 -- 172.21.15.102:0/1603863509 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7ff38c004210 con 0x7ff3700092f0 2021-05-25T12:07:11.846 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.837+0000 7ff396964700 5 monclient: authenticate success, global_id 204312 2021-05-25T12:07:11.846 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.837+0000 7ff396964700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:07:11.847 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.837+0000 7ff394960700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:11.847 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.837+0000 7ff394960700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:07:11.847 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.837+0000 7ff394960700 10 monclient: dump: 2021-05-25T12:07:11.847 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:11.848 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:11.848 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:11.848 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:11.848 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:11.848 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:11.849 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:11.849 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:11.849 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:11.849 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:11.849 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.837+0000 7ff396964700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:07:11.850 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.837+0000 7ff394960700 1 -- 172.21.15.102:0/1603863509 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff38c0044b0 con 0x7ff3700092f0 2021-05-25T12:07:11.850 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.837+0000 7ff394960700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:07:11.850 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.837+0000 7ff394960700 1 -- 172.21.15.102:0/1603863509 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7ff38c004880 con 0x7ff3700092f0 2021-05-25T12:07:11.850 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.837+0000 7ff396964700 10 monclient: get_monmap_and_config success 2021-05-25T12:07:11.851 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.837+0000 7ff396964700 4 set_mon_vals no callback set 2021-05-25T12:07:11.851 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.837+0000 7ff394960700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:11.851 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.837+0000 7ff394960700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:07:11.851 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.837+0000 7ff394960700 10 monclient: dump: 2021-05-25T12:07:11.851 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:11.852 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:11.852 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:11.852 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:11.852 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:11.853 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:11.853 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:11.853 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:11.853 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:11.853 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:11.854 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.838+0000 7ff396964700 10 monclient: shutdown 2021-05-25T12:07:11.854 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.838+0000 7ff396964700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:07:11.854 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.838+0000 7ff396964700 1 -- 172.21.15.102:0/1603863509 >> v1:172.21.15.102:6789/0 conn(0x7ff3700092f0 legacy=0x7ff370008bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:11.854 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.838+0000 7ff396964700 1 -- 172.21.15.102:0/1603863509 shutdown_connections 2021-05-25T12:07:11.854 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.838+0000 7ff396964700 1 -- 172.21.15.102:0/1603863509 >> 172.21.15.102:0/1603863509 conn(0x7ff39009eb80 msgr2=0x7ff39009ef90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:07:11.855 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.838+0000 7ff396964700 1 -- 172.21.15.102:0/1603863509 shutdown_connections 2021-05-25T12:07:11.855 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.838+0000 7ff396964700 1 -- 172.21.15.102:0/1603863509 wait complete. 2021-05-25T12:07:11.855 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.839+0000 7ff396964700 10 monclient: build_initial_monmap 2021-05-25T12:07:11.855 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.839+0000 7ff396964700 10 monclient: monmap: 2021-05-25T12:07:11.856 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:07:11.856 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:07:11.856 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:07:11.856 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:07:11.856 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:07:11.857 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:07:11.857 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:07:11.857 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:07:11.857 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:07:11.857 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:11.858 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.839+0000 7ff396964700 1 librados: starting msgr at 2021-05-25T12:07:11.858 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.839+0000 7ff396964700 1 librados: starting objecter 2021-05-25T12:07:11.858 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.839+0000 7ff396964700 1 Processor -- start 2021-05-25T12:07:11.858 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.839+0000 7ff396964700 1 -- start start 2021-05-25T12:07:11.859 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.839+0000 7ff396964700 1 librados: setting wanted keys 2021-05-25T12:07:11.859 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.839+0000 7ff396964700 1 librados: calling monclient init 2021-05-25T12:07:11.859 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.839+0000 7ff396964700 10 monclient: init 2021-05-25T12:07:11.859 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.840+0000 7ff396964700 10 monclient: _reopen_session rank -1 2021-05-25T12:07:11.859 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.840+0000 7ff396964700 10 monclient: _add_conns ranks=[0,1,2] 2021-05-25T12:07:11.860 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.840+0000 7ff396964700 10 monclient(hunting): picked mon.noname-a con 0x7ff3700092f0 addr v1:172.21.15.102:6789/0 2021-05-25T12:07:11.860 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.840+0000 7ff396964700 10 monclient(hunting): picked mon.noname-b con 0x7ff37000fe20 addr v1:172.21.15.181:6789/0 2021-05-25T12:07:11.860 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.840+0000 7ff396964700 10 monclient(hunting): picked mon.noname-c con 0x7ff390098b70 addr v1:172.21.15.102:6790/0 2021-05-25T12:07:11.860 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.840+0000 7ff396964700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3901195b0 con 0x7ff3700092f0 2021-05-25T12:07:11.860 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.840+0000 7ff396964700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff390118a50 con 0x7ff37000fe20 2021-05-25T12:07:11.861 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.840+0000 7ff396964700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3901281e0 con 0x7ff390098b70 2021-05-25T12:07:11.861 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.840+0000 7ff396964700 10 monclient(hunting): _renew_subs 2021-05-25T12:07:11.861 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.840+0000 7ff396964700 10 monclient(hunting): authenticate will time out at 1232.437310s 2021-05-25T12:07:11.861 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.840+0000 7ff396163700 1 -- >> v1:172.21.15.102:6790/0 conn(0x7ff390098b70 legacy=0x7ff390143860 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.102:6790/0 2021-05-25T12:07:11.861 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.840+0000 7ff395962700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7ff3700092f0 0x7ff390115890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33702/0 (socket says 172.21.15.102:33702) 2021-05-25T12:07:11.862 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.840+0000 7ff395962700 1 -- 172.21.15.102:0/4168630074 learned_addr learned my addr 172.21.15.102:0/4168630074 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:07:11.862 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.840+0000 7ff37dffb700 10 client.?.objecter ms_handle_connect 0x7ff3700092f0 2021-05-25T12:07:11.862 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.840+0000 7ff37dffb700 10 client.?.objecter resend_mon_ops 2021-05-25T12:07:11.862 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.840+0000 7ff37dffb700 10 client.?.objecter ms_handle_connect 0x7ff37000fe20 2021-05-25T12:07:11.863 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.840+0000 7ff37dffb700 10 client.?.objecter resend_mon_ops 2021-05-25T12:07:11.868 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.844+0000 7ff37dffb700 1 -- 172.21.15.102:0/4168630074 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3290648486 0 0) 0x7ff390118a50 con 0x7ff37000fe20 2021-05-25T12:07:11.868 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.844+0000 7ff37dffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:11.868 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.844+0000 7ff37dffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:11.868 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.844+0000 7ff37dffb700 10 monclient(hunting): my global_id is 224179 2021-05-25T12:07:11.869 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.844+0000 7ff37dffb700 1 -- 172.21.15.102:0/4168630074 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff370009750 con 0x7ff37000fe20 2021-05-25T12:07:11.869 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.846+0000 7ff37dffb700 1 -- 172.21.15.102:0/4168630074 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 839206302 0 0) 0x7ff3901195b0 con 0x7ff3700092f0 2021-05-25T12:07:11.869 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.846+0000 7ff37dffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:11.869 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.846+0000 7ff37dffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:11.870 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.846+0000 7ff37dffb700 10 monclient(hunting): my global_id is 204315 2021-05-25T12:07:11.870 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.846+0000 7ff37dffb700 1 -- 172.21.15.102:0/4168630074 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff390118a50 con 0x7ff3700092f0 2021-05-25T12:07:11.870 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.846+0000 7ff37dffb700 1 -- 172.21.15.102:0/4168630074 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3736311093 0 0) 0x7ff370009750 con 0x7ff37000fe20 2021-05-25T12:07:11.870 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.846+0000 7ff37dffb700 1 -- 172.21.15.102:0/4168630074 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff3901195b0 con 0x7ff37000fe20 2021-05-25T12:07:11.871 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.846+0000 7ff37dffb700 1 -- 172.21.15.102:0/4168630074 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7ff380001730 con 0x7ff37000fe20 2021-05-25T12:07:11.871 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.846+0000 7ff37dffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:11.871 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.846+0000 7ff37dffb700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:07:11.871 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.846+0000 7ff37dffb700 10 monclient(hunting): dump: 2021-05-25T12:07:11.872 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:11.872 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:11.872 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:11.872 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:11.873 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:11.873 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:11.873 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:11.873 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:11.874 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:11.874 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:11.874 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.846+0000 7ff37dffb700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:07:11.874 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.847+0000 7ff37dffb700 1 -- 172.21.15.102:0/4168630074 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2987014517 0 0) 0x7ff390118a50 con 0x7ff3700092f0 2021-05-25T12:07:11.874 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.847+0000 7ff37dffb700 1 -- 172.21.15.102:0/4168630074 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff370009750 con 0x7ff3700092f0 2021-05-25T12:07:11.875 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.847+0000 7ff37dffb700 1 -- 172.21.15.102:0/4168630074 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7ff3840018c0 con 0x7ff3700092f0 2021-05-25T12:07:11.875 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.847+0000 7ff37dffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:11.875 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.847+0000 7ff37dffb700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2021-05-25T12:07:11.875 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.847+0000 7ff37dffb700 10 monclient(hunting): dump: 2021-05-25T12:07:11.876 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:11.876 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:11.876 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:11.876 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:11.877 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:11.877 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:11.877 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:11.877 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:11.878 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:11.878 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:11.878 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.847+0000 7ff37dffb700 1 -- 172.21.15.102:0/4168630074 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3266348976 0 0) 0x7ff3901195b0 con 0x7ff37000fe20 2021-05-25T12:07:11.878 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.847+0000 7ff37dffb700 1 -- 172.21.15.102:0/4168630074 >> v1:172.21.15.102:6790/0 conn(0x7ff390098b70 legacy=0x7ff390143860 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-05-25T12:07:11.878 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.847+0000 7ff37dffb700 1 -- 172.21.15.102:0/4168630074 >> v1:172.21.15.102:6789/0 conn(0x7ff3700092f0 legacy=0x7ff390115890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:11.879 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.847+0000 7ff37dffb700 10 monclient: _finish_hunting 0 2021-05-25T12:07:11.879 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.847+0000 7ff37dffb700 1 monclient: found mon.b 2021-05-25T12:07:11.879 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.847+0000 7ff37dffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:07:11.879 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.847+0000 7ff37dffb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T12:07:11.880 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.847+0000 7ff37dffb700 1 -- 172.21.15.102:0/4168630074 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff390117440 con 0x7ff37000fe20 2021-05-25T12:07:11.880 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.847+0000 7ff37dffb700 10 monclient: _finish_auth 0 2021-05-25T12:07:11.880 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.847+0000 7ff37dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:07:11.880 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.847+0000 7ff396964700 5 monclient: authenticate success, global_id 224179 2021-05-25T12:07:11.880 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.847+0000 7ff396964700 10 monclient: _renew_subs 2021-05-25T12:07:11.881 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.847+0000 7ff396964700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T12:07:11.881 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.847+0000 7ff396964700 1 -- 172.21.15.102:0/4168630074 --> v1:172.21.15.181:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff390128f20 con 0x7ff37000fe20 2021-05-25T12:07:11.881 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.847+0000 7ff37dffb700 1 -- 172.21.15.102:0/4168630074 <== mon.1 v1:172.21.15.181:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff380001cc0 con 0x7ff37000fe20 2021-05-25T12:07:11.881 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.847+0000 7ff37dffb700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:07:11.881 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.847+0000 7ff37dffb700 1 -- 172.21.15.102:0/4168630074 <== mon.1 v1:172.21.15.181:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7ff3800046d0 con 0x7ff37000fe20 2021-05-25T12:07:11.882 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.847+0000 7ff37dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:11.882 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.847+0000 7ff37dffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T12:07:11.882 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.847+0000 7ff37dffb700 10 monclient: dump: 2021-05-25T12:07:11.882 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:11.883 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:11.883 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:11.883 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:11.883 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:11.884 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:11.884 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:11.884 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:11.884 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:11.884 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:11.885 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.847+0000 7ff37ffff700 4 set_mon_vals no callback set 2021-05-25T12:07:11.885 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.847+0000 7ff396964700 10 client.224179.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:07:11.885 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.847+0000 7ff396964700 10 monclient: _renew_subs 2021-05-25T12:07:11.885 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.847+0000 7ff396964700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T12:07:11.886 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.847+0000 7ff396964700 1 -- 172.21.15.102:0/4168630074 --> v1:172.21.15.181:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff390119b30 con 0x7ff37000fe20 2021-05-25T12:07:11.886 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.847+0000 7ff396964700 1 librados: init done 2021-05-25T12:07:11.886 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.848+0000 7ff37dffb700 1 -- 172.21.15.102:0/4168630074 <== mon.1 v1:172.21.15.181:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7ff380004930 con 0x7ff37000fe20 2021-05-25T12:07:11.886 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.848+0000 7ff37dffb700 1 -- 172.21.15.102:0/4168630074 <== mon.1 v1:172.21.15.181:6789/0 8 ==== osd_map(221..221 src has 1..221) v4 ==== 6648+0+0 (unknown 2376337231 0 0) 0x7ff38003d820 con 0x7ff37000fe20 2021-05-25T12:07:11.887 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.848+0000 7ff37dffb700 10 client.224179.objecter ms_dispatch 0x7ff390001550 osd_map(221..221 src has 1..221) v4 2021-05-25T12:07:11.887 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.848+0000 7ff37dffb700 3 client.224179.objecter handle_osd_map got epochs [221,221] > 0 2021-05-25T12:07:11.887 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.848+0000 7ff37dffb700 3 client.224179.objecter handle_osd_map decoding full epoch 221 2021-05-25T12:07:11.887 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.848+0000 7ff395962700 10 monclient: get_auth_request con 0x7ff3700342a0 auth_method 0 2021-05-25T12:07:11.887 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.848+0000 7ff37dffb700 20 client.224179.objecter dump_active .. 0 homeless 2021-05-25T12:07:11.888 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.848+0000 7ff396964700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:07:11.888 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.848+0000 7ff37dffb700 10 client.224179.objecter ms_handle_connect 0x7ff3700342a0 2021-05-25T12:07:11.888 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.848+0000 7ff396964700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:07:11.888 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.848+0000 7ff396964700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T12:07:11.889 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.848+0000 7ff396964700 1 -- 172.21.15.102:0/4168630074 --> v1:172.21.15.181:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff390063a10 con 0x7ff37000fe20 2021-05-25T12:07:11.889 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.853+0000 7ff37dffb700 1 -- 172.21.15.102:0/4168630074 <== mon.1 v1:172.21.15.181:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7ff380012930 con 0x7ff37000fe20 2021-05-25T12:07:11.889 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.853+0000 7ff37dffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:07:11.889 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:11.853+0000 7ff37dffb700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:07:12.073 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.071+0000 7ff396964700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2021-05-25T12:07:12.073 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.072+0000 7ff396964700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2021-05-25T12:07:12.074 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.072+0000 7ff396964700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T12:07:12.074 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.072+0000 7ff396964700 1 -- 172.21.15.102:0/4168630074 --> v1:172.21.15.181:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7ff390059730 con 0x7ff37000fe20 2021-05-25T12:07:12.075 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.072+0000 7ff37dffb700 1 -- 172.21.15.102:0/4168630074 <== mon.1 v1:172.21.15.181:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1336 (unknown 254746657 0 1105958856) 0x7ff380012750 con 0x7ff37000fe20 2021-05-25T12:07:12.075 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.072+0000 7ff37dffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2021-05-25T12:07:12.075 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.072+0000 7ff37dffb700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:07:12.076 INFO:teuthology.orchestra.run.smithi102.stdout:{"election_epoch":60,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138303579357183","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"20ee91b6-8935-4b98-b868-85c03789ccc4","modified":"2021-05-25T11:59:11.011522Z","created":"2021-05-25T11:59:11.011522Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6790","nonce":0}]},"addr":"172.21.15.102:6790/0","public_addr":"172.21.15.102:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2021-05-25T12:07:12.077 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.074+0000 7ff396964700 10 librados: watch_flush enter 2021-05-25T12:07:12.077 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.074+0000 7ff396964700 10 librados: watch_flush exit 2021-05-25T12:07:12.078 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.074+0000 7ff396964700 20 client.224179.objecter shutdown clearing up homeless session... 2021-05-25T12:07:12.078 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.074+0000 7ff396964700 10 client.224179.objecter successfully canceled tick 2021-05-25T12:07:12.078 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.074+0000 7ff396964700 1 -- 172.21.15.102:0/4168630074 >> v1:172.21.15.181:6812/29582 conn(0x7ff3700342a0 legacy=0x7ff370036760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:12.078 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.074+0000 7ff396964700 10 monclient: shutdown 2021-05-25T12:07:12.079 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.074+0000 7ff396964700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:07:12.079 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.074+0000 7ff396964700 1 -- 172.21.15.102:0/4168630074 >> v1:172.21.15.181:6789/0 conn(0x7ff37000fe20 legacy=0x7ff390115fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:12.079 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.074+0000 7ff396964700 1 -- 172.21.15.102:0/4168630074 shutdown_connections 2021-05-25T12:07:12.079 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.074+0000 7ff396964700 1 -- 172.21.15.102:0/4168630074 >> 172.21.15.102:0/4168630074 conn(0x7ff39009eb80 msgr2=0x7ff3900966e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:07:12.080 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.074+0000 7ff396964700 1 -- 172.21.15.102:0/4168630074 shutdown_connections 2021-05-25T12:07:12.080 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.074+0000 7ff396964700 1 -- 172.21.15.102:0/4168630074 wait complete. 2021-05-25T12:07:12.080 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.074+0000 7ff396964700 1 librados: shutdown 2021-05-25T12:07:12.087 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2021-05-25T12:07:12.087 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2021-05-25T12:07:12.199 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.198+0000 7f99201d3700 10 monclient: get_monmap_and_config 2021-05-25T12:07:12.199 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.198+0000 7f99201d3700 10 monclient: build_initial_monmap 2021-05-25T12:07:12.200 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.198+0000 7f99201d3700 10 monclient: monmap: 2021-05-25T12:07:12.200 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:07:12.200 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:12.200 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:07:12.198985+0000 2021-05-25T12:07:12.201 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:07:12.198985+0000 2021-05-25T12:07:12.201 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:07:12.201 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:12.201 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:07:12.201 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:07:12.202 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:07:12.202 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:12.215 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.198+0000 7f99201d3700 1 Processor -- start 2021-05-25T12:07:12.215 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.198+0000 7f99201d3700 1 -- start start 2021-05-25T12:07:12.216 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.198+0000 7f99201d3700 10 monclient: init 2021-05-25T12:07:12.216 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.199+0000 7f99201d3700 10 monclient: _reopen_session rank -1 2021-05-25T12:07:12.216 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.199+0000 7f99201d3700 10 monclient: _add_conns ranks=[1,0,2] 2021-05-25T12:07:12.216 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.199+0000 7f99201d3700 10 monclient(hunting): picked mon.noname-c con 0x7f9918090990 addr v1:172.21.15.181:6789/0 2021-05-25T12:07:12.216 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.199+0000 7f99201d3700 10 monclient(hunting): picked mon.noname-a con 0x7f9918090d70 addr v1:172.21.15.102:6789/0 2021-05-25T12:07:12.217 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.199+0000 7f99201d3700 10 monclient(hunting): picked mon.noname-b con 0x7f991812e330 addr v1:172.21.15.102:6790/0 2021-05-25T12:07:12.217 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.199+0000 7f99201d3700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9918096fa0 con 0x7f9918090d70 2021-05-25T12:07:12.217 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.199+0000 7f99201d3700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99180926a0 con 0x7f9918090990 2021-05-25T12:07:12.217 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.199+0000 7f99201d3700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9918092940 con 0x7f991812e330 2021-05-25T12:07:12.218 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.199+0000 7f99201d3700 10 monclient(hunting): _renew_subs 2021-05-25T12:07:12.218 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.199+0000 7f99201d3700 10 monclient(hunting): authenticate will time out at 1232.796322s 2021-05-25T12:07:12.218 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.199+0000 7f991e770700 1 -- >> v1:172.21.15.102:6790/0 conn(0x7f991812e330 legacy=0x7f9918130720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6790/0 2021-05-25T12:07:12.218 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.199+0000 7f991d76e700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f9918090d70 0x7f9918091200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33712/0 (socket says 172.21.15.102:33712) 2021-05-25T12:07:12.218 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.199+0000 7f991d76e700 1 -- 172.21.15.102:0/2803686852 learned_addr learned my addr 172.21.15.102:0/2803686852 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:07:12.219 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.199+0000 7f991cf6d700 1 -- 172.21.15.102:0/2803686852 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 724547392 0 0) 0x7f9918096fa0 con 0x7f9918090d70 2021-05-25T12:07:12.219 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.199+0000 7f991cf6d700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:12.219 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.199+0000 7f991cf6d700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:12.219 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.199+0000 7f991cf6d700 10 monclient(hunting): my global_id is 204318 2021-05-25T12:07:12.220 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.199+0000 7f991cf6d700 1 -- 172.21.15.102:0/2803686852 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9904003270 con 0x7f9918090d70 2021-05-25T12:07:12.220 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.199+0000 7f991cf6d700 1 -- 172.21.15.102:0/2803686852 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2106018880 0 0) 0x7f9904003270 con 0x7f9918090d70 2021-05-25T12:07:12.220 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.199+0000 7f991cf6d700 1 -- 172.21.15.102:0/2803686852 >> v1:172.21.15.102:6790/0 conn(0x7f991812e330 legacy=0x7f9918130720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-05-25T12:07:12.220 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.199+0000 7f991cf6d700 1 -- 172.21.15.102:0/2803686852 >> v1:172.21.15.181:6789/0 conn(0x7f9918090990 legacy=0x7f9918093240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:12.220 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.199+0000 7f991cf6d700 10 monclient: _finish_hunting 0 2021-05-25T12:07:12.221 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.199+0000 7f991cf6d700 1 monclient: found mon.noname-a 2021-05-25T12:07:12.221 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.199+0000 7f991cf6d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:07:12.221 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.199+0000 7f991cf6d700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T12:07:12.221 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.199+0000 7f991cf6d700 1 -- 172.21.15.102:0/2803686852 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9918092be0 con 0x7f9918090d70 2021-05-25T12:07:12.222 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.199+0000 7f991cf6d700 10 monclient: _finish_auth 0 2021-05-25T12:07:12.222 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.199+0000 7f991cf6d700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:07:12.222 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.199+0000 7f99201d3700 5 monclient: authenticate success, global_id 204318 2021-05-25T12:07:12.222 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.199+0000 7f991cf6d700 1 -- 172.21.15.102:0/2803686852 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f9900001610 con 0x7f9918090d70 2021-05-25T12:07:12.222 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.199+0000 7f99201d3700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:07:12.223 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.199+0000 7f991cf6d700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:12.223 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.199+0000 7f991cf6d700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:07:12.223 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.199+0000 7f991cf6d700 10 monclient: dump: 2021-05-25T12:07:12.223 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:12.224 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:12.224 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:12.224 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:12.224 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:12.224 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:12.225 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:12.225 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:12.225 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:12.225 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:12.225 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.199+0000 7f99201d3700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:07:12.226 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.200+0000 7f991cf6d700 1 -- 172.21.15.102:0/2803686852 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f99000018c0 con 0x7f9918090d70 2021-05-25T12:07:12.226 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.200+0000 7f991cf6d700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:07:12.226 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.200+0000 7f99201d3700 10 monclient: get_monmap_and_config success 2021-05-25T12:07:12.226 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.200+0000 7f991cf6d700 1 -- 172.21.15.102:0/2803686852 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f9900001c50 con 0x7f9918090d70 2021-05-25T12:07:12.227 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.200+0000 7f991cf6d700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:12.227 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.200+0000 7f99201d3700 4 set_mon_vals no callback set 2021-05-25T12:07:12.227 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.200+0000 7f991cf6d700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:07:12.227 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.200+0000 7f991cf6d700 10 monclient: dump: 2021-05-25T12:07:12.227 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:12.228 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:12.228 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:12.228 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:12.228 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:12.228 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:12.229 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:12.229 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:12.229 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:12.229 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:12.230 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.200+0000 7f99201d3700 10 monclient: shutdown 2021-05-25T12:07:12.230 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.200+0000 7f99201d3700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:07:12.230 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.200+0000 7f99201d3700 1 -- 172.21.15.102:0/2803686852 >> v1:172.21.15.102:6789/0 conn(0x7f9918090d70 legacy=0x7f9918091200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:12.230 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.200+0000 7f99201d3700 1 -- 172.21.15.102:0/2803686852 shutdown_connections 2021-05-25T12:07:12.230 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.200+0000 7f99201d3700 1 -- 172.21.15.102:0/2803686852 >> 172.21.15.102:0/2803686852 conn(0x7f991809b370 msgr2=0x7f9918097430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:07:12.231 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.200+0000 7f99201d3700 1 -- 172.21.15.102:0/2803686852 shutdown_connections 2021-05-25T12:07:12.231 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.200+0000 7f99201d3700 1 -- 172.21.15.102:0/2803686852 wait complete. 2021-05-25T12:07:12.231 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.201+0000 7f99201d3700 10 monclient: build_initial_monmap 2021-05-25T12:07:12.231 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.201+0000 7f99201d3700 10 monclient: monmap: 2021-05-25T12:07:12.232 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:07:12.232 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:07:12.232 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:07:12.232 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:07:12.232 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:07:12.233 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:07:12.233 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:07:12.233 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:07:12.233 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:07:12.233 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:12.234 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.201+0000 7f99201d3700 1 librados: starting msgr at 2021-05-25T12:07:12.234 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.201+0000 7f99201d3700 1 librados: starting objecter 2021-05-25T12:07:12.234 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.201+0000 7f99201d3700 1 Processor -- start 2021-05-25T12:07:12.234 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.201+0000 7f99201d3700 1 -- start start 2021-05-25T12:07:12.235 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.201+0000 7f99201d3700 1 librados: setting wanted keys 2021-05-25T12:07:12.235 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.201+0000 7f99201d3700 1 librados: calling monclient init 2021-05-25T12:07:12.235 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.201+0000 7f99201d3700 10 monclient: init 2021-05-25T12:07:12.235 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.201+0000 7f99201d3700 10 monclient: _reopen_session rank -1 2021-05-25T12:07:12.235 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.201+0000 7f99201d3700 10 monclient: _add_conns ranks=[0,2,1] 2021-05-25T12:07:12.236 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.201+0000 7f99201d3700 10 monclient(hunting): picked mon.noname-a con 0x7f9918090990 addr v1:172.21.15.102:6789/0 2021-05-25T12:07:12.236 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.201+0000 7f99201d3700 10 monclient(hunting): picked mon.noname-c con 0x7f9918090d70 addr v1:172.21.15.102:6790/0 2021-05-25T12:07:12.236 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.201+0000 7f99201d3700 10 monclient(hunting): picked mon.noname-b con 0x7f991812e330 addr v1:172.21.15.181:6789/0 2021-05-25T12:07:12.236 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.201+0000 7f99201d3700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99180658f0 con 0x7f9918090990 2021-05-25T12:07:12.237 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.201+0000 7f99201d3700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9918091fe0 con 0x7f991812e330 2021-05-25T12:07:12.237 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.201+0000 7f99201d3700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f991812a250 con 0x7f9918090d70 2021-05-25T12:07:12.237 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.201+0000 7f99201d3700 10 monclient(hunting): _renew_subs 2021-05-25T12:07:12.237 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.201+0000 7f99201d3700 10 monclient(hunting): authenticate will time out at 1232.798830s 2021-05-25T12:07:12.237 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.201+0000 7f991d76e700 1 -- >> v1:172.21.15.102:6790/0 conn(0x7f9918090d70 legacy=0x7f9918067fe0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.102:6790/0 2021-05-25T12:07:12.238 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.201+0000 7f991df6f700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f9918090990 0x7f9918136940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33716/0 (socket says 172.21.15.102:33716) 2021-05-25T12:07:12.238 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.201+0000 7f991df6f700 1 -- 172.21.15.102:0/3400293872 learned_addr learned my addr 172.21.15.102:0/3400293872 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:07:12.238 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.201+0000 7f990dffb700 10 client.?.objecter ms_handle_connect 0x7f9918090990 2021-05-25T12:07:12.238 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.201+0000 7f990dffb700 10 client.?.objecter resend_mon_ops 2021-05-25T12:07:12.238 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.202+0000 7f990dffb700 1 -- 172.21.15.102:0/3400293872 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4141414088 0 0) 0x7f99180658f0 con 0x7f9918090990 2021-05-25T12:07:12.239 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.202+0000 7f990dffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:12.239 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.202+0000 7f990dffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:12.239 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.202+0000 7f990dffb700 10 monclient(hunting): my global_id is 204321 2021-05-25T12:07:12.239 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.202+0000 7f990dffb700 1 -- 172.21.15.102:0/3400293872 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98f8004c10 con 0x7f9918090990 2021-05-25T12:07:12.240 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.202+0000 7f990dffb700 10 client.?.objecter ms_handle_connect 0x7f991812e330 2021-05-25T12:07:12.240 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.202+0000 7f990dffb700 10 client.?.objecter resend_mon_ops 2021-05-25T12:07:12.240 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.202+0000 7f990dffb700 1 -- 172.21.15.102:0/3400293872 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2254848812 0 0) 0x7f9918091fe0 con 0x7f991812e330 2021-05-25T12:07:12.240 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.202+0000 7f990dffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:12.240 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.202+0000 7f990dffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:12.241 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.202+0000 7f990dffb700 10 monclient(hunting): my global_id is 224185 2021-05-25T12:07:12.241 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.202+0000 7f990dffb700 1 -- 172.21.15.102:0/3400293872 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99180658f0 con 0x7f991812e330 2021-05-25T12:07:12.241 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.202+0000 7f990dffb700 1 -- 172.21.15.102:0/3400293872 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 785962383 0 0) 0x7f98f8004c10 con 0x7f9918090990 2021-05-25T12:07:12.241 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.202+0000 7f990dffb700 1 -- 172.21.15.102:0/3400293872 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9918091fe0 con 0x7f9918090990 2021-05-25T12:07:12.242 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.202+0000 7f990dffb700 1 -- 172.21.15.102:0/3400293872 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f9908001940 con 0x7f9918090990 2021-05-25T12:07:12.242 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.202+0000 7f990dffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:12.242 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.202+0000 7f990dffb700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:07:12.242 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.202+0000 7f990dffb700 10 monclient(hunting): dump: 2021-05-25T12:07:12.242 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:12.243 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:12.243 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:12.243 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:12.243 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:12.244 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:12.244 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:12.244 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:12.244 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:12.244 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:12.245 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.202+0000 7f990dffb700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:07:12.245 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.202+0000 7f990dffb700 1 -- 172.21.15.102:0/3400293872 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1596860136 0 0) 0x7f9918091fe0 con 0x7f9918090990 2021-05-25T12:07:12.245 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.202+0000 7f990dffb700 1 -- 172.21.15.102:0/3400293872 >> v1:172.21.15.102:6790/0 conn(0x7f9918090d70 legacy=0x7f9918067fe0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-05-25T12:07:12.245 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.202+0000 7f990dffb700 1 -- 172.21.15.102:0/3400293872 >> v1:172.21.15.181:6789/0 conn(0x7f991812e330 legacy=0x7f991813ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:12.246 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.202+0000 7f990dffb700 10 monclient: _finish_hunting 0 2021-05-25T12:07:12.246 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.202+0000 7f990dffb700 1 monclient: found mon.a 2021-05-25T12:07:12.246 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.202+0000 7f990dffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:07:12.246 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.202+0000 7f990dffb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:07:12.246 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.202+0000 7f990dffb700 1 -- 172.21.15.102:0/3400293872 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99181377f0 con 0x7f9918090990 2021-05-25T12:07:12.247 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.202+0000 7f990dffb700 10 monclient: _finish_auth 0 2021-05-25T12:07:12.247 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.202+0000 7f990dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:07:12.247 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.202+0000 7f99201d3700 5 monclient: authenticate success, global_id 204321 2021-05-25T12:07:12.247 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.202+0000 7f99201d3700 10 monclient: _renew_subs 2021-05-25T12:07:12.248 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.202+0000 7f99201d3700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:07:12.248 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.202+0000 7f99201d3700 1 -- 172.21.15.102:0/3400293872 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9918137050 con 0x7f9918090990 2021-05-25T12:07:12.248 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.202+0000 7f990dffb700 1 -- 172.21.15.102:0/3400293872 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9908001cd0 con 0x7f9918090990 2021-05-25T12:07:12.248 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.202+0000 7f990dffb700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:07:12.248 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.202+0000 7f990dffb700 1 -- 172.21.15.102:0/3400293872 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f9908002410 con 0x7f9918090990 2021-05-25T12:07:12.249 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.202+0000 7f990dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:12.249 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.202+0000 7f990dffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:07:12.249 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.202+0000 7f990dffb700 10 monclient: dump: 2021-05-25T12:07:12.249 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:12.250 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:12.250 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:12.250 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:12.250 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:12.250 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:12.251 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:12.251 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:12.251 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:12.251 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:12.252 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.202+0000 7f990ffff700 4 set_mon_vals no callback set 2021-05-25T12:07:12.252 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.202+0000 7f99201d3700 10 client.204321.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:07:12.252 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.202+0000 7f99201d3700 10 monclient: _renew_subs 2021-05-25T12:07:12.252 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.202+0000 7f99201d3700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:07:12.253 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.202+0000 7f99201d3700 1 -- 172.21.15.102:0/3400293872 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f99180686f0 con 0x7f9918090990 2021-05-25T12:07:12.253 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.202+0000 7f99201d3700 1 librados: init done 2021-05-25T12:07:12.253 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.203+0000 7f990dffb700 1 -- 172.21.15.102:0/3400293872 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f99080025f0 con 0x7f9918090990 2021-05-25T12:07:12.253 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.203+0000 7f990dffb700 1 -- 172.21.15.102:0/3400293872 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(222..222 src has 1..222) v4 ==== 6358+0+0 (unknown 207553742 0 0) 0x7f990803d920 con 0x7f9918090990 2021-05-25T12:07:12.253 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.203+0000 7f990dffb700 10 client.204321.objecter ms_dispatch 0x7f9918001550 osd_map(222..222 src has 1..222) v4 2021-05-25T12:07:12.254 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.203+0000 7f99201d3700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:07:12.254 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.203+0000 7f990dffb700 3 client.204321.objecter handle_osd_map got epochs [222,222] > 0 2021-05-25T12:07:12.254 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.203+0000 7f990dffb700 3 client.204321.objecter handle_osd_map decoding full epoch 222 2021-05-25T12:07:12.254 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.203+0000 7f99201d3700 1 -- 172.21.15.102:0/3400293872 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f9918059730 con 0x7f9918127570 2021-05-25T12:07:12.255 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.203+0000 7f990dffb700 20 client.204321.objecter dump_active .. 0 homeless 2021-05-25T12:07:12.255 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.203+0000 7f991d76e700 10 monclient: get_auth_request con 0x7f98f80331a0 auth_method 0 2021-05-25T12:07:12.255 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.203+0000 7f990dffb700 10 client.204321.objecter ms_handle_connect 0x7f9918127570 2021-05-25T12:07:12.255 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.203+0000 7f990dffb700 10 client.204321.objecter resend_mon_ops 2021-05-25T12:07:12.256 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.203+0000 7f990dffb700 1 -- 172.21.15.102:0/3400293872 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 655534263 0 0) 0x7f9918059730 con 0x7f9918127570 2021-05-25T12:07:12.256 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.203+0000 7f990dffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:12.256 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.203+0000 7f990dffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:12.256 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.203+0000 7f990dffb700 10 monclient(hunting): my global_id is 204324 2021-05-25T12:07:12.256 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.203+0000 7f990dffb700 1 -- 172.21.15.102:0/3400293872 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9918091fe0 con 0x7f9918127570 2021-05-25T12:07:12.257 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.203+0000 7f990dffb700 10 client.204321.objecter ms_handle_connect 0x7f98f80331a0 2021-05-25T12:07:12.257 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.203+0000 7f990dffb700 1 -- 172.21.15.102:0/3400293872 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3499871214 0 0) 0x7f9918091fe0 con 0x7f9918127570 2021-05-25T12:07:12.257 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.203+0000 7f990dffb700 1 -- 172.21.15.102:0/3400293872 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9918059730 con 0x7f9918127570 2021-05-25T12:07:12.257 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.203+0000 7f990dffb700 1 -- 172.21.15.102:0/3400293872 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f99140040a0 con 0x7f9918127570 2021-05-25T12:07:12.258 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.203+0000 7f990dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:12.258 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.203+0000 7f990dffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:07:12.258 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.203+0000 7f990dffb700 10 monclient: dump: 2021-05-25T12:07:12.258 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:12.258 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:12.259 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:12.259 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:12.259 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:12.259 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:12.260 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:12.260 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:12.260 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:12.260 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:12.260 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.203+0000 7f990dffb700 1 -- 172.21.15.102:0/3400293872 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 399299279 0 0) 0x7f9918059730 con 0x7f9918127570 2021-05-25T12:07:12.261 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.203+0000 7f990dffb700 1 -- 172.21.15.102:0/3400293872 --> v1:172.21.15.102:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9918096500 con 0x7f9918127570 2021-05-25T12:07:12.261 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.204+0000 7f990dffb700 1 -- 172.21.15.102:0/3400293872 <== mon.0 v1:172.21.15.102:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7f9914005580 con 0x7f9918127570 2021-05-25T12:07:12.261 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.204+0000 7f990dffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:07:12.261 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.204+0000 7f990dffb700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:07:12.262 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.204+0000 7f990dffb700 1 -- 172.21.15.102:0/3400293872 >> v1:172.21.15.102:6789/0 conn(0x7f9918127570 legacy=0x7f9918127950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:12.262 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.204+0000 7f990dffb700 1 -- 172.21.15.102:0/3400293872 >> v1:172.21.15.102:6789/0 conn(0x7f9918127570 legacy=0x7f9918127950 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:07:12.262 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.210+0000 7f99201d3700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:07:12.262 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.210+0000 7f99201d3700 1 -- 172.21.15.102:0/3400293872 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f9918065b80 con 0x7f9918027ce0 2021-05-25T12:07:12.262 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.211+0000 7f990dffb700 10 client.204321.objecter ms_handle_connect 0x7f9918027ce0 2021-05-25T12:07:12.263 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.211+0000 7f990dffb700 10 client.204321.objecter resend_mon_ops 2021-05-25T12:07:12.263 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.211+0000 7f990dffb700 1 -- 172.21.15.102:0/3400293872 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1630719900 0 0) 0x7f9918065b80 con 0x7f9918027ce0 2021-05-25T12:07:12.263 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.211+0000 7f990dffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:12.263 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.211+0000 7f990dffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:12.263 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.211+0000 7f990dffb700 10 monclient(hunting): my global_id is 204327 2021-05-25T12:07:12.264 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.211+0000 7f990dffb700 1 -- 172.21.15.102:0/3400293872 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9918059730 con 0x7f9918027ce0 2021-05-25T12:07:12.264 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.211+0000 7f990dffb700 1 -- 172.21.15.102:0/3400293872 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2855332692 0 0) 0x7f9918059730 con 0x7f9918027ce0 2021-05-25T12:07:12.264 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.211+0000 7f990dffb700 1 -- 172.21.15.102:0/3400293872 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9918065b80 con 0x7f9918027ce0 2021-05-25T12:07:12.264 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.211+0000 7f990dffb700 1 -- 172.21.15.102:0/3400293872 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f9914004750 con 0x7f9918027ce0 2021-05-25T12:07:12.265 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.211+0000 7f990dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:12.265 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.211+0000 7f990dffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:07:12.265 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.211+0000 7f990dffb700 10 monclient: dump: 2021-05-25T12:07:12.265 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:12.265 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:12.266 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:12.266 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:12.266 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:12.266 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:12.266 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:12.267 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:12.267 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:12.267 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:12.267 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.211+0000 7f990dffb700 1 -- 172.21.15.102:0/3400293872 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 113587133 0 0) 0x7f9918065b80 con 0x7f9918027ce0 2021-05-25T12:07:12.267 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.211+0000 7f990dffb700 1 -- 172.21.15.102:0/3400293872 --> v1:172.21.15.102:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f991805d870 con 0x7f9918027ce0 2021-05-25T12:07:12.268 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.211+0000 7f990dffb700 1 -- 172.21.15.102:0/3400293872 <== mon.0 v1:172.21.15.102:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3929 (unknown 0 0 3513654527) 0x7f9914005830 con 0x7f9918027ce0 2021-05-25T12:07:12.268 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.211+0000 7f990dffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:07:12.268 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.211+0000 7f990dffb700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:07:12.268 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.211+0000 7f990dffb700 1 -- 172.21.15.102:0/3400293872 >> v1:172.21.15.102:6789/0 conn(0x7f9918027ce0 legacy=0x7f99181443f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:12.269 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.211+0000 7f990dffb700 1 -- 172.21.15.102:0/3400293872 >> v1:172.21.15.102:6789/0 conn(0x7f9918027ce0 legacy=0x7f99181443f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:07:12.269 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:07:12.269 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:07:12.270 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:07:12.270 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "peon", 2021-05-25T12:07:12.270 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 60, 2021-05-25T12:07:12.270 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:07:12.270 INFO:teuthology.orchestra.run.smithi102.stdout: 0, 2021-05-25T12:07:12.271 INFO:teuthology.orchestra.run.smithi102.stdout: 1 2021-05-25T12:07:12.271 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:07:12.271 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 0, 2021-05-25T12:07:12.271 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:07:12.272 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:07:12.272 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:07:12.272 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:07:12.272 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:07:12.272 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:07:12.273 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:07:12.273 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:07:12.273 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:07:12.273 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:07:12.273 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:07:12.274 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:07:12.274 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:07:12.274 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:07:12.274 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:07:12.274 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:07:12.275 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:07:12.275 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:07:12.275 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:07:12.275 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:07:12.276 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:07:12.276 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:07:12.276 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:07:12.276 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:07:12.276 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:07:12.277 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:12.277 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:07:12.277 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:07:12.277 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:07:12.277 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:07:12.278 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:07:12.278 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:07:12.278 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:07:12.278 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:07:12.279 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:07:12.279 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:07:12.279 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:07:12.279 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:07:12.279 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:07:12.280 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:07:12.280 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:07:12.280 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:07:12.280 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:07:12.280 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:07:12.281 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:07:12.281 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:07:12.281 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:07:12.281 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:07:12.281 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:07:12.282 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:07:12.282 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:07:12.282 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:07:12.282 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:12.283 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:07:12.283 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:12.283 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:07:12.283 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:07:12.283 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:07:12.284 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:07:12.284 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:12.284 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:07:12.284 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:07:12.285 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:07:12.285 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:07:12.285 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:07:12.285 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:12.285 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:07:12.286 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:07:12.286 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:07:12.286 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:07:12.286 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:07:12.286 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:12.287 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:12.287 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:07:12.287 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:07:12.287 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:07:12.287 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:07:12.288 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:12.288 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:07:12.288 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:07:12.288 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:07:12.288 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:07:12.289 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:07:12.289 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:12.289 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:07:12.289 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:07:12.290 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:07:12.290 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:07:12.290 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:07:12.290 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:12.290 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:12.291 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:07:12.291 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:07:12.291 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:07:12.291 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:07:12.291 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:12.292 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:07:12.292 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:07:12.292 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:07:12.292 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:07:12.292 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:07:12.293 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:12.293 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:07:12.293 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:07:12.293 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:07:12.293 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:07:12.294 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:07:12.294 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:07:12.294 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:07:12.294 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:12.294 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:07:12.295 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:07:12.295 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:12.295 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:07:12.295 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:07:12.295 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:07:12.296 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:07:12.296 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:07:12.296 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": [ 2021-05-25T12:07:12.296 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:12.297 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:07:12.297 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:07:12.297 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 6 2021-05-25T12:07:12.297 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:07:12.297 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:07:12.298 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:07:12.298 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:12.298 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:07:12.298 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:07:12.299 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 4 2021-05-25T12:07:12.299 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:07:12.299 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:07:12.299 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:12.300 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:07:12.300 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:07:12.302 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.211+0000 7f99201d3700 10 librados: watch_flush enter 2021-05-25T12:07:12.302 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.211+0000 7f99201d3700 10 librados: watch_flush exit 2021-05-25T12:07:12.302 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.211+0000 7f99201d3700 20 client.204321.objecter shutdown clearing up homeless session... 2021-05-25T12:07:12.303 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.211+0000 7f99201d3700 10 client.204321.objecter successfully canceled tick 2021-05-25T12:07:12.303 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.212+0000 7f99201d3700 1 -- 172.21.15.102:0/3400293872 >> v1:172.21.15.181:6812/29582 conn(0x7f98f80331a0 legacy=0x7f98f8035660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:12.303 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.212+0000 7f99201d3700 10 monclient: shutdown 2021-05-25T12:07:12.303 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.212+0000 7f99201d3700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:07:12.304 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.212+0000 7f99201d3700 1 -- 172.21.15.102:0/3400293872 >> v1:172.21.15.102:6789/0 conn(0x7f9918090990 legacy=0x7f9918136940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:12.304 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.212+0000 7f991e770700 1 -- 172.21.15.102:0/3400293872 reap_dead start 2021-05-25T12:07:12.304 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.212+0000 7f99201d3700 1 -- 172.21.15.102:0/3400293872 shutdown_connections 2021-05-25T12:07:12.304 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.212+0000 7f99201d3700 1 -- 172.21.15.102:0/3400293872 >> 172.21.15.102:0/3400293872 conn(0x7f991809b370 msgr2=0x7f9918093a60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:07:12.304 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.212+0000 7f99201d3700 1 -- 172.21.15.102:0/3400293872 shutdown_connections 2021-05-25T12:07:12.305 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.212+0000 7f99201d3700 1 -- 172.21.15.102:0/3400293872 wait complete. 2021-05-25T12:07:12.305 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.212+0000 7f99201d3700 1 librados: shutdown 2021-05-25T12:07:12.305 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2021-05-25T12:07:12.433 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.413+0000 7f7b7cdf1700 10 monclient: get_monmap_and_config 2021-05-25T12:07:12.434 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.413+0000 7f7b7cdf1700 10 monclient: build_initial_monmap 2021-05-25T12:07:12.434 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.413+0000 7f7b7cdf1700 10 monclient: monmap: 2021-05-25T12:07:12.434 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:07:12.434 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:12.435 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:07:12.415484+0000 2021-05-25T12:07:12.435 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:07:12.415484+0000 2021-05-25T12:07:12.435 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:07:12.435 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:12.436 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:07:12.436 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:07:12.436 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:07:12.436 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:12.436 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.414+0000 7f7b7cdf1700 1 Processor -- start 2021-05-25T12:07:12.437 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.414+0000 7f7b7cdf1700 1 -- start start 2021-05-25T12:07:12.437 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.414+0000 7f7b7cdf1700 10 monclient: init 2021-05-25T12:07:12.437 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.414+0000 7f7b7cdf1700 10 monclient: _reopen_session rank -1 2021-05-25T12:07:12.437 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.414+0000 7f7b7cdf1700 10 monclient: _add_conns ranks=[1,0,2] 2021-05-25T12:07:12.438 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.414+0000 7f7b7cdf1700 10 monclient(hunting): picked mon.noname-c con 0x7f7b780699f0 addr v1:172.21.15.181:6789/0 2021-05-25T12:07:12.438 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.414+0000 7f7b7cdf1700 10 monclient(hunting): picked mon.noname-a con 0x7f7b78096790 addr v1:172.21.15.102:6789/0 2021-05-25T12:07:12.438 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.414+0000 7f7b7cdf1700 10 monclient(hunting): picked mon.noname-b con 0x7f7b78130520 addr v1:172.21.15.102:6790/0 2021-05-25T12:07:12.438 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.414+0000 7f7b7cdf1700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b78094690 con 0x7f7b78096790 2021-05-25T12:07:12.438 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.414+0000 7f7b7cdf1700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b78096c40 con 0x7f7b780699f0 2021-05-25T12:07:12.439 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.414+0000 7f7b7cdf1700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b78097180 con 0x7f7b78130520 2021-05-25T12:07:12.439 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.414+0000 7f7b7cdf1700 10 monclient(hunting): _renew_subs 2021-05-25T12:07:12.439 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.414+0000 7f7b7cdf1700 10 monclient(hunting): authenticate will time out at 1233.012864s 2021-05-25T12:07:12.439 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.414+0000 7f7b76d9d700 1 -- >> v1:172.21.15.102:6790/0 conn(0x7f7b78130520 legacy=0x7f7b78132910 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6790/0 2021-05-25T12:07:12.440 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.414+0000 7f7b75d9b700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f7b78096790 0x7f7b78098290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33728/0 (socket says 172.21.15.102:33728) 2021-05-25T12:07:12.440 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.414+0000 7f7b75d9b700 1 -- 172.21.15.102:0/4125600473 learned_addr learned my addr 172.21.15.102:0/4125600473 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:07:12.440 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.415+0000 7f7b7559a700 1 -- 172.21.15.102:0/4125600473 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1879509248 0 0) 0x7f7b78094690 con 0x7f7b78096790 2021-05-25T12:07:12.440 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.415+0000 7f7b7559a700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:12.440 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.415+0000 7f7b7559a700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:12.441 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.415+0000 7f7b7559a700 10 monclient(hunting): my global_id is 204330 2021-05-25T12:07:12.441 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.415+0000 7f7b7559a700 1 -- 172.21.15.102:0/4125600473 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b5c003270 con 0x7f7b78096790 2021-05-25T12:07:12.441 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.415+0000 7f7b7559a700 1 -- 172.21.15.102:0/4125600473 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2790650694 0 0) 0x7f7b78096c40 con 0x7f7b780699f0 2021-05-25T12:07:12.441 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.415+0000 7f7b7559a700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:12.442 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.415+0000 7f7b7559a700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:12.442 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.415+0000 7f7b7559a700 10 monclient(hunting): my global_id is 224188 2021-05-25T12:07:12.442 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.415+0000 7f7b7559a700 1 -- 172.21.15.102:0/4125600473 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b78094690 con 0x7f7b780699f0 2021-05-25T12:07:12.442 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.415+0000 7f7b7559a700 1 -- 172.21.15.102:0/4125600473 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4226370380 0 0) 0x7f7b5c003270 con 0x7f7b78096790 2021-05-25T12:07:12.442 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.415+0000 7f7b7559a700 1 -- 172.21.15.102:0/4125600473 >> v1:172.21.15.102:6790/0 conn(0x7f7b78130520 legacy=0x7f7b78132910 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-05-25T12:07:12.443 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.415+0000 7f7b7559a700 1 -- 172.21.15.102:0/4125600473 >> v1:172.21.15.181:6789/0 conn(0x7f7b780699f0 legacy=0x7f7b78097b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:12.443 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.415+0000 7f7b7559a700 10 monclient: _finish_hunting 0 2021-05-25T12:07:12.443 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.415+0000 7f7b7559a700 1 monclient: found mon.noname-a 2021-05-25T12:07:12.443 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.415+0000 7f7b7559a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:07:12.444 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.415+0000 7f7b7559a700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T12:07:12.444 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.415+0000 7f7b7559a700 1 -- 172.21.15.102:0/4125600473 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b780989a0 con 0x7f7b78096790 2021-05-25T12:07:12.444 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.415+0000 7f7b7559a700 10 monclient: _finish_auth 0 2021-05-25T12:07:12.444 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.415+0000 7f7b7559a700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:07:12.444 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.415+0000 7f7b7cdf1700 5 monclient: authenticate success, global_id 204330 2021-05-25T12:07:12.445 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.415+0000 7f7b7559a700 1 -- 172.21.15.102:0/4125600473 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f7b60001610 con 0x7f7b78096790 2021-05-25T12:07:12.445 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.415+0000 7f7b7cdf1700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:07:12.445 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.415+0000 7f7b7559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:12.445 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.415+0000 7f7b7559a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:07:12.446 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.415+0000 7f7b7559a700 10 monclient: dump: 2021-05-25T12:07:12.446 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:12.446 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:12.446 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:12.447 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:12.447 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:12.447 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:12.447 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:12.447 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:12.448 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:12.448 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:12.448 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.415+0000 7f7b7cdf1700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:07:12.448 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.415+0000 7f7b7559a700 1 -- 172.21.15.102:0/4125600473 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7b600018f0 con 0x7f7b78096790 2021-05-25T12:07:12.448 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.415+0000 7f7b7559a700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:07:12.449 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.415+0000 7f7b7559a700 1 -- 172.21.15.102:0/4125600473 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f7b60001c80 con 0x7f7b78096790 2021-05-25T12:07:12.449 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.415+0000 7f7b7cdf1700 10 monclient: get_monmap_and_config success 2021-05-25T12:07:12.449 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.415+0000 7f7b7cdf1700 4 set_mon_vals no callback set 2021-05-25T12:07:12.449 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.415+0000 7f7b7559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:12.450 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.415+0000 7f7b7559a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:07:12.450 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.415+0000 7f7b7559a700 10 monclient: dump: 2021-05-25T12:07:12.450 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:12.450 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:12.451 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:12.451 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:12.451 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:12.451 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:12.451 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:12.452 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:12.452 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:12.452 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:12.452 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.415+0000 7f7b7cdf1700 10 monclient: shutdown 2021-05-25T12:07:12.452 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.415+0000 7f7b7cdf1700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:07:12.453 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.415+0000 7f7b7cdf1700 1 -- 172.21.15.102:0/4125600473 >> v1:172.21.15.102:6789/0 conn(0x7f7b78096790 legacy=0x7f7b78098290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:12.453 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.416+0000 7f7b7cdf1700 1 -- 172.21.15.102:0/4125600473 shutdown_connections 2021-05-25T12:07:12.453 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.416+0000 7f7b7cdf1700 1 -- 172.21.15.102:0/4125600473 >> 172.21.15.102:0/4125600473 conn(0x7f7b7809cd10 msgr2=0x7f7b7809d120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:07:12.453 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.416+0000 7f7b7cdf1700 1 -- 172.21.15.102:0/4125600473 shutdown_connections 2021-05-25T12:07:12.454 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.416+0000 7f7b7cdf1700 1 -- 172.21.15.102:0/4125600473 wait complete. 2021-05-25T12:07:12.454 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.416+0000 7f7b7cdf1700 10 monclient: build_initial_monmap 2021-05-25T12:07:12.454 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.416+0000 7f7b7cdf1700 10 monclient: monmap: 2021-05-25T12:07:12.454 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:07:12.455 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:07:12.455 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:07:12.455 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:07:12.455 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:07:12.455 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:07:12.456 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:07:12.456 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:07:12.456 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:07:12.456 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:12.457 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.416+0000 7f7b7cdf1700 1 librados: starting msgr at 2021-05-25T12:07:12.457 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.416+0000 7f7b7cdf1700 1 librados: starting objecter 2021-05-25T12:07:12.457 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.416+0000 7f7b7cdf1700 1 Processor -- start 2021-05-25T12:07:12.457 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.416+0000 7f7b7cdf1700 1 -- start start 2021-05-25T12:07:12.457 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.416+0000 7f7b7cdf1700 1 librados: setting wanted keys 2021-05-25T12:07:12.458 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.416+0000 7f7b7cdf1700 1 librados: calling monclient init 2021-05-25T12:07:12.458 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.416+0000 7f7b7cdf1700 10 monclient: init 2021-05-25T12:07:12.458 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.417+0000 7f7b7cdf1700 10 monclient: _reopen_session rank -1 2021-05-25T12:07:12.458 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.417+0000 7f7b7cdf1700 10 monclient: _add_conns ranks=[2,0,1] 2021-05-25T12:07:12.459 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.417+0000 7f7b7cdf1700 10 monclient(hunting): picked mon.noname-c con 0x7f7b780699f0 addr v1:172.21.15.102:6790/0 2021-05-25T12:07:12.459 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.417+0000 7f7b7cdf1700 10 monclient(hunting): picked mon.noname-a con 0x7f7b78096790 addr v1:172.21.15.102:6789/0 2021-05-25T12:07:12.459 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.417+0000 7f7b7cdf1700 10 monclient(hunting): picked mon.noname-b con 0x7f7b78130520 addr v1:172.21.15.181:6789/0 2021-05-25T12:07:12.459 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.417+0000 7f7b7cdf1700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b78094cc0 con 0x7f7b78096790 2021-05-25T12:07:12.459 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.417+0000 7f7b7cdf1700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b7812c5a0 con 0x7f7b78130520 2021-05-25T12:07:12.460 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.417+0000 7f7b7cdf1700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b78118260 con 0x7f7b780699f0 2021-05-25T12:07:12.460 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.417+0000 7f7b7cdf1700 10 monclient(hunting): _renew_subs 2021-05-25T12:07:12.460 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.417+0000 7f7b7cdf1700 10 monclient(hunting): authenticate will time out at 1233.015405s 2021-05-25T12:07:12.460 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.417+0000 7f7b7659c700 1 -- >> v1:172.21.15.102:6790/0 conn(0x7f7b780699f0 legacy=0x7f7b78062c30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.102:6790/0 2021-05-25T12:07:12.461 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.417+0000 7f7b75d9b700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f7b78096790 0x7f7b7812b830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33736/0 (socket says 172.21.15.102:33736) 2021-05-25T12:07:12.461 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.417+0000 7f7b75d9b700 1 -- 172.21.15.102:0/3493365763 learned_addr learned my addr 172.21.15.102:0/3493365763 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:07:12.461 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.417+0000 7f7b667fc700 10 client.?.objecter ms_handle_connect 0x7f7b78096790 2021-05-25T12:07:12.461 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.417+0000 7f7b667fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:07:12.461 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.417+0000 7f7b667fc700 1 -- 172.21.15.102:0/3493365763 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3727659985 0 0) 0x7f7b78094cc0 con 0x7f7b78096790 2021-05-25T12:07:12.462 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.417+0000 7f7b667fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:12.462 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.417+0000 7f7b667fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:12.462 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.417+0000 7f7b667fc700 10 monclient(hunting): my global_id is 204333 2021-05-25T12:07:12.462 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.417+0000 7f7b667fc700 1 -- 172.21.15.102:0/3493365763 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b54004c10 con 0x7f7b78096790 2021-05-25T12:07:12.463 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.417+0000 7f7b667fc700 10 client.?.objecter ms_handle_connect 0x7f7b78130520 2021-05-25T12:07:12.463 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.417+0000 7f7b667fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:07:12.463 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.417+0000 7f7b667fc700 1 -- 172.21.15.102:0/3493365763 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 472043588 0 0) 0x7f7b54004c10 con 0x7f7b78096790 2021-05-25T12:07:12.463 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.417+0000 7f7b667fc700 1 -- 172.21.15.102:0/3493365763 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7b78094cc0 con 0x7f7b78096790 2021-05-25T12:07:12.463 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.417+0000 7f7b667fc700 1 -- 172.21.15.102:0/3493365763 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f7b600015d0 con 0x7f7b78096790 2021-05-25T12:07:12.464 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.417+0000 7f7b667fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:12.464 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.417+0000 7f7b667fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:07:12.464 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.417+0000 7f7b667fc700 10 monclient(hunting): dump: 2021-05-25T12:07:12.464 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:12.465 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:12.465 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:12.465 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:12.465 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:12.465 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:12.466 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:12.466 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:12.466 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:12.466 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:12.467 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.417+0000 7f7b667fc700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:07:12.467 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.417+0000 7f7b667fc700 1 -- 172.21.15.102:0/3493365763 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3452030678 0 0) 0x7f7b7812c5a0 con 0x7f7b78130520 2021-05-25T12:07:12.467 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.417+0000 7f7b667fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:12.467 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.417+0000 7f7b667fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:12.467 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.417+0000 7f7b667fc700 10 monclient(hunting): my global_id is 224191 2021-05-25T12:07:12.468 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.417+0000 7f7b667fc700 1 -- 172.21.15.102:0/3493365763 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b54004c10 con 0x7f7b78130520 2021-05-25T12:07:12.468 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.418+0000 7f7b667fc700 1 -- 172.21.15.102:0/3493365763 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2288259163 0 0) 0x7f7b78094cc0 con 0x7f7b78096790 2021-05-25T12:07:12.468 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.418+0000 7f7b667fc700 1 -- 172.21.15.102:0/3493365763 >> v1:172.21.15.102:6790/0 conn(0x7f7b780699f0 legacy=0x7f7b78062c30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-05-25T12:07:12.468 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.418+0000 7f7b667fc700 1 -- 172.21.15.102:0/3493365763 >> v1:172.21.15.181:6789/0 conn(0x7f7b78130520 legacy=0x7f7b78143040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:12.469 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.418+0000 7f7b667fc700 10 monclient: _finish_hunting 0 2021-05-25T12:07:12.469 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.418+0000 7f7b667fc700 1 monclient: found mon.a 2021-05-25T12:07:12.469 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.418+0000 7f7b667fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:07:12.469 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.418+0000 7f7b667fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:07:12.469 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.418+0000 7f7b667fc700 1 -- 172.21.15.102:0/3493365763 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b7812bf40 con 0x7f7b78096790 2021-05-25T12:07:12.470 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.418+0000 7f7b667fc700 10 monclient: _finish_auth 0 2021-05-25T12:07:12.470 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.418+0000 7f7b667fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:07:12.470 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.418+0000 7f7b7cdf1700 5 monclient: authenticate success, global_id 204333 2021-05-25T12:07:12.470 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.418+0000 7f7b7cdf1700 10 monclient: _renew_subs 2021-05-25T12:07:12.471 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.418+0000 7f7b7cdf1700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:07:12.471 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.418+0000 7f7b7cdf1700 1 -- 172.21.15.102:0/3493365763 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b7812f730 con 0x7f7b78096790 2021-05-25T12:07:12.471 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.418+0000 7f7b7cdf1700 10 client.204333.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:07:12.471 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.418+0000 7f7b7cdf1700 10 monclient: _renew_subs 2021-05-25T12:07:12.471 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.418+0000 7f7b7cdf1700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:07:12.472 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.418+0000 7f7b7cdf1700 1 -- 172.21.15.102:0/3493365763 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b78063b30 con 0x7f7b78096790 2021-05-25T12:07:12.472 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.418+0000 7f7b7cdf1700 1 librados: init done 2021-05-25T12:07:12.472 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.418+0000 7f7b667fc700 1 -- 172.21.15.102:0/3493365763 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7b60004ab0 con 0x7f7b78096790 2021-05-25T12:07:12.472 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.418+0000 7f7b667fc700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:07:12.473 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.418+0000 7f7b667fc700 1 -- 172.21.15.102:0/3493365763 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f7b60004e40 con 0x7f7b78096790 2021-05-25T12:07:12.473 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.418+0000 7f7b667fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:12.473 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.418+0000 7f7b667fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:07:12.473 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.418+0000 7f7b74d99700 4 set_mon_vals no callback set 2021-05-25T12:07:12.473 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.418+0000 7f7b667fc700 10 monclient: dump: 2021-05-25T12:07:12.474 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:12.474 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:12.474 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:12.474 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:12.475 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:12.475 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:12.475 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:12.475 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:12.475 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:12.476 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:12.476 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.418+0000 7f7b7cdf1700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:07:12.476 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.418+0000 7f7b7cdf1700 1 -- 172.21.15.102:0/3493365763 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f7b580065a0 con 0x7f7b58002840 2021-05-25T12:07:12.476 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.418+0000 7f7b667fc700 1 -- 172.21.15.102:0/3493365763 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f7b60004fa0 con 0x7f7b78096790 2021-05-25T12:07:12.477 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.418+0000 7f7b667fc700 1 -- 172.21.15.102:0/3493365763 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(222..222 src has 1..222) v4 ==== 6358+0+0 (unknown 207553742 0 0) 0x7f7b6003dc70 con 0x7f7b78096790 2021-05-25T12:07:12.477 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.418+0000 7f7b667fc700 10 client.204333.objecter ms_dispatch 0x7f7b78001550 osd_map(222..222 src has 1..222) v4 2021-05-25T12:07:12.477 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.418+0000 7f7b667fc700 3 client.204333.objecter handle_osd_map got epochs [222,222] > 0 2021-05-25T12:07:12.477 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.418+0000 7f7b667fc700 3 client.204333.objecter handle_osd_map decoding full epoch 222 2021-05-25T12:07:12.477 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.419+0000 7f7b667fc700 20 client.204333.objecter dump_active .. 0 homeless 2021-05-25T12:07:12.478 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.419+0000 7f7b667fc700 10 client.204333.objecter ms_handle_connect 0x7f7b58002840 2021-05-25T12:07:12.478 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.419+0000 7f7b667fc700 10 client.204333.objecter resend_mon_ops 2021-05-25T12:07:12.478 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.419+0000 7f7b76d9d700 10 monclient: get_auth_request con 0x7f7b54033150 auth_method 0 2021-05-25T12:07:12.478 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.419+0000 7f7b667fc700 1 -- 172.21.15.102:0/3493365763 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1275187134 0 0) 0x7f7b580065a0 con 0x7f7b58002840 2021-05-25T12:07:12.479 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.419+0000 7f7b667fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:12.479 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.419+0000 7f7b667fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:12.479 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.419+0000 7f7b667fc700 10 monclient(hunting): my global_id is 224194 2021-05-25T12:07:12.479 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.419+0000 7f7b667fc700 1 -- 172.21.15.102:0/3493365763 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b78094cc0 con 0x7f7b58002840 2021-05-25T12:07:12.480 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.419+0000 7f7b667fc700 10 client.204333.objecter ms_handle_connect 0x7f7b54033150 2021-05-25T12:07:12.480 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.419+0000 7f7b667fc700 1 -- 172.21.15.102:0/3493365763 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1608119251 0 0) 0x7f7b78094cc0 con 0x7f7b58002840 2021-05-25T12:07:12.480 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.419+0000 7f7b667fc700 1 -- 172.21.15.102:0/3493365763 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7b580065a0 con 0x7f7b58002840 2021-05-25T12:07:12.480 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.419+0000 7f7b667fc700 1 -- 172.21.15.102:0/3493365763 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f7b6c002c20 con 0x7f7b58002840 2021-05-25T12:07:12.480 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.419+0000 7f7b667fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:12.481 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.419+0000 7f7b667fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T12:07:12.481 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.419+0000 7f7b667fc700 10 monclient: dump: 2021-05-25T12:07:12.481 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:12.481 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:12.481 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:12.482 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:12.482 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:12.482 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:12.482 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:12.483 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:12.483 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:12.483 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:12.483 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.419+0000 7f7b667fc700 1 -- 172.21.15.102:0/3493365763 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2499750620 0 0) 0x7f7b580065a0 con 0x7f7b58002840 2021-05-25T12:07:12.483 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.419+0000 7f7b667fc700 1 -- 172.21.15.102:0/3493365763 --> v1:172.21.15.181:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7b58009e40 con 0x7f7b58002840 2021-05-25T12:07:12.484 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.419+0000 7f7b667fc700 1 -- 172.21.15.102:0/3493365763 <== mon.1 v1:172.21.15.181:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7f7b6c006110 con 0x7f7b58002840 2021-05-25T12:07:12.484 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.419+0000 7f7b667fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:07:12.484 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.419+0000 7f7b667fc700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:07:12.484 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.419+0000 7f7b667fc700 1 -- 172.21.15.102:0/3493365763 >> v1:172.21.15.181:6789/0 conn(0x7f7b58002840 legacy=0x7f7b58004ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:12.485 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.419+0000 7f7b667fc700 1 -- 172.21.15.102:0/3493365763 >> v1:172.21.15.181:6789/0 conn(0x7f7b58002840 legacy=0x7f7b58004ce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:07:12.485 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.426+0000 7f7b7cdf1700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:07:12.485 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.426+0000 7f7b7cdf1700 1 -- 172.21.15.102:0/3493365763 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f7b58009b00 con 0x7f7b58008dc0 2021-05-25T12:07:12.485 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.427+0000 7f7b667fc700 10 client.204333.objecter ms_handle_connect 0x7f7b58008dc0 2021-05-25T12:07:12.485 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.427+0000 7f7b667fc700 10 client.204333.objecter resend_mon_ops 2021-05-25T12:07:12.486 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.427+0000 7f7b667fc700 1 -- 172.21.15.102:0/3493365763 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2444136454 0 0) 0x7f7b58009b00 con 0x7f7b58008dc0 2021-05-25T12:07:12.486 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.427+0000 7f7b667fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:12.486 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.427+0000 7f7b667fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:12.486 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.427+0000 7f7b667fc700 10 monclient(hunting): my global_id is 224197 2021-05-25T12:07:12.487 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.427+0000 7f7b667fc700 1 -- 172.21.15.102:0/3493365763 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b580065a0 con 0x7f7b58008dc0 2021-05-25T12:07:12.487 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.427+0000 7f7b667fc700 1 -- 172.21.15.102:0/3493365763 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3485229079 0 0) 0x7f7b580065a0 con 0x7f7b58008dc0 2021-05-25T12:07:12.487 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.427+0000 7f7b667fc700 1 -- 172.21.15.102:0/3493365763 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7b58009b00 con 0x7f7b58008dc0 2021-05-25T12:07:12.487 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.427+0000 7f7b667fc700 1 -- 172.21.15.102:0/3493365763 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f7b6c004670 con 0x7f7b58008dc0 2021-05-25T12:07:12.487 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.427+0000 7f7b667fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:12.488 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.427+0000 7f7b667fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T12:07:12.488 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.427+0000 7f7b667fc700 10 monclient: dump: 2021-05-25T12:07:12.488 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:12.488 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:12.489 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:12.489 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:12.489 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:12.489 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:12.489 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:12.490 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:12.490 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:12.490 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:12.490 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.427+0000 7f7b667fc700 1 -- 172.21.15.102:0/3493365763 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1475737591 0 0) 0x7f7b58009b00 con 0x7f7b58008dc0 2021-05-25T12:07:12.491 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.427+0000 7f7b667fc700 1 -- 172.21.15.102:0/3493365763 --> v1:172.21.15.181:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f7b58008740 con 0x7f7b58008dc0 2021-05-25T12:07:12.491 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.427+0000 7f7b667fc700 1 -- 172.21.15.102:0/3493365763 <== mon.1 v1:172.21.15.181:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3931 (unknown 0 0 3630803704) 0x7f7b6c005890 con 0x7f7b58008dc0 2021-05-25T12:07:12.491 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.427+0000 7f7b667fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:07:12.491 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.427+0000 7f7b667fc700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:07:12.491 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.427+0000 7f7b667fc700 1 -- 172.21.15.102:0/3493365763 >> v1:172.21.15.181:6789/0 conn(0x7f7b58008dc0 legacy=0x7f7b58009250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:12.492 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.427+0000 7f7b667fc700 1 -- 172.21.15.102:0/3493365763 >> v1:172.21.15.181:6789/0 conn(0x7f7b58008dc0 legacy=0x7f7b58009250 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:07:12.492 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:07:12.492 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:07:12.493 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:07:12.493 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "leader", 2021-05-25T12:07:12.493 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 60, 2021-05-25T12:07:12.493 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:07:12.494 INFO:teuthology.orchestra.run.smithi102.stdout: 0, 2021-05-25T12:07:12.494 INFO:teuthology.orchestra.run.smithi102.stdout: 1 2021-05-25T12:07:12.494 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:07:12.494 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 0, 2021-05-25T12:07:12.494 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:07:12.495 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:07:12.495 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:07:12.495 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:07:12.495 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:07:12.496 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:07:12.496 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:07:12.496 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:07:12.496 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:07:12.496 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:07:12.497 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:07:12.497 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:07:12.497 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:07:12.497 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:07:12.498 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:07:12.498 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:07:12.498 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:07:12.498 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:07:12.498 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:07:12.499 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:07:12.499 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:07:12.499 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:07:12.499 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:07:12.500 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:07:12.500 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:07:12.500 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:12.500 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:07:12.500 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:07:12.501 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:07:12.501 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:07:12.501 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:07:12.501 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:07:12.502 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:07:12.502 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:07:12.502 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:07:12.502 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:07:12.502 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:07:12.503 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:07:12.503 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:07:12.503 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:07:12.503 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:07:12.504 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:07:12.504 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:07:12.504 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:07:12.504 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:07:12.504 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:07:12.505 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:07:12.505 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:07:12.505 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:07:12.505 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:07:12.505 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:07:12.506 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:07:12.506 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:12.506 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:07:12.506 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:12.507 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:07:12.507 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:07:12.507 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:07:12.507 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:07:12.508 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:12.508 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:07:12.508 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:07:12.508 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:07:12.508 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:07:12.509 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:07:12.509 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:12.509 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:07:12.509 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:07:12.509 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:07:12.510 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:07:12.510 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:07:12.510 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:12.510 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:12.511 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:07:12.511 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:07:12.511 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:07:12.511 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:07:12.511 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:12.512 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:07:12.512 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:07:12.512 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:07:12.512 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:07:12.513 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:07:12.513 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:12.513 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:07:12.513 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:07:12.513 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:07:12.514 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:07:12.514 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:07:12.514 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:12.514 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:12.514 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:07:12.515 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:07:12.515 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:07:12.515 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:07:12.515 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:12.516 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:07:12.516 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:07:12.516 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:07:12.516 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:07:12.516 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:07:12.517 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:12.517 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:07:12.517 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:07:12.517 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:07:12.517 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:07:12.518 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:07:12.518 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:07:12.518 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:07:12.518 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:12.519 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:07:12.519 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:07:12.519 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:12.519 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:07:12.519 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:07:12.520 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:07:12.520 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:07:12.520 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:07:12.520 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:07:12.520 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:12.521 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:07:12.521 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:07:12.521 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 5 2021-05-25T12:07:12.521 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:07:12.521 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:07:12.522 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": [ 2021-05-25T12:07:12.522 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:12.522 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:07:12.522 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:07:12.523 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:07:12.523 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:07:12.523 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:07:12.523 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:12.523 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:07:12.524 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:07:12.526 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.428+0000 7f7b7cdf1700 10 librados: watch_flush enter 2021-05-25T12:07:12.526 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.428+0000 7f7b7cdf1700 10 librados: watch_flush exit 2021-05-25T12:07:12.526 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.428+0000 7f7b7cdf1700 20 client.204333.objecter shutdown clearing up homeless session... 2021-05-25T12:07:12.526 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.428+0000 7f7b7cdf1700 10 client.204333.objecter successfully canceled tick 2021-05-25T12:07:12.527 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.428+0000 7f7b7cdf1700 1 -- 172.21.15.102:0/3493365763 >> v1:172.21.15.181:6812/29582 conn(0x7f7b54033150 legacy=0x7f7b54035610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:12.527 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.428+0000 7f7b7cdf1700 10 monclient: shutdown 2021-05-25T12:07:12.527 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.428+0000 7f7b7cdf1700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:07:12.527 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.428+0000 7f7b7cdf1700 1 -- 172.21.15.102:0/3493365763 >> v1:172.21.15.102:6789/0 conn(0x7f7b78096790 legacy=0x7f7b7812b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:12.528 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.428+0000 7f7b76d9d700 1 -- 172.21.15.102:0/3493365763 reap_dead start 2021-05-25T12:07:12.528 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.428+0000 7f7b76d9d700 1 -- 172.21.15.102:0/3493365763 reap_dead start 2021-05-25T12:07:12.528 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.428+0000 7f7b7cdf1700 1 -- 172.21.15.102:0/3493365763 shutdown_connections 2021-05-25T12:07:12.528 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.428+0000 7f7b7cdf1700 1 -- 172.21.15.102:0/3493365763 >> 172.21.15.102:0/3493365763 conn(0x7f7b7809cd10 msgr2=0x7f7b78066130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:07:12.528 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.428+0000 7f7b7cdf1700 1 -- 172.21.15.102:0/3493365763 shutdown_connections 2021-05-25T12:07:12.529 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.428+0000 7f7b7cdf1700 1 -- 172.21.15.102:0/3493365763 wait complete. 2021-05-25T12:07:12.529 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:12.428+0000 7f7b7cdf1700 1 librados: shutdown 2021-05-25T12:07:12.529 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2021-05-25T12:07:21.954 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetRead 2021-05-25T12:07:21.955 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: hmm, no HitSet yet 2021-05-25T12:07:21.955 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: ok, hit_set contains 174:602f83fe:::foo:head 2021-05-25T12:07:21.955 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetRead (23172 ms) 2021-05-25T12:07:21.956 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetWrite 2021-05-25T12:07:21.956 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: pg_num = 32 2021-05-25T12:07:21.956 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: pg 0 ls 1621944442,0 2021-05-25T12:07:21.956 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: pg 1 ls 1621944442,0 2021-05-25T12:07:21.957 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: pg 2 ls 1621944442,0 2021-05-25T12:07:21.957 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: pg 3 ls 1621944442,0 2021-05-25T12:07:21.957 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: pg 4 ls 1621944442,0 2021-05-25T12:07:21.957 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: pg 5 ls 1621944442,0 2021-05-25T12:07:21.958 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: pg 6 ls 1621944442,0 2021-05-25T12:07:21.958 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: pg 7 ls 1621944442,0 2021-05-25T12:07:21.958 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: pg 8 ls 1621944442,0 2021-05-25T12:07:21.958 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: pg 9 ls 1621944442,0 2021-05-25T12:07:21.958 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: pg 10 ls 1621944442,0 2021-05-25T12:07:21.959 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: pg 11 ls 1621944442,0 2021-05-25T12:07:21.959 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: pg 12 ls 1621944442,0 2021-05-25T12:07:21.959 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: pg 13 ls 1621944442,0 2021-05-25T12:07:21.959 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: pg 14 ls 1621944442,0 2021-05-25T12:07:21.959 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: pg 15 ls 1621944442,0 2021-05-25T12:07:21.960 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: pg 16 ls 1621944442,0 2021-05-25T12:07:21.960 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: pg 17 ls 1621944442,0 2021-05-25T12:07:21.960 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: pg 18 ls 1621944442,0 2021-05-25T12:07:21.960 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: pg 19 ls 1621944442,0 2021-05-25T12:07:21.960 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: pg 20 ls 1621944442,0 2021-05-25T12:07:21.961 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: pg 21 ls 1621944442,0 2021-05-25T12:07:21.961 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: pg 22 ls 1621944442,0 2021-05-25T12:07:21.961 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: pg 23 ls 1621944442,0 2021-05-25T12:07:21.961 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: pg 24 ls 1621944442,0 2021-05-25T12:07:21.961 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: pg 25 ls 1621944442,0 2021-05-25T12:07:21.962 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: pg 26 ls 1621944442,0 2021-05-25T12:07:21.962 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: pg 27 ls 1621944442,0 2021-05-25T12:07:21.962 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: pg 28 ls 1621944442,0 2021-05-25T12:07:21.962 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: pg 29 ls 1621944442,0 2021-05-25T12:07:21.963 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: pg 30 ls 1621944442,0 2021-05-25T12:07:21.963 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: pg 31 ls 1621944442,0 2021-05-25T12:07:21.963 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: pg_num = 32 2021-05-25T12:07:21.963 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:6cac518f:::0:head 2021-05-25T12:07:21.964 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:02547ec2:::1:head 2021-05-25T12:07:21.964 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:f905c69b:::2:head 2021-05-25T12:07:21.964 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:cfc208b3:::3:head 2021-05-25T12:07:21.964 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:d83876eb:::4:head 2021-05-25T12:07:21.964 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:b29083e3:::5:head 2021-05-25T12:07:21.965 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:c4fdafeb:::6:head 2021-05-25T12:07:21.965 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:5c6b0b28:::7:head 2021-05-25T12:07:21.965 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:bd63b0f1:::8:head 2021-05-25T12:07:21.965 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:e960b815:::9:head 2021-05-25T12:07:21.966 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:52ea6a34:::10:head 2021-05-25T12:07:21.966 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:89d3ae78:::11:head 2021-05-25T12:07:21.966 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:de5d7c5f:::12:head 2021-05-25T12:07:21.966 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:566253c9:::13:head 2021-05-25T12:07:21.966 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:62a1935d:::14:head 2021-05-25T12:07:21.967 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:863748b0:::15:head 2021-05-25T12:07:21.967 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:3958e169:::16:head 2021-05-25T12:07:21.967 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:4d4dabf9:::17:head 2021-05-25T12:07:21.967 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:8391935d:::18:head 2021-05-25T12:07:21.967 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:28883081:::19:head 2021-05-25T12:07:21.968 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:69259c59:::20:head 2021-05-25T12:07:21.968 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:4bdb80b7:::21:head 2021-05-25T12:07:21.968 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:a11c5d71:::22:head 2021-05-25T12:07:21.968 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:271af37b:::23:head 2021-05-25T12:07:21.969 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:95b121be:::24:head 2021-05-25T12:07:21.969 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:58d1031b:::25:head 2021-05-25T12:07:21.969 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:0a050783:::26:head 2021-05-25T12:07:21.969 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:c709704c:::27:head 2021-05-25T12:07:21.969 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:cbe56eaf:::28:head 2021-05-25T12:07:21.970 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:86b4b162:::29:head 2021-05-25T12:07:21.970 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:70d89383:::30:head 2021-05-25T12:07:21.970 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:dd450c7c:::31:head 2021-05-25T12:07:21.970 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:6d5729b1:::32:head 2021-05-25T12:07:21.970 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:c388f3fb:::33:head 2021-05-25T12:07:21.971 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:56cfea31:::34:head 2021-05-25T12:07:21.971 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:9dbc1bf7:::35:head 2021-05-25T12:07:21.971 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:40b74ccd:::36:head 2021-05-25T12:07:21.971 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:4d5aaf42:::37:head 2021-05-25T12:07:21.971 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:920f362c:::38:head 2021-05-25T12:07:21.972 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:6cc53222:::39:head 2021-05-25T12:07:21.972 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:9cad833f:::40:head 2021-05-25T12:07:21.972 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:1ea84d41:::41:head 2021-05-25T12:07:21.972 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:c4480ef6:::42:head 2021-05-25T12:07:21.972 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:a694361e:::43:head 2021-05-25T12:07:21.973 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:d1bd33e9:::44:head 2021-05-25T12:07:21.973 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:ddc2cd5d:::45:head 2021-05-25T12:07:21.973 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:2b782207:::46:head 2021-05-25T12:07:21.973 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:7b187fca:::47:head 2021-05-25T12:07:21.974 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:90ecdf6f:::48:head 2021-05-25T12:07:21.974 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:a5ed95fe:::49:head 2021-05-25T12:07:21.974 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:ea0eaa55:::50:head 2021-05-25T12:07:21.974 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:f33ef17b:::51:head 2021-05-25T12:07:21.975 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:a0d1b2f6:::52:head 2021-05-25T12:07:21.975 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:60c5229e:::53:head 2021-05-25T12:07:21.975 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:edcbc575:::54:head 2021-05-25T12:07:21.975 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:102cf253:::55:head 2021-05-25T12:07:21.975 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:efb7fb0b:::56:head 2021-05-25T12:07:21.976 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:50d0a326:::57:head 2021-05-25T12:07:21.976 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:d4dc5daf:::58:head 2021-05-25T12:07:21.976 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:3a130462:::59:head 2021-05-25T12:07:21.976 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:ec87ed71:::60:head 2021-05-25T12:07:21.976 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:d5bc9454:::61:head 2021-05-25T12:07:21.977 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:3ddfe313:::62:head 2021-05-25T12:07:21.977 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:7c2816b9:::63:head 2021-05-25T12:07:21.977 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:47e00e4d:::64:head 2021-05-25T12:07:21.977 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:c6410c18:::65:head 2021-05-25T12:07:21.978 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:b48ed237:::66:head 2021-05-25T12:07:21.978 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:cd63ad31:::67:head 2021-05-25T12:07:21.978 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:b179e92b:::68:head 2021-05-25T12:07:21.978 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:0d9f741a:::69:head 2021-05-25T12:07:21.978 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:6d3352ae:::70:head 2021-05-25T12:07:21.979 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:c6d5c19e:::71:head 2021-05-25T12:07:21.979 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:bc4729c3:::72:head 2021-05-25T12:07:21.979 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:77e930b9:::73:head 2021-05-25T12:07:21.979 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:0abeecfd:::74:head 2021-05-25T12:07:21.979 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:b7c37e15:::75:head 2021-05-25T12:07:21.980 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:b6378398:::76:head 2021-05-25T12:07:21.980 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:02bd68de:::77:head 2021-05-25T12:07:21.980 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:cc795d2d:::78:head 2021-05-25T12:07:21.980 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:630d4fea:::79:head 2021-05-25T12:07:21.980 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:e0d29ef5:::80:head 2021-05-25T12:07:21.981 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:fd6f13d2:::81:head 2021-05-25T12:07:21.981 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:606461d5:::82:head 2021-05-25T12:07:21.981 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:eadbdc43:::83:head 2021-05-25T12:07:21.981 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:8761d0bb:::84:head 2021-05-25T12:07:21.981 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:9ef0186f:::85:head 2021-05-25T12:07:21.982 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:e0d41294:::86:head 2021-05-25T12:07:21.982 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:961de695:::87:head 2021-05-25T12:07:21.982 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:1423148f:::88:head 2021-05-25T12:07:21.982 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:633a8fa2:::89:head 2021-05-25T12:07:21.983 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:a8653809:::90:head 2021-05-25T12:07:21.983 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:3dac8b33:::91:head 2021-05-25T12:07:21.983 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:35aad435:::92:head 2021-05-25T12:07:21.983 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:f6dcc343:::93:head 2021-05-25T12:07:21.983 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:dbbdad87:::94:head 2021-05-25T12:07:21.984 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:1cb48ce0:::95:head 2021-05-25T12:07:21.984 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:03cd461c:::96:head 2021-05-25T12:07:21.984 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:17a4ea99:::97:head 2021-05-25T12:07:21.984 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:9993c9a7:::98:head 2021-05-25T12:07:21.984 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:6394211c:::99:head 2021-05-25T12:07:21.985 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:94c7ae57:::100:head 2021-05-25T12:07:21.985 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:6fdee5bb:::101:head 2021-05-25T12:07:21.985 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:9a477fd1:::102:head 2021-05-25T12:07:21.985 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:eb850916:::103:head 2021-05-25T12:07:21.986 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:affc56b9:::104:head 2021-05-25T12:07:21.986 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:b42dc814:::105:head 2021-05-25T12:07:21.986 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:f319f8f0:::106:head 2021-05-25T12:07:21.986 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:9a40b9de:::107:head 2021-05-25T12:07:21.986 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:8b524f28:::108:head 2021-05-25T12:07:21.987 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:e3de589f:::109:head 2021-05-25T12:07:21.987 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:90f90a5b:::110:head 2021-05-25T12:07:21.987 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:a7b4f1d7:::111:head 2021-05-25T12:07:21.987 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:af51766e:::112:head 2021-05-25T12:07:21.988 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:b6f90bd1:::113:head 2021-05-25T12:07:21.988 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:e0261208:::114:head 2021-05-25T12:07:21.988 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:c9569ef7:::115:head 2021-05-25T12:07:21.988 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:61bebe50:::116:head 2021-05-25T12:07:21.988 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:fe93412b:::117:head 2021-05-25T12:07:21.989 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:d3d38bee:::118:head 2021-05-25T12:07:21.989 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:3100ba0c:::119:head 2021-05-25T12:07:21.989 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:d0560ada:::120:head 2021-05-25T12:07:21.989 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:f0ea8b35:::121:head 2021-05-25T12:07:21.989 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:766f231a:::122:head 2021-05-25T12:07:21.990 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:a07a2582:::123:head 2021-05-25T12:07:21.990 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:bd7c6b3a:::124:head 2021-05-25T12:07:21.990 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:fb2ddaff:::125:head 2021-05-25T12:07:21.990 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:4408e1fe:::126:head 2021-05-25T12:07:21.991 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:ee1df7a7:::127:head 2021-05-25T12:07:21.991 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:c3002909:::128:head 2021-05-25T12:07:21.991 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:4f48ffa9:::129:head 2021-05-25T12:07:21.991 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:edf38733:::130:head 2021-05-25T12:07:21.991 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:c08425c0:::131:head 2021-05-25T12:07:21.992 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:5f902d98:::132:head 2021-05-25T12:07:21.992 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:41ea2c93:::133:head 2021-05-25T12:07:21.992 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:813cee13:::134:head 2021-05-25T12:07:21.992 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:0131818d:::135:head 2021-05-25T12:07:21.993 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:26ba5a85:::136:head 2021-05-25T12:07:21.993 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:381b8a5a:::137:head 2021-05-25T12:07:21.993 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:28797e47:::138:head 2021-05-25T12:07:21.993 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:bfca7f22:::139:head 2021-05-25T12:07:21.993 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:36807075:::140:head 2021-05-25T12:07:21.994 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:80b03975:::141:head 2021-05-25T12:07:21.994 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:5c15709b:::142:head 2021-05-25T12:07:21.994 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:f39ea15e:::143:head 2021-05-25T12:07:21.994 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:ea992956:::144:head 2021-05-25T12:07:21.995 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:48887b1c:::145:head 2021-05-25T12:07:21.995 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:9f24a9dd:::146:head 2021-05-25T12:07:21.995 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:987f100b:::147:head 2021-05-25T12:07:21.995 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:d2dd3581:::148:head 2021-05-25T12:07:21.995 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:7fed1808:::149:head 2021-05-25T12:07:21.996 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:c80b70e9:::150:head 2021-05-25T12:07:21.996 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:85ed90f9:::151:head 2021-05-25T12:07:21.996 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:36428b24:::152:head 2021-05-25T12:07:21.996 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:d044c34a:::153:head 2021-05-25T12:07:21.996 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:7c18bf58:::154:head 2021-05-25T12:07:21.997 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:d1c21232:::155:head 2021-05-25T12:07:21.997 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:a7a3c575:::156:head 2021-05-25T12:07:21.997 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:87da0633:::157:head 2021-05-25T12:07:21.997 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:d5ac3822:::158:head 2021-05-25T12:07:21.998 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:3f20522d:::159:head 2021-05-25T12:07:21.998 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:6ca26563:::160:head 2021-05-25T12:07:21.998 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:532ce135:::161:head 2021-05-25T12:07:21.998 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:c78863e6:::162:head 2021-05-25T12:07:28.639 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:7462ddf6:EC.RoundTrip 2021-05-25T12:07:28.639 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip (32123 ms) 2021-05-25T12:07:28.639 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip2 2021-05-25T12:07:28.640 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip2 (7027 ms) 2021-05-25T12:07:28.640 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripAppend 2021-05-25T12:07:28.640 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripAppend (6996 ms) 2021-05-25T12:07:28.640 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [ RUN ] LibRadosAioEC.IsComplete 2021-05-25T12:07:28.641 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [ OK ] LibRadosAioEC.IsComplete (40693 ms) 2021-05-25T12:07:28.641 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [ RUN ] LibRadosAioEC.IsSafe 2021-05-25T12:07:28.641 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [ OK ] LibRadosAioEC.IsSafe (20987 ms) 2021-05-25T12:07:28.641 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [ RUN ] LibRadosAioEC.ReturnValue 2021-05-25T12:07:28.641 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [ OK ] LibRadosAioEC.ReturnValue (7002 ms) 2021-05-25T12:07:28.642 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [ RUN ] LibRadosAioEC.Flush 2021-05-25T12:07:28.642 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [ OK ] LibRadosAioEC.Flush (7021 ms) 2021-05-25T12:07:28.642 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [ RUN ] LibRadosAioEC.FlushAsync 2021-05-25T12:07:28.642 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [ OK ] LibRadosAioEC.FlushAsync (21078 ms) 2021-05-25T12:07:28.642 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripWriteFull 2021-05-25T12:07:28.643 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripWriteFull (6903 ms) 2021-05-25T12:07:28.643 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStat 2021-05-25T12:07:28.643 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStat (7010 ms) 2021-05-25T12:07:28.643 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStatNS 2021-05-25T12:07:28.644 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStatNS (7083 ms) 2021-05-25T12:07:28.644 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [ RUN ] LibRadosAioEC.StatRemove 2021-05-25T12:07:28.644 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [ OK ] LibRadosAioEC.StatRemove (21097 ms) 2021-05-25T12:07:28.644 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [ RUN ] LibRadosAioEC.ExecuteClass 2021-05-25T12:07:28.644 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [ OK ] LibRadosAioEC.ExecuteClass (6715 ms) 2021-05-25T12:07:28.645 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [ RUN ] LibRadosAioEC.MultiWrite 2021-05-25T12:07:28.645 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [ OK ] LibRadosAioEC.MultiWrite (6993 ms) 2021-05-25T12:07:28.645 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [----------] 16 tests from LibRadosAioEC (243574 ms total) 2021-05-25T12:07:28.645 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: 2021-05-25T12:07:28.645 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [----------] Global test environment tear-down 2021-05-25T12:07:28.646 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [==========] 40 tests from 2 test suites ran. (408956 ms total) 2021-05-25T12:07:28.646 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio: [ PASSED ] 40 tests. 2021-05-25T12:07:32.530 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2021-05-25T12:07:32.531 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2021-05-25T12:07:32.531 INFO:tasks.ceph.mon.c:Restarting daemon 2021-05-25T12:07:32.532 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2021-05-25T12:07:32.535 INFO:tasks.ceph.mon.c:Started 2021-05-25T12:07:32.535 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2021-05-25T12:07:32.536 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2021-05-25T12:07:32.672 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.671+0000 7fb6006ef700 10 monclient: get_monmap_and_config 2021-05-25T12:07:32.672 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.671+0000 7fb6006ef700 10 monclient: build_initial_monmap 2021-05-25T12:07:32.673 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.671+0000 7fb6006ef700 10 monclient: monmap: 2021-05-25T12:07:32.673 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:07:32.673 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:32.673 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:07:32.672064+0000 2021-05-25T12:07:32.674 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:07:32.672064+0000 2021-05-25T12:07:32.674 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:07:32.674 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:32.674 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:07:32.674 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:07:32.675 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:07:32.675 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:32.677 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.671+0000 7fb6006ef700 1 Processor -- start 2021-05-25T12:07:32.677 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.671+0000 7fb6006ef700 1 -- start start 2021-05-25T12:07:32.677 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.671+0000 7fb6006ef700 10 monclient: init 2021-05-25T12:07:32.677 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.672+0000 7fb6006ef700 10 monclient: _reopen_session rank -1 2021-05-25T12:07:32.677 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.672+0000 7fb6006ef700 10 monclient: _add_conns ranks=[2,0,1] 2021-05-25T12:07:32.678 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.672+0000 7fb6006ef700 10 monclient(hunting): picked mon.noname-b con 0x7fb5f8093e10 addr v1:172.21.15.102:6790/0 2021-05-25T12:07:32.678 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.672+0000 7fb6006ef700 10 monclient(hunting): picked mon.noname-a con 0x7fb5f8091b00 addr v1:172.21.15.102:6789/0 2021-05-25T12:07:32.678 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.672+0000 7fb6006ef700 10 monclient(hunting): picked mon.noname-c con 0x7fb5f8094900 addr v1:172.21.15.181:6789/0 2021-05-25T12:07:32.678 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.672+0000 7fb6006ef700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5f8092a90 con 0x7fb5f8091b00 2021-05-25T12:07:32.679 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.672+0000 7fb6006ef700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5f8092d30 con 0x7fb5f8094900 2021-05-25T12:07:32.679 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.672+0000 7fb6006ef700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5f8092fd0 con 0x7fb5f8093e10 2021-05-25T12:07:32.679 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.672+0000 7fb6006ef700 10 monclient(hunting): _renew_subs 2021-05-25T12:07:32.679 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.672+0000 7fb6006ef700 10 monclient(hunting): authenticate will time out at 1253.269544s 2021-05-25T12:07:32.679 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.672+0000 7fb5fdc8a700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fb5f8091b00 0x7fb5f80941f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33766/0 (socket says 172.21.15.102:33766) 2021-05-25T12:07:32.680 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.672+0000 7fb5fe48b700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fb5f8093e10 0x7fb5f8093550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:56874/0 (socket says 172.21.15.102:56874) 2021-05-25T12:07:32.680 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.672+0000 7fb5fdc8a700 1 -- 172.21.15.102:0/2502425750 learned_addr learned my addr 172.21.15.102:0/2502425750 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:07:32.689 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.683+0000 7fb5fd489700 1 -- 172.21.15.102:0/2502425750 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 242171528 0 0) 0x7fb5f8092a90 con 0x7fb5f8091b00 2021-05-25T12:07:32.689 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.683+0000 7fb5fd489700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:32.689 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.683+0000 7fb5fd489700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:32.690 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.683+0000 7fb5fd489700 10 monclient(hunting): my global_id is 204366 2021-05-25T12:07:32.690 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.683+0000 7fb5fd489700 1 -- 172.21.15.102:0/2502425750 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5e0004380 con 0x7fb5f8091b00 2021-05-25T12:07:32.690 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.683+0000 7fb5fd489700 1 -- 172.21.15.102:0/2502425750 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 571128337 0 0) 0x7fb5f8092d30 con 0x7fb5f8094900 2021-05-25T12:07:32.690 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.683+0000 7fb5fd489700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:32.691 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.683+0000 7fb5fd489700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:32.691 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.683+0000 7fb5fd489700 10 monclient(hunting): my global_id is 224230 2021-05-25T12:07:32.691 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.683+0000 7fb5fd489700 1 -- 172.21.15.102:0/2502425750 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5f8092a90 con 0x7fb5f8094900 2021-05-25T12:07:32.691 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.683+0000 7fb5fd489700 1 -- 172.21.15.102:0/2502425750 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 743247292 0 0) 0x7fb5e0004380 con 0x7fb5f8091b00 2021-05-25T12:07:32.692 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.683+0000 7fb5fd489700 1 -- 172.21.15.102:0/2502425750 >> v1:172.21.15.102:6790/0 conn(0x7fb5f8093e10 legacy=0x7fb5f8093550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:32.692 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.683+0000 7fb5fd489700 1 -- 172.21.15.102:0/2502425750 >> v1:172.21.15.181:6789/0 conn(0x7fb5f8094900 legacy=0x7fb5f8130130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:32.692 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.683+0000 7fb5fd489700 10 monclient: _finish_hunting 0 2021-05-25T12:07:32.692 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.683+0000 7fb5fd489700 1 monclient: found mon.noname-a 2021-05-25T12:07:32.692 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.683+0000 7fb5fd489700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:07:32.693 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.683+0000 7fb5fd489700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T12:07:32.693 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.683+0000 7fb5fd489700 1 -- 172.21.15.102:0/2502425750 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5f8134880 con 0x7fb5f8091b00 2021-05-25T12:07:32.693 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.683+0000 7fb5fd489700 10 monclient: _finish_auth 0 2021-05-25T12:07:32.693 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.683+0000 7fb5fd489700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:07:32.693 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.683+0000 7fb5fd489700 1 -- 172.21.15.102:0/2502425750 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fb5ec001610 con 0x7fb5f8091b00 2021-05-25T12:07:32.694 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.683+0000 7fb6006ef700 5 monclient: authenticate success, global_id 204366 2021-05-25T12:07:32.694 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.683+0000 7fb6006ef700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:07:32.694 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.684+0000 7fb5fd489700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:32.694 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.684+0000 7fb5fd489700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:07:32.695 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.684+0000 7fb5fd489700 10 monclient: dump: 2021-05-25T12:07:32.695 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:32.695 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:32.695 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:32.695 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:32.696 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:32.696 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:32.696 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:32.696 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:32.696 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:32.697 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:32.697 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.684+0000 7fb6006ef700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:07:32.697 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.684+0000 7fb5fd489700 1 -- 172.21.15.102:0/2502425750 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb5ec001880 con 0x7fb5f8091b00 2021-05-25T12:07:32.697 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.684+0000 7fb5fd489700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:07:32.698 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.684+0000 7fb6006ef700 10 monclient: get_monmap_and_config success 2021-05-25T12:07:32.698 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.684+0000 7fb6006ef700 4 set_mon_vals no callback set 2021-05-25T12:07:32.698 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.684+0000 7fb5fd489700 1 -- 172.21.15.102:0/2502425750 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fb5ec001c10 con 0x7fb5f8091b00 2021-05-25T12:07:32.698 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.684+0000 7fb5fd489700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:32.698 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.684+0000 7fb5fd489700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:07:32.699 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.684+0000 7fb5fd489700 10 monclient: dump: 2021-05-25T12:07:32.699 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:32.699 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:32.699 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:32.699 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:32.700 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:32.700 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:32.700 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:32.700 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:32.700 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:32.701 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:32.701 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.684+0000 7fb6006ef700 10 monclient: shutdown 2021-05-25T12:07:32.701 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.684+0000 7fb6006ef700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:07:32.701 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.684+0000 7fb6006ef700 1 -- 172.21.15.102:0/2502425750 >> v1:172.21.15.102:6789/0 conn(0x7fb5f8091b00 legacy=0x7fb5f80941f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:32.701 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.684+0000 7fb6006ef700 1 -- 172.21.15.102:0/2502425750 shutdown_connections 2021-05-25T12:07:32.702 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.684+0000 7fb6006ef700 1 -- 172.21.15.102:0/2502425750 >> 172.21.15.102:0/2502425750 conn(0x7fb5f809a910 msgr2=0x7fb5f809ad20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:07:32.702 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.684+0000 7fb6006ef700 1 -- 172.21.15.102:0/2502425750 shutdown_connections 2021-05-25T12:07:32.702 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.684+0000 7fb6006ef700 1 -- 172.21.15.102:0/2502425750 wait complete. 2021-05-25T12:07:32.702 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.685+0000 7fb6006ef700 10 monclient: build_initial_monmap 2021-05-25T12:07:32.702 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.685+0000 7fb6006ef700 10 monclient: monmap: 2021-05-25T12:07:32.703 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:07:32.703 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:07:32.703 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:07:32.703 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:07:32.703 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:07:32.704 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:07:32.704 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:07:32.704 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:07:32.704 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:07:32.704 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:32.705 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.685+0000 7fb6006ef700 1 librados: starting msgr at 2021-05-25T12:07:32.705 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.685+0000 7fb6006ef700 1 librados: starting objecter 2021-05-25T12:07:32.705 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.685+0000 7fb6006ef700 1 Processor -- start 2021-05-25T12:07:32.705 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.685+0000 7fb6006ef700 1 -- start start 2021-05-25T12:07:32.706 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.685+0000 7fb6006ef700 1 librados: setting wanted keys 2021-05-25T12:07:32.706 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.685+0000 7fb6006ef700 1 librados: calling monclient init 2021-05-25T12:07:32.706 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.685+0000 7fb6006ef700 10 monclient: init 2021-05-25T12:07:32.706 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.686+0000 7fb6006ef700 10 monclient: _reopen_session rank -1 2021-05-25T12:07:32.706 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.686+0000 7fb6006ef700 10 monclient: _add_conns ranks=[2,1,0] 2021-05-25T12:07:32.707 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.686+0000 7fb6006ef700 10 monclient(hunting): picked mon.noname-c con 0x7fb5f8091b00 addr v1:172.21.15.102:6790/0 2021-05-25T12:07:32.707 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.686+0000 7fb6006ef700 10 monclient(hunting): picked mon.noname-b con 0x7fb5f8093e10 addr v1:172.21.15.181:6789/0 2021-05-25T12:07:32.707 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.686+0000 7fb6006ef700 10 monclient(hunting): picked mon.noname-a con 0x7fb5f8094900 addr v1:172.21.15.102:6789/0 2021-05-25T12:07:32.707 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.686+0000 7fb5fec8c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fb5f8094900 0x7fb5f8140fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33774/0 (socket says 172.21.15.102:33774) 2021-05-25T12:07:32.707 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.686+0000 7fb5fec8c700 1 -- 172.21.15.102:0/3884068671 learned_addr learned my addr 172.21.15.102:0/3884068671 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:07:32.708 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.686+0000 7fb6006ef700 1 -- 172.21.15.102:0/3884068671 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5f8118310 con 0x7fb5f8094900 2021-05-25T12:07:32.708 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.686+0000 7fb5e67fc700 10 client.?.objecter ms_handle_connect 0x7fb5f8094900 2021-05-25T12:07:32.708 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.686+0000 7fb5e67fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:07:32.708 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.686+0000 7fb6006ef700 1 -- 172.21.15.102:0/3884068671 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5f81342b0 con 0x7fb5f8093e10 2021-05-25T12:07:32.708 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.686+0000 7fb5e67fc700 10 client.?.objecter ms_handle_connect 0x7fb5f8093e10 2021-05-25T12:07:32.709 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.686+0000 7fb5e67fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:07:32.709 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.686+0000 7fb6006ef700 1 -- 172.21.15.102:0/3884068671 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5f811a100 con 0x7fb5f8091b00 2021-05-25T12:07:32.709 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.686+0000 7fb6006ef700 10 monclient(hunting): _renew_subs 2021-05-25T12:07:32.709 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.686+0000 7fb6006ef700 10 monclient(hunting): authenticate will time out at 1253.283703s 2021-05-25T12:07:32.709 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.686+0000 7fb5e67fc700 10 client.?.objecter ms_handle_connect 0x7fb5f8091b00 2021-05-25T12:07:32.710 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.687+0000 7fb5e67fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:07:32.713 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.688+0000 7fb5e67fc700 1 -- 172.21.15.102:0/3884068671 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3767643578 0 0) 0x7fb5f81342b0 con 0x7fb5f8093e10 2021-05-25T12:07:32.714 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.688+0000 7fb5e67fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:32.714 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.688+0000 7fb5e67fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:32.714 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.688+0000 7fb5e67fc700 10 monclient(hunting): my global_id is 224233 2021-05-25T12:07:32.714 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.688+0000 7fb5e67fc700 1 -- 172.21.15.102:0/3884068671 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5d8004c10 con 0x7fb5f8093e10 2021-05-25T12:07:32.714 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.689+0000 7fb5e67fc700 1 -- 172.21.15.102:0/3884068671 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3859703245 0 0) 0x7fb5d8004c10 con 0x7fb5f8093e10 2021-05-25T12:07:32.715 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.689+0000 7fb5e67fc700 1 -- 172.21.15.102:0/3884068671 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb5f81342b0 con 0x7fb5f8093e10 2021-05-25T12:07:32.715 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.689+0000 7fb5e67fc700 1 -- 172.21.15.102:0/3884068671 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fb5ec001690 con 0x7fb5f8093e10 2021-05-25T12:07:32.715 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.689+0000 7fb5e67fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:32.715 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.689+0000 7fb5e67fc700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:07:32.716 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.689+0000 7fb5e67fc700 10 monclient(hunting): dump: 2021-05-25T12:07:32.716 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:32.716 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:32.716 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:32.716 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:32.717 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:32.717 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:32.717 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:32.717 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:32.717 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:32.718 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:32.718 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.689+0000 7fb5e67fc700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:07:32.718 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.689+0000 7fb5e67fc700 1 -- 172.21.15.102:0/3884068671 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3549366702 0 0) 0x7fb5f811a100 con 0x7fb5f8091b00 2021-05-25T12:07:32.718 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.689+0000 7fb5e67fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:32.718 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.689+0000 7fb5e67fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:32.719 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.689+0000 7fb5e67fc700 10 monclient(hunting): my global_id is 234101 2021-05-25T12:07:32.719 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.689+0000 7fb5e67fc700 1 -- 172.21.15.102:0/3884068671 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5d8004c10 con 0x7fb5f8091b00 2021-05-25T12:07:32.719 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.689+0000 7fb5e67fc700 1 -- 172.21.15.102:0/3884068671 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4028677737 0 0) 0x7fb5d8004c10 con 0x7fb5f8091b00 2021-05-25T12:07:32.719 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.689+0000 7fb5e67fc700 1 -- 172.21.15.102:0/3884068671 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb5f811a100 con 0x7fb5f8091b00 2021-05-25T12:07:32.719 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.689+0000 7fb5e67fc700 1 -- 172.21.15.102:0/3884068671 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fb5e80017b0 con 0x7fb5f8091b00 2021-05-25T12:07:32.720 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.689+0000 7fb5e67fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:32.720 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.689+0000 7fb5e67fc700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2021-05-25T12:07:32.720 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.689+0000 7fb5e67fc700 10 monclient(hunting): dump: 2021-05-25T12:07:32.720 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:32.720 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:32.721 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:32.721 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:32.721 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:32.721 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:32.721 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:32.722 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:32.722 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:32.722 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:32.722 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.689+0000 7fb5e67fc700 1 -- 172.21.15.102:0/3884068671 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3786914648 0 0) 0x7fb5f811a100 con 0x7fb5f8091b00 2021-05-25T12:07:32.722 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.689+0000 7fb5e67fc700 1 -- 172.21.15.102:0/3884068671 >> v1:172.21.15.181:6789/0 conn(0x7fb5f8093e10 legacy=0x7fb5f8133ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:32.723 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.689+0000 7fb5e67fc700 1 -- 172.21.15.102:0/3884068671 >> v1:172.21.15.102:6789/0 conn(0x7fb5f8094900 legacy=0x7fb5f8140fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:32.723 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.689+0000 7fb5e67fc700 10 monclient: _finish_hunting 0 2021-05-25T12:07:32.723 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.689+0000 7fb5e67fc700 1 monclient: found mon.c 2021-05-25T12:07:32.723 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.689+0000 7fb5e67fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:07:32.723 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.689+0000 7fb5e67fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:07:32.724 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.689+0000 7fb5e67fc700 1 -- 172.21.15.102:0/3884068671 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5f811a3a0 con 0x7fb5f8091b00 2021-05-25T12:07:32.724 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.689+0000 7fb5e67fc700 10 monclient: _finish_auth 0 2021-05-25T12:07:32.724 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.689+0000 7fb5e67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:07:32.724 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.689+0000 7fb6006ef700 5 monclient: authenticate success, global_id 234101 2021-05-25T12:07:32.724 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.689+0000 7fb6006ef700 10 monclient: _renew_subs 2021-05-25T12:07:32.724 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.689+0000 7fb6006ef700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:07:32.725 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.689+0000 7fb6006ef700 1 -- 172.21.15.102:0/3884068671 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5f8092220 con 0x7fb5f8091b00 2021-05-25T12:07:32.725 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.689+0000 7fb6006ef700 10 client.234101.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:07:32.725 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.689+0000 7fb6006ef700 10 monclient: _renew_subs 2021-05-25T12:07:32.725 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.689+0000 7fb6006ef700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:07:32.725 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.689+0000 7fb6006ef700 1 -- 172.21.15.102:0/3884068671 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5f8118d10 con 0x7fb5f8091b00 2021-05-25T12:07:32.726 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.689+0000 7fb6006ef700 1 librados: init done 2021-05-25T12:07:32.726 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.689+0000 7fb5e67fc700 1 -- 172.21.15.102:0/3884068671 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb5e8004530 con 0x7fb5f8091b00 2021-05-25T12:07:32.726 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.689+0000 7fb5e67fc700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:07:32.726 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.689+0000 7fb5e67fc700 1 -- 172.21.15.102:0/3884068671 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fb5e8004840 con 0x7fb5f8091b00 2021-05-25T12:07:32.727 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.689+0000 7fb5e67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:32.727 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.689+0000 7fb5e67fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:07:32.727 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.689+0000 7fb5fcc88700 4 set_mon_vals no callback set 2021-05-25T12:07:32.727 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.689+0000 7fb5e67fc700 10 monclient: dump: 2021-05-25T12:07:32.727 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:32.727 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:32.728 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:32.728 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:32.728 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:32.728 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:32.728 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:32.729 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:32.729 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:32.729 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:32.729 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.690+0000 7fb5e67fc700 1 -- 172.21.15.102:0/3884068671 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7fb5e8004a20 con 0x7fb5f8091b00 2021-05-25T12:07:32.729 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.690+0000 7fb5e67fc700 1 -- 172.21.15.102:0/3884068671 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(243..243 src has 1..243) v4 ==== 5568+0+0 (unknown 453324089 0 0) 0x7fb5e803d560 con 0x7fb5f8091b00 2021-05-25T12:07:32.730 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.690+0000 7fb5e67fc700 10 client.234101.objecter ms_dispatch 0x7fb5f8001550 osd_map(243..243 src has 1..243) v4 2021-05-25T12:07:32.730 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.690+0000 7fb5e67fc700 3 client.234101.objecter handle_osd_map got epochs [243,243] > 0 2021-05-25T12:07:32.730 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.690+0000 7fb5e67fc700 3 client.234101.objecter handle_osd_map decoding full epoch 243 2021-05-25T12:07:32.730 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.690+0000 7fb5e67fc700 20 client.234101.objecter dump_active .. 0 homeless 2021-05-25T12:07:32.730 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.690+0000 7fb6006ef700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:07:32.731 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.690+0000 7fb6006ef700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:07:32.731 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.690+0000 7fb6006ef700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:07:32.731 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.690+0000 7fb6006ef700 1 -- 172.21.15.102:0/3884068671 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5dc004db0 con 0x7fb5f8091b00 2021-05-25T12:07:32.731 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.690+0000 7fb5fdc8a700 10 monclient: get_auth_request con 0x7fb5d80332a0 auth_method 0 2021-05-25T12:07:32.731 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.691+0000 7fb5e67fc700 10 client.234101.objecter ms_handle_connect 0x7fb5d80332a0 2021-05-25T12:07:32.732 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.694+0000 7fb5e67fc700 1 -- 172.21.15.102:0/3884068671 <== mon.2 v1:172.21.15.102:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7fb5e8012cd0 con 0x7fb5f8091b00 2021-05-25T12:07:32.732 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.694+0000 7fb5e67fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:07:32.732 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.694+0000 7fb5e67fc700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:07:32.912 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.908+0000 7fb6006ef700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2021-05-25T12:07:32.912 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.908+0000 7fb6006ef700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2021-05-25T12:07:32.912 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.909+0000 7fb6006ef700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:07:32.912 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.909+0000 7fb6006ef700 1 -- 172.21.15.102:0/3884068671 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fb5dc005060 con 0x7fb5f8091b00 2021-05-25T12:07:32.913 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.909+0000 7fb5e67fc700 1 -- 172.21.15.102:0/3884068671 <== mon.2 v1:172.21.15.102:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1342 (unknown 254746657 0 2361137307) 0x7fb5e8012aa0 con 0x7fb5f8091b00 2021-05-25T12:07:32.913 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.909+0000 7fb5e67fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2021-05-25T12:07:32.913 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.909+0000 7fb5e67fc700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:07:32.913 INFO:teuthology.orchestra.run.smithi102.stdout:{"election_epoch":64,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138303579357183","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"20ee91b6-8935-4b98-b868-85c03789ccc4","modified":"2021-05-25T11:59:11.011522Z","created":"2021-05-25T11:59:11.011522Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6790","nonce":0}]},"addr":"172.21.15.102:6790/0","public_addr":"172.21.15.102:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2021-05-25T12:07:32.915 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.910+0000 7fb6006ef700 10 librados: watch_flush enter 2021-05-25T12:07:32.915 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.910+0000 7fb6006ef700 10 librados: watch_flush exit 2021-05-25T12:07:32.915 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.910+0000 7fb6006ef700 20 client.234101.objecter shutdown clearing up homeless session... 2021-05-25T12:07:32.916 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.910+0000 7fb6006ef700 10 client.234101.objecter successfully canceled tick 2021-05-25T12:07:32.916 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.910+0000 7fb6006ef700 1 -- 172.21.15.102:0/3884068671 >> v1:172.21.15.181:6812/29582 conn(0x7fb5d80332a0 legacy=0x7fb5d8035760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:32.916 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.910+0000 7fb6006ef700 10 monclient: shutdown 2021-05-25T12:07:32.916 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.910+0000 7fb6006ef700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:07:32.916 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.910+0000 7fb6006ef700 1 -- 172.21.15.102:0/3884068671 >> v1:172.21.15.102:6790/0 conn(0x7fb5f8091b00 legacy=0x7fb5f8133490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:32.917 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.910+0000 7fb6006ef700 1 -- 172.21.15.102:0/3884068671 shutdown_connections 2021-05-25T12:07:32.917 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.910+0000 7fb6006ef700 1 -- 172.21.15.102:0/3884068671 >> 172.21.15.102:0/3884068671 conn(0x7fb5f809a910 msgr2=0x7fb5f8067820 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:07:32.917 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.910+0000 7fb6006ef700 1 -- 172.21.15.102:0/3884068671 shutdown_connections 2021-05-25T12:07:32.917 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.910+0000 7fb6006ef700 1 -- 172.21.15.102:0/3884068671 wait complete. 2021-05-25T12:07:32.917 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:32.911+0000 7fb6006ef700 1 librados: shutdown 2021-05-25T12:07:32.923 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2021-05-25T12:07:32.923 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2021-05-25T12:07:33.034 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.033+0000 7ffa29c0d700 10 monclient: get_monmap_and_config 2021-05-25T12:07:33.034 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.033+0000 7ffa29c0d700 10 monclient: build_initial_monmap 2021-05-25T12:07:33.035 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.033+0000 7ffa29c0d700 10 monclient: monmap: 2021-05-25T12:07:33.035 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:07:33.035 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:33.035 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:07:33.034134+0000 2021-05-25T12:07:33.036 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:07:33.034134+0000 2021-05-25T12:07:33.036 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:07:33.036 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:33.036 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:07:33.036 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:07:33.037 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:07:33.037 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:33.050 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.033+0000 7ffa29c0d700 1 Processor -- start 2021-05-25T12:07:33.051 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.033+0000 7ffa29c0d700 1 -- start start 2021-05-25T12:07:33.051 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.033+0000 7ffa29c0d700 10 monclient: init 2021-05-25T12:07:33.051 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.034+0000 7ffa29c0d700 10 monclient: _reopen_session rank -1 2021-05-25T12:07:33.051 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.034+0000 7ffa29c0d700 10 monclient: _add_conns ranks=[1,2,0] 2021-05-25T12:07:33.051 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.034+0000 7ffa29c0d700 10 monclient(hunting): picked mon.noname-c con 0x7ffa24098930 addr v1:172.21.15.181:6789/0 2021-05-25T12:07:33.052 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.034+0000 7ffa29c0d700 10 monclient(hunting): picked mon.noname-b con 0x7ffa2412bca0 addr v1:172.21.15.102:6790/0 2021-05-25T12:07:33.052 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.034+0000 7ffa29c0d700 10 monclient(hunting): picked mon.noname-a con 0x7ffa2412c150 addr v1:172.21.15.102:6789/0 2021-05-25T12:07:33.052 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.034+0000 7ffa29c0d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa24099b20 con 0x7ffa2412c150 2021-05-25T12:07:33.052 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.034+0000 7ffa29c0d700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa2409a3a0 con 0x7ffa24098930 2021-05-25T12:07:33.052 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.034+0000 7ffa29c0d700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa2409ac60 con 0x7ffa2412bca0 2021-05-25T12:07:33.053 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.034+0000 7ffa29c0d700 10 monclient(hunting): _renew_subs 2021-05-25T12:07:33.053 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.034+0000 7ffa29c0d700 10 monclient(hunting): authenticate will time out at 1253.631517s 2021-05-25T12:07:33.053 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.034+0000 7ffa2940c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7ffa2412c150 0x7ffa24133ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33782/0 (socket says 172.21.15.102:33782) 2021-05-25T12:07:33.053 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.034+0000 7ffa1bfff700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7ffa2412bca0 0x7ffa241308c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:56888/0 (socket says 172.21.15.102:56888) 2021-05-25T12:07:33.054 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.034+0000 7ffa2940c700 1 -- 172.21.15.102:0/1530045478 learned_addr learned my addr 172.21.15.102:0/1530045478 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:07:33.054 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.034+0000 7ffa1b7fe700 1 -- 172.21.15.102:0/1530045478 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3598047417 0 0) 0x7ffa24099b20 con 0x7ffa2412c150 2021-05-25T12:07:33.054 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.034+0000 7ffa1b7fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:33.054 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.034+0000 7ffa1b7fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:33.054 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.034+0000 7ffa1b7fe700 10 monclient(hunting): my global_id is 204369 2021-05-25T12:07:33.055 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.034+0000 7ffa1b7fe700 1 -- 172.21.15.102:0/1530045478 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa10003270 con 0x7ffa2412c150 2021-05-25T12:07:33.055 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.034+0000 7ffa1b7fe700 1 -- 172.21.15.102:0/1530045478 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4236064030 0 0) 0x7ffa2409ac60 con 0x7ffa2412bca0 2021-05-25T12:07:33.055 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.034+0000 7ffa1b7fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:33.055 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.034+0000 7ffa1b7fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:33.055 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.034+0000 7ffa1b7fe700 10 monclient(hunting): my global_id is 234104 2021-05-25T12:07:33.056 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.034+0000 7ffa1b7fe700 1 -- 172.21.15.102:0/1530045478 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa24099b20 con 0x7ffa2412bca0 2021-05-25T12:07:33.056 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.035+0000 7ffa1b7fe700 1 -- 172.21.15.102:0/1530045478 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 94495685 0 0) 0x7ffa2409a3a0 con 0x7ffa24098930 2021-05-25T12:07:33.056 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.035+0000 7ffa1b7fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:33.056 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.035+0000 7ffa1b7fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:33.056 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.035+0000 7ffa1b7fe700 10 monclient(hunting): my global_id is 224236 2021-05-25T12:07:33.057 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.035+0000 7ffa1b7fe700 1 -- 172.21.15.102:0/1530045478 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa2409ac60 con 0x7ffa24098930 2021-05-25T12:07:33.057 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.035+0000 7ffa1b7fe700 1 -- 172.21.15.102:0/1530045478 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1126493813 0 0) 0x7ffa10003270 con 0x7ffa2412c150 2021-05-25T12:07:33.057 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.035+0000 7ffa1b7fe700 1 -- 172.21.15.102:0/1530045478 >> v1:172.21.15.102:6790/0 conn(0x7ffa2412bca0 legacy=0x7ffa241308c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:33.057 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.035+0000 7ffa1b7fe700 1 -- 172.21.15.102:0/1530045478 >> v1:172.21.15.181:6789/0 conn(0x7ffa24098930 legacy=0x7ffa2412b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:33.057 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.035+0000 7ffa1b7fe700 10 monclient: _finish_hunting 0 2021-05-25T12:07:33.058 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.035+0000 7ffa1b7fe700 1 monclient: found mon.noname-a 2021-05-25T12:07:33.058 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.035+0000 7ffa1b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:07:33.058 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.035+0000 7ffa1b7fe700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T12:07:33.058 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.035+0000 7ffa1b7fe700 1 -- 172.21.15.102:0/1530045478 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa2409af00 con 0x7ffa2412c150 2021-05-25T12:07:33.059 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.035+0000 7ffa1b7fe700 10 monclient: _finish_auth 0 2021-05-25T12:07:33.059 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.035+0000 7ffa1b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:07:33.059 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.035+0000 7ffa1b7fe700 1 -- 172.21.15.102:0/1530045478 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7ffa20001990 con 0x7ffa2412c150 2021-05-25T12:07:33.059 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.035+0000 7ffa29c0d700 5 monclient: authenticate success, global_id 204369 2021-05-25T12:07:33.059 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.035+0000 7ffa29c0d700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:07:33.060 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.035+0000 7ffa1b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:33.060 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.035+0000 7ffa1b7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:07:33.060 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.035+0000 7ffa1b7fe700 10 monclient: dump: 2021-05-25T12:07:33.060 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:33.060 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:33.061 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:33.061 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:33.061 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:33.061 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:33.061 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:33.062 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:33.062 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:33.062 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:33.062 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.035+0000 7ffa29c0d700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:07:33.062 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.035+0000 7ffa1b7fe700 1 -- 172.21.15.102:0/1530045478 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffa20001b80 con 0x7ffa2412c150 2021-05-25T12:07:33.063 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.035+0000 7ffa1b7fe700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:07:33.063 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.035+0000 7ffa29c0d700 10 monclient: get_monmap_and_config success 2021-05-25T12:07:33.063 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.035+0000 7ffa1b7fe700 1 -- 172.21.15.102:0/1530045478 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7ffa20001f10 con 0x7ffa2412c150 2021-05-25T12:07:33.063 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.035+0000 7ffa29c0d700 4 set_mon_vals no callback set 2021-05-25T12:07:33.064 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.035+0000 7ffa1b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:33.064 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.035+0000 7ffa1b7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:07:33.064 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.035+0000 7ffa1b7fe700 10 monclient: dump: 2021-05-25T12:07:33.064 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:33.064 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:33.065 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:33.065 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:33.065 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:33.065 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:33.065 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:33.066 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:33.066 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:33.066 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:33.066 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.035+0000 7ffa29c0d700 10 monclient: shutdown 2021-05-25T12:07:33.067 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.035+0000 7ffa29c0d700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:07:33.067 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.035+0000 7ffa29c0d700 1 -- 172.21.15.102:0/1530045478 >> v1:172.21.15.102:6789/0 conn(0x7ffa2412c150 legacy=0x7ffa24133ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:33.067 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.035+0000 7ffa29c0d700 1 -- 172.21.15.102:0/1530045478 shutdown_connections 2021-05-25T12:07:33.067 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.035+0000 7ffa29c0d700 1 -- 172.21.15.102:0/1530045478 >> 172.21.15.102:0/1530045478 conn(0x7ffa2409e9d0 msgr2=0x7ffa2409ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:07:33.067 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.035+0000 7ffa29c0d700 1 -- 172.21.15.102:0/1530045478 shutdown_connections 2021-05-25T12:07:33.068 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.035+0000 7ffa29c0d700 1 -- 172.21.15.102:0/1530045478 wait complete. 2021-05-25T12:07:33.068 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.036+0000 7ffa29c0d700 10 monclient: build_initial_monmap 2021-05-25T12:07:33.068 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.036+0000 7ffa29c0d700 10 monclient: monmap: 2021-05-25T12:07:33.068 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:07:33.069 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:07:33.069 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:07:33.069 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:07:33.069 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:07:33.069 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:07:33.070 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:07:33.070 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:07:33.070 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:07:33.070 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:33.071 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.036+0000 7ffa29c0d700 1 librados: starting msgr at 2021-05-25T12:07:33.071 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.036+0000 7ffa29c0d700 1 librados: starting objecter 2021-05-25T12:07:33.071 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.036+0000 7ffa29c0d700 1 Processor -- start 2021-05-25T12:07:33.071 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.036+0000 7ffa29c0d700 1 -- start start 2021-05-25T12:07:33.071 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.036+0000 7ffa29c0d700 1 librados: setting wanted keys 2021-05-25T12:07:33.072 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.036+0000 7ffa29c0d700 1 librados: calling monclient init 2021-05-25T12:07:33.072 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.036+0000 7ffa29c0d700 10 monclient: init 2021-05-25T12:07:33.072 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.037+0000 7ffa29c0d700 10 monclient: _reopen_session rank -1 2021-05-25T12:07:33.072 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.037+0000 7ffa29c0d700 10 monclient: _add_conns ranks=[1,2,0] 2021-05-25T12:07:33.073 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.037+0000 7ffa29c0d700 10 monclient(hunting): picked mon.noname-b con 0x7ffa24098930 addr v1:172.21.15.181:6789/0 2021-05-25T12:07:33.073 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.037+0000 7ffa29c0d700 10 monclient(hunting): picked mon.noname-c con 0x7ffa2412bca0 addr v1:172.21.15.102:6790/0 2021-05-25T12:07:33.073 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.037+0000 7ffa29c0d700 10 monclient(hunting): picked mon.noname-a con 0x7ffa2412c150 addr v1:172.21.15.102:6789/0 2021-05-25T12:07:33.073 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.037+0000 7ffa29c0d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa24094460 con 0x7ffa2412c150 2021-05-25T12:07:33.073 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.037+0000 7ffa29c0d700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa24092b60 con 0x7ffa24098930 2021-05-25T12:07:33.074 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.037+0000 7ffa29c0d700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa240946b0 con 0x7ffa2412bca0 2021-05-25T12:07:33.074 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.037+0000 7ffa29c0d700 10 monclient(hunting): _renew_subs 2021-05-25T12:07:33.074 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.037+0000 7ffa29c0d700 10 monclient(hunting): authenticate will time out at 1253.634147s 2021-05-25T12:07:33.074 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.037+0000 7ffa2940c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7ffa2412c150 0x7ffa24142cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33788/0 (socket says 172.21.15.102:33788) 2021-05-25T12:07:33.074 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.037+0000 7ffa2940c700 1 -- 172.21.15.102:0/2336475889 learned_addr learned my addr 172.21.15.102:0/2336475889 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:07:33.075 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.037+0000 7ffa18ff9700 10 client.?.objecter ms_handle_connect 0x7ffa2412bca0 2021-05-25T12:07:33.075 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.037+0000 7ffa18ff9700 10 client.?.objecter resend_mon_ops 2021-05-25T12:07:33.075 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.037+0000 7ffa18ff9700 10 client.?.objecter ms_handle_connect 0x7ffa2412c150 2021-05-25T12:07:33.075 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.037+0000 7ffa18ff9700 10 client.?.objecter resend_mon_ops 2021-05-25T12:07:33.075 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.037+0000 7ffa18ff9700 10 client.?.objecter ms_handle_connect 0x7ffa24098930 2021-05-25T12:07:33.076 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.037+0000 7ffa18ff9700 10 client.?.objecter resend_mon_ops 2021-05-25T12:07:33.076 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.037+0000 7ffa18ff9700 1 -- 172.21.15.102:0/2336475889 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 488038328 0 0) 0x7ffa240946b0 con 0x7ffa2412bca0 2021-05-25T12:07:33.076 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.037+0000 7ffa18ff9700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:33.076 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.037+0000 7ffa18ff9700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:33.076 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.037+0000 7ffa18ff9700 10 monclient(hunting): my global_id is 234107 2021-05-25T12:07:33.077 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.037+0000 7ffa18ff9700 1 -- 172.21.15.102:0/2336475889 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa04004c10 con 0x7ffa2412bca0 2021-05-25T12:07:33.077 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.037+0000 7ffa18ff9700 1 -- 172.21.15.102:0/2336475889 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3796411769 0 0) 0x7ffa24094460 con 0x7ffa2412c150 2021-05-25T12:07:33.077 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.037+0000 7ffa18ff9700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:33.077 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.037+0000 7ffa18ff9700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:33.078 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.037+0000 7ffa18ff9700 10 monclient(hunting): my global_id is 204372 2021-05-25T12:07:33.078 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.037+0000 7ffa18ff9700 1 -- 172.21.15.102:0/2336475889 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa240946b0 con 0x7ffa2412c150 2021-05-25T12:07:33.078 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.037+0000 7ffa18ff9700 1 -- 172.21.15.102:0/2336475889 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1341364405 0 0) 0x7ffa2409ac60 con 0x7ffa24098930 2021-05-25T12:07:33.078 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.037+0000 7ffa18ff9700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:33.078 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.037+0000 7ffa18ff9700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:33.079 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.037+0000 7ffa18ff9700 10 monclient(hunting): my global_id is 224239 2021-05-25T12:07:33.079 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.037+0000 7ffa18ff9700 1 -- 172.21.15.102:0/2336475889 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa24094460 con 0x7ffa24098930 2021-05-25T12:07:33.079 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.037+0000 7ffa18ff9700 1 -- 172.21.15.102:0/2336475889 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2649973225 0 0) 0x7ffa04004c10 con 0x7ffa2412bca0 2021-05-25T12:07:33.079 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.037+0000 7ffa18ff9700 1 -- 172.21.15.102:0/2336475889 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffa2409ac60 con 0x7ffa2412bca0 2021-05-25T12:07:33.080 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.037+0000 7ffa18ff9700 1 -- 172.21.15.102:0/2336475889 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7ffa0c0019d0 con 0x7ffa2412bca0 2021-05-25T12:07:33.080 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.037+0000 7ffa18ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:33.080 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.037+0000 7ffa18ff9700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:07:33.080 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.037+0000 7ffa18ff9700 10 monclient(hunting): dump: 2021-05-25T12:07:33.080 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:33.081 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:33.081 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:33.081 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:33.081 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:33.081 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:33.082 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:33.082 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:33.082 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:33.082 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:33.082 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.037+0000 7ffa18ff9700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:07:33.083 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.037+0000 7ffa18ff9700 1 -- 172.21.15.102:0/2336475889 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3615632532 0 0) 0x7ffa240946b0 con 0x7ffa2412c150 2021-05-25T12:07:33.083 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.037+0000 7ffa18ff9700 1 -- 172.21.15.102:0/2336475889 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffa04004c10 con 0x7ffa2412c150 2021-05-25T12:07:33.083 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.037+0000 7ffa18ff9700 1 -- 172.21.15.102:0/2336475889 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7ffa20001890 con 0x7ffa2412c150 2021-05-25T12:07:33.083 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.037+0000 7ffa18ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:33.084 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.037+0000 7ffa18ff9700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2021-05-25T12:07:33.084 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.037+0000 7ffa18ff9700 10 monclient(hunting): dump: 2021-05-25T12:07:33.084 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:33.084 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:33.084 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:33.085 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:33.085 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:33.085 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:33.085 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:33.085 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:33.086 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:33.086 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:33.086 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.037+0000 7ffa18ff9700 1 -- 172.21.15.102:0/2336475889 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2878815821 0 0) 0x7ffa2409ac60 con 0x7ffa2412bca0 2021-05-25T12:07:33.086 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.037+0000 7ffa18ff9700 1 -- 172.21.15.102:0/2336475889 >> v1:172.21.15.181:6789/0 conn(0x7ffa24098930 legacy=0x7ffa24097090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:33.086 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.037+0000 7ffa18ff9700 1 -- 172.21.15.102:0/2336475889 >> v1:172.21.15.102:6789/0 conn(0x7ffa2412c150 legacy=0x7ffa24142cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:33.087 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.037+0000 7ffa18ff9700 10 monclient: _finish_hunting 0 2021-05-25T12:07:33.087 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.037+0000 7ffa18ff9700 1 monclient: found mon.c 2021-05-25T12:07:33.087 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.037+0000 7ffa18ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:07:33.087 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.037+0000 7ffa18ff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:07:33.088 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.037+0000 7ffa18ff9700 1 -- 172.21.15.102:0/2336475889 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa24094970 con 0x7ffa2412bca0 2021-05-25T12:07:33.088 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.037+0000 7ffa18ff9700 10 monclient: _finish_auth 0 2021-05-25T12:07:33.088 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.037+0000 7ffa18ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:07:33.088 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.037+0000 7ffa29c0d700 5 monclient: authenticate success, global_id 234107 2021-05-25T12:07:33.088 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.037+0000 7ffa29c0d700 10 monclient: _renew_subs 2021-05-25T12:07:33.089 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.037+0000 7ffa29c0d700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:07:33.089 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.037+0000 7ffa29c0d700 1 -- 172.21.15.102:0/2336475889 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa240977a0 con 0x7ffa2412bca0 2021-05-25T12:07:33.089 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.037+0000 7ffa29c0d700 10 client.234107.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:07:33.089 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.037+0000 7ffa29c0d700 10 monclient: _renew_subs 2021-05-25T12:07:33.089 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.037+0000 7ffa29c0d700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:07:33.090 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.037+0000 7ffa29c0d700 1 -- 172.21.15.102:0/2336475889 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa2413b2c0 con 0x7ffa2412bca0 2021-05-25T12:07:33.090 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.037+0000 7ffa29c0d700 1 librados: init done 2021-05-25T12:07:33.090 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.038+0000 7ffa18ff9700 1 -- 172.21.15.102:0/2336475889 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffa0c001ca0 con 0x7ffa2412bca0 2021-05-25T12:07:33.090 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.038+0000 7ffa18ff9700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:07:33.091 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.038+0000 7ffa18ff9700 1 -- 172.21.15.102:0/2336475889 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7ffa0c004980 con 0x7ffa2412bca0 2021-05-25T12:07:33.091 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.038+0000 7ffa18ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:33.091 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.038+0000 7ffa18ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:07:33.091 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.038+0000 7ffa1affd700 4 set_mon_vals no callback set 2021-05-25T12:07:33.091 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.038+0000 7ffa18ff9700 10 monclient: dump: 2021-05-25T12:07:33.092 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:33.092 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:33.092 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:33.092 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:33.092 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:33.093 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:33.093 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:33.093 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:33.093 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:33.093 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:33.094 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.038+0000 7ffa29c0d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:07:33.094 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.038+0000 7ffa18ff9700 1 -- 172.21.15.102:0/2336475889 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7ffa0c00dca0 con 0x7ffa2412bca0 2021-05-25T12:07:33.094 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.038+0000 7ffa29c0d700 1 -- 172.21.15.102:0/2336475889 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff9fc0065a0 con 0x7ff9fc002840 2021-05-25T12:07:33.094 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.038+0000 7ffa18ff9700 1 -- 172.21.15.102:0/2336475889 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(243..243 src has 1..243) v4 ==== 5568+0+0 (unknown 453324089 0 0) 0x7ffa0c03d540 con 0x7ffa2412bca0 2021-05-25T12:07:33.094 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.038+0000 7ffa18ff9700 10 client.234107.objecter ms_dispatch 0x7ffa24001550 osd_map(243..243 src has 1..243) v4 2021-05-25T12:07:33.095 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.038+0000 7ffa18ff9700 3 client.234107.objecter handle_osd_map got epochs [243,243] > 0 2021-05-25T12:07:33.095 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.038+0000 7ffa18ff9700 3 client.234107.objecter handle_osd_map decoding full epoch 243 2021-05-25T12:07:33.095 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.039+0000 7ffa18ff9700 20 client.234107.objecter dump_active .. 0 homeless 2021-05-25T12:07:33.095 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.039+0000 7ffa18ff9700 10 client.234107.objecter ms_handle_connect 0x7ff9fc002840 2021-05-25T12:07:33.096 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.039+0000 7ffa18ff9700 10 client.234107.objecter resend_mon_ops 2021-05-25T12:07:33.096 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.039+0000 7ffa18ff9700 1 -- 172.21.15.102:0/2336475889 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 255453737 0 0) 0x7ff9fc0065a0 con 0x7ff9fc002840 2021-05-25T12:07:33.096 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.039+0000 7ffa18ff9700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:33.096 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.039+0000 7ffa18ff9700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:33.096 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.039+0000 7ffa18ff9700 10 monclient(hunting): my global_id is 204375 2021-05-25T12:07:33.097 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.039+0000 7ffa18ff9700 1 -- 172.21.15.102:0/2336475889 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa2409ac60 con 0x7ff9fc002840 2021-05-25T12:07:33.097 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.039+0000 7ffa2940c700 10 monclient: get_auth_request con 0x7ffa04033410 auth_method 0 2021-05-25T12:07:33.097 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.039+0000 7ffa18ff9700 1 -- 172.21.15.102:0/2336475889 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 999633409 0 0) 0x7ffa2409ac60 con 0x7ff9fc002840 2021-05-25T12:07:33.097 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.039+0000 7ffa18ff9700 1 -- 172.21.15.102:0/2336475889 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff9fc0065a0 con 0x7ff9fc002840 2021-05-25T12:07:33.097 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.039+0000 7ffa18ff9700 1 -- 172.21.15.102:0/2336475889 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7ffa14001600 con 0x7ff9fc002840 2021-05-25T12:07:33.098 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.039+0000 7ffa18ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:33.098 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.039+0000 7ffa18ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:07:33.098 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.039+0000 7ffa18ff9700 10 monclient: dump: 2021-05-25T12:07:33.098 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:33.099 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:33.099 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:33.099 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:33.099 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:33.099 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:33.100 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:33.100 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:33.100 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:33.100 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:33.100 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.039+0000 7ffa18ff9700 10 client.234107.objecter ms_handle_connect 0x7ffa04033410 2021-05-25T12:07:33.101 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.039+0000 7ffa18ff9700 1 -- 172.21.15.102:0/2336475889 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 411270223 0 0) 0x7ff9fc0065a0 con 0x7ff9fc002840 2021-05-25T12:07:33.101 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.039+0000 7ffa18ff9700 1 -- 172.21.15.102:0/2336475889 --> v1:172.21.15.102:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff9fc009e40 con 0x7ff9fc002840 2021-05-25T12:07:33.101 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.039+0000 7ffa18ff9700 1 -- 172.21.15.102:0/2336475889 <== mon.0 v1:172.21.15.102:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7ffa14002cc0 con 0x7ff9fc002840 2021-05-25T12:07:33.101 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.039+0000 7ffa18ff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:07:33.101 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.039+0000 7ffa18ff9700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:07:33.102 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.039+0000 7ffa18ff9700 1 -- 172.21.15.102:0/2336475889 >> v1:172.21.15.102:6789/0 conn(0x7ff9fc002840 legacy=0x7ff9fc004ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:33.102 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.039+0000 7ffa18ff9700 1 -- 172.21.15.102:0/2336475889 >> v1:172.21.15.102:6789/0 conn(0x7ff9fc002840 legacy=0x7ff9fc004ce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:07:33.102 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.046+0000 7ffa29c0d700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:07:33.102 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.046+0000 7ffa29c0d700 1 -- 172.21.15.102:0/2336475889 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff9fc009ad0 con 0x7ff9fc008d90 2021-05-25T12:07:33.102 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.046+0000 7ffa18ff9700 10 client.234107.objecter ms_handle_connect 0x7ff9fc008d90 2021-05-25T12:07:33.103 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.046+0000 7ffa18ff9700 10 client.234107.objecter resend_mon_ops 2021-05-25T12:07:33.103 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.046+0000 7ffa18ff9700 1 -- 172.21.15.102:0/2336475889 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3500576907 0 0) 0x7ff9fc009ad0 con 0x7ff9fc008d90 2021-05-25T12:07:33.103 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.046+0000 7ffa18ff9700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:33.103 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.046+0000 7ffa18ff9700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:33.103 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.046+0000 7ffa18ff9700 10 monclient(hunting): my global_id is 204378 2021-05-25T12:07:33.104 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.046+0000 7ffa18ff9700 1 -- 172.21.15.102:0/2336475889 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9fc0065a0 con 0x7ff9fc008d90 2021-05-25T12:07:33.104 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.046+0000 7ffa18ff9700 1 -- 172.21.15.102:0/2336475889 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1855569970 0 0) 0x7ff9fc0065a0 con 0x7ff9fc008d90 2021-05-25T12:07:33.104 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.046+0000 7ffa18ff9700 1 -- 172.21.15.102:0/2336475889 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff9fc009ad0 con 0x7ff9fc008d90 2021-05-25T12:07:33.104 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.046+0000 7ffa18ff9700 1 -- 172.21.15.102:0/2336475889 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7ffa14001d20 con 0x7ff9fc008d90 2021-05-25T12:07:33.104 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.046+0000 7ffa18ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:33.105 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.046+0000 7ffa18ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:07:33.105 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.046+0000 7ffa18ff9700 10 monclient: dump: 2021-05-25T12:07:33.105 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:33.105 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:33.105 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:33.106 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:33.106 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:33.106 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:33.106 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:33.107 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:33.107 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:33.107 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:33.107 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.046+0000 7ffa18ff9700 1 -- 172.21.15.102:0/2336475889 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2364646122 0 0) 0x7ff9fc009ad0 con 0x7ff9fc008d90 2021-05-25T12:07:33.107 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.046+0000 7ffa18ff9700 1 -- 172.21.15.102:0/2336475889 --> v1:172.21.15.102:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ff9fc008740 con 0x7ff9fc008d90 2021-05-25T12:07:33.108 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.046+0000 7ffa18ff9700 1 -- 172.21.15.102:0/2336475889 <== mon.0 v1:172.21.15.102:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3940 (unknown 0 0 4205694745) 0x7ffa14001fc0 con 0x7ff9fc008d90 2021-05-25T12:07:33.108 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.046+0000 7ffa18ff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:07:33.108 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.046+0000 7ffa18ff9700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:07:33.108 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.046+0000 7ffa18ff9700 1 -- 172.21.15.102:0/2336475889 >> v1:172.21.15.102:6789/0 conn(0x7ff9fc008d90 legacy=0x7ff9fc009220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:33.108 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.046+0000 7ffa18ff9700 1 -- 172.21.15.102:0/2336475889 >> v1:172.21.15.102:6789/0 conn(0x7ff9fc008d90 legacy=0x7ff9fc009220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:07:33.109 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:07:33.109 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:07:33.109 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:07:33.110 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "peon", 2021-05-25T12:07:33.110 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 64, 2021-05-25T12:07:33.110 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:07:33.110 INFO:teuthology.orchestra.run.smithi102.stdout: 0, 2021-05-25T12:07:33.111 INFO:teuthology.orchestra.run.smithi102.stdout: 1, 2021-05-25T12:07:33.111 INFO:teuthology.orchestra.run.smithi102.stdout: 2 2021-05-25T12:07:33.111 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:07:33.111 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 0, 2021-05-25T12:07:33.111 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:07:33.112 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:07:33.112 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:07:33.112 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:07:33.112 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:07:33.112 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:07:33.113 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:07:33.113 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:07:33.113 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:07:33.113 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:07:33.114 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:07:33.114 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:07:33.114 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:07:33.114 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:07:33.114 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:07:33.115 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:07:33.115 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:07:33.115 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:07:33.115 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:07:33.115 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:07:33.116 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:07:33.116 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:07:33.116 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:07:33.116 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:07:33.116 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:07:33.117 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:33.117 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:07:33.117 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:07:33.117 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:07:33.117 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:07:33.118 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:07:33.118 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:07:33.118 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:07:33.118 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:07:33.118 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:07:33.119 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:07:33.119 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:07:33.119 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:07:33.119 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:07:33.119 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:07:33.120 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:07:33.120 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:07:33.120 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:07:33.120 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:07:33.120 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:07:33.121 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:07:33.121 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:07:33.121 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:07:33.121 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:07:33.121 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:07:33.122 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:07:33.122 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:07:33.122 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:33.122 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:07:33.122 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:33.123 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:07:33.123 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:07:33.123 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:07:33.123 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:07:33.123 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:33.124 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:07:33.124 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:07:33.124 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:07:33.124 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:07:33.125 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:07:33.125 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:33.125 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:07:33.125 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:07:33.125 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:07:33.126 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:07:33.126 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:07:33.126 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:33.126 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:33.126 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:07:33.127 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:07:33.127 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:07:33.127 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:07:33.127 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:33.127 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:07:33.128 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:07:33.128 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:07:33.128 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:07:33.128 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:07:33.128 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:33.129 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:07:33.129 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:07:33.129 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:07:33.129 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:07:33.129 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:07:33.130 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:33.130 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:33.130 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:07:33.130 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:07:33.130 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:07:33.131 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:07:33.131 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:33.131 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:07:33.131 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:07:33.132 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:07:33.132 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:07:33.132 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:07:33.132 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:33.132 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:07:33.132 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:07:33.133 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:07:33.133 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:07:33.133 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:07:33.133 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:07:33.134 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:07:33.134 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:33.134 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:07:33.134 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:07:33.134 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:33.135 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:07:33.135 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:07:33.135 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:07:33.135 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:07:33.135 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:07:33.136 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": [ 2021-05-25T12:07:33.136 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:33.136 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:07:33.136 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:07:33.136 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 6 2021-05-25T12:07:33.137 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:07:33.137 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:07:33.137 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:07:33.137 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:33.137 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:07:33.138 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:07:33.138 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 3 2021-05-25T12:07:33.138 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:07:33.138 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:07:33.138 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:33.139 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:07:33.139 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:07:33.141 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.047+0000 7ffa29c0d700 10 librados: watch_flush enter 2021-05-25T12:07:33.142 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.047+0000 7ffa29c0d700 10 librados: watch_flush exit 2021-05-25T12:07:33.142 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.047+0000 7ffa29c0d700 20 client.234107.objecter shutdown clearing up homeless session... 2021-05-25T12:07:33.142 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.047+0000 7ffa29c0d700 10 client.234107.objecter successfully canceled tick 2021-05-25T12:07:33.142 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.047+0000 7ffa29c0d700 1 -- 172.21.15.102:0/2336475889 >> v1:172.21.15.181:6812/29582 conn(0x7ffa04033410 legacy=0x7ffa040358d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:33.142 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.047+0000 7ffa29c0d700 10 monclient: shutdown 2021-05-25T12:07:33.143 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.047+0000 7ffa29c0d700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:07:33.143 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.047+0000 7ffa29c0d700 1 -- 172.21.15.102:0/2336475889 >> v1:172.21.15.102:6790/0 conn(0x7ffa2412bca0 legacy=0x7ffa2413a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:33.143 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.047+0000 7ffa2940c700 1 -- 172.21.15.102:0/2336475889 reap_dead start 2021-05-25T12:07:33.143 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.047+0000 7ffa2940c700 1 -- 172.21.15.102:0/2336475889 reap_dead start 2021-05-25T12:07:33.144 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.047+0000 7ffa29c0d700 1 -- 172.21.15.102:0/2336475889 shutdown_connections 2021-05-25T12:07:33.144 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.047+0000 7ffa29c0d700 1 -- 172.21.15.102:0/2336475889 >> 172.21.15.102:0/2336475889 conn(0x7ffa2409e9d0 msgr2=0x7ffa2409c800 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:07:33.144 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.047+0000 7ffa29c0d700 1 -- 172.21.15.102:0/2336475889 shutdown_connections 2021-05-25T12:07:33.144 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.047+0000 7ffa29c0d700 1 -- 172.21.15.102:0/2336475889 wait complete. 2021-05-25T12:07:33.144 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.047+0000 7ffa29c0d700 1 librados: shutdown 2021-05-25T12:07:33.145 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2021-05-25T12:07:33.255 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.254+0000 7fbfc709d700 10 monclient: get_monmap_and_config 2021-05-25T12:07:33.256 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.254+0000 7fbfc709d700 10 monclient: build_initial_monmap 2021-05-25T12:07:33.256 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.254+0000 7fbfc709d700 10 monclient: monmap: 2021-05-25T12:07:33.256 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:07:33.256 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:33.257 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:07:33.255165+0000 2021-05-25T12:07:33.257 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:07:33.255165+0000 2021-05-25T12:07:33.257 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:07:33.257 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:33.257 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:07:33.258 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:07:33.258 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:07:33.258 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:33.271 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.254+0000 7fbfc709d700 1 Processor -- start 2021-05-25T12:07:33.271 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.254+0000 7fbfc709d700 1 -- start start 2021-05-25T12:07:33.271 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.254+0000 7fbfc709d700 10 monclient: init 2021-05-25T12:07:33.272 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.255+0000 7fbfc709d700 10 monclient: _reopen_session rank -1 2021-05-25T12:07:33.272 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.255+0000 7fbfc709d700 10 monclient: _add_conns ranks=[2,0,1] 2021-05-25T12:07:33.272 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.255+0000 7fbfc709d700 10 monclient(hunting): picked mon.noname-b con 0x7fbfc0090990 addr v1:172.21.15.102:6790/0 2021-05-25T12:07:33.272 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.255+0000 7fbfc709d700 10 monclient(hunting): picked mon.noname-a con 0x7fbfc0090d70 addr v1:172.21.15.102:6789/0 2021-05-25T12:07:33.273 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.255+0000 7fbfc709d700 10 monclient(hunting): picked mon.noname-c con 0x7fbfc012e330 addr v1:172.21.15.181:6789/0 2021-05-25T12:07:33.273 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.255+0000 7fbfc709d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfc0096fa0 con 0x7fbfc0090d70 2021-05-25T12:07:33.273 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.255+0000 7fbfc709d700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfc00926a0 con 0x7fbfc012e330 2021-05-25T12:07:33.273 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.255+0000 7fbfc709d700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfc0092940 con 0x7fbfc0090990 2021-05-25T12:07:33.273 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.255+0000 7fbfc709d700 10 monclient(hunting): _renew_subs 2021-05-25T12:07:33.274 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.255+0000 7fbfc709d700 10 monclient(hunting): authenticate will time out at 1253.852511s 2021-05-25T12:07:33.274 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.255+0000 7fbfb7fff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fbfc0090d70 0x7fbfc0091200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33798/0 (socket says 172.21.15.102:33798) 2021-05-25T12:07:33.274 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.255+0000 7fbfb7fff700 1 -- 172.21.15.102:0/2310516105 learned_addr learned my addr 172.21.15.102:0/2310516105 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:07:33.274 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.255+0000 7fbfc4e39700 1 --1- 172.21.15.102:0/2310516105 >> v1:172.21.15.102:6790/0 conn(0x7fbfc0090990 0x7fbfc0093240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:56906/0 (socket says 172.21.15.102:56906) 2021-05-25T12:07:33.274 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.255+0000 7fbfb77fe700 1 -- 172.21.15.102:0/2310516105 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2045702918 0 0) 0x7fbfc0092940 con 0x7fbfc0090990 2021-05-25T12:07:33.275 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.255+0000 7fbfb77fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:33.275 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.255+0000 7fbfb77fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:33.275 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.255+0000 7fbfb77fe700 10 monclient(hunting): my global_id is 234110 2021-05-25T12:07:33.275 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.255+0000 7fbfb77fe700 1 -- 172.21.15.102:0/2310516105 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfac003270 con 0x7fbfc0090990 2021-05-25T12:07:33.275 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.255+0000 7fbfb77fe700 1 -- 172.21.15.102:0/2310516105 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2679056663 0 0) 0x7fbfc0096fa0 con 0x7fbfc0090d70 2021-05-25T12:07:33.276 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.255+0000 7fbfb77fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:33.276 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.255+0000 7fbfb77fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:33.276 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.255+0000 7fbfb77fe700 10 monclient(hunting): my global_id is 204381 2021-05-25T12:07:33.276 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.255+0000 7fbfb77fe700 1 -- 172.21.15.102:0/2310516105 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfc0092940 con 0x7fbfc0090d70 2021-05-25T12:07:33.277 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.256+0000 7fbfb77fe700 1 -- 172.21.15.102:0/2310516105 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 347577805 0 0) 0x7fbfc00926a0 con 0x7fbfc012e330 2021-05-25T12:07:33.277 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.256+0000 7fbfb77fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:33.277 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.256+0000 7fbfb77fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:33.277 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.256+0000 7fbfb77fe700 10 monclient(hunting): my global_id is 224242 2021-05-25T12:07:33.277 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.256+0000 7fbfb77fe700 1 -- 172.21.15.102:0/2310516105 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfc0096fa0 con 0x7fbfc012e330 2021-05-25T12:07:33.278 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.256+0000 7fbfb77fe700 1 -- 172.21.15.102:0/2310516105 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3047774304 0 0) 0x7fbfc0092940 con 0x7fbfc0090d70 2021-05-25T12:07:33.278 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.256+0000 7fbfb77fe700 1 -- 172.21.15.102:0/2310516105 >> v1:172.21.15.102:6790/0 conn(0x7fbfc0090990 legacy=0x7fbfc0093240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:33.278 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.256+0000 7fbfb77fe700 1 -- 172.21.15.102:0/2310516105 >> v1:172.21.15.181:6789/0 conn(0x7fbfc012e330 legacy=0x7fbfc0130720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:33.278 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.256+0000 7fbfb77fe700 10 monclient: _finish_hunting 0 2021-05-25T12:07:33.278 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.256+0000 7fbfb77fe700 1 monclient: found mon.noname-a 2021-05-25T12:07:33.279 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.256+0000 7fbfb77fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:07:33.279 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.256+0000 7fbfb77fe700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T12:07:33.279 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.256+0000 7fbfb77fe700 1 -- 172.21.15.102:0/2310516105 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfc0092be0 con 0x7fbfc0090d70 2021-05-25T12:07:33.279 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.256+0000 7fbfb77fe700 10 monclient: _finish_auth 0 2021-05-25T12:07:33.280 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.256+0000 7fbfb77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:07:33.280 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.256+0000 7fbfc709d700 5 monclient: authenticate success, global_id 204381 2021-05-25T12:07:33.280 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.256+0000 7fbfb77fe700 1 -- 172.21.15.102:0/2310516105 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fbfa8001540 con 0x7fbfc0090d70 2021-05-25T12:07:33.280 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.256+0000 7fbfb77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:33.280 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.256+0000 7fbfb77fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:07:33.281 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.256+0000 7fbfb77fe700 10 monclient: dump: 2021-05-25T12:07:33.281 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:33.281 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:33.281 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:33.281 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:33.282 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:33.282 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:33.282 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:33.282 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:33.283 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:33.283 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:33.283 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.256+0000 7fbfc709d700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:07:33.283 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.256+0000 7fbfb77fe700 1 -- 172.21.15.102:0/2310516105 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbfa8001760 con 0x7fbfc0090d70 2021-05-25T12:07:33.283 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.256+0000 7fbfb77fe700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:07:33.284 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.256+0000 7fbfb77fe700 1 -- 172.21.15.102:0/2310516105 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fbfa8001a70 con 0x7fbfc0090d70 2021-05-25T12:07:33.284 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.256+0000 7fbfc709d700 10 monclient: get_monmap_and_config success 2021-05-25T12:07:33.284 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.256+0000 7fbfc709d700 4 set_mon_vals no callback set 2021-05-25T12:07:33.284 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.256+0000 7fbfb77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:33.284 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.256+0000 7fbfb77fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:07:33.285 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.256+0000 7fbfb77fe700 10 monclient: dump: 2021-05-25T12:07:33.285 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:33.285 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:33.285 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:33.285 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:33.286 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:33.286 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:33.286 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:33.286 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:33.286 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:33.287 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:33.287 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.256+0000 7fbfc709d700 10 monclient: shutdown 2021-05-25T12:07:33.287 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.256+0000 7fbfc709d700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:07:33.287 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.256+0000 7fbfc709d700 1 -- 172.21.15.102:0/2310516105 >> v1:172.21.15.102:6789/0 conn(0x7fbfc0090d70 legacy=0x7fbfc0091200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:33.287 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.256+0000 7fbfc709d700 1 -- 172.21.15.102:0/2310516105 shutdown_connections 2021-05-25T12:07:33.288 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.256+0000 7fbfc709d700 1 -- 172.21.15.102:0/2310516105 >> 172.21.15.102:0/2310516105 conn(0x7fbfc009b370 msgr2=0x7fbfc0097430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:07:33.288 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.256+0000 7fbfc709d700 1 -- 172.21.15.102:0/2310516105 shutdown_connections 2021-05-25T12:07:33.288 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.256+0000 7fbfc709d700 1 -- 172.21.15.102:0/2310516105 wait complete. 2021-05-25T12:07:33.288 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.257+0000 7fbfc709d700 10 monclient: build_initial_monmap 2021-05-25T12:07:33.289 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.257+0000 7fbfc709d700 10 monclient: monmap: 2021-05-25T12:07:33.289 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:07:33.289 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:07:33.289 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:07:33.289 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:07:33.290 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:07:33.290 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:07:33.290 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:07:33.290 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:07:33.290 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:07:33.291 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:33.291 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.257+0000 7fbfc709d700 1 librados: starting msgr at 2021-05-25T12:07:33.291 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.257+0000 7fbfc709d700 1 librados: starting objecter 2021-05-25T12:07:33.291 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.257+0000 7fbfc709d700 1 Processor -- start 2021-05-25T12:07:33.291 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.257+0000 7fbfc709d700 1 -- start start 2021-05-25T12:07:33.292 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.257+0000 7fbfc709d700 1 librados: setting wanted keys 2021-05-25T12:07:33.292 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.257+0000 7fbfc709d700 1 librados: calling monclient init 2021-05-25T12:07:33.292 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.257+0000 7fbfc709d700 10 monclient: init 2021-05-25T12:07:33.292 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.258+0000 7fbfc709d700 10 monclient: _reopen_session rank -1 2021-05-25T12:07:33.292 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.258+0000 7fbfc709d700 10 monclient: _add_conns ranks=[1,2,0] 2021-05-25T12:07:33.293 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.258+0000 7fbfc709d700 10 monclient(hunting): picked mon.noname-b con 0x7fbfc0090990 addr v1:172.21.15.181:6789/0 2021-05-25T12:07:33.293 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.258+0000 7fbfc709d700 10 monclient(hunting): picked mon.noname-c con 0x7fbfc0090d70 addr v1:172.21.15.102:6790/0 2021-05-25T12:07:33.293 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.258+0000 7fbfc709d700 10 monclient(hunting): picked mon.noname-a con 0x7fbfc012e330 addr v1:172.21.15.102:6789/0 2021-05-25T12:07:33.293 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.258+0000 7fbfc709d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfc01268b0 con 0x7fbfc012e330 2021-05-25T12:07:33.293 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.258+0000 7fbfc709d700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfc00684d0 con 0x7fbfc0090990 2021-05-25T12:07:33.293 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.258+0000 7fbfc709d700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfc0068770 con 0x7fbfc0090d70 2021-05-25T12:07:33.294 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.258+0000 7fbfc709d700 10 monclient(hunting): _renew_subs 2021-05-25T12:07:33.294 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.258+0000 7fbfc709d700 10 monclient(hunting): authenticate will time out at 1253.855189s 2021-05-25T12:07:33.294 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.258+0000 7fbfc563a700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fbfc012e330 0x7fbfc013f330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33806/0 (socket says 172.21.15.102:33806) 2021-05-25T12:07:33.294 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.258+0000 7fbfc563a700 1 -- 172.21.15.102:0/3626073948 learned_addr learned my addr 172.21.15.102:0/3626073948 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:07:33.294 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.258+0000 7fbfb7fff700 1 --1- 172.21.15.102:0/3626073948 >> v1:172.21.15.102:6790/0 conn(0x7fbfc0090d70 0x7fbfc013bc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:56914/0 (socket says 172.21.15.102:56914) 2021-05-25T12:07:33.295 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.258+0000 7fbfb4ff9700 10 client.?.objecter ms_handle_connect 0x7fbfc012e330 2021-05-25T12:07:33.295 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.258+0000 7fbfb4ff9700 10 client.?.objecter resend_mon_ops 2021-05-25T12:07:33.295 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.258+0000 7fbfb4ff9700 10 client.?.objecter ms_handle_connect 0x7fbfc0090d70 2021-05-25T12:07:33.295 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.258+0000 7fbfb4ff9700 10 client.?.objecter resend_mon_ops 2021-05-25T12:07:33.295 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.258+0000 7fbfb4ff9700 10 client.?.objecter ms_handle_connect 0x7fbfc0090990 2021-05-25T12:07:33.296 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.258+0000 7fbfb4ff9700 10 client.?.objecter resend_mon_ops 2021-05-25T12:07:33.296 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.258+0000 7fbfb4ff9700 1 -- 172.21.15.102:0/3626073948 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1337646892 0 0) 0x7fbfc0068770 con 0x7fbfc0090d70 2021-05-25T12:07:33.296 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.258+0000 7fbfb4ff9700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:33.296 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.258+0000 7fbfb4ff9700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:33.296 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.258+0000 7fbfb4ff9700 10 monclient(hunting): my global_id is 234113 2021-05-25T12:07:33.296 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.258+0000 7fbfb4ff9700 1 -- 172.21.15.102:0/3626073948 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfa0004c10 con 0x7fbfc0090d70 2021-05-25T12:07:33.297 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.258+0000 7fbfb4ff9700 1 -- 172.21.15.102:0/3626073948 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3573774615 0 0) 0x7fbfc01268b0 con 0x7fbfc012e330 2021-05-25T12:07:33.297 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.258+0000 7fbfb4ff9700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:33.297 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.258+0000 7fbfb4ff9700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:33.297 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.258+0000 7fbfb4ff9700 10 monclient(hunting): my global_id is 204384 2021-05-25T12:07:33.297 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.258+0000 7fbfb4ff9700 1 -- 172.21.15.102:0/3626073948 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfc0068770 con 0x7fbfc012e330 2021-05-25T12:07:33.298 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.258+0000 7fbfb4ff9700 1 -- 172.21.15.102:0/3626073948 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1497546750 0 0) 0x7fbfc00684d0 con 0x7fbfc0090990 2021-05-25T12:07:33.298 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.258+0000 7fbfb4ff9700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:33.298 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.258+0000 7fbfb4ff9700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:33.298 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.258+0000 7fbfb4ff9700 10 monclient(hunting): my global_id is 224245 2021-05-25T12:07:33.298 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.258+0000 7fbfb4ff9700 1 -- 172.21.15.102:0/3626073948 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfc01268b0 con 0x7fbfc0090990 2021-05-25T12:07:33.299 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.258+0000 7fbfb4ff9700 1 -- 172.21.15.102:0/3626073948 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4177112324 0 0) 0x7fbfa0004c10 con 0x7fbfc0090d70 2021-05-25T12:07:33.299 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.258+0000 7fbfb4ff9700 1 -- 172.21.15.102:0/3626073948 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbfc00684d0 con 0x7fbfc0090d70 2021-05-25T12:07:33.299 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.258+0000 7fbfb4ff9700 1 -- 172.21.15.102:0/3626073948 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fbfa8001440 con 0x7fbfc0090d70 2021-05-25T12:07:33.299 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.258+0000 7fbfb4ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:33.299 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.258+0000 7fbfb4ff9700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:07:33.300 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.258+0000 7fbfb4ff9700 10 monclient(hunting): dump: 2021-05-25T12:07:33.300 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:33.300 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:33.300 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:33.300 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:33.301 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:33.301 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:33.301 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:33.301 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:33.301 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:33.302 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:33.302 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.258+0000 7fbfb4ff9700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:07:33.302 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.258+0000 7fbfb4ff9700 1 -- 172.21.15.102:0/3626073948 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3386821809 0 0) 0x7fbfc0068770 con 0x7fbfc012e330 2021-05-25T12:07:33.302 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.258+0000 7fbfb4ff9700 1 -- 172.21.15.102:0/3626073948 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbfa0004c10 con 0x7fbfc012e330 2021-05-25T12:07:33.302 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.258+0000 7fbfb4ff9700 1 -- 172.21.15.102:0/3626073948 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fbfbc001d10 con 0x7fbfc012e330 2021-05-25T12:07:33.303 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.258+0000 7fbfb4ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:33.303 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.258+0000 7fbfb4ff9700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2021-05-25T12:07:33.303 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.258+0000 7fbfb4ff9700 10 monclient(hunting): dump: 2021-05-25T12:07:33.303 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:33.303 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:33.304 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:33.304 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:33.304 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:33.304 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:33.304 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:33.304 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:33.305 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:33.305 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:33.305 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.258+0000 7fbfb4ff9700 1 -- 172.21.15.102:0/3626073948 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2520801123 0 0) 0x7fbfc01268b0 con 0x7fbfc0090990 2021-05-25T12:07:33.305 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.258+0000 7fbfb4ff9700 1 -- 172.21.15.102:0/3626073948 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbfc0068770 con 0x7fbfc0090990 2021-05-25T12:07:33.306 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.258+0000 7fbfb4ff9700 1 -- 172.21.15.102:0/3626073948 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fbfb00019f0 con 0x7fbfc0090990 2021-05-25T12:07:33.306 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.258+0000 7fbfb4ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:33.306 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.258+0000 7fbfb4ff9700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2021-05-25T12:07:33.306 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.258+0000 7fbfb4ff9700 10 monclient(hunting): dump: 2021-05-25T12:07:33.306 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:33.307 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:33.307 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:33.307 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:33.307 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:33.308 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:33.308 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:33.308 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:33.308 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:33.308 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:33.309 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.258+0000 7fbfb4ff9700 1 -- 172.21.15.102:0/3626073948 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1918193320 0 0) 0x7fbfc00684d0 con 0x7fbfc0090d70 2021-05-25T12:07:33.309 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.258+0000 7fbfb4ff9700 1 -- 172.21.15.102:0/3626073948 >> v1:172.21.15.181:6789/0 conn(0x7fbfc0090990 legacy=0x7fbfc01261a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:33.309 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.258+0000 7fbfb4ff9700 1 -- 172.21.15.102:0/3626073948 >> v1:172.21.15.102:6789/0 conn(0x7fbfc012e330 legacy=0x7fbfc013f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:33.309 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.258+0000 7fbfb4ff9700 10 monclient: _finish_hunting 0 2021-05-25T12:07:33.309 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.258+0000 7fbfb4ff9700 1 monclient: found mon.c 2021-05-25T12:07:33.310 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.258+0000 7fbfb4ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:07:33.310 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.258+0000 7fbfb4ff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:07:33.310 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.258+0000 7fbfb4ff9700 1 -- 172.21.15.102:0/3626073948 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfc0068a10 con 0x7fbfc0090d70 2021-05-25T12:07:33.310 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.258+0000 7fbfb4ff9700 10 monclient: _finish_auth 0 2021-05-25T12:07:33.311 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.258+0000 7fbfb4ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:07:33.311 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.258+0000 7fbfc709d700 5 monclient: authenticate success, global_id 234113 2021-05-25T12:07:33.311 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.258+0000 7fbfc709d700 10 monclient: _renew_subs 2021-05-25T12:07:33.311 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.258+0000 7fbfc709d700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:07:33.311 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.258+0000 7fbfc709d700 1 -- 172.21.15.102:0/3626073948 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfc0066980 con 0x7fbfc0090d70 2021-05-25T12:07:33.312 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.259+0000 7fbfc709d700 10 client.234113.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:07:33.312 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.259+0000 7fbfc709d700 10 monclient: _renew_subs 2021-05-25T12:07:33.312 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.259+0000 7fbfc709d700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:07:33.312 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.259+0000 7fbfc709d700 1 -- 172.21.15.102:0/3626073948 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfc0137370 con 0x7fbfc0090d70 2021-05-25T12:07:33.312 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.259+0000 7fbfc709d700 1 librados: init done 2021-05-25T12:07:33.313 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.259+0000 7fbfb4ff9700 1 -- 172.21.15.102:0/3626073948 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbfa80047e0 con 0x7fbfc0090d70 2021-05-25T12:07:33.313 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.259+0000 7fbfb4ff9700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:07:33.313 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.259+0000 7fbfb4ff9700 1 -- 172.21.15.102:0/3626073948 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fbfa8004af0 con 0x7fbfc0090d70 2021-05-25T12:07:33.313 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.259+0000 7fbfb4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:33.313 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.259+0000 7fbfb4ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:07:33.314 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.259+0000 7fbfb6ffd700 4 set_mon_vals no callback set 2021-05-25T12:07:33.314 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.259+0000 7fbfb4ff9700 10 monclient: dump: 2021-05-25T12:07:33.314 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:33.314 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:33.314 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:33.315 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:33.315 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:33.315 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:33.315 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:33.315 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:33.316 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:33.316 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:33.316 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.259+0000 7fbfb4ff9700 1 -- 172.21.15.102:0/3626073948 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7fbfa800de10 con 0x7fbfc0090d70 2021-05-25T12:07:33.316 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.259+0000 7fbfc709d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:07:33.316 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.259+0000 7fbfc709d700 1 -- 172.21.15.102:0/3626073948 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fbfc0059730 con 0x7fbfc0136a90 2021-05-25T12:07:33.317 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.259+0000 7fbfb4ff9700 1 -- 172.21.15.102:0/3626073948 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(243..243 src has 1..243) v4 ==== 5568+0+0 (unknown 453324089 0 0) 0x7fbfa803d610 con 0x7fbfc0090d70 2021-05-25T12:07:33.317 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.259+0000 7fbfb4ff9700 10 client.234113.objecter ms_dispatch 0x7fbfc0001550 osd_map(243..243 src has 1..243) v4 2021-05-25T12:07:33.317 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.259+0000 7fbfb4ff9700 3 client.234113.objecter handle_osd_map got epochs [243,243] > 0 2021-05-25T12:07:33.317 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.259+0000 7fbfb4ff9700 3 client.234113.objecter handle_osd_map decoding full epoch 243 2021-05-25T12:07:33.317 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.259+0000 7fbfb4ff9700 20 client.234113.objecter dump_active .. 0 homeless 2021-05-25T12:07:33.318 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.259+0000 7fbfb4ff9700 10 client.234113.objecter ms_handle_connect 0x7fbfc0136a90 2021-05-25T12:07:33.318 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.259+0000 7fbfb4ff9700 10 client.234113.objecter resend_mon_ops 2021-05-25T12:07:33.318 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.259+0000 7fbfc563a700 10 monclient: get_auth_request con 0x7fbfa0033620 auth_method 0 2021-05-25T12:07:33.318 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.259+0000 7fbfb4ff9700 1 -- 172.21.15.102:0/3626073948 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2051108771 0 0) 0x7fbfc0059730 con 0x7fbfc0136a90 2021-05-25T12:07:33.318 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.259+0000 7fbfb4ff9700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:33.319 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.259+0000 7fbfb4ff9700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:33.319 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.259+0000 7fbfb4ff9700 10 monclient(hunting): my global_id is 234116 2021-05-25T12:07:33.319 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.259+0000 7fbfb4ff9700 1 -- 172.21.15.102:0/3626073948 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfc00684d0 con 0x7fbfc0136a90 2021-05-25T12:07:33.319 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.260+0000 7fbfb4ff9700 10 client.234113.objecter ms_handle_connect 0x7fbfa0033620 2021-05-25T12:07:33.320 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.260+0000 7fbfb4ff9700 1 -- 172.21.15.102:0/3626073948 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3741057622 0 0) 0x7fbfc00684d0 con 0x7fbfc0136a90 2021-05-25T12:07:33.320 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.260+0000 7fbfb4ff9700 1 -- 172.21.15.102:0/3626073948 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbfc0059730 con 0x7fbfc0136a90 2021-05-25T12:07:33.320 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.260+0000 7fbfb4ff9700 1 -- 172.21.15.102:0/3626073948 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fbfb00046d0 con 0x7fbfc0136a90 2021-05-25T12:07:33.320 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.260+0000 7fbfb4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:33.320 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.260+0000 7fbfb4ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:07:33.321 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.260+0000 7fbfb4ff9700 10 monclient: dump: 2021-05-25T12:07:33.321 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:33.321 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:33.321 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:33.321 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:33.322 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:33.322 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:33.322 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:33.322 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:33.323 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:33.323 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:33.323 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.260+0000 7fbfb4ff9700 1 -- 172.21.15.102:0/3626073948 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3984947010 0 0) 0x7fbfc0059730 con 0x7fbfc0136a90 2021-05-25T12:07:33.323 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.260+0000 7fbfb4ff9700 1 -- 172.21.15.102:0/3626073948 --> v1:172.21.15.102:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbfc0136e70 con 0x7fbfc0136a90 2021-05-25T12:07:33.323 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.260+0000 7fbfb4ff9700 1 -- 172.21.15.102:0/3626073948 <== mon.2 v1:172.21.15.102:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7fbfb00068d0 con 0x7fbfc0136a90 2021-05-25T12:07:33.324 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.260+0000 7fbfb4ff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:07:33.324 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.260+0000 7fbfb4ff9700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:07:33.324 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.260+0000 7fbfb4ff9700 1 -- 172.21.15.102:0/3626073948 >> v1:172.21.15.102:6790/0 conn(0x7fbfc0136a90 legacy=0x7fbfc0058d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:33.324 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.260+0000 7fbfb4ff9700 1 -- 172.21.15.102:0/3626073948 >> v1:172.21.15.102:6790/0 conn(0x7fbfc0136a90 legacy=0x7fbfc0058d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:07:33.325 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.267+0000 7fbfc709d700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:07:33.325 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.267+0000 7fbfc709d700 1 -- 172.21.15.102:0/3626073948 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fbfc005d870 con 0x7fbfc012b4a0 2021-05-25T12:07:33.325 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.267+0000 7fbfb4ff9700 10 client.234113.objecter ms_handle_connect 0x7fbfc012b4a0 2021-05-25T12:07:33.325 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.267+0000 7fbfb4ff9700 10 client.234113.objecter resend_mon_ops 2021-05-25T12:07:33.326 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.267+0000 7fbfb4ff9700 1 -- 172.21.15.102:0/3626073948 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1088763553 0 0) 0x7fbfc005d870 con 0x7fbfc012b4a0 2021-05-25T12:07:33.326 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.267+0000 7fbfb4ff9700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:33.326 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.267+0000 7fbfb4ff9700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:33.326 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.267+0000 7fbfb4ff9700 10 monclient(hunting): my global_id is 234119 2021-05-25T12:07:33.326 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.267+0000 7fbfb4ff9700 1 -- 172.21.15.102:0/3626073948 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfc0059730 con 0x7fbfc012b4a0 2021-05-25T12:07:33.327 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.267+0000 7fbfb4ff9700 1 -- 172.21.15.102:0/3626073948 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3405397626 0 0) 0x7fbfc0059730 con 0x7fbfc012b4a0 2021-05-25T12:07:33.327 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.267+0000 7fbfb4ff9700 1 -- 172.21.15.102:0/3626073948 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbfc005d870 con 0x7fbfc012b4a0 2021-05-25T12:07:33.327 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.267+0000 7fbfb4ff9700 1 -- 172.21.15.102:0/3626073948 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fbfb0004d70 con 0x7fbfc012b4a0 2021-05-25T12:07:33.327 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.267+0000 7fbfb4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:33.327 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.267+0000 7fbfb4ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:07:33.328 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.267+0000 7fbfb4ff9700 10 monclient: dump: 2021-05-25T12:07:33.328 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:33.328 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:33.328 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:33.328 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:33.329 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:33.329 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:33.329 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:33.329 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:33.329 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:33.330 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:33.330 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.267+0000 7fbfb4ff9700 1 -- 172.21.15.102:0/3626073948 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1287467242 0 0) 0x7fbfc005d870 con 0x7fbfc012b4a0 2021-05-25T12:07:33.330 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.267+0000 7fbfb4ff9700 1 -- 172.21.15.102:0/3626073948 --> v1:172.21.15.102:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fbfc0059420 con 0x7fbfc012b4a0 2021-05-25T12:07:33.330 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.267+0000 7fbfb4ff9700 1 -- 172.21.15.102:0/3626073948 <== mon.2 v1:172.21.15.102:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3772 (unknown 0 0 3204108823) 0x7fbfb0006b60 con 0x7fbfc012b4a0 2021-05-25T12:07:33.331 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.267+0000 7fbfb4ff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:07:33.331 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.267+0000 7fbfb4ff9700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:07:33.331 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.267+0000 7fbfb4ff9700 1 -- 172.21.15.102:0/3626073948 >> v1:172.21.15.102:6790/0 conn(0x7fbfc012b4a0 legacy=0x7fbfc0144a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:33.331 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.267+0000 7fbfb4ff9700 1 -- 172.21.15.102:0/3626073948 >> v1:172.21.15.102:6790/0 conn(0x7fbfc012b4a0 legacy=0x7fbfc0144a70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:07:33.332 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:07:33.332 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:07:33.332 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:07:33.332 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "leader", 2021-05-25T12:07:33.333 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 64, 2021-05-25T12:07:33.333 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:07:33.333 INFO:teuthology.orchestra.run.smithi102.stdout: 0, 2021-05-25T12:07:33.333 INFO:teuthology.orchestra.run.smithi102.stdout: 1, 2021-05-25T12:07:33.333 INFO:teuthology.orchestra.run.smithi102.stdout: 2 2021-05-25T12:07:33.334 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:07:33.334 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 0, 2021-05-25T12:07:33.334 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:07:33.334 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:07:33.334 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:07:33.335 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:07:33.335 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:07:33.335 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:07:33.335 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:07:33.335 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:07:33.336 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:07:33.336 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:07:33.336 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:07:33.336 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:07:33.336 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:07:33.337 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:07:33.337 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:07:33.337 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:07:33.337 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:07:33.337 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:07:33.338 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:07:33.338 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:07:33.338 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:07:33.338 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:07:33.338 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:07:33.339 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:07:33.339 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:07:33.339 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:33.339 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:07:33.340 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:07:33.340 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:07:33.340 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:07:33.340 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:07:33.340 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:07:33.341 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:07:33.341 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:07:33.341 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:07:33.341 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:07:33.341 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:07:33.342 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:07:33.342 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:07:33.342 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:07:33.342 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:07:33.342 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:07:33.343 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:07:33.343 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:07:33.343 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:07:33.343 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:07:33.343 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:07:33.344 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:07:33.344 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:07:33.344 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:07:33.344 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:07:33.345 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:07:33.345 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:33.345 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:07:33.345 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:33.345 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:07:33.346 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:07:33.346 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:07:33.346 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:07:33.346 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:33.346 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:07:33.347 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:07:33.347 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:07:33.347 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:07:33.347 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:07:33.347 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:33.348 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:07:33.348 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:07:33.348 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:07:33.348 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:07:33.348 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:07:33.349 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:33.349 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:33.349 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:07:33.349 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:07:33.350 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:07:33.350 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:07:33.350 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:33.350 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:07:33.350 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:07:33.351 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:07:33.351 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:07:33.351 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:07:33.351 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:33.351 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:07:33.352 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:07:33.352 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:07:33.352 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:07:33.352 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:07:33.352 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:33.353 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:33.353 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:07:33.353 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:07:33.353 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:07:33.353 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:07:33.354 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:33.354 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:07:33.354 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:07:33.354 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:07:33.354 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:07:33.355 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:07:33.355 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:33.355 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:07:33.355 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:07:33.355 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:07:33.356 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:07:33.356 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:07:33.356 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:07:33.356 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:07:33.356 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:33.357 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:07:33.357 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:07:33.357 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:33.357 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:07:33.357 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:07:33.358 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:07:33.358 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:07:33.358 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:07:33.358 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:07:33.358 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:33.359 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:07:33.359 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:07:33.359 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 2 2021-05-25T12:07:33.359 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:07:33.360 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:07:33.360 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:33.360 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:07:33.360 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:07:33.362 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.268+0000 7fbfc709d700 10 librados: watch_flush enter 2021-05-25T12:07:33.362 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.268+0000 7fbfc709d700 10 librados: watch_flush exit 2021-05-25T12:07:33.363 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.268+0000 7fbfc709d700 20 client.234113.objecter shutdown clearing up homeless session... 2021-05-25T12:07:33.363 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.268+0000 7fbfc709d700 10 client.234113.objecter successfully canceled tick 2021-05-25T12:07:33.363 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.268+0000 7fbfc709d700 1 -- 172.21.15.102:0/3626073948 >> v1:172.21.15.181:6812/29582 conn(0x7fbfa0033620 legacy=0x7fbfa0035ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:33.363 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.268+0000 7fbfc709d700 10 monclient: shutdown 2021-05-25T12:07:33.363 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.268+0000 7fbfc709d700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:07:33.364 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.268+0000 7fbfc709d700 1 -- 172.21.15.102:0/3626073948 >> v1:172.21.15.102:6790/0 conn(0x7fbfc0090d70 legacy=0x7fbfc013bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:33.364 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.268+0000 7fbfc563a700 1 -- 172.21.15.102:0/3626073948 reap_dead start 2021-05-25T12:07:33.364 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.268+0000 7fbfc563a700 1 -- 172.21.15.102:0/3626073948 reap_dead start 2021-05-25T12:07:33.364 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.268+0000 7fbfc709d700 1 -- 172.21.15.102:0/3626073948 shutdown_connections 2021-05-25T12:07:33.365 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.268+0000 7fbfc709d700 1 -- 172.21.15.102:0/3626073948 >> 172.21.15.102:0/3626073948 conn(0x7fbfc009b370 msgr2=0x7fbfc01292f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:07:33.365 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.268+0000 7fbfc709d700 1 -- 172.21.15.102:0/3626073948 shutdown_connections 2021-05-25T12:07:33.365 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.268+0000 7fbfc709d700 1 -- 172.21.15.102:0/3626073948 wait complete. 2021-05-25T12:07:33.365 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.268+0000 7fbfc709d700 1 librados: shutdown 2021-05-25T12:07:33.366 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2021-05-25T12:07:33.476 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.474+0000 7f69fe9ec700 10 monclient: get_monmap_and_config 2021-05-25T12:07:33.476 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.474+0000 7f69fe9ec700 10 monclient: build_initial_monmap 2021-05-25T12:07:33.476 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.474+0000 7f69fe9ec700 10 monclient: monmap: 2021-05-25T12:07:33.476 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:07:33.477 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:33.477 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:07:33.475685+0000 2021-05-25T12:07:33.477 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:07:33.475685+0000 2021-05-25T12:07:33.477 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:07:33.477 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:33.478 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:07:33.478 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:07:33.478 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:07:33.478 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:33.491 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.475+0000 7f69fe9ec700 1 Processor -- start 2021-05-25T12:07:33.492 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.475+0000 7f69fe9ec700 1 -- start start 2021-05-25T12:07:33.492 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.475+0000 7f69fe9ec700 10 monclient: init 2021-05-25T12:07:33.492 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.475+0000 7f69fe9ec700 10 monclient: _reopen_session rank -1 2021-05-25T12:07:33.492 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.475+0000 7f69fe9ec700 10 monclient: _add_conns ranks=[2,0,1] 2021-05-25T12:07:33.493 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.476+0000 7f69fe9ec700 10 monclient(hunting): picked mon.noname-b con 0x7f69f8095c40 addr v1:172.21.15.102:6790/0 2021-05-25T12:07:33.493 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.476+0000 7f69fe9ec700 10 monclient(hunting): picked mon.noname-a con 0x7f69f8096020 addr v1:172.21.15.102:6789/0 2021-05-25T12:07:33.493 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.476+0000 7f69fe9ec700 10 monclient(hunting): picked mon.noname-c con 0x7f69f812e5c0 addr v1:172.21.15.181:6789/0 2021-05-25T12:07:33.493 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.476+0000 7f69fe9ec700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69f8092990 con 0x7f69f8096020 2021-05-25T12:07:33.493 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.476+0000 7f69fe9ec700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69f8092c30 con 0x7f69f812e5c0 2021-05-25T12:07:33.494 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.476+0000 7f69fe9ec700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69f8092ed0 con 0x7f69f8095c40 2021-05-25T12:07:33.494 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.476+0000 7f69fe9ec700 10 monclient(hunting): _renew_subs 2021-05-25T12:07:33.494 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.476+0000 7f69fe9ec700 10 monclient(hunting): authenticate will time out at 1254.073103s 2021-05-25T12:07:33.494 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.476+0000 7f69fd9ea700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f69f8095c40 0x7f69f8093290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:56924/0 (socket says 172.21.15.102:56924) 2021-05-25T12:07:33.494 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.476+0000 7f69fd1e9700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f69f8096020 0x7f69f80964d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33816/0 (socket says 172.21.15.102:33816) 2021-05-25T12:07:33.495 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.476+0000 7f69fd9ea700 1 -- 172.21.15.102:0/1501871714 learned_addr learned my addr 172.21.15.102:0/1501871714 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:07:33.495 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.476+0000 7f69fc9e8700 1 -- 172.21.15.102:0/1501871714 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2387061126 0 0) 0x7f69f8092ed0 con 0x7f69f8095c40 2021-05-25T12:07:33.495 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.476+0000 7f69fc9e8700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:33.495 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.476+0000 7f69fc9e8700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:33.496 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.476+0000 7f69fc9e8700 10 monclient(hunting): my global_id is 234122 2021-05-25T12:07:33.496 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.476+0000 7f69fc9e8700 1 -- 172.21.15.102:0/1501871714 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69e0004380 con 0x7f69f8095c40 2021-05-25T12:07:33.496 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.476+0000 7f69fc9e8700 1 -- 172.21.15.102:0/1501871714 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1902300310 0 0) 0x7f69f8092990 con 0x7f69f8096020 2021-05-25T12:07:33.496 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.476+0000 7f69fc9e8700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:33.496 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.476+0000 7f69fc9e8700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:33.497 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.476+0000 7f69fc9e8700 10 monclient(hunting): my global_id is 204387 2021-05-25T12:07:33.497 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.476+0000 7f69fc9e8700 1 -- 172.21.15.102:0/1501871714 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69f8092ed0 con 0x7f69f8096020 2021-05-25T12:07:33.497 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.476+0000 7f69fc9e8700 1 -- 172.21.15.102:0/1501871714 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 644284295 0 0) 0x7f69f8092c30 con 0x7f69f812e5c0 2021-05-25T12:07:33.497 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.476+0000 7f69fc9e8700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:33.497 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.476+0000 7f69fc9e8700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:33.498 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.476+0000 7f69fc9e8700 10 monclient(hunting): my global_id is 224248 2021-05-25T12:07:33.498 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.476+0000 7f69fc9e8700 1 -- 172.21.15.102:0/1501871714 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69f8092990 con 0x7f69f812e5c0 2021-05-25T12:07:33.498 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.476+0000 7f69fc9e8700 1 -- 172.21.15.102:0/1501871714 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3316142961 0 0) 0x7f69e0004380 con 0x7f69f8095c40 2021-05-25T12:07:33.498 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.476+0000 7f69fc9e8700 1 -- 172.21.15.102:0/1501871714 >> v1:172.21.15.181:6789/0 conn(0x7f69f812e5c0 legacy=0x7f69f81309b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:33.498 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.476+0000 7f69fc9e8700 1 -- 172.21.15.102:0/1501871714 >> v1:172.21.15.102:6789/0 conn(0x7f69f8096020 legacy=0x7f69f80964d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:33.499 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.476+0000 7f69fc9e8700 10 monclient: _finish_hunting 0 2021-05-25T12:07:33.499 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.476+0000 7f69fc9e8700 1 monclient: found mon.noname-b 2021-05-25T12:07:33.499 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.476+0000 7f69fc9e8700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:07:33.499 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.476+0000 7f69fc9e8700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T12:07:33.500 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.476+0000 7f69fc9e8700 1 -- 172.21.15.102:0/1501871714 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69f8092430 con 0x7f69f8095c40 2021-05-25T12:07:33.500 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.476+0000 7f69fc9e8700 10 monclient: _finish_auth 0 2021-05-25T12:07:33.500 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.476+0000 7f69fc9e8700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:07:33.500 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.476+0000 7f69fc9e8700 1 -- 172.21.15.102:0/1501871714 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f69ec001610 con 0x7f69f8095c40 2021-05-25T12:07:33.500 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.476+0000 7f69fc9e8700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:33.501 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.476+0000 7f69fc9e8700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:07:33.501 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.476+0000 7f69fc9e8700 10 monclient: dump: 2021-05-25T12:07:33.501 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:33.501 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:33.501 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:33.502 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:33.502 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:33.502 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:33.502 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:33.502 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:33.503 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:33.503 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:33.503 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.476+0000 7f69fe9ec700 5 monclient: authenticate success, global_id 234122 2021-05-25T12:07:33.503 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.476+0000 7f69fe9ec700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:07:33.503 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.476+0000 7f69fc9e8700 1 -- 172.21.15.102:0/1501871714 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f69ec001880 con 0x7f69f8095c40 2021-05-25T12:07:33.504 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.476+0000 7f69fc9e8700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:07:33.504 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.476+0000 7f69fe9ec700 10 monclient: get_monmap_and_config success 2021-05-25T12:07:33.504 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.476+0000 7f69fc9e8700 1 -- 172.21.15.102:0/1501871714 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f69ec001c10 con 0x7f69f8095c40 2021-05-25T12:07:33.504 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.476+0000 7f69fe9ec700 4 set_mon_vals no callback set 2021-05-25T12:07:33.504 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.476+0000 7f69fc9e8700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:33.505 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.476+0000 7f69fc9e8700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:07:33.505 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.476+0000 7f69fc9e8700 10 monclient: dump: 2021-05-25T12:07:33.505 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:33.505 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:33.505 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:33.506 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:33.506 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:33.506 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:33.506 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:33.506 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:33.507 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:33.507 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:33.507 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.477+0000 7f69fe9ec700 10 monclient: shutdown 2021-05-25T12:07:33.507 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.477+0000 7f69fe9ec700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:07:33.507 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.477+0000 7f69fe9ec700 1 -- 172.21.15.102:0/1501871714 >> v1:172.21.15.102:6790/0 conn(0x7f69f8095c40 legacy=0x7f69f8093290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:33.508 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.477+0000 7f69fe9ec700 1 -- 172.21.15.102:0/1501871714 shutdown_connections 2021-05-25T12:07:33.508 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.477+0000 7f69fe9ec700 1 -- 172.21.15.102:0/1501871714 >> 172.21.15.102:0/1501871714 conn(0x7f69f80974b0 msgr2=0x7f69f80978c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:07:33.508 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.477+0000 7f69fe9ec700 1 -- 172.21.15.102:0/1501871714 shutdown_connections 2021-05-25T12:07:33.508 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.477+0000 7f69fe9ec700 1 -- 172.21.15.102:0/1501871714 wait complete. 2021-05-25T12:07:33.508 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.477+0000 7f69fe9ec700 10 monclient: build_initial_monmap 2021-05-25T12:07:33.509 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.477+0000 7f69fe9ec700 10 monclient: monmap: 2021-05-25T12:07:33.509 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:07:33.509 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:07:33.509 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:07:33.509 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:07:33.510 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:07:33.510 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:07:33.510 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:07:33.510 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:07:33.511 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:07:33.511 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:33.511 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.478+0000 7f69fe9ec700 1 librados: starting msgr at 2021-05-25T12:07:33.511 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.478+0000 7f69fe9ec700 1 librados: starting objecter 2021-05-25T12:07:33.511 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.478+0000 7f69fe9ec700 1 Processor -- start 2021-05-25T12:07:33.512 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.478+0000 7f69fe9ec700 1 -- start start 2021-05-25T12:07:33.512 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.478+0000 7f69fe9ec700 1 librados: setting wanted keys 2021-05-25T12:07:33.512 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.478+0000 7f69fe9ec700 1 librados: calling monclient init 2021-05-25T12:07:33.512 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.478+0000 7f69fe9ec700 10 monclient: init 2021-05-25T12:07:33.512 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.478+0000 7f69fe9ec700 10 monclient: _reopen_session rank -1 2021-05-25T12:07:33.513 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.478+0000 7f69fe9ec700 10 monclient: _add_conns ranks=[2,0,1] 2021-05-25T12:07:33.513 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.478+0000 7f69fe9ec700 10 monclient(hunting): picked mon.noname-c con 0x7f69f8095c40 addr v1:172.21.15.102:6790/0 2021-05-25T12:07:33.513 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.478+0000 7f69fe9ec700 10 monclient(hunting): picked mon.noname-a con 0x7f69f8096020 addr v1:172.21.15.102:6789/0 2021-05-25T12:07:33.513 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.478+0000 7f69fe9ec700 10 monclient(hunting): picked mon.noname-b con 0x7f69f812e5c0 addr v1:172.21.15.181:6789/0 2021-05-25T12:07:33.513 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.478+0000 7f69fe9ec700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69f8113030 con 0x7f69f8096020 2021-05-25T12:07:33.514 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.478+0000 7f69fe9ec700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69f8128f60 con 0x7f69f812e5c0 2021-05-25T12:07:33.514 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.478+0000 7f69fe9ec700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69f81132a0 con 0x7f69f8095c40 2021-05-25T12:07:33.514 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.478+0000 7f69fe9ec700 10 monclient(hunting): _renew_subs 2021-05-25T12:07:33.514 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.478+0000 7f69fe9ec700 10 monclient(hunting): authenticate will time out at 1254.075739s 2021-05-25T12:07:33.514 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.478+0000 7f69fd9ea700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f69f8095c40 0x7f69f8128850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:56930/0 (socket says 172.21.15.102:56930) 2021-05-25T12:07:33.515 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.478+0000 7f69fd9ea700 1 -- 172.21.15.102:0/3809855660 learned_addr learned my addr 172.21.15.102:0/3809855660 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:07:33.515 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.478+0000 7f69e5ffb700 10 client.?.objecter ms_handle_connect 0x7f69f8095c40 2021-05-25T12:07:33.515 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.478+0000 7f69e5ffb700 10 client.?.objecter resend_mon_ops 2021-05-25T12:07:33.515 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.478+0000 7f69e5ffb700 10 client.?.objecter ms_handle_connect 0x7f69f8096020 2021-05-25T12:07:33.515 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.478+0000 7f69e5ffb700 10 client.?.objecter resend_mon_ops 2021-05-25T12:07:33.516 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.478+0000 7f69e5ffb700 1 -- 172.21.15.102:0/3809855660 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1307772173 0 0) 0x7f69f81132a0 con 0x7f69f8095c40 2021-05-25T12:07:33.516 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.478+0000 7f69e5ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:33.516 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.479+0000 7f69e5ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:33.516 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.479+0000 7f69e5ffb700 10 monclient(hunting): my global_id is 234125 2021-05-25T12:07:33.516 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.479+0000 7f69e5ffb700 1 -- 172.21.15.102:0/3809855660 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69d8004c10 con 0x7f69f8095c40 2021-05-25T12:07:33.517 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.479+0000 7f69e5ffb700 10 client.?.objecter ms_handle_connect 0x7f69f812e5c0 2021-05-25T12:07:33.517 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.479+0000 7f69e5ffb700 10 client.?.objecter resend_mon_ops 2021-05-25T12:07:33.517 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.479+0000 7f69e5ffb700 1 -- 172.21.15.102:0/3809855660 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 638952811 0 0) 0x7f69f8113030 con 0x7f69f8096020 2021-05-25T12:07:33.517 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.479+0000 7f69e5ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:33.517 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.479+0000 7f69e5ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:33.518 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.479+0000 7f69e5ffb700 10 monclient(hunting): my global_id is 204390 2021-05-25T12:07:33.518 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.479+0000 7f69e5ffb700 1 -- 172.21.15.102:0/3809855660 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69f81132a0 con 0x7f69f8096020 2021-05-25T12:07:33.518 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.479+0000 7f69e5ffb700 1 -- 172.21.15.102:0/3809855660 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2126037460 0 0) 0x7f69f8128f60 con 0x7f69f812e5c0 2021-05-25T12:07:33.518 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.479+0000 7f69e5ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:33.518 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.479+0000 7f69e5ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:33.519 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.479+0000 7f69e5ffb700 10 monclient(hunting): my global_id is 224251 2021-05-25T12:07:33.519 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.479+0000 7f69e5ffb700 1 -- 172.21.15.102:0/3809855660 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69f8113030 con 0x7f69f812e5c0 2021-05-25T12:07:33.519 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.479+0000 7f69e5ffb700 1 -- 172.21.15.102:0/3809855660 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3465398128 0 0) 0x7f69d8004c10 con 0x7f69f8095c40 2021-05-25T12:07:33.519 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.479+0000 7f69e5ffb700 1 -- 172.21.15.102:0/3809855660 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f69f8128f60 con 0x7f69f8095c40 2021-05-25T12:07:33.519 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.479+0000 7f69e5ffb700 1 -- 172.21.15.102:0/3809855660 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f69ec001690 con 0x7f69f8095c40 2021-05-25T12:07:33.520 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.479+0000 7f69e5ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:33.520 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.479+0000 7f69e5ffb700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:07:33.520 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.479+0000 7f69e5ffb700 10 monclient(hunting): dump: 2021-05-25T12:07:33.520 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:33.520 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:33.521 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:33.521 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:33.521 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:33.521 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:33.521 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:33.522 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:33.522 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:33.522 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:33.522 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.479+0000 7f69e5ffb700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:07:33.522 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.479+0000 7f69e5ffb700 1 -- 172.21.15.102:0/3809855660 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 877596547 0 0) 0x7f69f81132a0 con 0x7f69f8096020 2021-05-25T12:07:33.522 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.479+0000 7f69e5ffb700 1 -- 172.21.15.102:0/3809855660 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f69d8004c10 con 0x7f69f8096020 2021-05-25T12:07:33.523 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.479+0000 7f69e5ffb700 1 -- 172.21.15.102:0/3809855660 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f69e8001840 con 0x7f69f8096020 2021-05-25T12:07:33.523 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.479+0000 7f69e5ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:33.523 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.479+0000 7f69e5ffb700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2021-05-25T12:07:33.523 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.479+0000 7f69e5ffb700 10 monclient(hunting): dump: 2021-05-25T12:07:33.524 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:33.524 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:33.524 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:33.524 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:33.524 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:33.525 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:33.525 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:33.525 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:33.525 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:33.526 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:33.526 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.479+0000 7f69e5ffb700 1 -- 172.21.15.102:0/3809855660 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 618543288 0 0) 0x7f69f8128f60 con 0x7f69f8095c40 2021-05-25T12:07:33.526 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.479+0000 7f69e5ffb700 1 -- 172.21.15.102:0/3809855660 >> v1:172.21.15.181:6789/0 conn(0x7f69f812e5c0 legacy=0x7f69f813f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:33.526 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.479+0000 7f69e5ffb700 1 -- 172.21.15.102:0/3809855660 >> v1:172.21.15.102:6789/0 conn(0x7f69f8096020 legacy=0x7f69f813c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:33.526 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.479+0000 7f69e5ffb700 10 monclient: _finish_hunting 0 2021-05-25T12:07:33.527 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.479+0000 7f69e5ffb700 1 monclient: found mon.c 2021-05-25T12:07:33.527 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.479+0000 7f69e5ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:07:33.527 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.479+0000 7f69e5ffb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:07:33.527 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.479+0000 7f69e5ffb700 1 -- 172.21.15.102:0/3809855660 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69f81293c0 con 0x7f69f8095c40 2021-05-25T12:07:33.527 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.479+0000 7f69e5ffb700 10 monclient: _finish_auth 0 2021-05-25T12:07:33.528 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.479+0000 7f69e5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:07:33.528 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.479+0000 7f69fe9ec700 5 monclient: authenticate success, global_id 234125 2021-05-25T12:07:33.528 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.479+0000 7f69fe9ec700 10 monclient: _renew_subs 2021-05-25T12:07:33.528 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.479+0000 7f69fe9ec700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:07:33.528 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.479+0000 7f69fe9ec700 1 -- 172.21.15.102:0/3809855660 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69f8112740 con 0x7f69f8095c40 2021-05-25T12:07:33.529 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.479+0000 7f69e5ffb700 1 -- 172.21.15.102:0/3809855660 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f69ec004860 con 0x7f69f8095c40 2021-05-25T12:07:33.529 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.479+0000 7f69e5ffb700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:07:33.529 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.479+0000 7f69e5ffb700 1 -- 172.21.15.102:0/3809855660 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f69ec004bf0 con 0x7f69f8095c40 2021-05-25T12:07:33.529 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.479+0000 7f69e5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:33.529 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.479+0000 7f69e5ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:07:33.530 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.479+0000 7f69e5ffb700 10 monclient: dump: 2021-05-25T12:07:33.530 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:33.530 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:33.530 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:33.530 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:33.531 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:33.531 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:33.531 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:33.531 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:33.531 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:33.532 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:33.532 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.479+0000 7f69e7fff700 4 set_mon_vals no callback set 2021-05-25T12:07:33.532 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.479+0000 7f69fe9ec700 10 client.234125.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:07:33.532 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.479+0000 7f69fe9ec700 10 monclient: _renew_subs 2021-05-25T12:07:33.532 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.479+0000 7f69fe9ec700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:07:33.533 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.479+0000 7f69fe9ec700 1 -- 172.21.15.102:0/3809855660 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f69f81112f0 con 0x7f69f8095c40 2021-05-25T12:07:33.533 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.479+0000 7f69fe9ec700 1 librados: init done 2021-05-25T12:07:33.533 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.480+0000 7f69e5ffb700 1 -- 172.21.15.102:0/3809855660 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f69ec00df90 con 0x7f69f8095c40 2021-05-25T12:07:33.533 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.480+0000 7f69e5ffb700 1 -- 172.21.15.102:0/3809855660 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(243..243 src has 1..243) v4 ==== 5568+0+0 (unknown 453324089 0 0) 0x7f69ec03d810 con 0x7f69f8095c40 2021-05-25T12:07:33.533 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.480+0000 7f69e5ffb700 10 client.234125.objecter ms_dispatch 0x7f69f8001550 osd_map(243..243 src has 1..243) v4 2021-05-25T12:07:33.534 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.480+0000 7f69e5ffb700 3 client.234125.objecter handle_osd_map got epochs [243,243] > 0 2021-05-25T12:07:33.534 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.480+0000 7f69e5ffb700 3 client.234125.objecter handle_osd_map decoding full epoch 243 2021-05-25T12:07:33.534 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.480+0000 7f69fe9ec700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:07:33.534 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.480+0000 7f69fe9ec700 1 -- 172.21.15.102:0/3809855660 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f69f8111f60 con 0x7f69f8133f70 2021-05-25T12:07:33.534 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.480+0000 7f69e5ffb700 20 client.234125.objecter dump_active .. 0 homeless 2021-05-25T12:07:33.535 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.480+0000 7f69fd1e9700 10 monclient: get_auth_request con 0x7f69d8033190 auth_method 0 2021-05-25T12:07:33.535 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.480+0000 7f69e5ffb700 10 client.234125.objecter ms_handle_connect 0x7f69f8133f70 2021-05-25T12:07:33.535 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.480+0000 7f69e5ffb700 10 client.234125.objecter resend_mon_ops 2021-05-25T12:07:33.535 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.480+0000 7f69e5ffb700 10 client.234125.objecter ms_handle_connect 0x7f69d8033190 2021-05-25T12:07:33.536 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.480+0000 7f69e5ffb700 1 -- 172.21.15.102:0/3809855660 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 121290314 0 0) 0x7f69f8111f60 con 0x7f69f8133f70 2021-05-25T12:07:33.536 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.480+0000 7f69e5ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:33.536 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.480+0000 7f69e5ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:33.536 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.480+0000 7f69e5ffb700 10 monclient(hunting): my global_id is 224254 2021-05-25T12:07:33.536 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.480+0000 7f69e5ffb700 1 -- 172.21.15.102:0/3809855660 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69f8128f60 con 0x7f69f8133f70 2021-05-25T12:07:33.537 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.481+0000 7f69e5ffb700 1 -- 172.21.15.102:0/3809855660 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2015831455 0 0) 0x7f69f8128f60 con 0x7f69f8133f70 2021-05-25T12:07:33.537 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.481+0000 7f69e5ffb700 1 -- 172.21.15.102:0/3809855660 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f69f8111f60 con 0x7f69f8133f70 2021-05-25T12:07:33.537 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.481+0000 7f69e5ffb700 1 -- 172.21.15.102:0/3809855660 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f69f4002280 con 0x7f69f8133f70 2021-05-25T12:07:33.537 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.481+0000 7f69e5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:33.537 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.481+0000 7f69e5ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T12:07:33.538 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.481+0000 7f69e5ffb700 10 monclient: dump: 2021-05-25T12:07:33.538 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:33.538 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:33.538 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:33.539 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:33.539 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:33.539 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:33.539 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:33.539 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:33.540 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:33.540 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:33.540 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.481+0000 7f69e5ffb700 1 -- 172.21.15.102:0/3809855660 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 552181819 0 0) 0x7f69f8111f60 con 0x7f69f8133f70 2021-05-25T12:07:33.540 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.481+0000 7f69e5ffb700 1 -- 172.21.15.102:0/3809855660 --> v1:172.21.15.181:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f69f8059730 con 0x7f69f8133f70 2021-05-25T12:07:33.540 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.481+0000 7f69e5ffb700 1 -- 172.21.15.102:0/3809855660 <== mon.1 v1:172.21.15.181:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7f69f4002700 con 0x7f69f8133f70 2021-05-25T12:07:33.541 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.481+0000 7f69e5ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:07:33.541 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.481+0000 7f69e5ffb700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:07:33.541 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.481+0000 7f69e5ffb700 1 -- 172.21.15.102:0/3809855660 >> v1:172.21.15.181:6789/0 conn(0x7f69f8133f70 legacy=0x7f69f8063e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:33.541 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.481+0000 7f69e5ffb700 1 -- 172.21.15.102:0/3809855660 >> v1:172.21.15.181:6789/0 conn(0x7f69f8133f70 legacy=0x7f69f8063e20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:07:33.541 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.488+0000 7f69fe9ec700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:07:33.542 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.488+0000 7f69fe9ec700 1 -- 172.21.15.102:0/3809855660 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f69f8069c40 con 0x7f69f8027ce0 2021-05-25T12:07:33.542 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.488+0000 7f69e5ffb700 10 client.234125.objecter ms_handle_connect 0x7f69f8027ce0 2021-05-25T12:07:33.542 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.488+0000 7f69e5ffb700 10 client.234125.objecter resend_mon_ops 2021-05-25T12:07:33.542 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.488+0000 7f69e5ffb700 1 -- 172.21.15.102:0/3809855660 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3727559811 0 0) 0x7f69f8069c40 con 0x7f69f8027ce0 2021-05-25T12:07:33.542 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.488+0000 7f69e5ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:33.543 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.488+0000 7f69e5ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:33.543 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.488+0000 7f69e5ffb700 10 monclient(hunting): my global_id is 224257 2021-05-25T12:07:33.543 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.488+0000 7f69e5ffb700 1 -- 172.21.15.102:0/3809855660 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69f8111f60 con 0x7f69f8027ce0 2021-05-25T12:07:33.543 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.489+0000 7f69e5ffb700 1 -- 172.21.15.102:0/3809855660 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1407802875 0 0) 0x7f69f8111f60 con 0x7f69f8027ce0 2021-05-25T12:07:33.543 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.489+0000 7f69e5ffb700 1 -- 172.21.15.102:0/3809855660 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f69f8069c40 con 0x7f69f8027ce0 2021-05-25T12:07:33.544 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.489+0000 7f69e5ffb700 1 -- 172.21.15.102:0/3809855660 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f69f4002d80 con 0x7f69f8027ce0 2021-05-25T12:07:33.544 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.489+0000 7f69e5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:33.544 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.489+0000 7f69e5ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T12:07:33.544 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.489+0000 7f69e5ffb700 10 monclient: dump: 2021-05-25T12:07:33.545 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:33.545 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:33.545 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:33.545 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:33.545 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:33.546 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:33.546 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:33.546 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:33.546 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:33.546 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:33.547 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.489+0000 7f69e5ffb700 1 -- 172.21.15.102:0/3809855660 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1283174816 0 0) 0x7f69f8069c40 con 0x7f69f8027ce0 2021-05-25T12:07:33.547 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.489+0000 7f69e5ffb700 1 -- 172.21.15.102:0/3809855660 --> v1:172.21.15.181:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f69f806c9a0 con 0x7f69f8027ce0 2021-05-25T12:07:33.547 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.489+0000 7f69e5ffb700 1 -- 172.21.15.102:0/3809855660 <== mon.1 v1:172.21.15.181:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3940 (unknown 0 0 164232421) 0x7f69f4003060 con 0x7f69f8027ce0 2021-05-25T12:07:33.547 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.489+0000 7f69e5ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:07:33.548 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.489+0000 7f69e5ffb700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:07:33.548 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.489+0000 7f69e5ffb700 1 -- 172.21.15.102:0/3809855660 >> v1:172.21.15.181:6789/0 conn(0x7f69f8027ce0 legacy=0x7f69f8064530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:33.548 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.489+0000 7f69e5ffb700 1 -- 172.21.15.102:0/3809855660 >> v1:172.21.15.181:6789/0 conn(0x7f69f8027ce0 legacy=0x7f69f8064530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:07:33.549 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:07:33.549 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:07:33.549 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:07:33.549 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "peon", 2021-05-25T12:07:33.550 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 64, 2021-05-25T12:07:33.550 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:07:33.550 INFO:teuthology.orchestra.run.smithi102.stdout: 0, 2021-05-25T12:07:33.550 INFO:teuthology.orchestra.run.smithi102.stdout: 1, 2021-05-25T12:07:33.550 INFO:teuthology.orchestra.run.smithi102.stdout: 2 2021-05-25T12:07:33.551 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:07:33.551 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 0, 2021-05-25T12:07:33.551 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:07:33.551 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:07:33.552 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:07:33.552 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:07:33.552 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:07:33.552 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:07:33.552 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:07:33.553 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:07:33.553 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:07:33.553 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:07:33.553 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:07:33.554 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:07:33.554 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:07:33.554 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:07:33.554 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:07:33.554 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:07:33.555 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:07:33.555 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:07:33.555 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:07:33.555 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:07:33.555 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:07:33.556 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:07:33.556 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:07:33.556 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:07:33.556 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:07:33.556 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:33.557 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:07:33.557 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:07:33.557 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:07:33.557 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:07:33.557 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:07:33.558 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:07:33.558 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:07:33.558 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:07:33.558 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:07:33.558 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:07:33.559 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:07:33.559 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:07:33.559 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:07:33.559 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:07:33.559 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:07:33.560 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:07:33.560 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:07:33.560 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:07:33.560 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:07:33.560 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:07:33.561 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:07:33.561 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:07:33.561 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:07:33.561 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:07:33.561 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:07:33.562 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:07:33.562 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:33.562 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:07:33.562 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:33.563 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:07:33.563 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:07:33.563 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:07:33.563 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:07:33.563 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:33.564 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:07:33.564 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:07:33.564 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:07:33.564 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:07:33.564 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:07:33.565 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:33.565 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:07:33.565 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:07:33.565 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:07:33.565 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:07:33.566 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:07:33.566 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:33.566 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:33.566 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:07:33.566 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:07:33.567 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:07:33.567 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:07:33.567 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:33.567 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:07:33.567 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:07:33.568 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:07:33.568 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:07:33.568 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:07:33.568 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:33.568 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:07:33.569 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:07:33.569 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:07:33.569 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:07:33.569 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:07:33.569 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:33.569 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:33.570 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:07:33.570 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:07:33.570 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:07:33.570 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:07:33.570 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:33.571 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:07:33.571 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:07:33.571 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:07:33.571 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:07:33.571 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:07:33.572 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:33.572 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:07:33.572 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:07:33.572 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:07:33.572 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:07:33.573 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:07:33.573 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:07:33.573 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:07:33.573 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:33.573 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:07:33.574 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:07:33.574 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:33.574 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:07:33.574 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:07:33.574 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:07:33.575 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:07:33.575 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:07:33.575 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:07:33.575 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:33.576 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:07:33.576 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:07:33.576 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 4 2021-05-25T12:07:33.576 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:07:33.576 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:07:33.577 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": [ 2021-05-25T12:07:33.577 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:33.577 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:07:33.577 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:07:33.577 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:07:33.578 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:07:33.578 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:07:33.578 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:33.578 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:07:33.578 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:07:33.580 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.489+0000 7f69fe9ec700 10 librados: watch_flush enter 2021-05-25T12:07:33.580 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.489+0000 7f69fe9ec700 10 librados: watch_flush exit 2021-05-25T12:07:33.581 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.489+0000 7f69fe9ec700 20 client.234125.objecter shutdown clearing up homeless session... 2021-05-25T12:07:33.581 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.489+0000 7f69fe9ec700 10 client.234125.objecter successfully canceled tick 2021-05-25T12:07:33.581 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.489+0000 7f69fe9ec700 1 -- 172.21.15.102:0/3809855660 >> v1:172.21.15.181:6812/29582 conn(0x7f69d8033190 legacy=0x7f69d8035650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:33.581 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.489+0000 7f69fe9ec700 10 monclient: shutdown 2021-05-25T12:07:33.581 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.489+0000 7f69fe9ec700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:07:33.582 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.489+0000 7f69fe9ec700 1 -- 172.21.15.102:0/3809855660 >> v1:172.21.15.102:6790/0 conn(0x7f69f8095c40 legacy=0x7f69f8128850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:33.582 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.489+0000 7f69fe1eb700 1 -- 172.21.15.102:0/3809855660 reap_dead start 2021-05-25T12:07:33.582 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.489+0000 7f69fe9ec700 1 -- 172.21.15.102:0/3809855660 shutdown_connections 2021-05-25T12:07:33.582 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.489+0000 7f69fe9ec700 1 -- 172.21.15.102:0/3809855660 >> 172.21.15.102:0/3809855660 conn(0x7f69f80974b0 msgr2=0x7f69f812d300 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:07:33.583 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.490+0000 7f69fe9ec700 1 -- 172.21.15.102:0/3809855660 shutdown_connections 2021-05-25T12:07:33.583 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.490+0000 7f69fe9ec700 1 -- 172.21.15.102:0/3809855660 wait complete. 2021-05-25T12:07:33.583 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.490+0000 7f69fe9ec700 1 librados: shutdown 2021-05-25T12:07:33.583 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2021-05-25T12:07:33.584 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2021-05-25T12:07:33.695 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.694+0000 7efc5c3ef700 10 monclient: get_monmap_and_config 2021-05-25T12:07:33.696 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.694+0000 7efc5c3ef700 10 monclient: build_initial_monmap 2021-05-25T12:07:33.696 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.694+0000 7efc5c3ef700 10 monclient: monmap: 2021-05-25T12:07:33.696 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:07:33.696 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:33.697 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:07:33.695470+0000 2021-05-25T12:07:33.697 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:07:33.695470+0000 2021-05-25T12:07:33.697 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:07:33.697 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:33.697 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:07:33.698 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:07:33.698 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:07:33.698 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:33.708 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.695+0000 7efc5c3ef700 1 Processor -- start 2021-05-25T12:07:33.708 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.695+0000 7efc5c3ef700 1 -- start start 2021-05-25T12:07:33.708 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.695+0000 7efc5c3ef700 10 monclient: init 2021-05-25T12:07:33.709 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.695+0000 7efc5c3ef700 10 monclient: _reopen_session rank -1 2021-05-25T12:07:33.709 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.695+0000 7efc5c3ef700 10 monclient: _add_conns ranks=[1,0,2] 2021-05-25T12:07:33.709 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.695+0000 7efc5c3ef700 10 monclient(hunting): picked mon.noname-c con 0x7efc54097fa0 addr v1:172.21.15.181:6789/0 2021-05-25T12:07:33.709 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.695+0000 7efc5c3ef700 10 monclient(hunting): picked mon.noname-a con 0x7efc54063b70 addr v1:172.21.15.102:6789/0 2021-05-25T12:07:33.710 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.695+0000 7efc5c3ef700 10 monclient(hunting): picked mon.noname-b con 0x7efc541325c0 addr v1:172.21.15.102:6790/0 2021-05-25T12:07:33.710 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.695+0000 7efc5c3ef700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc54096b20 con 0x7efc54063b70 2021-05-25T12:07:33.710 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.695+0000 7efc5c3ef700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc5409b460 con 0x7efc54097fa0 2021-05-25T12:07:33.710 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.695+0000 7efc5c3ef700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc540964c0 con 0x7efc541325c0 2021-05-25T12:07:33.710 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.695+0000 7efc5c3ef700 10 monclient(hunting): _renew_subs 2021-05-25T12:07:33.711 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.695+0000 7efc5c3ef700 10 monclient(hunting): authenticate will time out at 1254.292858s 2021-05-25T12:07:33.711 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.695+0000 7efc5998a700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7efc54063b70 0x7efc54064020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33838/0 (socket says 172.21.15.102:33838) 2021-05-25T12:07:33.711 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.695+0000 7efc5998a700 1 -- 172.21.15.102:0/3886345249 learned_addr learned my addr 172.21.15.102:0/3886345249 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:07:33.711 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.696+0000 7efc59189700 1 -- 172.21.15.102:0/3886345249 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2784428771 0 0) 0x7efc54096b20 con 0x7efc54063b70 2021-05-25T12:07:33.712 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.696+0000 7efc59189700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:33.712 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.696+0000 7efc59189700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:33.712 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.696+0000 7efc59189700 10 monclient(hunting): my global_id is 204393 2021-05-25T12:07:33.712 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.696+0000 7efc59189700 1 -- 172.21.15.102:0/3886345249 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc3c003270 con 0x7efc54063b70 2021-05-25T12:07:33.712 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.696+0000 7efc59189700 1 -- 172.21.15.102:0/3886345249 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 107176881 0 0) 0x7efc540964c0 con 0x7efc541325c0 2021-05-25T12:07:33.712 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.696+0000 7efc59189700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:33.713 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.696+0000 7efc59189700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:33.713 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.696+0000 7efc59189700 10 monclient(hunting): my global_id is 234128 2021-05-25T12:07:33.713 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.696+0000 7efc59189700 1 -- 172.21.15.102:0/3886345249 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc54096b20 con 0x7efc541325c0 2021-05-25T12:07:33.713 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.696+0000 7efc59189700 1 -- 172.21.15.102:0/3886345249 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 74191831 0 0) 0x7efc5409b460 con 0x7efc54097fa0 2021-05-25T12:07:33.713 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.696+0000 7efc59189700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:33.714 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.696+0000 7efc59189700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:33.714 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.696+0000 7efc59189700 10 monclient(hunting): my global_id is 224260 2021-05-25T12:07:33.714 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.696+0000 7efc59189700 1 -- 172.21.15.102:0/3886345249 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc540964c0 con 0x7efc54097fa0 2021-05-25T12:07:33.714 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.696+0000 7efc59189700 1 -- 172.21.15.102:0/3886345249 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2532471367 0 0) 0x7efc3c003270 con 0x7efc54063b70 2021-05-25T12:07:33.714 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.696+0000 7efc59189700 1 -- 172.21.15.102:0/3886345249 >> v1:172.21.15.102:6790/0 conn(0x7efc541325c0 legacy=0x7efc541349b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:33.714 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.696+0000 7efc59189700 1 -- 172.21.15.102:0/3886345249 >> v1:172.21.15.181:6789/0 conn(0x7efc54097fa0 legacy=0x7efc54063460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:33.715 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.696+0000 7efc59189700 10 monclient: _finish_hunting 0 2021-05-25T12:07:33.715 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.696+0000 7efc59189700 1 monclient: found mon.noname-a 2021-05-25T12:07:33.715 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.696+0000 7efc59189700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:07:33.715 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.696+0000 7efc59189700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T12:07:33.715 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.696+0000 7efc59189700 1 -- 172.21.15.102:0/3886345249 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc54096760 con 0x7efc54063b70 2021-05-25T12:07:33.716 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.696+0000 7efc59189700 10 monclient: _finish_auth 0 2021-05-25T12:07:33.716 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.696+0000 7efc59189700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:07:33.716 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.696+0000 7efc5c3ef700 5 monclient: authenticate success, global_id 204393 2021-05-25T12:07:33.716 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.696+0000 7efc59189700 1 -- 172.21.15.102:0/3886345249 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7efc48001610 con 0x7efc54063b70 2021-05-25T12:07:33.716 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.696+0000 7efc5c3ef700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:07:33.716 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.696+0000 7efc59189700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:33.717 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.696+0000 7efc59189700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:07:33.717 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.696+0000 7efc59189700 10 monclient: dump: 2021-05-25T12:07:33.717 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:33.717 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:33.717 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:33.718 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:33.718 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:33.718 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:33.718 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:33.718 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:33.719 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:33.719 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:33.719 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.696+0000 7efc5c3ef700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:07:33.719 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.696+0000 7efc59189700 1 -- 172.21.15.102:0/3886345249 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efc48001800 con 0x7efc54063b70 2021-05-25T12:07:33.719 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.696+0000 7efc59189700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:07:33.720 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.696+0000 7efc5c3ef700 10 monclient: get_monmap_and_config success 2021-05-25T12:07:33.720 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.696+0000 7efc59189700 1 -- 172.21.15.102:0/3886345249 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7efc48001b90 con 0x7efc54063b70 2021-05-25T12:07:33.720 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.696+0000 7efc59189700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:33.720 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.696+0000 7efc5c3ef700 4 set_mon_vals no callback set 2021-05-25T12:07:33.720 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.696+0000 7efc59189700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:07:33.721 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.696+0000 7efc59189700 10 monclient: dump: 2021-05-25T12:07:33.721 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:33.721 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:33.721 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:33.721 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:33.721 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:33.722 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:33.722 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:33.722 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:33.722 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:33.722 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:33.723 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.697+0000 7efc5c3ef700 10 monclient: shutdown 2021-05-25T12:07:33.723 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.697+0000 7efc5c3ef700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:07:33.723 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.697+0000 7efc5c3ef700 1 -- 172.21.15.102:0/3886345249 >> v1:172.21.15.102:6789/0 conn(0x7efc54063b70 legacy=0x7efc54064020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:33.723 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.697+0000 7efc5c3ef700 1 -- 172.21.15.102:0/3886345249 shutdown_connections 2021-05-25T12:07:33.723 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.697+0000 7efc5c3ef700 1 -- 172.21.15.102:0/3886345249 >> 172.21.15.102:0/3886345249 conn(0x7efc5409ef50 msgr2=0x7efc5409f360 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:07:33.723 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.697+0000 7efc5c3ef700 1 -- 172.21.15.102:0/3886345249 shutdown_connections 2021-05-25T12:07:33.724 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.697+0000 7efc5c3ef700 1 -- 172.21.15.102:0/3886345249 wait complete. 2021-05-25T12:07:33.724 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.697+0000 7efc5c3ef700 10 monclient: build_initial_monmap 2021-05-25T12:07:33.724 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.697+0000 7efc5c3ef700 10 monclient: monmap: 2021-05-25T12:07:33.724 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:07:33.724 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:07:33.725 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:07:33.725 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:07:33.725 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:07:33.725 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:07:33.725 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:07:33.726 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:07:33.726 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:07:33.726 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:33.726 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.697+0000 7efc5c3ef700 1 librados: starting msgr at 2021-05-25T12:07:33.726 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.697+0000 7efc5c3ef700 1 librados: starting objecter 2021-05-25T12:07:33.727 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.697+0000 7efc5c3ef700 1 Processor -- start 2021-05-25T12:07:33.727 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.697+0000 7efc5c3ef700 1 -- start start 2021-05-25T12:07:33.727 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.697+0000 7efc5c3ef700 1 librados: setting wanted keys 2021-05-25T12:07:33.727 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.697+0000 7efc5c3ef700 1 librados: calling monclient init 2021-05-25T12:07:33.727 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.697+0000 7efc5c3ef700 10 monclient: init 2021-05-25T12:07:33.727 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.698+0000 7efc5c3ef700 10 monclient: _reopen_session rank -1 2021-05-25T12:07:33.728 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.698+0000 7efc5c3ef700 10 monclient: _add_conns ranks=[2,0,1] 2021-05-25T12:07:33.728 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.698+0000 7efc5c3ef700 10 monclient(hunting): picked mon.noname-c con 0x7efc54063b70 addr v1:172.21.15.102:6790/0 2021-05-25T12:07:33.728 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.698+0000 7efc5c3ef700 10 monclient(hunting): picked mon.noname-a con 0x7efc54097fa0 addr v1:172.21.15.102:6789/0 2021-05-25T12:07:33.728 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.698+0000 7efc5c3ef700 10 monclient(hunting): picked mon.noname-b con 0x7efc541325c0 addr v1:172.21.15.181:6789/0 2021-05-25T12:07:33.728 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.698+0000 7efc5c3ef700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc5409ab70 con 0x7efc54097fa0 2021-05-25T12:07:33.729 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.698+0000 7efc5c3ef700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc54094ff0 con 0x7efc541325c0 2021-05-25T12:07:33.729 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.698+0000 7efc5c3ef700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc54132320 con 0x7efc54063b70 2021-05-25T12:07:33.729 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.698+0000 7efc5c3ef700 10 monclient(hunting): _renew_subs 2021-05-25T12:07:33.729 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.698+0000 7efc5c3ef700 10 monclient(hunting): authenticate will time out at 1254.295405s 2021-05-25T12:07:33.729 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.698+0000 7efc5a18b700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7efc54063b70 0x7efc5413b540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:56950/0 (socket says 172.21.15.102:56950) 2021-05-25T12:07:33.730 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.698+0000 7efc5998a700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7efc54097fa0 0x7efc54131c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33842/0 (socket says 172.21.15.102:33842) 2021-05-25T12:07:33.730 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.698+0000 7efc5a18b700 1 -- 172.21.15.102:0/4170620436 learned_addr learned my addr 172.21.15.102:0/4170620436 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:07:33.730 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.698+0000 7efc427fc700 10 client.?.objecter ms_handle_connect 0x7efc54097fa0 2021-05-25T12:07:33.730 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.698+0000 7efc427fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:07:33.730 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.698+0000 7efc427fc700 10 client.?.objecter ms_handle_connect 0x7efc54063b70 2021-05-25T12:07:33.731 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.698+0000 7efc427fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:07:33.731 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.698+0000 7efc427fc700 1 -- 172.21.15.102:0/4170620436 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1100898712 0 0) 0x7efc5409ab70 con 0x7efc54097fa0 2021-05-25T12:07:33.731 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.698+0000 7efc427fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:33.731 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.698+0000 7efc427fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:33.731 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.698+0000 7efc427fc700 10 monclient(hunting): my global_id is 204396 2021-05-25T12:07:33.731 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.698+0000 7efc427fc700 1 -- 172.21.15.102:0/4170620436 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc34004c10 con 0x7efc54097fa0 2021-05-25T12:07:33.732 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.698+0000 7efc427fc700 10 client.?.objecter ms_handle_connect 0x7efc541325c0 2021-05-25T12:07:33.732 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.698+0000 7efc427fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:07:33.732 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.698+0000 7efc427fc700 1 -- 172.21.15.102:0/4170620436 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 626366503 0 0) 0x7efc54132320 con 0x7efc54063b70 2021-05-25T12:07:33.732 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.698+0000 7efc427fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:33.732 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.698+0000 7efc427fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:33.733 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.698+0000 7efc427fc700 10 monclient(hunting): my global_id is 234131 2021-05-25T12:07:33.733 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.698+0000 7efc427fc700 1 -- 172.21.15.102:0/4170620436 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc5409ab70 con 0x7efc54063b70 2021-05-25T12:07:33.733 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.698+0000 7efc5a18b700 1 -- 172.21.15.102:0/4170620436 >> v1:172.21.15.102:6790/0 conn(0x7efc54063b70 legacy=0x7efc5413b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-05-25T12:07:33.733 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.698+0000 7efc5a18b700 1 -- 172.21.15.102:0/4170620436 >> v1:172.21.15.102:6790/0 conn(0x7efc54063b70 legacy=0x7efc5413b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-05-25T12:07:33.733 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.698+0000 7efc5a18b700 1 --1- 172.21.15.102:0/4170620436 >> v1:172.21.15.102:6790/0 conn(0x7efc54063b70 0x7efc5413b540 :-1 s=OPENED pgs=7504 cs=1 l=1).handle_message read tag failed 2021-05-25T12:07:33.734 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.698+0000 7efc5a18b700 1 --1- 172.21.15.102:0/4170620436 >> v1:172.21.15.102:6790/0 conn(0x7efc54063b70 0x7efc5413b540 :-1 s=OPENED pgs=7504 cs=1 l=1).fault on lossy channel, failing 2021-05-25T12:07:33.734 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.698+0000 7efc427fc700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.102:6790/0 2021-05-25T12:07:33.734 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.698+0000 7efc427fc700 1 -- 172.21.15.102:0/4170620436 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3896451828 0 0) 0x7efc54094ff0 con 0x7efc541325c0 2021-05-25T12:07:33.734 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.698+0000 7efc427fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:33.734 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.698+0000 7efc427fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:33.734 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.698+0000 7efc427fc700 10 monclient(hunting): my global_id is 224263 2021-05-25T12:07:33.735 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.698+0000 7efc427fc700 1 -- 172.21.15.102:0/4170620436 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc54132320 con 0x7efc541325c0 2021-05-25T12:07:33.735 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.698+0000 7efc427fc700 1 -- 172.21.15.102:0/4170620436 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1332241021 0 0) 0x7efc34004c10 con 0x7efc54097fa0 2021-05-25T12:07:33.735 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.698+0000 7efc427fc700 1 -- 172.21.15.102:0/4170620436 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efc54094ff0 con 0x7efc54097fa0 2021-05-25T12:07:33.735 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.698+0000 7efc427fc700 1 -- 172.21.15.102:0/4170620436 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7efc48001510 con 0x7efc54097fa0 2021-05-25T12:07:33.735 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.698+0000 7efc427fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:33.736 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.698+0000 7efc427fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:07:33.736 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.698+0000 7efc427fc700 10 monclient(hunting): dump: 2021-05-25T12:07:33.736 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:33.736 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:33.736 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:33.737 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:33.737 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:33.737 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:33.737 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:33.737 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:33.738 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:33.738 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:33.738 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.698+0000 7efc427fc700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:07:33.738 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.699+0000 7efc427fc700 1 -- 172.21.15.102:0/4170620436 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3028318702 0 0) 0x7efc54094ff0 con 0x7efc54097fa0 2021-05-25T12:07:33.738 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.699+0000 7efc427fc700 1 -- 172.21.15.102:0/4170620436 >> v1:172.21.15.102:6790/0 conn(0x7efc54063b70 legacy=0x7efc5413b540 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:07:33.738 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.699+0000 7efc427fc700 1 -- 172.21.15.102:0/4170620436 >> v1:172.21.15.181:6789/0 conn(0x7efc541325c0 legacy=0x7efc54143430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:33.739 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.699+0000 7efc427fc700 10 monclient: _finish_hunting 0 2021-05-25T12:07:33.739 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.699+0000 7efc427fc700 1 monclient: found mon.a 2021-05-25T12:07:33.739 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.699+0000 7efc427fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:07:33.739 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.699+0000 7efc427fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:07:33.739 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.699+0000 7efc427fc700 1 -- 172.21.15.102:0/4170620436 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc5411c380 con 0x7efc54097fa0 2021-05-25T12:07:33.740 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.699+0000 7efc427fc700 10 monclient: _finish_auth 0 2021-05-25T12:07:33.740 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.699+0000 7efc427fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:07:33.740 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.699+0000 7efc5c3ef700 5 monclient: authenticate success, global_id 204396 2021-05-25T12:07:33.740 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.699+0000 7efc5c3ef700 10 monclient: _renew_subs 2021-05-25T12:07:33.740 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.699+0000 7efc5c3ef700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:07:33.741 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.699+0000 7efc5c3ef700 1 -- 172.21.15.102:0/4170620436 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc5411b150 con 0x7efc54097fa0 2021-05-25T12:07:33.741 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.699+0000 7efc5c3ef700 10 client.204396.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:07:33.741 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.699+0000 7efc5c3ef700 10 monclient: _renew_subs 2021-05-25T12:07:33.741 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.699+0000 7efc5c3ef700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:07:33.741 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.699+0000 7efc5c3ef700 1 -- 172.21.15.102:0/4170620436 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efc54119b10 con 0x7efc54097fa0 2021-05-25T12:07:33.742 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.699+0000 7efc5c3ef700 1 librados: init done 2021-05-25T12:07:33.742 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.699+0000 7efc427fc700 1 -- 172.21.15.102:0/4170620436 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efc480047e0 con 0x7efc54097fa0 2021-05-25T12:07:33.742 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.699+0000 7efc427fc700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:07:33.742 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.699+0000 7efc427fc700 1 -- 172.21.15.102:0/4170620436 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7efc48004b70 con 0x7efc54097fa0 2021-05-25T12:07:33.742 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.699+0000 7efc427fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:33.742 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.699+0000 7efc427fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:07:33.743 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.699+0000 7efc58988700 4 set_mon_vals no callback set 2021-05-25T12:07:33.743 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.699+0000 7efc427fc700 10 monclient: dump: 2021-05-25T12:07:33.743 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:33.743 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:33.743 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:33.744 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:33.744 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:33.744 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:33.744 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:33.744 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:33.745 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:33.745 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:33.745 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.699+0000 7efc427fc700 1 -- 172.21.15.102:0/4170620436 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7efc48004d50 con 0x7efc54097fa0 2021-05-25T12:07:33.745 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.699+0000 7efc5c3ef700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:07:33.745 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.699+0000 7efc5c3ef700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:07:33.746 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.699+0000 7efc5c3ef700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:07:33.746 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.699+0000 7efc5c3ef700 1 -- 172.21.15.102:0/4170620436 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc38004db0 con 0x7efc54097fa0 2021-05-25T12:07:33.746 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.700+0000 7efc427fc700 1 -- 172.21.15.102:0/4170620436 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(244..244 src has 1..244) v4 ==== 5459+0+0 (unknown 3351300352 0 0) 0x7efc4803d6d0 con 0x7efc54097fa0 2021-05-25T12:07:33.746 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.700+0000 7efc427fc700 10 client.204396.objecter ms_dispatch 0x7efc54001550 osd_map(244..244 src has 1..244) v4 2021-05-25T12:07:33.746 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.700+0000 7efc427fc700 3 client.204396.objecter handle_osd_map got epochs [244,244] > 0 2021-05-25T12:07:33.747 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.700+0000 7efc427fc700 3 client.204396.objecter handle_osd_map decoding full epoch 244 2021-05-25T12:07:33.747 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.700+0000 7efc427fc700 20 client.204396.objecter dump_active .. 0 homeless 2021-05-25T12:07:33.747 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.700+0000 7efc5a18b700 10 monclient: get_auth_request con 0x7efc34033250 auth_method 0 2021-05-25T12:07:33.747 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.700+0000 7efc427fc700 10 client.204396.objecter ms_handle_connect 0x7efc34033250 2021-05-25T12:07:33.747 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.704+0000 7efc427fc700 1 -- 172.21.15.102:0/4170620436 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7efc48012f10 con 0x7efc54097fa0 2021-05-25T12:07:33.748 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.704+0000 7efc427fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:07:33.748 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.704+0000 7efc427fc700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:07:33.919 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.917+0000 7efc5c3ef700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2021-05-25T12:07:33.919 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.917+0000 7efc5c3ef700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2021-05-25T12:07:33.919 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.917+0000 7efc5c3ef700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:07:33.919 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.917+0000 7efc5c3ef700 1 -- 172.21.15.102:0/4170620436 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7efc38005870 con 0x7efc54097fa0 2021-05-25T12:07:33.920 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.918+0000 7efc427fc700 1 -- 172.21.15.102:0/4170620436 <== mon.0 v1:172.21.15.102:6789/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7efc48012ce0 con 0x7efc54097fa0 2021-05-25T12:07:33.921 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.918+0000 7efc427fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2021-05-25T12:07:33.921 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.918+0000 7efc427fc700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:07:33.922 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:07:33.919+0000 7fdd56bb4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:07:33.923 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:07:33.919+0000 7fdd56bb4700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=28,config=2,health=11,kv=5,logm=54} crc {auth=1853372904,config=3140514009,health=1116420093,kv=2716774850,logm=2381993882}) 2021-05-25T12:07:33.923 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:07:33.919+0000 7fdd56bb4700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=2,health=12,kv=5,logm=55} crc {auth=639597543,config=3140514009,health=506550936,kv=518131367,logm=2461849854}) 2021-05-25T12:07:33.923 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:07:33.919+0000 7fdd56bb4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:07:33.923 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:07:33.919+0000 7fdd56bb4700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=28,config=2,health=11,kv=5,logm=54} crc {auth=1853372904,config=3140514009,health=1116420093,kv=2716774850,logm=2381993882}) 2021-05-25T12:07:33.924 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:07:33.919+0000 7fdd56bb4700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=28,config=2,health=11,kv=5,logm=54} crc {auth=2215873390,config=3140514009,health=2595898560,kv=3785212936,logm=2350374804}) 2021-05-25T12:07:33.924 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:07:33.920+0000 7fdd56bb4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:07:33.924 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:07:33.920+0000 7fdd56bb4700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4093993555}) 2021-05-25T12:07:33.924 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:07:33.920+0000 7fdd56bb4700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3658223674}) 2021-05-25T12:07:33.924 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:07:33.920+0000 7fdd56bb4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:07:33.925 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:07:33.920+0000 7fdd56bb4700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4093993555}) 2021-05-25T12:07:33.925 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:07:33.920+0000 7fdd56bb4700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=981374277}) 2021-05-25T12:07:33.927 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.921+0000 7efc5c3ef700 10 librados: watch_flush enter 2021-05-25T12:07:33.927 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.921+0000 7efc5c3ef700 10 librados: watch_flush exit 2021-05-25T12:07:33.927 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.921+0000 7efc5c3ef700 20 client.204396.objecter shutdown clearing up homeless session... 2021-05-25T12:07:33.928 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.921+0000 7efc5c3ef700 10 client.204396.objecter successfully canceled tick 2021-05-25T12:07:33.928 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.921+0000 7efc5c3ef700 1 -- 172.21.15.102:0/4170620436 >> v1:172.21.15.181:6812/29582 conn(0x7efc34033250 legacy=0x7efc34035710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:33.928 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.921+0000 7efc5c3ef700 10 monclient: shutdown 2021-05-25T12:07:33.928 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.921+0000 7efc5c3ef700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:07:33.928 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.921+0000 7efc5c3ef700 1 -- 172.21.15.102:0/4170620436 >> v1:172.21.15.102:6789/0 conn(0x7efc54097fa0 legacy=0x7efc54131c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:33.929 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.921+0000 7efc5c3ef700 1 -- 172.21.15.102:0/4170620436 shutdown_connections 2021-05-25T12:07:33.929 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.921+0000 7efc5c3ef700 1 -- 172.21.15.102:0/4170620436 >> 172.21.15.102:0/4170620436 conn(0x7efc5409ef50 msgr2=0x7efc54096a00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:07:33.929 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.921+0000 7efc5c3ef700 1 -- 172.21.15.102:0/4170620436 shutdown_connections 2021-05-25T12:07:33.929 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.921+0000 7efc5c3ef700 1 -- 172.21.15.102:0/4170620436 wait complete. 2021-05-25T12:07:33.930 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:33.921+0000 7efc5c3ef700 1 librados: shutdown 2021-05-25T12:07:33.932 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:07:33.922+0000 7fdd56bb4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:07:33.932 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:07:33.922+0000 7fdd56bb4700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3025522388}) 2021-05-25T12:07:33.932 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:07:33.922+0000 7fdd56bb4700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3918316478}) 2021-05-25T12:07:33.932 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:07:33.922+0000 7fdd56bb4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:07:33.933 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:07:33.922+0000 7fdd56bb4700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3025522388}) 2021-05-25T12:07:33.933 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:07:33.922+0000 7fdd56bb4700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2822301883}) 2021-05-25T12:07:33.933 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:07:33.926+0000 7fdd56bb4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:07:33.933 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:07:33.926+0000 7fdd56bb4700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1367855527}) 2021-05-25T12:07:33.934 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:07:33.926+0000 7fdd56bb4700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=890198799}) 2021-05-25T12:07:33.934 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:07:33.926+0000 7fdd56bb4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:07:33.934 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:07:33.926+0000 7fdd56bb4700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1367855527}) 2021-05-25T12:07:33.934 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:07:33.926+0000 7fdd56bb4700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3917768455}) 2021-05-25T12:07:33.935 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:07:33.929+0000 7fdd56bb4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:07:33.935 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:07:33.929+0000 7fdd56bb4700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=99,mdsmap=1} crc {logm=2438042306,mdsmap=2826658659}) 2021-05-25T12:07:33.935 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:07:33.929+0000 7fdd56bb4700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2381328103}) 2021-05-25T12:07:33.935 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:07:33.929+0000 7fdd56bb4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:07:33.936 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:07:33.929+0000 7fdd56bb4700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=99,mdsmap=1} crc {logm=2438042306,mdsmap=2826658659}) 2021-05-25T12:07:33.936 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:07:33.929+0000 7fdd56bb4700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=97,mdsmap=3} crc {logm=3993647676,mdsmap=1531151339}) 2021-05-25T12:07:33.936 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:07:33.930+0000 7fdd56bb4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:07:33.936 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:07:33.930+0000 7fdd56bb4700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=4,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=29,osdmap=33} crc {mdsmap=3140514009,mgr=2996579194,mgr_command_descs=3824783157,mgr_metadata=1267308364,mgrstat=276363833,mon_config_key=4118394475,monmap=3809614708,osd_metadata=725478305,osd_pg_creating=1639835525,osd_snap=290684650,osdmap=2927684342}) 2021-05-25T12:07:33.936 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:07:33.930+0000 7fdd56bb4700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mdsmap=2,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=4,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=26,osdmap=38} crc {mdsmap=3140514009,mgr=3925590689,mgr_command_descs=3824783157,mgr_metadata=1267308364,mgrstat=276363833,mon_config_key=4118394475,monmap=1140339822,osd_metadata=725478305,osd_pg_creating=1639835525,osd_snap=1298132734,osdmap=2490139425}) 2021-05-25T12:07:33.937 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:07:33.930+0000 7fdd56bb4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:07:33.937 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:07:33.930+0000 7fdd56bb4700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=4,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=29,osdmap=33} crc {mdsmap=3140514009,mgr=2996579194,mgr_command_descs=3824783157,mgr_metadata=1267308364,mgrstat=276363833,mon_config_key=4118394475,monmap=3809614708,osd_metadata=725478305,osd_pg_creating=1639835525,osd_snap=290684650,osdmap=2927684342}) 2021-05-25T12:07:33.937 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:07:33.930+0000 7fdd56bb4700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mdsmap=2,mgr=5,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=3,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=30,osdmap=36} crc {mdsmap=3140514009,mgr=3103237869,mgr_command_descs=3824783157,mgr_metadata=1267308364,mgrstat=276363833,mon_config_key=2093747287,monmap=1140339822,osd_metadata=725478305,osd_pg_creating=1639835525,osd_snap=4215295085,osdmap=4216279139}) 2021-05-25T12:07:33.939 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:07:33.931+0000 7fdd56bb4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:07:33.939 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:07:33.931+0000 7fdd56bb4700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=645679556}) 2021-05-25T12:07:33.939 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:07:33.931+0000 7fdd56bb4700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3258403648}) 2021-05-25T12:07:33.939 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:07:33.931+0000 7fdd56bb4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:07:33.940 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:07:33.931+0000 7fdd56bb4700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=645679556}) 2021-05-25T12:07:33.940 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:07:33.931+0000 7fdd56bb4700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3257894326}) 2021-05-25T12:07:33.940 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:07:33.932+0000 7fdd56bb4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:07:33.940 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:07:33.932+0000 7fdd56bb4700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=4003734621}) 2021-05-25T12:07:33.941 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:07:33.932+0000 7fdd56bb4700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2630461111}) 2021-05-25T12:07:33.941 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:07:33.932+0000 7fdd56bb4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:07:33.941 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:07:33.932+0000 7fdd56bb4700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=4003734621}) 2021-05-25T12:07:33.941 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:07:33.932+0000 7fdd56bb4700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=299943602}) 2021-05-25T12:07:33.941 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:07:33.934+0000 7fdd56bb4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:07:33.942 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:07:33.934+0000 7fdd56bb4700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=798487862}) 2021-05-25T12:07:33.942 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:07:33.934+0000 7fdd56bb4700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2554184535}) 2021-05-25T12:07:33.942 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:07:33.934+0000 7fdd56bb4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:07:33.942 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:07:33.934+0000 7fdd56bb4700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=798487862}) 2021-05-25T12:07:33.942 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:07:33.934+0000 7fdd56bb4700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=900846188}) 2021-05-25T12:07:33.944 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2021-05-25T12:07:33.945 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:07:33.943+0000 7fdd56bb4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:07:33.945 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:07:33.943+0000 7fdd56bb4700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=38} crc {osdmap=2047741184}) 2021-05-25T12:07:33.945 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:07:33.943+0000 7fdd56bb4700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=38} crc {osdmap=617087001}) 2021-05-25T12:07:33.946 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:07:33.943+0000 7fdd56bb4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:07:33.946 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:07:33.943+0000 7fdd56bb4700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=38} crc {osdmap=2047741184}) 2021-05-25T12:07:33.946 INFO:tasks.ceph.mon.c.smithi102.stderr:2021-05-25T12:07:33.943+0000 7fdd56bb4700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=37} crc {osdmap=3023828194}) 2021-05-25T12:07:34.945 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2021-05-25T12:07:34.945 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2021-05-25T12:07:35.069 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.068+0000 7f7239f18700 10 monclient: get_monmap_and_config 2021-05-25T12:07:35.070 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.068+0000 7f7239f18700 10 monclient: build_initial_monmap 2021-05-25T12:07:35.070 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.068+0000 7f7239f18700 10 monclient: monmap: 2021-05-25T12:07:35.070 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:07:35.070 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:35.071 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:07:35.069426+0000 2021-05-25T12:07:35.071 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:07:35.069426+0000 2021-05-25T12:07:35.071 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:07:35.071 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:35.072 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:07:35.072 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:07:35.072 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:07:35.072 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:35.084 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.069+0000 7f7239f18700 1 Processor -- start 2021-05-25T12:07:35.084 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.069+0000 7f7239f18700 1 -- start start 2021-05-25T12:07:35.085 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.069+0000 7f7239f18700 10 monclient: init 2021-05-25T12:07:35.085 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.069+0000 7f7239f18700 10 monclient: _reopen_session rank -1 2021-05-25T12:07:35.085 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.069+0000 7f7239f18700 10 monclient: _add_conns ranks=[2,0,1] 2021-05-25T12:07:35.085 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.069+0000 7f7239f18700 10 monclient(hunting): picked mon.noname-b con 0x7f7234093fc0 addr v1:172.21.15.102:6790/0 2021-05-25T12:07:35.086 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.069+0000 7f7239f18700 10 monclient(hunting): picked mon.noname-a con 0x7f72340692d0 addr v1:172.21.15.102:6789/0 2021-05-25T12:07:35.086 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.069+0000 7f7239f18700 10 monclient(hunting): picked mon.noname-c con 0x7f723412f130 addr v1:172.21.15.181:6789/0 2021-05-25T12:07:35.086 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.069+0000 7f7239f18700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7234097160 con 0x7f72340692d0 2021-05-25T12:07:35.086 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.069+0000 7f7239f18700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7234092fe0 con 0x7f723412f130 2021-05-25T12:07:35.087 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.069+0000 7f7239f18700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7234093280 con 0x7f7234093fc0 2021-05-25T12:07:35.087 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.069+0000 7f7239f18700 10 monclient(hunting): _renew_subs 2021-05-25T12:07:35.087 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.069+0000 7f7239f18700 10 monclient(hunting): authenticate will time out at 1255.666949s 2021-05-25T12:07:35.087 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.070+0000 7f72337fe700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f7234093fc0 0x7f7234096340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:56958/0 (socket says 172.21.15.102:56958) 2021-05-25T12:07:35.087 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.070+0000 7f7232ffd700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f72340692d0 0x7f7234096a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33852/0 (socket says 172.21.15.102:33852) 2021-05-25T12:07:35.088 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.070+0000 7f72337fe700 1 -- 172.21.15.102:0/4147260466 learned_addr learned my addr 172.21.15.102:0/4147260466 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:07:35.088 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.070+0000 7f72327fc700 1 -- 172.21.15.102:0/4147260466 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3899735830 0 0) 0x7f7234097160 con 0x7f72340692d0 2021-05-25T12:07:35.088 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.070+0000 7f72327fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:35.088 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.070+0000 7f72327fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:35.089 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.070+0000 7f72327fc700 10 monclient(hunting): my global_id is 204405 2021-05-25T12:07:35.089 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.070+0000 7f72327fc700 1 -- 172.21.15.102:0/4147260466 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7218003270 con 0x7f72340692d0 2021-05-25T12:07:35.089 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.070+0000 7f72327fc700 1 -- 172.21.15.102:0/4147260466 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3825464762 0 0) 0x7f7234093280 con 0x7f7234093fc0 2021-05-25T12:07:35.089 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.070+0000 7f72327fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:35.089 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.070+0000 7f72327fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:35.090 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.070+0000 7f72327fc700 10 monclient(hunting): my global_id is 234140 2021-05-25T12:07:35.090 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.070+0000 7f72327fc700 1 -- 172.21.15.102:0/4147260466 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7234097160 con 0x7f7234093fc0 2021-05-25T12:07:35.090 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.070+0000 7f72327fc700 1 -- 172.21.15.102:0/4147260466 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1584722866 0 0) 0x7f7218003270 con 0x7f72340692d0 2021-05-25T12:07:35.091 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.070+0000 7f72327fc700 1 -- 172.21.15.102:0/4147260466 >> v1:172.21.15.102:6790/0 conn(0x7f7234093fc0 legacy=0x7f7234096340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:35.091 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.070+0000 7f72327fc700 1 -- 172.21.15.102:0/4147260466 >> v1:172.21.15.181:6789/0 conn(0x7f723412f130 legacy=0x7f7234131520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:35.091 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.070+0000 7f72327fc700 10 monclient: _finish_hunting 0 2021-05-25T12:07:35.091 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.070+0000 7f72327fc700 1 monclient: found mon.noname-a 2021-05-25T12:07:35.091 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.070+0000 7f72327fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:07:35.092 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.070+0000 7f72327fc700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T12:07:35.092 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.070+0000 7f72327fc700 1 -- 172.21.15.102:0/4147260466 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7234093520 con 0x7f72340692d0 2021-05-25T12:07:35.092 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.070+0000 7f72327fc700 10 monclient: _finish_auth 0 2021-05-25T12:07:35.092 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.070+0000 7f72327fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:07:35.093 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.070+0000 7f7239f18700 5 monclient: authenticate success, global_id 204405 2021-05-25T12:07:35.093 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.070+0000 7f72327fc700 1 -- 172.21.15.102:0/4147260466 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f7224001610 con 0x7f72340692d0 2021-05-25T12:07:35.093 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.070+0000 7f7239f18700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:07:35.093 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.070+0000 7f72327fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:35.093 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.070+0000 7f72327fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:07:35.094 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.070+0000 7f72327fc700 10 monclient: dump: 2021-05-25T12:07:35.094 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:35.094 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:35.095 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:35.095 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:35.095 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:35.095 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:35.096 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:35.096 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:35.096 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:35.096 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:35.096 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.070+0000 7f7239f18700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:07:35.097 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.070+0000 7f72327fc700 1 -- 172.21.15.102:0/4147260466 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7224001830 con 0x7f72340692d0 2021-05-25T12:07:35.097 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.070+0000 7f72327fc700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:07:35.097 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.070+0000 7f7239f18700 10 monclient: get_monmap_and_config success 2021-05-25T12:07:35.097 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.070+0000 7f72327fc700 1 -- 172.21.15.102:0/4147260466 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f7224001bc0 con 0x7f72340692d0 2021-05-25T12:07:35.098 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.070+0000 7f7239f18700 4 set_mon_vals no callback set 2021-05-25T12:07:35.098 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.070+0000 7f72327fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:35.098 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.070+0000 7f72327fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:07:35.098 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.070+0000 7f72327fc700 10 monclient: dump: 2021-05-25T12:07:35.099 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:35.099 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:35.099 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:35.099 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:35.099 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:35.100 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:35.100 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:35.100 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:35.100 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:35.100 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:35.101 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.071+0000 7f7239f18700 10 monclient: shutdown 2021-05-25T12:07:35.101 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.071+0000 7f7239f18700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:07:35.101 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.071+0000 7f7239f18700 1 -- 172.21.15.102:0/4147260466 >> v1:172.21.15.102:6789/0 conn(0x7f72340692d0 legacy=0x7f7234096a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:35.101 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.071+0000 7f7239f18700 1 -- 172.21.15.102:0/4147260466 shutdown_connections 2021-05-25T12:07:35.101 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.071+0000 7f7239f18700 1 -- 172.21.15.102:0/4147260466 >> 172.21.15.102:0/4147260466 conn(0x7f723409b370 msgr2=0x7f7234097430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:07:35.102 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.071+0000 7f7239f18700 1 -- 172.21.15.102:0/4147260466 shutdown_connections 2021-05-25T12:07:35.102 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.071+0000 7f7239f18700 1 -- 172.21.15.102:0/4147260466 wait complete. 2021-05-25T12:07:35.102 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.071+0000 7f7239f18700 10 monclient: build_initial_monmap 2021-05-25T12:07:35.103 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.071+0000 7f7239f18700 10 monclient: monmap: 2021-05-25T12:07:35.103 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:07:35.103 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:07:35.103 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:07:35.104 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:07:35.104 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:07:35.104 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:07:35.104 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:07:35.104 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:07:35.105 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:07:35.105 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:35.105 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.072+0000 7f7239f18700 1 librados: starting msgr at 2021-05-25T12:07:35.105 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.072+0000 7f7239f18700 1 librados: starting objecter 2021-05-25T12:07:35.105 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.072+0000 7f7239f18700 1 Processor -- start 2021-05-25T12:07:35.106 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.072+0000 7f7239f18700 1 -- start start 2021-05-25T12:07:35.106 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.072+0000 7f7239f18700 1 librados: setting wanted keys 2021-05-25T12:07:35.106 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.072+0000 7f7239f18700 1 librados: calling monclient init 2021-05-25T12:07:35.106 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.072+0000 7f7239f18700 10 monclient: init 2021-05-25T12:07:35.107 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.072+0000 7f7239f18700 10 monclient: _reopen_session rank -1 2021-05-25T12:07:35.107 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.072+0000 7f7239f18700 10 monclient: _add_conns ranks=[1,2,0] 2021-05-25T12:07:35.107 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.072+0000 7f7239f18700 10 monclient(hunting): picked mon.noname-b con 0x7f72340692d0 addr v1:172.21.15.181:6789/0 2021-05-25T12:07:35.107 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.072+0000 7f7239f18700 10 monclient(hunting): picked mon.noname-c con 0x7f7234093fc0 addr v1:172.21.15.102:6790/0 2021-05-25T12:07:35.107 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.072+0000 7f7239f18700 10 monclient(hunting): picked mon.noname-a con 0x7f723412f130 addr v1:172.21.15.102:6789/0 2021-05-25T12:07:35.108 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.072+0000 7f7239f18700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7234116300 con 0x7f723412f130 2021-05-25T12:07:35.108 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.072+0000 7f7239f18700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7234127f60 con 0x7f72340692d0 2021-05-25T12:07:35.108 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.072+0000 7f7239f18700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7234092350 con 0x7f7234093fc0 2021-05-25T12:07:35.108 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.072+0000 7f7239f18700 10 monclient(hunting): _renew_subs 2021-05-25T12:07:35.109 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.072+0000 7f7239f18700 10 monclient(hunting): authenticate will time out at 1255.669805s 2021-05-25T12:07:35.109 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.072+0000 7f7232ffd700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f7234093fc0 0x7f7234112dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:56966/0 (socket says 172.21.15.102:56966) 2021-05-25T12:07:35.109 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.072+0000 7f7233fff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f723412f130 0x7f7234143620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33858/0 (socket says 172.21.15.102:33858) 2021-05-25T12:07:35.109 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.072+0000 7f7232ffd700 1 -- 172.21.15.102:0/750365997 learned_addr learned my addr 172.21.15.102:0/750365997 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:07:35.109 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.073+0000 7f721f7fe700 10 client.?.objecter ms_handle_connect 0x7f723412f130 2021-05-25T12:07:35.110 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.073+0000 7f721f7fe700 10 client.?.objecter resend_mon_ops 2021-05-25T12:07:35.110 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.073+0000 7f721f7fe700 10 client.?.objecter ms_handle_connect 0x7f7234093fc0 2021-05-25T12:07:35.110 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.073+0000 7f721f7fe700 10 client.?.objecter resend_mon_ops 2021-05-25T12:07:35.110 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.073+0000 7f721f7fe700 10 client.?.objecter ms_handle_connect 0x7f72340692d0 2021-05-25T12:07:35.111 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.073+0000 7f721f7fe700 10 client.?.objecter resend_mon_ops 2021-05-25T12:07:35.111 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.073+0000 7f721f7fe700 1 -- 172.21.15.102:0/750365997 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3063750103 0 0) 0x7f7234116300 con 0x7f723412f130 2021-05-25T12:07:35.111 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.073+0000 7f721f7fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:35.112 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.073+0000 7f721f7fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:35.112 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.073+0000 7f721f7fe700 10 monclient(hunting): my global_id is 204408 2021-05-25T12:07:35.112 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.073+0000 7f721f7fe700 1 -- 172.21.15.102:0/750365997 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7214004c10 con 0x7f723412f130 2021-05-25T12:07:35.113 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.073+0000 7f721f7fe700 1 -- 172.21.15.102:0/750365997 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1058386500 0 0) 0x7f7234092350 con 0x7f7234093fc0 2021-05-25T12:07:35.113 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.073+0000 7f721f7fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:35.113 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.073+0000 7f721f7fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:35.113 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.073+0000 7f721f7fe700 10 monclient(hunting): my global_id is 234143 2021-05-25T12:07:35.113 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.073+0000 7f721f7fe700 1 -- 172.21.15.102:0/750365997 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7234116300 con 0x7f7234093fc0 2021-05-25T12:07:35.114 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.073+0000 7f721f7fe700 1 -- 172.21.15.102:0/750365997 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1438040571 0 0) 0x7f7234127f60 con 0x7f72340692d0 2021-05-25T12:07:35.114 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.073+0000 7f721f7fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:35.114 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.073+0000 7f721f7fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:35.114 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.073+0000 7f721f7fe700 10 monclient(hunting): my global_id is 224275 2021-05-25T12:07:35.114 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.073+0000 7f721f7fe700 1 -- 172.21.15.102:0/750365997 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7234092350 con 0x7f72340692d0 2021-05-25T12:07:35.115 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.073+0000 7f721f7fe700 1 -- 172.21.15.102:0/750365997 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3433094553 0 0) 0x7f7214004c10 con 0x7f723412f130 2021-05-25T12:07:35.115 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.073+0000 7f721f7fe700 1 -- 172.21.15.102:0/750365997 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7234127f60 con 0x7f723412f130 2021-05-25T12:07:35.115 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.073+0000 7f721f7fe700 1 -- 172.21.15.102:0/750365997 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2311301472 0 0) 0x7f7234116300 con 0x7f7234093fc0 2021-05-25T12:07:35.115 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.073+0000 7f721f7fe700 1 -- 172.21.15.102:0/750365997 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7214004c10 con 0x7f7234093fc0 2021-05-25T12:07:35.115 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.073+0000 7f721f7fe700 1 -- 172.21.15.102:0/750365997 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f7228001bc0 con 0x7f723412f130 2021-05-25T12:07:35.116 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.073+0000 7f721f7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:35.116 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.073+0000 7f721f7fe700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:07:35.116 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.073+0000 7f721f7fe700 10 monclient(hunting): dump: 2021-05-25T12:07:35.116 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:35.117 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:35.117 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:35.117 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:35.117 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:35.117 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:35.118 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:35.118 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:35.118 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:35.118 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:35.118 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.073+0000 7f721f7fe700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:07:35.119 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.073+0000 7f721f7fe700 1 -- 172.21.15.102:0/750365997 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f72240015d0 con 0x7f7234093fc0 2021-05-25T12:07:35.119 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.073+0000 7f721f7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:35.119 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.073+0000 7f721f7fe700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2021-05-25T12:07:35.119 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.073+0000 7f721f7fe700 10 monclient(hunting): dump: 2021-05-25T12:07:35.120 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:35.120 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:35.120 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:35.120 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:35.120 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:35.121 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:35.121 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:35.121 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:35.121 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:35.121 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:35.122 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.073+0000 7f721f7fe700 1 -- 172.21.15.102:0/750365997 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1464522721 0 0) 0x7f7234092350 con 0x7f72340692d0 2021-05-25T12:07:35.122 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.073+0000 7f721f7fe700 1 -- 172.21.15.102:0/750365997 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7234116300 con 0x7f72340692d0 2021-05-25T12:07:35.122 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.073+0000 7f721f7fe700 1 -- 172.21.15.102:0/750365997 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f7220001a70 con 0x7f72340692d0 2021-05-25T12:07:35.122 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.073+0000 7f721f7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:35.123 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.073+0000 7f721f7fe700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2021-05-25T12:07:35.123 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.073+0000 7f721f7fe700 10 monclient(hunting): dump: 2021-05-25T12:07:35.123 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:35.123 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:35.123 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:35.124 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:35.124 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:35.124 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:35.124 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:35.124 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:35.125 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:35.125 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:35.125 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.073+0000 7f721f7fe700 1 -- 172.21.15.102:0/750365997 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3548828451 0 0) 0x7f7214004c10 con 0x7f7234093fc0 2021-05-25T12:07:35.125 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.073+0000 7f721f7fe700 1 -- 172.21.15.102:0/750365997 >> v1:172.21.15.181:6789/0 conn(0x7f72340692d0 legacy=0x7f72341126c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:35.126 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.073+0000 7f721f7fe700 1 -- 172.21.15.102:0/750365997 >> v1:172.21.15.102:6789/0 conn(0x7f723412f130 legacy=0x7f7234143620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:35.126 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.073+0000 7f721f7fe700 10 monclient: _finish_hunting 0 2021-05-25T12:07:35.126 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.073+0000 7f721f7fe700 1 monclient: found mon.c 2021-05-25T12:07:35.126 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.073+0000 7f721f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:07:35.126 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.073+0000 7f721f7fe700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:07:35.127 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.073+0000 7f721f7fe700 1 -- 172.21.15.102:0/750365997 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f723412a330 con 0x7f7234093fc0 2021-05-25T12:07:35.127 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.073+0000 7f721f7fe700 10 monclient: _finish_auth 0 2021-05-25T12:07:35.127 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.073+0000 7f721f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:07:35.127 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.073+0000 7f7239f18700 5 monclient: authenticate success, global_id 234143 2021-05-25T12:07:35.128 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.073+0000 7f7239f18700 10 monclient: _renew_subs 2021-05-25T12:07:35.128 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.073+0000 7f7239f18700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:07:35.128 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.073+0000 7f7239f18700 1 -- 172.21.15.102:0/750365997 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72341166c0 con 0x7f7234093fc0 2021-05-25T12:07:35.128 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.073+0000 7f7239f18700 10 client.234143.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:07:35.128 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.073+0000 7f7239f18700 10 monclient: _renew_subs 2021-05-25T12:07:35.129 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.073+0000 7f7239f18700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:07:35.129 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.073+0000 7f7239f18700 1 -- 172.21.15.102:0/750365997 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7234128df0 con 0x7f7234093fc0 2021-05-25T12:07:35.129 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.073+0000 7f7239f18700 1 librados: init done 2021-05-25T12:07:35.129 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.073+0000 7f721f7fe700 1 -- 172.21.15.102:0/750365997 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f72240011e0 con 0x7f7234093fc0 2021-05-25T12:07:35.129 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.073+0000 7f721f7fe700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:07:35.130 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.073+0000 7f721f7fe700 1 -- 172.21.15.102:0/750365997 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f7224004a80 con 0x7f7234093fc0 2021-05-25T12:07:35.130 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.073+0000 7f721f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:35.130 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.073+0000 7f721f7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:07:35.130 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.073+0000 7f7231ffb700 4 set_mon_vals no callback set 2021-05-25T12:07:35.131 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.073+0000 7f721f7fe700 10 monclient: dump: 2021-05-25T12:07:35.131 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:35.131 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:35.131 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:35.131 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:35.132 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:35.132 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:35.132 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:35.132 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:35.132 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:35.133 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:35.133 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.074+0000 7f721f7fe700 1 -- 172.21.15.102:0/750365997 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f7224004c60 con 0x7f7234093fc0 2021-05-25T12:07:35.133 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.074+0000 7f7239f18700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:07:35.133 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.074+0000 7f7239f18700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:07:35.133 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.074+0000 7f7239f18700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:07:35.134 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.074+0000 7f7239f18700 1 -- 172.21.15.102:0/750365997 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f720c004db0 con 0x7f7234093fc0 2021-05-25T12:07:35.134 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.074+0000 7f721f7fe700 1 -- 172.21.15.102:0/750365997 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(245..245 src has 1..245) v4 ==== 5749+0+0 (unknown 3061073547 0 0) 0x7f722403d7f0 con 0x7f7234093fc0 2021-05-25T12:07:35.134 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.074+0000 7f721f7fe700 10 client.234143.objecter ms_dispatch 0x7f7234001550 osd_map(245..245 src has 1..245) v4 2021-05-25T12:07:35.134 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.074+0000 7f721f7fe700 3 client.234143.objecter handle_osd_map got epochs [245,245] > 0 2021-05-25T12:07:35.135 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.074+0000 7f721f7fe700 3 client.234143.objecter handle_osd_map decoding full epoch 245 2021-05-25T12:07:35.135 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.074+0000 7f721f7fe700 20 client.234143.objecter dump_active .. 0 homeless 2021-05-25T12:07:35.135 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.074+0000 7f72337fe700 10 monclient: get_auth_request con 0x7f7214033560 auth_method 0 2021-05-25T12:07:35.135 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.075+0000 7f721f7fe700 10 client.234143.objecter ms_handle_connect 0x7f7214033560 2021-05-25T12:07:35.135 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.079+0000 7f721f7fe700 1 -- 172.21.15.102:0/750365997 <== mon.2 v1:172.21.15.102:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7f72240062a0 con 0x7f7234093fc0 2021-05-25T12:07:35.136 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.079+0000 7f721f7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:07:35.136 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.079+0000 7f721f7fe700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:07:35.289 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.287+0000 7f7239f18700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2021-05-25T12:07:35.289 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.287+0000 7f7239f18700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2021-05-25T12:07:35.290 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.287+0000 7f7239f18700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:07:35.290 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.287+0000 7f7239f18700 1 -- 172.21.15.102:0/750365997 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f720c005060 con 0x7f7234093fc0 2021-05-25T12:07:35.290 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.288+0000 7f721f7fe700 1 -- 172.21.15.102:0/750365997 <== mon.2 v1:172.21.15.102:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1342 (unknown 254746657 0 942516091) 0x7f7224012bb0 con 0x7f7234093fc0 2021-05-25T12:07:35.290 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.288+0000 7f721f7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2021-05-25T12:07:35.291 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.288+0000 7f721f7fe700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:07:35.291 INFO:teuthology.orchestra.run.smithi102.stdout:{"election_epoch":64,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":2,"features":{"quorum_con":"4540138303579357183","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"20ee91b6-8935-4b98-b868-85c03789ccc4","modified":"2021-05-25T11:59:11.011522Z","created":"2021-05-25T11:59:11.011522Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6790","nonce":0}]},"addr":"172.21.15.102:6790/0","public_addr":"172.21.15.102:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2021-05-25T12:07:35.293 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.289+0000 7f7239f18700 10 librados: watch_flush enter 2021-05-25T12:07:35.293 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.289+0000 7f7239f18700 10 librados: watch_flush exit 2021-05-25T12:07:35.294 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.289+0000 7f7239f18700 20 client.234143.objecter shutdown clearing up homeless session... 2021-05-25T12:07:35.294 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.289+0000 7f7239f18700 10 client.234143.objecter successfully canceled tick 2021-05-25T12:07:35.294 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.289+0000 7f7239f18700 1 -- 172.21.15.102:0/750365997 >> v1:172.21.15.181:6812/29582 conn(0x7f7214033560 legacy=0x7f7214035a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:35.295 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.289+0000 7f7239f18700 10 monclient: shutdown 2021-05-25T12:07:35.295 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.289+0000 7f7239f18700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:07:35.295 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.289+0000 7f7239f18700 1 -- 172.21.15.102:0/750365997 >> v1:172.21.15.102:6790/0 conn(0x7f7234093fc0 legacy=0x7f7234112dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:35.295 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.289+0000 7f7239f18700 1 -- 172.21.15.102:0/750365997 shutdown_connections 2021-05-25T12:07:35.295 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.289+0000 7f7239f18700 1 -- 172.21.15.102:0/750365997 >> 172.21.15.102:0/750365997 conn(0x7f723409b370 msgr2=0x7f72340988b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:07:35.296 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.289+0000 7f7239f18700 1 -- 172.21.15.102:0/750365997 shutdown_connections 2021-05-25T12:07:35.296 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.289+0000 7f7239f18700 1 -- 172.21.15.102:0/750365997 wait complete. 2021-05-25T12:07:35.296 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.289+0000 7f7239f18700 1 librados: shutdown 2021-05-25T12:07:35.302 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2021-05-25T12:07:35.302 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2021-05-25T12:07:35.302 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2021-05-25T12:07:35.411 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.410+0000 7f9bdd2c5700 10 monclient: get_monmap_and_config 2021-05-25T12:07:35.412 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.410+0000 7f9bdd2c5700 10 monclient: build_initial_monmap 2021-05-25T12:07:35.412 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.410+0000 7f9bdd2c5700 10 monclient: monmap: 2021-05-25T12:07:35.412 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:07:35.413 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:35.413 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:07:35.411408+0000 2021-05-25T12:07:35.413 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:07:35.411408+0000 2021-05-25T12:07:35.413 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:07:35.413 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:35.414 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:07:35.414 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:07:35.414 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:07:35.414 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:35.438 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.411+0000 7f9bdd2c5700 1 Processor -- start 2021-05-25T12:07:35.439 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.411+0000 7f9bdd2c5700 1 -- start start 2021-05-25T12:07:35.439 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.411+0000 7f9bdd2c5700 10 monclient: init 2021-05-25T12:07:35.439 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.411+0000 7f9bdd2c5700 10 monclient: _reopen_session rank -1 2021-05-25T12:07:35.440 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.411+0000 7f9bdd2c5700 10 monclient: _add_conns ranks=[2,0,1] 2021-05-25T12:07:35.440 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.411+0000 7f9bdd2c5700 10 monclient(hunting): picked mon.noname-b con 0x7f9bd8096d50 addr v1:172.21.15.102:6790/0 2021-05-25T12:07:35.441 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.411+0000 7f9bdd2c5700 10 monclient(hunting): picked mon.noname-a con 0x7f9bd812b180 addr v1:172.21.15.102:6789/0 2021-05-25T12:07:35.441 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.411+0000 7f9bdd2c5700 10 monclient(hunting): picked mon.noname-c con 0x7f9bd812ec90 addr v1:172.21.15.181:6789/0 2021-05-25T12:07:35.441 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.411+0000 7f9bdd2c5700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bd8093230 con 0x7f9bd812b180 2021-05-25T12:07:35.442 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.411+0000 7f9bdd2c5700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bd80934d0 con 0x7f9bd812ec90 2021-05-25T12:07:35.442 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.411+0000 7f9bdd2c5700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bd8093770 con 0x7f9bd8096d50 2021-05-25T12:07:35.443 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.411+0000 7f9bdd2c5700 10 monclient(hunting): _renew_subs 2021-05-25T12:07:35.443 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.411+0000 7f9bdd2c5700 10 monclient(hunting): authenticate will time out at 1256.008749s 2021-05-25T12:07:35.443 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.411+0000 7f9bd7fff700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f9bd8096d50 0x7f9bd80924d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:56972/0 (socket says 172.21.15.102:56972) 2021-05-25T12:07:35.444 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.411+0000 7f9bd77fe700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f9bd812b180 0x7f9bd812d570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33866/0 (socket says 172.21.15.102:33866) 2021-05-25T12:07:35.444 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.411+0000 7f9bd7fff700 1 -- 172.21.15.102:0/2145376528 learned_addr learned my addr 172.21.15.102:0/2145376528 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:07:35.444 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.412+0000 7f9bd6ffd700 1 -- 172.21.15.102:0/2145376528 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3582069142 0 0) 0x7f9bd8093770 con 0x7f9bd8096d50 2021-05-25T12:07:35.444 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.412+0000 7f9bd6ffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:35.445 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.412+0000 7f9bd6ffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:35.445 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.412+0000 7f9bd6ffd700 10 monclient(hunting): my global_id is 234146 2021-05-25T12:07:35.445 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.412+0000 7f9bd6ffd700 1 -- 172.21.15.102:0/2145376528 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bbc003270 con 0x7f9bd8096d50 2021-05-25T12:07:35.445 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.412+0000 7f9bd6ffd700 1 -- 172.21.15.102:0/2145376528 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3413230407 0 0) 0x7f9bd8093230 con 0x7f9bd812b180 2021-05-25T12:07:35.445 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.412+0000 7f9bd6ffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:35.446 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.412+0000 7f9bd6ffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:35.446 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.412+0000 7f9bd6ffd700 10 monclient(hunting): my global_id is 204411 2021-05-25T12:07:35.446 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.412+0000 7f9bd6ffd700 1 -- 172.21.15.102:0/2145376528 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bd8093770 con 0x7f9bd812b180 2021-05-25T12:07:35.446 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.412+0000 7f9bd6ffd700 1 -- 172.21.15.102:0/2145376528 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2791781091 0 0) 0x7f9bd80934d0 con 0x7f9bd812ec90 2021-05-25T12:07:35.446 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.412+0000 7f9bd6ffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:35.447 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.412+0000 7f9bd6ffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:35.447 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.412+0000 7f9bd6ffd700 10 monclient(hunting): my global_id is 224278 2021-05-25T12:07:35.447 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.412+0000 7f9bd6ffd700 1 -- 172.21.15.102:0/2145376528 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bd8093230 con 0x7f9bd812ec90 2021-05-25T12:07:35.447 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.412+0000 7f9bd6ffd700 1 -- 172.21.15.102:0/2145376528 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1136963841 0 0) 0x7f9bbc003270 con 0x7f9bd8096d50 2021-05-25T12:07:35.448 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.412+0000 7f9bd6ffd700 1 -- 172.21.15.102:0/2145376528 >> v1:172.21.15.181:6789/0 conn(0x7f9bd812ec90 legacy=0x7f9bd8131080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:35.448 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.412+0000 7f9bd6ffd700 1 -- 172.21.15.102:0/2145376528 >> v1:172.21.15.102:6789/0 conn(0x7f9bd812b180 legacy=0x7f9bd812d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:35.448 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.412+0000 7f9bd6ffd700 10 monclient: _finish_hunting 0 2021-05-25T12:07:35.448 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.412+0000 7f9bd6ffd700 1 monclient: found mon.noname-b 2021-05-25T12:07:35.448 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.412+0000 7f9bd6ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:07:35.449 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.412+0000 7f9bd6ffd700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T12:07:35.449 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.412+0000 7f9bd6ffd700 1 -- 172.21.15.102:0/2145376528 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bd8092cd0 con 0x7f9bd8096d50 2021-05-25T12:07:35.449 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.412+0000 7f9bd6ffd700 10 monclient: _finish_auth 0 2021-05-25T12:07:35.449 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.412+0000 7f9bd6ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:07:35.449 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.412+0000 7f9bd6ffd700 1 -- 172.21.15.102:0/2145376528 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f9bc4001540 con 0x7f9bd8096d50 2021-05-25T12:07:35.450 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.412+0000 7f9bdd2c5700 5 monclient: authenticate success, global_id 234146 2021-05-25T12:07:35.450 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.412+0000 7f9bdd2c5700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:07:35.450 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.412+0000 7f9bd6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:35.450 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.412+0000 7f9bd6ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:07:35.451 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.412+0000 7f9bd6ffd700 10 monclient: dump: 2021-05-25T12:07:35.451 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:35.451 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:35.451 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:35.451 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:35.452 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:35.452 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:35.452 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:35.452 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:35.452 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:35.453 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:35.453 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.412+0000 7f9bdd2c5700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:07:35.453 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.412+0000 7f9bd6ffd700 1 -- 172.21.15.102:0/2145376528 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9bc4001760 con 0x7f9bd8096d50 2021-05-25T12:07:35.453 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.412+0000 7f9bd6ffd700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:07:35.454 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.412+0000 7f9bd6ffd700 1 -- 172.21.15.102:0/2145376528 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f9bc4001af0 con 0x7f9bd8096d50 2021-05-25T12:07:35.454 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.412+0000 7f9bdd2c5700 10 monclient: get_monmap_and_config success 2021-05-25T12:07:35.454 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.412+0000 7f9bdd2c5700 4 set_mon_vals no callback set 2021-05-25T12:07:35.454 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.412+0000 7f9bd6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:35.455 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.412+0000 7f9bd6ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:07:35.455 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.412+0000 7f9bd6ffd700 10 monclient: dump: 2021-05-25T12:07:35.455 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:35.455 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:35.455 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:35.456 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:35.456 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:35.456 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:35.456 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:35.457 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:35.457 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:35.457 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:35.457 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.412+0000 7f9bdd2c5700 10 monclient: shutdown 2021-05-25T12:07:35.457 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.412+0000 7f9bdd2c5700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:07:35.458 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.412+0000 7f9bdd2c5700 1 -- 172.21.15.102:0/2145376528 >> v1:172.21.15.102:6790/0 conn(0x7f9bd8096d50 legacy=0x7f9bd80924d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:35.458 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.412+0000 7f9bdd2c5700 1 -- 172.21.15.102:0/2145376528 shutdown_connections 2021-05-25T12:07:35.458 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.412+0000 7f9bdd2c5700 1 -- 172.21.15.102:0/2145376528 >> 172.21.15.102:0/2145376528 conn(0x7f9bd80974b0 msgr2=0x7f9bd80978c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:07:35.458 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.412+0000 7f9bdd2c5700 1 -- 172.21.15.102:0/2145376528 shutdown_connections 2021-05-25T12:07:35.459 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.413+0000 7f9bdd2c5700 1 -- 172.21.15.102:0/2145376528 wait complete. 2021-05-25T12:07:35.459 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.413+0000 7f9bdd2c5700 10 monclient: build_initial_monmap 2021-05-25T12:07:35.459 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.413+0000 7f9bdd2c5700 10 monclient: monmap: 2021-05-25T12:07:35.459 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:07:35.459 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:07:35.460 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:07:35.460 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:07:35.460 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:07:35.460 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:07:35.460 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:07:35.461 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:07:35.461 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:07:35.461 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:35.461 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.413+0000 7f9bdd2c5700 1 librados: starting msgr at 2021-05-25T12:07:35.461 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.413+0000 7f9bdd2c5700 1 librados: starting objecter 2021-05-25T12:07:35.462 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.413+0000 7f9bdd2c5700 1 Processor -- start 2021-05-25T12:07:35.462 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.413+0000 7f9bdd2c5700 1 -- start start 2021-05-25T12:07:35.462 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.413+0000 7f9bdd2c5700 1 librados: setting wanted keys 2021-05-25T12:07:35.462 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.413+0000 7f9bdd2c5700 1 librados: calling monclient init 2021-05-25T12:07:35.463 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.413+0000 7f9bdd2c5700 10 monclient: init 2021-05-25T12:07:35.463 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.414+0000 7f9bdd2c5700 10 monclient: _reopen_session rank -1 2021-05-25T12:07:35.463 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.414+0000 7f9bdd2c5700 10 monclient: _add_conns ranks=[0,1,2] 2021-05-25T12:07:35.463 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.414+0000 7f9bdd2c5700 10 monclient(hunting): picked mon.noname-a con 0x7f9bd8096d50 addr v1:172.21.15.102:6789/0 2021-05-25T12:07:35.463 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.414+0000 7f9bdd2c5700 10 monclient(hunting): picked mon.noname-b con 0x7f9bd812b180 addr v1:172.21.15.181:6789/0 2021-05-25T12:07:35.464 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.414+0000 7f9bdd2c5700 10 monclient(hunting): picked mon.noname-c con 0x7f9bd812ec90 addr v1:172.21.15.102:6790/0 2021-05-25T12:07:35.464 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.414+0000 7f9bdd2c5700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bd8093e20 con 0x7f9bd8096d50 2021-05-25T12:07:35.464 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.414+0000 7f9bdd2c5700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bd8117790 con 0x7f9bd812b180 2021-05-25T12:07:35.464 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.414+0000 7f9bdd2c5700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bd8117a30 con 0x7f9bd812ec90 2021-05-25T12:07:35.464 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.414+0000 7f9bdd2c5700 10 monclient(hunting): _renew_subs 2021-05-25T12:07:35.465 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.414+0000 7f9bdd2c5700 10 monclient(hunting): authenticate will time out at 1256.011413s 2021-05-25T12:07:35.465 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.414+0000 7f9bd7fff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f9bd8096d50 0x7f9bd8128fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33868/0 (socket says 172.21.15.102:33868) 2021-05-25T12:07:35.465 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.414+0000 7f9bd7fff700 1 -- 172.21.15.102:0/2342104115 learned_addr learned my addr 172.21.15.102:0/2342104115 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:07:35.465 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.414+0000 7f9bc3fff700 10 client.?.objecter ms_handle_connect 0x7f9bd8096d50 2021-05-25T12:07:35.466 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.414+0000 7f9bc3fff700 10 client.?.objecter resend_mon_ops 2021-05-25T12:07:35.466 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.414+0000 7f9bc3fff700 10 client.?.objecter ms_handle_connect 0x7f9bd812ec90 2021-05-25T12:07:35.466 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.414+0000 7f9bc3fff700 10 client.?.objecter resend_mon_ops 2021-05-25T12:07:35.466 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.414+0000 7f9bc3fff700 10 client.?.objecter ms_handle_connect 0x7f9bd812b180 2021-05-25T12:07:35.466 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.414+0000 7f9bc3fff700 10 client.?.objecter resend_mon_ops 2021-05-25T12:07:35.467 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.414+0000 7f9bc3fff700 1 -- 172.21.15.102:0/2342104115 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2535414607 0 0) 0x7f9bd8093e20 con 0x7f9bd8096d50 2021-05-25T12:07:35.467 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.414+0000 7f9bc3fff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:35.467 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.414+0000 7f9bc3fff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:35.467 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.414+0000 7f9bc3fff700 10 monclient(hunting): my global_id is 204414 2021-05-25T12:07:35.467 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.414+0000 7f9bc3fff700 1 -- 172.21.15.102:0/2342104115 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bb8004c10 con 0x7f9bd8096d50 2021-05-25T12:07:35.468 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.414+0000 7f9bc3fff700 1 -- 172.21.15.102:0/2342104115 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 330223448 0 0) 0x7f9bd8117a30 con 0x7f9bd812ec90 2021-05-25T12:07:35.468 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.414+0000 7f9bc3fff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:35.468 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.414+0000 7f9bc3fff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:35.468 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.414+0000 7f9bc3fff700 10 monclient(hunting): my global_id is 234149 2021-05-25T12:07:35.468 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.414+0000 7f9bc3fff700 1 -- 172.21.15.102:0/2342104115 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bd8093e20 con 0x7f9bd812ec90 2021-05-25T12:07:35.469 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.414+0000 7f9bc3fff700 1 -- 172.21.15.102:0/2342104115 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3816487193 0 0) 0x7f9bd8117790 con 0x7f9bd812b180 2021-05-25T12:07:35.469 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.414+0000 7f9bc3fff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:35.469 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.414+0000 7f9bc3fff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:35.469 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.414+0000 7f9bc3fff700 10 monclient(hunting): my global_id is 224281 2021-05-25T12:07:35.470 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.414+0000 7f9bc3fff700 1 -- 172.21.15.102:0/2342104115 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bd8117a30 con 0x7f9bd812b180 2021-05-25T12:07:35.470 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.414+0000 7f9bc3fff700 1 -- 172.21.15.102:0/2342104115 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4181296160 0 0) 0x7f9bb8004c10 con 0x7f9bd8096d50 2021-05-25T12:07:35.470 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.414+0000 7f9bc3fff700 1 -- 172.21.15.102:0/2342104115 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9bd8117790 con 0x7f9bd8096d50 2021-05-25T12:07:35.470 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.414+0000 7f9bc3fff700 1 -- 172.21.15.102:0/2342104115 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f9bc4001440 con 0x7f9bd8096d50 2021-05-25T12:07:35.470 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.414+0000 7f9bc3fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:35.471 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.415+0000 7f9bc3fff700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:07:35.471 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.415+0000 7f9bc3fff700 10 monclient(hunting): dump: 2021-05-25T12:07:35.471 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:35.471 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:35.471 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:35.472 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:35.472 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:35.472 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:35.472 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:35.473 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:35.473 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:35.473 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:35.473 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.415+0000 7f9bc3fff700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:07:35.473 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.415+0000 7f9bc3fff700 1 -- 172.21.15.102:0/2342104115 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2521413642 0 0) 0x7f9bd8093e20 con 0x7f9bd812ec90 2021-05-25T12:07:35.474 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.415+0000 7f9bc3fff700 1 -- 172.21.15.102:0/2342104115 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9bb8004c10 con 0x7f9bd812ec90 2021-05-25T12:07:35.474 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.415+0000 7f9bc3fff700 1 -- 172.21.15.102:0/2342104115 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f9bcc001bd0 con 0x7f9bd812ec90 2021-05-25T12:07:35.474 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.415+0000 7f9bc3fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:35.474 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.415+0000 7f9bc3fff700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2021-05-25T12:07:35.474 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.415+0000 7f9bc3fff700 10 monclient(hunting): dump: 2021-05-25T12:07:35.475 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:35.475 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:35.475 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:35.475 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:35.475 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:35.476 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:35.476 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:35.476 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:35.476 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:35.476 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:35.477 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.415+0000 7f9bc3fff700 1 -- 172.21.15.102:0/2342104115 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2133127205 0 0) 0x7f9bd8117790 con 0x7f9bd8096d50 2021-05-25T12:07:35.477 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.415+0000 7f9bc3fff700 1 -- 172.21.15.102:0/2342104115 >> v1:172.21.15.102:6790/0 conn(0x7f9bd812ec90 legacy=0x7f9bd8143aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:35.477 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.415+0000 7f9bc3fff700 1 -- 172.21.15.102:0/2342104115 >> v1:172.21.15.181:6789/0 conn(0x7f9bd812b180 legacy=0x7f9bd8140370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:35.477 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.415+0000 7f9bc3fff700 10 monclient: _finish_hunting 0 2021-05-25T12:07:35.478 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.415+0000 7f9bc3fff700 1 monclient: found mon.a 2021-05-25T12:07:35.478 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.415+0000 7f9bc3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:07:35.478 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.415+0000 7f9bc3fff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:07:35.478 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.415+0000 7f9bc3fff700 1 -- 172.21.15.102:0/2342104115 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bd81158c0 con 0x7f9bd8096d50 2021-05-25T12:07:35.478 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.415+0000 7f9bc3fff700 10 monclient: _finish_auth 0 2021-05-25T12:07:35.479 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.415+0000 7f9bc3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:07:35.479 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.415+0000 7f9bdd2c5700 5 monclient: authenticate success, global_id 204414 2021-05-25T12:07:35.479 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.415+0000 7f9bdd2c5700 10 monclient: _renew_subs 2021-05-25T12:07:35.479 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.415+0000 7f9bdd2c5700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:07:35.479 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.415+0000 7f9bdd2c5700 1 -- 172.21.15.102:0/2342104115 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bd81296f0 con 0x7f9bd8096d50 2021-05-25T12:07:35.480 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.415+0000 7f9bdd2c5700 10 client.204414.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:07:35.480 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.415+0000 7f9bdd2c5700 10 monclient: _renew_subs 2021-05-25T12:07:35.480 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.415+0000 7f9bdd2c5700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:07:35.480 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.415+0000 7f9bdd2c5700 1 -- 172.21.15.102:0/2342104115 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bd8114d60 con 0x7f9bd8096d50 2021-05-25T12:07:35.481 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.415+0000 7f9bdd2c5700 1 librados: init done 2021-05-25T12:07:35.481 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.415+0000 7f9bc3fff700 1 -- 172.21.15.102:0/2342104115 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9bc4004910 con 0x7f9bd8096d50 2021-05-25T12:07:35.481 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.415+0000 7f9bc3fff700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:07:35.481 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.415+0000 7f9bc3fff700 1 -- 172.21.15.102:0/2342104115 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f9bc4004ca0 con 0x7f9bd8096d50 2021-05-25T12:07:35.481 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.415+0000 7f9bc3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:35.482 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.415+0000 7f9bc3fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:07:35.482 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.415+0000 7f9bd67fc700 4 set_mon_vals no callback set 2021-05-25T12:07:35.482 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.415+0000 7f9bc3fff700 10 monclient: dump: 2021-05-25T12:07:35.482 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:35.482 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:35.483 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:35.483 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:35.483 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:35.483 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:35.484 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:35.484 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:35.484 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:35.484 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:35.484 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.415+0000 7f9bc3fff700 1 -- 172.21.15.102:0/2342104115 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f9bc4004e80 con 0x7f9bd8096d50 2021-05-25T12:07:35.485 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.415+0000 7f9bdd2c5700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:07:35.485 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.415+0000 7f9bdd2c5700 1 -- 172.21.15.102:0/2342104115 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f9bb00065a0 con 0x7f9bb0002840 2021-05-25T12:07:35.485 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.416+0000 7f9bc3fff700 1 -- 172.21.15.102:0/2342104115 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(245..245 src has 1..245) v4 ==== 5749+0+0 (unknown 3061073547 0 0) 0x7f9bc403d8e0 con 0x7f9bd8096d50 2021-05-25T12:07:35.485 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.416+0000 7f9bc3fff700 10 client.204414.objecter ms_dispatch 0x7f9bd8001550 osd_map(245..245 src has 1..245) v4 2021-05-25T12:07:35.485 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.416+0000 7f9bc3fff700 3 client.204414.objecter handle_osd_map got epochs [245,245] > 0 2021-05-25T12:07:35.486 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.416+0000 7f9bc3fff700 3 client.204414.objecter handle_osd_map decoding full epoch 245 2021-05-25T12:07:35.486 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.416+0000 7f9bc3fff700 20 client.204414.objecter dump_active .. 0 homeless 2021-05-25T12:07:35.486 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.416+0000 7f9bc3fff700 10 client.204414.objecter ms_handle_connect 0x7f9bb0002840 2021-05-25T12:07:35.486 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.416+0000 7f9bc3fff700 10 client.204414.objecter resend_mon_ops 2021-05-25T12:07:35.487 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.416+0000 7f9bc3fff700 1 -- 172.21.15.102:0/2342104115 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3765450200 0 0) 0x7f9bb00065a0 con 0x7f9bb0002840 2021-05-25T12:07:35.487 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.416+0000 7f9bc3fff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:35.487 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.416+0000 7f9bc3fff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:35.487 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.416+0000 7f9bc3fff700 10 monclient(hunting): my global_id is 204417 2021-05-25T12:07:35.487 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.416+0000 7f9bc3fff700 1 -- 172.21.15.102:0/2342104115 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bd8117790 con 0x7f9bb0002840 2021-05-25T12:07:35.488 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.416+0000 7f9bdcac4700 10 monclient: get_auth_request con 0x7f9bb8033270 auth_method 0 2021-05-25T12:07:35.488 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.416+0000 7f9bc3fff700 1 -- 172.21.15.102:0/2342104115 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3046559782 0 0) 0x7f9bd8117790 con 0x7f9bb0002840 2021-05-25T12:07:35.488 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.416+0000 7f9bc3fff700 1 -- 172.21.15.102:0/2342104115 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9bb00065a0 con 0x7f9bb0002840 2021-05-25T12:07:35.488 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.416+0000 7f9bc3fff700 1 -- 172.21.15.102:0/2342104115 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f9bc80046d0 con 0x7f9bb0002840 2021-05-25T12:07:35.488 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.416+0000 7f9bc3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:35.489 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.416+0000 7f9bc3fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:07:35.489 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.416+0000 7f9bc3fff700 10 monclient: dump: 2021-05-25T12:07:35.489 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:35.489 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:35.489 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:35.490 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:35.490 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:35.490 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:35.490 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:35.491 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:35.491 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:35.491 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:35.491 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.416+0000 7f9bc3fff700 10 client.204414.objecter ms_handle_connect 0x7f9bb8033270 2021-05-25T12:07:35.491 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.416+0000 7f9bc3fff700 1 -- 172.21.15.102:0/2342104115 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4036785064 0 0) 0x7f9bb00065a0 con 0x7f9bb0002840 2021-05-25T12:07:35.492 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.416+0000 7f9bc3fff700 1 -- 172.21.15.102:0/2342104115 --> v1:172.21.15.102:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9bb0009e40 con 0x7f9bb0002840 2021-05-25T12:07:35.492 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.416+0000 7f9bc3fff700 1 -- 172.21.15.102:0/2342104115 <== mon.0 v1:172.21.15.102:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7f9bc8006a20 con 0x7f9bb0002840 2021-05-25T12:07:35.492 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.416+0000 7f9bc3fff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:07:35.492 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.416+0000 7f9bc3fff700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:07:35.492 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.416+0000 7f9bc3fff700 1 -- 172.21.15.102:0/2342104115 >> v1:172.21.15.102:6789/0 conn(0x7f9bb0002840 legacy=0x7f9bb0004ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:35.493 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.416+0000 7f9bc3fff700 1 -- 172.21.15.102:0/2342104115 >> v1:172.21.15.102:6789/0 conn(0x7f9bb0002840 legacy=0x7f9bb0004ce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:07:35.493 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.423+0000 7f9bdd2c5700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:07:35.493 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.423+0000 7f9bdd2c5700 1 -- 172.21.15.102:0/2342104115 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f9bb0007510 con 0x7f9bb00068b0 2021-05-25T12:07:35.493 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.423+0000 7f9bc3fff700 10 client.204414.objecter ms_handle_connect 0x7f9bb00068b0 2021-05-25T12:07:35.493 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.423+0000 7f9bc3fff700 10 client.204414.objecter resend_mon_ops 2021-05-25T12:07:35.494 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.423+0000 7f9bc3fff700 1 -- 172.21.15.102:0/2342104115 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4199443390 0 0) 0x7f9bb0007510 con 0x7f9bb00068b0 2021-05-25T12:07:35.494 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.423+0000 7f9bc3fff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:35.494 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.423+0000 7f9bc3fff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:35.495 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.423+0000 7f9bc3fff700 10 monclient(hunting): my global_id is 204420 2021-05-25T12:07:35.495 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.423+0000 7f9bc3fff700 1 -- 172.21.15.102:0/2342104115 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bb00065a0 con 0x7f9bb00068b0 2021-05-25T12:07:35.495 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.423+0000 7f9bc3fff700 1 -- 172.21.15.102:0/2342104115 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 253553114 0 0) 0x7f9bb00065a0 con 0x7f9bb00068b0 2021-05-25T12:07:35.495 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.423+0000 7f9bc3fff700 1 -- 172.21.15.102:0/2342104115 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9bb0007510 con 0x7f9bb00068b0 2021-05-25T12:07:35.495 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.423+0000 7f9bc3fff700 1 -- 172.21.15.102:0/2342104115 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f9bc8004e80 con 0x7f9bb00068b0 2021-05-25T12:07:35.496 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.423+0000 7f9bc3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:35.496 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.423+0000 7f9bc3fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:07:35.496 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.423+0000 7f9bc3fff700 10 monclient: dump: 2021-05-25T12:07:35.496 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:35.497 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:35.497 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:35.497 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:35.497 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:35.497 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:35.498 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:35.498 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:35.498 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:35.498 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:35.498 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.423+0000 7f9bc3fff700 1 -- 172.21.15.102:0/2342104115 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1630450413 0 0) 0x7f9bb0007510 con 0x7f9bb00068b0 2021-05-25T12:07:35.499 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.424+0000 7f9bc3fff700 1 -- 172.21.15.102:0/2342104115 --> v1:172.21.15.102:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f9bb0008770 con 0x7f9bb00068b0 2021-05-25T12:07:35.499 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.424+0000 7f9bc3fff700 1 -- 172.21.15.102:0/2342104115 <== mon.0 v1:172.21.15.102:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3940 (unknown 0 0 1640280671) 0x7f9bc8006300 con 0x7f9bb00068b0 2021-05-25T12:07:35.499 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.424+0000 7f9bc3fff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:07:35.499 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.424+0000 7f9bc3fff700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:07:35.499 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.424+0000 7f9bc3fff700 1 -- 172.21.15.102:0/2342104115 >> v1:172.21.15.102:6789/0 conn(0x7f9bb00068b0 legacy=0x7f9bb0006d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:35.500 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.424+0000 7f9bc3fff700 1 -- 172.21.15.102:0/2342104115 >> v1:172.21.15.102:6789/0 conn(0x7f9bb00068b0 legacy=0x7f9bb0006d40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:07:35.500 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:07:35.500 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:07:35.501 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:07:35.501 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "peon", 2021-05-25T12:07:35.501 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 64, 2021-05-25T12:07:35.501 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:07:35.502 INFO:teuthology.orchestra.run.smithi102.stdout: 0, 2021-05-25T12:07:35.502 INFO:teuthology.orchestra.run.smithi102.stdout: 1, 2021-05-25T12:07:35.502 INFO:teuthology.orchestra.run.smithi102.stdout: 2 2021-05-25T12:07:35.502 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:07:35.502 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 2, 2021-05-25T12:07:35.503 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:07:35.503 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:07:35.503 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:07:35.503 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:07:35.503 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:07:35.504 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:07:35.504 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:07:35.504 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:07:35.504 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:07:35.505 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:07:35.505 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:07:35.505 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:07:35.505 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:07:35.505 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:07:35.506 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:07:35.506 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:07:35.506 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:07:35.506 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:07:35.506 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:07:35.507 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:07:35.507 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:07:35.507 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:07:35.507 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:07:35.507 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:07:35.508 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:07:35.508 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:35.508 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:07:35.508 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:07:35.509 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:07:35.509 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:07:35.509 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:07:35.509 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:07:35.509 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:07:35.510 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:07:35.510 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:07:35.510 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:07:35.510 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:07:35.510 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:07:35.511 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:07:35.511 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:07:35.511 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:07:35.511 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:07:35.511 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:07:35.512 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:07:35.512 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:07:35.512 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:07:35.512 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:07:35.512 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:07:35.513 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:07:35.513 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:07:35.513 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:07:35.513 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:07:35.514 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:35.514 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:07:35.514 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:35.514 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:07:35.514 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:07:35.515 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:07:35.515 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:07:35.515 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:35.515 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:07:35.515 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:07:35.516 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:07:35.516 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:07:35.516 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:07:35.516 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:35.516 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:07:35.517 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:07:35.517 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:07:35.517 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:07:35.517 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:07:35.518 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:35.518 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:35.518 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:07:35.518 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:07:35.518 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:07:35.519 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:07:35.519 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:35.519 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:07:35.519 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:07:35.519 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:07:35.520 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:07:35.520 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:07:35.520 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:35.520 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:07:35.520 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:07:35.521 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:07:35.521 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:07:35.521 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:07:35.521 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:35.521 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:35.522 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:07:35.522 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:07:35.522 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:07:35.522 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:07:35.523 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:35.523 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:07:35.523 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:07:35.523 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:07:35.523 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:07:35.524 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:07:35.524 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:35.524 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:07:35.524 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:07:35.524 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:07:35.525 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:07:35.525 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:07:35.525 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:07:35.525 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:07:35.525 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:35.526 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:07:35.526 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:07:35.526 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:35.526 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:07:35.526 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:07:35.527 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:07:35.527 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:07:35.527 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:07:35.527 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": [ 2021-05-25T12:07:35.528 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:35.528 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:07:35.528 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:07:35.528 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 6 2021-05-25T12:07:35.528 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:07:35.529 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:07:35.529 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:07:35.529 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:35.529 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:07:35.529 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:07:35.530 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 4 2021-05-25T12:07:35.530 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:07:35.530 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:07:35.530 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:35.530 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:07:35.531 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:07:35.533 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.424+0000 7f9bdd2c5700 10 librados: watch_flush enter 2021-05-25T12:07:35.533 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.424+0000 7f9bdd2c5700 10 librados: watch_flush exit 2021-05-25T12:07:35.534 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.424+0000 7f9bdd2c5700 20 client.204414.objecter shutdown clearing up homeless session... 2021-05-25T12:07:35.534 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.424+0000 7f9bdd2c5700 10 client.204414.objecter successfully canceled tick 2021-05-25T12:07:35.534 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.424+0000 7f9bdd2c5700 1 -- 172.21.15.102:0/2342104115 >> v1:172.21.15.181:6812/29582 conn(0x7f9bb8033270 legacy=0x7f9bb8035730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:35.534 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.424+0000 7f9bdd2c5700 10 monclient: shutdown 2021-05-25T12:07:35.534 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.424+0000 7f9bdd2c5700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:07:35.535 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.424+0000 7f9bdd2c5700 1 -- 172.21.15.102:0/2342104115 >> v1:172.21.15.102:6789/0 conn(0x7f9bd8096d50 legacy=0x7f9bd8128fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:35.535 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.424+0000 7f9bdcac4700 1 -- 172.21.15.102:0/2342104115 reap_dead start 2021-05-25T12:07:35.535 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.424+0000 7f9bdcac4700 1 -- 172.21.15.102:0/2342104115 reap_dead start 2021-05-25T12:07:35.535 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.424+0000 7f9bdd2c5700 1 -- 172.21.15.102:0/2342104115 shutdown_connections 2021-05-25T12:07:35.536 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.424+0000 7f9bdd2c5700 1 -- 172.21.15.102:0/2342104115 >> 172.21.15.102:0/2342104115 conn(0x7f9bd80974b0 msgr2=0x7f9bd8068230 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:07:35.536 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.424+0000 7f9bdd2c5700 1 -- 172.21.15.102:0/2342104115 shutdown_connections 2021-05-25T12:07:35.536 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.424+0000 7f9bdd2c5700 1 -- 172.21.15.102:0/2342104115 wait complete. 2021-05-25T12:07:35.536 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.424+0000 7f9bdd2c5700 1 librados: shutdown 2021-05-25T12:07:35.537 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2021-05-25T12:07:35.646 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.645+0000 7fef659ab700 10 monclient: get_monmap_and_config 2021-05-25T12:07:35.647 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.645+0000 7fef659ab700 10 monclient: build_initial_monmap 2021-05-25T12:07:35.647 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.645+0000 7fef659ab700 10 monclient: monmap: 2021-05-25T12:07:35.647 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:07:35.647 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:35.648 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:07:35.646447+0000 2021-05-25T12:07:35.648 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:07:35.646447+0000 2021-05-25T12:07:35.648 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:07:35.648 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:35.649 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:07:35.649 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:07:35.650 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:07:35.650 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:35.668 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.646+0000 7fef659ab700 1 Processor -- start 2021-05-25T12:07:35.668 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.646+0000 7fef659ab700 1 -- start start 2021-05-25T12:07:35.668 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.646+0000 7fef659ab700 10 monclient: init 2021-05-25T12:07:35.669 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.646+0000 7fef659ab700 10 monclient: _reopen_session rank -1 2021-05-25T12:07:35.669 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.646+0000 7fef659ab700 10 monclient: _add_conns ranks=[2,0,1] 2021-05-25T12:07:35.669 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.646+0000 7fef659ab700 10 monclient(hunting): picked mon.noname-b con 0x7fef60090bc0 addr v1:172.21.15.102:6790/0 2021-05-25T12:07:35.669 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.646+0000 7fef659ab700 10 monclient(hunting): picked mon.noname-a con 0x7fef60090fa0 addr v1:172.21.15.102:6789/0 2021-05-25T12:07:35.669 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.646+0000 7fef659ab700 10 monclient(hunting): picked mon.noname-c con 0x7fef60131e10 addr v1:172.21.15.181:6789/0 2021-05-25T12:07:35.670 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.646+0000 7fef659ab700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef600977a0 con 0x7fef60090fa0 2021-05-25T12:07:35.670 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.646+0000 7fef659ab700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef60095eb0 con 0x7fef60131e10 2021-05-25T12:07:35.670 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.646+0000 7fef659ab700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef60096150 con 0x7fef60090bc0 2021-05-25T12:07:35.670 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.646+0000 7fef659ab700 10 monclient(hunting): _renew_subs 2021-05-25T12:07:35.671 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.646+0000 7fef659ab700 10 monclient(hunting): authenticate will time out at 1256.243732s 2021-05-25T12:07:35.671 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.646+0000 7fef57fff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fef60090fa0 0x7fef60091420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33880/0 (socket says 172.21.15.102:33880) 2021-05-25T12:07:35.671 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.646+0000 7fef649a9700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fef60090bc0 0x7fef60097090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:56990/0 (socket says 172.21.15.102:56990) 2021-05-25T12:07:35.671 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.646+0000 7fef57fff700 1 -- 172.21.15.102:0/1300801437 learned_addr learned my addr 172.21.15.102:0/1300801437 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:07:35.671 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.647+0000 7fef577fe700 1 -- 172.21.15.102:0/1300801437 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3216480906 0 0) 0x7fef60096150 con 0x7fef60090bc0 2021-05-25T12:07:35.672 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.647+0000 7fef577fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:35.672 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.647+0000 7fef577fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:35.672 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.647+0000 7fef577fe700 10 monclient(hunting): my global_id is 234152 2021-05-25T12:07:35.672 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.647+0000 7fef577fe700 1 -- 172.21.15.102:0/1300801437 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef4c003270 con 0x7fef60090bc0 2021-05-25T12:07:35.673 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.647+0000 7fef577fe700 1 -- 172.21.15.102:0/1300801437 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 517029949 0 0) 0x7fef600977a0 con 0x7fef60090fa0 2021-05-25T12:07:35.673 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.647+0000 7fef577fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:35.673 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.647+0000 7fef577fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:35.673 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.647+0000 7fef577fe700 10 monclient(hunting): my global_id is 204423 2021-05-25T12:07:35.674 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.647+0000 7fef577fe700 1 -- 172.21.15.102:0/1300801437 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef60096150 con 0x7fef60090fa0 2021-05-25T12:07:35.674 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.647+0000 7fef577fe700 1 -- 172.21.15.102:0/1300801437 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2723599309 0 0) 0x7fef4c003270 con 0x7fef60090bc0 2021-05-25T12:07:35.674 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.647+0000 7fef577fe700 1 -- 172.21.15.102:0/1300801437 >> v1:172.21.15.181:6789/0 conn(0x7fef60131e10 legacy=0x7fef60134200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:35.674 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.647+0000 7fef577fe700 1 -- 172.21.15.102:0/1300801437 >> v1:172.21.15.102:6789/0 conn(0x7fef60090fa0 legacy=0x7fef60091420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:35.675 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.647+0000 7fef577fe700 10 monclient: _finish_hunting 0 2021-05-25T12:07:35.675 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.647+0000 7fef577fe700 1 monclient: found mon.noname-b 2021-05-25T12:07:35.675 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.647+0000 7fef577fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:07:35.675 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.647+0000 7fef577fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T12:07:35.675 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.647+0000 7fef577fe700 1 -- 172.21.15.102:0/1300801437 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef6006b8d0 con 0x7fef60090bc0 2021-05-25T12:07:35.676 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.647+0000 7fef577fe700 10 monclient: _finish_auth 0 2021-05-25T12:07:35.676 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.647+0000 7fef577fe700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:07:35.676 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.647+0000 7fef577fe700 1 -- 172.21.15.102:0/1300801437 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fef50001650 con 0x7fef60090bc0 2021-05-25T12:07:35.676 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.647+0000 7fef659ab700 5 monclient: authenticate success, global_id 234152 2021-05-25T12:07:35.676 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.647+0000 7fef577fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:35.677 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.647+0000 7fef577fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:07:35.677 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.647+0000 7fef577fe700 10 monclient: dump: 2021-05-25T12:07:35.677 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:35.677 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:35.678 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:35.678 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:35.678 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:35.678 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:35.678 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:35.679 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:35.679 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:35.679 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:35.679 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.647+0000 7fef659ab700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:07:35.679 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.647+0000 7fef577fe700 1 -- 172.21.15.102:0/1300801437 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fef50001820 con 0x7fef60090bc0 2021-05-25T12:07:35.680 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.647+0000 7fef577fe700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:07:35.680 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.647+0000 7fef577fe700 1 -- 172.21.15.102:0/1300801437 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fef50001bb0 con 0x7fef60090bc0 2021-05-25T12:07:35.680 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.647+0000 7fef659ab700 10 monclient: get_monmap_and_config success 2021-05-25T12:07:35.680 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.647+0000 7fef659ab700 4 set_mon_vals no callback set 2021-05-25T12:07:35.681 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.647+0000 7fef577fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:35.681 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.647+0000 7fef577fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:07:35.681 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.647+0000 7fef577fe700 10 monclient: dump: 2021-05-25T12:07:35.681 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:35.681 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:35.682 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:35.682 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:35.682 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:35.682 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:35.683 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:35.683 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:35.683 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:35.684 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:35.684 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.647+0000 7fef659ab700 10 monclient: shutdown 2021-05-25T12:07:35.684 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.647+0000 7fef659ab700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:07:35.684 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.647+0000 7fef659ab700 1 -- 172.21.15.102:0/1300801437 >> v1:172.21.15.102:6790/0 conn(0x7fef60090bc0 legacy=0x7fef60097090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:35.684 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.647+0000 7fef659ab700 1 -- 172.21.15.102:0/1300801437 shutdown_connections 2021-05-25T12:07:35.685 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.647+0000 7fef659ab700 1 -- 172.21.15.102:0/1300801437 >> 172.21.15.102:0/1300801437 conn(0x7fef6009b070 msgr2=0x7fef6009b450 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:07:35.685 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.648+0000 7fef659ab700 1 -- 172.21.15.102:0/1300801437 shutdown_connections 2021-05-25T12:07:35.685 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.648+0000 7fef659ab700 1 -- 172.21.15.102:0/1300801437 wait complete. 2021-05-25T12:07:35.685 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.648+0000 7fef659ab700 10 monclient: build_initial_monmap 2021-05-25T12:07:35.685 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.648+0000 7fef659ab700 10 monclient: monmap: 2021-05-25T12:07:35.686 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:07:35.686 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:07:35.686 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:07:35.686 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:07:35.687 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:07:35.687 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:07:35.687 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:07:35.687 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:07:35.687 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:07:35.688 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:35.688 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.649+0000 7fef659ab700 1 librados: starting msgr at 2021-05-25T12:07:35.688 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.649+0000 7fef659ab700 1 librados: starting objecter 2021-05-25T12:07:35.688 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.649+0000 7fef659ab700 1 Processor -- start 2021-05-25T12:07:35.689 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.649+0000 7fef659ab700 1 -- start start 2021-05-25T12:07:35.689 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.649+0000 7fef659ab700 1 librados: setting wanted keys 2021-05-25T12:07:35.689 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.649+0000 7fef659ab700 1 librados: calling monclient init 2021-05-25T12:07:35.689 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.649+0000 7fef659ab700 10 monclient: init 2021-05-25T12:07:35.690 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.650+0000 7fef659ab700 10 monclient: _reopen_session rank -1 2021-05-25T12:07:35.690 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.650+0000 7fef659ab700 10 monclient: _add_conns ranks=[1,2,0] 2021-05-25T12:07:35.690 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.650+0000 7fef659ab700 10 monclient(hunting): picked mon.noname-b con 0x7fef60090bc0 addr v1:172.21.15.181:6789/0 2021-05-25T12:07:35.690 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.650+0000 7fef659ab700 10 monclient(hunting): picked mon.noname-c con 0x7fef60090fa0 addr v1:172.21.15.102:6790/0 2021-05-25T12:07:35.690 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.650+0000 7fef659ab700 10 monclient(hunting): picked mon.noname-a con 0x7fef60131e10 addr v1:172.21.15.102:6789/0 2021-05-25T12:07:35.691 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.650+0000 7fef659ab700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef60117e70 con 0x7fef60131e10 2021-05-25T12:07:35.691 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.650+0000 7fef659ab700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef60118110 con 0x7fef60090bc0 2021-05-25T12:07:35.691 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.650+0000 7fef659ab700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef601183b0 con 0x7fef60090fa0 2021-05-25T12:07:35.691 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.650+0000 7fef659ab700 10 monclient(hunting): _renew_subs 2021-05-25T12:07:35.691 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.650+0000 7fef659ab700 10 monclient(hunting): authenticate will time out at 1256.247358s 2021-05-25T12:07:35.692 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.650+0000 7fef57fff700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fef60090fa0 0x7fef6013b240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:56998/0 (socket says 172.21.15.102:56998) 2021-05-25T12:07:35.692 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.650+0000 7fef57fff700 1 -- 172.21.15.102:0/1346278890 learned_addr learned my addr 172.21.15.102:0/1346278890 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:07:35.692 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.650+0000 7fef54ff9700 10 client.?.objecter ms_handle_connect 0x7fef60090fa0 2021-05-25T12:07:35.692 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.650+0000 7fef54ff9700 10 client.?.objecter resend_mon_ops 2021-05-25T12:07:35.693 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.650+0000 7fef54ff9700 10 client.?.objecter ms_handle_connect 0x7fef60131e10 2021-05-25T12:07:35.693 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.650+0000 7fef54ff9700 10 client.?.objecter resend_mon_ops 2021-05-25T12:07:35.693 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.650+0000 7fef54ff9700 10 client.?.objecter ms_handle_connect 0x7fef60090bc0 2021-05-25T12:07:35.693 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.650+0000 7fef54ff9700 10 client.?.objecter resend_mon_ops 2021-05-25T12:07:35.693 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.650+0000 7fef54ff9700 1 -- 172.21.15.102:0/1346278890 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2940017957 0 0) 0x7fef60117e70 con 0x7fef60131e10 2021-05-25T12:07:35.694 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.650+0000 7fef54ff9700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:35.694 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.650+0000 7fef54ff9700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:35.694 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.650+0000 7fef54ff9700 10 monclient(hunting): my global_id is 204426 2021-05-25T12:07:35.695 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.650+0000 7fef54ff9700 1 -- 172.21.15.102:0/1346278890 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef40004c10 con 0x7fef60131e10 2021-05-25T12:07:35.695 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.650+0000 7fef54ff9700 1 -- 172.21.15.102:0/1346278890 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4225163361 0 0) 0x7fef601183b0 con 0x7fef60090fa0 2021-05-25T12:07:35.695 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.650+0000 7fef54ff9700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:35.695 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.650+0000 7fef54ff9700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:35.695 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.650+0000 7fef54ff9700 10 monclient(hunting): my global_id is 234155 2021-05-25T12:07:35.696 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.650+0000 7fef54ff9700 1 -- 172.21.15.102:0/1346278890 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef60117e70 con 0x7fef60090fa0 2021-05-25T12:07:35.696 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.650+0000 7fef54ff9700 1 -- 172.21.15.102:0/1346278890 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1376542759 0 0) 0x7fef60118110 con 0x7fef60090bc0 2021-05-25T12:07:35.696 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.650+0000 7fef54ff9700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:35.696 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.650+0000 7fef54ff9700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:35.696 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.650+0000 7fef54ff9700 10 monclient(hunting): my global_id is 224287 2021-05-25T12:07:35.697 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.650+0000 7fef54ff9700 1 -- 172.21.15.102:0/1346278890 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef601183b0 con 0x7fef60090bc0 2021-05-25T12:07:35.697 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.650+0000 7fef54ff9700 1 -- 172.21.15.102:0/1346278890 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2956086818 0 0) 0x7fef40004c10 con 0x7fef60131e10 2021-05-25T12:07:35.697 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.650+0000 7fef54ff9700 1 -- 172.21.15.102:0/1346278890 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fef60118110 con 0x7fef60131e10 2021-05-25T12:07:35.697 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.650+0000 7fef54ff9700 1 -- 172.21.15.102:0/1346278890 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fef5c001c80 con 0x7fef60131e10 2021-05-25T12:07:35.698 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.650+0000 7fef54ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:35.698 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.650+0000 7fef54ff9700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:07:35.698 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.650+0000 7fef54ff9700 10 monclient(hunting): dump: 2021-05-25T12:07:35.698 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:35.698 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:35.699 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:35.699 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:35.699 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:35.699 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:35.699 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:35.700 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:35.700 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:35.700 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:35.700 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.651+0000 7fef54ff9700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:07:35.700 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.651+0000 7fef54ff9700 1 -- 172.21.15.102:0/1346278890 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 218745187 0 0) 0x7fef601183b0 con 0x7fef60090bc0 2021-05-25T12:07:35.701 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.651+0000 7fef54ff9700 1 -- 172.21.15.102:0/1346278890 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fef40004c10 con 0x7fef60090bc0 2021-05-25T12:07:35.701 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.651+0000 7fef54ff9700 1 -- 172.21.15.102:0/1346278890 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fef50001b30 con 0x7fef60090bc0 2021-05-25T12:07:35.701 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.651+0000 7fef54ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:35.701 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.651+0000 7fef54ff9700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2021-05-25T12:07:35.702 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.651+0000 7fef54ff9700 10 monclient(hunting): dump: 2021-05-25T12:07:35.702 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:35.702 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:35.702 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:35.702 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:35.703 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:35.703 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:35.703 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:35.703 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:35.703 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:35.704 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:35.704 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.651+0000 7fef54ff9700 1 -- 172.21.15.102:0/1346278890 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 724508417 0 0) 0x7fef60117e70 con 0x7fef60090fa0 2021-05-25T12:07:35.704 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.651+0000 7fef54ff9700 1 -- 172.21.15.102:0/1346278890 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fef601183b0 con 0x7fef60090fa0 2021-05-25T12:07:35.705 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.651+0000 7fef54ff9700 1 -- 172.21.15.102:0/1346278890 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fef480017e0 con 0x7fef60090fa0 2021-05-25T12:07:35.705 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.651+0000 7fef54ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:35.705 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.651+0000 7fef54ff9700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2021-05-25T12:07:35.705 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.651+0000 7fef54ff9700 10 monclient(hunting): dump: 2021-05-25T12:07:35.705 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:35.706 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:35.706 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:35.706 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:35.706 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:35.707 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:35.707 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:35.707 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:35.707 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:35.707 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:35.708 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.651+0000 7fef54ff9700 1 -- 172.21.15.102:0/1346278890 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2248294732 0 0) 0x7fef60118110 con 0x7fef60131e10 2021-05-25T12:07:35.708 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.651+0000 7fef54ff9700 1 -- 172.21.15.102:0/1346278890 >> v1:172.21.15.102:6790/0 conn(0x7fef60090fa0 legacy=0x7fef6013b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:35.708 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.651+0000 7fef54ff9700 1 -- 172.21.15.102:0/1346278890 >> v1:172.21.15.181:6789/0 conn(0x7fef60090bc0 legacy=0x7fef6013ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:35.708 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.651+0000 7fef54ff9700 10 monclient: _finish_hunting 0 2021-05-25T12:07:35.709 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.651+0000 7fef54ff9700 1 monclient: found mon.a 2021-05-25T12:07:35.709 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.651+0000 7fef54ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:07:35.709 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.651+0000 7fef54ff9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:07:35.709 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.651+0000 7fef54ff9700 1 -- 172.21.15.102:0/1346278890 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef601162b0 con 0x7fef60131e10 2021-05-25T12:07:35.709 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.651+0000 7fef54ff9700 10 monclient: _finish_auth 0 2021-05-25T12:07:35.710 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.651+0000 7fef54ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:07:35.710 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.651+0000 7fef659ab700 5 monclient: authenticate success, global_id 204426 2021-05-25T12:07:35.710 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.651+0000 7fef659ab700 10 monclient: _renew_subs 2021-05-25T12:07:35.710 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.651+0000 7fef659ab700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:07:35.711 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.651+0000 7fef659ab700 1 -- 172.21.15.102:0/1346278890 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef60116ee0 con 0x7fef60131e10 2021-05-25T12:07:35.711 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.651+0000 7fef659ab700 10 client.204426.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:07:35.711 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.651+0000 7fef659ab700 10 monclient: _renew_subs 2021-05-25T12:07:35.711 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.651+0000 7fef659ab700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:07:35.711 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.651+0000 7fef659ab700 1 -- 172.21.15.102:0/1346278890 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fef60130ef0 con 0x7fef60131e10 2021-05-25T12:07:35.712 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.651+0000 7fef659ab700 1 librados: init done 2021-05-25T12:07:35.712 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.651+0000 7fef54ff9700 1 -- 172.21.15.102:0/1346278890 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fef5c001ef0 con 0x7fef60131e10 2021-05-25T12:07:35.712 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.651+0000 7fef54ff9700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:07:35.713 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.651+0000 7fef54ff9700 1 -- 172.21.15.102:0/1346278890 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fef5c001470 con 0x7fef60131e10 2021-05-25T12:07:35.713 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.651+0000 7fef54ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:35.713 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.651+0000 7fef54ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:07:35.713 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.651+0000 7fef54ff9700 10 monclient: dump: 2021-05-25T12:07:35.714 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:35.714 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:35.714 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:35.714 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:35.715 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:35.715 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:35.715 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:35.715 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:35.716 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:35.716 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:35.716 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.651+0000 7fef56ffd700 4 set_mon_vals no callback set 2021-05-25T12:07:35.716 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.651+0000 7fef54ff9700 1 -- 172.21.15.102:0/1346278890 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7fef5c002300 con 0x7fef60131e10 2021-05-25T12:07:35.716 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.651+0000 7fef659ab700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:07:35.717 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.651+0000 7fef659ab700 1 -- 172.21.15.102:0/1346278890 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fef380065a0 con 0x7fef38002840 2021-05-25T12:07:35.717 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.652+0000 7fef54ff9700 1 -- 172.21.15.102:0/1346278890 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(245..245 src has 1..245) v4 ==== 5749+0+0 (unknown 3061073547 0 0) 0x7fef5c03b470 con 0x7fef60131e10 2021-05-25T12:07:35.717 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.652+0000 7fef54ff9700 10 client.204426.objecter ms_dispatch 0x7fef60001550 osd_map(245..245 src has 1..245) v4 2021-05-25T12:07:35.717 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.652+0000 7fef54ff9700 3 client.204426.objecter handle_osd_map got epochs [245,245] > 0 2021-05-25T12:07:35.717 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.652+0000 7fef54ff9700 3 client.204426.objecter handle_osd_map decoding full epoch 245 2021-05-25T12:07:35.718 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.652+0000 7fef54ff9700 20 client.204426.objecter dump_active .. 0 homeless 2021-05-25T12:07:35.718 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.652+0000 7fef54ff9700 10 client.204426.objecter ms_handle_connect 0x7fef38002840 2021-05-25T12:07:35.718 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.652+0000 7fef54ff9700 10 client.204426.objecter resend_mon_ops 2021-05-25T12:07:35.718 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.652+0000 7fef54ff9700 1 -- 172.21.15.102:0/1346278890 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1103146441 0 0) 0x7fef380065a0 con 0x7fef38002840 2021-05-25T12:07:35.719 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.652+0000 7fef54ff9700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:35.719 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.652+0000 7fef54ff9700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:35.719 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.652+0000 7fef54ff9700 10 monclient(hunting): my global_id is 234158 2021-05-25T12:07:35.719 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.652+0000 7fef54ff9700 1 -- 172.21.15.102:0/1346278890 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef60118110 con 0x7fef38002840 2021-05-25T12:07:35.719 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.652+0000 7fef57fff700 10 monclient: get_auth_request con 0x7fef40033570 auth_method 0 2021-05-25T12:07:35.720 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.652+0000 7fef54ff9700 1 -- 172.21.15.102:0/1346278890 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 60068905 0 0) 0x7fef60118110 con 0x7fef38002840 2021-05-25T12:07:35.720 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.652+0000 7fef54ff9700 1 -- 172.21.15.102:0/1346278890 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fef380065a0 con 0x7fef38002840 2021-05-25T12:07:35.720 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.652+0000 7fef54ff9700 1 -- 172.21.15.102:0/1346278890 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fef500049c0 con 0x7fef38002840 2021-05-25T12:07:35.720 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.652+0000 7fef54ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:35.720 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.652+0000 7fef54ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:07:35.721 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.652+0000 7fef54ff9700 10 monclient: dump: 2021-05-25T12:07:35.721 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:35.721 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:35.721 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:35.721 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:35.722 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:35.722 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:35.722 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:35.722 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:35.722 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:35.723 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:35.723 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.652+0000 7fef54ff9700 1 -- 172.21.15.102:0/1346278890 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1210668023 0 0) 0x7fef380065a0 con 0x7fef38002840 2021-05-25T12:07:35.723 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.652+0000 7fef54ff9700 1 -- 172.21.15.102:0/1346278890 --> v1:172.21.15.102:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fef38009e40 con 0x7fef38002840 2021-05-25T12:07:35.723 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.652+0000 7fef54ff9700 10 client.204426.objecter ms_handle_connect 0x7fef40033570 2021-05-25T12:07:35.724 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.652+0000 7fef54ff9700 1 -- 172.21.15.102:0/1346278890 <== mon.2 v1:172.21.15.102:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7fef50006f80 con 0x7fef38002840 2021-05-25T12:07:35.724 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.652+0000 7fef54ff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:07:35.724 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.652+0000 7fef54ff9700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:07:35.724 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.652+0000 7fef54ff9700 1 -- 172.21.15.102:0/1346278890 >> v1:172.21.15.102:6790/0 conn(0x7fef38002840 legacy=0x7fef38004ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:35.724 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.652+0000 7fef54ff9700 1 -- 172.21.15.102:0/1346278890 >> v1:172.21.15.102:6790/0 conn(0x7fef38002840 legacy=0x7fef38004ce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:07:35.725 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.659+0000 7fef659ab700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:07:35.725 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.659+0000 7fef659ab700 1 -- 172.21.15.102:0/1346278890 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fef38009ad0 con 0x7fef38008d90 2021-05-25T12:07:35.725 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.659+0000 7fef54ff9700 10 client.204426.objecter ms_handle_connect 0x7fef38008d90 2021-05-25T12:07:35.725 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.659+0000 7fef54ff9700 10 client.204426.objecter resend_mon_ops 2021-05-25T12:07:35.726 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.659+0000 7fef54ff9700 1 -- 172.21.15.102:0/1346278890 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3042941088 0 0) 0x7fef38009ad0 con 0x7fef38008d90 2021-05-25T12:07:35.726 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.659+0000 7fef54ff9700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:35.726 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.659+0000 7fef54ff9700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:35.726 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.659+0000 7fef54ff9700 10 monclient(hunting): my global_id is 234161 2021-05-25T12:07:35.727 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.659+0000 7fef54ff9700 1 -- 172.21.15.102:0/1346278890 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef380065a0 con 0x7fef38008d90 2021-05-25T12:07:35.727 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.659+0000 7fef54ff9700 1 -- 172.21.15.102:0/1346278890 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3794956763 0 0) 0x7fef380065a0 con 0x7fef38008d90 2021-05-25T12:07:35.727 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.659+0000 7fef54ff9700 1 -- 172.21.15.102:0/1346278890 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fef38009ad0 con 0x7fef38008d90 2021-05-25T12:07:35.727 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.659+0000 7fef54ff9700 1 -- 172.21.15.102:0/1346278890 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fef50005390 con 0x7fef38008d90 2021-05-25T12:07:35.727 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.659+0000 7fef54ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:35.728 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.659+0000 7fef54ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:07:35.728 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.659+0000 7fef54ff9700 10 monclient: dump: 2021-05-25T12:07:35.728 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:35.728 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:35.728 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:35.729 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:35.729 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:35.729 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:35.729 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:35.729 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:35.730 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:35.730 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:35.730 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.659+0000 7fef54ff9700 1 -- 172.21.15.102:0/1346278890 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 602727559 0 0) 0x7fef38009ad0 con 0x7fef38008d90 2021-05-25T12:07:35.730 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.660+0000 7fef54ff9700 1 -- 172.21.15.102:0/1346278890 --> v1:172.21.15.102:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fef38008740 con 0x7fef38008d90 2021-05-25T12:07:35.731 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.660+0000 7fef54ff9700 1 -- 172.21.15.102:0/1346278890 <== mon.2 v1:172.21.15.102:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3772 (unknown 0 0 667832650) 0x7fef50007210 con 0x7fef38008d90 2021-05-25T12:07:35.731 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.660+0000 7fef54ff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:07:35.731 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.660+0000 7fef54ff9700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:07:35.731 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.660+0000 7fef54ff9700 1 -- 172.21.15.102:0/1346278890 >> v1:172.21.15.102:6790/0 conn(0x7fef38008d90 legacy=0x7fef38009220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:35.731 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.660+0000 7fef54ff9700 1 -- 172.21.15.102:0/1346278890 >> v1:172.21.15.102:6790/0 conn(0x7fef38008d90 legacy=0x7fef38009220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:07:35.732 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:07:35.732 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:07:35.732 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:07:35.733 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "leader", 2021-05-25T12:07:35.733 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 64, 2021-05-25T12:07:35.733 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:07:35.733 INFO:teuthology.orchestra.run.smithi102.stdout: 0, 2021-05-25T12:07:35.734 INFO:teuthology.orchestra.run.smithi102.stdout: 1, 2021-05-25T12:07:35.734 INFO:teuthology.orchestra.run.smithi102.stdout: 2 2021-05-25T12:07:35.734 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:07:35.734 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 2, 2021-05-25T12:07:35.734 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:07:35.735 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:07:35.735 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:07:35.735 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:07:35.735 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:07:35.736 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:07:35.736 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:07:35.736 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:07:35.737 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:07:35.737 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:07:35.737 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:07:35.737 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:07:35.737 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:07:35.738 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:07:35.738 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:07:35.738 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:07:35.738 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:07:35.739 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:07:35.739 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:07:35.739 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:07:35.739 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:07:35.740 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:07:35.740 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:07:35.740 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:07:35.740 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:07:35.740 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:35.741 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:07:35.741 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:07:35.741 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:07:35.741 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:07:35.742 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:07:35.742 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:07:35.742 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:07:35.742 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:07:35.742 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:07:35.743 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:07:35.743 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:07:35.743 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:07:35.743 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:07:35.743 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:07:35.744 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:07:35.744 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:07:35.744 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:07:35.744 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:07:35.744 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:07:35.745 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:07:35.745 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:07:35.745 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:07:35.745 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:07:35.745 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:07:35.746 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:07:35.746 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:07:35.746 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:35.747 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:07:35.747 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:35.747 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:07:35.747 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:07:35.747 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:07:35.748 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:07:35.748 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:35.748 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:07:35.748 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:07:35.748 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:07:35.749 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:07:35.749 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:07:35.749 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:35.749 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:07:35.750 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:07:35.750 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:07:35.750 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:07:35.750 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:07:35.750 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:35.751 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:35.751 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:07:35.751 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:07:35.751 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:07:35.751 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:07:35.752 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:35.752 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:07:35.752 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:07:35.752 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:07:35.753 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:07:35.753 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:07:35.753 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:35.753 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:07:35.753 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:07:35.754 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:07:35.754 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:07:35.754 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:07:35.754 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:35.754 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:35.755 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:07:35.755 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:07:35.755 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:07:35.755 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:07:35.756 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:35.756 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:07:35.756 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:07:35.756 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:07:35.756 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:07:35.757 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:07:35.757 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:35.757 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:07:35.757 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:07:35.758 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:07:35.758 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:07:35.758 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:07:35.758 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:07:35.759 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:07:35.759 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:35.759 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:07:35.759 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:07:35.759 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:35.760 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:07:35.760 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:07:35.760 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:07:35.760 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:07:35.760 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:07:35.761 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:07:35.761 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:35.761 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:07:35.761 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:07:35.761 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 2 2021-05-25T12:07:35.762 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:07:35.762 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:07:35.762 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:35.762 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:07:35.762 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:07:35.765 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.660+0000 7fef659ab700 10 librados: watch_flush enter 2021-05-25T12:07:35.765 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.660+0000 7fef659ab700 10 librados: watch_flush exit 2021-05-25T12:07:35.765 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.660+0000 7fef659ab700 20 client.204426.objecter shutdown clearing up homeless session... 2021-05-25T12:07:35.765 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.660+0000 7fef659ab700 10 client.204426.objecter successfully canceled tick 2021-05-25T12:07:35.766 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.660+0000 7fef659ab700 1 -- 172.21.15.102:0/1346278890 >> v1:172.21.15.181:6812/29582 conn(0x7fef40033570 legacy=0x7fef40035a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:35.766 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.660+0000 7fef659ab700 10 monclient: shutdown 2021-05-25T12:07:35.766 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.660+0000 7fef659ab700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:07:35.766 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.660+0000 7fef659ab700 1 -- 172.21.15.102:0/1346278890 >> v1:172.21.15.102:6789/0 conn(0x7fef60131e10 legacy=0x7fef60142e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:35.766 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.660+0000 7fef651aa700 1 -- 172.21.15.102:0/1346278890 reap_dead start 2021-05-25T12:07:35.767 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.660+0000 7fef651aa700 1 -- 172.21.15.102:0/1346278890 reap_dead start 2021-05-25T12:07:35.767 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.660+0000 7fef659ab700 1 -- 172.21.15.102:0/1346278890 shutdown_connections 2021-05-25T12:07:35.767 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.660+0000 7fef659ab700 1 -- 172.21.15.102:0/1346278890 >> 172.21.15.102:0/1346278890 conn(0x7fef6009b070 msgr2=0x7fef600995d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:07:35.768 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.660+0000 7fef659ab700 1 -- 172.21.15.102:0/1346278890 shutdown_connections 2021-05-25T12:07:35.768 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.660+0000 7fef659ab700 1 -- 172.21.15.102:0/1346278890 wait complete. 2021-05-25T12:07:35.768 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.660+0000 7fef659ab700 1 librados: shutdown 2021-05-25T12:07:35.768 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2021-05-25T12:07:35.878 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.877+0000 7fbc5e80a700 10 monclient: get_monmap_and_config 2021-05-25T12:07:35.879 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.877+0000 7fbc5e80a700 10 monclient: build_initial_monmap 2021-05-25T12:07:35.879 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.877+0000 7fbc5e80a700 10 monclient: monmap: 2021-05-25T12:07:35.879 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:07:35.880 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:35.880 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:07:35.878191+0000 2021-05-25T12:07:35.881 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:07:35.878191+0000 2021-05-25T12:07:35.881 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:07:35.881 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:35.882 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:07:35.882 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:07:35.882 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:07:35.883 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:35.896 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.877+0000 7fbc5e80a700 1 Processor -- start 2021-05-25T12:07:35.896 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.877+0000 7fbc5e80a700 1 -- start start 2021-05-25T12:07:35.896 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.877+0000 7fbc5e80a700 10 monclient: init 2021-05-25T12:07:35.896 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.878+0000 7fbc5e80a700 10 monclient: _reopen_session rank -1 2021-05-25T12:07:35.897 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.878+0000 7fbc5e80a700 10 monclient: _add_conns ranks=[2,0,1] 2021-05-25T12:07:35.897 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.878+0000 7fbc5e80a700 10 monclient(hunting): picked mon.noname-b con 0x7fbc58093fc0 addr v1:172.21.15.102:6790/0 2021-05-25T12:07:35.897 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.878+0000 7fbc5e80a700 10 monclient(hunting): picked mon.noname-a con 0x7fbc580692d0 addr v1:172.21.15.102:6789/0 2021-05-25T12:07:35.897 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.878+0000 7fbc5e80a700 10 monclient(hunting): picked mon.noname-c con 0x7fbc5812f130 addr v1:172.21.15.181:6789/0 2021-05-25T12:07:35.897 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.878+0000 7fbc5e80a700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc58097160 con 0x7fbc580692d0 2021-05-25T12:07:35.898 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.878+0000 7fbc5e80a700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc58092fe0 con 0x7fbc5812f130 2021-05-25T12:07:35.898 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.878+0000 7fbc5e80a700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc58093280 con 0x7fbc58093fc0 2021-05-25T12:07:35.898 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.878+0000 7fbc5e80a700 10 monclient(hunting): _renew_subs 2021-05-25T12:07:35.898 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.878+0000 7fbc5e80a700 10 monclient(hunting): authenticate will time out at 1256.475406s 2021-05-25T12:07:35.899 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.878+0000 7fbc577fe700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fbc580692d0 0x7fbc58096a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33898/0 (socket says 172.21.15.102:33898) 2021-05-25T12:07:35.899 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.878+0000 7fbc57fff700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fbc58093fc0 0x7fbc58096340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:57010/0 (socket says 172.21.15.102:57010) 2021-05-25T12:07:35.899 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.878+0000 7fbc577fe700 1 -- 172.21.15.102:0/4210694929 learned_addr learned my addr 172.21.15.102:0/4210694929 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:07:35.899 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.878+0000 7fbc56ffd700 1 -- 172.21.15.102:0/4210694929 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 827424442 0 0) 0x7fbc58093280 con 0x7fbc58093fc0 2021-05-25T12:07:35.899 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.878+0000 7fbc56ffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:35.900 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.878+0000 7fbc56ffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:35.900 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.878+0000 7fbc56ffd700 10 monclient(hunting): my global_id is 234164 2021-05-25T12:07:35.900 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.878+0000 7fbc56ffd700 1 -- 172.21.15.102:0/4210694929 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc40003270 con 0x7fbc58093fc0 2021-05-25T12:07:35.900 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.878+0000 7fbc56ffd700 1 -- 172.21.15.102:0/4210694929 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2842271866 0 0) 0x7fbc58097160 con 0x7fbc580692d0 2021-05-25T12:07:35.900 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.878+0000 7fbc56ffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:35.901 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.878+0000 7fbc56ffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:35.901 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.878+0000 7fbc56ffd700 10 monclient(hunting): my global_id is 204429 2021-05-25T12:07:35.901 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.878+0000 7fbc56ffd700 1 -- 172.21.15.102:0/4210694929 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc58093280 con 0x7fbc580692d0 2021-05-25T12:07:35.901 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.878+0000 7fbc56ffd700 1 -- 172.21.15.102:0/4210694929 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3254766907 0 0) 0x7fbc58093280 con 0x7fbc580692d0 2021-05-25T12:07:35.902 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.878+0000 7fbc56ffd700 1 -- 172.21.15.102:0/4210694929 >> v1:172.21.15.102:6790/0 conn(0x7fbc58093fc0 legacy=0x7fbc58096340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:35.902 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.879+0000 7fbc56ffd700 1 -- 172.21.15.102:0/4210694929 >> v1:172.21.15.181:6789/0 conn(0x7fbc5812f130 legacy=0x7fbc58131520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:35.902 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.879+0000 7fbc56ffd700 10 monclient: _finish_hunting 0 2021-05-25T12:07:35.902 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.879+0000 7fbc56ffd700 1 monclient: found mon.noname-a 2021-05-25T12:07:35.903 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.879+0000 7fbc56ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:07:35.903 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.879+0000 7fbc56ffd700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T12:07:35.903 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.879+0000 7fbc56ffd700 1 -- 172.21.15.102:0/4210694929 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc58093520 con 0x7fbc580692d0 2021-05-25T12:07:35.903 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.879+0000 7fbc56ffd700 10 monclient: _finish_auth 0 2021-05-25T12:07:35.904 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.879+0000 7fbc56ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:07:35.904 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.879+0000 7fbc56ffd700 1 -- 172.21.15.102:0/4210694929 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fbc3c001610 con 0x7fbc580692d0 2021-05-25T12:07:35.904 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.879+0000 7fbc5e80a700 5 monclient: authenticate success, global_id 204429 2021-05-25T12:07:35.904 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.879+0000 7fbc5e80a700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:07:35.905 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.879+0000 7fbc56ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:35.905 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.879+0000 7fbc56ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:07:35.905 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.879+0000 7fbc56ffd700 10 monclient: dump: 2021-05-25T12:07:35.905 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:35.906 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:35.906 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:35.906 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:35.906 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:35.907 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:35.907 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:35.907 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:35.907 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:35.908 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:35.908 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.879+0000 7fbc5e80a700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:07:35.908 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.879+0000 7fbc56ffd700 1 -- 172.21.15.102:0/4210694929 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbc3c001830 con 0x7fbc580692d0 2021-05-25T12:07:35.908 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.879+0000 7fbc56ffd700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:07:35.908 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.879+0000 7fbc5e80a700 10 monclient: get_monmap_and_config success 2021-05-25T12:07:35.909 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.879+0000 7fbc56ffd700 1 -- 172.21.15.102:0/4210694929 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fbc3c001bc0 con 0x7fbc580692d0 2021-05-25T12:07:35.909 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.879+0000 7fbc56ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:35.909 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.879+0000 7fbc5e80a700 4 set_mon_vals no callback set 2021-05-25T12:07:35.909 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.879+0000 7fbc56ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:07:35.909 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.879+0000 7fbc56ffd700 10 monclient: dump: 2021-05-25T12:07:35.910 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:35.910 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:35.910 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:35.910 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:35.911 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:35.911 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:35.911 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:35.911 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:35.911 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:35.912 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:35.912 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.879+0000 7fbc5e80a700 10 monclient: shutdown 2021-05-25T12:07:35.912 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.879+0000 7fbc5e80a700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:07:35.912 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.879+0000 7fbc5e80a700 1 -- 172.21.15.102:0/4210694929 >> v1:172.21.15.102:6789/0 conn(0x7fbc580692d0 legacy=0x7fbc58096a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:35.913 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.879+0000 7fbc5e80a700 1 -- 172.21.15.102:0/4210694929 shutdown_connections 2021-05-25T12:07:35.913 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.879+0000 7fbc5e80a700 1 -- 172.21.15.102:0/4210694929 >> 172.21.15.102:0/4210694929 conn(0x7fbc5809b370 msgr2=0x7fbc58097430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:07:35.913 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.879+0000 7fbc5e80a700 1 -- 172.21.15.102:0/4210694929 shutdown_connections 2021-05-25T12:07:35.913 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.879+0000 7fbc5e80a700 1 -- 172.21.15.102:0/4210694929 wait complete. 2021-05-25T12:07:35.913 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.880+0000 7fbc5e80a700 10 monclient: build_initial_monmap 2021-05-25T12:07:35.914 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.880+0000 7fbc5e80a700 10 monclient: monmap: 2021-05-25T12:07:35.914 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:07:35.914 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:07:35.915 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:07:35.915 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:07:35.915 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:07:35.915 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:07:35.915 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:07:35.916 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:07:35.916 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:07:35.916 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:35.916 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.880+0000 7fbc5e80a700 1 librados: starting msgr at 2021-05-25T12:07:35.916 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.880+0000 7fbc5e80a700 1 librados: starting objecter 2021-05-25T12:07:35.917 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.880+0000 7fbc5e80a700 1 Processor -- start 2021-05-25T12:07:35.917 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.880+0000 7fbc5e80a700 1 -- start start 2021-05-25T12:07:35.917 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.880+0000 7fbc5e80a700 1 librados: setting wanted keys 2021-05-25T12:07:35.917 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.880+0000 7fbc5e80a700 1 librados: calling monclient init 2021-05-25T12:07:35.918 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.880+0000 7fbc5e80a700 10 monclient: init 2021-05-25T12:07:35.918 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.880+0000 7fbc5e80a700 10 monclient: _reopen_session rank -1 2021-05-25T12:07:35.918 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.880+0000 7fbc5e80a700 10 monclient: _add_conns ranks=[0,1,2] 2021-05-25T12:07:35.918 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.880+0000 7fbc5e80a700 10 monclient(hunting): picked mon.noname-a con 0x7fbc580692d0 addr v1:172.21.15.102:6789/0 2021-05-25T12:07:35.918 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.881+0000 7fbc5e80a700 10 monclient(hunting): picked mon.noname-b con 0x7fbc58093fc0 addr v1:172.21.15.181:6789/0 2021-05-25T12:07:35.919 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.881+0000 7fbc5e80a700 10 monclient(hunting): picked mon.noname-c con 0x7fbc5812f130 addr v1:172.21.15.102:6790/0 2021-05-25T12:07:35.919 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.881+0000 7fbc5e80a700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc58116300 con 0x7fbc580692d0 2021-05-25T12:07:35.919 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.881+0000 7fbc5e80a700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc58127f60 con 0x7fbc58093fc0 2021-05-25T12:07:35.919 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.881+0000 7fbc5e80a700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc58092350 con 0x7fbc5812f130 2021-05-25T12:07:35.920 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.881+0000 7fbc5e80a700 10 monclient(hunting): _renew_subs 2021-05-25T12:07:35.920 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.881+0000 7fbc5e80a700 10 monclient(hunting): authenticate will time out at 1256.478045s 2021-05-25T12:07:35.920 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.881+0000 7fbc57fff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fbc580692d0 0x7fbc581126c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33904/0 (socket says 172.21.15.102:33904) 2021-05-25T12:07:35.920 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.881+0000 7fbc57fff700 1 -- 172.21.15.102:0/435510340 learned_addr learned my addr 172.21.15.102:0/435510340 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:07:35.920 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.881+0000 7fbc4bfff700 10 client.?.objecter ms_handle_connect 0x7fbc5812f130 2021-05-25T12:07:35.921 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.881+0000 7fbc4bfff700 10 client.?.objecter resend_mon_ops 2021-05-25T12:07:35.921 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.881+0000 7fbc4bfff700 10 client.?.objecter ms_handle_connect 0x7fbc580692d0 2021-05-25T12:07:35.921 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.881+0000 7fbc4bfff700 10 client.?.objecter resend_mon_ops 2021-05-25T12:07:35.922 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.881+0000 7fbc4bfff700 10 client.?.objecter ms_handle_connect 0x7fbc58093fc0 2021-05-25T12:07:35.922 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.881+0000 7fbc4bfff700 10 client.?.objecter resend_mon_ops 2021-05-25T12:07:35.922 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.881+0000 7fbc4bfff700 1 -- 172.21.15.102:0/435510340 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3426361661 0 0) 0x7fbc58092350 con 0x7fbc5812f130 2021-05-25T12:07:35.922 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.881+0000 7fbc4bfff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:35.923 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.881+0000 7fbc4bfff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:35.923 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.881+0000 7fbc4bfff700 10 monclient(hunting): my global_id is 234167 2021-05-25T12:07:35.924 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.881+0000 7fbc4bfff700 1 -- 172.21.15.102:0/435510340 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc34004c10 con 0x7fbc5812f130 2021-05-25T12:07:35.924 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.881+0000 7fbc4bfff700 1 -- 172.21.15.102:0/435510340 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1246378075 0 0) 0x7fbc58116300 con 0x7fbc580692d0 2021-05-25T12:07:35.924 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.881+0000 7fbc4bfff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:35.924 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.881+0000 7fbc4bfff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:35.924 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.881+0000 7fbc4bfff700 10 monclient(hunting): my global_id is 204432 2021-05-25T12:07:35.925 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.881+0000 7fbc4bfff700 1 -- 172.21.15.102:0/435510340 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc58092350 con 0x7fbc580692d0 2021-05-25T12:07:35.925 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.881+0000 7fbc4bfff700 1 -- 172.21.15.102:0/435510340 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1568054044 0 0) 0x7fbc58127f60 con 0x7fbc58093fc0 2021-05-25T12:07:35.925 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.881+0000 7fbc4bfff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:35.925 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.881+0000 7fbc4bfff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:35.926 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.881+0000 7fbc4bfff700 10 monclient(hunting): my global_id is 224293 2021-05-25T12:07:35.926 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.881+0000 7fbc4bfff700 1 -- 172.21.15.102:0/435510340 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc58116300 con 0x7fbc58093fc0 2021-05-25T12:07:35.926 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.881+0000 7fbc4bfff700 1 -- 172.21.15.102:0/435510340 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1408628539 0 0) 0x7fbc34004c10 con 0x7fbc5812f130 2021-05-25T12:07:35.926 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.881+0000 7fbc4bfff700 1 -- 172.21.15.102:0/435510340 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbc58127f60 con 0x7fbc5812f130 2021-05-25T12:07:35.926 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.881+0000 7fbc4bfff700 1 -- 172.21.15.102:0/435510340 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3371213892 0 0) 0x7fbc58092350 con 0x7fbc580692d0 2021-05-25T12:07:35.927 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.881+0000 7fbc4bfff700 1 -- 172.21.15.102:0/435510340 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbc34004c10 con 0x7fbc580692d0 2021-05-25T12:07:35.927 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.881+0000 7fbc4bfff700 1 -- 172.21.15.102:0/435510340 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fbc4c001b80 con 0x7fbc5812f130 2021-05-25T12:07:35.927 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.881+0000 7fbc4bfff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:35.927 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.881+0000 7fbc4bfff700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:07:35.928 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.881+0000 7fbc4bfff700 10 monclient(hunting): dump: 2021-05-25T12:07:35.928 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:35.928 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:35.928 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:35.928 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:35.929 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:35.929 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:35.929 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:35.929 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:35.929 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:35.930 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:35.930 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.881+0000 7fbc4bfff700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:07:35.930 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.881+0000 7fbc4bfff700 1 -- 172.21.15.102:0/435510340 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fbc44001a70 con 0x7fbc580692d0 2021-05-25T12:07:35.930 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.881+0000 7fbc4bfff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:35.931 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.881+0000 7fbc4bfff700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2021-05-25T12:07:35.931 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.881+0000 7fbc4bfff700 10 monclient(hunting): dump: 2021-05-25T12:07:35.931 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:35.931 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:35.931 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:35.932 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:35.932 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:35.932 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:35.932 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:35.933 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:35.933 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:35.933 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:35.933 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.881+0000 7fbc4bfff700 1 -- 172.21.15.102:0/435510340 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 211589508 0 0) 0x7fbc58116300 con 0x7fbc58093fc0 2021-05-25T12:07:35.934 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.881+0000 7fbc4bfff700 1 -- 172.21.15.102:0/435510340 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbc58092350 con 0x7fbc58093fc0 2021-05-25T12:07:35.934 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.881+0000 7fbc4bfff700 1 -- 172.21.15.102:0/435510340 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fbc3c0015d0 con 0x7fbc58093fc0 2021-05-25T12:07:35.934 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.881+0000 7fbc4bfff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:35.934 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.881+0000 7fbc4bfff700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2021-05-25T12:07:35.935 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.881+0000 7fbc4bfff700 10 monclient(hunting): dump: 2021-05-25T12:07:35.935 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:35.935 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:35.935 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:35.935 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:35.936 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:35.936 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:35.936 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:35.936 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:35.937 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:35.937 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:35.937 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.881+0000 7fbc4bfff700 1 -- 172.21.15.102:0/435510340 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3114944399 0 0) 0x7fbc58127f60 con 0x7fbc5812f130 2021-05-25T12:07:35.937 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.881+0000 7fbc4bfff700 1 -- 172.21.15.102:0/435510340 >> v1:172.21.15.181:6789/0 conn(0x7fbc58093fc0 legacy=0x7fbc58112dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:35.937 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.881+0000 7fbc4bfff700 1 -- 172.21.15.102:0/435510340 >> v1:172.21.15.102:6789/0 conn(0x7fbc580692d0 legacy=0x7fbc581126c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:35.938 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.881+0000 7fbc4bfff700 10 monclient: _finish_hunting 0 2021-05-25T12:07:35.938 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.881+0000 7fbc4bfff700 1 monclient: found mon.c 2021-05-25T12:07:35.938 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.881+0000 7fbc4bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:07:35.938 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.881+0000 7fbc4bfff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:07:35.938 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.881+0000 7fbc4bfff700 1 -- 172.21.15.102:0/435510340 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc5812a330 con 0x7fbc5812f130 2021-05-25T12:07:35.939 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.881+0000 7fbc4bfff700 10 monclient: _finish_auth 0 2021-05-25T12:07:35.939 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.881+0000 7fbc4bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:07:35.939 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.881+0000 7fbc5e80a700 5 monclient: authenticate success, global_id 234167 2021-05-25T12:07:35.939 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.881+0000 7fbc5e80a700 10 monclient: _renew_subs 2021-05-25T12:07:35.939 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.881+0000 7fbc5e80a700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:07:35.940 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.881+0000 7fbc5e80a700 1 -- 172.21.15.102:0/435510340 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc581166c0 con 0x7fbc5812f130 2021-05-25T12:07:35.940 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.881+0000 7fbc5e80a700 10 client.234167.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:07:35.940 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.881+0000 7fbc5e80a700 10 monclient: _renew_subs 2021-05-25T12:07:35.941 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.881+0000 7fbc5e80a700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:07:35.941 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.881+0000 7fbc5e80a700 1 -- 172.21.15.102:0/435510340 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc58128df0 con 0x7fbc5812f130 2021-05-25T12:07:35.941 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.881+0000 7fbc5e80a700 1 librados: init done 2021-05-25T12:07:35.942 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.881+0000 7fbc4bfff700 1 -- 172.21.15.102:0/435510340 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbc4c001d70 con 0x7fbc5812f130 2021-05-25T12:07:35.942 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.881+0000 7fbc4bfff700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:07:35.942 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.881+0000 7fbc4bfff700 1 -- 172.21.15.102:0/435510340 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fbc4c002100 con 0x7fbc5812f130 2021-05-25T12:07:35.942 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.881+0000 7fbc4bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:35.942 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.881+0000 7fbc4bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:07:35.943 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.881+0000 7fbc567fc700 4 set_mon_vals no callback set 2021-05-25T12:07:35.943 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.882+0000 7fbc4bfff700 10 monclient: dump: 2021-05-25T12:07:35.943 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:35.944 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:35.944 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:35.944 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:35.944 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:35.944 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:35.945 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:35.945 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:35.945 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:35.945 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:35.946 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.882+0000 7fbc5e80a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:07:35.946 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.882+0000 7fbc5e80a700 1 -- 172.21.15.102:0/435510340 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fbc380065a0 con 0x7fbc38002840 2021-05-25T12:07:35.946 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.882+0000 7fbc4bfff700 1 -- 172.21.15.102:0/435510340 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7fbc4c0022e0 con 0x7fbc5812f130 2021-05-25T12:07:35.946 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.882+0000 7fbc4bfff700 10 client.234167.objecter ms_handle_connect 0x7fbc38002840 2021-05-25T12:07:35.946 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.882+0000 7fbc4bfff700 10 client.234167.objecter resend_mon_ops 2021-05-25T12:07:35.947 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.882+0000 7fbc4bfff700 1 -- 172.21.15.102:0/435510340 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(246..246 src has 1..246) v4 ==== 5858+0+0 (unknown 4267717717 0 0) 0x7fbc4c03b2e0 con 0x7fbc5812f130 2021-05-25T12:07:35.947 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.882+0000 7fbc4bfff700 10 client.234167.objecter ms_dispatch 0x7fbc58001550 osd_map(246..246 src has 1..246) v4 2021-05-25T12:07:35.947 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.882+0000 7fbc4bfff700 3 client.234167.objecter handle_osd_map got epochs [246,246] > 0 2021-05-25T12:07:35.947 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.882+0000 7fbc4bfff700 3 client.234167.objecter handle_osd_map decoding full epoch 246 2021-05-25T12:07:35.948 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.883+0000 7fbc4bfff700 20 client.234167.objecter dump_active .. 0 homeless 2021-05-25T12:07:35.948 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.883+0000 7fbc4bfff700 1 -- 172.21.15.102:0/435510340 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2073471539 0 0) 0x7fbc380065a0 con 0x7fbc38002840 2021-05-25T12:07:35.948 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.883+0000 7fbc4bfff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:35.948 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.883+0000 7fbc4bfff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:35.948 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.883+0000 7fbc4bfff700 10 monclient(hunting): my global_id is 224296 2021-05-25T12:07:35.949 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.883+0000 7fbc4bfff700 1 -- 172.21.15.102:0/435510340 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc58127f60 con 0x7fbc38002840 2021-05-25T12:07:35.949 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.883+0000 7fbc577fe700 10 monclient: get_auth_request con 0x7fbc34033520 auth_method 0 2021-05-25T12:07:35.949 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.883+0000 7fbc4bfff700 1 -- 172.21.15.102:0/435510340 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1193633459 0 0) 0x7fbc58127f60 con 0x7fbc38002840 2021-05-25T12:07:35.949 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.883+0000 7fbc4bfff700 1 -- 172.21.15.102:0/435510340 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbc380065a0 con 0x7fbc38002840 2021-05-25T12:07:35.949 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.883+0000 7fbc4bfff700 1 -- 172.21.15.102:0/435510340 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fbc44004980 con 0x7fbc38002840 2021-05-25T12:07:35.950 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.883+0000 7fbc4bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:35.950 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.883+0000 7fbc4bfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T12:07:35.950 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.883+0000 7fbc4bfff700 10 monclient: dump: 2021-05-25T12:07:35.950 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:35.950 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:35.951 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:35.951 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:35.951 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:35.951 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:35.952 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:35.952 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:35.952 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:35.952 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:35.952 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.883+0000 7fbc4bfff700 10 client.234167.objecter ms_handle_connect 0x7fbc34033520 2021-05-25T12:07:35.953 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.883+0000 7fbc4bfff700 1 -- 172.21.15.102:0/435510340 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3306344711 0 0) 0x7fbc380065a0 con 0x7fbc38002840 2021-05-25T12:07:35.953 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.883+0000 7fbc4bfff700 1 -- 172.21.15.102:0/435510340 --> v1:172.21.15.181:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbc38009e40 con 0x7fbc38002840 2021-05-25T12:07:35.953 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.883+0000 7fbc4bfff700 1 -- 172.21.15.102:0/435510340 <== mon.1 v1:172.21.15.181:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7fbc44006be0 con 0x7fbc38002840 2021-05-25T12:07:35.953 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.883+0000 7fbc4bfff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:07:35.954 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.883+0000 7fbc4bfff700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:07:35.954 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.883+0000 7fbc4bfff700 1 -- 172.21.15.102:0/435510340 >> v1:172.21.15.181:6789/0 conn(0x7fbc38002840 legacy=0x7fbc38004ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:35.954 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.883+0000 7fbc4bfff700 1 -- 172.21.15.102:0/435510340 >> v1:172.21.15.181:6789/0 conn(0x7fbc38002840 legacy=0x7fbc38004ce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:07:35.955 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.890+0000 7fbc5e80a700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:07:35.955 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.890+0000 7fbc5e80a700 1 -- 172.21.15.102:0/435510340 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fbc38007540 con 0x7fbc380068e0 2021-05-25T12:07:35.955 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.890+0000 7fbc4bfff700 10 client.234167.objecter ms_handle_connect 0x7fbc380068e0 2021-05-25T12:07:35.955 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.890+0000 7fbc4bfff700 10 client.234167.objecter resend_mon_ops 2021-05-25T12:07:35.956 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.890+0000 7fbc4bfff700 1 -- 172.21.15.102:0/435510340 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1691222684 0 0) 0x7fbc38007540 con 0x7fbc380068e0 2021-05-25T12:07:35.956 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.890+0000 7fbc4bfff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:35.956 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.890+0000 7fbc4bfff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:35.956 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.890+0000 7fbc4bfff700 10 monclient(hunting): my global_id is 224299 2021-05-25T12:07:35.956 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.890+0000 7fbc4bfff700 1 -- 172.21.15.102:0/435510340 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc380065a0 con 0x7fbc380068e0 2021-05-25T12:07:35.957 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.891+0000 7fbc4bfff700 1 -- 172.21.15.102:0/435510340 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3925122741 0 0) 0x7fbc380065a0 con 0x7fbc380068e0 2021-05-25T12:07:35.957 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.891+0000 7fbc4bfff700 1 -- 172.21.15.102:0/435510340 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbc38007540 con 0x7fbc380068e0 2021-05-25T12:07:35.957 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.891+0000 7fbc4bfff700 1 -- 172.21.15.102:0/435510340 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fbc44005060 con 0x7fbc380068e0 2021-05-25T12:07:35.957 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.891+0000 7fbc4bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:35.958 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.891+0000 7fbc4bfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T12:07:35.958 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.891+0000 7fbc4bfff700 10 monclient: dump: 2021-05-25T12:07:35.958 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:35.958 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:35.958 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:35.959 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:35.959 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:35.959 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:35.959 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:35.959 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:35.960 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:35.960 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:35.960 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.891+0000 7fbc4bfff700 1 -- 172.21.15.102:0/435510340 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 176303804 0 0) 0x7fbc38007540 con 0x7fbc380068e0 2021-05-25T12:07:35.960 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.891+0000 7fbc4bfff700 1 -- 172.21.15.102:0/435510340 --> v1:172.21.15.181:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fbc38008770 con 0x7fbc380068e0 2021-05-25T12:07:35.960 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.891+0000 7fbc4bfff700 1 -- 172.21.15.102:0/435510340 <== mon.1 v1:172.21.15.181:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3940 (unknown 0 0 3137019786) 0x7fbc440064e0 con 0x7fbc380068e0 2021-05-25T12:07:35.961 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.891+0000 7fbc4bfff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:07:35.961 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.891+0000 7fbc4bfff700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:07:35.961 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.891+0000 7fbc4bfff700 1 -- 172.21.15.102:0/435510340 >> v1:172.21.15.181:6789/0 conn(0x7fbc380068e0 legacy=0x7fbc38006d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:35.961 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.891+0000 7fbc4bfff700 1 -- 172.21.15.102:0/435510340 >> v1:172.21.15.181:6789/0 conn(0x7fbc380068e0 legacy=0x7fbc38006d70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:07:35.962 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:07:35.962 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:07:35.962 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:07:35.963 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "peon", 2021-05-25T12:07:35.963 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 64, 2021-05-25T12:07:35.963 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:07:35.963 INFO:teuthology.orchestra.run.smithi102.stdout: 0, 2021-05-25T12:07:35.963 INFO:teuthology.orchestra.run.smithi102.stdout: 1, 2021-05-25T12:07:35.964 INFO:teuthology.orchestra.run.smithi102.stdout: 2 2021-05-25T12:07:35.964 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:07:35.964 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 3, 2021-05-25T12:07:35.964 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:07:35.965 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:07:35.965 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:07:35.965 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:07:35.965 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:07:35.966 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:07:35.966 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:07:35.966 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:07:35.966 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:07:35.966 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:07:35.967 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:07:35.967 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:07:35.967 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:07:35.967 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:07:35.968 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:07:35.968 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:07:35.968 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:07:35.968 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:07:35.968 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:07:35.969 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:07:35.969 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:07:35.969 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:07:35.969 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:07:35.970 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:07:35.970 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:07:35.970 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:35.970 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:07:35.970 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:07:35.971 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:07:35.971 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:07:35.971 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:07:35.971 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:07:35.971 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:07:35.972 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:07:35.972 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:07:35.972 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:07:35.973 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:07:35.973 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:07:35.974 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:07:35.974 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:07:35.974 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:07:35.974 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:07:35.975 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:07:35.975 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:07:35.975 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:07:35.975 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:07:35.975 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:07:35.976 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:07:35.976 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:07:35.976 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:07:35.976 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:07:35.977 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:07:35.977 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:35.977 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:07:35.977 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:35.977 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:07:35.978 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:07:35.978 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:07:35.978 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:07:35.978 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:35.978 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:07:35.979 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:07:35.979 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:07:35.979 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:07:35.979 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:07:35.980 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:35.980 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:07:35.980 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:07:35.980 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:07:35.980 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:07:35.981 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:07:35.981 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:35.981 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:35.981 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:07:35.981 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:07:35.982 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:07:35.982 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:07:35.982 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:35.982 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:07:35.982 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:07:35.983 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:07:35.983 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:07:35.983 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:07:35.983 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:35.984 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:07:35.984 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:07:35.984 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:07:35.984 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:07:35.984 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:07:35.985 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:35.985 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:35.985 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:07:35.985 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:07:35.986 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:07:35.986 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:07:35.986 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:35.986 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:07:35.986 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:07:35.987 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:07:35.987 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:07:35.987 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:07:35.987 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:35.987 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:07:35.988 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:07:35.988 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:07:35.988 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:07:35.988 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:07:35.988 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:07:35.989 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:07:35.989 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:35.989 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:07:35.989 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:07:35.989 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:35.990 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:07:35.990 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:07:35.990 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:07:35.991 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:07:35.991 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:07:35.991 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:07:35.991 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:35.991 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:07:35.992 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:07:35.992 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 3 2021-05-25T12:07:35.992 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:07:35.992 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:07:35.992 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": [ 2021-05-25T12:07:35.993 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:35.993 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:07:35.993 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:07:35.993 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:07:35.994 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:07:35.994 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:07:35.994 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:35.994 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:07:35.994 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:07:35.997 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.891+0000 7fbc5e80a700 10 librados: watch_flush enter 2021-05-25T12:07:35.997 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.891+0000 7fbc5e80a700 10 librados: watch_flush exit 2021-05-25T12:07:35.997 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.891+0000 7fbc5e80a700 20 client.234167.objecter shutdown clearing up homeless session... 2021-05-25T12:07:35.997 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.891+0000 7fbc5e80a700 10 client.234167.objecter successfully canceled tick 2021-05-25T12:07:35.998 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.891+0000 7fbc5e80a700 1 -- 172.21.15.102:0/435510340 >> v1:172.21.15.181:6812/29582 conn(0x7fbc34033520 legacy=0x7fbc340359e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:35.998 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.891+0000 7fbc5e80a700 10 monclient: shutdown 2021-05-25T12:07:35.998 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.891+0000 7fbc5e80a700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:07:35.998 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.891+0000 7fbc5e80a700 1 -- 172.21.15.102:0/435510340 >> v1:172.21.15.102:6790/0 conn(0x7fbc5812f130 legacy=0x7fbc58143620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:35.999 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.892+0000 7fbc5cda7700 1 -- 172.21.15.102:0/435510340 reap_dead start 2021-05-25T12:07:35.999 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.892+0000 7fbc5cda7700 1 -- 172.21.15.102:0/435510340 reap_dead start 2021-05-25T12:07:35.999 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.892+0000 7fbc5e80a700 1 -- 172.21.15.102:0/435510340 shutdown_connections 2021-05-25T12:07:35.999 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.892+0000 7fbc5e80a700 1 -- 172.21.15.102:0/435510340 >> 172.21.15.102:0/435510340 conn(0x7fbc5809b370 msgr2=0x7fbc580988b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:07:35.999 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.892+0000 7fbc5e80a700 1 -- 172.21.15.102:0/435510340 shutdown_connections 2021-05-25T12:07:36.000 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.892+0000 7fbc5e80a700 1 -- 172.21.15.102:0/435510340 wait complete. 2021-05-25T12:07:36.000 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:35.892+0000 7fbc5e80a700 1 librados: shutdown 2021-05-25T12:07:36.000 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2021-05-25T12:07:36.001 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2021-05-25T12:07:36.001 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2021-05-25T12:07:36.001 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2021-05-25T12:07:36.002 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2021-05-25T12:07:36.002 INFO:teuthology.orchestra.run:waiting for 300 2021-05-25T12:07:36.011 INFO:tasks.ceph.mon.c:Stopped 2021-05-25T12:07:36.011 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2021-05-25T12:07:36.011 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2021-05-25T12:07:36.159 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.158+0000 7fb42f03b700 10 monclient: get_monmap_and_config 2021-05-25T12:07:36.159 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.158+0000 7fb42f03b700 10 monclient: build_initial_monmap 2021-05-25T12:07:36.160 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.158+0000 7fb42f03b700 10 monclient: monmap: 2021-05-25T12:07:36.160 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:07:36.160 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:36.160 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:07:36.159057+0000 2021-05-25T12:07:36.160 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:07:36.159057+0000 2021-05-25T12:07:36.161 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:07:36.161 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:36.161 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:07:36.161 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:07:36.162 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:07:36.162 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:36.173 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.158+0000 7fb42f03b700 1 Processor -- start 2021-05-25T12:07:36.173 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.158+0000 7fb42f03b700 1 -- start start 2021-05-25T12:07:36.173 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.158+0000 7fb42f03b700 10 monclient: init 2021-05-25T12:07:36.174 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.159+0000 7fb42f03b700 10 monclient: _reopen_session rank -1 2021-05-25T12:07:36.174 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.159+0000 7fb42f03b700 10 monclient: _add_conns ranks=[1,0,2] 2021-05-25T12:07:36.174 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.159+0000 7fb42f03b700 10 monclient(hunting): picked mon.noname-c con 0x7fb428090990 addr v1:172.21.15.181:6789/0 2021-05-25T12:07:36.174 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.159+0000 7fb42f03b700 10 monclient(hunting): picked mon.noname-a con 0x7fb428090d70 addr v1:172.21.15.102:6789/0 2021-05-25T12:07:36.175 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.159+0000 7fb42f03b700 10 monclient(hunting): picked mon.noname-b con 0x7fb42812e330 addr v1:172.21.15.102:6790/0 2021-05-25T12:07:36.175 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.159+0000 7fb42f03b700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb428096fa0 con 0x7fb428090d70 2021-05-25T12:07:36.175 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.159+0000 7fb42f03b700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4280926a0 con 0x7fb428090990 2021-05-25T12:07:36.175 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.159+0000 7fb42f03b700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb428092940 con 0x7fb42812e330 2021-05-25T12:07:36.176 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.159+0000 7fb42f03b700 10 monclient(hunting): _renew_subs 2021-05-25T12:07:36.176 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.159+0000 7fb42f03b700 10 monclient(hunting): authenticate will time out at 1256.756318s 2021-05-25T12:07:36.176 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.159+0000 7fb42d5d8700 1 -- >> v1:172.21.15.102:6790/0 conn(0x7fb42812e330 legacy=0x7fb428130720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6790/0 2021-05-25T12:07:36.176 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.159+0000 7fb41ffff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fb428090d70 0x7fb428091200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33920/0 (socket says 172.21.15.102:33920) 2021-05-25T12:07:36.176 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.159+0000 7fb41ffff700 1 -- 172.21.15.102:0/2161469393 learned_addr learned my addr 172.21.15.102:0/2161469393 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:07:36.177 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.159+0000 7fb41f7fe700 1 -- 172.21.15.102:0/2161469393 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 441058056 0 0) 0x7fb428096fa0 con 0x7fb428090d70 2021-05-25T12:07:36.177 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.159+0000 7fb41f7fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:36.177 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.159+0000 7fb41f7fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:36.177 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.159+0000 7fb41f7fe700 10 monclient(hunting): my global_id is 204435 2021-05-25T12:07:36.177 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.159+0000 7fb41f7fe700 1 -- 172.21.15.102:0/2161469393 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb414004380 con 0x7fb428090d70 2021-05-25T12:07:36.178 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.159+0000 7fb41f7fe700 1 -- 172.21.15.102:0/2161469393 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3402116729 0 0) 0x7fb4280926a0 con 0x7fb428090990 2021-05-25T12:07:36.178 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.159+0000 7fb41f7fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:36.178 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.159+0000 7fb41f7fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:36.178 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.159+0000 7fb41f7fe700 10 monclient(hunting): my global_id is 224302 2021-05-25T12:07:36.179 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.159+0000 7fb41f7fe700 1 -- 172.21.15.102:0/2161469393 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb428096fa0 con 0x7fb428090990 2021-05-25T12:07:36.179 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.159+0000 7fb41f7fe700 1 -- 172.21.15.102:0/2161469393 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 426727068 0 0) 0x7fb414004380 con 0x7fb428090d70 2021-05-25T12:07:36.179 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.159+0000 7fb41f7fe700 1 -- 172.21.15.102:0/2161469393 >> v1:172.21.15.102:6790/0 conn(0x7fb42812e330 legacy=0x7fb428130720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-05-25T12:07:36.179 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.159+0000 7fb41f7fe700 1 -- 172.21.15.102:0/2161469393 >> v1:172.21.15.181:6789/0 conn(0x7fb428090990 legacy=0x7fb428093240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:36.179 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.159+0000 7fb41f7fe700 10 monclient: _finish_hunting 0 2021-05-25T12:07:36.180 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.159+0000 7fb41f7fe700 1 monclient: found mon.noname-a 2021-05-25T12:07:36.180 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.159+0000 7fb41f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:07:36.180 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.159+0000 7fb41f7fe700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T12:07:36.180 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.159+0000 7fb41f7fe700 1 -- 172.21.15.102:0/2161469393 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb428092be0 con 0x7fb428090d70 2021-05-25T12:07:36.181 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.159+0000 7fb41f7fe700 10 monclient: _finish_auth 0 2021-05-25T12:07:36.181 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.159+0000 7fb41f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:07:36.181 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.159+0000 7fb42f03b700 5 monclient: authenticate success, global_id 204435 2021-05-25T12:07:36.181 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.159+0000 7fb41f7fe700 1 -- 172.21.15.102:0/2161469393 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fb410001610 con 0x7fb428090d70 2021-05-25T12:07:36.181 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.159+0000 7fb41f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:36.182 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.159+0000 7fb41f7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:07:36.182 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.159+0000 7fb41f7fe700 10 monclient: dump: 2021-05-25T12:07:36.182 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:36.182 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:36.183 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:36.183 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:36.183 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:36.183 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:36.183 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:36.184 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:36.184 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:36.184 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:36.184 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.159+0000 7fb42f03b700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:07:36.185 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.159+0000 7fb41f7fe700 1 -- 172.21.15.102:0/2161469393 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb410001880 con 0x7fb428090d70 2021-05-25T12:07:36.185 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.159+0000 7fb41f7fe700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:07:36.185 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.159+0000 7fb41f7fe700 1 -- 172.21.15.102:0/2161469393 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fb410001c10 con 0x7fb428090d70 2021-05-25T12:07:36.185 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.160+0000 7fb41f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:36.185 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.160+0000 7fb41f7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:07:36.186 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.160+0000 7fb41f7fe700 10 monclient: dump: 2021-05-25T12:07:36.186 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:36.186 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:36.186 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:36.187 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:36.187 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:36.187 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:36.187 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:36.188 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:36.189 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:36.189 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:36.189 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.160+0000 7fb42f03b700 10 monclient: get_monmap_and_config success 2021-05-25T12:07:36.189 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.160+0000 7fb42f03b700 4 set_mon_vals no callback set 2021-05-25T12:07:36.189 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.160+0000 7fb42f03b700 10 monclient: shutdown 2021-05-25T12:07:36.190 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.160+0000 7fb42f03b700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:07:36.190 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.160+0000 7fb42f03b700 1 -- 172.21.15.102:0/2161469393 >> v1:172.21.15.102:6789/0 conn(0x7fb428090d70 legacy=0x7fb428091200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:36.190 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.160+0000 7fb42f03b700 1 -- 172.21.15.102:0/2161469393 shutdown_connections 2021-05-25T12:07:36.190 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.160+0000 7fb42f03b700 1 -- 172.21.15.102:0/2161469393 >> 172.21.15.102:0/2161469393 conn(0x7fb42809b370 msgr2=0x7fb428097430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:07:36.191 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.160+0000 7fb42f03b700 1 -- 172.21.15.102:0/2161469393 shutdown_connections 2021-05-25T12:07:36.191 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.160+0000 7fb42f03b700 1 -- 172.21.15.102:0/2161469393 wait complete. 2021-05-25T12:07:36.191 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.161+0000 7fb42f03b700 10 monclient: build_initial_monmap 2021-05-25T12:07:36.191 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.161+0000 7fb42f03b700 10 monclient: monmap: 2021-05-25T12:07:36.191 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:07:36.192 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:07:36.192 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:07:36.192 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:07:36.192 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:07:36.192 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:07:36.193 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:07:36.193 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:07:36.193 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:07:36.193 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:36.194 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.161+0000 7fb42f03b700 1 librados: starting msgr at 2021-05-25T12:07:36.194 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.161+0000 7fb42f03b700 1 librados: starting objecter 2021-05-25T12:07:36.194 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.161+0000 7fb42f03b700 1 Processor -- start 2021-05-25T12:07:36.194 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.161+0000 7fb42f03b700 1 -- start start 2021-05-25T12:07:36.194 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.161+0000 7fb42f03b700 1 librados: setting wanted keys 2021-05-25T12:07:36.195 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.161+0000 7fb42f03b700 1 librados: calling monclient init 2021-05-25T12:07:36.195 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.161+0000 7fb42f03b700 10 monclient: init 2021-05-25T12:07:36.195 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.161+0000 7fb42f03b700 10 monclient: _reopen_session rank -1 2021-05-25T12:07:36.195 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.161+0000 7fb42f03b700 10 monclient: _add_conns ranks=[1,0,2] 2021-05-25T12:07:36.196 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.162+0000 7fb42f03b700 10 monclient(hunting): picked mon.noname-b con 0x7fb428090990 addr v1:172.21.15.181:6789/0 2021-05-25T12:07:36.196 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.162+0000 7fb42f03b700 10 monclient(hunting): picked mon.noname-a con 0x7fb428090d70 addr v1:172.21.15.102:6789/0 2021-05-25T12:07:36.196 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.162+0000 7fb42f03b700 10 monclient(hunting): picked mon.noname-c con 0x7fb42812e330 addr v1:172.21.15.102:6790/0 2021-05-25T12:07:36.196 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.162+0000 7fb42f03b700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb428137760 con 0x7fb428090d70 2021-05-25T12:07:36.196 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.162+0000 7fb42f03b700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb428068750 con 0x7fb428090990 2021-05-25T12:07:36.197 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.162+0000 7fb42f03b700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4280689f0 con 0x7fb42812e330 2021-05-25T12:07:36.197 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.162+0000 7fb42f03b700 10 monclient(hunting): _renew_subs 2021-05-25T12:07:36.197 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.162+0000 7fb42f03b700 10 monclient(hunting): authenticate will time out at 1256.759070s 2021-05-25T12:07:36.197 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.162+0000 7fb42d5d8700 1 -- >> v1:172.21.15.102:6790/0 conn(0x7fb42812e330 legacy=0x7fb42813ec90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.102:6790/0 2021-05-25T12:07:36.197 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.162+0000 7fb41ffff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fb428090d70 0x7fb428137050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33922/0 (socket says 172.21.15.102:33922) 2021-05-25T12:07:36.198 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.162+0000 7fb41ffff700 1 -- 172.21.15.102:0/3342375067 learned_addr learned my addr 172.21.15.102:0/3342375067 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:07:36.198 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.162+0000 7fb41cff9700 10 client.?.objecter ms_handle_connect 0x7fb428090d70 2021-05-25T12:07:36.198 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.162+0000 7fb41cff9700 10 client.?.objecter resend_mon_ops 2021-05-25T12:07:36.198 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.162+0000 7fb41cff9700 10 client.?.objecter ms_handle_connect 0x7fb428090990 2021-05-25T12:07:36.199 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.162+0000 7fb41cff9700 10 client.?.objecter resend_mon_ops 2021-05-25T12:07:36.199 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.162+0000 7fb41cff9700 1 -- 172.21.15.102:0/3342375067 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4112105463 0 0) 0x7fb428137760 con 0x7fb428090d70 2021-05-25T12:07:36.199 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.162+0000 7fb41cff9700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:36.199 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.162+0000 7fb41cff9700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:36.199 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.162+0000 7fb41cff9700 10 monclient(hunting): my global_id is 204438 2021-05-25T12:07:36.200 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.162+0000 7fb41cff9700 1 -- 172.21.15.102:0/3342375067 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb408004c10 con 0x7fb428090d70 2021-05-25T12:07:36.200 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.162+0000 7fb41cff9700 1 -- 172.21.15.102:0/3342375067 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3559017645 0 0) 0x7fb428068750 con 0x7fb428090990 2021-05-25T12:07:36.200 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.162+0000 7fb41cff9700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:36.200 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.162+0000 7fb41cff9700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:36.200 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.162+0000 7fb41cff9700 10 monclient(hunting): my global_id is 224305 2021-05-25T12:07:36.201 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.162+0000 7fb41cff9700 1 -- 172.21.15.102:0/3342375067 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb428137760 con 0x7fb428090990 2021-05-25T12:07:36.201 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.162+0000 7fb41cff9700 1 -- 172.21.15.102:0/3342375067 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1244209674 0 0) 0x7fb408004c10 con 0x7fb428090d70 2021-05-25T12:07:36.201 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.162+0000 7fb41cff9700 1 -- 172.21.15.102:0/3342375067 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb428068750 con 0x7fb428090d70 2021-05-25T12:07:36.201 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.162+0000 7fb41cff9700 1 -- 172.21.15.102:0/3342375067 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fb410001690 con 0x7fb428090d70 2021-05-25T12:07:36.201 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.162+0000 7fb41cff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:36.202 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.162+0000 7fb41cff9700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:07:36.202 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.162+0000 7fb41cff9700 10 monclient(hunting): dump: 2021-05-25T12:07:36.202 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:36.202 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:36.203 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:36.203 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:36.203 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:36.203 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:36.203 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:36.204 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:36.204 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:36.204 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:36.204 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.162+0000 7fb41cff9700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:07:36.204 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.162+0000 7fb41cff9700 1 -- 172.21.15.102:0/3342375067 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 899147348 0 0) 0x7fb428068750 con 0x7fb428090d70 2021-05-25T12:07:36.205 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.162+0000 7fb41cff9700 1 -- 172.21.15.102:0/3342375067 >> v1:172.21.15.102:6790/0 conn(0x7fb42812e330 legacy=0x7fb42813ec90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-05-25T12:07:36.205 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.162+0000 7fb41cff9700 1 -- 172.21.15.102:0/3342375067 >> v1:172.21.15.181:6789/0 conn(0x7fb428090990 legacy=0x7fb428136940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:36.205 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.162+0000 7fb41cff9700 10 monclient: _finish_hunting 0 2021-05-25T12:07:36.205 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.162+0000 7fb41cff9700 1 monclient: found mon.a 2021-05-25T12:07:36.205 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.162+0000 7fb41cff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:07:36.206 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.162+0000 7fb41cff9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:07:36.206 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.162+0000 7fb41cff9700 1 -- 172.21.15.102:0/3342375067 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb428066880 con 0x7fb428090d70 2021-05-25T12:07:36.206 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.162+0000 7fb41cff9700 10 monclient: _finish_auth 0 2021-05-25T12:07:36.206 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.162+0000 7fb41cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:07:36.206 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.162+0000 7fb42f03b700 5 monclient: authenticate success, global_id 204438 2021-05-25T12:07:36.207 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.162+0000 7fb42f03b700 10 monclient: _renew_subs 2021-05-25T12:07:36.207 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.162+0000 7fb42f03b700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:07:36.207 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.162+0000 7fb42f03b700 1 -- 172.21.15.102:0/3342375067 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb428067520 con 0x7fb428090d70 2021-05-25T12:07:36.207 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.162+0000 7fb42f03b700 10 client.204438.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:07:36.208 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.162+0000 7fb42f03b700 10 monclient: _renew_subs 2021-05-25T12:07:36.208 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.162+0000 7fb42f03b700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:07:36.208 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.162+0000 7fb42f03b700 1 -- 172.21.15.102:0/3342375067 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4281266c0 con 0x7fb428090d70 2021-05-25T12:07:36.208 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.162+0000 7fb42f03b700 1 librados: init done 2021-05-25T12:07:36.208 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.162+0000 7fb41cff9700 1 -- 172.21.15.102:0/3342375067 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb4100049e0 con 0x7fb428090d70 2021-05-25T12:07:36.209 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.162+0000 7fb41cff9700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:07:36.209 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.162+0000 7fb41cff9700 1 -- 172.21.15.102:0/3342375067 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fb410004d70 con 0x7fb428090d70 2021-05-25T12:07:36.209 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.162+0000 7fb41cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:36.209 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.162+0000 7fb41effd700 4 set_mon_vals no callback set 2021-05-25T12:07:36.209 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.162+0000 7fb41cff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:07:36.210 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.162+0000 7fb41cff9700 10 monclient: dump: 2021-05-25T12:07:36.210 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:36.210 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:36.210 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:36.210 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:36.211 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:36.211 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:36.211 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:36.211 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:36.211 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:36.212 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:36.212 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.163+0000 7fb41cff9700 1 -- 172.21.15.102:0/3342375067 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7fb410004f50 con 0x7fb428090d70 2021-05-25T12:07:36.212 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.163+0000 7fb42f03b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:07:36.212 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.163+0000 7fb42f03b700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:07:36.213 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.163+0000 7fb42f03b700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:07:36.213 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.163+0000 7fb42f03b700 1 -- 172.21.15.102:0/3342375067 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb400004db0 con 0x7fb428090d70 2021-05-25T12:07:36.213 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.163+0000 7fb41cff9700 1 -- 172.21.15.102:0/3342375067 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(246..246 src has 1..246) v4 ==== 5858+0+0 (unknown 4267717717 0 0) 0x7fb41003dad0 con 0x7fb428090d70 2021-05-25T12:07:36.213 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.163+0000 7fb41cff9700 10 client.204438.objecter ms_dispatch 0x7fb428001550 osd_map(246..246 src has 1..246) v4 2021-05-25T12:07:36.213 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.163+0000 7fb41cff9700 3 client.204438.objecter handle_osd_map got epochs [246,246] > 0 2021-05-25T12:07:36.214 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.163+0000 7fb41cff9700 3 client.204438.objecter handle_osd_map decoding full epoch 246 2021-05-25T12:07:36.214 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.163+0000 7fb41cff9700 20 client.204438.objecter dump_active .. 0 homeless 2021-05-25T12:07:36.214 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.163+0000 7fb42cdd7700 10 monclient: get_auth_request con 0x7fb408032fb0 auth_method 0 2021-05-25T12:07:36.214 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.164+0000 7fb41cff9700 10 client.204438.objecter ms_handle_connect 0x7fb408032fb0 2021-05-25T12:07:36.214 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.168+0000 7fb41cff9700 1 -- 172.21.15.102:0/3342375067 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7fb410013120 con 0x7fb428090d70 2021-05-25T12:07:36.215 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.168+0000 7fb41cff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:07:36.215 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.168+0000 7fb41cff9700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:07:36.377 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.375+0000 7fb42f03b700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2021-05-25T12:07:36.377 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.375+0000 7fb42f03b700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2021-05-25T12:07:36.377 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.375+0000 7fb42f03b700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:07:36.378 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.375+0000 7fb42f03b700 1 -- 172.21.15.102:0/3342375067 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fb400005060 con 0x7fb428090d70 2021-05-25T12:07:36.378 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.375+0000 7fb41cff9700 1 -- 172.21.15.102:0/3342375067 <== mon.0 v1:172.21.15.102:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1342 (unknown 254746657 0 1650817163) 0x7fb410012ef0 con 0x7fb428090d70 2021-05-25T12:07:36.378 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.375+0000 7fb41cff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2021-05-25T12:07:36.378 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.376+0000 7fb41cff9700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:07:36.379 INFO:teuthology.orchestra.run.smithi102.stdout:{"election_epoch":64,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":3,"features":{"quorum_con":"4540138303579357183","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"20ee91b6-8935-4b98-b868-85c03789ccc4","modified":"2021-05-25T11:59:11.011522Z","created":"2021-05-25T11:59:11.011522Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6790","nonce":0}]},"addr":"172.21.15.102:6790/0","public_addr":"172.21.15.102:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2021-05-25T12:07:36.380 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.377+0000 7fb42f03b700 10 librados: watch_flush enter 2021-05-25T12:07:36.381 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.377+0000 7fb42f03b700 10 librados: watch_flush exit 2021-05-25T12:07:36.381 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.377+0000 7fb42f03b700 20 client.204438.objecter shutdown clearing up homeless session... 2021-05-25T12:07:36.381 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.377+0000 7fb42f03b700 10 client.204438.objecter successfully canceled tick 2021-05-25T12:07:36.382 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.377+0000 7fb42f03b700 1 -- 172.21.15.102:0/3342375067 >> v1:172.21.15.181:6812/29582 conn(0x7fb408032fb0 legacy=0x7fb408035470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:36.382 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.377+0000 7fb42f03b700 10 monclient: shutdown 2021-05-25T12:07:36.382 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.377+0000 7fb42f03b700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:07:36.382 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.377+0000 7fb42f03b700 1 -- 172.21.15.102:0/3342375067 >> v1:172.21.15.102:6789/0 conn(0x7fb428090d70 legacy=0x7fb428137050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:36.383 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.377+0000 7fb42f03b700 1 -- 172.21.15.102:0/3342375067 shutdown_connections 2021-05-25T12:07:36.383 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.377+0000 7fb42f03b700 1 -- 172.21.15.102:0/3342375067 >> 172.21.15.102:0/3342375067 conn(0x7fb42809b370 msgr2=0x7fb4281292f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:07:36.383 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.377+0000 7fb42f03b700 1 -- 172.21.15.102:0/3342375067 shutdown_connections 2021-05-25T12:07:36.383 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.377+0000 7fb42f03b700 1 -- 172.21.15.102:0/3342375067 wait complete. 2021-05-25T12:07:36.383 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:36.377+0000 7fb42f03b700 1 librados: shutdown 2021-05-25T12:07:39.391 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2021-05-25T12:07:39.510 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.508+0000 7f1bb4a23700 10 monclient: get_monmap_and_config 2021-05-25T12:07:39.511 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.508+0000 7f1bb4a23700 10 monclient: build_initial_monmap 2021-05-25T12:07:39.511 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.508+0000 7f1bb4a23700 10 monclient: monmap: 2021-05-25T12:07:39.511 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:07:39.512 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:39.512 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:07:39.509728+0000 2021-05-25T12:07:39.512 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:07:39.509728+0000 2021-05-25T12:07:39.512 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:07:39.513 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:39.513 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:07:39.513 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:07:39.513 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:07:39.514 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:39.514 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.509+0000 7f1bb4a23700 1 Processor -- start 2021-05-25T12:07:39.514 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.509+0000 7f1bb4a23700 1 -- start start 2021-05-25T12:07:39.514 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.509+0000 7f1bb4a23700 10 monclient: init 2021-05-25T12:07:39.529 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.510+0000 7f1bb4a23700 10 monclient: _reopen_session rank -1 2021-05-25T12:07:39.529 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.510+0000 7f1bb4a23700 10 monclient: _add_conns ranks=[2,1,0] 2021-05-25T12:07:39.529 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.510+0000 7f1bb4a23700 10 monclient(hunting): picked mon.noname-b con 0x7f1bb0090990 addr v1:172.21.15.102:6790/0 2021-05-25T12:07:39.530 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.510+0000 7f1bb4a23700 10 monclient(hunting): picked mon.noname-c con 0x7f1bb0090d70 addr v1:172.21.15.181:6789/0 2021-05-25T12:07:39.530 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.510+0000 7f1bb4a23700 10 monclient(hunting): picked mon.noname-a con 0x7f1bb012e330 addr v1:172.21.15.102:6789/0 2021-05-25T12:07:39.530 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.510+0000 7f1bb4a23700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bb0096fa0 con 0x7f1bb012e330 2021-05-25T12:07:39.530 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.510+0000 7f1bb4a23700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bb00926a0 con 0x7f1bb0090d70 2021-05-25T12:07:39.531 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.510+0000 7f1bb4a23700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bb0092940 con 0x7f1bb0090990 2021-05-25T12:07:39.531 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.510+0000 7f1bb4a23700 10 monclient(hunting): _renew_subs 2021-05-25T12:07:39.531 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.510+0000 7f1bb4a23700 10 monclient(hunting): authenticate will time out at 1260.107128s 2021-05-25T12:07:39.531 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.510+0000 7f1bae59c700 1 -- >> v1:172.21.15.102:6790/0 conn(0x7f1bb0090990 legacy=0x7f1bb0093240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6790/0 2021-05-25T12:07:39.531 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.510+0000 7f1baed9d700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f1bb012e330 0x7f1bb0130720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33936/0 (socket says 172.21.15.102:33936) 2021-05-25T12:07:39.532 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.510+0000 7f1baed9d700 1 -- 172.21.15.102:0/1718416540 learned_addr learned my addr 172.21.15.102:0/1718416540 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:07:39.532 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.510+0000 7f1bad59a700 1 -- 172.21.15.102:0/1718416540 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2896344278 0 0) 0x7f1bb0096fa0 con 0x7f1bb012e330 2021-05-25T12:07:39.532 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.510+0000 7f1bad59a700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:39.532 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.510+0000 7f1bad59a700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:39.532 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.510+0000 7f1bad59a700 10 monclient(hunting): my global_id is 204441 2021-05-25T12:07:39.533 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.510+0000 7f1bad59a700 1 -- 172.21.15.102:0/1718416540 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b94004380 con 0x7f1bb012e330 2021-05-25T12:07:39.533 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.510+0000 7f1bad59a700 1 -- 172.21.15.102:0/1718416540 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4276864790 0 0) 0x7f1b94004380 con 0x7f1bb012e330 2021-05-25T12:07:39.533 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.510+0000 7f1bad59a700 1 -- 172.21.15.102:0/1718416540 >> v1:172.21.15.102:6790/0 conn(0x7f1bb0090990 legacy=0x7f1bb0093240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-05-25T12:07:39.533 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.510+0000 7f1bad59a700 1 -- 172.21.15.102:0/1718416540 >> v1:172.21.15.181:6789/0 conn(0x7f1bb0090d70 legacy=0x7f1bb0091200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:39.533 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.510+0000 7f1bad59a700 10 monclient: _finish_hunting 0 2021-05-25T12:07:39.534 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.510+0000 7f1bad59a700 1 monclient: found mon.noname-a 2021-05-25T12:07:39.534 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.510+0000 7f1bad59a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:07:39.534 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.510+0000 7f1bad59a700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T12:07:39.534 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.510+0000 7f1bad59a700 1 -- 172.21.15.102:0/1718416540 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bb0092be0 con 0x7f1bb012e330 2021-05-25T12:07:39.534 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.510+0000 7f1bad59a700 10 monclient: _finish_auth 0 2021-05-25T12:07:39.535 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.510+0000 7f1bad59a700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:07:39.535 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.510+0000 7f1bb4a23700 5 monclient: authenticate success, global_id 204441 2021-05-25T12:07:39.535 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.510+0000 7f1bad59a700 1 -- 172.21.15.102:0/1718416540 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f1ba0001990 con 0x7f1bb012e330 2021-05-25T12:07:39.536 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.510+0000 7f1bb4a23700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:07:39.536 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.510+0000 7f1bad59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:39.536 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.510+0000 7f1bad59a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:07:39.536 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.510+0000 7f1bad59a700 10 monclient: dump: 2021-05-25T12:07:39.536 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:39.537 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:39.537 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:39.537 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:39.537 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:39.538 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:39.538 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:39.538 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:39.538 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:39.538 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:39.539 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.510+0000 7f1bb4a23700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:07:39.539 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.510+0000 7f1bad59a700 1 -- 172.21.15.102:0/1718416540 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1ba0001c40 con 0x7f1bb012e330 2021-05-25T12:07:39.539 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.510+0000 7f1bad59a700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:07:39.539 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.510+0000 7f1bb4a23700 10 monclient: get_monmap_and_config success 2021-05-25T12:07:39.540 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.510+0000 7f1bb4a23700 4 set_mon_vals no callback set 2021-05-25T12:07:39.540 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.510+0000 7f1bad59a700 1 -- 172.21.15.102:0/1718416540 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f1ba0001fd0 con 0x7f1bb012e330 2021-05-25T12:07:39.540 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.510+0000 7f1bad59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:39.540 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.510+0000 7f1bad59a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:07:39.540 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.510+0000 7f1bad59a700 10 monclient: dump: 2021-05-25T12:07:39.541 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:39.541 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:39.541 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:39.541 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:39.542 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:39.542 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:39.542 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:39.542 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:39.542 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:39.543 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:39.543 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.511+0000 7f1bb4a23700 10 monclient: shutdown 2021-05-25T12:07:39.543 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.511+0000 7f1bb4a23700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:07:39.543 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.511+0000 7f1bb4a23700 1 -- 172.21.15.102:0/1718416540 >> v1:172.21.15.102:6789/0 conn(0x7f1bb012e330 legacy=0x7f1bb0130720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:39.543 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.511+0000 7f1bb4a23700 1 -- 172.21.15.102:0/1718416540 shutdown_connections 2021-05-25T12:07:39.544 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.511+0000 7f1bb4a23700 1 -- 172.21.15.102:0/1718416540 >> 172.21.15.102:0/1718416540 conn(0x7f1bb009b370 msgr2=0x7f1bb0097430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:07:39.544 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.511+0000 7f1bb4a23700 1 -- 172.21.15.102:0/1718416540 shutdown_connections 2021-05-25T12:07:39.544 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.511+0000 7f1bb4a23700 1 -- 172.21.15.102:0/1718416540 wait complete. 2021-05-25T12:07:39.544 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.511+0000 7f1bb4a23700 10 monclient: build_initial_monmap 2021-05-25T12:07:39.545 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.511+0000 7f1bb4a23700 10 monclient: monmap: 2021-05-25T12:07:39.545 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:07:39.545 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:07:39.545 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:07:39.545 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:07:39.546 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:07:39.546 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:07:39.546 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:07:39.546 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:07:39.547 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:07:39.547 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:39.547 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.512+0000 7f1bb4a23700 1 librados: starting msgr at 2021-05-25T12:07:39.547 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.512+0000 7f1bb4a23700 1 librados: starting objecter 2021-05-25T12:07:39.548 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.512+0000 7f1bb4a23700 1 Processor -- start 2021-05-25T12:07:39.548 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.512+0000 7f1bb4a23700 1 -- start start 2021-05-25T12:07:39.548 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.512+0000 7f1bb4a23700 1 librados: setting wanted keys 2021-05-25T12:07:39.548 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.512+0000 7f1bb4a23700 1 librados: calling monclient init 2021-05-25T12:07:39.549 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.512+0000 7f1bb4a23700 10 monclient: init 2021-05-25T12:07:39.549 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.512+0000 7f1bb4a23700 10 monclient: _reopen_session rank -1 2021-05-25T12:07:39.549 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.512+0000 7f1bb4a23700 10 monclient: _add_conns ranks=[0,2,1] 2021-05-25T12:07:39.550 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.512+0000 7f1bb4a23700 10 monclient(hunting): picked mon.noname-a con 0x7f1bb0090990 addr v1:172.21.15.102:6789/0 2021-05-25T12:07:39.550 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.512+0000 7f1bb4a23700 10 monclient(hunting): picked mon.noname-c con 0x7f1bb0090d70 addr v1:172.21.15.102:6790/0 2021-05-25T12:07:39.550 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.512+0000 7f1bb4a23700 10 monclient(hunting): picked mon.noname-b con 0x7f1bb012e330 addr v1:172.21.15.181:6789/0 2021-05-25T12:07:39.550 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.512+0000 7f1bb4a23700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bb0114ce0 con 0x7f1bb0090990 2021-05-25T12:07:39.551 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.512+0000 7f1bb4a23700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bb0091fe0 con 0x7f1bb012e330 2021-05-25T12:07:39.551 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.512+0000 7f1bb4a23700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bb012a230 con 0x7f1bb0090d70 2021-05-25T12:07:39.551 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.512+0000 7f1bb4a23700 10 monclient(hunting): _renew_subs 2021-05-25T12:07:39.551 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.512+0000 7f1bb4a23700 10 monclient(hunting): authenticate will time out at 1260.109698s 2021-05-25T12:07:39.551 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.512+0000 7f1badd9b700 1 -- >> v1:172.21.15.102:6790/0 conn(0x7f1bb0090d70 legacy=0x7f1bb0139300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.102:6790/0 2021-05-25T12:07:39.552 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.512+0000 7f1bae59c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f1bb0090990 0x7f1bb0138bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33938/0 (socket says 172.21.15.102:33938) 2021-05-25T12:07:39.552 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.512+0000 7f1bae59c700 1 -- 172.21.15.102:0/3947809978 learned_addr learned my addr 172.21.15.102:0/3947809978 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:07:39.552 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.512+0000 7f1b9e7fc700 10 client.?.objecter ms_handle_connect 0x7f1bb0090990 2021-05-25T12:07:39.552 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.512+0000 7f1b9e7fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:07:39.553 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.512+0000 7f1b9e7fc700 1 -- 172.21.15.102:0/3947809978 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3705352039 0 0) 0x7f1bb0114ce0 con 0x7f1bb0090990 2021-05-25T12:07:39.553 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.512+0000 7f1b9e7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:39.553 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.512+0000 7f1b9e7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:39.553 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.512+0000 7f1b9e7fc700 10 monclient(hunting): my global_id is 204444 2021-05-25T12:07:39.553 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.512+0000 7f1b9e7fc700 1 -- 172.21.15.102:0/3947809978 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b8c004c10 con 0x7f1bb0090990 2021-05-25T12:07:39.554 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.512+0000 7f1b9e7fc700 10 client.?.objecter ms_handle_connect 0x7f1bb012e330 2021-05-25T12:07:39.554 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.512+0000 7f1b9e7fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:07:39.554 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.513+0000 7f1b9e7fc700 1 -- 172.21.15.102:0/3947809978 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3890994190 0 0) 0x7f1bb0091fe0 con 0x7f1bb012e330 2021-05-25T12:07:39.554 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.513+0000 7f1b9e7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:39.555 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.513+0000 7f1b9e7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:39.555 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.513+0000 7f1b9e7fc700 10 monclient(hunting): my global_id is 224311 2021-05-25T12:07:39.555 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.513+0000 7f1b9e7fc700 1 -- 172.21.15.102:0/3947809978 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bb0114ce0 con 0x7f1bb012e330 2021-05-25T12:07:39.555 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.513+0000 7f1b9e7fc700 1 -- 172.21.15.102:0/3947809978 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 353175455 0 0) 0x7f1b8c004c10 con 0x7f1bb0090990 2021-05-25T12:07:39.555 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.513+0000 7f1b9e7fc700 1 -- 172.21.15.102:0/3947809978 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1bb0091fe0 con 0x7f1bb0090990 2021-05-25T12:07:39.556 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.513+0000 7f1b9e7fc700 1 -- 172.21.15.102:0/3947809978 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f1b98003eb0 con 0x7f1bb0090990 2021-05-25T12:07:39.556 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.513+0000 7f1b9e7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:39.556 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.513+0000 7f1b9e7fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:07:39.556 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.513+0000 7f1b9e7fc700 10 monclient(hunting): dump: 2021-05-25T12:07:39.556 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:39.557 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:39.557 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:39.557 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:39.557 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:39.557 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:39.558 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:39.558 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:39.558 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:39.558 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:39.558 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.513+0000 7f1b9e7fc700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:07:39.559 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.513+0000 7f1b9e7fc700 1 -- 172.21.15.102:0/3947809978 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2081080323 0 0) 0x7f1bb0091fe0 con 0x7f1bb0090990 2021-05-25T12:07:39.559 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.513+0000 7f1b9e7fc700 1 -- 172.21.15.102:0/3947809978 >> v1:172.21.15.102:6790/0 conn(0x7f1bb0090d70 legacy=0x7f1bb0139300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-05-25T12:07:39.559 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.513+0000 7f1b9e7fc700 1 -- 172.21.15.102:0/3947809978 >> v1:172.21.15.181:6789/0 conn(0x7f1bb012e330 legacy=0x7f1bb0140d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:39.560 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.513+0000 7f1b9e7fc700 10 monclient: _finish_hunting 0 2021-05-25T12:07:39.560 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.513+0000 7f1b9e7fc700 1 monclient: found mon.a 2021-05-25T12:07:39.560 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.513+0000 7f1b9e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:07:39.560 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.513+0000 7f1b9e7fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:07:39.560 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.513+0000 7f1b9e7fc700 1 -- 172.21.15.102:0/3947809978 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bb0112be0 con 0x7f1bb0090990 2021-05-25T12:07:39.561 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.513+0000 7f1b9e7fc700 10 monclient: _finish_auth 0 2021-05-25T12:07:39.561 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.513+0000 7f1b9e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:07:39.561 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.513+0000 7f1bb4a23700 5 monclient: authenticate success, global_id 204444 2021-05-25T12:07:39.561 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.513+0000 7f1bb4a23700 10 monclient: _renew_subs 2021-05-25T12:07:39.562 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.513+0000 7f1bb4a23700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:07:39.562 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.513+0000 7f1bb4a23700 1 -- 172.21.15.102:0/3947809978 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bb0113650 con 0x7f1bb0090990 2021-05-25T12:07:39.562 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.513+0000 7f1bb4a23700 10 client.204444.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:07:39.562 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.513+0000 7f1bb4a23700 10 monclient: _renew_subs 2021-05-25T12:07:39.562 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.513+0000 7f1bb4a23700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:07:39.563 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.513+0000 7f1bb4a23700 1 -- 172.21.15.102:0/3947809978 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bb0128850 con 0x7f1bb0090990 2021-05-25T12:07:39.563 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.513+0000 7f1bb4a23700 1 librados: init done 2021-05-25T12:07:39.563 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.513+0000 7f1b9e7fc700 1 -- 172.21.15.102:0/3947809978 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1b98004190 con 0x7f1bb0090990 2021-05-25T12:07:39.563 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.513+0000 7f1b9e7fc700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:07:39.564 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.513+0000 7f1b9e7fc700 1 -- 172.21.15.102:0/3947809978 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f1b98004520 con 0x7f1bb0090990 2021-05-25T12:07:39.564 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.513+0000 7f1b9e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:39.564 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.513+0000 7f1b9e7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:07:39.564 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.513+0000 7f1bacd99700 4 set_mon_vals no callback set 2021-05-25T12:07:39.564 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.513+0000 7f1b9e7fc700 10 monclient: dump: 2021-05-25T12:07:39.565 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:39.565 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:39.565 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:39.565 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:39.566 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:39.566 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:39.566 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:39.566 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:39.567 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:39.567 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:39.567 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.514+0000 7f1b9e7fc700 1 -- 172.21.15.102:0/3947809978 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f1b98004700 con 0x7f1bb0090990 2021-05-25T12:07:39.567 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.514+0000 7f1bb4a23700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:07:39.567 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.514+0000 7f1bb4a23700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:07:39.568 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.514+0000 7f1bb4a23700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:07:39.568 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.514+0000 7f1b9e7fc700 1 -- 172.21.15.102:0/3947809978 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(246..246 src has 1..246) v4 ==== 5858+0+0 (unknown 4267717717 0 0) 0x7f1b9803d7c0 con 0x7f1bb0090990 2021-05-25T12:07:39.568 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.514+0000 7f1b9e7fc700 10 client.204444.objecter ms_dispatch 0x7f1bb0001550 osd_map(246..246 src has 1..246) v4 2021-05-25T12:07:39.568 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.514+0000 7f1b9e7fc700 3 client.204444.objecter handle_osd_map got epochs [246,246] > 0 2021-05-25T12:07:39.569 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.514+0000 7f1b9e7fc700 3 client.204444.objecter handle_osd_map decoding full epoch 246 2021-05-25T12:07:39.569 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.514+0000 7f1bb4a23700 1 -- 172.21.15.102:0/3947809978 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b90004db0 con 0x7f1bb0090990 2021-05-25T12:07:39.569 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.514+0000 7f1b9e7fc700 20 client.204444.objecter dump_active .. 0 homeless 2021-05-25T12:07:39.569 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.514+0000 7f1badd9b700 10 monclient: get_auth_request con 0x7f1b8c033180 auth_method 0 2021-05-25T12:07:39.569 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.514+0000 7f1b9e7fc700 10 client.204444.objecter ms_handle_connect 0x7f1b8c033180 2021-05-25T12:07:39.570 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.518+0000 7f1b9e7fc700 1 -- 172.21.15.102:0/3947809978 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7f1b98004ee0 con 0x7f1bb0090990 2021-05-25T12:07:39.570 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.518+0000 7f1b9e7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:07:39.570 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.518+0000 7f1b9e7fc700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:07:39.728 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.726+0000 7f1bb4a23700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2021-05-25T12:07:39.728 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.726+0000 7f1bb4a23700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2021-05-25T12:07:39.728 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.726+0000 7f1bb4a23700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:07:39.729 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.726+0000 7f1bb4a23700 1 -- 172.21.15.102:0/3947809978 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f1b90005060 con 0x7f1bb0090990 2021-05-25T12:07:39.729 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.726+0000 7f1b9e7fc700 1 -- 172.21.15.102:0/3947809978 <== mon.0 v1:172.21.15.102:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1342 (unknown 254746657 0 246051258) 0x7f1b98012b80 con 0x7f1bb0090990 2021-05-25T12:07:39.729 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.726+0000 7f1b9e7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2021-05-25T12:07:39.729 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.726+0000 7f1b9e7fc700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:07:39.730 INFO:teuthology.orchestra.run.smithi102.stdout:{"election_epoch":64,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":7,"features":{"quorum_con":"4540138303579357183","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"20ee91b6-8935-4b98-b868-85c03789ccc4","modified":"2021-05-25T11:59:11.011522Z","created":"2021-05-25T11:59:11.011522Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6790","nonce":0}]},"addr":"172.21.15.102:6790/0","public_addr":"172.21.15.102:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2021-05-25T12:07:39.731 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.728+0000 7f1bb4a23700 10 librados: watch_flush enter 2021-05-25T12:07:39.731 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.728+0000 7f1bb4a23700 10 librados: watch_flush exit 2021-05-25T12:07:39.732 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.728+0000 7f1bb4a23700 20 client.204444.objecter shutdown clearing up homeless session... 2021-05-25T12:07:39.732 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.728+0000 7f1bb4a23700 10 client.204444.objecter successfully canceled tick 2021-05-25T12:07:39.732 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.728+0000 7f1bb4a23700 1 -- 172.21.15.102:0/3947809978 >> v1:172.21.15.181:6812/29582 conn(0x7f1b8c033180 legacy=0x7f1b8c035640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:39.732 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.728+0000 7f1bb4a23700 10 monclient: shutdown 2021-05-25T12:07:39.733 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.728+0000 7f1bb4a23700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:07:39.733 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.728+0000 7f1bb4a23700 1 -- 172.21.15.102:0/3947809978 >> v1:172.21.15.102:6789/0 conn(0x7f1bb0090990 legacy=0x7f1bb0138bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:39.733 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.728+0000 7f1bb4a23700 1 -- 172.21.15.102:0/3947809978 shutdown_connections 2021-05-25T12:07:39.733 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.728+0000 7f1bb4a23700 1 -- 172.21.15.102:0/3947809978 >> 172.21.15.102:0/3947809978 conn(0x7f1bb009b370 msgr2=0x7f1bb01293a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:07:39.734 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.728+0000 7f1bb4a23700 1 -- 172.21.15.102:0/3947809978 shutdown_connections 2021-05-25T12:07:39.734 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.728+0000 7f1bb4a23700 1 -- 172.21.15.102:0/3947809978 wait complete. 2021-05-25T12:07:39.734 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:39.728+0000 7f1bb4a23700 1 librados: shutdown 2021-05-25T12:07:42.741 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2021-05-25T12:07:42.865 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:42.863+0000 7f769d224700 10 monclient: get_monmap_and_config 2021-05-25T12:07:42.865 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:42.863+0000 7f769d224700 10 monclient: build_initial_monmap 2021-05-25T12:07:42.865 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:42.863+0000 7f769d224700 10 monclient: monmap: 2021-05-25T12:07:42.866 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:07:42.866 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:42.866 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:07:42.864446+0000 2021-05-25T12:07:42.866 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:07:42.864446+0000 2021-05-25T12:07:42.867 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:07:42.867 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:42.867 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:07:42.867 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:07:42.867 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:07:42.868 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:42.868 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:42.864+0000 7f769d224700 1 Processor -- start 2021-05-25T12:07:42.868 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:42.864+0000 7f769d224700 1 -- start start 2021-05-25T12:07:42.868 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:42.864+0000 7f769d224700 10 monclient: init 2021-05-25T12:07:42.870 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:42.864+0000 7f769d224700 10 monclient: _reopen_session rank -1 2021-05-25T12:07:42.870 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:42.864+0000 7f769d224700 10 monclient: _add_conns ranks=[0,2,1] 2021-05-25T12:07:42.870 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:42.864+0000 7f769d224700 10 monclient(hunting): picked mon.noname-a con 0x7f7698069190 addr v1:172.21.15.102:6789/0 2021-05-25T12:07:42.870 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:42.864+0000 7f769d224700 10 monclient(hunting): picked mon.noname-b con 0x7f7698069c80 addr v1:172.21.15.102:6790/0 2021-05-25T12:07:42.871 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:42.864+0000 7f769d224700 10 monclient(hunting): picked mon.noname-c con 0x7f7698096ab0 addr v1:172.21.15.181:6789/0 2021-05-25T12:07:42.871 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:42.864+0000 7f769d224700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76980959b0 con 0x7f7698069190 2021-05-25T12:07:42.871 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:42.864+0000 7f769d224700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76980914e0 con 0x7f7698096ab0 2021-05-25T12:07:42.871 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:42.864+0000 7f769d224700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7698095c10 con 0x7f7698069c80 2021-05-25T12:07:42.872 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:42.864+0000 7f769d224700 10 monclient(hunting): _renew_subs 2021-05-25T12:07:42.872 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:42.864+0000 7f769d224700 10 monclient(hunting): authenticate will time out at 1263.461718s 2021-05-25T12:07:42.872 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:42.864+0000 7f769659c700 1 -- >> v1:172.21.15.102:6790/0 conn(0x7f7698069c80 legacy=0x7f7698130e30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6790/0 2021-05-25T12:07:42.872 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:42.864+0000 7f7696d9d700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f7698069190 0x7f7698069570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33946/0 (socket says 172.21.15.102:33946) 2021-05-25T12:07:42.872 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:42.864+0000 7f7696d9d700 1 -- 172.21.15.102:0/3970834671 learned_addr learned my addr 172.21.15.102:0/3970834671 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:07:43.066 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:43.065+0000 7f769659c700 1 -- 172.21.15.102:0/3970834671 >> v1:172.21.15.102:6790/0 conn(0x7f7698069c80 legacy=0x7f7698130e30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6790/0 2021-05-25T12:07:43.467 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:43.466+0000 7f769659c700 1 -- 172.21.15.102:0/3970834671 >> v1:172.21.15.102:6790/0 conn(0x7f7698069c80 legacy=0x7f7698130e30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6790/0 2021-05-25T12:07:44.268 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:44.267+0000 7f769659c700 1 -- 172.21.15.102:0/3970834671 >> v1:172.21.15.102:6790/0 conn(0x7f7698069c80 legacy=0x7f7698130e30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6790/0 2021-05-25T12:07:45.866 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:45.865+0000 7f7694d99700 10 monclient(hunting): tick 2021-05-25T12:07:45.867 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:45.865+0000 7f7694d99700 1 monclient(hunting): continuing hunt 2021-05-25T12:07:45.867 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:45.865+0000 7f7694d99700 10 monclient(hunting): _reopen_session rank -1 2021-05-25T12:07:45.867 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:45.865+0000 7f7694d99700 1 -- 172.21.15.102:0/3970834671 >> v1:172.21.15.102:6790/0 conn(0x7f7698069c80 legacy=0x7f7698130e30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-05-25T12:07:45.868 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:45.865+0000 7f7694d99700 1 -- 172.21.15.102:0/3970834671 >> v1:172.21.15.181:6789/0 conn(0x7f7698096ab0 legacy=0x7f7698134560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:45.868 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:45.865+0000 7f7694d99700 1 -- 172.21.15.102:0/3970834671 >> v1:172.21.15.102:6789/0 conn(0x7f7698069190 legacy=0x7f7698069570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:45.868 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:45.865+0000 7f7694d99700 10 monclient: _add_conns ranks=[2,0,1] 2021-05-25T12:07:45.868 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:45.865+0000 7f7694d99700 10 monclient(hunting): picked mon.noname-b con 0x7f7698069c80 addr v1:172.21.15.102:6790/0 2021-05-25T12:07:45.869 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:45.865+0000 7f7694d99700 10 monclient(hunting): picked mon.noname-a con 0x7f7678006b70 addr v1:172.21.15.102:6789/0 2021-05-25T12:07:45.869 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:45.865+0000 7f7694d99700 10 monclient(hunting): picked mon.noname-c con 0x7f7698096ab0 addr v1:172.21.15.181:6789/0 2021-05-25T12:07:45.869 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:45.865+0000 7f7694d99700 1 -- 172.21.15.102:0/3970834671 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7698095eb0 con 0x7f7678006b70 2021-05-25T12:07:45.869 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:45.865+0000 7f7694d99700 1 -- 172.21.15.102:0/3970834671 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7698095c10 con 0x7f7698096ab0 2021-05-25T12:07:45.869 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:45.865+0000 7f7694d99700 1 -- 172.21.15.102:0/3970834671 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76780100c0 con 0x7f7698069c80 2021-05-25T12:07:45.870 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:45.865+0000 7f7694d99700 10 monclient(hunting): _renew_subs 2021-05-25T12:07:45.870 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:45.865+0000 7f7696d9d700 1 -- 172.21.15.102:0/3970834671 >> v1:172.21.15.102:6790/0 conn(0x7f7698069c80 legacy=0x7f7678005430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6790/0 2021-05-25T12:07:46.067 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:46.066+0000 7f7696d9d700 1 -- 172.21.15.102:0/3970834671 >> v1:172.21.15.102:6790/0 conn(0x7f7698069c80 legacy=0x7f7678005430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6790/0 2021-05-25T12:07:46.467 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:46.467+0000 7f7696d9d700 1 -- 172.21.15.102:0/3970834671 >> v1:172.21.15.102:6790/0 conn(0x7f7698069c80 legacy=0x7f7678005430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6790/0 2021-05-25T12:07:47.269 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:47.268+0000 7f7696d9d700 1 -- 172.21.15.102:0/3970834671 >> v1:172.21.15.102:6790/0 conn(0x7f7698069c80 legacy=0x7f7678005430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6790/0 2021-05-25T12:07:48.867 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:48.865+0000 7f7694d99700 10 monclient(hunting): tick 2021-05-25T12:07:48.867 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:48.865+0000 7f7694d99700 1 monclient(hunting): continuing hunt 2021-05-25T12:07:48.867 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:48.865+0000 7f7694d99700 10 monclient(hunting): _reopen_session rank -1 2021-05-25T12:07:48.868 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:48.865+0000 7f7694d99700 1 -- 172.21.15.102:0/3970834671 >> v1:172.21.15.102:6790/0 conn(0x7f7698069c80 legacy=0x7f7678005430 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-05-25T12:07:48.868 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:48.865+0000 7f7694d99700 1 -- 172.21.15.102:0/3970834671 >> v1:172.21.15.181:6789/0 conn(0x7f7698096ab0 legacy=0x7f767800c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:48.868 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:48.865+0000 7f7694d99700 1 -- 172.21.15.102:0/3970834671 >> v1:172.21.15.102:6789/0 conn(0x7f7678006b70 legacy=0x7f7678009000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:48.868 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:48.865+0000 7f7694d99700 10 monclient: _add_conns ranks=[0,2,1] 2021-05-25T12:07:48.868 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:48.865+0000 7f7694d99700 10 monclient(hunting): picked mon.noname-a con 0x7f7678010380 addr v1:172.21.15.102:6789/0 2021-05-25T12:07:48.869 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:48.865+0000 7f7694d99700 10 monclient(hunting): picked mon.noname-b con 0x7f7698069c80 addr v1:172.21.15.102:6790/0 2021-05-25T12:07:48.869 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:48.865+0000 7f7694d99700 10 monclient(hunting): picked mon.noname-c con 0x7f7698096ab0 addr v1:172.21.15.181:6789/0 2021-05-25T12:07:48.869 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:48.865+0000 7f7694d99700 1 -- 172.21.15.102:0/3970834671 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76780112d0 con 0x7f7678010380 2021-05-25T12:07:48.869 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:48.865+0000 7f7694d99700 1 -- 172.21.15.102:0/3970834671 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76780100c0 con 0x7f7698096ab0 2021-05-25T12:07:48.870 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:48.865+0000 7f7694d99700 1 -- 172.21.15.102:0/3970834671 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7678011010 con 0x7f7698069c80 2021-05-25T12:07:48.870 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:48.865+0000 7f7694d99700 10 monclient(hunting): _renew_subs 2021-05-25T12:07:48.870 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:48.865+0000 7f769659c700 1 -- 172.21.15.102:0/3970834671 >> v1:172.21.15.102:6790/0 conn(0x7f7698069c80 legacy=0x7f7678009000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6790/0 2021-05-25T12:07:49.067 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:49.066+0000 7f769659c700 1 -- 172.21.15.102:0/3970834671 >> v1:172.21.15.102:6790/0 conn(0x7f7698069c80 legacy=0x7f7678009000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6790/0 2021-05-25T12:07:49.468 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:49.467+0000 7f769659c700 1 -- 172.21.15.102:0/3970834671 >> v1:172.21.15.102:6790/0 conn(0x7f7698069c80 legacy=0x7f7678009000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6790/0 2021-05-25T12:07:50.270 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.268+0000 7f769659c700 1 -- 172.21.15.102:0/3970834671 >> v1:172.21.15.102:6790/0 conn(0x7f7698069c80 legacy=0x7f7678009000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6790/0 2021-05-25T12:07:50.787 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.783+0000 7f7695d9b700 1 -- 172.21.15.102:0/3970834671 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4200199406 0 0) 0x7f76780112d0 con 0x7f7678010380 2021-05-25T12:07:50.787 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.783+0000 7f7695d9b700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:50.787 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.783+0000 7f7695d9b700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:50.788 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.783+0000 7f7695d9b700 10 monclient(hunting): my global_id is 204447 2021-05-25T12:07:50.788 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.783+0000 7f7695d9b700 1 -- 172.21.15.102:0/3970834671 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7684003270 con 0x7f7678010380 2021-05-25T12:07:50.788 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.783+0000 7f7695d9b700 1 -- 172.21.15.102:0/3970834671 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 334290734 0 0) 0x7f7684003270 con 0x7f7678010380 2021-05-25T12:07:50.788 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.783+0000 7f7695d9b700 1 -- 172.21.15.102:0/3970834671 >> v1:172.21.15.102:6790/0 conn(0x7f7698069c80 legacy=0x7f7678009000 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-05-25T12:07:50.789 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.783+0000 7f7695d9b700 1 -- 172.21.15.102:0/3970834671 >> v1:172.21.15.181:6789/0 conn(0x7f7698096ab0 legacy=0x7f767800c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:50.789 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.783+0000 7f7695d9b700 10 monclient: _finish_hunting 0 2021-05-25T12:07:50.789 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.783+0000 7f7695d9b700 1 monclient: found mon.noname-a 2021-05-25T12:07:50.789 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.783+0000 7f7695d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:07:50.789 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.783+0000 7f7695d9b700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T12:07:50.790 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.783+0000 7f7695d9b700 1 -- 172.21.15.102:0/3970834671 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7678005440 con 0x7f7678010380 2021-05-25T12:07:50.790 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.783+0000 7f7695d9b700 10 monclient: _finish_auth 0 2021-05-25T12:07:50.790 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.783+0000 7f7695d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:07:50.790 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.783+0000 7f7695d9b700 1 -- 172.21.15.102:0/3970834671 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f76800015e0 con 0x7f7678010380 2021-05-25T12:07:50.791 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.783+0000 7f769d224700 5 monclient: authenticate success, global_id 204447 2021-05-25T12:07:50.791 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.783+0000 7f769d224700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:07:50.791 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.783+0000 7f7695d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:50.791 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.783+0000 7f7695d9b700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:07:50.791 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.783+0000 7f7695d9b700 10 monclient: dump: 2021-05-25T12:07:50.792 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:50.792 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:50.792 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:50.792 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:50.793 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:50.793 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:50.793 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:50.793 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:50.793 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:50.794 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:50.794 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.783+0000 7f769d224700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:07:50.794 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.784+0000 7f7695d9b700 1 -- 172.21.15.102:0/3970834671 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7680001880 con 0x7f7678010380 2021-05-25T12:07:50.794 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.784+0000 7f7695d9b700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:07:50.794 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.784+0000 7f7695d9b700 1 -- 172.21.15.102:0/3970834671 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f7680001c50 con 0x7f7678010380 2021-05-25T12:07:50.795 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.784+0000 7f769d224700 10 monclient: get_monmap_and_config success 2021-05-25T12:07:50.795 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.784+0000 7f769d224700 4 set_mon_vals no callback set 2021-05-25T12:07:50.795 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.784+0000 7f7695d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:50.795 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.784+0000 7f7695d9b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:07:50.795 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.784+0000 7f7695d9b700 10 monclient: dump: 2021-05-25T12:07:50.796 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:50.796 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:50.796 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:50.796 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:50.797 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:50.797 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:50.797 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:50.797 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:50.797 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:50.798 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:50.798 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.784+0000 7f769d224700 10 monclient: shutdown 2021-05-25T12:07:50.798 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.784+0000 7f769d224700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:07:50.798 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.784+0000 7f769d224700 1 -- 172.21.15.102:0/3970834671 >> v1:172.21.15.102:6789/0 conn(0x7f7678010380 legacy=0x7f7678010760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:50.798 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.784+0000 7f769d224700 1 -- 172.21.15.102:0/3970834671 shutdown_connections 2021-05-25T12:07:50.799 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.784+0000 7f769d224700 1 -- 172.21.15.102:0/3970834671 >> 172.21.15.102:0/3970834671 conn(0x7f769809af90 msgr2=0x7f769809b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:07:50.799 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.784+0000 7f769d224700 1 -- 172.21.15.102:0/3970834671 shutdown_connections 2021-05-25T12:07:50.799 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.784+0000 7f769d224700 1 -- 172.21.15.102:0/3970834671 wait complete. 2021-05-25T12:07:50.799 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.786+0000 7f769d224700 10 monclient: build_initial_monmap 2021-05-25T12:07:50.799 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.786+0000 7f769d224700 10 monclient: monmap: 2021-05-25T12:07:50.800 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:07:50.800 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:07:50.800 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:07:50.800 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:07:50.801 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:07:50.801 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:07:50.801 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:07:50.801 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:07:50.801 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:07:50.802 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:50.802 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.786+0000 7f769d224700 1 librados: starting msgr at 2021-05-25T12:07:50.802 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.786+0000 7f769d224700 1 librados: starting objecter 2021-05-25T12:07:50.806 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.786+0000 7f769d224700 1 Processor -- start 2021-05-25T12:07:50.806 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.786+0000 7f769d224700 1 -- start start 2021-05-25T12:07:50.806 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.786+0000 7f769d224700 1 librados: setting wanted keys 2021-05-25T12:07:50.806 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.786+0000 7f769d224700 1 librados: calling monclient init 2021-05-25T12:07:50.807 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.786+0000 7f769d224700 10 monclient: init 2021-05-25T12:07:50.807 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.787+0000 7f769d224700 10 monclient: _reopen_session rank -1 2021-05-25T12:07:50.807 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.787+0000 7f769d224700 10 monclient: _add_conns ranks=[0,1,2] 2021-05-25T12:07:50.807 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.787+0000 7f769d224700 10 monclient(hunting): picked mon.noname-a con 0x7f7678010380 addr v1:172.21.15.102:6789/0 2021-05-25T12:07:50.807 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.787+0000 7f769d224700 10 monclient(hunting): picked mon.noname-b con 0x7f7698069c80 addr v1:172.21.15.181:6789/0 2021-05-25T12:07:50.808 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.787+0000 7f769d224700 10 monclient(hunting): picked mon.noname-c con 0x7f7698096ab0 addr v1:172.21.15.102:6790/0 2021-05-25T12:07:50.808 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.787+0000 7f769d224700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7698119fd0 con 0x7f7678010380 2021-05-25T12:07:50.808 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.787+0000 7f769d224700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7698118be0 con 0x7f7698069c80 2021-05-25T12:07:50.808 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.787+0000 7f769d224700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7698118100 con 0x7f7698096ab0 2021-05-25T12:07:50.809 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.787+0000 7f769d224700 10 monclient(hunting): _renew_subs 2021-05-25T12:07:50.809 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.787+0000 7f769d224700 10 monclient(hunting): authenticate will time out at 1271.384507s 2021-05-25T12:07:50.809 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.787+0000 7f769759e700 1 -- >> v1:172.21.15.102:6790/0 conn(0x7f7698096ab0 legacy=0x7f7698143a00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.102:6790/0 2021-05-25T12:07:50.809 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.787+0000 7f7696d9d700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f7678010380 0x7f769812c830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33988/0 (socket says 172.21.15.102:33988) 2021-05-25T12:07:50.809 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.787+0000 7f7696d9d700 1 -- 172.21.15.102:0/1927627071 learned_addr learned my addr 172.21.15.102:0/1927627071 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:07:50.810 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.787+0000 7f767effd700 10 client.?.objecter ms_handle_connect 0x7f7678010380 2021-05-25T12:07:50.810 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.787+0000 7f767effd700 10 client.?.objecter resend_mon_ops 2021-05-25T12:07:50.810 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.787+0000 7f767effd700 10 client.?.objecter ms_handle_connect 0x7f7698069c80 2021-05-25T12:07:50.810 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.787+0000 7f767effd700 10 client.?.objecter resend_mon_ops 2021-05-25T12:07:50.815 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.810+0000 7f767effd700 1 -- 172.21.15.102:0/1927627071 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2377996414 0 0) 0x7f7698119fd0 con 0x7f7678010380 2021-05-25T12:07:50.815 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.810+0000 7f767effd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:50.816 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.810+0000 7f767effd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:50.816 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.810+0000 7f767effd700 10 monclient(hunting): my global_id is 204450 2021-05-25T12:07:50.816 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.810+0000 7f767effd700 1 -- 172.21.15.102:0/1927627071 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7678009550 con 0x7f7678010380 2021-05-25T12:07:50.816 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.810+0000 7f767effd700 1 -- 172.21.15.102:0/1927627071 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2979272385 0 0) 0x7f7698118be0 con 0x7f7698069c80 2021-05-25T12:07:50.816 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.810+0000 7f767effd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:50.817 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.810+0000 7f767effd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:50.817 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.810+0000 7f767effd700 10 monclient(hunting): my global_id is 224314 2021-05-25T12:07:50.817 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.810+0000 7f767effd700 1 -- 172.21.15.102:0/1927627071 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7698119fd0 con 0x7f7698069c80 2021-05-25T12:07:50.817 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.810+0000 7f767effd700 1 -- 172.21.15.102:0/1927627071 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3194245163 0 0) 0x7f7698119fd0 con 0x7f7698069c80 2021-05-25T12:07:50.818 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.810+0000 7f767effd700 1 -- 172.21.15.102:0/1927627071 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7698118be0 con 0x7f7698069c80 2021-05-25T12:07:50.818 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.810+0000 7f767effd700 1 -- 172.21.15.102:0/1927627071 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f768c004090 con 0x7f7698069c80 2021-05-25T12:07:50.818 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.810+0000 7f767effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:50.818 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.810+0000 7f767effd700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:07:50.818 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.810+0000 7f767effd700 10 monclient(hunting): dump: 2021-05-25T12:07:50.819 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:50.819 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:50.819 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:50.819 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:50.820 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:50.820 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:50.820 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:50.820 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:50.820 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:50.821 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:50.821 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.810+0000 7f767effd700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:07:50.821 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.810+0000 7f767effd700 1 -- 172.21.15.102:0/1927627071 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1504759223 0 0) 0x7f7678009550 con 0x7f7678010380 2021-05-25T12:07:50.821 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.810+0000 7f767effd700 1 -- 172.21.15.102:0/1927627071 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7698119fd0 con 0x7f7678010380 2021-05-25T12:07:50.821 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.810+0000 7f767effd700 1 -- 172.21.15.102:0/1927627071 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f7680001b90 con 0x7f7678010380 2021-05-25T12:07:50.822 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.810+0000 7f767effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:50.822 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.810+0000 7f767effd700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2021-05-25T12:07:50.822 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.810+0000 7f767effd700 10 monclient(hunting): dump: 2021-05-25T12:07:50.822 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:50.823 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:50.823 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:50.823 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:50.823 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:50.823 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:50.824 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:50.824 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:50.824 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:50.824 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:50.825 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.811+0000 7f767effd700 1 -- 172.21.15.102:0/1927627071 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 158728261 0 0) 0x7f7698119fd0 con 0x7f7678010380 2021-05-25T12:07:50.825 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.811+0000 7f767effd700 1 -- 172.21.15.102:0/1927627071 >> v1:172.21.15.102:6790/0 conn(0x7f7698096ab0 legacy=0x7f7698143a00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-05-25T12:07:50.825 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.811+0000 7f767effd700 1 -- 172.21.15.102:0/1927627071 >> v1:172.21.15.181:6789/0 conn(0x7f7698069c80 legacy=0x7f76981402d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:50.825 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.811+0000 7f767effd700 10 monclient: _finish_hunting 0 2021-05-25T12:07:50.826 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.811+0000 7f767effd700 1 monclient: found mon.a 2021-05-25T12:07:50.826 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.811+0000 7f767effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:07:50.826 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.811+0000 7f767effd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:07:50.826 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.811+0000 7f767effd700 1 -- 172.21.15.102:0/1927627071 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f769812cf40 con 0x7f7678010380 2021-05-25T12:07:50.826 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.811+0000 7f767effd700 10 monclient: _finish_auth 0 2021-05-25T12:07:50.827 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.811+0000 7f767effd700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:07:50.827 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.811+0000 7f769d224700 5 monclient: authenticate success, global_id 204450 2021-05-25T12:07:50.827 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.811+0000 7f769d224700 10 monclient: _renew_subs 2021-05-25T12:07:50.827 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.811+0000 7f769d224700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:07:50.828 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.811+0000 7f769d224700 1 -- 172.21.15.102:0/1927627071 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f769812d1b0 con 0x7f7678010380 2021-05-25T12:07:50.828 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.811+0000 7f769d224700 10 client.204450.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:07:50.828 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.811+0000 7f769d224700 10 monclient: _renew_subs 2021-05-25T12:07:50.828 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.811+0000 7f769d224700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:07:50.828 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.811+0000 7f769d224700 1 -- 172.21.15.102:0/1927627071 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7698117700 con 0x7f7678010380 2021-05-25T12:07:50.829 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.811+0000 7f769d224700 1 librados: init done 2021-05-25T12:07:50.829 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.811+0000 7f767effd700 1 -- 172.21.15.102:0/1927627071 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f76800047e0 con 0x7f7678010380 2021-05-25T12:07:50.829 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.811+0000 7f767effd700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:07:50.829 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.811+0000 7f767effd700 1 -- 172.21.15.102:0/1927627071 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f7680004af0 con 0x7f7678010380 2021-05-25T12:07:50.830 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.811+0000 7f767effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:50.830 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.811+0000 7f767effd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:07:50.830 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.811+0000 7f769559a700 4 set_mon_vals no callback set 2021-05-25T12:07:50.830 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.811+0000 7f767effd700 10 monclient: dump: 2021-05-25T12:07:50.830 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:50.831 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:50.831 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:50.831 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:50.831 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:50.832 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:50.832 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:50.832 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:50.832 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:50.832 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:50.833 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.811+0000 7f767effd700 1 -- 172.21.15.102:0/1927627071 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f768000de10 con 0x7f7678010380 2021-05-25T12:07:50.833 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.812+0000 7f767effd700 1 -- 172.21.15.102:0/1927627071 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(246..246 src has 1..246) v4 ==== 5858+0+0 (unknown 4267717717 0 0) 0x7f768003d760 con 0x7f7678010380 2021-05-25T12:07:50.833 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.812+0000 7f767effd700 10 client.204450.objecter ms_dispatch 0x7f7698001550 osd_map(246..246 src has 1..246) v4 2021-05-25T12:07:50.833 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.812+0000 7f767effd700 3 client.204450.objecter handle_osd_map got epochs [246,246] > 0 2021-05-25T12:07:50.833 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.812+0000 7f767effd700 3 client.204450.objecter handle_osd_map decoding full epoch 246 2021-05-25T12:07:50.834 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.812+0000 7f769d224700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:07:50.834 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.812+0000 7f769d224700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:07:50.834 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.812+0000 7f769d224700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:07:50.834 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.812+0000 7f769d224700 1 -- 172.21.15.102:0/1927627071 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7674004db0 con 0x7f7678010380 2021-05-25T12:07:50.834 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.812+0000 7f767effd700 20 client.204450.objecter dump_active .. 0 homeless 2021-05-25T12:07:50.835 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.812+0000 7f769659c700 10 monclient: get_auth_request con 0x7f7678034200 auth_method 0 2021-05-25T12:07:50.835 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.812+0000 7f767effd700 10 client.204450.objecter ms_handle_connect 0x7f7678034200 2021-05-25T12:07:50.836 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.816+0000 7f767effd700 1 -- 172.21.15.102:0/1927627071 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7f7680012db0 con 0x7f7678010380 2021-05-25T12:07:50.836 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.816+0000 7f767effd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:07:50.836 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:50.816+0000 7f767effd700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:07:51.039 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.037+0000 7f769d224700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2021-05-25T12:07:51.039 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.037+0000 7f769d224700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2021-05-25T12:07:51.039 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.037+0000 7f769d224700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:07:51.040 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.037+0000 7f769d224700 1 -- 172.21.15.102:0/1927627071 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f7674005060 con 0x7f7678010380 2021-05-25T12:07:51.040 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.037+0000 7f767effd700 1 -- 172.21.15.102:0/1927627071 <== mon.0 v1:172.21.15.102:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1336 (unknown 254746657 0 773971734) 0x7f76800129a0 con 0x7f7678010380 2021-05-25T12:07:51.040 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.038+0000 7f767effd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2021-05-25T12:07:51.040 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.038+0000 7f767effd700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:07:51.041 INFO:teuthology.orchestra.run.smithi102.stdout:{"election_epoch":66,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138303579357183","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"20ee91b6-8935-4b98-b868-85c03789ccc4","modified":"2021-05-25T11:59:11.011522Z","created":"2021-05-25T11:59:11.011522Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6790","nonce":0}]},"addr":"172.21.15.102:6790/0","public_addr":"172.21.15.102:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2021-05-25T12:07:51.042 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.039+0000 7f769d224700 10 librados: watch_flush enter 2021-05-25T12:07:51.043 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.039+0000 7f769d224700 10 librados: watch_flush exit 2021-05-25T12:07:51.043 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.039+0000 7f769d224700 20 client.204450.objecter shutdown clearing up homeless session... 2021-05-25T12:07:51.043 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.039+0000 7f769d224700 10 client.204450.objecter successfully canceled tick 2021-05-25T12:07:51.043 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.039+0000 7f769d224700 1 -- 172.21.15.102:0/1927627071 >> v1:172.21.15.181:6812/29582 conn(0x7f7678034200 legacy=0x7f76780366c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:51.044 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.039+0000 7f769d224700 10 monclient: shutdown 2021-05-25T12:07:51.044 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.039+0000 7f769d224700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:07:51.044 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.039+0000 7f769d224700 1 -- 172.21.15.102:0/1927627071 >> v1:172.21.15.102:6789/0 conn(0x7f7678010380 legacy=0x7f769812c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:51.044 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.039+0000 7f769d224700 1 -- 172.21.15.102:0/1927627071 shutdown_connections 2021-05-25T12:07:51.045 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.039+0000 7f769d224700 1 -- 172.21.15.102:0/1927627071 >> 172.21.15.102:0/1927627071 conn(0x7f769809af90 msgr2=0x7f769812ec90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:07:51.045 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.039+0000 7f769d224700 1 -- 172.21.15.102:0/1927627071 shutdown_connections 2021-05-25T12:07:51.045 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.039+0000 7f769d224700 1 -- 172.21.15.102:0/1927627071 wait complete. 2021-05-25T12:07:51.045 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.039+0000 7f769d224700 1 librados: shutdown 2021-05-25T12:07:51.055 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2021-05-25T12:07:51.056 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2021-05-25T12:07:51.165 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.164+0000 7f57ce217700 10 monclient: get_monmap_and_config 2021-05-25T12:07:51.166 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.164+0000 7f57ce217700 10 monclient: build_initial_monmap 2021-05-25T12:07:51.166 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.164+0000 7f57ce217700 10 monclient: monmap: 2021-05-25T12:07:51.166 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:07:51.167 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:51.167 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:07:51.165516+0000 2021-05-25T12:07:51.167 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:07:51.165516+0000 2021-05-25T12:07:51.168 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:07:51.168 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:51.168 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:07:51.168 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:07:51.168 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:07:51.169 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:51.181 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.165+0000 7f57ce217700 1 Processor -- start 2021-05-25T12:07:51.182 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.165+0000 7f57ce217700 1 -- start start 2021-05-25T12:07:51.182 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.165+0000 7f57ce217700 10 monclient: init 2021-05-25T12:07:51.182 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.165+0000 7f57ce217700 10 monclient: _reopen_session rank -1 2021-05-25T12:07:51.182 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.165+0000 7f57ce217700 10 monclient: _add_conns ranks=[0,1,2] 2021-05-25T12:07:51.183 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.165+0000 7f57ce217700 10 monclient(hunting): picked mon.noname-a con 0x7f57c8068f80 addr v1:172.21.15.102:6789/0 2021-05-25T12:07:51.183 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.165+0000 7f57ce217700 10 monclient(hunting): picked mon.noname-c con 0x7f57c8126790 addr v1:172.21.15.181:6789/0 2021-05-25T12:07:51.183 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.165+0000 7f57ce217700 10 monclient(hunting): picked mon.noname-b con 0x7f57c8126c10 addr v1:172.21.15.102:6790/0 2021-05-25T12:07:51.183 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.165+0000 7f57ce217700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57c8094310 con 0x7f57c8068f80 2021-05-25T12:07:51.184 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.165+0000 7f57ce217700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57c80945b0 con 0x7f57c8126790 2021-05-25T12:07:51.184 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.165+0000 7f57ce217700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57c8094850 con 0x7f57c8126c10 2021-05-25T12:07:51.184 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.165+0000 7f57ce217700 10 monclient(hunting): _renew_subs 2021-05-25T12:07:51.184 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.165+0000 7f57ce217700 10 monclient(hunting): authenticate will time out at 1271.762965s 2021-05-25T12:07:51.184 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.166+0000 7f57c7fff700 1 -- >> v1:172.21.15.102:6790/0 conn(0x7f57c8126c10 legacy=0x7f57c81321a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6790/0 2021-05-25T12:07:51.185 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.166+0000 7f57c77fe700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f57c8068f80 0x7f57c8069380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33996/0 (socket says 172.21.15.102:33996) 2021-05-25T12:07:51.185 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.166+0000 7f57c77fe700 1 -- 172.21.15.102:0/697690484 learned_addr learned my addr 172.21.15.102:0/697690484 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:07:51.185 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.166+0000 7f57c67fc700 1 -- 172.21.15.102:0/697690484 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3865270430 0 0) 0x7f57c8094310 con 0x7f57c8068f80 2021-05-25T12:07:51.185 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.166+0000 7f57c67fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:51.186 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.166+0000 7f57c67fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:51.186 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.166+0000 7f57c67fc700 10 monclient(hunting): my global_id is 204453 2021-05-25T12:07:51.186 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.166+0000 7f57c67fc700 1 -- 172.21.15.102:0/697690484 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57ac003270 con 0x7f57c8068f80 2021-05-25T12:07:51.186 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.166+0000 7f57c67fc700 1 -- 172.21.15.102:0/697690484 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1753745265 0 0) 0x7f57ac003270 con 0x7f57c8068f80 2021-05-25T12:07:51.186 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.166+0000 7f57c67fc700 1 -- 172.21.15.102:0/697690484 >> v1:172.21.15.102:6790/0 conn(0x7f57c8126c10 legacy=0x7f57c81321a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-05-25T12:07:51.187 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.166+0000 7f57c67fc700 1 -- 172.21.15.102:0/697690484 >> v1:172.21.15.181:6789/0 conn(0x7f57c8126790 legacy=0x7f57c812ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:51.187 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.166+0000 7f57c67fc700 10 monclient: _finish_hunting 0 2021-05-25T12:07:51.187 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.166+0000 7f57c67fc700 1 monclient: found mon.noname-a 2021-05-25T12:07:51.188 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.166+0000 7f57c67fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:07:51.188 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.166+0000 7f57c67fc700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T12:07:51.188 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.166+0000 7f57c67fc700 1 -- 172.21.15.102:0/697690484 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57c8094c10 con 0x7f57c8068f80 2021-05-25T12:07:51.188 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.166+0000 7f57c67fc700 10 monclient: _finish_auth 0 2021-05-25T12:07:51.189 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.166+0000 7f57c67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:07:51.189 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.166+0000 7f57ce217700 5 monclient: authenticate success, global_id 204453 2021-05-25T12:07:51.189 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.166+0000 7f57ce217700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:07:51.189 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.166+0000 7f57c67fc700 1 -- 172.21.15.102:0/697690484 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f57b4001610 con 0x7f57c8068f80 2021-05-25T12:07:51.190 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.166+0000 7f57c67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:51.190 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.166+0000 7f57c67fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:07:51.190 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.166+0000 7f57c67fc700 10 monclient: dump: 2021-05-25T12:07:51.190 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:51.190 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:51.191 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:51.191 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:51.191 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:51.191 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:51.191 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:51.192 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:51.192 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:51.192 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:51.192 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.166+0000 7f57ce217700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:07:51.193 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.166+0000 7f57c67fc700 1 -- 172.21.15.102:0/697690484 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f57b4001930 con 0x7f57c8068f80 2021-05-25T12:07:51.193 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.166+0000 7f57c67fc700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:07:51.193 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.166+0000 7f57ce217700 10 monclient: get_monmap_and_config success 2021-05-25T12:07:51.193 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.166+0000 7f57ce217700 4 set_mon_vals no callback set 2021-05-25T12:07:51.193 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.166+0000 7f57c67fc700 1 -- 172.21.15.102:0/697690484 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f57b4001cc0 con 0x7f57c8068f80 2021-05-25T12:07:51.194 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.166+0000 7f57c67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:51.194 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.166+0000 7f57c67fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:07:51.194 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.166+0000 7f57c67fc700 10 monclient: dump: 2021-05-25T12:07:51.194 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:51.195 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:51.195 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:51.195 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:51.195 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:51.195 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:51.196 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:51.196 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:51.196 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:51.196 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:51.197 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.166+0000 7f57ce217700 10 monclient: shutdown 2021-05-25T12:07:51.197 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.166+0000 7f57ce217700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:07:51.197 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.166+0000 7f57ce217700 1 -- 172.21.15.102:0/697690484 >> v1:172.21.15.102:6789/0 conn(0x7f57c8068f80 legacy=0x7f57c8069380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:51.197 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.166+0000 7f57ce217700 1 -- 172.21.15.102:0/697690484 shutdown_connections 2021-05-25T12:07:51.197 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.166+0000 7f57ce217700 1 -- 172.21.15.102:0/697690484 >> 172.21.15.102:0/697690484 conn(0x7f57c809b370 msgr2=0x7f57c8097430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:07:51.198 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.167+0000 7f57ce217700 1 -- 172.21.15.102:0/697690484 shutdown_connections 2021-05-25T12:07:51.198 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.167+0000 7f57ce217700 1 -- 172.21.15.102:0/697690484 wait complete. 2021-05-25T12:07:51.198 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.167+0000 7f57ce217700 10 monclient: build_initial_monmap 2021-05-25T12:07:51.198 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.167+0000 7f57ce217700 10 monclient: monmap: 2021-05-25T12:07:51.198 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:07:51.199 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:07:51.199 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:07:51.199 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:07:51.199 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:07:51.200 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:07:51.200 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:07:51.200 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:07:51.200 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:07:51.200 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:51.201 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.167+0000 7f57ce217700 1 librados: starting msgr at 2021-05-25T12:07:51.201 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.167+0000 7f57ce217700 1 librados: starting objecter 2021-05-25T12:07:51.201 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.167+0000 7f57ce217700 1 Processor -- start 2021-05-25T12:07:51.201 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.167+0000 7f57ce217700 1 -- start start 2021-05-25T12:07:51.202 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.167+0000 7f57ce217700 1 librados: setting wanted keys 2021-05-25T12:07:51.202 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.167+0000 7f57ce217700 1 librados: calling monclient init 2021-05-25T12:07:51.202 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.167+0000 7f57ce217700 10 monclient: init 2021-05-25T12:07:51.202 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.168+0000 7f57ce217700 10 monclient: _reopen_session rank -1 2021-05-25T12:07:51.202 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.168+0000 7f57ce217700 10 monclient: _add_conns ranks=[2,1,0] 2021-05-25T12:07:51.203 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.168+0000 7f57ce217700 10 monclient(hunting): picked mon.noname-c con 0x7f57c8068f80 addr v1:172.21.15.102:6790/0 2021-05-25T12:07:51.203 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.168+0000 7f57ce217700 10 monclient(hunting): picked mon.noname-b con 0x7f57c8126790 addr v1:172.21.15.181:6789/0 2021-05-25T12:07:51.203 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.168+0000 7f57ce217700 10 monclient(hunting): picked mon.noname-a con 0x7f57c8126c10 addr v1:172.21.15.102:6789/0 2021-05-25T12:07:51.203 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.168+0000 7f57ce217700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57c8132fa0 con 0x7f57c8126c10 2021-05-25T12:07:51.204 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.168+0000 7f57ce217700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57c8124070 con 0x7f57c8126790 2021-05-25T12:07:51.204 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.168+0000 7f57ce217700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57c81280b0 con 0x7f57c8068f80 2021-05-25T12:07:51.204 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.168+0000 7f57ce217700 10 monclient(hunting): _renew_subs 2021-05-25T12:07:51.204 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.168+0000 7f57ce217700 10 monclient(hunting): authenticate will time out at 1271.765388s 2021-05-25T12:07:51.204 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.168+0000 7f57c77fe700 1 -- >> v1:172.21.15.102:6790/0 conn(0x7f57c8068f80 legacy=0x7f57c8132890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.102:6790/0 2021-05-25T12:07:51.205 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.168+0000 7f57c7fff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f57c8126c10 0x7f57c81418d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:34004/0 (socket says 172.21.15.102:34004) 2021-05-25T12:07:51.205 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.168+0000 7f57c7fff700 1 -- 172.21.15.102:0/2528022184 learned_addr learned my addr 172.21.15.102:0/2528022184 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:07:51.205 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.168+0000 7f57b37fe700 10 client.?.objecter ms_handle_connect 0x7f57c8126c10 2021-05-25T12:07:51.205 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.168+0000 7f57b37fe700 10 client.?.objecter resend_mon_ops 2021-05-25T12:07:51.206 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.168+0000 7f57b37fe700 10 client.?.objecter ms_handle_connect 0x7f57c8126790 2021-05-25T12:07:51.206 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.168+0000 7f57b37fe700 10 client.?.objecter resend_mon_ops 2021-05-25T12:07:51.206 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.168+0000 7f57b37fe700 1 -- 172.21.15.102:0/2528022184 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3657629454 0 0) 0x7f57c8132fa0 con 0x7f57c8126c10 2021-05-25T12:07:51.206 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.168+0000 7f57b37fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:51.206 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.168+0000 7f57b37fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:51.207 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.168+0000 7f57b37fe700 10 monclient(hunting): my global_id is 204456 2021-05-25T12:07:51.207 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.168+0000 7f57b37fe700 1 -- 172.21.15.102:0/2528022184 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57a8004c10 con 0x7f57c8126c10 2021-05-25T12:07:51.207 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.168+0000 7f57b37fe700 1 -- 172.21.15.102:0/2528022184 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2619559850 0 0) 0x7f57c8124070 con 0x7f57c8126790 2021-05-25T12:07:51.207 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.168+0000 7f57b37fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:51.207 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.168+0000 7f57b37fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:51.208 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.168+0000 7f57b37fe700 10 monclient(hunting): my global_id is 224320 2021-05-25T12:07:51.208 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.168+0000 7f57b37fe700 1 -- 172.21.15.102:0/2528022184 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57c8132fa0 con 0x7f57c8126790 2021-05-25T12:07:51.208 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.168+0000 7f57b37fe700 1 -- 172.21.15.102:0/2528022184 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1670304951 0 0) 0x7f57a8004c10 con 0x7f57c8126c10 2021-05-25T12:07:51.208 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.168+0000 7f57b37fe700 1 -- 172.21.15.102:0/2528022184 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f57c8124070 con 0x7f57c8126c10 2021-05-25T12:07:51.209 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.168+0000 7f57b37fe700 1 -- 172.21.15.102:0/2528022184 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f57bc004110 con 0x7f57c8126c10 2021-05-25T12:07:51.209 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.168+0000 7f57b37fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:51.209 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.168+0000 7f57b37fe700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:07:51.209 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.168+0000 7f57b37fe700 10 monclient(hunting): dump: 2021-05-25T12:07:51.209 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:51.210 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:51.210 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:51.210 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:51.210 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:51.211 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:51.211 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:51.211 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:51.211 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:51.211 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:51.212 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.168+0000 7f57b37fe700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:07:51.212 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.169+0000 7f57b37fe700 1 -- 172.21.15.102:0/2528022184 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1584470304 0 0) 0x7f57c8124070 con 0x7f57c8126c10 2021-05-25T12:07:51.212 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.169+0000 7f57b37fe700 1 -- 172.21.15.102:0/2528022184 >> v1:172.21.15.102:6790/0 conn(0x7f57c8068f80 legacy=0x7f57c8132890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-05-25T12:07:51.212 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.169+0000 7f57b37fe700 1 -- 172.21.15.102:0/2528022184 >> v1:172.21.15.181:6789/0 conn(0x7f57c8126790 legacy=0x7f57c813e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:51.212 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.169+0000 7f57b37fe700 10 monclient: _finish_hunting 0 2021-05-25T12:07:51.213 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.169+0000 7f57b37fe700 1 monclient: found mon.a 2021-05-25T12:07:51.213 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.169+0000 7f57b37fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:07:51.213 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.169+0000 7f57b37fe700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:07:51.213 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.169+0000 7f57b37fe700 1 -- 172.21.15.102:0/2528022184 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57c8128350 con 0x7f57c8126c10 2021-05-25T12:07:51.214 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.169+0000 7f57b37fe700 10 monclient: _finish_auth 0 2021-05-25T12:07:51.214 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.169+0000 7f57b37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:07:51.214 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.169+0000 7f57ce217700 5 monclient: authenticate success, global_id 204456 2021-05-25T12:07:51.214 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.169+0000 7f57ce217700 10 monclient: _renew_subs 2021-05-25T12:07:51.214 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.169+0000 7f57ce217700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:07:51.215 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.169+0000 7f57ce217700 1 -- 172.21.15.102:0/2528022184 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57c8091240 con 0x7f57c8126c10 2021-05-25T12:07:51.215 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.169+0000 7f57ce217700 10 client.204456.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:07:51.215 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.169+0000 7f57ce217700 10 monclient: _renew_subs 2021-05-25T12:07:51.215 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.169+0000 7f57ce217700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:07:51.216 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.169+0000 7f57ce217700 1 -- 172.21.15.102:0/2528022184 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f57c8139950 con 0x7f57c8126c10 2021-05-25T12:07:51.216 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.169+0000 7f57ce217700 1 librados: init done 2021-05-25T12:07:51.216 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.169+0000 7f57b37fe700 1 -- 172.21.15.102:0/2528022184 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f57bc004380 con 0x7f57c8126c10 2021-05-25T12:07:51.216 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.169+0000 7f57b37fe700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:07:51.216 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.169+0000 7f57b37fe700 1 -- 172.21.15.102:0/2528022184 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f57bc004710 con 0x7f57c8126c10 2021-05-25T12:07:51.217 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.169+0000 7f57b37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:51.217 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.169+0000 7f57b37fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:07:51.217 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.169+0000 7f57c5ffb700 4 set_mon_vals no callback set 2021-05-25T12:07:51.217 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.169+0000 7f57b37fe700 10 monclient: dump: 2021-05-25T12:07:51.218 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:51.218 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:51.218 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:51.219 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:51.219 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:51.219 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:51.219 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:51.220 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:51.220 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:51.220 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:51.220 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.169+0000 7f57ce217700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:07:51.220 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.169+0000 7f57ce217700 1 -- 172.21.15.102:0/2528022184 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f57a00065a0 con 0x7f57a0002840 2021-05-25T12:07:51.221 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.169+0000 7f57b37fe700 1 -- 172.21.15.102:0/2528022184 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f57bc0048f0 con 0x7f57c8126c10 2021-05-25T12:07:51.221 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.170+0000 7f57b37fe700 10 client.204456.objecter ms_handle_connect 0x7f57a0002840 2021-05-25T12:07:51.221 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.170+0000 7f57b37fe700 10 client.204456.objecter resend_mon_ops 2021-05-25T12:07:51.221 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.170+0000 7f57b37fe700 1 -- 172.21.15.102:0/2528022184 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(247..247 src has 1..247) v4 ==== 6241+0+0 (unknown 3601026478 0 0) 0x7f57bc03daf0 con 0x7f57c8126c10 2021-05-25T12:07:51.222 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.170+0000 7f57b37fe700 10 client.204456.objecter ms_dispatch 0x7f57c8001550 osd_map(247..247 src has 1..247) v4 2021-05-25T12:07:51.222 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.170+0000 7f57b37fe700 3 client.204456.objecter handle_osd_map got epochs [247,247] > 0 2021-05-25T12:07:51.222 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.170+0000 7f57b37fe700 3 client.204456.objecter handle_osd_map decoding full epoch 247 2021-05-25T12:07:51.222 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.170+0000 7f57b37fe700 20 client.204456.objecter dump_active .. 0 homeless 2021-05-25T12:07:51.222 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.170+0000 7f57b37fe700 1 -- 172.21.15.102:0/2528022184 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 997274150 0 0) 0x7f57a00065a0 con 0x7f57a0002840 2021-05-25T12:07:51.223 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.170+0000 7f57b37fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:51.223 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.170+0000 7f57b37fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:51.223 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.170+0000 7f57b37fe700 10 monclient(hunting): my global_id is 204459 2021-05-25T12:07:51.223 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.170+0000 7f57b37fe700 1 -- 172.21.15.102:0/2528022184 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57c8124070 con 0x7f57a0002840 2021-05-25T12:07:51.224 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.170+0000 7f57c6ffd700 10 monclient: get_auth_request con 0x7f57a8032f70 auth_method 0 2021-05-25T12:07:51.224 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.170+0000 7f57b37fe700 1 -- 172.21.15.102:0/2528022184 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1829815831 0 0) 0x7f57c8124070 con 0x7f57a0002840 2021-05-25T12:07:51.224 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.170+0000 7f57b37fe700 1 -- 172.21.15.102:0/2528022184 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f57a00065a0 con 0x7f57a0002840 2021-05-25T12:07:51.224 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.170+0000 7f57b37fe700 1 -- 172.21.15.102:0/2528022184 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f57b4001210 con 0x7f57a0002840 2021-05-25T12:07:51.224 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.170+0000 7f57b37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:51.225 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.170+0000 7f57b37fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:07:51.225 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.170+0000 7f57b37fe700 10 monclient: dump: 2021-05-25T12:07:51.225 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:51.225 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:51.226 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:51.226 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:51.226 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:51.226 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:51.227 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:51.227 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:51.227 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:51.227 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:51.227 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.170+0000 7f57b37fe700 1 -- 172.21.15.102:0/2528022184 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3776739266 0 0) 0x7f57a00065a0 con 0x7f57a0002840 2021-05-25T12:07:51.228 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.170+0000 7f57b37fe700 1 -- 172.21.15.102:0/2528022184 --> v1:172.21.15.102:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f57a0009e40 con 0x7f57a0002840 2021-05-25T12:07:51.228 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.170+0000 7f57b37fe700 10 client.204456.objecter ms_handle_connect 0x7f57a8032f70 2021-05-25T12:07:51.228 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.170+0000 7f57b37fe700 1 -- 172.21.15.102:0/2528022184 <== mon.0 v1:172.21.15.102:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7f57b4001d10 con 0x7f57a0002840 2021-05-25T12:07:51.228 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.170+0000 7f57b37fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:07:51.228 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.170+0000 7f57b37fe700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:07:51.229 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.170+0000 7f57b37fe700 1 -- 172.21.15.102:0/2528022184 >> v1:172.21.15.102:6789/0 conn(0x7f57a0002840 legacy=0x7f57a0004ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:51.229 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.170+0000 7f57b37fe700 1 -- 172.21.15.102:0/2528022184 >> v1:172.21.15.102:6789/0 conn(0x7f57a0002840 legacy=0x7f57a0004ce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:07:51.229 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.177+0000 7f57ce217700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:07:51.229 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.177+0000 7f57ce217700 1 -- 172.21.15.102:0/2528022184 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f57a0007540 con 0x7f57a00068e0 2021-05-25T12:07:51.230 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.177+0000 7f57b37fe700 10 client.204456.objecter ms_handle_connect 0x7f57a00068e0 2021-05-25T12:07:51.230 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.177+0000 7f57b37fe700 10 client.204456.objecter resend_mon_ops 2021-05-25T12:07:51.230 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.177+0000 7f57b37fe700 1 -- 172.21.15.102:0/2528022184 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 949292871 0 0) 0x7f57a0007540 con 0x7f57a00068e0 2021-05-25T12:07:51.230 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.177+0000 7f57b37fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:51.230 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.177+0000 7f57b37fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:51.231 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.177+0000 7f57b37fe700 10 monclient(hunting): my global_id is 204462 2021-05-25T12:07:51.231 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.177+0000 7f57b37fe700 1 -- 172.21.15.102:0/2528022184 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57a00065a0 con 0x7f57a00068e0 2021-05-25T12:07:51.231 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.177+0000 7f57b37fe700 1 -- 172.21.15.102:0/2528022184 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 130474330 0 0) 0x7f57a00065a0 con 0x7f57a00068e0 2021-05-25T12:07:51.231 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.177+0000 7f57b37fe700 1 -- 172.21.15.102:0/2528022184 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f57a0007540 con 0x7f57a00068e0 2021-05-25T12:07:51.232 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.177+0000 7f57b37fe700 1 -- 172.21.15.102:0/2528022184 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f57b4004b10 con 0x7f57a00068e0 2021-05-25T12:07:51.232 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.177+0000 7f57b37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:51.232 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.177+0000 7f57b37fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:07:51.232 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.177+0000 7f57b37fe700 10 monclient: dump: 2021-05-25T12:07:51.233 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:51.233 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:51.233 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:51.233 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:51.233 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:51.234 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:51.234 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:51.234 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:51.234 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:51.234 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:51.235 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.177+0000 7f57b37fe700 1 -- 172.21.15.102:0/2528022184 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 896957417 0 0) 0x7f57a0007540 con 0x7f57a00068e0 2021-05-25T12:07:51.235 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.177+0000 7f57b37fe700 1 -- 172.21.15.102:0/2528022184 --> v1:172.21.15.102:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f57a0008770 con 0x7f57a00068e0 2021-05-25T12:07:51.235 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.178+0000 7f57b37fe700 1 -- 172.21.15.102:0/2528022184 <== mon.0 v1:172.21.15.102:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3929 (unknown 0 0 3665060231) 0x7f57b4005f70 con 0x7f57a00068e0 2021-05-25T12:07:51.235 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.178+0000 7f57b37fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:07:51.236 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.178+0000 7f57b37fe700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:07:51.236 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.178+0000 7f57b37fe700 1 -- 172.21.15.102:0/2528022184 >> v1:172.21.15.102:6789/0 conn(0x7f57a00068e0 legacy=0x7f57a0006d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:51.236 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.178+0000 7f57b37fe700 1 -- 172.21.15.102:0/2528022184 >> v1:172.21.15.102:6789/0 conn(0x7f57a00068e0 legacy=0x7f57a0006d70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:07:51.237 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:07:51.237 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:07:51.237 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:07:51.237 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "peon", 2021-05-25T12:07:51.238 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 66, 2021-05-25T12:07:51.238 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:07:51.238 INFO:teuthology.orchestra.run.smithi102.stdout: 0, 2021-05-25T12:07:51.238 INFO:teuthology.orchestra.run.smithi102.stdout: 1 2021-05-25T12:07:51.238 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:07:51.239 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 0, 2021-05-25T12:07:51.239 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:07:51.239 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:07:51.239 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:07:51.240 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:07:51.240 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:07:51.240 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:07:51.240 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:07:51.241 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:07:51.241 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:07:51.241 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:07:51.241 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:07:51.241 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:07:51.242 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:07:51.242 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:07:51.242 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:07:51.242 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:07:51.242 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:07:51.243 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:07:51.243 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:07:51.243 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:07:51.243 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:07:51.244 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:07:51.244 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:07:51.244 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:07:51.244 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:07:51.244 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:51.245 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:07:51.245 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:07:51.245 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:07:51.245 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:07:51.246 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:07:51.246 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:07:51.246 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:07:51.246 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:07:51.246 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:07:51.247 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:07:51.247 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:07:51.247 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:07:51.247 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:07:51.247 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:07:51.248 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:07:51.248 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:07:51.248 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:07:51.248 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:07:51.249 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:07:51.249 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:07:51.249 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:07:51.249 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:07:51.249 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:07:51.250 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:07:51.250 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:07:51.250 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:07:51.250 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:51.251 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:07:51.251 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:51.251 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:07:51.251 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:07:51.252 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:07:51.252 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:07:51.252 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:51.252 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:07:51.252 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:07:51.253 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:07:51.253 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:07:51.253 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:07:51.253 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:51.254 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:07:51.254 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:07:51.254 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:07:51.254 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:07:51.254 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:07:51.255 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:51.255 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:51.255 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:07:51.255 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:07:51.255 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:07:51.256 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:07:51.256 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:51.256 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:07:51.256 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:07:51.257 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:07:51.257 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:07:51.257 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:07:51.257 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:51.257 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:07:51.258 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:07:51.258 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:07:51.258 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:07:51.258 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:07:51.259 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:51.259 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:51.259 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:07:51.259 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:07:51.259 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:07:51.260 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:07:51.260 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:51.260 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:07:51.260 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:07:51.260 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:07:51.261 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:07:51.261 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:07:51.261 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:51.261 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:07:51.261 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:07:51.262 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:07:51.262 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:07:51.262 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:07:51.262 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:07:51.263 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:07:51.263 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:51.263 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:07:51.263 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:07:51.263 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:51.264 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:07:51.264 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:07:51.264 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:07:51.264 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:07:51.265 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:07:51.265 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": [ 2021-05-25T12:07:51.265 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:51.265 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:07:51.265 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:07:51.266 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 6 2021-05-25T12:07:51.266 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:07:51.266 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:07:51.266 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:07:51.266 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:51.267 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:07:51.267 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:07:51.267 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 4 2021-05-25T12:07:51.267 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:07:51.268 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:07:51.268 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:51.268 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:07:51.268 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:07:51.271 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.178+0000 7f57ce217700 10 librados: watch_flush enter 2021-05-25T12:07:51.271 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.178+0000 7f57ce217700 10 librados: watch_flush exit 2021-05-25T12:07:51.271 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.178+0000 7f57ce217700 20 client.204456.objecter shutdown clearing up homeless session... 2021-05-25T12:07:51.271 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.178+0000 7f57ce217700 10 client.204456.objecter successfully canceled tick 2021-05-25T12:07:51.272 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.178+0000 7f57ce217700 1 -- 172.21.15.102:0/2528022184 >> v1:172.21.15.181:6812/29582 conn(0x7f57a8032f70 legacy=0x7f57a8035430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:51.272 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.178+0000 7f57ce217700 10 monclient: shutdown 2021-05-25T12:07:51.272 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.178+0000 7f57ce217700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:07:51.272 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.178+0000 7f57ce217700 1 -- 172.21.15.102:0/2528022184 >> v1:172.21.15.102:6789/0 conn(0x7f57c8126c10 legacy=0x7f57c81418d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:51.272 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.178+0000 7f57c7fff700 1 -- 172.21.15.102:0/2528022184 reap_dead start 2021-05-25T12:07:51.273 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.178+0000 7f57c7fff700 1 -- 172.21.15.102:0/2528022184 reap_dead start 2021-05-25T12:07:51.273 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.178+0000 7f57ce217700 1 -- 172.21.15.102:0/2528022184 shutdown_connections 2021-05-25T12:07:51.273 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.178+0000 7f57ce217700 1 -- 172.21.15.102:0/2528022184 >> 172.21.15.102:0/2528022184 conn(0x7f57c809b370 msgr2=0x7f57c81254d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:07:51.273 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.178+0000 7f57ce217700 1 -- 172.21.15.102:0/2528022184 shutdown_connections 2021-05-25T12:07:51.274 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.178+0000 7f57ce217700 1 -- 172.21.15.102:0/2528022184 wait complete. 2021-05-25T12:07:51.274 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.178+0000 7f57ce217700 1 librados: shutdown 2021-05-25T12:07:51.274 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2021-05-25T12:07:51.385 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.384+0000 7f4f9f30f700 10 monclient: get_monmap_and_config 2021-05-25T12:07:51.385 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.384+0000 7f4f9f30f700 10 monclient: build_initial_monmap 2021-05-25T12:07:51.385 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.384+0000 7f4f9f30f700 10 monclient: monmap: 2021-05-25T12:07:51.386 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:07:51.386 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:51.386 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:07:51.385019+0000 2021-05-25T12:07:51.386 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:07:51.385019+0000 2021-05-25T12:07:51.387 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:07:51.387 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:51.387 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:07:51.387 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:07:51.387 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:07:51.388 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:51.401 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.384+0000 7f4f9f30f700 1 Processor -- start 2021-05-25T12:07:51.401 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.384+0000 7f4f9f30f700 1 -- start start 2021-05-25T12:07:51.402 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.384+0000 7f4f9f30f700 10 monclient: init 2021-05-25T12:07:51.402 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.385+0000 7f4f9f30f700 10 monclient: _reopen_session rank -1 2021-05-25T12:07:51.402 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.385+0000 7f4f9f30f700 10 monclient: _add_conns ranks=[1,2,0] 2021-05-25T12:07:51.403 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.385+0000 7f4f9f30f700 10 monclient(hunting): picked mon.noname-c con 0x7f4f98090990 addr v1:172.21.15.181:6789/0 2021-05-25T12:07:51.403 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.385+0000 7f4f9f30f700 10 monclient(hunting): picked mon.noname-b con 0x7f4f98090d70 addr v1:172.21.15.102:6790/0 2021-05-25T12:07:51.403 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.385+0000 7f4f9f30f700 10 monclient(hunting): picked mon.noname-a con 0x7f4f9812e330 addr v1:172.21.15.102:6789/0 2021-05-25T12:07:51.403 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.385+0000 7f4f9f30f700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f98096fa0 con 0x7f4f9812e330 2021-05-25T12:07:51.404 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.385+0000 7f4f9f30f700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f980926a0 con 0x7f4f98090990 2021-05-25T12:07:51.404 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.385+0000 7f4f9f30f700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f98092940 con 0x7f4f98090d70 2021-05-25T12:07:51.404 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.385+0000 7f4f9f30f700 10 monclient(hunting): _renew_subs 2021-05-25T12:07:51.404 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.385+0000 7f4f9f30f700 10 monclient(hunting): authenticate will time out at 1271.982258s 2021-05-25T12:07:51.405 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.385+0000 7f4f9c8aa700 1 -- >> v1:172.21.15.102:6790/0 conn(0x7f4f98090d70 legacy=0x7f4f98091200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6790/0 2021-05-25T12:07:51.405 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.385+0000 7f4f9d8ac700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f4f9812e330 0x7f4f98130720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:34018/0 (socket says 172.21.15.102:34018) 2021-05-25T12:07:51.405 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.385+0000 7f4f9d8ac700 1 -- 172.21.15.102:0/3949506314 learned_addr learned my addr 172.21.15.102:0/3949506314 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:07:51.405 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.385+0000 7f4f87fff700 1 -- 172.21.15.102:0/3949506314 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4158845784 0 0) 0x7f4f98096fa0 con 0x7f4f9812e330 2021-05-25T12:07:51.405 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.385+0000 7f4f87fff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:51.406 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.385+0000 7f4f87fff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:51.406 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.385+0000 7f4f87fff700 10 monclient(hunting): my global_id is 204465 2021-05-25T12:07:51.406 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.385+0000 7f4f87fff700 1 -- 172.21.15.102:0/3949506314 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f80003270 con 0x7f4f9812e330 2021-05-25T12:07:51.406 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.385+0000 7f4f87fff700 1 -- 172.21.15.102:0/3949506314 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1394643389 0 0) 0x7f4f80003270 con 0x7f4f9812e330 2021-05-25T12:07:51.407 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.385+0000 7f4f87fff700 1 -- 172.21.15.102:0/3949506314 >> v1:172.21.15.102:6790/0 conn(0x7f4f98090d70 legacy=0x7f4f98091200 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-05-25T12:07:51.407 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.385+0000 7f4f87fff700 1 -- 172.21.15.102:0/3949506314 >> v1:172.21.15.181:6789/0 conn(0x7f4f98090990 legacy=0x7f4f98093240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:51.407 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.385+0000 7f4f87fff700 10 monclient: _finish_hunting 0 2021-05-25T12:07:51.407 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.385+0000 7f4f87fff700 1 monclient: found mon.noname-a 2021-05-25T12:07:51.407 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.385+0000 7f4f87fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:07:51.408 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.385+0000 7f4f87fff700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T12:07:51.408 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.385+0000 7f4f87fff700 1 -- 172.21.15.102:0/3949506314 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f98092be0 con 0x7f4f9812e330 2021-05-25T12:07:51.408 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.385+0000 7f4f87fff700 10 monclient: _finish_auth 0 2021-05-25T12:07:51.408 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.385+0000 7f4f87fff700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:07:51.409 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.385+0000 7f4f87fff700 1 -- 172.21.15.102:0/3949506314 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f4f94001990 con 0x7f4f9812e330 2021-05-25T12:07:51.409 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.385+0000 7f4f9f30f700 5 monclient: authenticate success, global_id 204465 2021-05-25T12:07:51.409 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.385+0000 7f4f9f30f700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:07:51.409 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.385+0000 7f4f87fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:51.409 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.385+0000 7f4f87fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:07:51.410 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.385+0000 7f4f87fff700 10 monclient: dump: 2021-05-25T12:07:51.410 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:51.410 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:51.410 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:51.410 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:51.411 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:51.411 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:51.411 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:51.411 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:51.412 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:51.412 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:51.412 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.385+0000 7f4f9f30f700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:07:51.412 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.385+0000 7f4f87fff700 1 -- 172.21.15.102:0/3949506314 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4f94001c40 con 0x7f4f9812e330 2021-05-25T12:07:51.412 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.385+0000 7f4f87fff700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:07:51.413 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.385+0000 7f4f87fff700 1 -- 172.21.15.102:0/3949506314 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f4f94002010 con 0x7f4f9812e330 2021-05-25T12:07:51.413 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.385+0000 7f4f9f30f700 10 monclient: get_monmap_and_config success 2021-05-25T12:07:51.413 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.385+0000 7f4f87fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:51.413 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.385+0000 7f4f9f30f700 4 set_mon_vals no callback set 2021-05-25T12:07:51.414 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.385+0000 7f4f87fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:07:51.414 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.385+0000 7f4f87fff700 10 monclient: dump: 2021-05-25T12:07:51.414 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:51.414 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:51.414 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:51.415 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:51.415 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:51.415 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:51.415 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:51.416 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:51.416 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:51.416 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:51.416 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.386+0000 7f4f9f30f700 10 monclient: shutdown 2021-05-25T12:07:51.417 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.386+0000 7f4f9f30f700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:07:51.417 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.386+0000 7f4f9f30f700 1 -- 172.21.15.102:0/3949506314 >> v1:172.21.15.102:6789/0 conn(0x7f4f9812e330 legacy=0x7f4f98130720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:51.417 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.386+0000 7f4f9f30f700 1 -- 172.21.15.102:0/3949506314 shutdown_connections 2021-05-25T12:07:51.417 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.386+0000 7f4f9f30f700 1 -- 172.21.15.102:0/3949506314 >> 172.21.15.102:0/3949506314 conn(0x7f4f9809b370 msgr2=0x7f4f98097430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:07:51.417 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.386+0000 7f4f9f30f700 1 -- 172.21.15.102:0/3949506314 shutdown_connections 2021-05-25T12:07:51.418 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.386+0000 7f4f9f30f700 1 -- 172.21.15.102:0/3949506314 wait complete. 2021-05-25T12:07:51.418 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.386+0000 7f4f9f30f700 10 monclient: build_initial_monmap 2021-05-25T12:07:51.418 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.387+0000 7f4f9f30f700 10 monclient: monmap: 2021-05-25T12:07:51.418 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:07:51.419 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:07:51.419 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:07:51.419 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:07:51.419 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:07:51.419 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:07:51.420 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:07:51.420 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:07:51.420 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:07:51.420 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:51.421 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.387+0000 7f4f9f30f700 1 librados: starting msgr at 2021-05-25T12:07:51.421 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.387+0000 7f4f9f30f700 1 librados: starting objecter 2021-05-25T12:07:51.421 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.387+0000 7f4f9f30f700 1 Processor -- start 2021-05-25T12:07:51.421 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.387+0000 7f4f9f30f700 1 -- start start 2021-05-25T12:07:51.421 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.387+0000 7f4f9f30f700 1 librados: setting wanted keys 2021-05-25T12:07:51.422 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.387+0000 7f4f9f30f700 1 librados: calling monclient init 2021-05-25T12:07:51.422 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.387+0000 7f4f9f30f700 10 monclient: init 2021-05-25T12:07:51.422 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.387+0000 7f4f9f30f700 10 monclient: _reopen_session rank -1 2021-05-25T12:07:51.422 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.387+0000 7f4f9f30f700 10 monclient: _add_conns ranks=[2,1,0] 2021-05-25T12:07:51.423 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.387+0000 7f4f9f30f700 10 monclient(hunting): picked mon.noname-c con 0x7f4f98090990 addr v1:172.21.15.102:6790/0 2021-05-25T12:07:51.423 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.387+0000 7f4f9f30f700 10 monclient(hunting): picked mon.noname-b con 0x7f4f98090d70 addr v1:172.21.15.181:6789/0 2021-05-25T12:07:51.423 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.387+0000 7f4f9f30f700 10 monclient(hunting): picked mon.noname-a con 0x7f4f9812e330 addr v1:172.21.15.102:6789/0 2021-05-25T12:07:51.423 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.387+0000 7f4f9f30f700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f98068650 con 0x7f4f9812e330 2021-05-25T12:07:51.423 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.387+0000 7f4f9f30f700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f980667f0 con 0x7f4f98090d70 2021-05-25T12:07:51.424 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.387+0000 7f4f9f30f700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f9812ebf0 con 0x7f4f98090990 2021-05-25T12:07:51.424 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.387+0000 7f4f9f30f700 10 monclient(hunting): _renew_subs 2021-05-25T12:07:51.424 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.387+0000 7f4f9d0ab700 1 -- >> v1:172.21.15.102:6790/0 conn(0x7f4f98090990 legacy=0x7f4f98136ea0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.102:6790/0 2021-05-25T12:07:51.424 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.387+0000 7f4f9f30f700 10 monclient(hunting): authenticate will time out at 1271.984997s 2021-05-25T12:07:51.425 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.388+0000 7f4f9d8ac700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f4f9812e330 0x7f4f9813f230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:34024/0 (socket says 172.21.15.102:34024) 2021-05-25T12:07:51.425 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.388+0000 7f4f9d8ac700 1 -- 172.21.15.102:0/1362518470 learned_addr learned my addr 172.21.15.102:0/1362518470 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:07:51.425 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.388+0000 7f4f857fa700 10 client.?.objecter ms_handle_connect 0x7f4f9812e330 2021-05-25T12:07:51.425 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.388+0000 7f4f857fa700 10 client.?.objecter resend_mon_ops 2021-05-25T12:07:51.425 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.388+0000 7f4f857fa700 10 client.?.objecter ms_handle_connect 0x7f4f98090d70 2021-05-25T12:07:51.426 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.388+0000 7f4f857fa700 10 client.?.objecter resend_mon_ops 2021-05-25T12:07:51.426 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.388+0000 7f4f857fa700 1 -- 172.21.15.102:0/1362518470 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3681911670 0 0) 0x7f4f98068650 con 0x7f4f9812e330 2021-05-25T12:07:51.426 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.388+0000 7f4f857fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:51.426 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.388+0000 7f4f857fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:51.427 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.388+0000 7f4f857fa700 10 monclient(hunting): my global_id is 204468 2021-05-25T12:07:51.427 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.388+0000 7f4f857fa700 1 -- 172.21.15.102:0/1362518470 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f78004c10 con 0x7f4f9812e330 2021-05-25T12:07:51.427 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.388+0000 7f4f857fa700 1 -- 172.21.15.102:0/1362518470 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2851559146 0 0) 0x7f4f980667f0 con 0x7f4f98090d70 2021-05-25T12:07:51.427 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.388+0000 7f4f857fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:51.427 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.388+0000 7f4f857fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:51.428 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.388+0000 7f4f857fa700 10 monclient(hunting): my global_id is 224326 2021-05-25T12:07:51.428 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.388+0000 7f4f857fa700 1 -- 172.21.15.102:0/1362518470 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f98068650 con 0x7f4f98090d70 2021-05-25T12:07:51.428 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.388+0000 7f4f857fa700 1 -- 172.21.15.102:0/1362518470 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 484256104 0 0) 0x7f4f78004c10 con 0x7f4f9812e330 2021-05-25T12:07:51.428 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.388+0000 7f4f857fa700 1 -- 172.21.15.102:0/1362518470 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4f980667f0 con 0x7f4f9812e330 2021-05-25T12:07:51.428 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.388+0000 7f4f857fa700 1 -- 172.21.15.102:0/1362518470 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f4f94001f50 con 0x7f4f9812e330 2021-05-25T12:07:51.429 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.388+0000 7f4f857fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:51.429 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.388+0000 7f4f857fa700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:07:51.429 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.388+0000 7f4f857fa700 10 monclient(hunting): dump: 2021-05-25T12:07:51.429 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:51.430 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:51.430 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:51.430 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:51.430 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:51.430 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:51.431 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:51.431 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:51.431 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:51.431 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:51.432 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.388+0000 7f4f857fa700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:07:51.432 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.388+0000 7f4f857fa700 1 -- 172.21.15.102:0/1362518470 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 444316577 0 0) 0x7f4f98068650 con 0x7f4f98090d70 2021-05-25T12:07:51.432 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.388+0000 7f4f857fa700 1 -- 172.21.15.102:0/1362518470 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4f78004c10 con 0x7f4f98090d70 2021-05-25T12:07:51.432 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.388+0000 7f4f857fa700 1 -- 172.21.15.102:0/1362518470 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f4f8c003e20 con 0x7f4f98090d70 2021-05-25T12:07:51.432 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.388+0000 7f4f857fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:51.433 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.388+0000 7f4f857fa700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2021-05-25T12:07:51.433 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.388+0000 7f4f857fa700 10 monclient(hunting): dump: 2021-05-25T12:07:51.433 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:51.433 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:51.434 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:51.434 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:51.434 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:51.434 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:51.435 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:51.435 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:51.435 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:51.435 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:51.436 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.388+0000 7f4f857fa700 1 -- 172.21.15.102:0/1362518470 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3249472088 0 0) 0x7f4f980667f0 con 0x7f4f9812e330 2021-05-25T12:07:51.436 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.388+0000 7f4f857fa700 1 -- 172.21.15.102:0/1362518470 >> v1:172.21.15.102:6790/0 conn(0x7f4f98090990 legacy=0x7f4f98136ea0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-05-25T12:07:51.436 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.388+0000 7f4f857fa700 1 -- 172.21.15.102:0/1362518470 >> v1:172.21.15.181:6789/0 conn(0x7f4f98090d70 legacy=0x7f4f9813bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:51.436 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.388+0000 7f4f857fa700 10 monclient: _finish_hunting 0 2021-05-25T12:07:51.437 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.388+0000 7f4f857fa700 1 monclient: found mon.a 2021-05-25T12:07:51.437 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.388+0000 7f4f857fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:07:51.437 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.388+0000 7f4f857fa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:07:51.437 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.388+0000 7f4f857fa700 1 -- 172.21.15.102:0/1362518470 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f981375b0 con 0x7f4f9812e330 2021-05-25T12:07:51.438 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.388+0000 7f4f857fa700 10 monclient: _finish_auth 0 2021-05-25T12:07:51.438 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.388+0000 7f4f857fa700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:07:51.438 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.388+0000 7f4f9f30f700 5 monclient: authenticate success, global_id 204468 2021-05-25T12:07:51.438 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.388+0000 7f4f9f30f700 10 monclient: _renew_subs 2021-05-25T12:07:51.438 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.388+0000 7f4f9f30f700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:07:51.439 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.388+0000 7f4f9f30f700 1 -- 172.21.15.102:0/1362518470 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f98067260 con 0x7f4f9812e330 2021-05-25T12:07:51.439 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.388+0000 7f4f9f30f700 10 client.204468.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:07:51.439 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.388+0000 7f4f9f30f700 10 monclient: _renew_subs 2021-05-25T12:07:51.439 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.388+0000 7f4f9f30f700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:07:51.440 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.388+0000 7f4f9f30f700 1 -- 172.21.15.102:0/1362518470 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f981266e0 con 0x7f4f9812e330 2021-05-25T12:07:51.440 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.388+0000 7f4f9f30f700 1 librados: init done 2021-05-25T12:07:51.440 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.388+0000 7f4f857fa700 1 -- 172.21.15.102:0/1362518470 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4f94004d80 con 0x7f4f9812e330 2021-05-25T12:07:51.441 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.388+0000 7f4f857fa700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:07:51.441 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.388+0000 7f4f857fa700 1 -- 172.21.15.102:0/1362518470 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f4f94005110 con 0x7f4f9812e330 2021-05-25T12:07:51.441 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.388+0000 7f4f857fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:51.441 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.388+0000 7f4f857fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:07:51.442 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.388+0000 7f4f877fe700 4 set_mon_vals no callback set 2021-05-25T12:07:51.442 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.388+0000 7f4f857fa700 10 monclient: dump: 2021-05-25T12:07:51.442 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:51.442 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:51.442 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:51.443 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:51.443 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:51.443 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:51.443 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:51.444 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:51.444 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:51.444 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:51.444 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.389+0000 7f4f9f30f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:07:51.445 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.389+0000 7f4f9f30f700 1 -- 172.21.15.102:0/1362518470 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f4f700065a0 con 0x7f4f70002840 2021-05-25T12:07:51.445 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.389+0000 7f4f857fa700 1 -- 172.21.15.102:0/1362518470 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f4f94005270 con 0x7f4f9812e330 2021-05-25T12:07:51.445 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.389+0000 7f4f857fa700 1 -- 172.21.15.102:0/1362518470 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(247..247 src has 1..247) v4 ==== 6241+0+0 (unknown 3601026478 0 0) 0x7f4f9403ddf0 con 0x7f4f9812e330 2021-05-25T12:07:51.445 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.389+0000 7f4f857fa700 10 client.204468.objecter ms_dispatch 0x7f4f98001550 osd_map(247..247 src has 1..247) v4 2021-05-25T12:07:51.445 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.389+0000 7f4f857fa700 3 client.204468.objecter handle_osd_map got epochs [247,247] > 0 2021-05-25T12:07:51.446 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.389+0000 7f4f857fa700 3 client.204468.objecter handle_osd_map decoding full epoch 247 2021-05-25T12:07:51.446 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.389+0000 7f4f857fa700 20 client.204468.objecter dump_active .. 0 homeless 2021-05-25T12:07:51.446 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.389+0000 7f4f857fa700 10 client.204468.objecter ms_handle_connect 0x7f4f70002840 2021-05-25T12:07:51.446 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.389+0000 7f4f857fa700 10 client.204468.objecter resend_mon_ops 2021-05-25T12:07:51.446 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.389+0000 7f4f9c8aa700 10 monclient: get_auth_request con 0x7f4f780332a0 auth_method 0 2021-05-25T12:07:51.447 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.389+0000 7f4f857fa700 1 -- 172.21.15.102:0/1362518470 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2227124052 0 0) 0x7f4f700065a0 con 0x7f4f70002840 2021-05-25T12:07:51.447 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.389+0000 7f4f857fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:51.447 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.389+0000 7f4f857fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:51.448 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.389+0000 7f4f857fa700 10 monclient(hunting): my global_id is 224329 2021-05-25T12:07:51.448 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.389+0000 7f4f857fa700 1 -- 172.21.15.102:0/1362518470 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f980667f0 con 0x7f4f70002840 2021-05-25T12:07:51.448 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.390+0000 7f4f857fa700 10 client.204468.objecter ms_handle_connect 0x7f4f780332a0 2021-05-25T12:07:51.448 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.390+0000 7f4f857fa700 1 -- 172.21.15.102:0/1362518470 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1189770603 0 0) 0x7f4f980667f0 con 0x7f4f70002840 2021-05-25T12:07:51.448 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.390+0000 7f4f857fa700 1 -- 172.21.15.102:0/1362518470 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4f700065a0 con 0x7f4f70002840 2021-05-25T12:07:51.449 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.390+0000 7f4f857fa700 1 -- 172.21.15.102:0/1362518470 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f4f88002790 con 0x7f4f70002840 2021-05-25T12:07:51.449 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.390+0000 7f4f857fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:51.449 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.390+0000 7f4f857fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T12:07:51.449 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.390+0000 7f4f857fa700 10 monclient: dump: 2021-05-25T12:07:51.450 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:51.450 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:51.450 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:51.450 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:51.450 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:51.451 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:51.451 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:51.451 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:51.451 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:51.451 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:51.452 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.390+0000 7f4f857fa700 1 -- 172.21.15.102:0/1362518470 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1949736801 0 0) 0x7f4f700065a0 con 0x7f4f70002840 2021-05-25T12:07:51.452 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.390+0000 7f4f857fa700 1 -- 172.21.15.102:0/1362518470 --> v1:172.21.15.181:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4f70009e40 con 0x7f4f70002840 2021-05-25T12:07:51.452 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.390+0000 7f4f857fa700 1 -- 172.21.15.102:0/1362518470 <== mon.1 v1:172.21.15.181:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7f4f88002c40 con 0x7f4f70002840 2021-05-25T12:07:51.452 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.390+0000 7f4f857fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:07:51.453 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.390+0000 7f4f857fa700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:07:51.453 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.390+0000 7f4f857fa700 1 -- 172.21.15.102:0/1362518470 >> v1:172.21.15.181:6789/0 conn(0x7f4f70002840 legacy=0x7f4f70004ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:51.453 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.390+0000 7f4f857fa700 1 -- 172.21.15.102:0/1362518470 >> v1:172.21.15.181:6789/0 conn(0x7f4f70002840 legacy=0x7f4f70004ce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:07:51.453 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.397+0000 7f4f9f30f700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:07:51.453 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.397+0000 7f4f9f30f700 1 -- 172.21.15.102:0/1362518470 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f4f70007540 con 0x7f4f700068e0 2021-05-25T12:07:51.454 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.397+0000 7f4f857fa700 10 client.204468.objecter ms_handle_connect 0x7f4f700068e0 2021-05-25T12:07:51.454 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.397+0000 7f4f857fa700 10 client.204468.objecter resend_mon_ops 2021-05-25T12:07:51.454 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.397+0000 7f4f857fa700 1 -- 172.21.15.102:0/1362518470 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3665352101 0 0) 0x7f4f70007540 con 0x7f4f700068e0 2021-05-25T12:07:51.454 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.397+0000 7f4f857fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:07:51.454 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.397+0000 7f4f857fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:07:51.455 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.397+0000 7f4f857fa700 10 monclient(hunting): my global_id is 224332 2021-05-25T12:07:51.455 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.397+0000 7f4f857fa700 1 -- 172.21.15.102:0/1362518470 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f700065a0 con 0x7f4f700068e0 2021-05-25T12:07:51.455 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.397+0000 7f4f857fa700 1 -- 172.21.15.102:0/1362518470 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3947319654 0 0) 0x7f4f700065a0 con 0x7f4f700068e0 2021-05-25T12:07:51.455 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.397+0000 7f4f857fa700 1 -- 172.21.15.102:0/1362518470 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4f70007540 con 0x7f4f700068e0 2021-05-25T12:07:51.455 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.397+0000 7f4f857fa700 1 -- 172.21.15.102:0/1362518470 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f4f880016c0 con 0x7f4f700068e0 2021-05-25T12:07:51.456 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.397+0000 7f4f857fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:07:51.456 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.397+0000 7f4f857fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T12:07:51.456 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.397+0000 7f4f857fa700 10 monclient: dump: 2021-05-25T12:07:51.456 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:07:51.456 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:07:51.457 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:51.457 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:07:51.457 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:07:51.457 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:07:51.458 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:07:51.458 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:07:51.458 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:07:51.458 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:07:51.458 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.398+0000 7f4f857fa700 1 -- 172.21.15.102:0/1362518470 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2838370932 0 0) 0x7f4f70007540 con 0x7f4f700068e0 2021-05-25T12:07:51.459 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.398+0000 7f4f857fa700 1 -- 172.21.15.102:0/1362518470 --> v1:172.21.15.181:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f4f70008770 con 0x7f4f700068e0 2021-05-25T12:07:51.459 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.398+0000 7f4f857fa700 1 -- 172.21.15.102:0/1362518470 <== mon.1 v1:172.21.15.181:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3931 (unknown 0 0 3694468569) 0x7f4f880019d0 con 0x7f4f700068e0 2021-05-25T12:07:51.459 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.398+0000 7f4f857fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:07:51.459 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.398+0000 7f4f857fa700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:07:51.460 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.398+0000 7f4f857fa700 1 -- 172.21.15.102:0/1362518470 >> v1:172.21.15.181:6789/0 conn(0x7f4f700068e0 legacy=0x7f4f70006d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:51.460 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.398+0000 7f4f857fa700 1 -- 172.21.15.102:0/1362518470 >> v1:172.21.15.181:6789/0 conn(0x7f4f700068e0 legacy=0x7f4f70006d70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:07:51.460 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:07:51.461 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:07:51.461 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:07:51.461 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "leader", 2021-05-25T12:07:51.461 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 66, 2021-05-25T12:07:51.461 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:07:51.462 INFO:teuthology.orchestra.run.smithi102.stdout: 0, 2021-05-25T12:07:51.462 INFO:teuthology.orchestra.run.smithi102.stdout: 1 2021-05-25T12:07:51.462 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:07:51.462 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 0, 2021-05-25T12:07:51.463 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:07:51.463 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:07:51.463 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:07:51.463 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:07:51.463 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:07:51.464 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:07:51.464 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:07:51.464 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:07:51.464 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:07:51.465 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:07:51.465 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:07:51.465 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:07:51.465 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:07:51.465 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:07:51.466 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:07:51.466 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:07:51.466 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:07:51.466 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:07:51.467 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:07:51.467 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:07:51.467 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:07:51.467 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:07:51.467 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:07:51.468 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:07:51.468 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:07:51.468 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:51.468 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:07:51.468 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:07:51.469 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:07:51.469 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:07:51.469 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:07:51.469 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:07:51.469 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:07:51.470 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:07:51.470 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:07:51.470 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:07:51.470 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:07:51.471 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:07:51.471 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:07:51.471 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:07:51.471 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:07:51.471 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:07:51.472 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:07:51.472 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:07:51.472 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:07:51.472 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:07:51.472 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:07:51.473 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:07:51.473 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:07:51.473 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:07:51.473 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:07:51.474 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:07:51.474 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:51.474 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:07:51.474 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:51.474 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:07:51.475 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:07:51.475 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:07:51.475 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:07:51.475 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:51.475 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:07:51.476 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:07:51.476 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:07:51.476 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:07:51.476 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:07:51.476 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:51.477 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:07:51.477 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:07:51.477 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:07:51.477 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:07:51.478 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:07:51.478 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:51.478 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:51.478 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:07:51.478 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:07:51.479 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:07:51.479 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:07:51.479 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:51.479 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:07:51.480 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:07:51.480 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:07:51.480 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:07:51.480 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:07:51.480 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:51.480 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:07:51.481 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:07:51.481 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:07:51.481 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:07:51.481 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:07:51.482 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:51.482 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:51.482 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:07:51.482 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:07:51.482 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:07:51.483 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:07:51.483 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:51.483 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:07:51.483 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:07:51.483 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:07:51.484 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:07:51.484 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:07:51.484 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:51.484 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:07:51.485 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:07:51.485 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:07:51.485 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:07:51.485 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:07:51.485 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:07:51.486 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:07:51.486 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:51.486 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:07:51.486 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:07:51.486 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:51.487 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:07:51.487 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:07:51.487 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:07:51.487 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:07:51.488 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:07:51.488 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:07:51.488 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:51.488 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:07:51.488 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:07:51.489 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 4 2021-05-25T12:07:51.489 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:07:51.489 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:07:51.489 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": [ 2021-05-25T12:07:51.489 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:07:51.490 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:07:51.490 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:07:51.490 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:07:51.490 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:07:51.491 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:07:51.491 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:07:51.491 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:07:51.491 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:07:51.493 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.398+0000 7f4f9f30f700 10 librados: watch_flush enter 2021-05-25T12:07:51.493 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.398+0000 7f4f9f30f700 10 librados: watch_flush exit 2021-05-25T12:07:51.494 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.398+0000 7f4f9f30f700 20 client.204468.objecter shutdown clearing up homeless session... 2021-05-25T12:07:51.494 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.398+0000 7f4f9f30f700 10 client.204468.objecter successfully canceled tick 2021-05-25T12:07:51.494 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.398+0000 7f4f9f30f700 1 -- 172.21.15.102:0/1362518470 >> v1:172.21.15.181:6812/29582 conn(0x7f4f780332a0 legacy=0x7f4f78035760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:51.494 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.398+0000 7f4f9f30f700 10 monclient: shutdown 2021-05-25T12:07:51.495 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.398+0000 7f4f9f30f700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:07:51.495 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.398+0000 7f4f9f30f700 1 -- 172.21.15.102:0/1362518470 >> v1:172.21.15.102:6789/0 conn(0x7f4f9812e330 legacy=0x7f4f9813f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:07:51.495 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.398+0000 7f4f9d8ac700 1 -- 172.21.15.102:0/1362518470 reap_dead start 2021-05-25T12:07:51.495 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.398+0000 7f4f9f30f700 1 -- 172.21.15.102:0/1362518470 shutdown_connections 2021-05-25T12:07:51.495 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.398+0000 7f4f9f30f700 1 -- 172.21.15.102:0/1362518470 >> 172.21.15.102:0/1362518470 conn(0x7f4f9809b370 msgr2=0x7f4f981292f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:07:51.496 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.398+0000 7f4f9f30f700 1 -- 172.21.15.102:0/1362518470 shutdown_connections 2021-05-25T12:07:51.496 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.398+0000 7f4f9f30f700 1 -- 172.21.15.102:0/1362518470 wait complete. 2021-05-25T12:07:51.496 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:07:51.398+0000 7f4f9f30f700 1 librados: shutdown 2021-05-25T12:07:51.496 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2021-05-25T12:08:11.497 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2021-05-25T12:08:11.498 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2021-05-25T12:08:11.498 INFO:tasks.ceph.mon.c:Restarting daemon 2021-05-25T12:08:11.499 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2021-05-25T12:08:11.502 INFO:tasks.ceph.mon.c:Started 2021-05-25T12:08:11.503 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2021-05-25T12:08:11.503 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2021-05-25T12:08:11.632 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.630+0000 7f85924f6700 10 monclient: get_monmap_and_config 2021-05-25T12:08:11.632 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.630+0000 7f85924f6700 10 monclient: build_initial_monmap 2021-05-25T12:08:11.632 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.630+0000 7f85924f6700 10 monclient: monmap: 2021-05-25T12:08:11.633 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:08:11.633 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:11.633 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:08:11.631691+0000 2021-05-25T12:08:11.633 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:08:11.631691+0000 2021-05-25T12:08:11.634 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:08:11.634 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:11.634 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:08:11.634 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:08:11.635 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:08:11.635 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:11.636 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.631+0000 7f85924f6700 1 Processor -- start 2021-05-25T12:08:11.637 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.631+0000 7f85924f6700 1 -- start start 2021-05-25T12:08:11.637 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.631+0000 7f85924f6700 10 monclient: init 2021-05-25T12:08:11.637 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.631+0000 7f85924f6700 10 monclient: _reopen_session rank -1 2021-05-25T12:08:11.637 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.631+0000 7f85924f6700 10 monclient: _add_conns ranks=[0,1,2] 2021-05-25T12:08:11.638 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.631+0000 7f85924f6700 10 monclient(hunting): picked mon.noname-a con 0x7f858c090990 addr v1:172.21.15.102:6789/0 2021-05-25T12:08:11.638 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.631+0000 7f85924f6700 10 monclient(hunting): picked mon.noname-c con 0x7f858c090d70 addr v1:172.21.15.181:6789/0 2021-05-25T12:08:11.638 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.632+0000 7f85924f6700 10 monclient(hunting): picked mon.noname-b con 0x7f858c12e330 addr v1:172.21.15.102:6790/0 2021-05-25T12:08:11.638 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.632+0000 7f85924f6700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f858c096fa0 con 0x7f858c090990 2021-05-25T12:08:11.639 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.632+0000 7f85924f6700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f858c0926a0 con 0x7f858c090d70 2021-05-25T12:08:11.639 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.632+0000 7f85924f6700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f858c092940 con 0x7f858c12e330 2021-05-25T12:08:11.639 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.632+0000 7f85924f6700 10 monclient(hunting): _renew_subs 2021-05-25T12:08:11.639 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.632+0000 7f85924f6700 10 monclient(hunting): authenticate will time out at 1292.229038s 2021-05-25T12:08:11.640 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.632+0000 7f858bfff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f858c090990 0x7f858c093240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:34060/0 (socket says 172.21.15.102:34060) 2021-05-25T12:08:11.640 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.632+0000 7f8590a93700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f858c12e330 0x7f858c130720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:57172/0 (socket says 172.21.15.102:57172) 2021-05-25T12:08:11.640 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.632+0000 7f858bfff700 1 -- 172.21.15.102:0/610772312 learned_addr learned my addr 172.21.15.102:0/610772312 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:08:11.650 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.644+0000 7f858affd700 1 -- 172.21.15.102:0/610772312 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 856980805 0 0) 0x7f858c096fa0 con 0x7f858c090990 2021-05-25T12:08:11.650 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.644+0000 7f858affd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:11.650 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.644+0000 7f858affd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:11.650 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.644+0000 7f858affd700 10 monclient(hunting): my global_id is 204489 2021-05-25T12:08:11.651 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.644+0000 7f858affd700 1 -- 172.21.15.102:0/610772312 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8574003270 con 0x7f858c090990 2021-05-25T12:08:11.651 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.644+0000 7f858affd700 1 -- 172.21.15.102:0/610772312 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2251565741 0 0) 0x7f8574003270 con 0x7f858c090990 2021-05-25T12:08:11.651 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.644+0000 7f858affd700 1 -- 172.21.15.102:0/610772312 >> v1:172.21.15.102:6790/0 conn(0x7f858c12e330 legacy=0x7f858c130720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:11.651 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.644+0000 7f858affd700 1 -- 172.21.15.102:0/610772312 >> v1:172.21.15.181:6789/0 conn(0x7f858c090d70 legacy=0x7f858c091200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:11.652 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.644+0000 7f858affd700 10 monclient: _finish_hunting 0 2021-05-25T12:08:11.652 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.644+0000 7f858affd700 1 monclient: found mon.noname-a 2021-05-25T12:08:11.652 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.644+0000 7f858affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:08:11.652 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.644+0000 7f858affd700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T12:08:11.652 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.644+0000 7f858affd700 1 -- 172.21.15.102:0/610772312 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f858c092be0 con 0x7f858c090990 2021-05-25T12:08:11.653 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.644+0000 7f858affd700 10 monclient: _finish_auth 0 2021-05-25T12:08:11.653 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.644+0000 7f858affd700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:08:11.653 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.644+0000 7f858affd700 1 -- 172.21.15.102:0/610772312 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f8578001610 con 0x7f858c090990 2021-05-25T12:08:11.653 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.644+0000 7f858affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:11.653 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.644+0000 7f858affd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:08:11.654 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.644+0000 7f858affd700 10 monclient: dump: 2021-05-25T12:08:11.654 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:11.654 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:11.654 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:11.655 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:11.655 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:11.655 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:11.655 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:11.655 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:11.656 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:11.656 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:11.656 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.644+0000 7f85924f6700 5 monclient: authenticate success, global_id 204489 2021-05-25T12:08:11.656 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.644+0000 7f85924f6700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:08:11.656 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.645+0000 7f858affd700 1 -- 172.21.15.102:0/610772312 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f85780018c0 con 0x7f858c090990 2021-05-25T12:08:11.657 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.645+0000 7f858affd700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:08:11.657 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.645+0000 7f85924f6700 10 monclient: get_monmap_and_config success 2021-05-25T12:08:11.657 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.645+0000 7f858affd700 1 -- 172.21.15.102:0/610772312 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f8578001c10 con 0x7f858c090990 2021-05-25T12:08:11.657 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.645+0000 7f85924f6700 4 set_mon_vals no callback set 2021-05-25T12:08:11.657 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.645+0000 7f858affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:11.658 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.645+0000 7f858affd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:08:11.658 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.645+0000 7f858affd700 10 monclient: dump: 2021-05-25T12:08:11.658 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:11.658 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:11.659 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:11.659 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:11.659 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:11.659 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:11.659 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:11.660 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:11.660 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:11.660 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:11.660 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.645+0000 7f85924f6700 10 monclient: shutdown 2021-05-25T12:08:11.660 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.645+0000 7f85924f6700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:08:11.661 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.645+0000 7f85924f6700 1 -- 172.21.15.102:0/610772312 >> v1:172.21.15.102:6789/0 conn(0x7f858c090990 legacy=0x7f858c093240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:11.661 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.645+0000 7f85924f6700 1 -- 172.21.15.102:0/610772312 shutdown_connections 2021-05-25T12:08:11.661 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.645+0000 7f85924f6700 1 -- 172.21.15.102:0/610772312 >> 172.21.15.102:0/610772312 conn(0x7f858c09b370 msgr2=0x7f858c097430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:08:11.661 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.645+0000 7f85924f6700 1 -- 172.21.15.102:0/610772312 shutdown_connections 2021-05-25T12:08:11.662 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.645+0000 7f85924f6700 1 -- 172.21.15.102:0/610772312 wait complete. 2021-05-25T12:08:11.662 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.646+0000 7f85924f6700 10 monclient: build_initial_monmap 2021-05-25T12:08:11.662 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.646+0000 7f85924f6700 10 monclient: monmap: 2021-05-25T12:08:11.662 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:08:11.663 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:08:11.663 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:08:11.663 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:08:11.663 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:08:11.664 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:08:11.664 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:08:11.664 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:08:11.664 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:08:11.665 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:11.665 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.646+0000 7f85924f6700 1 librados: starting msgr at 2021-05-25T12:08:11.665 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.646+0000 7f85924f6700 1 librados: starting objecter 2021-05-25T12:08:11.665 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.646+0000 7f85924f6700 1 Processor -- start 2021-05-25T12:08:11.665 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.646+0000 7f85924f6700 1 -- start start 2021-05-25T12:08:11.666 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.646+0000 7f85924f6700 1 librados: setting wanted keys 2021-05-25T12:08:11.666 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.646+0000 7f85924f6700 1 librados: calling monclient init 2021-05-25T12:08:11.666 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.646+0000 7f85924f6700 10 monclient: init 2021-05-25T12:08:11.666 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.647+0000 7f85924f6700 10 monclient: _reopen_session rank -1 2021-05-25T12:08:11.666 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.647+0000 7f85924f6700 10 monclient: _add_conns ranks=[2,0,1] 2021-05-25T12:08:11.667 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.647+0000 7f85924f6700 10 monclient(hunting): picked mon.noname-c con 0x7f858c090990 addr v1:172.21.15.102:6790/0 2021-05-25T12:08:11.667 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.647+0000 7f85924f6700 10 monclient(hunting): picked mon.noname-a con 0x7f858c090d70 addr v1:172.21.15.102:6789/0 2021-05-25T12:08:11.667 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.647+0000 7f85924f6700 10 monclient(hunting): picked mon.noname-b con 0x7f858c12e330 addr v1:172.21.15.181:6789/0 2021-05-25T12:08:11.667 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.647+0000 7f85924f6700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f858c0685f0 con 0x7f858c090d70 2021-05-25T12:08:11.667 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.647+0000 7f85924f6700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f858c068890 con 0x7f858c12e330 2021-05-25T12:08:11.668 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.647+0000 7f85924f6700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f858c068b30 con 0x7f858c090990 2021-05-25T12:08:11.668 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.647+0000 7f85924f6700 10 monclient(hunting): _renew_subs 2021-05-25T12:08:11.668 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.647+0000 7f85924f6700 10 monclient(hunting): authenticate will time out at 1292.244273s 2021-05-25T12:08:11.668 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.647+0000 7f858bfff700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f858c090990 0x7f858c13b2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:57174/0 (socket says 172.21.15.102:57174) 2021-05-25T12:08:11.669 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.647+0000 7f858bfff700 1 -- 172.21.15.102:0/2186333455 learned_addr learned my addr 172.21.15.102:0/2186333455 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:08:11.669 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.647+0000 7f857ffff700 10 client.?.objecter ms_handle_connect 0x7f858c090d70 2021-05-25T12:08:11.669 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.647+0000 7f857ffff700 10 client.?.objecter resend_mon_ops 2021-05-25T12:08:11.669 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.647+0000 7f857ffff700 10 client.?.objecter ms_handle_connect 0x7f858c090990 2021-05-25T12:08:11.670 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.647+0000 7f857ffff700 10 client.?.objecter resend_mon_ops 2021-05-25T12:08:11.670 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.648+0000 7f857ffff700 10 client.?.objecter ms_handle_connect 0x7f858c12e330 2021-05-25T12:08:11.670 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.648+0000 7f857ffff700 10 client.?.objecter resend_mon_ops 2021-05-25T12:08:11.675 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.667+0000 7f857ffff700 1 -- 172.21.15.102:0/2186333455 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3914086023 0 0) 0x7f858c068890 con 0x7f858c12e330 2021-05-25T12:08:11.675 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.667+0000 7f857ffff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:11.675 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.667+0000 7f857ffff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:11.676 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.667+0000 7f857ffff700 10 monclient(hunting): my global_id is 224353 2021-05-25T12:08:11.676 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.667+0000 7f857ffff700 1 -- 172.21.15.102:0/2186333455 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8568004c10 con 0x7f858c12e330 2021-05-25T12:08:11.676 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.669+0000 7f857ffff700 1 -- 172.21.15.102:0/2186333455 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2234991123 0 0) 0x7f858c0685f0 con 0x7f858c090d70 2021-05-25T12:08:11.676 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.669+0000 7f857ffff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:11.676 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.669+0000 7f857ffff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:11.677 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.669+0000 7f857ffff700 10 monclient(hunting): my global_id is 204492 2021-05-25T12:08:11.677 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.669+0000 7f857ffff700 1 -- 172.21.15.102:0/2186333455 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f858c068890 con 0x7f858c090d70 2021-05-25T12:08:11.677 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.669+0000 7f857ffff700 1 -- 172.21.15.102:0/2186333455 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1847333047 0 0) 0x7f858c068890 con 0x7f858c090d70 2021-05-25T12:08:11.677 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.669+0000 7f857ffff700 1 -- 172.21.15.102:0/2186333455 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f858c0685f0 con 0x7f858c090d70 2021-05-25T12:08:11.678 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.669+0000 7f857ffff700 1 -- 172.21.15.102:0/2186333455 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f8570001880 con 0x7f858c090d70 2021-05-25T12:08:11.678 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.669+0000 7f857ffff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:11.678 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.669+0000 7f857ffff700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:08:11.678 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.669+0000 7f857ffff700 10 monclient(hunting): dump: 2021-05-25T12:08:11.678 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:11.679 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:11.679 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:11.679 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:11.679 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:11.679 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:11.680 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:11.680 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:11.680 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:11.680 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:11.680 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.669+0000 7f857ffff700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:08:11.681 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.669+0000 7f857ffff700 1 -- 172.21.15.102:0/2186333455 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3145841254 0 0) 0x7f8568004c10 con 0x7f858c12e330 2021-05-25T12:08:11.681 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.669+0000 7f857ffff700 1 -- 172.21.15.102:0/2186333455 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f858c068890 con 0x7f858c12e330 2021-05-25T12:08:11.681 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.669+0000 7f857ffff700 1 -- 172.21.15.102:0/2186333455 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f8580001ab0 con 0x7f858c12e330 2021-05-25T12:08:11.681 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.669+0000 7f857ffff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:11.682 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.669+0000 7f857ffff700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2021-05-25T12:08:11.682 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.669+0000 7f857ffff700 10 monclient(hunting): dump: 2021-05-25T12:08:11.682 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:11.682 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:11.683 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:11.683 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:11.683 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:11.683 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:11.684 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:11.684 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:11.684 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:11.684 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:11.684 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.669+0000 7f857ffff700 1 -- 172.21.15.102:0/2186333455 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 882353748 0 0) 0x7f858c068890 con 0x7f858c12e330 2021-05-25T12:08:11.685 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.669+0000 7f857ffff700 1 -- 172.21.15.102:0/2186333455 >> v1:172.21.15.102:6790/0 conn(0x7f858c090990 legacy=0x7f858c13b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:11.685 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.669+0000 7f857ffff700 1 -- 172.21.15.102:0/2186333455 >> v1:172.21.15.102:6789/0 conn(0x7f858c090d70 legacy=0x7f858c140130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:11.685 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.669+0000 7f857ffff700 10 monclient: _finish_hunting 0 2021-05-25T12:08:11.685 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.669+0000 7f857ffff700 1 monclient: found mon.b 2021-05-25T12:08:11.686 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.669+0000 7f857ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:08:11.686 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.669+0000 7f857ffff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T12:08:11.686 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.669+0000 7f857ffff700 1 -- 172.21.15.102:0/2186333455 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f858c0669c0 con 0x7f858c12e330 2021-05-25T12:08:11.686 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.669+0000 7f857ffff700 10 monclient: _finish_auth 0 2021-05-25T12:08:11.686 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.669+0000 7f857ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:08:11.687 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.669+0000 7f85924f6700 5 monclient: authenticate success, global_id 224353 2021-05-25T12:08:11.687 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.669+0000 7f85924f6700 10 monclient: _renew_subs 2021-05-25T12:08:11.687 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.669+0000 7f85924f6700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T12:08:11.687 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.669+0000 7f85924f6700 1 -- 172.21.15.102:0/2186333455 --> v1:172.21.15.181:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f858c067670 con 0x7f858c12e330 2021-05-25T12:08:11.687 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.669+0000 7f85924f6700 10 client.224353.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:08:11.688 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.669+0000 7f85924f6700 10 monclient: _renew_subs 2021-05-25T12:08:11.688 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.669+0000 7f85924f6700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T12:08:11.688 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.669+0000 7f85924f6700 1 -- 172.21.15.102:0/2186333455 --> v1:172.21.15.181:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f858c13bb80 con 0x7f858c12e330 2021-05-25T12:08:11.688 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.669+0000 7f85924f6700 1 librados: init done 2021-05-25T12:08:11.688 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.670+0000 7f85924f6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:08:11.689 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.670+0000 7f85924f6700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:08:11.689 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.670+0000 7f85924f6700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T12:08:11.689 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.670+0000 7f85924f6700 1 -- 172.21.15.102:0/2186333455 --> v1:172.21.15.181:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f858c059730 con 0x7f858c12e330 2021-05-25T12:08:11.689 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.671+0000 7f857ffff700 1 -- 172.21.15.102:0/2186333455 <== mon.1 v1:172.21.15.181:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f85800020b0 con 0x7f858c12e330 2021-05-25T12:08:11.689 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.671+0000 7f857ffff700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:08:11.690 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.671+0000 7f857ffff700 1 -- 172.21.15.102:0/2186333455 <== mon.1 v1:172.21.15.181:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f8580004a50 con 0x7f858c12e330 2021-05-25T12:08:11.690 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.671+0000 7f857ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:11.690 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.671+0000 7f857ffff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T12:08:11.690 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.671+0000 7f857ffff700 10 monclient: dump: 2021-05-25T12:08:11.691 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:11.691 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:11.691 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:11.691 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:11.692 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:11.692 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:11.692 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:11.692 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:11.692 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:11.693 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:11.693 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.671+0000 7f858a7fc700 4 set_mon_vals no callback set 2021-05-25T12:08:11.693 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.672+0000 7f857ffff700 1 -- 172.21.15.102:0/2186333455 <== mon.1 v1:172.21.15.181:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f8580004bb0 con 0x7f858c12e330 2021-05-25T12:08:11.693 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.672+0000 7f857ffff700 1 -- 172.21.15.102:0/2186333455 <== mon.1 v1:172.21.15.181:6789/0 8 ==== osd_map(267..267 src has 1..267) v4 ==== 5832+0+0 (unknown 1812453150 0 0) 0x7f858003d7c0 con 0x7f858c12e330 2021-05-25T12:08:11.693 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.672+0000 7f857ffff700 10 client.224353.objecter ms_dispatch 0x7f858c001550 osd_map(267..267 src has 1..267) v4 2021-05-25T12:08:11.694 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.672+0000 7f857ffff700 3 client.224353.objecter handle_osd_map got epochs [267,267] > 0 2021-05-25T12:08:11.694 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.672+0000 7f857ffff700 3 client.224353.objecter handle_osd_map decoding full epoch 267 2021-05-25T12:08:11.694 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.672+0000 7f857ffff700 20 client.224353.objecter dump_active .. 0 homeless 2021-05-25T12:08:11.694 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.672+0000 7f858bfff700 10 monclient: get_auth_request con 0x7f85680332a0 auth_method 0 2021-05-25T12:08:11.695 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.672+0000 7f857ffff700 10 client.224353.objecter ms_handle_connect 0x7f85680332a0 2021-05-25T12:08:11.695 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.676+0000 7f857ffff700 1 -- 172.21.15.102:0/2186333455 <== mon.1 v1:172.21.15.181:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7f8580012c00 con 0x7f858c12e330 2021-05-25T12:08:11.695 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.676+0000 7f857ffff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:08:11.695 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.676+0000 7f857ffff700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:08:11.891 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.890+0000 7f85924f6700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2021-05-25T12:08:11.892 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.890+0000 7f85924f6700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2021-05-25T12:08:11.892 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.890+0000 7f85924f6700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T12:08:11.892 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.890+0000 7f85924f6700 1 -- 172.21.15.102:0/2186333455 --> v1:172.21.15.181:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f858c066250 con 0x7f858c12e330 2021-05-25T12:08:11.893 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.891+0000 7f857ffff700 1 -- 172.21.15.102:0/2186333455 <== mon.1 v1:172.21.15.181:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1342 (unknown 254746657 0 1845516103) 0x7f8580012a20 con 0x7f858c12e330 2021-05-25T12:08:11.893 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.891+0000 7f857ffff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2021-05-25T12:08:11.893 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.891+0000 7f857ffff700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:08:11.894 INFO:teuthology.orchestra.run.smithi102.stdout:{"election_epoch":68,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138303579357183","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"20ee91b6-8935-4b98-b868-85c03789ccc4","modified":"2021-05-25T11:59:11.011522Z","created":"2021-05-25T11:59:11.011522Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6790","nonce":0}]},"addr":"172.21.15.102:6790/0","public_addr":"172.21.15.102:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2021-05-25T12:08:11.895 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.892+0000 7f85924f6700 10 librados: watch_flush enter 2021-05-25T12:08:11.896 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.893+0000 7f85924f6700 10 librados: watch_flush exit 2021-05-25T12:08:11.896 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.893+0000 7f85924f6700 20 client.224353.objecter shutdown clearing up homeless session... 2021-05-25T12:08:11.896 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.893+0000 7f85924f6700 10 client.224353.objecter successfully canceled tick 2021-05-25T12:08:11.896 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.893+0000 7f85924f6700 1 -- 172.21.15.102:0/2186333455 >> v1:172.21.15.181:6812/29582 conn(0x7f85680332a0 legacy=0x7f8568035760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:11.897 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.893+0000 7f85924f6700 10 monclient: shutdown 2021-05-25T12:08:11.897 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.893+0000 7f85924f6700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:08:11.897 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.893+0000 7f85924f6700 1 -- 172.21.15.102:0/2186333455 >> v1:172.21.15.181:6789/0 conn(0x7f858c12e330 legacy=0x7f858c143860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:11.897 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.893+0000 7f85924f6700 1 -- 172.21.15.102:0/2186333455 shutdown_connections 2021-05-25T12:08:11.897 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.893+0000 7f85924f6700 1 -- 172.21.15.102:0/2186333455 >> 172.21.15.102:0/2186333455 conn(0x7f858c09b370 msgr2=0x7f858c129450 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:08:11.898 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.893+0000 7f85924f6700 1 -- 172.21.15.102:0/2186333455 shutdown_connections 2021-05-25T12:08:11.898 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.893+0000 7f85924f6700 1 -- 172.21.15.102:0/2186333455 wait complete. 2021-05-25T12:08:11.898 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:11.893+0000 7f85924f6700 1 librados: shutdown 2021-05-25T12:08:11.905 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2021-05-25T12:08:11.906 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2021-05-25T12:08:12.015 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.013+0000 7f09248d1700 10 monclient: get_monmap_and_config 2021-05-25T12:08:12.015 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.013+0000 7f09248d1700 10 monclient: build_initial_monmap 2021-05-25T12:08:12.016 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.013+0000 7f09248d1700 10 monclient: monmap: 2021-05-25T12:08:12.016 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:08:12.016 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:12.016 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:08:12.014625+0000 2021-05-25T12:08:12.017 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:08:12.014625+0000 2021-05-25T12:08:12.017 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:08:12.017 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:12.017 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:08:12.017 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:08:12.018 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:08:12.018 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:12.018 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.014+0000 7f09248d1700 1 Processor -- start 2021-05-25T12:08:12.018 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.014+0000 7f09248d1700 1 -- start start 2021-05-25T12:08:12.019 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.014+0000 7f09248d1700 10 monclient: init 2021-05-25T12:08:12.034 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.014+0000 7f09248d1700 10 monclient: _reopen_session rank -1 2021-05-25T12:08:12.035 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.014+0000 7f09248d1700 10 monclient: _add_conns ranks=[1,2,0] 2021-05-25T12:08:12.035 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.014+0000 7f09248d1700 10 monclient(hunting): picked mon.noname-c con 0x7f0920068d50 addr v1:172.21.15.181:6789/0 2021-05-25T12:08:12.035 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.014+0000 7f09248d1700 10 monclient(hunting): picked mon.noname-b con 0x7f0920069130 addr v1:172.21.15.102:6790/0 2021-05-25T12:08:12.035 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.014+0000 7f09248d1700 10 monclient(hunting): picked mon.noname-a con 0x7f09200695e0 addr v1:172.21.15.102:6789/0 2021-05-25T12:08:12.036 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.014+0000 7f09248d1700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0920094380 con 0x7f09200695e0 2021-05-25T12:08:12.036 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.014+0000 7f09248d1700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0920094620 con 0x7f0920068d50 2021-05-25T12:08:12.036 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.014+0000 7f09248d1700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09200948c0 con 0x7f0920069130 2021-05-25T12:08:12.036 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.014+0000 7f09248d1700 10 monclient(hunting): _renew_subs 2021-05-25T12:08:12.036 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.014+0000 7f09248d1700 10 monclient(hunting): authenticate will time out at 1292.611871s 2021-05-25T12:08:12.037 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.015+0000 7f091ed9d700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f09200695e0 0x7f0920132160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:34072/0 (socket says 172.21.15.102:34072) 2021-05-25T12:08:12.037 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.015+0000 7f091dd9b700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f0920069130 0x7f092012ea30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:57182/0 (socket says 172.21.15.102:57182) 2021-05-25T12:08:12.037 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.015+0000 7f091ed9d700 1 -- 172.21.15.102:0/413398794 learned_addr learned my addr 172.21.15.102:0/413398794 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:08:12.037 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.015+0000 7f091d59a700 1 -- 172.21.15.102:0/413398794 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4168052899 0 0) 0x7f09200948c0 con 0x7f0920069130 2021-05-25T12:08:12.037 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.015+0000 7f091d59a700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:12.038 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.015+0000 7f091d59a700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:12.038 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.015+0000 7f091d59a700 10 monclient(hunting): my global_id is 244100 2021-05-25T12:08:12.038 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.015+0000 7f091d59a700 1 -- 172.21.15.102:0/413398794 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0904003270 con 0x7f0920069130 2021-05-25T12:08:12.038 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.015+0000 7f091d59a700 1 -- 172.21.15.102:0/413398794 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 471128742 0 0) 0x7f0920094380 con 0x7f09200695e0 2021-05-25T12:08:12.039 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.015+0000 7f091d59a700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:12.039 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.015+0000 7f091d59a700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:12.039 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.015+0000 7f091d59a700 10 monclient(hunting): my global_id is 204495 2021-05-25T12:08:12.039 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.015+0000 7f091d59a700 1 -- 172.21.15.102:0/413398794 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09200948c0 con 0x7f09200695e0 2021-05-25T12:08:12.039 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.015+0000 7f091d59a700 1 -- 172.21.15.102:0/413398794 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 771598363 0 0) 0x7f0920094620 con 0x7f0920068d50 2021-05-25T12:08:12.040 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.015+0000 7f091d59a700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:12.040 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.015+0000 7f091d59a700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:12.040 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.015+0000 7f091d59a700 10 monclient(hunting): my global_id is 224356 2021-05-25T12:08:12.040 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.015+0000 7f091d59a700 1 -- 172.21.15.102:0/413398794 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0920094380 con 0x7f0920068d50 2021-05-25T12:08:12.041 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.015+0000 7f091d59a700 1 -- 172.21.15.102:0/413398794 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 267852398 0 0) 0x7f0904003270 con 0x7f0920069130 2021-05-25T12:08:12.041 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.015+0000 7f091d59a700 1 -- 172.21.15.102:0/413398794 >> v1:172.21.15.181:6789/0 conn(0x7f0920068d50 legacy=0x7f0920129c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:12.041 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.015+0000 7f091d59a700 1 -- 172.21.15.102:0/413398794 >> v1:172.21.15.102:6789/0 conn(0x7f09200695e0 legacy=0x7f0920132160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:12.041 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.015+0000 7f091d59a700 10 monclient: _finish_hunting 0 2021-05-25T12:08:12.042 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.015+0000 7f091d59a700 1 monclient: found mon.noname-b 2021-05-25T12:08:12.042 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.015+0000 7f091d59a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:08:12.042 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.015+0000 7f091d59a700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T12:08:12.042 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.015+0000 7f091d59a700 1 -- 172.21.15.102:0/413398794 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0920094c80 con 0x7f0920069130 2021-05-25T12:08:12.042 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.015+0000 7f091d59a700 10 monclient: _finish_auth 0 2021-05-25T12:08:12.043 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.015+0000 7f091d59a700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:08:12.043 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.015+0000 7f09248d1700 5 monclient: authenticate success, global_id 244100 2021-05-25T12:08:12.043 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.015+0000 7f091d59a700 1 -- 172.21.15.102:0/413398794 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f0908001610 con 0x7f0920069130 2021-05-25T12:08:12.043 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.015+0000 7f091d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:12.043 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.015+0000 7f091d59a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:08:12.044 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.015+0000 7f091d59a700 10 monclient: dump: 2021-05-25T12:08:12.044 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:12.044 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:12.044 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:12.045 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:12.045 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:12.045 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:12.045 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:12.046 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:12.046 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:12.046 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:12.046 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.015+0000 7f09248d1700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:08:12.046 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.015+0000 7f091d59a700 1 -- 172.21.15.102:0/413398794 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f09080018f0 con 0x7f0920069130 2021-05-25T12:08:12.047 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.015+0000 7f091d59a700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:08:12.047 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.015+0000 7f091d59a700 1 -- 172.21.15.102:0/413398794 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f0908001c80 con 0x7f0920069130 2021-05-25T12:08:12.047 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.015+0000 7f09248d1700 10 monclient: get_monmap_and_config success 2021-05-25T12:08:12.047 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.015+0000 7f09248d1700 4 set_mon_vals no callback set 2021-05-25T12:08:12.048 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.015+0000 7f091d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:12.048 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.015+0000 7f091d59a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:08:12.048 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.015+0000 7f091d59a700 10 monclient: dump: 2021-05-25T12:08:12.048 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:12.048 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:12.049 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:12.049 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:12.049 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:12.049 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:12.050 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:12.050 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:12.050 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:12.050 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:12.051 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.016+0000 7f09248d1700 10 monclient: shutdown 2021-05-25T12:08:12.051 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.016+0000 7f09248d1700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:08:12.051 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.016+0000 7f09248d1700 1 -- 172.21.15.102:0/413398794 >> v1:172.21.15.102:6790/0 conn(0x7f0920069130 legacy=0x7f092012ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:12.051 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.016+0000 7f09248d1700 1 -- 172.21.15.102:0/413398794 shutdown_connections 2021-05-25T12:08:12.051 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.016+0000 7f09248d1700 1 -- 172.21.15.102:0/413398794 >> 172.21.15.102:0/413398794 conn(0x7f092009b370 msgr2=0x7f0920097430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:08:12.052 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.016+0000 7f09248d1700 1 -- 172.21.15.102:0/413398794 shutdown_connections 2021-05-25T12:08:12.052 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.016+0000 7f09248d1700 1 -- 172.21.15.102:0/413398794 wait complete. 2021-05-25T12:08:12.052 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.016+0000 7f09248d1700 10 monclient: build_initial_monmap 2021-05-25T12:08:12.052 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.016+0000 7f09248d1700 10 monclient: monmap: 2021-05-25T12:08:12.053 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:08:12.053 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:08:12.053 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:08:12.053 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:08:12.053 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:08:12.054 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:08:12.054 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:08:12.054 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:08:12.054 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:08:12.055 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:12.055 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.016+0000 7f09248d1700 1 librados: starting msgr at 2021-05-25T12:08:12.055 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.016+0000 7f09248d1700 1 librados: starting objecter 2021-05-25T12:08:12.055 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.016+0000 7f09248d1700 1 Processor -- start 2021-05-25T12:08:12.056 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.016+0000 7f09248d1700 1 -- start start 2021-05-25T12:08:12.056 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.016+0000 7f09248d1700 1 librados: setting wanted keys 2021-05-25T12:08:12.056 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.016+0000 7f09248d1700 1 librados: calling monclient init 2021-05-25T12:08:12.056 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.016+0000 7f09248d1700 10 monclient: init 2021-05-25T12:08:12.056 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.017+0000 7f09248d1700 10 monclient: _reopen_session rank -1 2021-05-25T12:08:12.057 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.017+0000 7f09248d1700 10 monclient: _add_conns ranks=[2,0,1] 2021-05-25T12:08:12.057 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.017+0000 7f09248d1700 10 monclient(hunting): picked mon.noname-c con 0x7f0920068d50 addr v1:172.21.15.102:6790/0 2021-05-25T12:08:12.057 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.017+0000 7f09248d1700 10 monclient(hunting): picked mon.noname-a con 0x7f0920069130 addr v1:172.21.15.102:6789/0 2021-05-25T12:08:12.057 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.017+0000 7f09248d1700 10 monclient(hunting): picked mon.noname-b con 0x7f09200695e0 addr v1:172.21.15.181:6789/0 2021-05-25T12:08:12.057 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.017+0000 7f09248d1700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0920128af0 con 0x7f0920069130 2021-05-25T12:08:12.058 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.017+0000 7f09248d1700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f092012f5a0 con 0x7f09200695e0 2021-05-25T12:08:12.058 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.017+0000 7f09248d1700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f092012ea00 con 0x7f0920068d50 2021-05-25T12:08:12.058 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.017+0000 7f09248d1700 10 monclient(hunting): _renew_subs 2021-05-25T12:08:12.058 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.017+0000 7f09248d1700 10 monclient(hunting): authenticate will time out at 1292.614347s 2021-05-25T12:08:12.059 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.017+0000 7f091dd9b700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f0920069130 0x7f0920132a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:34078/0 (socket says 172.21.15.102:34078) 2021-05-25T12:08:12.059 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.017+0000 7f091dd9b700 1 -- 172.21.15.102:0/1080030418 learned_addr learned my addr 172.21.15.102:0/1080030418 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:08:12.059 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.017+0000 7f091e59c700 1 --1- 172.21.15.102:0/1080030418 >> v1:172.21.15.102:6790/0 conn(0x7f0920068d50 0x7f0920138e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:57186/0 (socket says 172.21.15.102:57186) 2021-05-25T12:08:12.059 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.017+0000 7f090e7fc700 10 client.?.objecter ms_handle_connect 0x7f0920069130 2021-05-25T12:08:12.059 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.017+0000 7f090e7fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:08:12.060 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.017+0000 7f090e7fc700 10 client.?.objecter ms_handle_connect 0x7f0920068d50 2021-05-25T12:08:12.060 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.017+0000 7f090e7fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:08:12.060 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.017+0000 7f090e7fc700 1 -- 172.21.15.102:0/1080030418 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1935223726 0 0) 0x7f0920128af0 con 0x7f0920069130 2021-05-25T12:08:12.060 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.017+0000 7f090e7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:12.061 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.017+0000 7f090e7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:12.061 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.017+0000 7f090e7fc700 10 monclient(hunting): my global_id is 204498 2021-05-25T12:08:12.061 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.017+0000 7f090e7fc700 1 -- 172.21.15.102:0/1080030418 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08fc004c10 con 0x7f0920069130 2021-05-25T12:08:12.061 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.017+0000 7f090e7fc700 10 client.?.objecter ms_handle_connect 0x7f09200695e0 2021-05-25T12:08:12.061 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.017+0000 7f090e7fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:08:12.062 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.017+0000 7f090e7fc700 1 -- 172.21.15.102:0/1080030418 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 252637427 0 0) 0x7f092012ea00 con 0x7f0920068d50 2021-05-25T12:08:12.062 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.017+0000 7f090e7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:12.062 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.017+0000 7f090e7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:12.062 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.017+0000 7f090e7fc700 10 monclient(hunting): my global_id is 244103 2021-05-25T12:08:12.063 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.017+0000 7f090e7fc700 1 -- 172.21.15.102:0/1080030418 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0920128af0 con 0x7f0920068d50 2021-05-25T12:08:12.063 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.017+0000 7f090e7fc700 1 -- 172.21.15.102:0/1080030418 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 405459842 0 0) 0x7f092012f5a0 con 0x7f09200695e0 2021-05-25T12:08:12.063 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.017+0000 7f090e7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:12.063 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.017+0000 7f090e7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:12.063 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.017+0000 7f090e7fc700 10 monclient(hunting): my global_id is 224359 2021-05-25T12:08:12.064 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.017+0000 7f090e7fc700 1 -- 172.21.15.102:0/1080030418 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f092012ea00 con 0x7f09200695e0 2021-05-25T12:08:12.064 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.017+0000 7f090e7fc700 1 -- 172.21.15.102:0/1080030418 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3145596985 0 0) 0x7f08fc004c10 con 0x7f0920069130 2021-05-25T12:08:12.064 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.017+0000 7f090e7fc700 1 -- 172.21.15.102:0/1080030418 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f092012f5a0 con 0x7f0920069130 2021-05-25T12:08:12.064 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.017+0000 7f090e7fc700 1 -- 172.21.15.102:0/1080030418 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f09080015d0 con 0x7f0920069130 2021-05-25T12:08:12.065 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.017+0000 7f090e7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:12.065 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.017+0000 7f090e7fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:08:12.065 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.017+0000 7f090e7fc700 10 monclient(hunting): dump: 2021-05-25T12:08:12.065 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:12.065 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:12.066 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:12.066 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:12.066 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:12.066 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:12.067 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:12.067 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:12.067 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:12.067 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:12.067 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.017+0000 7f090e7fc700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:08:12.068 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.017+0000 7f090e7fc700 1 -- 172.21.15.102:0/1080030418 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3057212496 0 0) 0x7f0920128af0 con 0x7f0920068d50 2021-05-25T12:08:12.068 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.017+0000 7f090e7fc700 1 -- 172.21.15.102:0/1080030418 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f08fc004c10 con 0x7f0920068d50 2021-05-25T12:08:12.068 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.017+0000 7f090e7fc700 1 -- 172.21.15.102:0/1080030418 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f0914002910 con 0x7f0920068d50 2021-05-25T12:08:12.068 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.017+0000 7f090e7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:12.068 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.017+0000 7f090e7fc700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2021-05-25T12:08:12.069 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.017+0000 7f090e7fc700 10 monclient(hunting): dump: 2021-05-25T12:08:12.069 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:12.069 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:12.069 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:12.069 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:12.070 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:12.070 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:12.070 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:12.070 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:12.071 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:12.071 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:12.071 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.018+0000 7f090e7fc700 1 -- 172.21.15.102:0/1080030418 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3917826494 0 0) 0x7f092012ea00 con 0x7f09200695e0 2021-05-25T12:08:12.071 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.018+0000 7f090e7fc700 1 -- 172.21.15.102:0/1080030418 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0920128af0 con 0x7f09200695e0 2021-05-25T12:08:12.071 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.018+0000 7f090e7fc700 1 -- 172.21.15.102:0/1080030418 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1453935563 0 0) 0x7f092012f5a0 con 0x7f0920069130 2021-05-25T12:08:12.072 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.018+0000 7f090e7fc700 1 -- 172.21.15.102:0/1080030418 >> v1:172.21.15.102:6790/0 conn(0x7f0920068d50 legacy=0x7f0920138e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:12.072 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.018+0000 7f090e7fc700 1 -- 172.21.15.102:0/1080030418 >> v1:172.21.15.181:6789/0 conn(0x7f09200695e0 legacy=0x7f0920141190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:12.072 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.018+0000 7f090e7fc700 10 monclient: _finish_hunting 0 2021-05-25T12:08:12.072 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.018+0000 7f090e7fc700 1 monclient: found mon.a 2021-05-25T12:08:12.073 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.018+0000 7f090e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:08:12.073 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.018+0000 7f090e7fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:08:12.073 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.018+0000 7f090e7fc700 1 -- 172.21.15.102:0/1080030418 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f092012d430 con 0x7f0920069130 2021-05-25T12:08:12.073 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.018+0000 7f090e7fc700 10 monclient: _finish_auth 0 2021-05-25T12:08:12.073 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.018+0000 7f090e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:08:12.074 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.018+0000 7f09248d1700 5 monclient: authenticate success, global_id 204498 2021-05-25T12:08:12.074 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.018+0000 7f09248d1700 10 monclient: _renew_subs 2021-05-25T12:08:12.074 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.018+0000 7f09248d1700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:08:12.074 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.018+0000 7f09248d1700 1 -- 172.21.15.102:0/1080030418 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0920139530 con 0x7f0920069130 2021-05-25T12:08:12.074 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.018+0000 7f09248d1700 10 client.204498.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:08:12.075 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.018+0000 7f09248d1700 10 monclient: _renew_subs 2021-05-25T12:08:12.075 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.018+0000 7f09248d1700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:08:12.075 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.018+0000 7f09248d1700 1 -- 172.21.15.102:0/1080030418 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0920133340 con 0x7f0920069130 2021-05-25T12:08:12.075 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.018+0000 7f09248d1700 1 librados: init done 2021-05-25T12:08:12.076 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.018+0000 7f090e7fc700 1 -- 172.21.15.102:0/1080030418 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0908004980 con 0x7f0920069130 2021-05-25T12:08:12.076 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.018+0000 7f090e7fc700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:08:12.076 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.018+0000 7f090e7fc700 1 -- 172.21.15.102:0/1080030418 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f0908004d10 con 0x7f0920069130 2021-05-25T12:08:12.076 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.018+0000 7f090e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:12.076 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.018+0000 7f091cd99700 4 set_mon_vals no callback set 2021-05-25T12:08:12.077 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.018+0000 7f090e7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:08:12.077 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.018+0000 7f090e7fc700 10 monclient: dump: 2021-05-25T12:08:12.077 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:12.077 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:12.077 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:12.078 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:12.078 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:12.078 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:12.078 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:12.079 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:12.079 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:12.079 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:12.079 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.018+0000 7f09248d1700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:08:12.079 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.018+0000 7f090e7fc700 1 -- 172.21.15.102:0/1080030418 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f0908004e70 con 0x7f0920069130 2021-05-25T12:08:12.080 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.018+0000 7f09248d1700 1 -- 172.21.15.102:0/1080030418 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f09000065a0 con 0x7f0900002840 2021-05-25T12:08:12.080 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.019+0000 7f090e7fc700 10 client.204498.objecter ms_handle_connect 0x7f0900002840 2021-05-25T12:08:12.080 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.019+0000 7f090e7fc700 10 client.204498.objecter resend_mon_ops 2021-05-25T12:08:12.080 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.019+0000 7f090e7fc700 1 -- 172.21.15.102:0/1080030418 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(268..268 src has 1..268) v4 ==== 6249+0+0 (unknown 118496375 0 0) 0x7f090803db30 con 0x7f0920069130 2021-05-25T12:08:12.081 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.019+0000 7f090e7fc700 10 client.204498.objecter ms_dispatch 0x7f0920001550 osd_map(268..268 src has 1..268) v4 2021-05-25T12:08:12.081 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.019+0000 7f090e7fc700 3 client.204498.objecter handle_osd_map got epochs [268,268] > 0 2021-05-25T12:08:12.081 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.019+0000 7f090e7fc700 3 client.204498.objecter handle_osd_map decoding full epoch 268 2021-05-25T12:08:12.081 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.019+0000 7f090e7fc700 20 client.204498.objecter dump_active .. 0 homeless 2021-05-25T12:08:12.081 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.019+0000 7f090e7fc700 1 -- 172.21.15.102:0/1080030418 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1862695552 0 0) 0x7f09000065a0 con 0x7f0900002840 2021-05-25T12:08:12.082 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.019+0000 7f090e7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:12.082 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.019+0000 7f090e7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:12.082 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.019+0000 7f090e7fc700 10 monclient(hunting): my global_id is 204501 2021-05-25T12:08:12.082 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.019+0000 7f090e7fc700 1 -- 172.21.15.102:0/1080030418 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f092012f5a0 con 0x7f0900002840 2021-05-25T12:08:12.083 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.019+0000 7f091ed9d700 10 monclient: get_auth_request con 0x7f08fc0334e0 auth_method 0 2021-05-25T12:08:12.083 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.019+0000 7f090e7fc700 1 -- 172.21.15.102:0/1080030418 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2424431673 0 0) 0x7f092012f5a0 con 0x7f0900002840 2021-05-25T12:08:12.083 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.019+0000 7f090e7fc700 1 -- 172.21.15.102:0/1080030418 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f09000065a0 con 0x7f0900002840 2021-05-25T12:08:12.083 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.019+0000 7f090e7fc700 1 -- 172.21.15.102:0/1080030418 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f0914001450 con 0x7f0900002840 2021-05-25T12:08:12.083 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.019+0000 7f090e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:12.084 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.019+0000 7f090e7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:08:12.084 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.019+0000 7f090e7fc700 10 monclient: dump: 2021-05-25T12:08:12.084 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:12.084 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:12.084 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:12.085 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:12.085 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:12.085 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:12.085 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:12.085 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:12.086 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:12.086 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:12.086 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.019+0000 7f090e7fc700 10 client.204498.objecter ms_handle_connect 0x7f08fc0334e0 2021-05-25T12:08:12.086 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.019+0000 7f090e7fc700 1 -- 172.21.15.102:0/1080030418 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1035488317 0 0) 0x7f09000065a0 con 0x7f0900002840 2021-05-25T12:08:12.087 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.019+0000 7f090e7fc700 1 -- 172.21.15.102:0/1080030418 --> v1:172.21.15.102:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0900009e40 con 0x7f0900002840 2021-05-25T12:08:12.087 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.019+0000 7f090e7fc700 1 -- 172.21.15.102:0/1080030418 <== mon.0 v1:172.21.15.102:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7f0914005510 con 0x7f0900002840 2021-05-25T12:08:12.087 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.019+0000 7f090e7fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:08:12.087 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.019+0000 7f090e7fc700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:08:12.087 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.019+0000 7f090e7fc700 1 -- 172.21.15.102:0/1080030418 >> v1:172.21.15.102:6789/0 conn(0x7f0900002840 legacy=0x7f0900004ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:12.088 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.019+0000 7f090e7fc700 1 -- 172.21.15.102:0/1080030418 >> v1:172.21.15.102:6789/0 conn(0x7f0900002840 legacy=0x7f0900004ce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:08:12.088 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.026+0000 7f09248d1700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:08:12.088 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.026+0000 7f09248d1700 1 -- 172.21.15.102:0/1080030418 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f0900009b00 con 0x7f0900008dc0 2021-05-25T12:08:12.088 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.026+0000 7f090e7fc700 10 client.204498.objecter ms_handle_connect 0x7f0900008dc0 2021-05-25T12:08:12.089 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.026+0000 7f090e7fc700 10 client.204498.objecter resend_mon_ops 2021-05-25T12:08:12.089 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.026+0000 7f090e7fc700 1 -- 172.21.15.102:0/1080030418 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4245749798 0 0) 0x7f0900009b00 con 0x7f0900008dc0 2021-05-25T12:08:12.089 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.026+0000 7f090e7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:12.089 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.026+0000 7f090e7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:12.089 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.026+0000 7f090e7fc700 10 monclient(hunting): my global_id is 204504 2021-05-25T12:08:12.090 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.026+0000 7f090e7fc700 1 -- 172.21.15.102:0/1080030418 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09000065a0 con 0x7f0900008dc0 2021-05-25T12:08:12.090 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.026+0000 7f090e7fc700 1 -- 172.21.15.102:0/1080030418 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3674490877 0 0) 0x7f09000065a0 con 0x7f0900008dc0 2021-05-25T12:08:12.090 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.026+0000 7f090e7fc700 1 -- 172.21.15.102:0/1080030418 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0900009b00 con 0x7f0900008dc0 2021-05-25T12:08:12.090 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.026+0000 7f090e7fc700 1 -- 172.21.15.102:0/1080030418 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f0914001ac0 con 0x7f0900008dc0 2021-05-25T12:08:12.091 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.026+0000 7f090e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:12.091 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.026+0000 7f090e7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:08:12.091 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.026+0000 7f090e7fc700 10 monclient: dump: 2021-05-25T12:08:12.091 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:12.091 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:12.092 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:12.092 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:12.092 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:12.092 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:12.093 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:12.093 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:12.093 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:12.093 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:12.093 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.026+0000 7f090e7fc700 1 -- 172.21.15.102:0/1080030418 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1832103616 0 0) 0x7f0900009b00 con 0x7f0900008dc0 2021-05-25T12:08:12.094 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.026+0000 7f090e7fc700 1 -- 172.21.15.102:0/1080030418 --> v1:172.21.15.102:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f0900008740 con 0x7f0900008dc0 2021-05-25T12:08:12.094 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.027+0000 7f090e7fc700 1 -- 172.21.15.102:0/1080030418 <== mon.0 v1:172.21.15.102:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3940 (unknown 0 0 2386733128) 0x7f0914001e40 con 0x7f0900008dc0 2021-05-25T12:08:12.094 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.027+0000 7f090e7fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:08:12.094 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.027+0000 7f090e7fc700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:08:12.095 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.027+0000 7f090e7fc700 1 -- 172.21.15.102:0/1080030418 >> v1:172.21.15.102:6789/0 conn(0x7f0900008dc0 legacy=0x7f0900009250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:12.095 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.027+0000 7f090e7fc700 1 -- 172.21.15.102:0/1080030418 >> v1:172.21.15.102:6789/0 conn(0x7f0900008dc0 legacy=0x7f0900009250 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:08:12.095 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:08:12.096 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:08:12.096 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:08:12.096 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "peon", 2021-05-25T12:08:12.096 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 68, 2021-05-25T12:08:12.096 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:08:12.097 INFO:teuthology.orchestra.run.smithi102.stdout: 0, 2021-05-25T12:08:12.097 INFO:teuthology.orchestra.run.smithi102.stdout: 1, 2021-05-25T12:08:12.097 INFO:teuthology.orchestra.run.smithi102.stdout: 2 2021-05-25T12:08:12.097 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:08:12.098 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 0, 2021-05-25T12:08:12.098 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:08:12.098 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:08:12.098 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:08:12.098 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:08:12.099 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:08:12.099 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:08:12.099 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:08:12.099 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:08:12.099 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:08:12.100 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:08:12.100 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:08:12.100 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:08:12.100 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:08:12.101 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:08:12.101 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:08:12.101 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:08:12.101 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:08:12.101 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:08:12.102 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:08:12.102 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:08:12.102 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:08:12.102 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:08:12.102 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:08:12.103 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:08:12.103 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:12.103 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:12.103 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:08:12.104 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:08:12.104 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:08:12.104 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:08:12.104 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:08:12.104 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:08:12.105 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:08:12.105 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:08:12.105 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:08:12.105 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:08:12.105 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:08:12.106 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:08:12.106 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:08:12.106 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:08:12.106 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:08:12.107 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:08:12.107 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:08:12.107 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:08:12.107 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:08:12.107 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:08:12.108 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:08:12.108 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:08:12.108 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:08:12.108 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:08:12.108 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:08:12.109 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:08:12.109 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:12.109 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:08:12.109 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:12.109 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:08:12.110 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:08:12.110 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:08:12.110 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:08:12.110 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:12.111 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:08:12.111 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:08:12.111 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:08:12.111 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:12.111 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:12.112 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:12.112 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:08:12.112 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:08:12.112 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:08:12.112 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:08:12.113 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:08:12.113 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:12.113 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:12.113 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:08:12.114 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:08:12.114 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:08:12.114 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:08:12.114 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:12.114 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:08:12.115 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:08:12.115 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:08:12.115 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:12.115 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:12.115 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:12.116 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:08:12.116 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:08:12.116 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:08:12.116 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:08:12.117 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:08:12.117 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:12.117 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:12.117 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:08:12.117 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:08:12.118 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:08:12.118 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:08:12.118 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:12.118 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:08:12.119 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:08:12.119 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:08:12.119 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:12.119 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:12.119 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:12.120 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:08:12.120 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:08:12.120 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:08:12.120 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:08:12.120 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:08:12.121 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:12.121 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:12.121 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:12.121 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:08:12.121 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:08:12.122 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:12.122 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:08:12.122 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:08:12.122 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:08:12.123 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:12.123 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:08:12.123 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": [ 2021-05-25T12:08:12.123 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:12.123 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:08:12.124 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:08:12.124 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 6 2021-05-25T12:08:12.124 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:12.124 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:08:12.124 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:08:12.125 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:12.125 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:08:12.125 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:08:12.125 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 4 2021-05-25T12:08:12.126 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:12.126 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:12.126 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:12.126 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:08:12.126 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:08:12.129 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.027+0000 7f09248d1700 10 librados: watch_flush enter 2021-05-25T12:08:12.129 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.027+0000 7f09248d1700 10 librados: watch_flush exit 2021-05-25T12:08:12.129 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.027+0000 7f09248d1700 20 client.204498.objecter shutdown clearing up homeless session... 2021-05-25T12:08:12.130 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.027+0000 7f09248d1700 10 client.204498.objecter successfully canceled tick 2021-05-25T12:08:12.130 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.027+0000 7f09248d1700 1 -- 172.21.15.102:0/1080030418 >> v1:172.21.15.181:6812/29582 conn(0x7f08fc0334e0 legacy=0x7f08fc0359a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:12.130 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.027+0000 7f09248d1700 10 monclient: shutdown 2021-05-25T12:08:12.130 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.027+0000 7f09248d1700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:08:12.131 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.027+0000 7f09248d1700 1 -- 172.21.15.102:0/1080030418 >> v1:172.21.15.102:6789/0 conn(0x7f0920069130 legacy=0x7f0920132a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:12.131 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.027+0000 7f091ed9d700 1 -- 172.21.15.102:0/1080030418 reap_dead start 2021-05-25T12:08:12.131 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.027+0000 7f091ed9d700 1 -- 172.21.15.102:0/1080030418 reap_dead start 2021-05-25T12:08:12.131 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.027+0000 7f09248d1700 1 -- 172.21.15.102:0/1080030418 shutdown_connections 2021-05-25T12:08:12.131 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.027+0000 7f09248d1700 1 -- 172.21.15.102:0/1080030418 >> 172.21.15.102:0/1080030418 conn(0x7f092009b370 msgr2=0x7f0920128150 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:08:12.132 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.027+0000 7f09248d1700 1 -- 172.21.15.102:0/1080030418 shutdown_connections 2021-05-25T12:08:12.132 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.027+0000 7f09248d1700 1 -- 172.21.15.102:0/1080030418 wait complete. 2021-05-25T12:08:12.132 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.027+0000 7f09248d1700 1 librados: shutdown 2021-05-25T12:08:12.133 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2021-05-25T12:08:12.250 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.249+0000 7f92ed089700 10 monclient: get_monmap_and_config 2021-05-25T12:08:12.251 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.249+0000 7f92ed089700 10 monclient: build_initial_monmap 2021-05-25T12:08:12.251 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.249+0000 7f92ed089700 10 monclient: monmap: 2021-05-25T12:08:12.251 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:08:12.252 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:12.252 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:08:12.250025+0000 2021-05-25T12:08:12.252 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:08:12.250025+0000 2021-05-25T12:08:12.252 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:08:12.252 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:12.253 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:08:12.253 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:08:12.253 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:08:12.253 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:12.253 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.249+0000 7f92ed089700 1 Processor -- start 2021-05-25T12:08:12.254 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.249+0000 7f92ed089700 1 -- start start 2021-05-25T12:08:12.254 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.249+0000 7f92ed089700 10 monclient: init 2021-05-25T12:08:12.268 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.250+0000 7f92ed089700 10 monclient: _reopen_session rank -1 2021-05-25T12:08:12.268 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.250+0000 7f92ed089700 10 monclient: _add_conns ranks=[1,2,0] 2021-05-25T12:08:12.268 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.250+0000 7f92ed089700 10 monclient(hunting): picked mon.noname-c con 0x7f92e8063840 addr v1:172.21.15.181:6789/0 2021-05-25T12:08:12.269 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.250+0000 7f92ed089700 10 monclient(hunting): picked mon.noname-b con 0x7f92e8094da0 addr v1:172.21.15.102:6790/0 2021-05-25T12:08:12.269 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.250+0000 7f92ed089700 10 monclient(hunting): picked mon.noname-a con 0x7f92e81323c0 addr v1:172.21.15.102:6789/0 2021-05-25T12:08:12.269 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.250+0000 7f92ed089700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92e8063fb0 con 0x7f92e81323c0 2021-05-25T12:08:12.269 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.250+0000 7f92ed089700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92e80982f0 con 0x7f92e8063840 2021-05-25T12:08:12.269 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.250+0000 7f92ed089700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92e8064510 con 0x7f92e8094da0 2021-05-25T12:08:12.270 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.250+0000 7f92ed089700 10 monclient(hunting): _renew_subs 2021-05-25T12:08:12.270 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.250+0000 7f92ed089700 10 monclient(hunting): authenticate will time out at 1292.847251s 2021-05-25T12:08:12.270 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.250+0000 7f92e759e700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f92e81323c0 0x7f92e81347b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:34092/0 (socket says 172.21.15.102:34092) 2021-05-25T12:08:12.270 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.250+0000 7f92e759e700 1 -- 172.21.15.102:0/937273942 learned_addr learned my addr 172.21.15.102:0/937273942 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:08:12.271 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.250+0000 7f92e659c700 1 --1- 172.21.15.102:0/937273942 >> v1:172.21.15.102:6790/0 conn(0x7f92e8094da0 0x7f92e8097be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:57200/0 (socket says 172.21.15.102:57200) 2021-05-25T12:08:12.271 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.250+0000 7f92e5d9b700 1 -- 172.21.15.102:0/937273942 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2107724977 0 0) 0x7f92e8064510 con 0x7f92e8094da0 2021-05-25T12:08:12.271 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.250+0000 7f92e5d9b700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:12.271 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.250+0000 7f92e5d9b700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:12.271 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.250+0000 7f92e5d9b700 10 monclient(hunting): my global_id is 244106 2021-05-25T12:08:12.272 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.250+0000 7f92e5d9b700 1 -- 172.21.15.102:0/937273942 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92cc003270 con 0x7f92e8094da0 2021-05-25T12:08:12.272 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.250+0000 7f92e5d9b700 1 -- 172.21.15.102:0/937273942 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2212721434 0 0) 0x7f92e8063fb0 con 0x7f92e81323c0 2021-05-25T12:08:12.272 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.250+0000 7f92e5d9b700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:12.272 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.250+0000 7f92e5d9b700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:12.272 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.250+0000 7f92e5d9b700 10 monclient(hunting): my global_id is 204507 2021-05-25T12:08:12.273 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.250+0000 7f92e5d9b700 1 -- 172.21.15.102:0/937273942 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92e8064510 con 0x7f92e81323c0 2021-05-25T12:08:12.273 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.250+0000 7f92e5d9b700 1 -- 172.21.15.102:0/937273942 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1682099825 0 0) 0x7f92e80982f0 con 0x7f92e8063840 2021-05-25T12:08:12.273 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.250+0000 7f92e5d9b700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:12.273 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.250+0000 7f92e5d9b700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:12.274 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.250+0000 7f92e5d9b700 10 monclient(hunting): my global_id is 224362 2021-05-25T12:08:12.274 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.250+0000 7f92e5d9b700 1 -- 172.21.15.102:0/937273942 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92e8063fb0 con 0x7f92e8063840 2021-05-25T12:08:12.274 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.250+0000 7f92e5d9b700 1 -- 172.21.15.102:0/937273942 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 474684112 0 0) 0x7f92cc003270 con 0x7f92e8094da0 2021-05-25T12:08:12.274 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.250+0000 7f92e5d9b700 1 -- 172.21.15.102:0/937273942 >> v1:172.21.15.181:6789/0 conn(0x7f92e8063840 legacy=0x7f92e80974d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:12.274 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.250+0000 7f92e5d9b700 1 -- 172.21.15.102:0/937273942 >> v1:172.21.15.102:6789/0 conn(0x7f92e81323c0 legacy=0x7f92e81347b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:12.275 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.250+0000 7f92e5d9b700 10 monclient: _finish_hunting 0 2021-05-25T12:08:12.275 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.250+0000 7f92e5d9b700 1 monclient: found mon.noname-b 2021-05-25T12:08:12.275 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.250+0000 7f92e5d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:08:12.275 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.250+0000 7f92e5d9b700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T12:08:12.275 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.250+0000 7f92e5d9b700 1 -- 172.21.15.102:0/937273942 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92e80647b0 con 0x7f92e8094da0 2021-05-25T12:08:12.276 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.250+0000 7f92e5d9b700 10 monclient: _finish_auth 0 2021-05-25T12:08:12.276 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.250+0000 7f92e5d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:08:12.276 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.250+0000 7f92e5d9b700 1 -- 172.21.15.102:0/937273942 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f92dc001610 con 0x7f92e8094da0 2021-05-25T12:08:12.276 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.250+0000 7f92ed089700 5 monclient: authenticate success, global_id 244106 2021-05-25T12:08:12.276 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.250+0000 7f92ed089700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:08:12.277 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.250+0000 7f92e5d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:12.277 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.250+0000 7f92e5d9b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:08:12.277 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.250+0000 7f92e5d9b700 10 monclient: dump: 2021-05-25T12:08:12.277 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:12.277 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:12.278 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:12.278 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:12.278 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:12.278 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:12.279 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:12.279 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:12.279 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:12.279 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:12.279 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.250+0000 7f92ed089700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:08:12.280 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.250+0000 7f92e5d9b700 1 -- 172.21.15.102:0/937273942 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f92dc001880 con 0x7f92e8094da0 2021-05-25T12:08:12.280 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.250+0000 7f92e5d9b700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:08:12.280 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.250+0000 7f92ed089700 10 monclient: get_monmap_and_config success 2021-05-25T12:08:12.280 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.250+0000 7f92e5d9b700 1 -- 172.21.15.102:0/937273942 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f92dc001c10 con 0x7f92e8094da0 2021-05-25T12:08:12.280 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.250+0000 7f92ed089700 4 set_mon_vals no callback set 2021-05-25T12:08:12.281 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.250+0000 7f92e5d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:12.281 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.250+0000 7f92e5d9b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:08:12.281 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.250+0000 7f92e5d9b700 10 monclient: dump: 2021-05-25T12:08:12.281 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:12.281 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:12.282 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:12.282 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:12.282 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:12.282 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:12.283 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:12.283 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:12.283 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:12.283 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:12.283 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.251+0000 7f92ed089700 10 monclient: shutdown 2021-05-25T12:08:12.284 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.251+0000 7f92ed089700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:08:12.284 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.251+0000 7f92ed089700 1 -- 172.21.15.102:0/937273942 >> v1:172.21.15.102:6790/0 conn(0x7f92e8094da0 legacy=0x7f92e8097be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:12.284 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.251+0000 7f92ed089700 1 -- 172.21.15.102:0/937273942 shutdown_connections 2021-05-25T12:08:12.284 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.251+0000 7f92ed089700 1 -- 172.21.15.102:0/937273942 >> 172.21.15.102:0/937273942 conn(0x7f92e809cb30 msgr2=0x7f92e809cf40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:08:12.284 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.251+0000 7f92ed089700 1 -- 172.21.15.102:0/937273942 shutdown_connections 2021-05-25T12:08:12.285 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.251+0000 7f92ed089700 1 -- 172.21.15.102:0/937273942 wait complete. 2021-05-25T12:08:12.285 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.252+0000 7f92ed089700 10 monclient: build_initial_monmap 2021-05-25T12:08:12.285 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.252+0000 7f92ed089700 10 monclient: monmap: 2021-05-25T12:08:12.285 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:08:12.285 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:08:12.286 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:08:12.286 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:08:12.286 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:08:12.286 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:08:12.286 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:08:12.287 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:08:12.287 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:08:12.287 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:12.287 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.252+0000 7f92ed089700 1 librados: starting msgr at 2021-05-25T12:08:12.288 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.252+0000 7f92ed089700 1 librados: starting objecter 2021-05-25T12:08:12.288 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.252+0000 7f92ed089700 1 Processor -- start 2021-05-25T12:08:12.288 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.252+0000 7f92ed089700 1 -- start start 2021-05-25T12:08:12.288 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.252+0000 7f92ed089700 1 librados: setting wanted keys 2021-05-25T12:08:12.288 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.252+0000 7f92ed089700 1 librados: calling monclient init 2021-05-25T12:08:12.289 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.252+0000 7f92ed089700 10 monclient: init 2021-05-25T12:08:12.289 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.252+0000 7f92ed089700 10 monclient: _reopen_session rank -1 2021-05-25T12:08:12.289 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.252+0000 7f92ed089700 10 monclient: _add_conns ranks=[2,0,1] 2021-05-25T12:08:12.289 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.252+0000 7f92ed089700 10 monclient(hunting): picked mon.noname-c con 0x7f92e8063840 addr v1:172.21.15.102:6790/0 2021-05-25T12:08:12.289 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.252+0000 7f92ed089700 10 monclient(hunting): picked mon.noname-a con 0x7f92e8094da0 addr v1:172.21.15.102:6789/0 2021-05-25T12:08:12.290 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.252+0000 7f92ed089700 10 monclient(hunting): picked mon.noname-b con 0x7f92e81323c0 addr v1:172.21.15.181:6789/0 2021-05-25T12:08:12.290 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.252+0000 7f92ed089700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92e80685b0 con 0x7f92e8094da0 2021-05-25T12:08:12.290 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.252+0000 7f92ed089700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92e8092510 con 0x7f92e81323c0 2021-05-25T12:08:12.290 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.252+0000 7f92ed089700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92e8091a80 con 0x7f92e8063840 2021-05-25T12:08:12.290 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.252+0000 7f92ed089700 10 monclient(hunting): _renew_subs 2021-05-25T12:08:12.291 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.252+0000 7f92ed089700 10 monclient(hunting): authenticate will time out at 1292.849945s 2021-05-25T12:08:12.291 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.253+0000 7f92e659c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f92e8094da0 0x7f92e80910a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:34094/0 (socket says 172.21.15.102:34094) 2021-05-25T12:08:12.291 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.253+0000 7f92e659c700 1 -- 172.21.15.102:0/229887476 learned_addr learned my addr 172.21.15.102:0/229887476 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:08:12.291 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.253+0000 7f92e6d9d700 1 --1- 172.21.15.102:0/229887476 >> v1:172.21.15.102:6790/0 conn(0x7f92e8063840 0x7f92e8090990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:57206/0 (socket says 172.21.15.102:57206) 2021-05-25T12:08:12.291 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.253+0000 7f92d2ffd700 10 client.?.objecter ms_handle_connect 0x7f92e8094da0 2021-05-25T12:08:12.292 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.253+0000 7f92d2ffd700 10 client.?.objecter resend_mon_ops 2021-05-25T12:08:12.292 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.253+0000 7f92d2ffd700 10 client.?.objecter ms_handle_connect 0x7f92e8063840 2021-05-25T12:08:12.292 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.253+0000 7f92d2ffd700 10 client.?.objecter resend_mon_ops 2021-05-25T12:08:12.292 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.253+0000 7f92d2ffd700 10 client.?.objecter ms_handle_connect 0x7f92e81323c0 2021-05-25T12:08:12.293 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.253+0000 7f92d2ffd700 10 client.?.objecter resend_mon_ops 2021-05-25T12:08:12.293 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.253+0000 7f92d2ffd700 1 -- 172.21.15.102:0/229887476 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2839644096 0 0) 0x7f92e80685b0 con 0x7f92e8094da0 2021-05-25T12:08:12.293 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.253+0000 7f92d2ffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:12.293 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.253+0000 7f92d2ffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:12.293 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.253+0000 7f92d2ffd700 10 monclient(hunting): my global_id is 204510 2021-05-25T12:08:12.294 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.253+0000 7f92d2ffd700 1 -- 172.21.15.102:0/229887476 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92c8004c10 con 0x7f92e8094da0 2021-05-25T12:08:12.294 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.253+0000 7f92d2ffd700 1 -- 172.21.15.102:0/229887476 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2567142334 0 0) 0x7f92e8091a80 con 0x7f92e8063840 2021-05-25T12:08:12.294 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.253+0000 7f92d2ffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:12.294 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.253+0000 7f92d2ffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:12.294 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.253+0000 7f92d2ffd700 10 monclient(hunting): my global_id is 244109 2021-05-25T12:08:12.295 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.253+0000 7f92d2ffd700 1 -- 172.21.15.102:0/229887476 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92e80685b0 con 0x7f92e8063840 2021-05-25T12:08:12.295 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.253+0000 7f92d2ffd700 1 -- 172.21.15.102:0/229887476 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1572119550 0 0) 0x7f92e8092510 con 0x7f92e81323c0 2021-05-25T12:08:12.295 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.253+0000 7f92d2ffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:12.295 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.253+0000 7f92d2ffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:12.295 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.253+0000 7f92d2ffd700 10 monclient(hunting): my global_id is 224365 2021-05-25T12:08:12.296 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.253+0000 7f92d2ffd700 1 -- 172.21.15.102:0/229887476 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92e8091a80 con 0x7f92e81323c0 2021-05-25T12:08:12.296 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.253+0000 7f92d2ffd700 1 -- 172.21.15.102:0/229887476 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 44072851 0 0) 0x7f92c8004c10 con 0x7f92e8094da0 2021-05-25T12:08:12.296 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.253+0000 7f92d2ffd700 1 -- 172.21.15.102:0/229887476 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f92e8092510 con 0x7f92e8094da0 2021-05-25T12:08:12.296 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.253+0000 7f92d2ffd700 1 -- 172.21.15.102:0/229887476 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f92dc001610 con 0x7f92e8094da0 2021-05-25T12:08:12.297 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.253+0000 7f92d2ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:12.297 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.253+0000 7f92d2ffd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:08:12.297 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.253+0000 7f92d2ffd700 10 monclient(hunting): dump: 2021-05-25T12:08:12.297 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:12.297 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:12.298 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:12.298 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:12.298 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:12.298 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:12.298 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:12.299 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:12.299 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:12.299 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:12.299 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.253+0000 7f92d2ffd700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:08:12.299 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.253+0000 7f92d2ffd700 1 -- 172.21.15.102:0/229887476 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2458534896 0 0) 0x7f92e80685b0 con 0x7f92e8063840 2021-05-25T12:08:12.300 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.253+0000 7f92d2ffd700 1 -- 172.21.15.102:0/229887476 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f92c8004c10 con 0x7f92e8063840 2021-05-25T12:08:12.300 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.253+0000 7f92d2ffd700 1 -- 172.21.15.102:0/229887476 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f92d4001880 con 0x7f92e8063840 2021-05-25T12:08:12.300 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.253+0000 7f92d2ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:12.300 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.253+0000 7f92d2ffd700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2021-05-25T12:08:12.300 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.253+0000 7f92d2ffd700 10 monclient(hunting): dump: 2021-05-25T12:08:12.301 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:12.301 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:12.301 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:12.301 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:12.302 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:12.302 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:12.302 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:12.302 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:12.302 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:12.303 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:12.303 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.253+0000 7f92d2ffd700 1 -- 172.21.15.102:0/229887476 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2707696506 0 0) 0x7f92e8091a80 con 0x7f92e81323c0 2021-05-25T12:08:12.303 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.253+0000 7f92d2ffd700 1 -- 172.21.15.102:0/229887476 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f92e80685b0 con 0x7f92e81323c0 2021-05-25T12:08:12.303 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.253+0000 7f92d2ffd700 1 -- 172.21.15.102:0/229887476 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f92d8001d50 con 0x7f92e81323c0 2021-05-25T12:08:12.303 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.253+0000 7f92d2ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:12.304 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.253+0000 7f92d2ffd700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2021-05-25T12:08:12.304 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.253+0000 7f92d2ffd700 10 monclient(hunting): dump: 2021-05-25T12:08:12.304 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:12.304 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:12.304 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:12.305 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:12.305 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:12.305 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:12.305 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:12.306 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:12.306 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:12.306 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:12.306 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.253+0000 7f92d2ffd700 1 -- 172.21.15.102:0/229887476 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 336016866 0 0) 0x7f92e8092510 con 0x7f92e8094da0 2021-05-25T12:08:12.306 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.253+0000 7f92d2ffd700 1 -- 172.21.15.102:0/229887476 >> v1:172.21.15.102:6790/0 conn(0x7f92e8063840 legacy=0x7f92e8090990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:12.307 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.253+0000 7f92d2ffd700 1 -- 172.21.15.102:0/229887476 >> v1:172.21.15.181:6789/0 conn(0x7f92e81323c0 legacy=0x7f92e81478c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:12.307 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.253+0000 7f92d2ffd700 10 monclient: _finish_hunting 0 2021-05-25T12:08:12.307 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.253+0000 7f92d2ffd700 1 monclient: found mon.a 2021-05-25T12:08:12.307 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.253+0000 7f92d2ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:08:12.307 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.253+0000 7f92d2ffd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:08:12.308 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.253+0000 7f92d2ffd700 1 -- 172.21.15.102:0/229887476 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92e80917b0 con 0x7f92e8094da0 2021-05-25T12:08:12.308 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.253+0000 7f92d2ffd700 10 monclient: _finish_auth 0 2021-05-25T12:08:12.308 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.253+0000 7f92d2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:08:12.308 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.253+0000 7f92ed089700 5 monclient: authenticate success, global_id 204510 2021-05-25T12:08:12.308 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.253+0000 7f92ed089700 10 monclient: _renew_subs 2021-05-25T12:08:12.309 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.253+0000 7f92ed089700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:08:12.309 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.253+0000 7f92ed089700 1 -- 172.21.15.102:0/229887476 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92e8093b60 con 0x7f92e8094da0 2021-05-25T12:08:12.309 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.253+0000 7f92ed089700 10 client.204510.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:08:12.309 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.253+0000 7f92ed089700 10 monclient: _renew_subs 2021-05-25T12:08:12.309 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.253+0000 7f92ed089700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:08:12.310 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.253+0000 7f92ed089700 1 -- 172.21.15.102:0/229887476 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f92e8093390 con 0x7f92e8094da0 2021-05-25T12:08:12.310 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.253+0000 7f92ed089700 1 librados: init done 2021-05-25T12:08:12.310 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.253+0000 7f92d2ffd700 1 -- 172.21.15.102:0/229887476 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f92dc0011b0 con 0x7f92e8094da0 2021-05-25T12:08:12.310 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.253+0000 7f92d2ffd700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:08:12.311 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.253+0000 7f92d2ffd700 1 -- 172.21.15.102:0/229887476 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f92dc0048f0 con 0x7f92e8094da0 2021-05-25T12:08:12.311 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.253+0000 7f92d2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:12.311 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.253+0000 7f92d2ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:08:12.311 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.253+0000 7f92e559a700 4 set_mon_vals no callback set 2021-05-25T12:08:12.311 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.253+0000 7f92d2ffd700 10 monclient: dump: 2021-05-25T12:08:12.312 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:12.312 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:12.312 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:12.312 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:12.312 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:12.313 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:12.313 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:12.313 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:12.313 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:12.313 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:12.314 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.254+0000 7f92ed089700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:08:12.314 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.254+0000 7f92d2ffd700 1 -- 172.21.15.102:0/229887476 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f92dc004ad0 con 0x7f92e8094da0 2021-05-25T12:08:12.314 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.254+0000 7f92ed089700 1 -- 172.21.15.102:0/229887476 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f92c00065a0 con 0x7f92c0002840 2021-05-25T12:08:12.314 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.254+0000 7f92d2ffd700 1 -- 172.21.15.102:0/229887476 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(268..268 src has 1..268) v4 ==== 6249+0+0 (unknown 118496375 0 0) 0x7f92dc03d900 con 0x7f92e8094da0 2021-05-25T12:08:12.314 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.254+0000 7f92d2ffd700 10 client.204510.objecter ms_dispatch 0x7f92e8001550 osd_map(268..268 src has 1..268) v4 2021-05-25T12:08:12.315 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.254+0000 7f92d2ffd700 3 client.204510.objecter handle_osd_map got epochs [268,268] > 0 2021-05-25T12:08:12.315 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.254+0000 7f92d2ffd700 3 client.204510.objecter handle_osd_map decoding full epoch 268 2021-05-25T12:08:12.315 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.254+0000 7f92d2ffd700 20 client.204510.objecter dump_active .. 0 homeless 2021-05-25T12:08:12.315 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.254+0000 7f92d2ffd700 10 client.204510.objecter ms_handle_connect 0x7f92c0002840 2021-05-25T12:08:12.315 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.254+0000 7f92d2ffd700 10 client.204510.objecter resend_mon_ops 2021-05-25T12:08:12.316 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.254+0000 7f92d2ffd700 1 -- 172.21.15.102:0/229887476 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1709790590 0 0) 0x7f92c00065a0 con 0x7f92c0002840 2021-05-25T12:08:12.316 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.254+0000 7f92d2ffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:12.316 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.254+0000 7f92d2ffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:12.316 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.254+0000 7f92d2ffd700 10 monclient(hunting): my global_id is 244112 2021-05-25T12:08:12.316 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.254+0000 7f92d2ffd700 1 -- 172.21.15.102:0/229887476 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92e8092510 con 0x7f92c0002840 2021-05-25T12:08:12.317 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.254+0000 7f92e759e700 10 monclient: get_auth_request con 0x7f92c8033500 auth_method 0 2021-05-25T12:08:12.317 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.254+0000 7f92d2ffd700 1 -- 172.21.15.102:0/229887476 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1984072945 0 0) 0x7f92e8092510 con 0x7f92c0002840 2021-05-25T12:08:12.317 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.254+0000 7f92d2ffd700 1 -- 172.21.15.102:0/229887476 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f92c00065a0 con 0x7f92c0002840 2021-05-25T12:08:12.317 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.254+0000 7f92d2ffd700 1 -- 172.21.15.102:0/229887476 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f92d40022c0 con 0x7f92c0002840 2021-05-25T12:08:12.318 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.254+0000 7f92d2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:12.318 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.254+0000 7f92d2ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:08:12.318 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.254+0000 7f92d2ffd700 10 monclient: dump: 2021-05-25T12:08:12.318 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:12.318 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:12.319 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:12.319 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:12.319 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:12.319 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:12.319 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:12.320 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:12.320 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:12.320 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:12.320 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.255+0000 7f92d2ffd700 10 client.204510.objecter ms_handle_connect 0x7f92c8033500 2021-05-25T12:08:12.320 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.255+0000 7f92d2ffd700 1 -- 172.21.15.102:0/229887476 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 948974861 0 0) 0x7f92c00065a0 con 0x7f92c0002840 2021-05-25T12:08:12.321 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.255+0000 7f92d2ffd700 1 -- 172.21.15.102:0/229887476 --> v1:172.21.15.102:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f92c0009e40 con 0x7f92c0002840 2021-05-25T12:08:12.321 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.255+0000 7f92d2ffd700 1 -- 172.21.15.102:0/229887476 <== mon.2 v1:172.21.15.102:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7f92d4002700 con 0x7f92c0002840 2021-05-25T12:08:12.321 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.255+0000 7f92d2ffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:08:12.321 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.255+0000 7f92d2ffd700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:08:12.321 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.255+0000 7f92d2ffd700 1 -- 172.21.15.102:0/229887476 >> v1:172.21.15.102:6790/0 conn(0x7f92c0002840 legacy=0x7f92c0004ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:12.322 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.255+0000 7f92d2ffd700 1 -- 172.21.15.102:0/229887476 >> v1:172.21.15.102:6790/0 conn(0x7f92c0002840 legacy=0x7f92c0004ce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:08:12.322 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.261+0000 7f92ed089700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:08:12.322 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.261+0000 7f92ed089700 1 -- 172.21.15.102:0/229887476 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f92c0007540 con 0x7f92c00068e0 2021-05-25T12:08:12.322 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.262+0000 7f92d2ffd700 10 client.204510.objecter ms_handle_connect 0x7f92c00068e0 2021-05-25T12:08:12.323 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.262+0000 7f92d2ffd700 10 client.204510.objecter resend_mon_ops 2021-05-25T12:08:12.323 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.262+0000 7f92d2ffd700 1 -- 172.21.15.102:0/229887476 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1003033839 0 0) 0x7f92c0007540 con 0x7f92c00068e0 2021-05-25T12:08:12.323 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.262+0000 7f92d2ffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:12.323 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.262+0000 7f92d2ffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:12.323 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.262+0000 7f92d2ffd700 10 monclient(hunting): my global_id is 244115 2021-05-25T12:08:12.324 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.262+0000 7f92d2ffd700 1 -- 172.21.15.102:0/229887476 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92c00065a0 con 0x7f92c00068e0 2021-05-25T12:08:12.324 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.262+0000 7f92d2ffd700 1 -- 172.21.15.102:0/229887476 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4236875493 0 0) 0x7f92c00065a0 con 0x7f92c00068e0 2021-05-25T12:08:12.324 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.262+0000 7f92d2ffd700 1 -- 172.21.15.102:0/229887476 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f92c0007540 con 0x7f92c00068e0 2021-05-25T12:08:12.324 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.262+0000 7f92d2ffd700 1 -- 172.21.15.102:0/229887476 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f92d4005510 con 0x7f92c00068e0 2021-05-25T12:08:12.324 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.262+0000 7f92d2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:12.325 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.262+0000 7f92d2ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:08:12.325 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.262+0000 7f92d2ffd700 10 monclient: dump: 2021-05-25T12:08:12.325 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:12.325 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:12.325 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:12.326 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:12.326 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:12.326 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:12.326 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:12.326 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:12.327 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:12.327 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:12.327 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.262+0000 7f92d2ffd700 1 -- 172.21.15.102:0/229887476 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 579437737 0 0) 0x7f92c0007540 con 0x7f92c00068e0 2021-05-25T12:08:12.327 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.262+0000 7f92d2ffd700 1 -- 172.21.15.102:0/229887476 --> v1:172.21.15.102:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f92c0008770 con 0x7f92c00068e0 2021-05-25T12:08:12.327 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.262+0000 7f92d2ffd700 1 -- 172.21.15.102:0/229887476 <== mon.2 v1:172.21.15.102:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3770 (unknown 0 0 77972888) 0x7f92d4006ed0 con 0x7f92c00068e0 2021-05-25T12:08:12.328 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.262+0000 7f92d2ffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:08:12.328 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.262+0000 7f92d2ffd700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:08:12.328 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.262+0000 7f92d2ffd700 1 -- 172.21.15.102:0/229887476 >> v1:172.21.15.102:6790/0 conn(0x7f92c00068e0 legacy=0x7f92c0006d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:12.328 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.262+0000 7f92d2ffd700 1 -- 172.21.15.102:0/229887476 >> v1:172.21.15.102:6790/0 conn(0x7f92c00068e0 legacy=0x7f92c0006d70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:08:12.329 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:08:12.329 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:08:12.329 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:08:12.329 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "peon", 2021-05-25T12:08:12.330 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 68, 2021-05-25T12:08:12.330 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:08:12.330 INFO:teuthology.orchestra.run.smithi102.stdout: 0, 2021-05-25T12:08:12.330 INFO:teuthology.orchestra.run.smithi102.stdout: 1, 2021-05-25T12:08:12.331 INFO:teuthology.orchestra.run.smithi102.stdout: 2 2021-05-25T12:08:12.331 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:08:12.331 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 0, 2021-05-25T12:08:12.331 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:08:12.331 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:08:12.332 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:08:12.332 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:08:12.332 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:08:12.332 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:08:12.332 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:08:12.333 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:08:12.333 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:08:12.333 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:08:12.333 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:08:12.333 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:08:12.334 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:08:12.334 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:08:12.334 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:08:12.334 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:08:12.334 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:08:12.335 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:08:12.335 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:08:12.335 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:08:12.335 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:08:12.335 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:08:12.336 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:08:12.336 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:08:12.336 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:12.336 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:12.336 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:08:12.337 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:08:12.337 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:08:12.337 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:08:12.337 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:08:12.338 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:08:12.338 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:08:12.338 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:08:12.338 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:08:12.338 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:08:12.339 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:08:12.339 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:08:12.339 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:08:12.339 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:08:12.339 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:08:12.340 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:08:12.340 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:08:12.340 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:08:12.340 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:08:12.340 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:08:12.341 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:08:12.341 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:08:12.341 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:08:12.341 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:08:12.341 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:08:12.342 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:08:12.342 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:12.342 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:08:12.342 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:12.342 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:08:12.343 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:08:12.343 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:08:12.343 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:08:12.343 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:12.344 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:08:12.344 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:08:12.344 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:08:12.344 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:12.344 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:12.345 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:12.345 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:08:12.345 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:08:12.345 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:08:12.345 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:08:12.346 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:08:12.346 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:12.346 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:12.346 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:08:12.346 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:08:12.347 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:08:12.347 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:08:12.347 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:12.347 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:08:12.347 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:08:12.348 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:08:12.348 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:12.348 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:12.348 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:12.348 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:08:12.349 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:08:12.349 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:08:12.349 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:08:12.349 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:08:12.349 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:12.350 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:12.350 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:08:12.350 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:08:12.350 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:08:12.351 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:08:12.351 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:12.351 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:08:12.351 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:08:12.352 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:08:12.352 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:12.352 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:12.352 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:12.352 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:08:12.353 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:08:12.353 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:08:12.353 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:08:12.353 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:08:12.354 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:12.354 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:12.354 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:12.355 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:08:12.355 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:08:12.355 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:12.355 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:08:12.356 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:08:12.356 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:08:12.356 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:12.357 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:08:12.357 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:08:12.357 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:12.357 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:08:12.358 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:08:12.358 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:08:12.358 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:12.359 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:12.359 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:12.359 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:08:12.359 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:08:12.362 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.262+0000 7f92ed089700 10 librados: watch_flush enter 2021-05-25T12:08:12.362 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.262+0000 7f92ed089700 10 librados: watch_flush exit 2021-05-25T12:08:12.362 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.262+0000 7f92ed089700 20 client.204510.objecter shutdown clearing up homeless session... 2021-05-25T12:08:12.363 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.262+0000 7f92ed089700 10 client.204510.objecter successfully canceled tick 2021-05-25T12:08:12.363 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.263+0000 7f92ed089700 1 -- 172.21.15.102:0/229887476 >> v1:172.21.15.181:6812/29582 conn(0x7f92c8033500 legacy=0x7f92c80359c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:12.363 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.263+0000 7f92ed089700 10 monclient: shutdown 2021-05-25T12:08:12.363 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.263+0000 7f92ed089700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:08:12.364 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.263+0000 7f92ed089700 1 -- 172.21.15.102:0/229887476 >> v1:172.21.15.102:6789/0 conn(0x7f92e8094da0 legacy=0x7f92e80910a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:12.364 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.263+0000 7f92e759e700 1 -- 172.21.15.102:0/229887476 reap_dead start 2021-05-25T12:08:12.364 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.263+0000 7f92e759e700 1 -- 172.21.15.102:0/229887476 reap_dead start 2021-05-25T12:08:12.364 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.263+0000 7f92ed089700 1 -- 172.21.15.102:0/229887476 shutdown_connections 2021-05-25T12:08:12.364 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.263+0000 7f92ed089700 1 -- 172.21.15.102:0/229887476 >> 172.21.15.102:0/229887476 conn(0x7f92e809cb30 msgr2=0x7f92e8095300 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:08:12.365 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.263+0000 7f92ed089700 1 -- 172.21.15.102:0/229887476 shutdown_connections 2021-05-25T12:08:12.365 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.263+0000 7f92ed089700 1 -- 172.21.15.102:0/229887476 wait complete. 2021-05-25T12:08:12.365 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.263+0000 7f92ed089700 1 librados: shutdown 2021-05-25T12:08:12.366 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2021-05-25T12:08:12.477 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.475+0000 7fb0f92e0700 10 monclient: get_monmap_and_config 2021-05-25T12:08:12.477 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.475+0000 7fb0f92e0700 10 monclient: build_initial_monmap 2021-05-25T12:08:12.477 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.475+0000 7fb0f92e0700 10 monclient: monmap: 2021-05-25T12:08:12.478 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:08:12.478 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:12.478 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:08:12.476841+0000 2021-05-25T12:08:12.478 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:08:12.476841+0000 2021-05-25T12:08:12.479 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:08:12.479 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:12.479 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:08:12.480 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:08:12.480 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:08:12.480 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:12.494 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.476+0000 7fb0f92e0700 1 Processor -- start 2021-05-25T12:08:12.495 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.476+0000 7fb0f92e0700 1 -- start start 2021-05-25T12:08:12.495 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.476+0000 7fb0f92e0700 10 monclient: init 2021-05-25T12:08:12.495 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.476+0000 7fb0f92e0700 10 monclient: _reopen_session rank -1 2021-05-25T12:08:12.495 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.476+0000 7fb0f92e0700 10 monclient: _add_conns ranks=[2,1,0] 2021-05-25T12:08:12.496 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.476+0000 7fb0f92e0700 10 monclient(hunting): picked mon.noname-b con 0x7fb0f4090990 addr v1:172.21.15.102:6790/0 2021-05-25T12:08:12.496 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.476+0000 7fb0f92e0700 10 monclient(hunting): picked mon.noname-c con 0x7fb0f4090d70 addr v1:172.21.15.181:6789/0 2021-05-25T12:08:12.496 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.477+0000 7fb0f92e0700 10 monclient(hunting): picked mon.noname-a con 0x7fb0f412e330 addr v1:172.21.15.102:6789/0 2021-05-25T12:08:12.496 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.477+0000 7fb0f92e0700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0f4096fa0 con 0x7fb0f412e330 2021-05-25T12:08:12.496 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.477+0000 7fb0f92e0700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0f40926a0 con 0x7fb0f4090d70 2021-05-25T12:08:12.497 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.477+0000 7fb0f92e0700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0f4092940 con 0x7fb0f4090990 2021-05-25T12:08:12.497 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.477+0000 7fb0f92e0700 10 monclient(hunting): _renew_subs 2021-05-25T12:08:12.497 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.477+0000 7fb0f92e0700 10 monclient(hunting): authenticate will time out at 1293.074057s 2021-05-25T12:08:12.683 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.477+0000 7fb0f37fe700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fb0f412e330 0x7fb0f4130720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:34110/0 (socket says 172.21.15.102:34110) 2021-05-25T12:08:12.683 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.477+0000 7fb0f2ffd700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fb0f4090990 0x7fb0f4093240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:57216/0 (socket says 172.21.15.102:57216) 2021-05-25T12:08:12.683 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.477+0000 7fb0f37fe700 1 -- 172.21.15.102:0/3570715774 learned_addr learned my addr 172.21.15.102:0/3570715774 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:08:12.684 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.477+0000 7fb0f1ffb700 1 -- 172.21.15.102:0/3570715774 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1153584421 0 0) 0x7fb0f4096fa0 con 0x7fb0f412e330 2021-05-25T12:08:12.684 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.477+0000 7fb0f1ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:12.684 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.477+0000 7fb0f1ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:12.684 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.477+0000 7fb0f1ffb700 10 monclient(hunting): my global_id is 204513 2021-05-25T12:08:12.685 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.477+0000 7fb0f1ffb700 1 -- 172.21.15.102:0/3570715774 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0d8004380 con 0x7fb0f412e330 2021-05-25T12:08:12.685 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.477+0000 7fb0f1ffb700 1 -- 172.21.15.102:0/3570715774 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 820983680 0 0) 0x7fb0f4092940 con 0x7fb0f4090990 2021-05-25T12:08:12.685 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.477+0000 7fb0f1ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:12.685 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.477+0000 7fb0f1ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:12.685 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.477+0000 7fb0f1ffb700 10 monclient(hunting): my global_id is 244118 2021-05-25T12:08:12.686 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.477+0000 7fb0f1ffb700 1 -- 172.21.15.102:0/3570715774 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0f4096fa0 con 0x7fb0f4090990 2021-05-25T12:08:12.686 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.477+0000 7fb0f1ffb700 1 -- 172.21.15.102:0/3570715774 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3833441607 0 0) 0x7fb0d8004380 con 0x7fb0f412e330 2021-05-25T12:08:12.686 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.477+0000 7fb0f1ffb700 1 -- 172.21.15.102:0/3570715774 >> v1:172.21.15.102:6790/0 conn(0x7fb0f4090990 legacy=0x7fb0f4093240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:12.686 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.477+0000 7fb0f1ffb700 1 -- 172.21.15.102:0/3570715774 >> v1:172.21.15.181:6789/0 conn(0x7fb0f4090d70 legacy=0x7fb0f4091200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:12.687 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.477+0000 7fb0f1ffb700 10 monclient: _finish_hunting 0 2021-05-25T12:08:12.687 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.477+0000 7fb0f1ffb700 1 monclient: found mon.noname-a 2021-05-25T12:08:12.687 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.477+0000 7fb0f1ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:08:12.687 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.477+0000 7fb0f1ffb700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T12:08:12.687 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.477+0000 7fb0f1ffb700 1 -- 172.21.15.102:0/3570715774 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0f4092be0 con 0x7fb0f412e330 2021-05-25T12:08:12.688 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.477+0000 7fb0f1ffb700 10 monclient: _finish_auth 0 2021-05-25T12:08:12.688 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.477+0000 7fb0f1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:08:12.688 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.477+0000 7fb0f92e0700 5 monclient: authenticate success, global_id 204513 2021-05-25T12:08:12.688 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.477+0000 7fb0f1ffb700 1 -- 172.21.15.102:0/3570715774 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fb0e4001990 con 0x7fb0f412e330 2021-05-25T12:08:12.688 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.477+0000 7fb0f92e0700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:08:12.689 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.477+0000 7fb0f1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:12.689 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.477+0000 7fb0f1ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:08:12.689 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.477+0000 7fb0f1ffb700 10 monclient: dump: 2021-05-25T12:08:12.690 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:12.690 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:12.690 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:12.690 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:12.690 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:12.691 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:12.691 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:12.691 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:12.691 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:12.692 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:12.692 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.477+0000 7fb0f92e0700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:08:12.692 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.477+0000 7fb0f1ffb700 1 -- 172.21.15.102:0/3570715774 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb0e4001c00 con 0x7fb0f412e330 2021-05-25T12:08:12.692 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.477+0000 7fb0f1ffb700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:08:12.692 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.477+0000 7fb0f92e0700 10 monclient: get_monmap_and_config success 2021-05-25T12:08:12.693 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.477+0000 7fb0f1ffb700 1 -- 172.21.15.102:0/3570715774 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fb0e4001f90 con 0x7fb0f412e330 2021-05-25T12:08:12.693 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.477+0000 7fb0f92e0700 4 set_mon_vals no callback set 2021-05-25T12:08:12.693 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.477+0000 7fb0f1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:12.693 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.477+0000 7fb0f1ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:08:12.694 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.477+0000 7fb0f1ffb700 10 monclient: dump: 2021-05-25T12:08:12.694 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:12.694 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:12.694 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:12.695 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:12.695 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:12.695 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:12.695 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:12.696 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:12.696 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:12.696 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:12.696 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.478+0000 7fb0f92e0700 10 monclient: shutdown 2021-05-25T12:08:12.696 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.478+0000 7fb0f92e0700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:08:12.697 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.478+0000 7fb0f92e0700 1 -- 172.21.15.102:0/3570715774 >> v1:172.21.15.102:6789/0 conn(0x7fb0f412e330 legacy=0x7fb0f4130720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:12.697 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.478+0000 7fb0f92e0700 1 -- 172.21.15.102:0/3570715774 shutdown_connections 2021-05-25T12:08:12.697 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.478+0000 7fb0f92e0700 1 -- 172.21.15.102:0/3570715774 >> 172.21.15.102:0/3570715774 conn(0x7fb0f409b370 msgr2=0x7fb0f4097430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:08:12.697 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.478+0000 7fb0f92e0700 1 -- 172.21.15.102:0/3570715774 shutdown_connections 2021-05-25T12:08:12.698 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.478+0000 7fb0f92e0700 1 -- 172.21.15.102:0/3570715774 wait complete. 2021-05-25T12:08:12.698 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.478+0000 7fb0f92e0700 10 monclient: build_initial_monmap 2021-05-25T12:08:12.698 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.478+0000 7fb0f92e0700 10 monclient: monmap: 2021-05-25T12:08:12.698 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:08:12.698 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:08:12.699 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:08:12.699 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:08:12.699 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:08:12.699 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:08:12.700 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:08:12.700 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:08:12.700 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:08:12.700 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:12.700 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.479+0000 7fb0f92e0700 1 librados: starting msgr at 2021-05-25T12:08:12.701 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.479+0000 7fb0f92e0700 1 librados: starting objecter 2021-05-25T12:08:12.701 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.479+0000 7fb0f92e0700 1 Processor -- start 2021-05-25T12:08:12.701 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.479+0000 7fb0f92e0700 1 -- start start 2021-05-25T12:08:12.701 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.479+0000 7fb0f92e0700 1 librados: setting wanted keys 2021-05-25T12:08:12.701 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.479+0000 7fb0f92e0700 1 librados: calling monclient init 2021-05-25T12:08:12.702 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.479+0000 7fb0f92e0700 10 monclient: init 2021-05-25T12:08:12.702 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.479+0000 7fb0f92e0700 10 monclient: _reopen_session rank -1 2021-05-25T12:08:12.702 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.479+0000 7fb0f92e0700 10 monclient: _add_conns ranks=[1,0,2] 2021-05-25T12:08:12.702 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.479+0000 7fb0f92e0700 10 monclient(hunting): picked mon.noname-b con 0x7fb0f4090990 addr v1:172.21.15.181:6789/0 2021-05-25T12:08:12.703 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.479+0000 7fb0f92e0700 10 monclient(hunting): picked mon.noname-a con 0x7fb0f4090d70 addr v1:172.21.15.102:6789/0 2021-05-25T12:08:12.703 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.479+0000 7fb0f92e0700 10 monclient(hunting): picked mon.noname-c con 0x7fb0f412e330 addr v1:172.21.15.102:6790/0 2021-05-25T12:08:12.703 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.479+0000 7fb0f92e0700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0f4091fe0 con 0x7fb0f4090d70 2021-05-25T12:08:12.703 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.479+0000 7fb0f92e0700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0f4114790 con 0x7fb0f4090990 2021-05-25T12:08:12.703 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.479+0000 7fb0f92e0700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0f4114a30 con 0x7fb0f412e330 2021-05-25T12:08:12.704 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.479+0000 7fb0f92e0700 10 monclient(hunting): _renew_subs 2021-05-25T12:08:12.704 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.479+0000 7fb0f92e0700 10 monclient(hunting): authenticate will time out at 1293.076686s 2021-05-25T12:08:12.704 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.479+0000 7fb0f27fc700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fb0f4090d70 0x7fb0f413fd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:34114/0 (socket says 172.21.15.102:34114) 2021-05-25T12:08:12.704 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.479+0000 7fb0f37fe700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fb0f412e330 0x7fb0f4143460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:57226/0 (socket says 172.21.15.102:57226) 2021-05-25T12:08:12.705 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.479+0000 7fb0f27fc700 1 -- 172.21.15.102:0/2939790883 learned_addr learned my addr 172.21.15.102:0/2939790883 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:08:12.705 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.479+0000 7fb0deffd700 10 client.?.objecter ms_handle_connect 0x7fb0f412e330 2021-05-25T12:08:12.705 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.479+0000 7fb0deffd700 10 client.?.objecter resend_mon_ops 2021-05-25T12:08:12.705 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.479+0000 7fb0deffd700 10 client.?.objecter ms_handle_connect 0x7fb0f4090d70 2021-05-25T12:08:12.706 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.479+0000 7fb0deffd700 10 client.?.objecter resend_mon_ops 2021-05-25T12:08:12.706 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.479+0000 7fb0deffd700 1 -- 172.21.15.102:0/2939790883 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3529031587 0 0) 0x7fb0f4114a30 con 0x7fb0f412e330 2021-05-25T12:08:12.706 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.479+0000 7fb0deffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:12.706 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.479+0000 7fb0deffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:12.706 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.479+0000 7fb0deffd700 10 monclient(hunting): my global_id is 244121 2021-05-25T12:08:12.707 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.480+0000 7fb0deffd700 1 -- 172.21.15.102:0/2939790883 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0d4004c10 con 0x7fb0f412e330 2021-05-25T12:08:12.707 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.480+0000 7fb0deffd700 10 client.?.objecter ms_handle_connect 0x7fb0f4090990 2021-05-25T12:08:12.707 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.480+0000 7fb0deffd700 10 client.?.objecter resend_mon_ops 2021-05-25T12:08:12.707 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.480+0000 7fb0deffd700 1 -- 172.21.15.102:0/2939790883 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1543040674 0 0) 0x7fb0f4091fe0 con 0x7fb0f4090d70 2021-05-25T12:08:12.707 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.480+0000 7fb0deffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:12.708 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.480+0000 7fb0deffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:12.708 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.480+0000 7fb0deffd700 10 monclient(hunting): my global_id is 204516 2021-05-25T12:08:12.708 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.480+0000 7fb0deffd700 1 -- 172.21.15.102:0/2939790883 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0f4114a30 con 0x7fb0f4090d70 2021-05-25T12:08:12.708 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.480+0000 7fb0deffd700 1 -- 172.21.15.102:0/2939790883 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1864159989 0 0) 0x7fb0f4114790 con 0x7fb0f4090990 2021-05-25T12:08:12.708 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.480+0000 7fb0deffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:12.709 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.480+0000 7fb0deffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:12.709 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.480+0000 7fb0deffd700 10 monclient(hunting): my global_id is 224371 2021-05-25T12:08:12.709 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.480+0000 7fb0deffd700 1 -- 172.21.15.102:0/2939790883 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0f4091fe0 con 0x7fb0f4090990 2021-05-25T12:08:12.709 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.480+0000 7fb0deffd700 1 -- 172.21.15.102:0/2939790883 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1582591867 0 0) 0x7fb0d4004c10 con 0x7fb0f412e330 2021-05-25T12:08:12.710 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.480+0000 7fb0deffd700 1 -- 172.21.15.102:0/2939790883 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb0f4114790 con 0x7fb0f412e330 2021-05-25T12:08:12.711 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.480+0000 7fb0deffd700 1 -- 172.21.15.102:0/2939790883 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fb0e4001950 con 0x7fb0f412e330 2021-05-25T12:08:12.711 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.480+0000 7fb0deffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:12.711 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.480+0000 7fb0deffd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:08:12.711 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.480+0000 7fb0deffd700 10 monclient(hunting): dump: 2021-05-25T12:08:12.712 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:12.712 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:12.712 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:12.712 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:12.713 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:12.713 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:12.713 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:12.713 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:12.714 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:12.714 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:12.714 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.480+0000 7fb0deffd700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:08:12.714 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.480+0000 7fb0deffd700 1 -- 172.21.15.102:0/2939790883 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4174887540 0 0) 0x7fb0f4114a30 con 0x7fb0f4090d70 2021-05-25T12:08:12.715 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.480+0000 7fb0deffd700 1 -- 172.21.15.102:0/2939790883 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb0d4004c10 con 0x7fb0f4090d70 2021-05-25T12:08:12.715 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.480+0000 7fb0deffd700 1 -- 172.21.15.102:0/2939790883 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fb0e8001950 con 0x7fb0f4090d70 2021-05-25T12:08:12.715 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.480+0000 7fb0deffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:12.715 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.480+0000 7fb0deffd700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2021-05-25T12:08:12.716 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.480+0000 7fb0deffd700 10 monclient(hunting): dump: 2021-05-25T12:08:12.716 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:12.716 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:12.716 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:12.716 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:12.717 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:12.717 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:12.717 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:12.717 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:12.718 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:12.718 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:12.718 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.480+0000 7fb0deffd700 1 -- 172.21.15.102:0/2939790883 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2568550124 0 0) 0x7fb0f4091fe0 con 0x7fb0f4090990 2021-05-25T12:08:12.718 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.480+0000 7fb0deffd700 1 -- 172.21.15.102:0/2939790883 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb0f4114a30 con 0x7fb0f4090990 2021-05-25T12:08:12.719 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.480+0000 7fb0deffd700 1 -- 172.21.15.102:0/2939790883 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fb0e00019d0 con 0x7fb0f4090990 2021-05-25T12:08:12.719 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.480+0000 7fb0deffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:12.719 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.480+0000 7fb0deffd700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2021-05-25T12:08:12.719 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.480+0000 7fb0deffd700 10 monclient(hunting): dump: 2021-05-25T12:08:12.719 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:12.720 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:12.720 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:12.720 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:12.720 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:12.720 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:12.721 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:12.721 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:12.721 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:12.721 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:12.722 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.480+0000 7fb0deffd700 1 -- 172.21.15.102:0/2939790883 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 344397019 0 0) 0x7fb0f4114790 con 0x7fb0f412e330 2021-05-25T12:08:12.722 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.480+0000 7fb0deffd700 1 -- 172.21.15.102:0/2939790883 >> v1:172.21.15.181:6789/0 conn(0x7fb0f4090990 legacy=0x7fb0f4126280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:12.722 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.480+0000 7fb0deffd700 1 -- 172.21.15.102:0/2939790883 >> v1:172.21.15.102:6789/0 conn(0x7fb0f4090d70 legacy=0x7fb0f413fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:12.722 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.480+0000 7fb0deffd700 10 monclient: _finish_hunting 0 2021-05-25T12:08:12.722 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.480+0000 7fb0deffd700 1 monclient: found mon.c 2021-05-25T12:08:12.723 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.480+0000 7fb0deffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:08:12.723 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.480+0000 7fb0deffd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:08:12.723 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.480+0000 7fb0deffd700 1 -- 172.21.15.102:0/2939790883 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0f4112b60 con 0x7fb0f412e330 2021-05-25T12:08:12.723 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.480+0000 7fb0deffd700 10 monclient: _finish_auth 0 2021-05-25T12:08:12.724 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.480+0000 7fb0deffd700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:08:12.724 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.480+0000 7fb0f92e0700 5 monclient: authenticate success, global_id 244121 2021-05-25T12:08:12.724 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.480+0000 7fb0f92e0700 10 monclient: _renew_subs 2021-05-25T12:08:12.724 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.480+0000 7fb0f92e0700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:08:12.724 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.480+0000 7fb0f92e0700 1 -- 172.21.15.102:0/2939790883 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0f4126990 con 0x7fb0f412e330 2021-05-25T12:08:12.725 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.480+0000 7fb0f92e0700 10 client.244121.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:08:12.725 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.480+0000 7fb0f92e0700 10 monclient: _renew_subs 2021-05-25T12:08:12.725 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.480+0000 7fb0f92e0700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:08:12.725 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.480+0000 7fb0f92e0700 1 -- 172.21.15.102:0/2939790883 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0f4112000 con 0x7fb0f412e330 2021-05-25T12:08:12.726 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.480+0000 7fb0f92e0700 1 librados: init done 2021-05-25T12:08:12.726 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.480+0000 7fb0deffd700 1 -- 172.21.15.102:0/2939790883 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb0e4004b60 con 0x7fb0f412e330 2021-05-25T12:08:12.726 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.480+0000 7fb0deffd700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:08:12.727 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.480+0000 7fb0deffd700 1 -- 172.21.15.102:0/2939790883 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fb0e4004ef0 con 0x7fb0f412e330 2021-05-25T12:08:12.727 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.480+0000 7fb0deffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:12.727 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.480+0000 7fb0deffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:08:12.727 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.480+0000 7fb0f17fa700 4 set_mon_vals no callback set 2021-05-25T12:08:12.728 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.480+0000 7fb0deffd700 10 monclient: dump: 2021-05-25T12:08:12.728 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:12.728 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:12.728 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:12.728 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:12.729 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:12.729 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:12.729 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:12.729 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:12.730 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:12.730 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:12.730 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.481+0000 7fb0deffd700 1 -- 172.21.15.102:0/2939790883 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7fb0e40050d0 con 0x7fb0f412e330 2021-05-25T12:08:12.730 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.481+0000 7fb0f92e0700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:08:12.731 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.481+0000 7fb0f92e0700 1 -- 172.21.15.102:0/2939790883 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fb0cc0065a0 con 0x7fb0cc002840 2021-05-25T12:08:12.731 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.481+0000 7fb0deffd700 1 -- 172.21.15.102:0/2939790883 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(268..268 src has 1..268) v4 ==== 6249+0+0 (unknown 118496375 0 0) 0x7fb0e403dd70 con 0x7fb0f412e330 2021-05-25T12:08:12.731 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.481+0000 7fb0deffd700 10 client.244121.objecter ms_dispatch 0x7fb0f4001550 osd_map(268..268 src has 1..268) v4 2021-05-25T12:08:12.731 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.481+0000 7fb0deffd700 3 client.244121.objecter handle_osd_map got epochs [268,268] > 0 2021-05-25T12:08:12.732 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.481+0000 7fb0deffd700 3 client.244121.objecter handle_osd_map decoding full epoch 268 2021-05-25T12:08:12.732 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.481+0000 7fb0deffd700 20 client.244121.objecter dump_active .. 0 homeless 2021-05-25T12:08:12.732 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.481+0000 7fb0f27fc700 10 monclient: get_auth_request con 0x7fb0d4033500 auth_method 0 2021-05-25T12:08:12.732 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.481+0000 7fb0deffd700 10 client.244121.objecter ms_handle_connect 0x7fb0cc002840 2021-05-25T12:08:12.732 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.481+0000 7fb0deffd700 10 client.244121.objecter resend_mon_ops 2021-05-25T12:08:12.733 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.481+0000 7fb0deffd700 1 -- 172.21.15.102:0/2939790883 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2915523446 0 0) 0x7fb0cc0065a0 con 0x7fb0cc002840 2021-05-25T12:08:12.733 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.481+0000 7fb0deffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:12.733 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.481+0000 7fb0deffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:12.733 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.481+0000 7fb0deffd700 10 monclient(hunting): my global_id is 224374 2021-05-25T12:08:12.734 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.481+0000 7fb0deffd700 1 -- 172.21.15.102:0/2939790883 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0f4114790 con 0x7fb0cc002840 2021-05-25T12:08:12.734 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.481+0000 7fb0deffd700 10 client.244121.objecter ms_handle_connect 0x7fb0d4033500 2021-05-25T12:08:12.734 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.481+0000 7fb0deffd700 1 -- 172.21.15.102:0/2939790883 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4069701982 0 0) 0x7fb0f4114790 con 0x7fb0cc002840 2021-05-25T12:08:12.734 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.481+0000 7fb0deffd700 1 -- 172.21.15.102:0/2939790883 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb0cc0065a0 con 0x7fb0cc002840 2021-05-25T12:08:12.735 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.481+0000 7fb0deffd700 1 -- 172.21.15.102:0/2939790883 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fb0e00046d0 con 0x7fb0cc002840 2021-05-25T12:08:12.735 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.482+0000 7fb0deffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:12.735 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.482+0000 7fb0deffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T12:08:12.735 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.482+0000 7fb0deffd700 10 monclient: dump: 2021-05-25T12:08:12.736 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:12.736 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:12.736 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:12.736 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:12.736 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:12.737 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:12.737 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:12.737 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:12.737 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:12.738 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:12.738 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.482+0000 7fb0deffd700 1 -- 172.21.15.102:0/2939790883 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4077948384 0 0) 0x7fb0cc0065a0 con 0x7fb0cc002840 2021-05-25T12:08:12.738 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.482+0000 7fb0deffd700 1 -- 172.21.15.102:0/2939790883 --> v1:172.21.15.181:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb0cc009e40 con 0x7fb0cc002840 2021-05-25T12:08:12.738 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.482+0000 7fb0deffd700 1 -- 172.21.15.102:0/2939790883 <== mon.1 v1:172.21.15.181:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7fb0e0006890 con 0x7fb0cc002840 2021-05-25T12:08:12.739 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.482+0000 7fb0deffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:08:12.739 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.482+0000 7fb0deffd700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:08:12.739 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.482+0000 7fb0deffd700 1 -- 172.21.15.102:0/2939790883 >> v1:172.21.15.181:6789/0 conn(0x7fb0cc002840 legacy=0x7fb0cc004ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:12.739 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.482+0000 7fb0deffd700 1 -- 172.21.15.102:0/2939790883 >> v1:172.21.15.181:6789/0 conn(0x7fb0cc002840 legacy=0x7fb0cc004ce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:08:12.740 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.489+0000 7fb0f92e0700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:08:12.740 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.489+0000 7fb0f92e0700 1 -- 172.21.15.102:0/2939790883 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fb0cc009b00 con 0x7fb0cc008dc0 2021-05-25T12:08:12.740 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.489+0000 7fb0deffd700 10 client.244121.objecter ms_handle_connect 0x7fb0cc008dc0 2021-05-25T12:08:12.740 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.489+0000 7fb0deffd700 10 client.244121.objecter resend_mon_ops 2021-05-25T12:08:12.741 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.489+0000 7fb0deffd700 1 -- 172.21.15.102:0/2939790883 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 73561107 0 0) 0x7fb0cc009b00 con 0x7fb0cc008dc0 2021-05-25T12:08:12.741 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.489+0000 7fb0deffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:12.741 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.489+0000 7fb0deffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:12.741 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.489+0000 7fb0deffd700 10 monclient(hunting): my global_id is 224377 2021-05-25T12:08:12.742 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.489+0000 7fb0deffd700 1 -- 172.21.15.102:0/2939790883 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0cc0065a0 con 0x7fb0cc008dc0 2021-05-25T12:08:12.742 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.489+0000 7fb0deffd700 1 -- 172.21.15.102:0/2939790883 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1477274016 0 0) 0x7fb0cc0065a0 con 0x7fb0cc008dc0 2021-05-25T12:08:12.742 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.489+0000 7fb0deffd700 1 -- 172.21.15.102:0/2939790883 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb0cc009b00 con 0x7fb0cc008dc0 2021-05-25T12:08:12.742 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.489+0000 7fb0deffd700 1 -- 172.21.15.102:0/2939790883 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fb0e0004dc0 con 0x7fb0cc008dc0 2021-05-25T12:08:12.742 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.489+0000 7fb0deffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:12.743 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.489+0000 7fb0deffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T12:08:12.743 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.489+0000 7fb0deffd700 10 monclient: dump: 2021-05-25T12:08:12.743 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:12.743 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:12.743 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:12.744 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:12.744 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:12.744 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:12.744 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:12.745 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:12.745 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:12.745 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:12.745 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.490+0000 7fb0deffd700 1 -- 172.21.15.102:0/2939790883 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3654180317 0 0) 0x7fb0cc009b00 con 0x7fb0cc008dc0 2021-05-25T12:08:12.745 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.490+0000 7fb0deffd700 1 -- 172.21.15.102:0/2939790883 --> v1:172.21.15.181:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fb0cc008740 con 0x7fb0cc008dc0 2021-05-25T12:08:12.746 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.490+0000 7fb0deffd700 1 -- 172.21.15.102:0/2939790883 <== mon.1 v1:172.21.15.181:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3942 (unknown 0 0 3187463066) 0x7fb0e0006b20 con 0x7fb0cc008dc0 2021-05-25T12:08:12.746 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.490+0000 7fb0deffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:08:12.746 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.490+0000 7fb0deffd700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:08:12.746 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.490+0000 7fb0deffd700 1 -- 172.21.15.102:0/2939790883 >> v1:172.21.15.181:6789/0 conn(0x7fb0cc008dc0 legacy=0x7fb0cc009250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:12.747 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.490+0000 7fb0deffd700 1 -- 172.21.15.102:0/2939790883 >> v1:172.21.15.181:6789/0 conn(0x7fb0cc008dc0 legacy=0x7fb0cc009250 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:08:12.747 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:08:12.748 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:08:12.748 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:08:12.748 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "leader", 2021-05-25T12:08:12.748 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 68, 2021-05-25T12:08:12.749 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:08:12.749 INFO:teuthology.orchestra.run.smithi102.stdout: 0, 2021-05-25T12:08:12.749 INFO:teuthology.orchestra.run.smithi102.stdout: 1, 2021-05-25T12:08:12.749 INFO:teuthology.orchestra.run.smithi102.stdout: 2 2021-05-25T12:08:12.750 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:08:12.750 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 0, 2021-05-25T12:08:12.750 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:08:12.750 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:08:12.751 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:08:12.751 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:08:12.751 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:08:12.751 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:08:12.751 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:08:12.752 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:08:12.752 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:08:12.752 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:08:12.752 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:08:12.753 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:08:12.753 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:08:12.753 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:08:12.753 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:08:12.754 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:08:12.754 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:08:12.754 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:08:12.754 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:08:12.755 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:08:12.755 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:08:12.755 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:08:12.755 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:08:12.756 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:08:12.756 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:12.756 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:12.756 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:08:12.756 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:08:12.757 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:08:12.757 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:08:12.757 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:08:12.757 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:08:12.757 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:08:12.758 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:08:12.758 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:08:12.758 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:08:12.758 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:08:12.759 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:08:12.759 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:08:12.759 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:08:12.759 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:08:12.759 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:08:12.760 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:08:12.760 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:08:12.760 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:08:12.760 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:08:12.761 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:08:12.761 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:08:12.761 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:08:12.761 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:08:12.761 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:08:12.762 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:08:12.762 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:12.762 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:08:12.762 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:12.763 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:08:12.763 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:08:12.763 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:08:12.763 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:08:12.763 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:12.764 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:08:12.764 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:08:12.764 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:08:12.764 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:12.765 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:12.765 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:12.765 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:08:12.765 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:08:12.765 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:08:12.766 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:08:12.766 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:08:12.766 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:12.766 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:12.767 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:08:12.767 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:08:12.767 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:08:12.767 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:08:12.767 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:12.768 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:08:12.768 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:08:12.768 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:08:12.768 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:12.768 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:12.769 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:12.769 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:08:12.769 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:08:12.769 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:08:12.770 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:08:12.770 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:08:12.770 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:12.770 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:12.771 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:08:12.771 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:08:12.771 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:08:12.771 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:08:12.772 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:12.772 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:08:12.772 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:08:12.772 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:08:12.772 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:12.773 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:12.773 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:12.773 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:08:12.773 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:08:12.774 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:08:12.774 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:08:12.774 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:08:12.774 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:12.774 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:12.775 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:12.775 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:08:12.775 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:08:12.775 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:12.776 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:08:12.776 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:08:12.776 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:08:12.776 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:12.776 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:08:12.777 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:08:12.777 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:12.777 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:08:12.777 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:08:12.778 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 4 2021-05-25T12:08:12.778 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:12.778 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:08:12.778 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": [ 2021-05-25T12:08:12.778 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:12.779 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:08:12.779 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:08:12.779 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:08:12.779 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:12.780 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:12.780 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:12.780 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:08:12.780 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:08:12.783 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.490+0000 7fb0f92e0700 10 librados: watch_flush enter 2021-05-25T12:08:12.783 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.490+0000 7fb0f92e0700 10 librados: watch_flush exit 2021-05-25T12:08:12.783 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.490+0000 7fb0f92e0700 20 client.244121.objecter shutdown clearing up homeless session... 2021-05-25T12:08:12.783 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.490+0000 7fb0f92e0700 10 client.244121.objecter successfully canceled tick 2021-05-25T12:08:12.784 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.490+0000 7fb0f92e0700 1 -- 172.21.15.102:0/2939790883 >> v1:172.21.15.181:6812/29582 conn(0x7fb0d4033500 legacy=0x7fb0d40359c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:12.784 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.490+0000 7fb0f92e0700 10 monclient: shutdown 2021-05-25T12:08:12.784 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.490+0000 7fb0f92e0700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:08:12.784 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.490+0000 7fb0f92e0700 1 -- 172.21.15.102:0/2939790883 >> v1:172.21.15.102:6790/0 conn(0x7fb0f412e330 legacy=0x7fb0f4143460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:12.784 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.490+0000 7fb0f37fe700 1 -- 172.21.15.102:0/2939790883 reap_dead start 2021-05-25T12:08:12.785 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.490+0000 7fb0f37fe700 1 -- 172.21.15.102:0/2939790883 reap_dead start 2021-05-25T12:08:12.785 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.490+0000 7fb0f92e0700 1 -- 172.21.15.102:0/2939790883 shutdown_connections 2021-05-25T12:08:12.785 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.490+0000 7fb0f92e0700 1 -- 172.21.15.102:0/2939790883 >> 172.21.15.102:0/2939790883 conn(0x7fb0f409b370 msgr2=0x7fb0f4095260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:08:12.785 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.490+0000 7fb0f92e0700 1 -- 172.21.15.102:0/2939790883 shutdown_connections 2021-05-25T12:08:12.786 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.490+0000 7fb0f92e0700 1 -- 172.21.15.102:0/2939790883 wait complete. 2021-05-25T12:08:12.786 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.490+0000 7fb0f92e0700 1 librados: shutdown 2021-05-25T12:08:12.786 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2021-05-25T12:08:12.787 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2021-05-25T12:08:12.895 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.894+0000 7f5a0608a700 10 monclient: get_monmap_and_config 2021-05-25T12:08:12.896 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.894+0000 7f5a0608a700 10 monclient: build_initial_monmap 2021-05-25T12:08:12.896 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.894+0000 7f5a0608a700 10 monclient: monmap: 2021-05-25T12:08:12.896 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:08:12.896 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:12.896 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:08:12.895437+0000 2021-05-25T12:08:12.897 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:08:12.895437+0000 2021-05-25T12:08:12.897 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:08:12.897 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:12.897 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:08:12.898 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:08:12.898 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:08:12.898 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:12.909 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.895+0000 7f5a0608a700 1 Processor -- start 2021-05-25T12:08:12.910 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.895+0000 7f5a0608a700 1 -- start start 2021-05-25T12:08:12.910 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.895+0000 7f5a0608a700 10 monclient: init 2021-05-25T12:08:12.910 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.895+0000 7f5a0608a700 10 monclient: _reopen_session rank -1 2021-05-25T12:08:12.910 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.895+0000 7f5a0608a700 10 monclient: _add_conns ranks=[1,2,0] 2021-05-25T12:08:12.911 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.895+0000 7f5a0608a700 10 monclient(hunting): picked mon.noname-c con 0x7f5a00095c40 addr v1:172.21.15.181:6789/0 2021-05-25T12:08:12.911 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.895+0000 7f5a0608a700 10 monclient(hunting): picked mon.noname-b con 0x7f5a00096020 addr v1:172.21.15.102:6790/0 2021-05-25T12:08:12.911 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.895+0000 7f5a0608a700 10 monclient(hunting): picked mon.noname-a con 0x7f5a0012e5c0 addr v1:172.21.15.102:6789/0 2021-05-25T12:08:12.911 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.895+0000 7f5a0608a700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a00092990 con 0x7f5a0012e5c0 2021-05-25T12:08:12.912 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.895+0000 7f5a0608a700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a00092c30 con 0x7f5a00095c40 2021-05-25T12:08:12.912 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.895+0000 7f5a0608a700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a00092ed0 con 0x7f5a00096020 2021-05-25T12:08:12.912 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.895+0000 7f5a0608a700 10 monclient(hunting): _renew_subs 2021-05-25T12:08:12.912 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.895+0000 7f5a0608a700 10 monclient(hunting): authenticate will time out at 1293.492830s 2021-05-25T12:08:12.912 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.895+0000 7f5a04887700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f5a00096020 0x7f5a000964d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:57236/0 (socket says 172.21.15.102:57236) 2021-05-25T12:08:12.913 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.895+0000 7f5a05889700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f5a0012e5c0 0x7f5a001309b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:34128/0 (socket says 172.21.15.102:34128) 2021-05-25T12:08:12.913 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.895+0000 7f5a04887700 1 -- 172.21.15.102:0/2721571300 learned_addr learned my addr 172.21.15.102:0/2721571300 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:08:12.913 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.896+0000 7f59effff700 1 -- 172.21.15.102:0/2721571300 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3758521522 0 0) 0x7f5a00092990 con 0x7f5a0012e5c0 2021-05-25T12:08:12.913 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.896+0000 7f59effff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:12.914 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.896+0000 7f59effff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:12.914 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.896+0000 7f59effff700 10 monclient(hunting): my global_id is 204519 2021-05-25T12:08:12.914 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.896+0000 7f59effff700 1 -- 172.21.15.102:0/2721571300 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59e8003270 con 0x7f5a0012e5c0 2021-05-25T12:08:12.914 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.896+0000 7f59effff700 1 -- 172.21.15.102:0/2721571300 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 198404055 0 0) 0x7f5a00092ed0 con 0x7f5a00096020 2021-05-25T12:08:12.914 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.896+0000 7f59effff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:12.915 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.896+0000 7f59effff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:12.915 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.896+0000 7f59effff700 10 monclient(hunting): my global_id is 244124 2021-05-25T12:08:12.915 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.896+0000 7f59effff700 1 -- 172.21.15.102:0/2721571300 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a00092990 con 0x7f5a00096020 2021-05-25T12:08:12.915 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.896+0000 7f5a04887700 1 -- 172.21.15.102:0/2721571300 >> v1:172.21.15.102:6790/0 conn(0x7f5a00096020 legacy=0x7f5a000964d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-05-25T12:08:12.916 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.896+0000 7f5a04887700 1 -- 172.21.15.102:0/2721571300 >> v1:172.21.15.102:6790/0 conn(0x7f5a00096020 legacy=0x7f5a000964d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-05-25T12:08:12.916 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.896+0000 7f5a04887700 1 --1- 172.21.15.102:0/2721571300 >> v1:172.21.15.102:6790/0 conn(0x7f5a00096020 0x7f5a000964d0 :-1 s=OPENED pgs=7515 cs=1 l=1).handle_message read tag failed 2021-05-25T12:08:12.916 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.896+0000 7f5a04887700 1 --1- 172.21.15.102:0/2721571300 >> v1:172.21.15.102:6790/0 conn(0x7f5a00096020 0x7f5a000964d0 :-1 s=OPENED pgs=7515 cs=1 l=1).fault on lossy channel, failing 2021-05-25T12:08:12.916 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.896+0000 7f59effff700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.102:6790/0 2021-05-25T12:08:12.916 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.896+0000 7f59effff700 1 -- 172.21.15.102:0/2721571300 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 302121516 0 0) 0x7f5a00092c30 con 0x7f5a00095c40 2021-05-25T12:08:12.917 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.896+0000 7f59effff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:12.917 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.896+0000 7f59effff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:12.917 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.896+0000 7f59effff700 10 monclient(hunting): my global_id is 224380 2021-05-25T12:08:12.917 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.896+0000 7f59effff700 1 -- 172.21.15.102:0/2721571300 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a00092ed0 con 0x7f5a00095c40 2021-05-25T12:08:12.918 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.896+0000 7f59effff700 1 -- 172.21.15.102:0/2721571300 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3207160946 0 0) 0x7f59e8003270 con 0x7f5a0012e5c0 2021-05-25T12:08:12.918 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.896+0000 7f59effff700 1 -- 172.21.15.102:0/2721571300 >> v1:172.21.15.102:6790/0 conn(0x7f5a00096020 legacy=0x7f5a000964d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:08:12.918 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.896+0000 7f59effff700 1 -- 172.21.15.102:0/2721571300 >> v1:172.21.15.181:6789/0 conn(0x7f5a00095c40 legacy=0x7f5a00093290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:12.918 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.896+0000 7f59effff700 10 monclient: _finish_hunting 0 2021-05-25T12:08:12.919 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.896+0000 7f59effff700 1 monclient: found mon.noname-a 2021-05-25T12:08:12.919 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.896+0000 7f59effff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:08:12.919 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.896+0000 7f59effff700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T12:08:12.919 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.896+0000 7f59effff700 1 -- 172.21.15.102:0/2721571300 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a00092430 con 0x7f5a0012e5c0 2021-05-25T12:08:12.919 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.896+0000 7f59effff700 10 monclient: _finish_auth 0 2021-05-25T12:08:12.920 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.896+0000 7f59effff700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:08:12.920 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.896+0000 7f59effff700 1 -- 172.21.15.102:0/2721571300 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f59fc0019e0 con 0x7f5a0012e5c0 2021-05-25T12:08:12.920 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.896+0000 7f5a0608a700 5 monclient: authenticate success, global_id 204519 2021-05-25T12:08:12.920 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.896+0000 7f5a0608a700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:08:12.920 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.896+0000 7f59effff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:12.921 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.896+0000 7f59effff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:08:12.921 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.896+0000 7f59effff700 10 monclient: dump: 2021-05-25T12:08:12.921 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:12.921 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:12.922 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:12.922 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:12.922 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:12.922 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:12.922 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:12.923 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:12.923 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:12.923 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:12.923 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.896+0000 7f5a0608a700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:08:12.923 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.896+0000 7f59effff700 1 -- 172.21.15.102:0/2721571300 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f59fc001bd0 con 0x7f5a0012e5c0 2021-05-25T12:08:12.924 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.896+0000 7f59effff700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:08:12.924 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.896+0000 7f5a0608a700 10 monclient: get_monmap_and_config success 2021-05-25T12:08:12.924 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.896+0000 7f59effff700 1 -- 172.21.15.102:0/2721571300 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f59fc001f60 con 0x7f5a0012e5c0 2021-05-25T12:08:12.924 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.896+0000 7f59effff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:12.925 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.896+0000 7f5a0608a700 4 set_mon_vals no callback set 2021-05-25T12:08:12.925 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.896+0000 7f59effff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:08:12.925 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.896+0000 7f59effff700 10 monclient: dump: 2021-05-25T12:08:12.925 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:12.925 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:12.926 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:12.926 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:12.926 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:12.926 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:12.927 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:12.927 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:12.927 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:12.927 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:12.928 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.896+0000 7f5a0608a700 10 monclient: shutdown 2021-05-25T12:08:12.928 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.896+0000 7f5a0608a700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:08:12.928 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.896+0000 7f5a0608a700 1 -- 172.21.15.102:0/2721571300 >> v1:172.21.15.102:6789/0 conn(0x7f5a0012e5c0 legacy=0x7f5a001309b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:12.928 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.896+0000 7f5a0608a700 1 -- 172.21.15.102:0/2721571300 shutdown_connections 2021-05-25T12:08:12.929 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.897+0000 7f5a0608a700 1 -- 172.21.15.102:0/2721571300 >> 172.21.15.102:0/2721571300 conn(0x7f5a000974b0 msgr2=0x7f5a000978c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:08:12.929 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.897+0000 7f5a0608a700 1 -- 172.21.15.102:0/2721571300 shutdown_connections 2021-05-25T12:08:12.929 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.897+0000 7f5a0608a700 1 -- 172.21.15.102:0/2721571300 wait complete. 2021-05-25T12:08:12.929 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.897+0000 7f5a0608a700 10 monclient: build_initial_monmap 2021-05-25T12:08:12.929 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.897+0000 7f5a0608a700 10 monclient: monmap: 2021-05-25T12:08:12.930 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:08:12.930 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:08:12.930 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:08:12.930 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:08:12.930 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:08:12.931 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:08:12.931 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:08:12.931 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:08:12.931 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:08:12.932 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:12.932 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.897+0000 7f5a0608a700 1 librados: starting msgr at 2021-05-25T12:08:12.932 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.897+0000 7f5a0608a700 1 librados: starting objecter 2021-05-25T12:08:12.932 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.897+0000 7f5a0608a700 1 Processor -- start 2021-05-25T12:08:12.933 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.897+0000 7f5a0608a700 1 -- start start 2021-05-25T12:08:12.933 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.897+0000 7f5a0608a700 1 librados: setting wanted keys 2021-05-25T12:08:12.933 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.897+0000 7f5a0608a700 1 librados: calling monclient init 2021-05-25T12:08:12.933 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.897+0000 7f5a0608a700 10 monclient: init 2021-05-25T12:08:12.933 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.898+0000 7f5a0608a700 10 monclient: _reopen_session rank -1 2021-05-25T12:08:12.934 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.898+0000 7f5a0608a700 10 monclient: _add_conns ranks=[0,1,2] 2021-05-25T12:08:12.934 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.898+0000 7f5a0608a700 10 monclient(hunting): picked mon.noname-a con 0x7f5a00095c40 addr v1:172.21.15.102:6789/0 2021-05-25T12:08:12.934 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.898+0000 7f5a0608a700 10 monclient(hunting): picked mon.noname-b con 0x7f5a00096020 addr v1:172.21.15.181:6789/0 2021-05-25T12:08:12.934 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.898+0000 7f5a0608a700 10 monclient(hunting): picked mon.noname-c con 0x7f5a0012e5c0 addr v1:172.21.15.102:6790/0 2021-05-25T12:08:12.935 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.898+0000 7f5a0608a700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a00137b30 con 0x7f5a00095c40 2021-05-25T12:08:12.935 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.898+0000 7f5a0608a700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a00113040 con 0x7f5a00096020 2021-05-25T12:08:12.935 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.898+0000 7f5a0608a700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a0012da30 con 0x7f5a0012e5c0 2021-05-25T12:08:12.935 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.898+0000 7f5a0608a700 10 monclient(hunting): _renew_subs 2021-05-25T12:08:12.936 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.898+0000 7f5a0608a700 10 monclient(hunting): authenticate will time out at 1293.495489s 2021-05-25T12:08:12.936 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.898+0000 7f5a05889700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f5a0012e5c0 0x7f5a0013f080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:57244/0 (socket says 172.21.15.102:57244) 2021-05-25T12:08:12.936 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.898+0000 7f5a05088700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f5a00095c40 0x7f5a00136d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:34130/0 (socket says 172.21.15.102:34130) 2021-05-25T12:08:12.936 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.898+0000 7f5a05889700 1 -- 172.21.15.102:0/1029983626 learned_addr learned my addr 172.21.15.102:0/1029983626 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:08:12.937 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.898+0000 7f59ed7fa700 10 client.?.objecter ms_handle_connect 0x7f5a0012e5c0 2021-05-25T12:08:12.937 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.898+0000 7f59ed7fa700 10 client.?.objecter resend_mon_ops 2021-05-25T12:08:12.937 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.898+0000 7f59ed7fa700 10 client.?.objecter ms_handle_connect 0x7f5a00095c40 2021-05-25T12:08:12.937 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.898+0000 7f59ed7fa700 10 client.?.objecter resend_mon_ops 2021-05-25T12:08:12.938 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.898+0000 7f59ed7fa700 10 client.?.objecter ms_handle_connect 0x7f5a00096020 2021-05-25T12:08:12.938 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.898+0000 7f59ed7fa700 10 client.?.objecter resend_mon_ops 2021-05-25T12:08:12.938 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.898+0000 7f59ed7fa700 1 -- 172.21.15.102:0/1029983626 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2654705155 0 0) 0x7f5a0012da30 con 0x7f5a0012e5c0 2021-05-25T12:08:12.938 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.898+0000 7f59ed7fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:12.938 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.898+0000 7f59ed7fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:12.939 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.898+0000 7f59ed7fa700 10 monclient(hunting): my global_id is 244127 2021-05-25T12:08:12.939 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.898+0000 7f59ed7fa700 1 -- 172.21.15.102:0/1029983626 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59e0004c10 con 0x7f5a0012e5c0 2021-05-25T12:08:12.939 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.898+0000 7f59ed7fa700 1 -- 172.21.15.102:0/1029983626 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2939489020 0 0) 0x7f5a00137b30 con 0x7f5a00095c40 2021-05-25T12:08:12.939 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.898+0000 7f59ed7fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:12.940 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.898+0000 7f59ed7fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:12.940 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.898+0000 7f59ed7fa700 10 monclient(hunting): my global_id is 204522 2021-05-25T12:08:12.940 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.898+0000 7f59ed7fa700 1 -- 172.21.15.102:0/1029983626 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a0012da30 con 0x7f5a00095c40 2021-05-25T12:08:12.940 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.898+0000 7f59ed7fa700 1 -- 172.21.15.102:0/1029983626 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 492609206 0 0) 0x7f5a00113040 con 0x7f5a00096020 2021-05-25T12:08:12.940 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.898+0000 7f59ed7fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:12.941 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.898+0000 7f59ed7fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:12.941 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.898+0000 7f59ed7fa700 10 monclient(hunting): my global_id is 224383 2021-05-25T12:08:12.941 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.898+0000 7f59ed7fa700 1 -- 172.21.15.102:0/1029983626 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a00137b30 con 0x7f5a00096020 2021-05-25T12:08:12.941 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.899+0000 7f59ed7fa700 1 -- 172.21.15.102:0/1029983626 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1635163218 0 0) 0x7f59e0004c10 con 0x7f5a0012e5c0 2021-05-25T12:08:12.942 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.899+0000 7f59ed7fa700 1 -- 172.21.15.102:0/1029983626 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a00113040 con 0x7f5a0012e5c0 2021-05-25T12:08:12.942 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.899+0000 7f59ed7fa700 1 -- 172.21.15.102:0/1029983626 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f59fc0018e0 con 0x7f5a0012e5c0 2021-05-25T12:08:12.942 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.899+0000 7f59ed7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:12.942 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.899+0000 7f59ed7fa700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:08:12.942 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.899+0000 7f59ed7fa700 10 monclient(hunting): dump: 2021-05-25T12:08:12.943 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:12.943 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:12.943 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:12.943 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:12.944 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:12.944 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:12.944 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:12.944 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:12.944 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:12.945 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:12.945 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.899+0000 7f59ed7fa700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:08:12.945 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.899+0000 7f59ed7fa700 1 -- 172.21.15.102:0/1029983626 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2773806074 0 0) 0x7f5a0012da30 con 0x7f5a00095c40 2021-05-25T12:08:12.945 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.899+0000 7f59ed7fa700 1 -- 172.21.15.102:0/1029983626 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f59e0004c10 con 0x7f5a00095c40 2021-05-25T12:08:12.945 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.899+0000 7f59ed7fa700 1 -- 172.21.15.102:0/1029983626 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f59f4001800 con 0x7f5a00095c40 2021-05-25T12:08:12.946 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.899+0000 7f59ed7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:12.946 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.899+0000 7f59ed7fa700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2021-05-25T12:08:12.946 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.899+0000 7f59ed7fa700 10 monclient(hunting): dump: 2021-05-25T12:08:12.946 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:12.946 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:12.947 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:12.947 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:12.947 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:12.947 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:12.947 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:12.948 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:12.948 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:12.948 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:12.948 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.899+0000 7f59ed7fa700 1 -- 172.21.15.102:0/1029983626 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2437752394 0 0) 0x7f5a00137b30 con 0x7f5a00096020 2021-05-25T12:08:12.949 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.899+0000 7f59ed7fa700 1 -- 172.21.15.102:0/1029983626 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a0012da30 con 0x7f5a00096020 2021-05-25T12:08:12.949 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.899+0000 7f59ed7fa700 1 -- 172.21.15.102:0/1029983626 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f59f0001870 con 0x7f5a00096020 2021-05-25T12:08:12.949 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.899+0000 7f59ed7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:12.949 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.899+0000 7f59ed7fa700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2021-05-25T12:08:12.950 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.899+0000 7f59ed7fa700 10 monclient(hunting): dump: 2021-05-25T12:08:12.950 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:12.950 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:12.950 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:12.950 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:12.951 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:12.951 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:12.951 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:12.951 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:12.951 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:12.952 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:12.952 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.899+0000 7f59ed7fa700 1 -- 172.21.15.102:0/1029983626 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1970507341 0 0) 0x7f5a00113040 con 0x7f5a0012e5c0 2021-05-25T12:08:12.952 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.899+0000 7f59ed7fa700 1 -- 172.21.15.102:0/1029983626 >> v1:172.21.15.181:6789/0 conn(0x7f5a00096020 legacy=0x7f5a00137420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:12.952 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.899+0000 7f59ed7fa700 1 -- 172.21.15.102:0/1029983626 >> v1:172.21.15.102:6789/0 conn(0x7f5a00095c40 legacy=0x7f5a00136d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:12.953 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.899+0000 7f59ed7fa700 10 monclient: _finish_hunting 0 2021-05-25T12:08:12.953 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.899+0000 7f59ed7fa700 1 monclient: found mon.c 2021-05-25T12:08:12.953 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.899+0000 7f59ed7fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:08:12.953 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.899+0000 7f59ed7fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:08:12.953 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.899+0000 7f59ed7fa700 1 -- 172.21.15.102:0/1029983626 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a00110ff0 con 0x7f5a0012e5c0 2021-05-25T12:08:12.954 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.899+0000 7f59ed7fa700 10 monclient: _finish_auth 0 2021-05-25T12:08:12.954 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.899+0000 7f59ed7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:08:12.954 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.899+0000 7f5a0608a700 5 monclient: authenticate success, global_id 244127 2021-05-25T12:08:12.954 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.899+0000 7f5a0608a700 10 monclient: _renew_subs 2021-05-25T12:08:12.955 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.899+0000 7f5a0608a700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:08:12.955 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.899+0000 7f5a0608a700 1 -- 172.21.15.102:0/1029983626 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a00111b70 con 0x7f5a0012e5c0 2021-05-25T12:08:12.955 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.899+0000 7f5a0608a700 10 client.244127.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:08:12.955 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.899+0000 7f5a0608a700 10 monclient: _renew_subs 2021-05-25T12:08:12.955 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.899+0000 7f5a0608a700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:08:12.956 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.899+0000 7f5a0608a700 1 -- 172.21.15.102:0/1029983626 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a00128ac0 con 0x7f5a0012e5c0 2021-05-25T12:08:12.956 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.899+0000 7f5a0608a700 1 librados: init done 2021-05-25T12:08:12.956 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.899+0000 7f59ed7fa700 1 -- 172.21.15.102:0/1029983626 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f59fc004c70 con 0x7f5a0012e5c0 2021-05-25T12:08:12.956 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.899+0000 7f59ed7fa700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:08:12.957 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.899+0000 7f59ed7fa700 1 -- 172.21.15.102:0/1029983626 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f59fc004f80 con 0x7f5a0012e5c0 2021-05-25T12:08:12.957 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.899+0000 7f59ed7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:12.957 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.899+0000 7f59ed7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:08:12.957 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.899+0000 7f59ef7fe700 4 set_mon_vals no callback set 2021-05-25T12:08:12.957 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.899+0000 7f59ed7fa700 10 monclient: dump: 2021-05-25T12:08:12.958 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:12.958 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:12.958 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:12.958 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:12.959 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:12.959 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:12.959 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:12.959 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:12.959 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:12.960 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:12.960 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.900+0000 7f59ed7fa700 1 -- 172.21.15.102:0/1029983626 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f59fc005160 con 0x7f5a0012e5c0 2021-05-25T12:08:12.960 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.900+0000 7f59ed7fa700 1 -- 172.21.15.102:0/1029983626 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(269..269 src has 1..269) v4 ==== 6249+0+0 (unknown 490464239 0 0) 0x7f59fc03dd70 con 0x7f5a0012e5c0 2021-05-25T12:08:12.960 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.900+0000 7f59ed7fa700 10 client.244127.objecter ms_dispatch 0x7f5a00001550 osd_map(269..269 src has 1..269) v4 2021-05-25T12:08:12.960 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.900+0000 7f5a0608a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:08:12.960 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.900+0000 7f59ed7fa700 3 client.244127.objecter handle_osd_map got epochs [269,269] > 0 2021-05-25T12:08:12.961 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.900+0000 7f59ed7fa700 3 client.244127.objecter handle_osd_map decoding full epoch 269 2021-05-25T12:08:12.961 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.900+0000 7f5a0608a700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:08:12.961 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.900+0000 7f5a0608a700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:08:12.961 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.900+0000 7f5a0608a700 1 -- 172.21.15.102:0/1029983626 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59d8004db0 con 0x7f5a0012e5c0 2021-05-25T12:08:12.962 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.900+0000 7f59ed7fa700 20 client.244127.objecter dump_active .. 0 homeless 2021-05-25T12:08:12.962 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.900+0000 7f5a05088700 10 monclient: get_auth_request con 0x7f59e0033500 auth_method 0 2021-05-25T12:08:12.962 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.900+0000 7f59ed7fa700 10 client.244127.objecter ms_handle_connect 0x7f59e0033500 2021-05-25T12:08:12.962 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.904+0000 7f59ed7fa700 1 -- 172.21.15.102:0/1029983626 <== mon.2 v1:172.21.15.102:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7f59fc013060 con 0x7f5a0012e5c0 2021-05-25T12:08:12.962 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.904+0000 7f59ed7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:08:12.963 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:12.904+0000 7f59ed7fa700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:08:13.112 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:13.111+0000 7f5a0608a700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2021-05-25T12:08:13.112 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:13.111+0000 7f5a0608a700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2021-05-25T12:08:13.112 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:13.111+0000 7f5a0608a700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:08:13.113 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:13.111+0000 7f5a0608a700 1 -- 172.21.15.102:0/1029983626 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f59d8005870 con 0x7f5a0012e5c0 2021-05-25T12:08:13.114 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:13.112+0000 7f59ed7fa700 1 -- 172.21.15.102:0/1029983626 <== mon.2 v1:172.21.15.102:6790/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7f59fc012e80 con 0x7f5a0012e5c0 2021-05-25T12:08:13.114 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:13.113+0000 7f59ed7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2021-05-25T12:08:13.114 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:13.113+0000 7f59ed7fa700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:08:13.117 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:13.113+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:08:13.117 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:13.113+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=31,config=2,health=12,kv=5,logm=50} crc {auth=2575645200,config=3140514009,health=1407995967,kv=825553405,logm=165242447}) 2021-05-25T12:08:13.117 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:13.113+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=31,config=2,health=10,kv=5,logm=52} crc {auth=1460755790,config=3140514009,health=184593953,kv=825553405,logm=1427571493}) 2021-05-25T12:08:13.117 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:13.113+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:08:13.118 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:13.113+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=31,config=2,health=12,kv=5,logm=50} crc {auth=2575645200,config=3140514009,health=1407995967,kv=825553405,logm=165242447}) 2021-05-25T12:08:13.118 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:13.113+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=31,config=2,health=11,kv=6,logm=50} crc {auth=3842649701,config=3140514009,health=3811324204,kv=1184736763,logm=1537473108}) 2021-05-25T12:08:13.118 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:13.114+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:08:13.119 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:13.114+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2520721860}) 2021-05-25T12:08:13.119 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:13.114+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=822236702}) 2021-05-25T12:08:13.120 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:13.114+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:08:13.120 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:13.114+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2520721860}) 2021-05-25T12:08:13.121 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:13.114+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=284999249}) 2021-05-25T12:08:13.121 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:13.115+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:08:13.121 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:13.115+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3696032465}) 2021-05-25T12:08:13.122 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:13.115+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3028537477}) 2021-05-25T12:08:13.122 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:13.115+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:08:13.122 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:13.115+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3696032465}) 2021-05-25T12:08:13.123 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:13.115+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2430329926}) 2021-05-25T12:08:13.124 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:13.114+0000 7f5a0608a700 10 librados: watch_flush enter 2021-05-25T12:08:13.124 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:13.114+0000 7f5a0608a700 10 librados: watch_flush exit 2021-05-25T12:08:13.124 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:13.114+0000 7f5a0608a700 20 client.244127.objecter shutdown clearing up homeless session... 2021-05-25T12:08:13.124 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:13.114+0000 7f5a0608a700 10 client.244127.objecter successfully canceled tick 2021-05-25T12:08:13.125 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:13.114+0000 7f5a0608a700 1 -- 172.21.15.102:0/1029983626 >> v1:172.21.15.181:6812/29582 conn(0x7f59e0033500 legacy=0x7f59e00359c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:13.125 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:13.114+0000 7f5a0608a700 10 monclient: shutdown 2021-05-25T12:08:13.125 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:13.114+0000 7f5a0608a700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:08:13.125 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:13.114+0000 7f5a0608a700 1 -- 172.21.15.102:0/1029983626 >> v1:172.21.15.102:6790/0 conn(0x7f5a0012e5c0 legacy=0x7f5a0013f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:13.126 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:13.114+0000 7f5a0608a700 1 -- 172.21.15.102:0/1029983626 shutdown_connections 2021-05-25T12:08:13.126 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:13.114+0000 7f5a0608a700 1 -- 172.21.15.102:0/1029983626 >> 172.21.15.102:0/1029983626 conn(0x7f5a000974b0 msgr2=0x7f5a0012cf40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:08:13.126 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:13.114+0000 7f5a0608a700 1 -- 172.21.15.102:0/1029983626 shutdown_connections 2021-05-25T12:08:13.126 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:13.114+0000 7f5a0608a700 1 -- 172.21.15.102:0/1029983626 wait complete. 2021-05-25T12:08:13.127 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:13.114+0000 7f5a0608a700 1 librados: shutdown 2021-05-25T12:08:13.129 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:13.119+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:08:13.130 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:13.119+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2719375150}) 2021-05-25T12:08:13.130 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:13.119+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2386921562}) 2021-05-25T12:08:13.130 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:13.119+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:08:13.130 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:13.119+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2719375150}) 2021-05-25T12:08:13.130 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:13.119+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1895798141}) 2021-05-25T12:08:13.131 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:13.123+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:08:13.131 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:13.123+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1902507674}) 2021-05-25T12:08:13.131 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:13.123+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2542125644}) 2021-05-25T12:08:13.131 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:13.123+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:08:13.132 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:13.123+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1902507674}) 2021-05-25T12:08:13.132 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:13.123+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=526276142}) 2021-05-25T12:08:13.132 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:13.125+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:08:13.132 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:13.125+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=42,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=4,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=22} crc {logm=2642110635,mdsmap=1531151339,mgr=2178563563,mgr_command_descs=3824783157,mgr_metadata=1267308364,mgrstat=223041154,mon_config_key=4118394475,monmap=3809614708,osd_metadata=2434534451,osd_pg_creating=1952540717,osd_snap=3185801450}) 2021-05-25T12:08:13.133 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:13.125+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=44,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=4,monmap=1,osd_metadata=6,osd_pg_creating=1,osd_snap=21} crc {logm=1119643665,mdsmap=1531151339,mgr=2996579194,mgr_command_descs=3824783157,mgr_metadata=1267308364,mgrstat=3254972636,mon_config_key=4118394475,monmap=1027366620,osd_metadata=725478305,osd_pg_creating=1952540717,osd_snap=2285772520}) 2021-05-25T12:08:13.133 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:13.125+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:08:13.133 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:13.125+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=42,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=4,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=22} crc {logm=2642110635,mdsmap=1531151339,mgr=2178563563,mgr_command_descs=3824783157,mgr_metadata=1267308364,mgrstat=223041154,mon_config_key=4118394475,monmap=3809614708,osd_metadata=2434534451,osd_pg_creating=1952540717,osd_snap=3185801450}) 2021-05-25T12:08:13.133 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:13.125+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=42,mdsmap=3,mgr=7,mgr_command_descs=1,mgrstat=13,mon_config_key=4,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=20} crc {logm=2936534906,mdsmap=1531151339,mgr=2996579194,mgr_command_descs=3824783157,mgrstat=3337918247,mon_config_key=4118394475,monmap=3809614708,osd_metadata=725478305,osd_pg_creating=1952540717,osd_snap=3885120060}) 2021-05-25T12:08:13.134 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:13.127+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:08:13.134 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:13.127+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=7,osdmap=93} crc {osd_snap=3261585773,osdmap=4095231532}) 2021-05-25T12:08:13.134 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:13.127+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=8,osdmap=92} crc {osd_snap=3526248523,osdmap=208898331}) 2021-05-25T12:08:13.134 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:13.127+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:08:13.135 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:13.127+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=7,osdmap=93} crc {osd_snap=3261585773,osdmap=4095231532}) 2021-05-25T12:08:13.135 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:13.127+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=8,osdmap=92} crc {osd_snap=3526248523,osdmap=3454262283}) 2021-05-25T12:08:13.135 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:13.128+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:08:13.135 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:13.128+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2574877073}) 2021-05-25T12:08:13.135 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:13.128+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1977590637}) 2021-05-25T12:08:13.136 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:13.128+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:08:13.136 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:13.128+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2574877073}) 2021-05-25T12:08:13.136 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:13.128+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1831107400}) 2021-05-25T12:08:13.139 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2021-05-25T12:08:13.140 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:13.129+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:08:13.140 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:13.129+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=695137460}) 2021-05-25T12:08:13.140 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:13.129+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=967168155}) 2021-05-25T12:08:13.141 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:13.129+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:08:13.141 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:13.129+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=695137460}) 2021-05-25T12:08:13.141 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:13.129+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1390737368}) 2021-05-25T12:08:13.142 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:13.131+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:08:13.142 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:13.131+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=736497174}) 2021-05-25T12:08:13.143 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:13.131+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3441571703}) 2021-05-25T12:08:13.143 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:13.131+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:08:13.143 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:13.131+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=736497174}) 2021-05-25T12:08:13.144 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:13.131+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=361681957}) 2021-05-25T12:08:13.144 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:13.137+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:08:13.144 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:13.137+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=6} crc {osdmap=333644190}) 2021-05-25T12:08:13.145 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:13.137+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=3} crc {osdmap=1547267167}) 2021-05-25T12:08:13.145 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:13.137+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:08:13.145 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:13.137+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=6} crc {osdmap=333644190}) 2021-05-25T12:08:13.145 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:13.137+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=5} crc {osdmap=2917709037}) 2021-05-25T12:08:14.139 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2021-05-25T12:08:14.140 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2021-05-25T12:08:14.251 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.250+0000 7f9062d2d700 10 monclient: get_monmap_and_config 2021-05-25T12:08:14.251 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.250+0000 7f9062d2d700 10 monclient: build_initial_monmap 2021-05-25T12:08:14.252 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.250+0000 7f9062d2d700 10 monclient: monmap: 2021-05-25T12:08:14.252 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:08:14.252 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:14.252 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:08:14.250990+0000 2021-05-25T12:08:14.253 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:08:14.250990+0000 2021-05-25T12:08:14.253 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:08:14.253 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:14.253 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:08:14.254 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:08:14.254 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:08:14.254 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:14.265 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.250+0000 7f9062d2d700 1 Processor -- start 2021-05-25T12:08:14.266 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.250+0000 7f9062d2d700 1 -- start start 2021-05-25T12:08:14.266 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.250+0000 7f9062d2d700 10 monclient: init 2021-05-25T12:08:14.266 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.251+0000 7f9062d2d700 10 monclient: _reopen_session rank -1 2021-05-25T12:08:14.266 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.251+0000 7f9062d2d700 10 monclient: _add_conns ranks=[1,0,2] 2021-05-25T12:08:14.267 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.251+0000 7f9062d2d700 10 monclient(hunting): picked mon.noname-c con 0x7f905c09a5e0 addr v1:172.21.15.181:6789/0 2021-05-25T12:08:14.267 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.251+0000 7f9062d2d700 10 monclient(hunting): picked mon.noname-a con 0x7f905c063dc0 addr v1:172.21.15.102:6789/0 2021-05-25T12:08:14.267 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.251+0000 7f9062d2d700 10 monclient(hunting): picked mon.noname-b con 0x7f905c1325c0 addr v1:172.21.15.102:6790/0 2021-05-25T12:08:14.267 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.251+0000 7f9062d2d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f905c099fe0 con 0x7f905c063dc0 2021-05-25T12:08:14.268 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.251+0000 7f9062d2d700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f905c09a1e0 con 0x7f905c09a5e0 2021-05-25T12:08:14.268 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.251+0000 7f9062d2d700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f905c09b400 con 0x7f905c1325c0 2021-05-25T12:08:14.268 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.251+0000 7f9062d2d700 10 monclient(hunting): _renew_subs 2021-05-25T12:08:14.268 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.251+0000 7f9062d2d700 10 monclient(hunting): authenticate will time out at 1294.848266s 2021-05-25T12:08:14.268 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.251+0000 7f9053fff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f905c063dc0 0x7f905c064240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:34144/0 (socket says 172.21.15.102:34144) 2021-05-25T12:08:14.269 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.251+0000 7f90612ca700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f905c1325c0 0x7f905c1349b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:57258/0 (socket says 172.21.15.102:57258) 2021-05-25T12:08:14.269 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.251+0000 7f9053fff700 1 -- 172.21.15.102:0/2413581151 learned_addr learned my addr 172.21.15.102:0/2413581151 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:08:14.269 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.251+0000 7f90537fe700 1 -- 172.21.15.102:0/2413581151 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3020415011 0 0) 0x7f905c099fe0 con 0x7f905c063dc0 2021-05-25T12:08:14.269 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.251+0000 7f90537fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:14.270 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.251+0000 7f90537fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:14.270 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.251+0000 7f90537fe700 10 monclient(hunting): my global_id is 204525 2021-05-25T12:08:14.270 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.251+0000 7f90537fe700 1 -- 172.21.15.102:0/2413581151 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9048003270 con 0x7f905c063dc0 2021-05-25T12:08:14.270 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.251+0000 7f90537fe700 1 -- 172.21.15.102:0/2413581151 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 781607741 0 0) 0x7f905c09b400 con 0x7f905c1325c0 2021-05-25T12:08:14.271 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.251+0000 7f90537fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:14.271 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.251+0000 7f90537fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:14.271 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.251+0000 7f90537fe700 10 monclient(hunting): my global_id is 244130 2021-05-25T12:08:14.271 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.251+0000 7f90537fe700 1 -- 172.21.15.102:0/2413581151 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f905c099fe0 con 0x7f905c1325c0 2021-05-25T12:08:14.271 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.251+0000 7f90537fe700 1 -- 172.21.15.102:0/2413581151 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2306180287 0 0) 0x7f905c09a1e0 con 0x7f905c09a5e0 2021-05-25T12:08:14.272 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.251+0000 7f90537fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:14.272 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.251+0000 7f90537fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:14.272 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.251+0000 7f90537fe700 10 monclient(hunting): my global_id is 224386 2021-05-25T12:08:14.272 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.251+0000 7f90537fe700 1 -- 172.21.15.102:0/2413581151 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f905c09b400 con 0x7f905c09a5e0 2021-05-25T12:08:14.273 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.251+0000 7f90537fe700 1 -- 172.21.15.102:0/2413581151 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1435889767 0 0) 0x7f9048003270 con 0x7f905c063dc0 2021-05-25T12:08:14.273 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.251+0000 7f90537fe700 1 -- 172.21.15.102:0/2413581151 >> v1:172.21.15.102:6790/0 conn(0x7f905c1325c0 legacy=0x7f905c1349b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:14.273 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.251+0000 7f90537fe700 1 -- 172.21.15.102:0/2413581151 >> v1:172.21.15.181:6789/0 conn(0x7f905c09a5e0 legacy=0x7f905c0636b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:14.273 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.251+0000 7f90537fe700 10 monclient: _finish_hunting 0 2021-05-25T12:08:14.274 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.251+0000 7f90537fe700 1 monclient: found mon.noname-a 2021-05-25T12:08:14.274 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.251+0000 7f90537fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:08:14.274 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.251+0000 7f90537fe700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T12:08:14.274 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.251+0000 7f90537fe700 1 -- 172.21.15.102:0/2413581151 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f905c0963f0 con 0x7f905c063dc0 2021-05-25T12:08:14.275 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.251+0000 7f90537fe700 10 monclient: _finish_auth 0 2021-05-25T12:08:14.275 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.251+0000 7f90537fe700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:08:14.275 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.251+0000 7f9062d2d700 5 monclient: authenticate success, global_id 204525 2021-05-25T12:08:14.276 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.251+0000 7f90537fe700 1 -- 172.21.15.102:0/2413581151 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f9044001590 con 0x7f905c063dc0 2021-05-25T12:08:14.276 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.251+0000 7f9062d2d700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:08:14.276 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.251+0000 7f90537fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:14.276 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.251+0000 7f90537fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:08:14.276 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.251+0000 7f90537fe700 10 monclient: dump: 2021-05-25T12:08:14.277 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:14.277 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:14.277 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:14.278 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:14.278 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:14.279 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:14.279 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:14.279 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:14.280 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:14.280 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:14.280 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.251+0000 7f9062d2d700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:08:14.281 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.252+0000 7f90537fe700 1 -- 172.21.15.102:0/2413581151 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f90440017b0 con 0x7f905c063dc0 2021-05-25T12:08:14.281 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.252+0000 7f90537fe700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:08:14.282 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.252+0000 7f90537fe700 1 -- 172.21.15.102:0/2413581151 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f9044001ac0 con 0x7f905c063dc0 2021-05-25T12:08:14.282 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.252+0000 7f9062d2d700 10 monclient: get_monmap_and_config success 2021-05-25T12:08:14.283 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.252+0000 7f9062d2d700 4 set_mon_vals no callback set 2021-05-25T12:08:14.283 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.252+0000 7f90537fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:14.283 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.252+0000 7f90537fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:08:14.283 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.252+0000 7f90537fe700 10 monclient: dump: 2021-05-25T12:08:14.284 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:14.284 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:14.284 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:14.284 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:14.285 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:14.285 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:14.285 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:14.285 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:14.286 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:14.286 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:14.286 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.252+0000 7f9062d2d700 10 monclient: shutdown 2021-05-25T12:08:14.286 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.252+0000 7f9062d2d700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:08:14.286 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.252+0000 7f9062d2d700 1 -- 172.21.15.102:0/2413581151 >> v1:172.21.15.102:6789/0 conn(0x7f905c063dc0 legacy=0x7f905c064240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:14.287 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.252+0000 7f9062d2d700 1 -- 172.21.15.102:0/2413581151 shutdown_connections 2021-05-25T12:08:14.287 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.252+0000 7f9062d2d700 1 -- 172.21.15.102:0/2413581151 >> 172.21.15.102:0/2413581151 conn(0x7f905c09bc30 msgr2=0x7f905c09c040 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:08:14.287 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.252+0000 7f9062d2d700 1 -- 172.21.15.102:0/2413581151 shutdown_connections 2021-05-25T12:08:14.287 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.252+0000 7f9062d2d700 1 -- 172.21.15.102:0/2413581151 wait complete. 2021-05-25T12:08:14.288 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.252+0000 7f9062d2d700 10 monclient: build_initial_monmap 2021-05-25T12:08:14.288 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.252+0000 7f9062d2d700 10 monclient: monmap: 2021-05-25T12:08:14.288 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:08:14.288 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:08:14.288 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:08:14.289 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:08:14.289 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:08:14.289 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:08:14.289 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:08:14.289 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:08:14.290 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:08:14.290 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:14.290 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.253+0000 7f9062d2d700 1 librados: starting msgr at 2021-05-25T12:08:14.290 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.253+0000 7f9062d2d700 1 librados: starting objecter 2021-05-25T12:08:14.291 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.253+0000 7f9062d2d700 1 Processor -- start 2021-05-25T12:08:14.291 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.253+0000 7f9062d2d700 1 -- start start 2021-05-25T12:08:14.291 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.253+0000 7f9062d2d700 1 librados: setting wanted keys 2021-05-25T12:08:14.291 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.253+0000 7f9062d2d700 1 librados: calling monclient init 2021-05-25T12:08:14.291 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.253+0000 7f9062d2d700 10 monclient: init 2021-05-25T12:08:14.292 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.253+0000 7f9062d2d700 10 monclient: _reopen_session rank -1 2021-05-25T12:08:14.292 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.253+0000 7f9062d2d700 10 monclient: _add_conns ranks=[2,0,1] 2021-05-25T12:08:14.292 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.253+0000 7f9062d2d700 10 monclient(hunting): picked mon.noname-c con 0x7f905c063dc0 addr v1:172.21.15.102:6790/0 2021-05-25T12:08:14.292 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.253+0000 7f9062d2d700 10 monclient(hunting): picked mon.noname-a con 0x7f905c09a5e0 addr v1:172.21.15.102:6789/0 2021-05-25T12:08:14.292 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.253+0000 7f9062d2d700 10 monclient(hunting): picked mon.noname-b con 0x7f905c1325c0 addr v1:172.21.15.181:6789/0 2021-05-25T12:08:14.293 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.253+0000 7f9062d2d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f905c115f60 con 0x7f905c09a5e0 2021-05-25T12:08:14.293 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.253+0000 7f9062d2d700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f905c116200 con 0x7f905c1325c0 2021-05-25T12:08:14.293 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.253+0000 7f9062d2d700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f905c1164a0 con 0x7f905c063dc0 2021-05-25T12:08:14.293 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.253+0000 7f9062d2d700 10 monclient(hunting): _renew_subs 2021-05-25T12:08:14.294 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.253+0000 7f9062d2d700 10 monclient(hunting): authenticate will time out at 1294.850864s 2021-05-25T12:08:14.294 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.253+0000 7f9053fff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f905c09a5e0 0x7f905c1402d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:34152/0 (socket says 172.21.15.102:34152) 2021-05-25T12:08:14.294 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.253+0000 7f9053fff700 1 -- 172.21.15.102:0/489441677 learned_addr learned my addr 172.21.15.102:0/489441677 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:08:14.294 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.253+0000 7f9060ac9700 1 --1- 172.21.15.102:0/489441677 >> v1:172.21.15.102:6790/0 conn(0x7f905c063dc0 0x7f905c12ba90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:57260/0 (socket says 172.21.15.102:57260) 2021-05-25T12:08:14.295 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.254+0000 7f9050ff9700 10 client.?.objecter ms_handle_connect 0x7f905c09a5e0 2021-05-25T12:08:14.295 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.254+0000 7f9050ff9700 10 client.?.objecter resend_mon_ops 2021-05-25T12:08:14.295 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.254+0000 7f9050ff9700 10 client.?.objecter ms_handle_connect 0x7f905c063dc0 2021-05-25T12:08:14.295 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.254+0000 7f9050ff9700 10 client.?.objecter resend_mon_ops 2021-05-25T12:08:14.296 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.254+0000 7f9050ff9700 10 client.?.objecter ms_handle_connect 0x7f905c1325c0 2021-05-25T12:08:14.296 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.254+0000 7f9050ff9700 10 client.?.objecter resend_mon_ops 2021-05-25T12:08:14.296 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.254+0000 7f9050ff9700 1 -- 172.21.15.102:0/489441677 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3853116065 0 0) 0x7f905c115f60 con 0x7f905c09a5e0 2021-05-25T12:08:14.296 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.254+0000 7f9050ff9700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:14.297 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.254+0000 7f9050ff9700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:14.297 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.254+0000 7f9050ff9700 10 monclient(hunting): my global_id is 204528 2021-05-25T12:08:14.297 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.254+0000 7f9050ff9700 1 -- 172.21.15.102:0/489441677 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f903c004c10 con 0x7f905c09a5e0 2021-05-25T12:08:14.297 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.254+0000 7f9050ff9700 1 -- 172.21.15.102:0/489441677 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1460037663 0 0) 0x7f905c1164a0 con 0x7f905c063dc0 2021-05-25T12:08:14.297 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.254+0000 7f9050ff9700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:14.298 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.254+0000 7f9050ff9700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:14.298 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.254+0000 7f9050ff9700 10 monclient(hunting): my global_id is 244133 2021-05-25T12:08:14.298 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.254+0000 7f9050ff9700 1 -- 172.21.15.102:0/489441677 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f905c115f60 con 0x7f905c063dc0 2021-05-25T12:08:14.298 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.254+0000 7f9050ff9700 1 -- 172.21.15.102:0/489441677 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3985495063 0 0) 0x7f905c116200 con 0x7f905c1325c0 2021-05-25T12:08:14.299 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.254+0000 7f9050ff9700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:14.299 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.254+0000 7f9050ff9700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:14.299 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.254+0000 7f9050ff9700 10 monclient(hunting): my global_id is 224389 2021-05-25T12:08:14.299 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.254+0000 7f9050ff9700 1 -- 172.21.15.102:0/489441677 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f905c1164a0 con 0x7f905c1325c0 2021-05-25T12:08:14.299 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.254+0000 7f9050ff9700 1 -- 172.21.15.102:0/489441677 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2755391458 0 0) 0x7f903c004c10 con 0x7f905c09a5e0 2021-05-25T12:08:14.300 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.254+0000 7f9050ff9700 1 -- 172.21.15.102:0/489441677 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f905c116200 con 0x7f905c09a5e0 2021-05-25T12:08:14.300 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.254+0000 7f9050ff9700 1 -- 172.21.15.102:0/489441677 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f9044001510 con 0x7f905c09a5e0 2021-05-25T12:08:14.300 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.254+0000 7f9050ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:14.300 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.254+0000 7f9050ff9700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:08:14.300 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.254+0000 7f9050ff9700 10 monclient(hunting): dump: 2021-05-25T12:08:14.301 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:14.301 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:14.301 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:14.301 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:14.302 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:14.302 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:14.302 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:14.302 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:14.302 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:14.303 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:14.303 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.254+0000 7f9050ff9700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:08:14.303 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.254+0000 7f9050ff9700 1 -- 172.21.15.102:0/489441677 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2271241386 0 0) 0x7f905c115f60 con 0x7f905c063dc0 2021-05-25T12:08:14.303 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.254+0000 7f9050ff9700 1 -- 172.21.15.102:0/489441677 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f903c004c10 con 0x7f905c063dc0 2021-05-25T12:08:14.303 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.254+0000 7f9050ff9700 1 -- 172.21.15.102:0/489441677 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f904c002910 con 0x7f905c063dc0 2021-05-25T12:08:14.304 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.254+0000 7f9050ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:14.304 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.254+0000 7f9050ff9700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2021-05-25T12:08:14.304 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.254+0000 7f9050ff9700 10 monclient(hunting): dump: 2021-05-25T12:08:14.304 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:14.304 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:14.305 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:14.305 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:14.305 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:14.306 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:14.306 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:14.306 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:14.306 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:14.307 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:14.307 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.254+0000 7f9050ff9700 1 -- 172.21.15.102:0/489441677 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3218258631 0 0) 0x7f905c1164a0 con 0x7f905c1325c0 2021-05-25T12:08:14.307 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.254+0000 7f9050ff9700 1 -- 172.21.15.102:0/489441677 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f905c115f60 con 0x7f905c1325c0 2021-05-25T12:08:14.307 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.254+0000 7f9050ff9700 1 -- 172.21.15.102:0/489441677 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f9058001db0 con 0x7f905c1325c0 2021-05-25T12:08:14.307 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.254+0000 7f9050ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:14.308 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.254+0000 7f9050ff9700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2021-05-25T12:08:14.308 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.254+0000 7f9050ff9700 10 monclient(hunting): dump: 2021-05-25T12:08:14.308 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:14.308 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:14.309 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:14.309 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:14.309 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:14.309 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:14.309 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:14.310 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:14.310 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:14.310 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:14.310 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.254+0000 7f9050ff9700 1 -- 172.21.15.102:0/489441677 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3985891424 0 0) 0x7f905c116200 con 0x7f905c09a5e0 2021-05-25T12:08:14.310 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.254+0000 7f9050ff9700 1 -- 172.21.15.102:0/489441677 >> v1:172.21.15.102:6790/0 conn(0x7f905c063dc0 legacy=0x7f905c12ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:14.311 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.254+0000 7f9050ff9700 1 -- 172.21.15.102:0/489441677 >> v1:172.21.15.181:6789/0 conn(0x7f905c1325c0 legacy=0x7f905c143a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:14.311 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.254+0000 7f9050ff9700 10 monclient: _finish_hunting 0 2021-05-25T12:08:14.311 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.254+0000 7f9050ff9700 1 monclient: found mon.a 2021-05-25T12:08:14.311 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.254+0000 7f9050ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:08:14.312 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.254+0000 7f9050ff9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:08:14.312 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.254+0000 7f9050ff9700 1 -- 172.21.15.102:0/489441677 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f905c093d50 con 0x7f905c09a5e0 2021-05-25T12:08:14.312 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.254+0000 7f9050ff9700 10 monclient: _finish_auth 0 2021-05-25T12:08:14.312 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.254+0000 7f9050ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:08:14.312 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.254+0000 7f9062d2d700 5 monclient: authenticate success, global_id 204528 2021-05-25T12:08:14.313 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.254+0000 7f9062d2d700 10 monclient: _renew_subs 2021-05-25T12:08:14.313 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.254+0000 7f9062d2d700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:08:14.313 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.254+0000 7f9062d2d700 1 -- 172.21.15.102:0/489441677 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f905c138470 con 0x7f905c09a5e0 2021-05-25T12:08:14.313 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.254+0000 7f9062d2d700 10 client.204528.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:08:14.313 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.254+0000 7f9062d2d700 10 monclient: _renew_subs 2021-05-25T12:08:14.314 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.254+0000 7f9062d2d700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:08:14.314 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.254+0000 7f9062d2d700 1 -- 172.21.15.102:0/489441677 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f905c094970 con 0x7f905c09a5e0 2021-05-25T12:08:14.314 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.254+0000 7f9062d2d700 1 librados: init done 2021-05-25T12:08:14.314 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.254+0000 7f9050ff9700 1 -- 172.21.15.102:0/489441677 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f90440047e0 con 0x7f905c09a5e0 2021-05-25T12:08:14.314 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.254+0000 7f9050ff9700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:08:14.315 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.254+0000 7f9050ff9700 1 -- 172.21.15.102:0/489441677 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f9044004b70 con 0x7f905c09a5e0 2021-05-25T12:08:14.315 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.254+0000 7f9050ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:14.315 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.254+0000 7f9050ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:08:14.315 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.254+0000 7f9052ffd700 4 set_mon_vals no callback set 2021-05-25T12:08:14.316 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.254+0000 7f9050ff9700 10 monclient: dump: 2021-05-25T12:08:14.316 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:14.316 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:14.316 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:14.317 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:14.317 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:14.317 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:14.317 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:14.318 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:14.318 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:14.318 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:14.318 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.255+0000 7f9062d2d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:08:14.318 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.255+0000 7f9050ff9700 1 -- 172.21.15.102:0/489441677 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f9044004cd0 con 0x7f905c09a5e0 2021-05-25T12:08:14.319 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.255+0000 7f9062d2d700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:08:14.319 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.255+0000 7f9062d2d700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:08:14.319 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.255+0000 7f9062d2d700 1 -- 172.21.15.102:0/489441677 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9034004db0 con 0x7f905c09a5e0 2021-05-25T12:08:14.319 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.255+0000 7f9050ff9700 1 -- 172.21.15.102:0/489441677 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(270..270 src has 1..270) v4 ==== 5832+0+0 (unknown 1409516269 0 0) 0x7f904403d740 con 0x7f905c09a5e0 2021-05-25T12:08:14.320 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.255+0000 7f9050ff9700 10 client.204528.objecter ms_dispatch 0x7f905c001550 osd_map(270..270 src has 1..270) v4 2021-05-25T12:08:14.320 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.255+0000 7f9050ff9700 3 client.204528.objecter handle_osd_map got epochs [270,270] > 0 2021-05-25T12:08:14.320 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.255+0000 7f9050ff9700 3 client.204528.objecter handle_osd_map decoding full epoch 270 2021-05-25T12:08:14.320 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.255+0000 7f9050ff9700 20 client.204528.objecter dump_active .. 0 homeless 2021-05-25T12:08:14.320 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.255+0000 7f9060ac9700 10 monclient: get_auth_request con 0x7f903c033500 auth_method 0 2021-05-25T12:08:14.321 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.256+0000 7f9050ff9700 10 client.204528.objecter ms_handle_connect 0x7f903c033500 2021-05-25T12:08:14.321 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.259+0000 7f9050ff9700 1 -- 172.21.15.102:0/489441677 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7f9044012e60 con 0x7f905c09a5e0 2021-05-25T12:08:14.321 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.259+0000 7f9050ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:08:14.321 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.259+0000 7f9050ff9700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:08:14.466 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.464+0000 7f9062d2d700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2021-05-25T12:08:14.466 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.464+0000 7f9062d2d700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2021-05-25T12:08:14.466 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.464+0000 7f9062d2d700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:08:14.467 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.464+0000 7f9062d2d700 1 -- 172.21.15.102:0/489441677 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f9034005060 con 0x7f905c09a5e0 2021-05-25T12:08:14.467 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.465+0000 7f9050ff9700 1 -- 172.21.15.102:0/489441677 <== mon.0 v1:172.21.15.102:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1342 (unknown 254746657 0 3666995367) 0x7f9044012a50 con 0x7f905c09a5e0 2021-05-25T12:08:14.467 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.465+0000 7f9050ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2021-05-25T12:08:14.467 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.465+0000 7f9050ff9700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:08:14.468 INFO:teuthology.orchestra.run.smithi102.stdout:{"election_epoch":68,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"b","quorum_age":2,"features":{"quorum_con":"4540138303579357183","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"20ee91b6-8935-4b98-b868-85c03789ccc4","modified":"2021-05-25T11:59:11.011522Z","created":"2021-05-25T11:59:11.011522Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6790","nonce":0}]},"addr":"172.21.15.102:6790/0","public_addr":"172.21.15.102:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2021-05-25T12:08:14.469 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.466+0000 7f9062d2d700 10 librados: watch_flush enter 2021-05-25T12:08:14.470 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.466+0000 7f9062d2d700 10 librados: watch_flush exit 2021-05-25T12:08:14.470 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.466+0000 7f9062d2d700 20 client.204528.objecter shutdown clearing up homeless session... 2021-05-25T12:08:14.470 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.466+0000 7f9062d2d700 10 client.204528.objecter successfully canceled tick 2021-05-25T12:08:14.470 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.466+0000 7f9062d2d700 1 -- 172.21.15.102:0/489441677 >> v1:172.21.15.181:6812/29582 conn(0x7f903c033500 legacy=0x7f903c0359c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:14.471 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.466+0000 7f9062d2d700 10 monclient: shutdown 2021-05-25T12:08:14.471 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.466+0000 7f9062d2d700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:08:14.471 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.466+0000 7f9062d2d700 1 -- 172.21.15.102:0/489441677 >> v1:172.21.15.102:6789/0 conn(0x7f905c09a5e0 legacy=0x7f905c1402d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:14.471 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.466+0000 7f9062d2d700 1 -- 172.21.15.102:0/489441677 shutdown_connections 2021-05-25T12:08:14.471 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.466+0000 7f9062d2d700 1 -- 172.21.15.102:0/489441677 >> 172.21.15.102:0/489441677 conn(0x7f905c09bc30 msgr2=0x7f905c12e8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:08:14.472 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.466+0000 7f9062d2d700 1 -- 172.21.15.102:0/489441677 shutdown_connections 2021-05-25T12:08:14.472 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.466+0000 7f9062d2d700 1 -- 172.21.15.102:0/489441677 wait complete. 2021-05-25T12:08:14.472 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.466+0000 7f9062d2d700 1 librados: shutdown 2021-05-25T12:08:14.479 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2021-05-25T12:08:14.479 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2021-05-25T12:08:14.479 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2021-05-25T12:08:14.589 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.588+0000 7f02b27c1700 10 monclient: get_monmap_and_config 2021-05-25T12:08:14.591 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.588+0000 7f02b27c1700 10 monclient: build_initial_monmap 2021-05-25T12:08:14.591 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.588+0000 7f02b27c1700 10 monclient: monmap: 2021-05-25T12:08:14.592 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:08:14.592 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:14.592 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:08:14.589196+0000 2021-05-25T12:08:14.593 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:08:14.589196+0000 2021-05-25T12:08:14.593 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:08:14.593 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:14.593 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:08:14.593 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:08:14.594 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:08:14.594 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:14.594 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.588+0000 7f02b27c1700 1 Processor -- start 2021-05-25T12:08:14.594 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.589+0000 7f02b27c1700 1 -- start start 2021-05-25T12:08:14.595 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.589+0000 7f02b27c1700 10 monclient: init 2021-05-25T12:08:14.609 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.589+0000 7f02b27c1700 10 monclient: _reopen_session rank -1 2021-05-25T12:08:14.610 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.589+0000 7f02b27c1700 10 monclient: _add_conns ranks=[1,2,0] 2021-05-25T12:08:14.610 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.589+0000 7f02b27c1700 10 monclient(hunting): picked mon.noname-c con 0x7f02ac096960 addr v1:172.21.15.181:6789/0 2021-05-25T12:08:14.611 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.589+0000 7f02b27c1700 10 monclient(hunting): picked mon.noname-b con 0x7f02ac069610 addr v1:172.21.15.102:6790/0 2021-05-25T12:08:14.611 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.589+0000 7f02b27c1700 10 monclient(hunting): picked mon.noname-a con 0x7f02ac1320f0 addr v1:172.21.15.102:6789/0 2021-05-25T12:08:14.612 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.589+0000 7f02b27c1700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02ac069ac0 con 0x7f02ac1320f0 2021-05-25T12:08:14.612 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.589+0000 7f02b27c1700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02ac096d60 con 0x7f02ac096960 2021-05-25T12:08:14.612 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.589+0000 7f02b27c1700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02ac063020 con 0x7f02ac069610 2021-05-25T12:08:14.613 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.589+0000 7f02b27c1700 10 monclient(hunting): _renew_subs 2021-05-25T12:08:14.613 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.589+0000 7f02b27c1700 10 monclient(hunting): authenticate will time out at 1295.186587s 2021-05-25T12:08:14.614 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.589+0000 7f02b0fbe700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f02ac069610 0x7f02ac09b1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:57270/0 (socket says 172.21.15.102:57270) 2021-05-25T12:08:14.614 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.589+0000 7f02b0fbe700 1 -- 172.21.15.102:0/2842986054 learned_addr learned my addr 172.21.15.102:0/2842986054 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:08:14.615 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.589+0000 7f029bfff700 1 -- 172.21.15.102:0/2842986054 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1713647351 0 0) 0x7f029c001110 con 0x7f02ac069610 2021-05-25T12:08:14.615 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.589+0000 7f029bfff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:14.615 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.589+0000 7f029bfff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:14.616 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.589+0000 7f029bfff700 10 monclient(hunting): my global_id is 244136 2021-05-25T12:08:14.616 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.589+0000 7f029bfff700 1 -- 172.21.15.102:0/2842986054 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0294003270 con 0x7f02ac069610 2021-05-25T12:08:14.616 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.589+0000 7f029bfff700 1 -- 172.21.15.102:0/2842986054 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1035624688 0 0) 0x7f02a8001490 con 0x7f02ac1320f0 2021-05-25T12:08:14.617 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.590+0000 7f029bfff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:14.617 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.590+0000 7f029bfff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:14.618 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.590+0000 7f029bfff700 10 monclient(hunting): my global_id is 204531 2021-05-25T12:08:14.618 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.590+0000 7f029bfff700 1 -- 172.21.15.102:0/2842986054 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f029c001110 con 0x7f02ac1320f0 2021-05-25T12:08:14.618 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.590+0000 7f029bfff700 1 -- 172.21.15.102:0/2842986054 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2087245665 0 0) 0x7f02a0001110 con 0x7f02ac096960 2021-05-25T12:08:14.619 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.590+0000 7f029bfff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:14.619 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.590+0000 7f029bfff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:14.619 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.590+0000 7f029bfff700 10 monclient(hunting): my global_id is 224392 2021-05-25T12:08:14.619 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.590+0000 7f029bfff700 1 -- 172.21.15.102:0/2842986054 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02a8001490 con 0x7f02ac096960 2021-05-25T12:08:14.620 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.590+0000 7f029bfff700 1 -- 172.21.15.102:0/2842986054 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2521639442 0 0) 0x7f0294003270 con 0x7f02ac069610 2021-05-25T12:08:14.620 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.590+0000 7f029bfff700 1 -- 172.21.15.102:0/2842986054 >> v1:172.21.15.181:6789/0 conn(0x7f02ac096960 legacy=0x7f02ac09aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:14.620 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.590+0000 7f029bfff700 1 -- 172.21.15.102:0/2842986054 >> v1:172.21.15.102:6789/0 conn(0x7f02ac1320f0 legacy=0x7f02ac1344e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:14.620 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.590+0000 7f029bfff700 10 monclient: _finish_hunting 0 2021-05-25T12:08:14.620 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.590+0000 7f029bfff700 1 monclient: found mon.noname-b 2021-05-25T12:08:14.621 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.590+0000 7f029bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:08:14.621 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.590+0000 7f029bfff700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T12:08:14.621 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.590+0000 7f029bfff700 1 -- 172.21.15.102:0/2842986054 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02ac09b940 con 0x7f02ac069610 2021-05-25T12:08:14.621 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.590+0000 7f029bfff700 10 monclient: _finish_auth 0 2021-05-25T12:08:14.622 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.590+0000 7f029bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:08:14.622 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.590+0000 7f029bfff700 1 -- 172.21.15.102:0/2842986054 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f029c001600 con 0x7f02ac069610 2021-05-25T12:08:14.622 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.590+0000 7f02b27c1700 5 monclient: authenticate success, global_id 244136 2021-05-25T12:08:14.622 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.590+0000 7f02b27c1700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:08:14.623 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.590+0000 7f029bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:14.623 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.590+0000 7f029bfff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:08:14.623 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.590+0000 7f029bfff700 10 monclient: dump: 2021-05-25T12:08:14.623 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:14.623 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:14.624 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:14.624 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:14.624 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:14.624 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:14.624 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:14.625 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:14.625 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:14.625 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:14.625 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.590+0000 7f02b27c1700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:08:14.626 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.590+0000 7f029bfff700 1 -- 172.21.15.102:0/2842986054 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f029c001860 con 0x7f02ac069610 2021-05-25T12:08:14.626 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.590+0000 7f029bfff700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:08:14.626 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.590+0000 7f029bfff700 1 -- 172.21.15.102:0/2842986054 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f029c001bf0 con 0x7f02ac069610 2021-05-25T12:08:14.626 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.590+0000 7f02b27c1700 10 monclient: get_monmap_and_config success 2021-05-25T12:08:14.626 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.590+0000 7f02b27c1700 4 set_mon_vals no callback set 2021-05-25T12:08:14.627 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.590+0000 7f029bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:14.627 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.590+0000 7f029bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:08:14.627 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.590+0000 7f029bfff700 10 monclient: dump: 2021-05-25T12:08:14.627 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:14.628 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:14.628 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:14.628 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:14.628 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:14.628 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:14.629 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:14.629 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:14.629 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:14.629 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:14.630 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.590+0000 7f02b27c1700 10 monclient: shutdown 2021-05-25T12:08:14.630 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.590+0000 7f02b27c1700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:08:14.630 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.590+0000 7f02b27c1700 1 -- 172.21.15.102:0/2842986054 >> v1:172.21.15.102:6790/0 conn(0x7f02ac069610 legacy=0x7f02ac09b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:14.630 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.590+0000 7f02b27c1700 1 -- 172.21.15.102:0/2842986054 shutdown_connections 2021-05-25T12:08:14.631 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.590+0000 7f02b27c1700 1 -- 172.21.15.102:0/2842986054 >> 172.21.15.102:0/2842986054 conn(0x7f02ac09c8c0 msgr2=0x7f02ac09ccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:08:14.631 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.590+0000 7f02b27c1700 1 -- 172.21.15.102:0/2842986054 shutdown_connections 2021-05-25T12:08:14.631 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.590+0000 7f02b27c1700 1 -- 172.21.15.102:0/2842986054 wait complete. 2021-05-25T12:08:14.631 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.591+0000 7f02b27c1700 10 monclient: build_initial_monmap 2021-05-25T12:08:14.632 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.591+0000 7f02b27c1700 10 monclient: monmap: 2021-05-25T12:08:14.632 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:08:14.632 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:08:14.632 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:08:14.633 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:08:14.633 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:08:14.633 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:08:14.633 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:08:14.633 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:08:14.634 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:08:14.634 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:14.634 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.591+0000 7f02b27c1700 1 librados: starting msgr at 2021-05-25T12:08:14.634 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.591+0000 7f02b27c1700 1 librados: starting objecter 2021-05-25T12:08:14.635 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.591+0000 7f02b27c1700 1 Processor -- start 2021-05-25T12:08:14.635 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.591+0000 7f02b27c1700 1 -- start start 2021-05-25T12:08:14.635 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.591+0000 7f02b27c1700 1 librados: setting wanted keys 2021-05-25T12:08:14.635 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.591+0000 7f02b27c1700 1 librados: calling monclient init 2021-05-25T12:08:14.635 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.591+0000 7f02b27c1700 10 monclient: init 2021-05-25T12:08:14.636 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.592+0000 7f02b27c1700 10 monclient: _reopen_session rank -1 2021-05-25T12:08:14.636 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.592+0000 7f02b27c1700 10 monclient: _add_conns ranks=[0,1,2] 2021-05-25T12:08:14.636 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.592+0000 7f02b27c1700 10 monclient(hunting): picked mon.noname-a con 0x7f02ac069610 addr v1:172.21.15.102:6789/0 2021-05-25T12:08:14.637 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.592+0000 7f02b27c1700 10 monclient(hunting): picked mon.noname-b con 0x7f02ac096960 addr v1:172.21.15.181:6789/0 2021-05-25T12:08:14.637 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.592+0000 7f02b27c1700 10 monclient(hunting): picked mon.noname-c con 0x7f02ac1320f0 addr v1:172.21.15.102:6790/0 2021-05-25T12:08:14.637 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.592+0000 7f02b27c1700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02ac12e360 con 0x7f02ac069610 2021-05-25T12:08:14.637 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.592+0000 7f02b27c1700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02ac12aaf0 con 0x7f02ac096960 2021-05-25T12:08:14.638 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.592+0000 7f02b27c1700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02ac114b50 con 0x7f02ac1320f0 2021-05-25T12:08:14.638 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.592+0000 7f02b27c1700 10 monclient(hunting): _renew_subs 2021-05-25T12:08:14.638 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.592+0000 7f02b27c1700 10 monclient(hunting): authenticate will time out at 1295.189259s 2021-05-25T12:08:14.639 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.592+0000 7f02b17bf700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f02ac069610 0x7f02ac098ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:34164/0 (socket says 172.21.15.102:34164) 2021-05-25T12:08:14.639 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.592+0000 7f02b1fc0700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f02ac1320f0 0x7f02ac142c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:57278/0 (socket says 172.21.15.102:57278) 2021-05-25T12:08:14.639 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.592+0000 7f02b17bf700 1 -- 172.21.15.102:0/2756968631 learned_addr learned my addr 172.21.15.102:0/2756968631 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:08:14.640 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.592+0000 7f02997fa700 10 client.?.objecter ms_handle_connect 0x7f02ac069610 2021-05-25T12:08:14.640 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.592+0000 7f02997fa700 10 client.?.objecter resend_mon_ops 2021-05-25T12:08:14.640 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.592+0000 7f02997fa700 10 client.?.objecter ms_handle_connect 0x7f02ac1320f0 2021-05-25T12:08:14.640 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.592+0000 7f02997fa700 10 client.?.objecter resend_mon_ops 2021-05-25T12:08:14.641 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.592+0000 7f02997fa700 1 -- 172.21.15.102:0/2756968631 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 451021804 0 0) 0x7f02ac12e360 con 0x7f02ac069610 2021-05-25T12:08:14.641 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.592+0000 7f02997fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:14.641 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.592+0000 7f02997fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:14.641 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.592+0000 7f02997fa700 10 monclient(hunting): my global_id is 204534 2021-05-25T12:08:14.642 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.592+0000 7f02997fa700 1 -- 172.21.15.102:0/2756968631 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f028c004c10 con 0x7f02ac069610 2021-05-25T12:08:14.642 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.592+0000 7f02997fa700 10 client.?.objecter ms_handle_connect 0x7f02ac096960 2021-05-25T12:08:14.642 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.592+0000 7f02997fa700 10 client.?.objecter resend_mon_ops 2021-05-25T12:08:14.642 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.592+0000 7f02997fa700 1 -- 172.21.15.102:0/2756968631 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2764623726 0 0) 0x7f02ac114b50 con 0x7f02ac1320f0 2021-05-25T12:08:14.643 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.592+0000 7f02997fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:14.643 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.592+0000 7f02997fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:14.643 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.592+0000 7f02997fa700 10 monclient(hunting): my global_id is 244139 2021-05-25T12:08:14.643 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.592+0000 7f02997fa700 1 -- 172.21.15.102:0/2756968631 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02ac12e360 con 0x7f02ac1320f0 2021-05-25T12:08:14.643 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.592+0000 7f02997fa700 1 -- 172.21.15.102:0/2756968631 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1184284025 0 0) 0x7f02ac12aaf0 con 0x7f02ac096960 2021-05-25T12:08:14.644 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.592+0000 7f02997fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:14.644 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.592+0000 7f02997fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:14.644 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.592+0000 7f02997fa700 10 monclient(hunting): my global_id is 224395 2021-05-25T12:08:14.644 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.592+0000 7f02997fa700 1 -- 172.21.15.102:0/2756968631 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02ac114b50 con 0x7f02ac096960 2021-05-25T12:08:14.645 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.592+0000 7f02997fa700 1 -- 172.21.15.102:0/2756968631 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2804855699 0 0) 0x7f02ac12e360 con 0x7f02ac1320f0 2021-05-25T12:08:14.645 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.592+0000 7f02997fa700 1 -- 172.21.15.102:0/2756968631 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f02ac12aaf0 con 0x7f02ac1320f0 2021-05-25T12:08:14.645 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.592+0000 7f02997fa700 1 -- 172.21.15.102:0/2756968631 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f02a8001e00 con 0x7f02ac1320f0 2021-05-25T12:08:14.645 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.592+0000 7f02997fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:14.645 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.592+0000 7f02997fa700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:08:14.646 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.592+0000 7f02997fa700 10 monclient(hunting): dump: 2021-05-25T12:08:14.646 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:14.646 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:14.646 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:14.647 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:14.647 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:14.647 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:14.647 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:14.648 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:14.648 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:14.648 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:14.648 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.592+0000 7f02997fa700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:08:14.648 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.592+0000 7f02997fa700 1 -- 172.21.15.102:0/2756968631 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 775644450 0 0) 0x7f028c004c10 con 0x7f02ac069610 2021-05-25T12:08:14.649 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.592+0000 7f02997fa700 1 -- 172.21.15.102:0/2756968631 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f02ac12e360 con 0x7f02ac069610 2021-05-25T12:08:14.649 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.592+0000 7f02997fa700 1 -- 172.21.15.102:0/2756968631 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f02a0001830 con 0x7f02ac069610 2021-05-25T12:08:14.649 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.592+0000 7f02997fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:14.649 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.592+0000 7f02997fa700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2021-05-25T12:08:14.649 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.592+0000 7f02997fa700 10 monclient(hunting): dump: 2021-05-25T12:08:14.650 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:14.650 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:14.650 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:14.650 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:14.651 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:14.651 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:14.651 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:14.651 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:14.651 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:14.652 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:14.652 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.592+0000 7f02997fa700 1 -- 172.21.15.102:0/2756968631 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3487448236 0 0) 0x7f02ac12aaf0 con 0x7f02ac1320f0 2021-05-25T12:08:14.652 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.592+0000 7f02997fa700 1 -- 172.21.15.102:0/2756968631 >> v1:172.21.15.181:6789/0 conn(0x7f02ac096960 legacy=0x7f02ac12a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:14.652 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.593+0000 7f02997fa700 1 -- 172.21.15.102:0/2756968631 >> v1:172.21.15.102:6789/0 conn(0x7f02ac069610 legacy=0x7f02ac098ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:14.653 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.593+0000 7f02997fa700 10 monclient: _finish_hunting 0 2021-05-25T12:08:14.653 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.593+0000 7f02997fa700 1 monclient: found mon.c 2021-05-25T12:08:14.653 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.593+0000 7f02997fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:08:14.653 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.593+0000 7f02997fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:08:14.653 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.593+0000 7f02997fa700 1 -- 172.21.15.102:0/2756968631 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02ac114df0 con 0x7f02ac1320f0 2021-05-25T12:08:14.654 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.593+0000 7f02997fa700 10 monclient: _finish_auth 0 2021-05-25T12:08:14.654 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.593+0000 7f02997fa700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:08:14.654 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.593+0000 7f02b27c1700 5 monclient: authenticate success, global_id 244139 2021-05-25T12:08:14.655 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.593+0000 7f02b27c1700 10 monclient: _renew_subs 2021-05-25T12:08:14.655 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.593+0000 7f02b27c1700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:08:14.655 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.593+0000 7f02b27c1700 1 -- 172.21.15.102:0/2756968631 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02ac113920 con 0x7f02ac1320f0 2021-05-25T12:08:14.655 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.593+0000 7f02b27c1700 10 client.244139.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:08:14.656 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.593+0000 7f02b27c1700 10 monclient: _renew_subs 2021-05-25T12:08:14.656 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.593+0000 7f02b27c1700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:08:14.656 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.593+0000 7f02b27c1700 1 -- 172.21.15.102:0/2756968631 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f02ac099780 con 0x7f02ac1320f0 2021-05-25T12:08:14.656 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.593+0000 7f02b27c1700 1 librados: init done 2021-05-25T12:08:14.657 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.593+0000 7f02997fa700 1 -- 172.21.15.102:0/2756968631 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f02a8002370 con 0x7f02ac1320f0 2021-05-25T12:08:14.657 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.593+0000 7f02997fa700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:08:14.657 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.593+0000 7f02997fa700 1 -- 172.21.15.102:0/2756968631 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f02a8002680 con 0x7f02ac1320f0 2021-05-25T12:08:14.657 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.593+0000 7f02997fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:14.657 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.593+0000 7f02997fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:08:14.658 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.593+0000 7f02997fa700 10 monclient: dump: 2021-05-25T12:08:14.658 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:14.658 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:14.659 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:14.659 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:14.659 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:14.659 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:14.660 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:14.660 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:14.660 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:14.660 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:14.661 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.593+0000 7f029b7fe700 4 set_mon_vals no callback set 2021-05-25T12:08:14.661 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.593+0000 7f02997fa700 1 -- 172.21.15.102:0/2756968631 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f02a80028a0 con 0x7f02ac1320f0 2021-05-25T12:08:14.661 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.593+0000 7f02b27c1700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:08:14.661 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.593+0000 7f02b27c1700 1 -- 172.21.15.102:0/2756968631 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f02ac113b10 con 0x7f02ac12e4f0 2021-05-25T12:08:14.662 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.593+0000 7f02997fa700 1 -- 172.21.15.102:0/2756968631 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(270..270 src has 1..270) v4 ==== 5832+0+0 (unknown 1409516269 0 0) 0x7f02a803db60 con 0x7f02ac1320f0 2021-05-25T12:08:14.662 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.593+0000 7f02997fa700 10 client.244139.objecter ms_dispatch 0x7f02ac001550 osd_map(270..270 src has 1..270) v4 2021-05-25T12:08:14.662 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.593+0000 7f02997fa700 3 client.244139.objecter handle_osd_map got epochs [270,270] > 0 2021-05-25T12:08:14.662 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.593+0000 7f02997fa700 3 client.244139.objecter handle_osd_map decoding full epoch 270 2021-05-25T12:08:14.663 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.594+0000 7f02997fa700 20 client.244139.objecter dump_active .. 0 homeless 2021-05-25T12:08:14.663 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.594+0000 7f02997fa700 10 client.244139.objecter ms_handle_connect 0x7f02ac12e4f0 2021-05-25T12:08:14.663 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.594+0000 7f02997fa700 10 client.244139.objecter resend_mon_ops 2021-05-25T12:08:14.663 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.594+0000 7f02997fa700 1 -- 172.21.15.102:0/2756968631 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 824814317 0 0) 0x7f02ac12e360 con 0x7f02ac12e4f0 2021-05-25T12:08:14.663 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.594+0000 7f02997fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:14.664 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.594+0000 7f02997fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:14.664 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.594+0000 7f02997fa700 10 monclient(hunting): my global_id is 204537 2021-05-25T12:08:14.664 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.594+0000 7f02997fa700 1 -- 172.21.15.102:0/2756968631 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02ac12aaf0 con 0x7f02ac12e4f0 2021-05-25T12:08:14.664 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.594+0000 7f02b0fbe700 10 monclient: get_auth_request con 0x7f028c033470 auth_method 0 2021-05-25T12:08:14.664 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.594+0000 7f02997fa700 1 -- 172.21.15.102:0/2756968631 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3337350684 0 0) 0x7f02ac12aaf0 con 0x7f02ac12e4f0 2021-05-25T12:08:14.665 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.594+0000 7f02997fa700 1 -- 172.21.15.102:0/2756968631 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f02ac12e360 con 0x7f02ac12e4f0 2021-05-25T12:08:14.665 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.594+0000 7f02997fa700 1 -- 172.21.15.102:0/2756968631 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f02a0001c80 con 0x7f02ac12e4f0 2021-05-25T12:08:14.665 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.594+0000 7f02997fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:14.665 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.594+0000 7f02997fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:08:14.666 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.594+0000 7f02997fa700 10 monclient: dump: 2021-05-25T12:08:14.666 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:14.666 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:14.666 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:14.666 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:14.667 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:14.667 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:14.667 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:14.667 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:14.668 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:14.668 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:14.668 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.594+0000 7f02997fa700 1 -- 172.21.15.102:0/2756968631 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3470078637 0 0) 0x7f02ac12e360 con 0x7f02ac12e4f0 2021-05-25T12:08:14.668 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.594+0000 7f02997fa700 1 -- 172.21.15.102:0/2756968631 --> v1:172.21.15.102:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f02ac059730 con 0x7f02ac12e4f0 2021-05-25T12:08:14.668 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.594+0000 7f02997fa700 10 client.244139.objecter ms_handle_connect 0x7f028c033470 2021-05-25T12:08:14.669 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.594+0000 7f02997fa700 1 -- 172.21.15.102:0/2756968631 <== mon.0 v1:172.21.15.102:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7f02a0005510 con 0x7f02ac12e4f0 2021-05-25T12:08:14.669 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.594+0000 7f02997fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:08:14.669 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.594+0000 7f02997fa700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:08:14.669 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.594+0000 7f02997fa700 1 -- 172.21.15.102:0/2756968631 >> v1:172.21.15.102:6789/0 conn(0x7f02ac12e4f0 legacy=0x7f02ac12e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:14.669 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.594+0000 7f02997fa700 1 -- 172.21.15.102:0/2756968631 >> v1:172.21.15.102:6789/0 conn(0x7f02ac12e4f0 legacy=0x7f02ac12e8d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:08:14.670 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.601+0000 7f02b27c1700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:08:14.670 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.601+0000 7f02b27c1700 1 -- 172.21.15.102:0/2756968631 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f02ac113b10 con 0x7f02ac027ce0 2021-05-25T12:08:14.670 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.601+0000 7f02997fa700 10 client.244139.objecter ms_handle_connect 0x7f02ac027ce0 2021-05-25T12:08:14.670 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.601+0000 7f02997fa700 10 client.244139.objecter resend_mon_ops 2021-05-25T12:08:14.671 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.601+0000 7f02997fa700 1 -- 172.21.15.102:0/2756968631 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2084239242 0 0) 0x7f02a8001490 con 0x7f02ac027ce0 2021-05-25T12:08:14.671 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.601+0000 7f02997fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:14.671 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.601+0000 7f02997fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:14.671 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.601+0000 7f02997fa700 10 monclient(hunting): my global_id is 204540 2021-05-25T12:08:14.672 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.601+0000 7f02997fa700 1 -- 172.21.15.102:0/2756968631 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02ac12e360 con 0x7f02ac027ce0 2021-05-25T12:08:14.672 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.601+0000 7f02997fa700 1 -- 172.21.15.102:0/2756968631 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3874941851 0 0) 0x7f02ac12e360 con 0x7f02ac027ce0 2021-05-25T12:08:14.672 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.601+0000 7f02997fa700 1 -- 172.21.15.102:0/2756968631 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f02a8001490 con 0x7f02ac027ce0 2021-05-25T12:08:14.672 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.601+0000 7f02997fa700 1 -- 172.21.15.102:0/2756968631 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f02a00024c0 con 0x7f02ac027ce0 2021-05-25T12:08:14.672 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.601+0000 7f02997fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:14.673 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.601+0000 7f02997fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:08:14.673 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.601+0000 7f02997fa700 10 monclient: dump: 2021-05-25T12:08:14.673 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:14.673 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:14.673 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:14.674 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:14.674 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:14.674 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:14.675 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:14.675 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:14.675 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:14.675 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:14.675 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.601+0000 7f02997fa700 1 -- 172.21.15.102:0/2756968631 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2124381770 0 0) 0x7f02a8001490 con 0x7f02ac027ce0 2021-05-25T12:08:14.676 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.601+0000 7f02997fa700 1 -- 172.21.15.102:0/2756968631 --> v1:172.21.15.102:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f02ac099a30 con 0x7f02ac027ce0 2021-05-25T12:08:14.676 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.602+0000 7f02997fa700 1 -- 172.21.15.102:0/2756968631 <== mon.0 v1:172.21.15.102:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3940 (unknown 0 0 355194126) 0x7f02a0002930 con 0x7f02ac027ce0 2021-05-25T12:08:14.676 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.602+0000 7f02997fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:08:14.676 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.602+0000 7f02997fa700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:08:14.676 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.602+0000 7f02997fa700 1 -- 172.21.15.102:0/2756968631 >> v1:172.21.15.102:6789/0 conn(0x7f02ac027ce0 legacy=0x7f02ac12efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:14.677 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.602+0000 7f02997fa700 1 -- 172.21.15.102:0/2756968631 >> v1:172.21.15.102:6789/0 conn(0x7f02ac027ce0 legacy=0x7f02ac12efe0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:08:14.677 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:08:14.678 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:08:14.678 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:08:14.678 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "peon", 2021-05-25T12:08:14.678 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 68, 2021-05-25T12:08:14.678 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:08:14.679 INFO:teuthology.orchestra.run.smithi102.stdout: 0, 2021-05-25T12:08:14.679 INFO:teuthology.orchestra.run.smithi102.stdout: 1, 2021-05-25T12:08:14.679 INFO:teuthology.orchestra.run.smithi102.stdout: 2 2021-05-25T12:08:14.679 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:08:14.680 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 2, 2021-05-25T12:08:14.680 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:08:14.680 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:08:14.680 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:08:14.680 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:08:14.681 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:08:14.681 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:08:14.681 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:08:14.681 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:08:14.682 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:08:14.682 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:08:14.682 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:08:14.682 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:08:14.683 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:08:14.683 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:08:14.683 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:08:14.683 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:08:14.684 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:08:14.684 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:08:14.684 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:08:14.684 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:08:14.684 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:08:14.685 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:08:14.685 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:08:14.685 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:08:14.685 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:14.685 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:14.686 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:08:14.686 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:08:14.686 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:08:14.686 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:08:14.687 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:08:14.687 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:08:14.687 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:08:14.687 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:08:14.687 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:08:14.688 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:08:14.688 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:08:14.688 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:08:14.688 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:08:14.689 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:08:14.689 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:08:14.689 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:08:14.689 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:08:14.689 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:08:14.690 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:08:14.690 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:08:14.690 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:08:14.690 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:08:14.690 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:08:14.691 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:08:14.691 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:08:14.691 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:08:14.691 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:14.692 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:08:14.692 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:14.692 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:08:14.692 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:08:14.692 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:08:14.693 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:08:14.693 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:14.693 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:08:14.693 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:08:14.693 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:08:14.694 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:14.694 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:14.694 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:14.694 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:08:14.695 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:08:14.695 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:08:14.695 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:08:14.695 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:08:14.695 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:14.696 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:14.696 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:08:14.696 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:08:14.696 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:08:14.696 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:08:14.697 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:14.697 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:08:14.697 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:08:14.697 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:08:14.697 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:14.698 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:14.698 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:14.698 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:08:14.698 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:08:14.699 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:08:14.699 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:08:14.699 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:08:14.699 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:14.699 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:14.700 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:08:14.700 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:08:14.700 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:08:14.700 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:08:14.700 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:14.701 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:08:14.701 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:08:14.701 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:08:14.701 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:14.701 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:14.702 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:14.702 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:08:14.702 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:08:14.702 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:08:14.703 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:08:14.703 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:08:14.703 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:14.703 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:14.704 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:14.704 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:08:14.704 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:08:14.704 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:14.704 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:08:14.705 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:08:14.705 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:08:14.705 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:14.705 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:08:14.705 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": [ 2021-05-25T12:08:14.706 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:14.706 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:08:14.706 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:08:14.706 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 6 2021-05-25T12:08:14.707 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:14.707 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:08:14.707 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:08:14.707 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:14.707 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:08:14.708 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:08:14.708 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 3 2021-05-25T12:08:14.708 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:14.708 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:14.708 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:14.709 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:08:14.709 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:08:14.711 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.602+0000 7f02b27c1700 10 librados: watch_flush enter 2021-05-25T12:08:14.712 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.602+0000 7f02b27c1700 10 librados: watch_flush exit 2021-05-25T12:08:14.712 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.602+0000 7f02b27c1700 20 client.244139.objecter shutdown clearing up homeless session... 2021-05-25T12:08:14.712 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.602+0000 7f02b27c1700 10 client.244139.objecter successfully canceled tick 2021-05-25T12:08:14.712 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.602+0000 7f02b27c1700 1 -- 172.21.15.102:0/2756968631 >> v1:172.21.15.181:6812/29582 conn(0x7f028c033470 legacy=0x7f028c035930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:14.713 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.602+0000 7f02b27c1700 10 monclient: shutdown 2021-05-25T12:08:14.713 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.602+0000 7f02b27c1700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:08:14.713 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.602+0000 7f02b27c1700 1 -- 172.21.15.102:0/2756968631 >> v1:172.21.15.102:6790/0 conn(0x7f02ac1320f0 legacy=0x7f02ac142c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:14.713 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.602+0000 7f02b1fc0700 1 -- 172.21.15.102:0/2756968631 reap_dead start 2021-05-25T12:08:14.713 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.602+0000 7f02b1fc0700 1 -- 172.21.15.102:0/2756968631 reap_dead start 2021-05-25T12:08:14.714 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.602+0000 7f02b27c1700 1 -- 172.21.15.102:0/2756968631 shutdown_connections 2021-05-25T12:08:14.714 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.602+0000 7f02b27c1700 1 -- 172.21.15.102:0/2756968631 >> 172.21.15.102:0/2756968631 conn(0x7f02ac09c8c0 msgr2=0x7f02ac12d420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:08:14.714 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.602+0000 7f02b27c1700 1 -- 172.21.15.102:0/2756968631 shutdown_connections 2021-05-25T12:08:14.714 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.602+0000 7f02b27c1700 1 -- 172.21.15.102:0/2756968631 wait complete. 2021-05-25T12:08:14.714 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.602+0000 7f02b27c1700 1 librados: shutdown 2021-05-25T12:08:14.715 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2021-05-25T12:08:14.843 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.824+0000 7f506d60d700 10 monclient: get_monmap_and_config 2021-05-25T12:08:14.844 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.824+0000 7f506d60d700 10 monclient: build_initial_monmap 2021-05-25T12:08:14.844 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.824+0000 7f506d60d700 10 monclient: monmap: 2021-05-25T12:08:14.844 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:08:14.845 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:14.845 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:08:14.825389+0000 2021-05-25T12:08:14.845 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:08:14.825389+0000 2021-05-25T12:08:14.846 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:08:14.846 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:14.846 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:08:14.847 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:08:14.847 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:08:14.847 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:14.847 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.824+0000 7f506d60d700 1 Processor -- start 2021-05-25T12:08:14.847 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.825+0000 7f506d60d700 1 -- start start 2021-05-25T12:08:14.848 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.825+0000 7f506d60d700 10 monclient: init 2021-05-25T12:08:14.848 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.825+0000 7f506d60d700 10 monclient: _reopen_session rank -1 2021-05-25T12:08:14.848 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.825+0000 7f506d60d700 10 monclient: _add_conns ranks=[2,1,0] 2021-05-25T12:08:14.848 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.825+0000 7f506d60d700 10 monclient(hunting): picked mon.noname-b con 0x7f5068093fc0 addr v1:172.21.15.102:6790/0 2021-05-25T12:08:14.848 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.825+0000 7f506d60d700 10 monclient(hunting): picked mon.noname-c con 0x7f50680692d0 addr v1:172.21.15.181:6789/0 2021-05-25T12:08:14.849 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.825+0000 7f506d60d700 10 monclient(hunting): picked mon.noname-a con 0x7f506812f130 addr v1:172.21.15.102:6789/0 2021-05-25T12:08:14.849 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.825+0000 7f506d60d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5068097160 con 0x7f506812f130 2021-05-25T12:08:14.849 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.825+0000 7f506d60d700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5068092fe0 con 0x7f50680692d0 2021-05-25T12:08:14.849 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.825+0000 7f506d60d700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5068093280 con 0x7f5068093fc0 2021-05-25T12:08:14.850 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.825+0000 7f506d60d700 10 monclient(hunting): _renew_subs 2021-05-25T12:08:14.850 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.825+0000 7f506d60d700 10 monclient(hunting): authenticate will time out at 1295.422491s 2021-05-25T12:08:14.850 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.825+0000 7f50677fe700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f506812f130 0x7f5068131520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:34178/0 (socket says 172.21.15.102:34178) 2021-05-25T12:08:14.850 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.825+0000 7f5066ffd700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f5068093fc0 0x7f5068096340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:57290/0 (socket says 172.21.15.102:57290) 2021-05-25T12:08:14.851 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.825+0000 7f50677fe700 1 -- 172.21.15.102:0/2860612837 learned_addr learned my addr 172.21.15.102:0/2860612837 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:08:14.851 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.825+0000 7f5065ffb700 1 -- 172.21.15.102:0/2860612837 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3592804660 0 0) 0x7f5068097160 con 0x7f506812f130 2021-05-25T12:08:14.851 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.825+0000 7f5065ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:14.851 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.825+0000 7f5065ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:14.851 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.825+0000 7f5065ffb700 10 monclient(hunting): my global_id is 204543 2021-05-25T12:08:14.852 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.825+0000 7f5065ffb700 1 -- 172.21.15.102:0/2860612837 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5054003270 con 0x7f506812f130 2021-05-25T12:08:14.852 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.825+0000 7f5065ffb700 1 -- 172.21.15.102:0/2860612837 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2312203819 0 0) 0x7f5068093280 con 0x7f5068093fc0 2021-05-25T12:08:14.852 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.825+0000 7f5065ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:14.852 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.825+0000 7f5065ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:14.853 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.825+0000 7f5065ffb700 10 monclient(hunting): my global_id is 244142 2021-05-25T12:08:14.853 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.825+0000 7f5065ffb700 1 -- 172.21.15.102:0/2860612837 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5068097160 con 0x7f5068093fc0 2021-05-25T12:08:14.853 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.826+0000 7f5065ffb700 1 -- 172.21.15.102:0/2860612837 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2763663097 0 0) 0x7f5068092fe0 con 0x7f50680692d0 2021-05-25T12:08:14.853 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.826+0000 7f5065ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:14.854 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.826+0000 7f5065ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:14.854 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.826+0000 7f5065ffb700 10 monclient(hunting): my global_id is 224398 2021-05-25T12:08:14.854 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.826+0000 7f5065ffb700 1 -- 172.21.15.102:0/2860612837 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5068093280 con 0x7f50680692d0 2021-05-25T12:08:14.854 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.826+0000 7f5065ffb700 1 -- 172.21.15.102:0/2860612837 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 132113663 0 0) 0x7f5068097160 con 0x7f5068093fc0 2021-05-25T12:08:14.854 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.826+0000 7f5065ffb700 1 -- 172.21.15.102:0/2860612837 >> v1:172.21.15.181:6789/0 conn(0x7f50680692d0 legacy=0x7f5068096a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:14.855 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.826+0000 7f5065ffb700 1 -- 172.21.15.102:0/2860612837 >> v1:172.21.15.102:6789/0 conn(0x7f506812f130 legacy=0x7f5068131520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:14.855 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.826+0000 7f5065ffb700 10 monclient: _finish_hunting 0 2021-05-25T12:08:14.855 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.826+0000 7f5065ffb700 1 monclient: found mon.noname-b 2021-05-25T12:08:14.855 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.826+0000 7f5065ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:08:14.856 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.826+0000 7f5065ffb700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T12:08:14.856 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.826+0000 7f5065ffb700 1 -- 172.21.15.102:0/2860612837 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5068093520 con 0x7f5068093fc0 2021-05-25T12:08:14.856 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.826+0000 7f5065ffb700 10 monclient: _finish_auth 0 2021-05-25T12:08:14.856 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.826+0000 7f5065ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:08:14.857 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.826+0000 7f5065ffb700 1 -- 172.21.15.102:0/2860612837 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f505c0015f0 con 0x7f5068093fc0 2021-05-25T12:08:14.857 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.826+0000 7f506d60d700 5 monclient: authenticate success, global_id 244142 2021-05-25T12:08:14.857 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.826+0000 7f506d60d700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:08:14.857 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.826+0000 7f5065ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:14.857 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.826+0000 7f5065ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:08:14.858 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.826+0000 7f5065ffb700 10 monclient: dump: 2021-05-25T12:08:14.858 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:14.858 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:14.858 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:14.859 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:14.859 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:14.859 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:14.859 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:14.860 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:14.860 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:14.860 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:14.860 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.826+0000 7f506d60d700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:08:14.861 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.826+0000 7f5065ffb700 1 -- 172.21.15.102:0/2860612837 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f505c001890 con 0x7f5068093fc0 2021-05-25T12:08:14.861 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.826+0000 7f5065ffb700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:08:14.861 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.826+0000 7f5065ffb700 1 -- 172.21.15.102:0/2860612837 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f505c001c20 con 0x7f5068093fc0 2021-05-25T12:08:14.861 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.826+0000 7f506d60d700 10 monclient: get_monmap_and_config success 2021-05-25T12:08:14.862 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.826+0000 7f506d60d700 4 set_mon_vals no callback set 2021-05-25T12:08:14.862 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.826+0000 7f5065ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:14.862 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.826+0000 7f5065ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:08:14.862 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.826+0000 7f5065ffb700 10 monclient: dump: 2021-05-25T12:08:14.863 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:14.863 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:14.863 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:14.863 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:14.863 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:14.864 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:14.864 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:14.864 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:14.864 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:14.865 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:14.865 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.826+0000 7f506d60d700 10 monclient: shutdown 2021-05-25T12:08:14.865 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.826+0000 7f506d60d700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:08:14.865 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.826+0000 7f506d60d700 1 -- 172.21.15.102:0/2860612837 >> v1:172.21.15.102:6790/0 conn(0x7f5068093fc0 legacy=0x7f5068096340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:14.865 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.826+0000 7f506d60d700 1 -- 172.21.15.102:0/2860612837 shutdown_connections 2021-05-25T12:08:14.866 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.826+0000 7f506d60d700 1 -- 172.21.15.102:0/2860612837 >> 172.21.15.102:0/2860612837 conn(0x7f506809b370 msgr2=0x7f5068097430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:08:14.866 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.826+0000 7f506d60d700 1 -- 172.21.15.102:0/2860612837 shutdown_connections 2021-05-25T12:08:14.866 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.826+0000 7f506d60d700 1 -- 172.21.15.102:0/2860612837 wait complete. 2021-05-25T12:08:14.866 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.827+0000 7f506d60d700 10 monclient: build_initial_monmap 2021-05-25T12:08:14.867 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.827+0000 7f506d60d700 10 monclient: monmap: 2021-05-25T12:08:14.867 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:08:14.867 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:08:14.867 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:08:14.868 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:08:14.868 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:08:14.868 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:08:14.868 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:08:14.868 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:08:14.869 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:08:14.869 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:14.869 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.827+0000 7f506d60d700 1 librados: starting msgr at 2021-05-25T12:08:14.869 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.827+0000 7f506d60d700 1 librados: starting objecter 2021-05-25T12:08:14.870 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.827+0000 7f506d60d700 1 Processor -- start 2021-05-25T12:08:14.870 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.827+0000 7f506d60d700 1 -- start start 2021-05-25T12:08:14.870 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.827+0000 7f506d60d700 1 librados: setting wanted keys 2021-05-25T12:08:14.870 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.827+0000 7f506d60d700 1 librados: calling monclient init 2021-05-25T12:08:14.870 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.827+0000 7f506d60d700 10 monclient: init 2021-05-25T12:08:14.871 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.828+0000 7f506d60d700 10 monclient: _reopen_session rank -1 2021-05-25T12:08:14.871 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.828+0000 7f506d60d700 10 monclient: _add_conns ranks=[1,0,2] 2021-05-25T12:08:14.871 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.828+0000 7f506d60d700 10 monclient(hunting): picked mon.noname-b con 0x7f50680692d0 addr v1:172.21.15.181:6789/0 2021-05-25T12:08:14.871 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.828+0000 7f506d60d700 10 monclient(hunting): picked mon.noname-a con 0x7f5068093fc0 addr v1:172.21.15.102:6789/0 2021-05-25T12:08:14.872 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.828+0000 7f506d60d700 10 monclient(hunting): picked mon.noname-c con 0x7f506812f130 addr v1:172.21.15.102:6790/0 2021-05-25T12:08:14.872 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.828+0000 7f506d60d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5068112240 con 0x7f5068093fc0 2021-05-25T12:08:14.872 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.828+0000 7f506d60d700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5068092350 con 0x7f50680692d0 2021-05-25T12:08:14.872 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.828+0000 7f506d60d700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f506812a330 con 0x7f506812f130 2021-05-25T12:08:14.873 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.828+0000 7f506d60d700 10 monclient(hunting): _renew_subs 2021-05-25T12:08:14.873 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.828+0000 7f506d60d700 10 monclient(hunting): authenticate will time out at 1295.425168s 2021-05-25T12:08:14.873 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.828+0000 7f50677fe700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f506812f130 0x7f506813f630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:57296/0 (socket says 172.21.15.102:57296) 2021-05-25T12:08:14.873 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.828+0000 7f50667fc700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f5068093fc0 0x7f506813bf00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:34184/0 (socket says 172.21.15.102:34184) 2021-05-25T12:08:14.874 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.828+0000 7f50677fe700 1 -- 172.21.15.102:0/91382385 learned_addr learned my addr 172.21.15.102:0/91382385 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:08:14.874 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.828+0000 7f504effd700 10 client.?.objecter ms_handle_connect 0x7f506812f130 2021-05-25T12:08:14.874 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.828+0000 7f504effd700 10 client.?.objecter resend_mon_ops 2021-05-25T12:08:14.874 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.828+0000 7f504effd700 10 client.?.objecter ms_handle_connect 0x7f5068093fc0 2021-05-25T12:08:14.874 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.828+0000 7f504effd700 10 client.?.objecter resend_mon_ops 2021-05-25T12:08:14.875 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.828+0000 7f504effd700 1 -- 172.21.15.102:0/91382385 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1382437025 0 0) 0x7f506812a330 con 0x7f506812f130 2021-05-25T12:08:14.875 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.828+0000 7f504effd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:14.875 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.828+0000 7f504effd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:14.876 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.828+0000 7f504effd700 10 monclient(hunting): my global_id is 244145 2021-05-25T12:08:14.876 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.828+0000 7f504effd700 1 -- 172.21.15.102:0/91382385 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5048004c10 con 0x7f506812f130 2021-05-25T12:08:14.876 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.828+0000 7f504effd700 1 -- 172.21.15.102:0/91382385 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1262141493 0 0) 0x7f5068112240 con 0x7f5068093fc0 2021-05-25T12:08:14.876 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.828+0000 7f504effd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:14.876 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.828+0000 7f504effd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:14.877 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.828+0000 7f504effd700 10 monclient(hunting): my global_id is 204546 2021-05-25T12:08:14.877 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.828+0000 7f504effd700 1 -- 172.21.15.102:0/91382385 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f506812a330 con 0x7f5068093fc0 2021-05-25T12:08:14.877 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.828+0000 7f504effd700 10 client.?.objecter ms_handle_connect 0x7f50680692d0 2021-05-25T12:08:14.877 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.828+0000 7f504effd700 10 client.?.objecter resend_mon_ops 2021-05-25T12:08:14.878 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.828+0000 7f504effd700 1 -- 172.21.15.102:0/91382385 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3848248440 0 0) 0x7f506812a330 con 0x7f5068093fc0 2021-05-25T12:08:14.878 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.828+0000 7f504effd700 1 -- 172.21.15.102:0/91382385 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5068112240 con 0x7f5068093fc0 2021-05-25T12:08:14.878 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.828+0000 7f504effd700 1 -- 172.21.15.102:0/91382385 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f50500018c0 con 0x7f5068093fc0 2021-05-25T12:08:14.878 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.828+0000 7f504effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:14.879 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.828+0000 7f504effd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:08:14.879 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.828+0000 7f504effd700 10 monclient(hunting): dump: 2021-05-25T12:08:14.879 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:14.879 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:14.879 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:14.880 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:14.880 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:14.880 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:14.880 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:14.880 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:14.881 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:14.881 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:14.881 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.828+0000 7f504effd700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:08:14.881 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.828+0000 7f504effd700 1 -- 172.21.15.102:0/91382385 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1615547366 0 0) 0x7f5048004c10 con 0x7f506812f130 2021-05-25T12:08:14.881 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.828+0000 7f504effd700 1 -- 172.21.15.102:0/91382385 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f506812a330 con 0x7f506812f130 2021-05-25T12:08:14.882 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.828+0000 7f504effd700 1 -- 172.21.15.102:0/91382385 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2937877692 0 0) 0x7f5068092350 con 0x7f50680692d0 2021-05-25T12:08:14.882 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.828+0000 7f504effd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:14.882 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.828+0000 7f504effd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:14.883 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.828+0000 7f504effd700 10 monclient(hunting): my global_id is 224401 2021-05-25T12:08:14.883 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.828+0000 7f504effd700 1 -- 172.21.15.102:0/91382385 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5048004c10 con 0x7f50680692d0 2021-05-25T12:08:14.883 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.828+0000 7f504effd700 1 -- 172.21.15.102:0/91382385 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f5058001540 con 0x7f506812f130 2021-05-25T12:08:14.883 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.828+0000 7f504effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:14.883 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.828+0000 7f504effd700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2021-05-25T12:08:14.884 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.828+0000 7f504effd700 10 monclient(hunting): dump: 2021-05-25T12:08:14.884 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:14.884 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:14.884 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:14.885 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:14.885 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:14.885 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:14.885 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:14.885 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:14.886 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:14.886 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:14.886 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.828+0000 7f504effd700 1 -- 172.21.15.102:0/91382385 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 133303536 0 0) 0x7f5068112240 con 0x7f5068093fc0 2021-05-25T12:08:14.886 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.828+0000 7f504effd700 1 -- 172.21.15.102:0/91382385 >> v1:172.21.15.102:6790/0 conn(0x7f506812f130 legacy=0x7f506813f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:14.887 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.828+0000 7f504effd700 1 -- 172.21.15.102:0/91382385 >> v1:172.21.15.181:6789/0 conn(0x7f50680692d0 legacy=0x7f50681287d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:14.887 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.828+0000 7f504effd700 10 monclient: _finish_hunting 0 2021-05-25T12:08:14.887 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.828+0000 7f504effd700 1 monclient: found mon.a 2021-05-25T12:08:14.887 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.828+0000 7f504effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:08:14.887 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.828+0000 7f504effd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:08:14.888 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.828+0000 7f504effd700 1 -- 172.21.15.102:0/91382385 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5068111660 con 0x7f5068093fc0 2021-05-25T12:08:14.888 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.828+0000 7f504effd700 10 monclient: _finish_auth 0 2021-05-25T12:08:14.888 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.828+0000 7f504effd700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:08:14.888 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.828+0000 7f506d60d700 5 monclient: authenticate success, global_id 204546 2021-05-25T12:08:14.889 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.828+0000 7f506d60d700 10 monclient: _renew_subs 2021-05-25T12:08:14.889 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.828+0000 7f506d60d700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:08:14.889 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.828+0000 7f506d60d700 1 -- 172.21.15.102:0/91382385 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5068067ec0 con 0x7f5068093fc0 2021-05-25T12:08:14.889 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.828+0000 7f506d60d700 10 client.204546.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:08:14.889 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.828+0000 7f506d60d700 10 monclient: _renew_subs 2021-05-25T12:08:14.890 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.828+0000 7f506d60d700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:08:14.890 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.828+0000 7f506d60d700 1 -- 172.21.15.102:0/91382385 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5068134070 con 0x7f5068093fc0 2021-05-25T12:08:14.890 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.828+0000 7f506d60d700 1 librados: init done 2021-05-25T12:08:14.890 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.829+0000 7f504effd700 1 -- 172.21.15.102:0/91382385 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5050001c20 con 0x7f5068093fc0 2021-05-25T12:08:14.890 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.829+0000 7f504effd700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:08:14.891 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.829+0000 7f504effd700 1 -- 172.21.15.102:0/91382385 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f5050001fb0 con 0x7f5068093fc0 2021-05-25T12:08:14.891 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.829+0000 7f504effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:14.891 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.829+0000 7f50657fa700 4 set_mon_vals no callback set 2021-05-25T12:08:14.891 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.829+0000 7f504effd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:08:14.892 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.829+0000 7f504effd700 10 monclient: dump: 2021-05-25T12:08:14.892 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:14.892 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:14.892 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:14.892 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:14.893 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:14.893 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:14.893 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:14.893 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:14.893 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:14.894 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:14.894 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.829+0000 7f506d60d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:08:14.894 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.829+0000 7f506d60d700 1 -- 172.21.15.102:0/91382385 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f50400065a0 con 0x7f5040002840 2021-05-25T12:08:14.894 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.829+0000 7f504effd700 1 -- 172.21.15.102:0/91382385 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f5050002110 con 0x7f5068093fc0 2021-05-25T12:08:14.895 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.829+0000 7f504effd700 10 client.204546.objecter ms_handle_connect 0x7f5040002840 2021-05-25T12:08:14.895 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.829+0000 7f504effd700 10 client.204546.objecter resend_mon_ops 2021-05-25T12:08:14.895 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.829+0000 7f504effd700 1 -- 172.21.15.102:0/91382385 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(271..271 src has 1..271) v4 ==== 5542+0+0 (unknown 782829896 0 0) 0x7f505003d530 con 0x7f5068093fc0 2021-05-25T12:08:14.895 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.829+0000 7f504effd700 10 client.204546.objecter ms_dispatch 0x7f5068001550 osd_map(271..271 src has 1..271) v4 2021-05-25T12:08:14.895 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.829+0000 7f504effd700 3 client.204546.objecter handle_osd_map got epochs [271,271] > 0 2021-05-25T12:08:14.896 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.829+0000 7f504effd700 3 client.204546.objecter handle_osd_map decoding full epoch 271 2021-05-25T12:08:14.896 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.830+0000 7f504effd700 20 client.204546.objecter dump_active .. 0 homeless 2021-05-25T12:08:14.896 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.830+0000 7f504effd700 1 -- 172.21.15.102:0/91382385 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 499687076 0 0) 0x7f50400065a0 con 0x7f5040002840 2021-05-25T12:08:14.896 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.830+0000 7f504effd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:14.896 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.830+0000 7f504effd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:14.897 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.830+0000 7f504effd700 10 monclient(hunting): my global_id is 244148 2021-05-25T12:08:14.897 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.830+0000 7f504effd700 1 -- 172.21.15.102:0/91382385 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5068112240 con 0x7f5040002840 2021-05-25T12:08:14.897 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.830+0000 7f50677fe700 10 monclient: get_auth_request con 0x7f50480334b0 auth_method 0 2021-05-25T12:08:14.897 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.830+0000 7f504effd700 1 -- 172.21.15.102:0/91382385 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1174601819 0 0) 0x7f5068112240 con 0x7f5040002840 2021-05-25T12:08:14.897 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.830+0000 7f504effd700 1 -- 172.21.15.102:0/91382385 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f50400065a0 con 0x7f5040002840 2021-05-25T12:08:14.898 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.830+0000 7f504effd700 1 -- 172.21.15.102:0/91382385 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f505c001a00 con 0x7f5040002840 2021-05-25T12:08:14.898 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.830+0000 7f504effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:14.898 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.830+0000 7f504effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:08:14.898 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.830+0000 7f504effd700 10 monclient: dump: 2021-05-25T12:08:14.899 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:14.899 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:14.899 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:14.899 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:14.899 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:14.900 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:14.900 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:14.900 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:14.900 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:14.901 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:14.901 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.830+0000 7f504effd700 10 client.204546.objecter ms_handle_connect 0x7f50480334b0 2021-05-25T12:08:14.901 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.830+0000 7f504effd700 1 -- 172.21.15.102:0/91382385 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 427042052 0 0) 0x7f50400065a0 con 0x7f5040002840 2021-05-25T12:08:14.901 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.830+0000 7f504effd700 1 -- 172.21.15.102:0/91382385 --> v1:172.21.15.102:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5040009e40 con 0x7f5040002840 2021-05-25T12:08:14.901 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.830+0000 7f504effd700 1 -- 172.21.15.102:0/91382385 <== mon.2 v1:172.21.15.102:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7f505c004530 con 0x7f5040002840 2021-05-25T12:08:14.902 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.830+0000 7f504effd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:08:14.902 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.830+0000 7f504effd700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:08:14.902 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.830+0000 7f504effd700 1 -- 172.21.15.102:0/91382385 >> v1:172.21.15.102:6790/0 conn(0x7f5040002840 legacy=0x7f5040004ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:14.902 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.830+0000 7f504effd700 1 -- 172.21.15.102:0/91382385 >> v1:172.21.15.102:6790/0 conn(0x7f5040002840 legacy=0x7f5040004ce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:08:14.903 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.837+0000 7f506d60d700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:08:14.903 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.837+0000 7f506d60d700 1 -- 172.21.15.102:0/91382385 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f5040009ad0 con 0x7f5040008d90 2021-05-25T12:08:14.903 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.837+0000 7f504effd700 10 client.204546.objecter ms_handle_connect 0x7f5040008d90 2021-05-25T12:08:14.903 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.837+0000 7f504effd700 10 client.204546.objecter resend_mon_ops 2021-05-25T12:08:14.903 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.837+0000 7f504effd700 1 -- 172.21.15.102:0/91382385 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3856506715 0 0) 0x7f5040009ad0 con 0x7f5040008d90 2021-05-25T12:08:14.904 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.837+0000 7f504effd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:14.904 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.837+0000 7f504effd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:14.904 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.837+0000 7f504effd700 10 monclient(hunting): my global_id is 244151 2021-05-25T12:08:14.904 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.837+0000 7f504effd700 1 -- 172.21.15.102:0/91382385 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50400065a0 con 0x7f5040008d90 2021-05-25T12:08:14.904 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.837+0000 7f504effd700 1 -- 172.21.15.102:0/91382385 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2546051684 0 0) 0x7f50400065a0 con 0x7f5040008d90 2021-05-25T12:08:14.905 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.837+0000 7f504effd700 1 -- 172.21.15.102:0/91382385 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5040009ad0 con 0x7f5040008d90 2021-05-25T12:08:14.905 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.837+0000 7f504effd700 1 -- 172.21.15.102:0/91382385 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f505c004bd0 con 0x7f5040008d90 2021-05-25T12:08:14.905 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.837+0000 7f504effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:14.905 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.837+0000 7f504effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:08:14.906 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.837+0000 7f504effd700 10 monclient: dump: 2021-05-25T12:08:14.906 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:14.906 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:14.906 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:14.906 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:14.907 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:14.907 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:14.907 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:14.907 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:14.908 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:14.908 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:14.908 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.837+0000 7f504effd700 1 -- 172.21.15.102:0/91382385 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2993961710 0 0) 0x7f5040009ad0 con 0x7f5040008d90 2021-05-25T12:08:14.908 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.837+0000 7f504effd700 1 -- 172.21.15.102:0/91382385 --> v1:172.21.15.102:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f5040008740 con 0x7f5040008d90 2021-05-25T12:08:14.908 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.838+0000 7f504effd700 1 -- 172.21.15.102:0/91382385 <== mon.2 v1:172.21.15.102:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3770 (unknown 0 0 1409104915) 0x7f505c005ed0 con 0x7f5040008d90 2021-05-25T12:08:14.909 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.838+0000 7f504effd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:08:14.909 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.838+0000 7f504effd700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:08:14.909 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.838+0000 7f504effd700 1 -- 172.21.15.102:0/91382385 >> v1:172.21.15.102:6790/0 conn(0x7f5040008d90 legacy=0x7f5040009220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:14.909 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.838+0000 7f504effd700 1 -- 172.21.15.102:0/91382385 >> v1:172.21.15.102:6790/0 conn(0x7f5040008d90 legacy=0x7f5040009220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:08:14.910 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:08:14.910 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:08:14.910 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:08:14.911 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "peon", 2021-05-25T12:08:14.911 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 68, 2021-05-25T12:08:14.911 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:08:14.911 INFO:teuthology.orchestra.run.smithi102.stdout: 0, 2021-05-25T12:08:14.912 INFO:teuthology.orchestra.run.smithi102.stdout: 1, 2021-05-25T12:08:14.912 INFO:teuthology.orchestra.run.smithi102.stdout: 2 2021-05-25T12:08:14.912 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:08:14.912 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 3, 2021-05-25T12:08:14.912 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:08:14.913 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:08:14.913 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:08:14.913 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:08:14.913 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:08:14.914 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:08:14.914 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:08:14.914 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:08:14.914 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:08:14.914 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:08:14.915 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:08:14.915 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:08:14.915 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:08:14.915 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:08:14.915 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:08:14.916 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:08:14.916 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:08:14.916 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:08:14.916 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:08:14.917 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:08:14.917 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:08:14.917 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:08:14.917 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:08:14.918 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:08:14.918 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:14.918 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:14.918 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:08:14.918 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:08:14.919 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:08:14.919 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:08:14.919 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:08:14.919 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:08:14.920 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:08:14.920 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:08:14.920 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:08:14.920 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:08:14.920 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:08:14.921 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:08:14.921 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:08:14.921 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:08:14.921 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:08:14.921 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:08:14.922 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:08:14.922 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:08:14.922 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:08:14.922 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:08:14.923 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:08:14.923 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:08:14.923 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:08:14.923 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:08:14.923 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:08:14.924 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:08:14.924 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:14.924 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:08:14.924 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:14.924 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:08:14.925 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:08:14.925 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:08:14.925 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:08:14.925 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:14.926 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:08:14.926 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:08:14.926 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:08:14.926 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:14.927 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:14.927 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:14.927 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:08:14.927 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:08:14.927 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:08:14.928 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:08:14.928 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:08:14.928 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:14.928 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:14.928 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:08:14.929 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:08:14.929 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:08:14.929 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:08:14.929 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:14.929 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:08:14.930 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:08:14.930 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:08:14.930 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:14.930 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:14.931 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:14.931 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:08:14.931 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:08:14.932 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:08:14.932 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:08:14.932 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:08:14.933 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:14.933 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:14.933 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:08:14.933 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:08:14.933 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:08:14.934 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:08:14.934 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:14.934 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:08:14.934 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:08:14.935 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:08:14.935 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:14.935 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:14.935 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:14.935 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:08:14.936 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:08:14.936 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:08:14.936 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:08:14.936 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:08:14.937 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:14.937 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:14.937 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:14.937 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:08:14.937 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:08:14.938 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:14.938 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:08:14.938 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:08:14.938 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:08:14.939 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:14.939 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:08:14.939 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:08:14.939 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:14.939 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:08:14.940 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:08:14.940 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:08:14.940 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:14.940 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:14.941 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:14.941 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:08:14.941 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:08:14.943 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.838+0000 7f506d60d700 10 librados: watch_flush enter 2021-05-25T12:08:14.944 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.838+0000 7f506d60d700 10 librados: watch_flush exit 2021-05-25T12:08:14.944 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.838+0000 7f506d60d700 20 client.204546.objecter shutdown clearing up homeless session... 2021-05-25T12:08:14.944 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.838+0000 7f506d60d700 10 client.204546.objecter successfully canceled tick 2021-05-25T12:08:14.945 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.838+0000 7f506d60d700 1 -- 172.21.15.102:0/91382385 >> v1:172.21.15.181:6812/29582 conn(0x7f50480334b0 legacy=0x7f5048035970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:14.945 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.838+0000 7f506d60d700 10 monclient: shutdown 2021-05-25T12:08:14.945 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.838+0000 7f506d60d700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:08:14.946 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.838+0000 7f506d60d700 1 -- 172.21.15.102:0/91382385 >> v1:172.21.15.102:6789/0 conn(0x7f5068093fc0 legacy=0x7f506813bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:14.946 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.838+0000 7f50677fe700 1 -- 172.21.15.102:0/91382385 reap_dead start 2021-05-25T12:08:14.947 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.838+0000 7f50677fe700 1 -- 172.21.15.102:0/91382385 reap_dead start 2021-05-25T12:08:14.947 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.838+0000 7f506d60d700 1 -- 172.21.15.102:0/91382385 shutdown_connections 2021-05-25T12:08:14.947 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.838+0000 7f506d60d700 1 -- 172.21.15.102:0/91382385 >> 172.21.15.102:0/91382385 conn(0x7f506809b370 msgr2=0x7f5068098fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:08:14.948 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.838+0000 7f506d60d700 1 -- 172.21.15.102:0/91382385 shutdown_connections 2021-05-25T12:08:14.948 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.838+0000 7f506d60d700 1 -- 172.21.15.102:0/91382385 wait complete. 2021-05-25T12:08:14.949 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:14.838+0000 7f506d60d700 1 librados: shutdown 2021-05-25T12:08:14.950 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2021-05-25T12:08:15.060 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.058+0000 7f3d86c41700 10 monclient: get_monmap_and_config 2021-05-25T12:08:15.060 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.058+0000 7f3d86c41700 10 monclient: build_initial_monmap 2021-05-25T12:08:15.060 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.058+0000 7f3d86c41700 10 monclient: monmap: 2021-05-25T12:08:15.060 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:08:15.061 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:15.061 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:08:15.059686+0000 2021-05-25T12:08:15.061 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:08:15.059686+0000 2021-05-25T12:08:15.061 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:08:15.062 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:15.062 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:08:15.062 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:08:15.062 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:08:15.062 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:15.079 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.059+0000 7f3d86c41700 1 Processor -- start 2021-05-25T12:08:15.080 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.059+0000 7f3d86c41700 1 -- start start 2021-05-25T12:08:15.080 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.059+0000 7f3d86c41700 10 monclient: init 2021-05-25T12:08:15.080 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.059+0000 7f3d86c41700 10 monclient: _reopen_session rank -1 2021-05-25T12:08:15.080 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.059+0000 7f3d86c41700 10 monclient: _add_conns ranks=[1,2,0] 2021-05-25T12:08:15.081 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.059+0000 7f3d86c41700 10 monclient(hunting): picked mon.noname-c con 0x7f3d80093fc0 addr v1:172.21.15.181:6789/0 2021-05-25T12:08:15.081 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.059+0000 7f3d86c41700 10 monclient(hunting): picked mon.noname-b con 0x7f3d800692d0 addr v1:172.21.15.102:6790/0 2021-05-25T12:08:15.081 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.059+0000 7f3d86c41700 10 monclient(hunting): picked mon.noname-a con 0x7f3d8012f130 addr v1:172.21.15.102:6789/0 2021-05-25T12:08:15.081 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.059+0000 7f3d86c41700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d80097160 con 0x7f3d8012f130 2021-05-25T12:08:15.082 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.059+0000 7f3d86c41700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d80092fe0 con 0x7f3d80093fc0 2021-05-25T12:08:15.082 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.059+0000 7f3d86c41700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d80093280 con 0x7f3d800692d0 2021-05-25T12:08:15.082 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.059+0000 7f3d86c41700 10 monclient(hunting): _renew_subs 2021-05-25T12:08:15.082 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.059+0000 7f3d86c41700 10 monclient(hunting): authenticate will time out at 1295.656924s 2021-05-25T12:08:15.083 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.060+0000 7f3d77fff700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f3d800692d0 0x7f3d80096a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:57306/0 (socket says 172.21.15.102:57306) 2021-05-25T12:08:15.083 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.060+0000 7f3d851de700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f3d8012f130 0x7f3d80131520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:34198/0 (socket says 172.21.15.102:34198) 2021-05-25T12:08:15.083 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.060+0000 7f3d77fff700 1 -- 172.21.15.102:0/2798612355 learned_addr learned my addr 172.21.15.102:0/2798612355 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:08:15.083 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.060+0000 7f3d777fe700 1 -- 172.21.15.102:0/2798612355 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1937190938 0 0) 0x7f3d80097160 con 0x7f3d8012f130 2021-05-25T12:08:15.084 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.060+0000 7f3d777fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:15.084 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.060+0000 7f3d777fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:15.084 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.060+0000 7f3d777fe700 10 monclient(hunting): my global_id is 204549 2021-05-25T12:08:15.084 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.060+0000 7f3d777fe700 1 -- 172.21.15.102:0/2798612355 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d6c003270 con 0x7f3d8012f130 2021-05-25T12:08:15.085 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.060+0000 7f3d777fe700 1 -- 172.21.15.102:0/2798612355 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1666936361 0 0) 0x7f3d80093280 con 0x7f3d800692d0 2021-05-25T12:08:15.085 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.060+0000 7f3d777fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:15.085 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.060+0000 7f3d777fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:15.085 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.060+0000 7f3d777fe700 10 monclient(hunting): my global_id is 244154 2021-05-25T12:08:15.085 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.060+0000 7f3d777fe700 1 -- 172.21.15.102:0/2798612355 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d80097160 con 0x7f3d800692d0 2021-05-25T12:08:15.086 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.060+0000 7f3d777fe700 1 -- 172.21.15.102:0/2798612355 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 253751587 0 0) 0x7f3d80092fe0 con 0x7f3d80093fc0 2021-05-25T12:08:15.086 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.060+0000 7f3d777fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:15.086 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.060+0000 7f3d777fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:15.086 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.060+0000 7f3d777fe700 10 monclient(hunting): my global_id is 224404 2021-05-25T12:08:15.087 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.060+0000 7f3d777fe700 1 -- 172.21.15.102:0/2798612355 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d80093280 con 0x7f3d80093fc0 2021-05-25T12:08:15.087 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.060+0000 7f3d777fe700 1 -- 172.21.15.102:0/2798612355 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3571274629 0 0) 0x7f3d6c003270 con 0x7f3d8012f130 2021-05-25T12:08:15.087 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.060+0000 7f3d777fe700 1 -- 172.21.15.102:0/2798612355 >> v1:172.21.15.102:6790/0 conn(0x7f3d800692d0 legacy=0x7f3d80096a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:15.087 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.060+0000 7f3d777fe700 1 -- 172.21.15.102:0/2798612355 >> v1:172.21.15.181:6789/0 conn(0x7f3d80093fc0 legacy=0x7f3d80096340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:15.088 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.060+0000 7f3d777fe700 10 monclient: _finish_hunting 0 2021-05-25T12:08:15.088 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.060+0000 7f3d777fe700 1 monclient: found mon.noname-a 2021-05-25T12:08:15.088 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.060+0000 7f3d777fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:08:15.088 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.060+0000 7f3d777fe700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T12:08:15.088 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.060+0000 7f3d777fe700 1 -- 172.21.15.102:0/2798612355 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d80093520 con 0x7f3d8012f130 2021-05-25T12:08:15.089 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.060+0000 7f3d777fe700 10 monclient: _finish_auth 0 2021-05-25T12:08:15.089 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.060+0000 7f3d777fe700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:08:15.089 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.060+0000 7f3d777fe700 1 -- 172.21.15.102:0/2798612355 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f3d7c001990 con 0x7f3d8012f130 2021-05-25T12:08:15.089 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.060+0000 7f3d86c41700 5 monclient: authenticate success, global_id 204549 2021-05-25T12:08:15.090 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.060+0000 7f3d86c41700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:08:15.090 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.060+0000 7f3d777fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:15.090 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.060+0000 7f3d777fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:08:15.090 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.060+0000 7f3d777fe700 10 monclient: dump: 2021-05-25T12:08:15.090 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:15.091 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:15.091 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:15.091 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:15.091 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:15.091 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:15.092 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:15.092 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:15.092 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:15.092 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:15.093 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.060+0000 7f3d86c41700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:08:15.093 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.060+0000 7f3d777fe700 1 -- 172.21.15.102:0/2798612355 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3d7c001bf0 con 0x7f3d8012f130 2021-05-25T12:08:15.093 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.060+0000 7f3d777fe700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:08:15.093 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.060+0000 7f3d86c41700 10 monclient: get_monmap_and_config success 2021-05-25T12:08:15.093 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.060+0000 7f3d777fe700 1 -- 172.21.15.102:0/2798612355 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f3d7c001f80 con 0x7f3d8012f130 2021-05-25T12:08:15.094 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.060+0000 7f3d777fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:15.094 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.060+0000 7f3d86c41700 4 set_mon_vals no callback set 2021-05-25T12:08:15.094 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.060+0000 7f3d777fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:08:15.094 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.060+0000 7f3d777fe700 10 monclient: dump: 2021-05-25T12:08:15.094 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:15.095 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:15.095 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:15.095 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:15.095 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:15.096 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:15.096 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:15.096 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:15.096 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:15.096 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:15.097 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.061+0000 7f3d86c41700 10 monclient: shutdown 2021-05-25T12:08:15.097 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.061+0000 7f3d86c41700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:08:15.097 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.061+0000 7f3d86c41700 1 -- 172.21.15.102:0/2798612355 >> v1:172.21.15.102:6789/0 conn(0x7f3d8012f130 legacy=0x7f3d80131520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:15.097 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.061+0000 7f3d86c41700 1 -- 172.21.15.102:0/2798612355 shutdown_connections 2021-05-25T12:08:15.097 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.061+0000 7f3d86c41700 1 -- 172.21.15.102:0/2798612355 >> 172.21.15.102:0/2798612355 conn(0x7f3d8009b370 msgr2=0x7f3d80097430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:08:15.098 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.061+0000 7f3d86c41700 1 -- 172.21.15.102:0/2798612355 shutdown_connections 2021-05-25T12:08:15.098 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.061+0000 7f3d86c41700 1 -- 172.21.15.102:0/2798612355 wait complete. 2021-05-25T12:08:15.098 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.061+0000 7f3d86c41700 10 monclient: build_initial_monmap 2021-05-25T12:08:15.098 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.061+0000 7f3d86c41700 10 monclient: monmap: 2021-05-25T12:08:15.099 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:08:15.099 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:08:15.099 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:08:15.099 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:08:15.099 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:08:15.100 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:08:15.100 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:08:15.100 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:08:15.100 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:08:15.100 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:15.101 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.061+0000 7f3d86c41700 1 librados: starting msgr at 2021-05-25T12:08:15.101 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.061+0000 7f3d86c41700 1 librados: starting objecter 2021-05-25T12:08:15.101 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.062+0000 7f3d86c41700 1 Processor -- start 2021-05-25T12:08:15.101 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.062+0000 7f3d86c41700 1 -- start start 2021-05-25T12:08:15.101 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.062+0000 7f3d86c41700 1 librados: setting wanted keys 2021-05-25T12:08:15.102 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.062+0000 7f3d86c41700 1 librados: calling monclient init 2021-05-25T12:08:15.102 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.062+0000 7f3d86c41700 10 monclient: init 2021-05-25T12:08:15.102 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.062+0000 7f3d86c41700 10 monclient: _reopen_session rank -1 2021-05-25T12:08:15.102 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.062+0000 7f3d86c41700 10 monclient: _add_conns ranks=[2,0,1] 2021-05-25T12:08:15.103 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.062+0000 7f3d86c41700 10 monclient(hunting): picked mon.noname-c con 0x7f3d800692d0 addr v1:172.21.15.102:6790/0 2021-05-25T12:08:15.103 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.062+0000 7f3d86c41700 10 monclient(hunting): picked mon.noname-a con 0x7f3d80093fc0 addr v1:172.21.15.102:6789/0 2021-05-25T12:08:15.103 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.062+0000 7f3d86c41700 10 monclient(hunting): picked mon.noname-b con 0x7f3d8012f130 addr v1:172.21.15.181:6789/0 2021-05-25T12:08:15.103 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.062+0000 7f3d86c41700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d800675c0 con 0x7f3d80093fc0 2021-05-25T12:08:15.103 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.062+0000 7f3d86c41700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d80134370 con 0x7f3d8012f130 2021-05-25T12:08:15.104 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.062+0000 7f3d86c41700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d80112090 con 0x7f3d800692d0 2021-05-25T12:08:15.104 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.062+0000 7f3d86c41700 10 monclient(hunting): _renew_subs 2021-05-25T12:08:15.104 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.062+0000 7f3d86c41700 10 monclient(hunting): authenticate will time out at 1295.659716s 2021-05-25T12:08:15.104 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.062+0000 7f3d77fff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f3d80093fc0 0x7f3d80128690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:34202/0 (socket says 172.21.15.102:34202) 2021-05-25T12:08:15.105 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.062+0000 7f3d849dd700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f3d800692d0 0x7f3d80133c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:57310/0 (socket says 172.21.15.102:57310) 2021-05-25T12:08:15.105 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.062+0000 7f3d77fff700 1 -- 172.21.15.102:0/4004403904 learned_addr learned my addr 172.21.15.102:0/4004403904 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:08:15.105 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.062+0000 7f3d74ff9700 10 client.?.objecter ms_handle_connect 0x7f3d800692d0 2021-05-25T12:08:15.105 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.062+0000 7f3d74ff9700 10 client.?.objecter resend_mon_ops 2021-05-25T12:08:15.105 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.062+0000 7f3d74ff9700 10 client.?.objecter ms_handle_connect 0x7f3d80093fc0 2021-05-25T12:08:15.106 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.062+0000 7f3d74ff9700 10 client.?.objecter resend_mon_ops 2021-05-25T12:08:15.106 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.062+0000 7f3d74ff9700 10 client.?.objecter ms_handle_connect 0x7f3d8012f130 2021-05-25T12:08:15.106 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.062+0000 7f3d74ff9700 10 client.?.objecter resend_mon_ops 2021-05-25T12:08:15.107 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.062+0000 7f3d74ff9700 1 -- 172.21.15.102:0/4004403904 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3803455508 0 0) 0x7f3d80112090 con 0x7f3d800692d0 2021-05-25T12:08:15.107 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.063+0000 7f3d74ff9700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:15.107 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.063+0000 7f3d74ff9700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:15.107 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.063+0000 7f3d74ff9700 10 monclient(hunting): my global_id is 244157 2021-05-25T12:08:15.107 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.063+0000 7f3d74ff9700 1 -- 172.21.15.102:0/4004403904 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d60004c10 con 0x7f3d800692d0 2021-05-25T12:08:15.108 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.063+0000 7f3d74ff9700 1 -- 172.21.15.102:0/4004403904 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2408853556 0 0) 0x7f3d800675c0 con 0x7f3d80093fc0 2021-05-25T12:08:15.108 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.063+0000 7f3d74ff9700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:15.108 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.063+0000 7f3d74ff9700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:15.108 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.063+0000 7f3d74ff9700 10 monclient(hunting): my global_id is 204552 2021-05-25T12:08:15.109 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.063+0000 7f3d74ff9700 1 -- 172.21.15.102:0/4004403904 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d80112090 con 0x7f3d80093fc0 2021-05-25T12:08:15.109 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.063+0000 7f3d74ff9700 1 -- 172.21.15.102:0/4004403904 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4086785855 0 0) 0x7f3d80134370 con 0x7f3d8012f130 2021-05-25T12:08:15.109 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.063+0000 7f3d74ff9700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:15.109 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.063+0000 7f3d74ff9700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:15.110 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.063+0000 7f3d74ff9700 10 monclient(hunting): my global_id is 224407 2021-05-25T12:08:15.110 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.063+0000 7f3d74ff9700 1 -- 172.21.15.102:0/4004403904 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d800675c0 con 0x7f3d8012f130 2021-05-25T12:08:15.110 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.063+0000 7f3d74ff9700 1 -- 172.21.15.102:0/4004403904 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2875992243 0 0) 0x7f3d80112090 con 0x7f3d80093fc0 2021-05-25T12:08:15.110 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.063+0000 7f3d74ff9700 1 -- 172.21.15.102:0/4004403904 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3d80134370 con 0x7f3d80093fc0 2021-05-25T12:08:15.110 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.063+0000 7f3d74ff9700 1 -- 172.21.15.102:0/4004403904 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3393709346 0 0) 0x7f3d60004c10 con 0x7f3d800692d0 2021-05-25T12:08:15.111 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.063+0000 7f3d74ff9700 1 -- 172.21.15.102:0/4004403904 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3d80112090 con 0x7f3d800692d0 2021-05-25T12:08:15.111 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.063+0000 7f3d74ff9700 1 -- 172.21.15.102:0/4004403904 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f3d68001a50 con 0x7f3d80093fc0 2021-05-25T12:08:15.111 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.063+0000 7f3d74ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:15.111 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.063+0000 7f3d74ff9700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:08:15.111 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.063+0000 7f3d74ff9700 10 monclient(hunting): dump: 2021-05-25T12:08:15.112 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:15.112 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:15.112 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:15.112 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:15.113 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:15.113 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:15.113 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:15.113 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:15.114 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:15.114 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:15.114 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.063+0000 7f3d74ff9700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:08:15.114 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.063+0000 7f3d74ff9700 1 -- 172.21.15.102:0/4004403904 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f3d70002890 con 0x7f3d800692d0 2021-05-25T12:08:15.115 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.063+0000 7f3d74ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:15.115 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.063+0000 7f3d74ff9700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2021-05-25T12:08:15.115 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.063+0000 7f3d74ff9700 10 monclient(hunting): dump: 2021-05-25T12:08:15.115 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:15.116 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:15.116 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:15.116 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:15.116 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:15.117 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:15.117 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:15.117 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:15.117 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:15.118 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:15.118 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.063+0000 7f3d74ff9700 1 -- 172.21.15.102:0/4004403904 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1812183958 0 0) 0x7f3d800675c0 con 0x7f3d8012f130 2021-05-25T12:08:15.118 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.063+0000 7f3d74ff9700 1 -- 172.21.15.102:0/4004403904 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3d60004c10 con 0x7f3d8012f130 2021-05-25T12:08:15.118 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.063+0000 7f3d74ff9700 1 -- 172.21.15.102:0/4004403904 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f3d7c001f00 con 0x7f3d8012f130 2021-05-25T12:08:15.119 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.063+0000 7f3d74ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:15.119 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.063+0000 7f3d74ff9700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2021-05-25T12:08:15.119 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.063+0000 7f3d74ff9700 10 monclient(hunting): dump: 2021-05-25T12:08:15.119 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:15.120 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:15.120 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:15.120 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:15.120 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:15.121 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:15.121 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:15.121 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:15.121 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:15.122 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:15.122 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.063+0000 7f3d74ff9700 1 -- 172.21.15.102:0/4004403904 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3667876393 0 0) 0x7f3d80134370 con 0x7f3d80093fc0 2021-05-25T12:08:15.122 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.063+0000 7f3d74ff9700 1 -- 172.21.15.102:0/4004403904 >> v1:172.21.15.102:6790/0 conn(0x7f3d800692d0 legacy=0x7f3d80133c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:15.122 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.063+0000 7f3d74ff9700 1 -- 172.21.15.102:0/4004403904 >> v1:172.21.15.181:6789/0 conn(0x7f3d8012f130 legacy=0x7f3d8013ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:15.123 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.063+0000 7f3d74ff9700 10 monclient: _finish_hunting 0 2021-05-25T12:08:15.123 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.063+0000 7f3d74ff9700 1 monclient: found mon.a 2021-05-25T12:08:15.123 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.063+0000 7f3d74ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:08:15.124 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.063+0000 7f3d74ff9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:08:15.124 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.063+0000 7f3d74ff9700 1 -- 172.21.15.102:0/4004403904 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d80067d70 con 0x7f3d80093fc0 2021-05-25T12:08:15.124 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.063+0000 7f3d74ff9700 10 monclient: _finish_auth 0 2021-05-25T12:08:15.124 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.063+0000 7f3d74ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:08:15.125 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.063+0000 7f3d86c41700 5 monclient: authenticate success, global_id 204552 2021-05-25T12:08:15.125 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.063+0000 7f3d86c41700 10 monclient: _renew_subs 2021-05-25T12:08:15.125 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.063+0000 7f3d86c41700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:08:15.125 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.063+0000 7f3d86c41700 1 -- 172.21.15.102:0/4004403904 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d80128da0 con 0x7f3d80093fc0 2021-05-25T12:08:15.125 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.063+0000 7f3d86c41700 10 client.204552.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:08:15.126 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.063+0000 7f3d86c41700 10 monclient: _renew_subs 2021-05-25T12:08:15.126 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.063+0000 7f3d86c41700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:08:15.126 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.063+0000 7f3d86c41700 1 -- 172.21.15.102:0/4004403904 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d80068a10 con 0x7f3d80093fc0 2021-05-25T12:08:15.126 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.063+0000 7f3d86c41700 1 librados: init done 2021-05-25T12:08:15.127 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.063+0000 7f3d74ff9700 1 -- 172.21.15.102:0/4004403904 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3d68001d80 con 0x7f3d80093fc0 2021-05-25T12:08:15.127 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.063+0000 7f3d74ff9700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:08:15.127 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.063+0000 7f3d74ff9700 1 -- 172.21.15.102:0/4004403904 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f3d68004980 con 0x7f3d80093fc0 2021-05-25T12:08:15.127 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.063+0000 7f3d74ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:15.128 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.063+0000 7f3d76ffd700 4 set_mon_vals no callback set 2021-05-25T12:08:15.128 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.063+0000 7f3d74ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:08:15.128 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.063+0000 7f3d74ff9700 10 monclient: dump: 2021-05-25T12:08:15.128 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:15.128 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:15.129 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:15.129 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:15.129 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:15.129 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:15.130 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:15.130 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:15.130 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:15.130 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:15.131 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.064+0000 7f3d74ff9700 1 -- 172.21.15.102:0/4004403904 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f3d6800dca0 con 0x7f3d80093fc0 2021-05-25T12:08:15.131 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.064+0000 7f3d86c41700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:08:15.131 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.064+0000 7f3d86c41700 1 -- 172.21.15.102:0/4004403904 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f3d580065a0 con 0x7f3d58002840 2021-05-25T12:08:15.132 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.064+0000 7f3d74ff9700 1 -- 172.21.15.102:0/4004403904 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(271..271 src has 1..271) v4 ==== 5542+0+0 (unknown 782829896 0 0) 0x7f3d6803d5e0 con 0x7f3d80093fc0 2021-05-25T12:08:15.132 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.064+0000 7f3d74ff9700 10 client.204552.objecter ms_dispatch 0x7f3d80001550 osd_map(271..271 src has 1..271) v4 2021-05-25T12:08:15.132 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.064+0000 7f3d74ff9700 3 client.204552.objecter handle_osd_map got epochs [271,271] > 0 2021-05-25T12:08:15.132 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.064+0000 7f3d74ff9700 3 client.204552.objecter handle_osd_map decoding full epoch 271 2021-05-25T12:08:15.132 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.064+0000 7f3d74ff9700 20 client.204552.objecter dump_active .. 0 homeless 2021-05-25T12:08:15.133 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.064+0000 7f3d74ff9700 10 client.204552.objecter ms_handle_connect 0x7f3d58002840 2021-05-25T12:08:15.133 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.064+0000 7f3d74ff9700 10 client.204552.objecter resend_mon_ops 2021-05-25T12:08:15.133 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.064+0000 7f3d851de700 10 monclient: get_auth_request con 0x7f3d60033530 auth_method 0 2021-05-25T12:08:15.133 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.064+0000 7f3d74ff9700 1 -- 172.21.15.102:0/4004403904 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 63895956 0 0) 0x7f3d580065a0 con 0x7f3d58002840 2021-05-25T12:08:15.134 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.064+0000 7f3d74ff9700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:15.134 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.064+0000 7f3d74ff9700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:15.134 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.064+0000 7f3d74ff9700 10 monclient(hunting): my global_id is 224410 2021-05-25T12:08:15.134 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.064+0000 7f3d74ff9700 1 -- 172.21.15.102:0/4004403904 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d80134370 con 0x7f3d58002840 2021-05-25T12:08:15.134 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.064+0000 7f3d74ff9700 10 client.204552.objecter ms_handle_connect 0x7f3d60033530 2021-05-25T12:08:15.135 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.064+0000 7f3d74ff9700 1 -- 172.21.15.102:0/4004403904 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4288879021 0 0) 0x7f3d80134370 con 0x7f3d58002840 2021-05-25T12:08:15.135 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.064+0000 7f3d74ff9700 1 -- 172.21.15.102:0/4004403904 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3d580065a0 con 0x7f3d58002840 2021-05-25T12:08:15.135 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.064+0000 7f3d74ff9700 1 -- 172.21.15.102:0/4004403904 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f3d70001740 con 0x7f3d58002840 2021-05-25T12:08:15.135 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.064+0000 7f3d74ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:15.136 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.064+0000 7f3d74ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T12:08:15.136 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.064+0000 7f3d74ff9700 10 monclient: dump: 2021-05-25T12:08:15.136 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:15.136 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:15.136 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:15.137 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:15.137 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:15.137 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:15.137 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:15.138 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:15.138 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:15.138 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:15.139 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.065+0000 7f3d74ff9700 1 -- 172.21.15.102:0/4004403904 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3993105869 0 0) 0x7f3d580065a0 con 0x7f3d58002840 2021-05-25T12:08:15.139 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.065+0000 7f3d74ff9700 1 -- 172.21.15.102:0/4004403904 --> v1:172.21.15.181:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3d58009e40 con 0x7f3d58002840 2021-05-25T12:08:15.139 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.065+0000 7f3d74ff9700 1 -- 172.21.15.102:0/4004403904 <== mon.1 v1:172.21.15.181:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7f3d70001970 con 0x7f3d58002840 2021-05-25T12:08:15.139 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.065+0000 7f3d74ff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:08:15.139 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.065+0000 7f3d74ff9700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:08:15.140 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.065+0000 7f3d74ff9700 1 -- 172.21.15.102:0/4004403904 >> v1:172.21.15.181:6789/0 conn(0x7f3d58002840 legacy=0x7f3d58004ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:15.140 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.065+0000 7f3d74ff9700 1 -- 172.21.15.102:0/4004403904 >> v1:172.21.15.181:6789/0 conn(0x7f3d58002840 legacy=0x7f3d58004ce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:08:15.140 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.071+0000 7f3d86c41700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:08:15.140 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.071+0000 7f3d86c41700 1 -- 172.21.15.102:0/4004403904 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f3d580074f0 con 0x7f3d58006890 2021-05-25T12:08:15.141 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.072+0000 7f3d74ff9700 10 client.204552.objecter ms_handle_connect 0x7f3d58006890 2021-05-25T12:08:15.141 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.072+0000 7f3d74ff9700 10 client.204552.objecter resend_mon_ops 2021-05-25T12:08:15.141 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.072+0000 7f3d74ff9700 1 -- 172.21.15.102:0/4004403904 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2823182124 0 0) 0x7f3d580074f0 con 0x7f3d58006890 2021-05-25T12:08:15.141 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.072+0000 7f3d74ff9700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:15.141 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.072+0000 7f3d74ff9700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:15.142 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.072+0000 7f3d74ff9700 10 monclient(hunting): my global_id is 224413 2021-05-25T12:08:15.142 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.072+0000 7f3d74ff9700 1 -- 172.21.15.102:0/4004403904 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d580065a0 con 0x7f3d58006890 2021-05-25T12:08:15.142 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.072+0000 7f3d74ff9700 1 -- 172.21.15.102:0/4004403904 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 511694091 0 0) 0x7f3d580065a0 con 0x7f3d58006890 2021-05-25T12:08:15.142 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.072+0000 7f3d74ff9700 1 -- 172.21.15.102:0/4004403904 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3d580074f0 con 0x7f3d58006890 2021-05-25T12:08:15.142 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.072+0000 7f3d74ff9700 1 -- 172.21.15.102:0/4004403904 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f3d70002050 con 0x7f3d58006890 2021-05-25T12:08:15.143 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.072+0000 7f3d74ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:15.143 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.072+0000 7f3d74ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T12:08:15.143 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.072+0000 7f3d74ff9700 10 monclient: dump: 2021-05-25T12:08:15.144 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:15.144 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:15.144 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:15.144 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:15.144 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:15.145 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:15.145 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:15.145 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:15.145 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:15.146 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:15.146 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.072+0000 7f3d74ff9700 1 -- 172.21.15.102:0/4004403904 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 511815028 0 0) 0x7f3d580074f0 con 0x7f3d58006890 2021-05-25T12:08:15.146 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.072+0000 7f3d74ff9700 1 -- 172.21.15.102:0/4004403904 --> v1:172.21.15.181:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f3d58008770 con 0x7f3d58006890 2021-05-25T12:08:15.146 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.073+0000 7f3d74ff9700 1 -- 172.21.15.102:0/4004403904 <== mon.1 v1:172.21.15.181:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3942 (unknown 0 0 572409722) 0x7f3d70005580 con 0x7f3d58006890 2021-05-25T12:08:15.147 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.073+0000 7f3d74ff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:08:15.147 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.073+0000 7f3d74ff9700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:08:15.147 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.073+0000 7f3d74ff9700 1 -- 172.21.15.102:0/4004403904 >> v1:172.21.15.181:6789/0 conn(0x7f3d58006890 legacy=0x7f3d58006d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:15.147 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.073+0000 7f3d74ff9700 1 -- 172.21.15.102:0/4004403904 >> v1:172.21.15.181:6789/0 conn(0x7f3d58006890 legacy=0x7f3d58006d20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:08:15.148 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:08:15.148 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:08:15.148 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:08:15.148 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "leader", 2021-05-25T12:08:15.149 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 68, 2021-05-25T12:08:15.149 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:08:15.149 INFO:teuthology.orchestra.run.smithi102.stdout: 0, 2021-05-25T12:08:15.149 INFO:teuthology.orchestra.run.smithi102.stdout: 1, 2021-05-25T12:08:15.150 INFO:teuthology.orchestra.run.smithi102.stdout: 2 2021-05-25T12:08:15.150 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:08:15.150 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 3, 2021-05-25T12:08:15.150 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:08:15.151 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:08:15.151 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:08:15.151 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:08:15.151 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:08:15.152 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:08:15.152 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:08:15.152 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:08:15.152 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:08:15.152 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:08:15.153 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:08:15.153 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:08:15.153 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:08:15.153 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:08:15.154 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:08:15.154 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:08:15.154 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:08:15.154 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:08:15.155 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:08:15.155 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:08:15.155 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:08:15.155 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:08:15.155 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:08:15.156 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:08:15.156 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:15.156 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:15.156 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:08:15.156 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:08:15.157 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:08:15.157 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:08:15.158 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:08:15.158 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:08:15.158 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:08:15.159 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:08:15.159 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:08:15.159 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:08:15.160 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:08:15.160 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:08:15.161 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:08:15.161 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:08:15.161 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:08:15.162 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:08:15.162 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:08:15.163 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:08:15.163 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:08:15.164 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:08:15.164 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:08:15.164 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:08:15.164 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:08:15.165 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:08:15.165 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:08:15.165 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:08:15.165 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:15.166 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:08:15.166 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:15.166 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:08:15.166 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:08:15.167 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:08:15.167 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:08:15.167 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:15.167 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:08:15.168 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:08:15.168 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:08:15.168 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:15.168 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:15.168 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:15.169 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:08:15.169 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:08:15.169 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:08:15.169 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:08:15.170 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:08:15.170 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:15.170 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:15.170 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:08:15.171 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:08:15.171 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:08:15.171 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:08:15.171 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:15.171 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:08:15.172 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:08:15.172 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:08:15.172 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:15.172 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:15.173 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:15.173 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:08:15.173 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:08:15.173 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:08:15.174 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:08:15.174 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:08:15.174 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:15.174 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:15.175 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:08:15.175 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:08:15.175 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:08:15.175 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:08:15.175 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:15.176 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:08:15.176 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:08:15.176 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:08:15.176 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:15.177 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:15.177 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:15.177 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:08:15.177 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:08:15.178 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:08:15.178 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:08:15.178 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:08:15.178 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:15.179 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:15.179 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:15.179 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:08:15.179 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:08:15.179 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:15.180 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:08:15.180 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:08:15.180 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:08:15.180 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:15.180 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:08:15.181 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:08:15.181 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:15.181 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:08:15.181 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:08:15.182 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 4 2021-05-25T12:08:15.182 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:15.182 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:08:15.182 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": [ 2021-05-25T12:08:15.183 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:15.183 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:08:15.183 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:08:15.183 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:08:15.183 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:15.184 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:15.184 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:15.184 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:08:15.184 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:08:15.187 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.073+0000 7f3d86c41700 10 librados: watch_flush enter 2021-05-25T12:08:15.187 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.073+0000 7f3d86c41700 10 librados: watch_flush exit 2021-05-25T12:08:15.187 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.073+0000 7f3d86c41700 20 client.204552.objecter shutdown clearing up homeless session... 2021-05-25T12:08:15.188 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.073+0000 7f3d86c41700 10 client.204552.objecter successfully canceled tick 2021-05-25T12:08:15.188 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.073+0000 7f3d86c41700 1 -- 172.21.15.102:0/4004403904 >> v1:172.21.15.181:6812/29582 conn(0x7f3d60033530 legacy=0x7f3d600359f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:15.188 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.073+0000 7f3d86c41700 10 monclient: shutdown 2021-05-25T12:08:15.188 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.073+0000 7f3d86c41700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:08:15.189 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.073+0000 7f3d86c41700 1 -- 172.21.15.102:0/4004403904 >> v1:172.21.15.102:6789/0 conn(0x7f3d80093fc0 legacy=0x7f3d80128690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:15.189 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.073+0000 7f3d851de700 1 -- 172.21.15.102:0/4004403904 reap_dead start 2021-05-25T12:08:15.189 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.073+0000 7f3d851de700 1 -- 172.21.15.102:0/4004403904 reap_dead start 2021-05-25T12:08:15.189 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.073+0000 7f3d86c41700 1 -- 172.21.15.102:0/4004403904 shutdown_connections 2021-05-25T12:08:15.190 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.073+0000 7f3d86c41700 1 -- 172.21.15.102:0/4004403904 >> 172.21.15.102:0/4004403904 conn(0x7f3d8009b370 msgr2=0x7f3d80095ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:08:15.190 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.073+0000 7f3d86c41700 1 -- 172.21.15.102:0/4004403904 shutdown_connections 2021-05-25T12:08:15.190 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.073+0000 7f3d86c41700 1 -- 172.21.15.102:0/4004403904 wait complete. 2021-05-25T12:08:15.191 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.073+0000 7f3d86c41700 1 librados: shutdown 2021-05-25T12:08:15.192 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2021-05-25T12:08:15.192 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2021-05-25T12:08:15.193 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2021-05-25T12:08:15.193 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2021-05-25T12:08:15.194 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2021-05-25T12:08:15.195 INFO:teuthology.orchestra.run:waiting for 300 2021-05-25T12:08:15.207 INFO:tasks.ceph.mon.a:Stopped 2021-05-25T12:08:15.207 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2021-05-25T12:08:15.208 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2021-05-25T12:08:15.356 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.354+0000 7efd74825700 10 monclient: get_monmap_and_config 2021-05-25T12:08:15.356 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.354+0000 7efd74825700 10 monclient: build_initial_monmap 2021-05-25T12:08:15.356 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.354+0000 7efd74825700 10 monclient: monmap: 2021-05-25T12:08:15.357 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:08:15.357 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:15.357 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:08:15.355479+0000 2021-05-25T12:08:15.357 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:08:15.355479+0000 2021-05-25T12:08:15.358 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:08:15.358 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:15.358 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:08:15.358 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:08:15.358 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:08:15.359 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:15.359 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.355+0000 7efd74825700 1 Processor -- start 2021-05-25T12:08:15.359 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.355+0000 7efd74825700 1 -- start start 2021-05-25T12:08:15.359 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.355+0000 7efd74825700 10 monclient: init 2021-05-25T12:08:15.369 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.355+0000 7efd74825700 10 monclient: _reopen_session rank -1 2021-05-25T12:08:15.369 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.355+0000 7efd74825700 10 monclient: _add_conns ranks=[1,2,0] 2021-05-25T12:08:15.370 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.355+0000 7efd74825700 10 monclient(hunting): picked mon.noname-c con 0x7efd70061b20 addr v1:172.21.15.181:6789/0 2021-05-25T12:08:15.370 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.355+0000 7efd74825700 10 monclient(hunting): picked mon.noname-b con 0x7efd700954b0 addr v1:172.21.15.102:6790/0 2021-05-25T12:08:15.370 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.355+0000 7efd74825700 10 monclient(hunting): picked mon.noname-a con 0x7efd70095960 addr v1:172.21.15.102:6789/0 2021-05-25T12:08:15.371 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.355+0000 7efd74825700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd7009ad80 con 0x7efd70095960 2021-05-25T12:08:15.371 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.355+0000 7efd74825700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd7009b020 con 0x7efd70061b20 2021-05-25T12:08:15.371 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.355+0000 7efd74825700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd7009b3e0 con 0x7efd700954b0 2021-05-25T12:08:15.371 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.355+0000 7efd74825700 10 monclient(hunting): _renew_subs 2021-05-25T12:08:15.371 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.355+0000 7efd74825700 10 monclient(hunting): authenticate will time out at 1295.952636s 2021-05-25T12:08:15.372 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.355+0000 7efd6f59e700 1 -- >> v1:172.21.15.102:6789/0 conn(0x7efd70095960 legacy=0x7efd70134130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:08:15.372 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.355+0000 7efd6e59c700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7efd700954b0 0x7efd70130a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:57342/0 (socket says 172.21.15.102:57342) 2021-05-25T12:08:15.372 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.355+0000 7efd6e59c700 1 -- 172.21.15.102:0/978591088 learned_addr learned my addr 172.21.15.102:0/978591088 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:08:15.372 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.355+0000 7efd6dd9b700 1 -- 172.21.15.102:0/978591088 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2966842255 0 0) 0x7efd7009b3e0 con 0x7efd700954b0 2021-05-25T12:08:15.372 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.355+0000 7efd6dd9b700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:15.373 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.355+0000 7efd6dd9b700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:15.373 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.355+0000 7efd6dd9b700 10 monclient(hunting): my global_id is 244160 2021-05-25T12:08:15.373 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.355+0000 7efd6dd9b700 1 -- 172.21.15.102:0/978591088 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd5c003270 con 0x7efd700954b0 2021-05-25T12:08:15.373 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.356+0000 7efd6dd9b700 1 -- 172.21.15.102:0/978591088 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 818133869 0 0) 0x7efd5c003270 con 0x7efd700954b0 2021-05-25T12:08:15.374 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.356+0000 7efd6dd9b700 1 -- 172.21.15.102:0/978591088 >> v1:172.21.15.181:6789/0 conn(0x7efd70061b20 legacy=0x7efd70094da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:15.374 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.356+0000 7efd6dd9b700 1 -- 172.21.15.102:0/978591088 >> v1:172.21.15.102:6789/0 conn(0x7efd70095960 legacy=0x7efd70134130 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-05-25T12:08:15.374 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.356+0000 7efd6dd9b700 10 monclient: _finish_hunting 0 2021-05-25T12:08:15.374 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.356+0000 7efd6dd9b700 1 monclient: found mon.noname-b 2021-05-25T12:08:15.374 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.356+0000 7efd6dd9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:08:15.375 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.356+0000 7efd6dd9b700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T12:08:15.375 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.356+0000 7efd6dd9b700 1 -- 172.21.15.102:0/978591088 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd700695e0 con 0x7efd700954b0 2021-05-25T12:08:15.375 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.356+0000 7efd6dd9b700 10 monclient: _finish_auth 0 2021-05-25T12:08:15.375 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.356+0000 7efd6dd9b700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:08:15.376 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.356+0000 7efd6dd9b700 1 -- 172.21.15.102:0/978591088 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7efd58001610 con 0x7efd700954b0 2021-05-25T12:08:15.376 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.356+0000 7efd74825700 5 monclient: authenticate success, global_id 244160 2021-05-25T12:08:15.376 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.356+0000 7efd74825700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:08:15.376 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.356+0000 7efd6dd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:15.376 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.356+0000 7efd6dd9b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:08:15.377 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.356+0000 7efd6dd9b700 10 monclient: dump: 2021-05-25T12:08:15.377 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:15.377 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:15.377 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:15.378 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:15.378 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:15.378 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:15.378 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:15.378 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:15.379 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:15.379 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:15.379 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.356+0000 7efd74825700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:08:15.379 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.356+0000 7efd6dd9b700 1 -- 172.21.15.102:0/978591088 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efd58001930 con 0x7efd700954b0 2021-05-25T12:08:15.380 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.356+0000 7efd6dd9b700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:08:15.380 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.356+0000 7efd74825700 10 monclient: get_monmap_and_config success 2021-05-25T12:08:15.380 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.356+0000 7efd74825700 4 set_mon_vals no callback set 2021-05-25T12:08:15.380 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.356+0000 7efd6dd9b700 1 -- 172.21.15.102:0/978591088 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7efd58001d00 con 0x7efd700954b0 2021-05-25T12:08:15.380 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.356+0000 7efd6dd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:15.381 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.356+0000 7efd6dd9b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:08:15.381 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.356+0000 7efd6dd9b700 10 monclient: dump: 2021-05-25T12:08:15.381 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:15.381 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:15.381 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:15.382 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:15.382 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:15.382 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:15.382 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:15.383 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:15.383 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:15.383 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:15.383 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.356+0000 7efd74825700 10 monclient: shutdown 2021-05-25T12:08:15.383 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.356+0000 7efd74825700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:08:15.384 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.356+0000 7efd74825700 1 -- 172.21.15.102:0/978591088 >> v1:172.21.15.102:6790/0 conn(0x7efd700954b0 legacy=0x7efd70130a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:15.384 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.356+0000 7efd74825700 1 -- 172.21.15.102:0/978591088 shutdown_connections 2021-05-25T12:08:15.384 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.356+0000 7efd74825700 1 -- 172.21.15.102:0/978591088 >> 172.21.15.102:0/978591088 conn(0x7efd70098300 msgr2=0x7efd70098710 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:08:15.384 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.356+0000 7efd74825700 1 -- 172.21.15.102:0/978591088 shutdown_connections 2021-05-25T12:08:15.384 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.356+0000 7efd74825700 1 -- 172.21.15.102:0/978591088 wait complete. 2021-05-25T12:08:15.385 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.357+0000 7efd74825700 10 monclient: build_initial_monmap 2021-05-25T12:08:15.385 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.357+0000 7efd74825700 10 monclient: monmap: 2021-05-25T12:08:15.385 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:08:15.385 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:08:15.386 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:08:15.386 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:08:15.386 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:08:15.386 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:08:15.386 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:08:15.387 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:08:15.387 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:08:15.387 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:15.387 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.357+0000 7efd74825700 1 librados: starting msgr at 2021-05-25T12:08:15.387 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.357+0000 7efd74825700 1 librados: starting objecter 2021-05-25T12:08:15.388 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.357+0000 7efd74825700 1 Processor -- start 2021-05-25T12:08:15.388 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.357+0000 7efd74825700 1 -- start start 2021-05-25T12:08:15.388 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.357+0000 7efd74825700 1 librados: setting wanted keys 2021-05-25T12:08:15.388 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.357+0000 7efd74825700 1 librados: calling monclient init 2021-05-25T12:08:15.389 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.357+0000 7efd74825700 10 monclient: init 2021-05-25T12:08:15.389 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.358+0000 7efd74825700 10 monclient: _reopen_session rank -1 2021-05-25T12:08:15.389 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.358+0000 7efd74825700 10 monclient: _add_conns ranks=[1,2,0] 2021-05-25T12:08:15.389 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.358+0000 7efd74825700 10 monclient(hunting): picked mon.noname-b con 0x7efd70061b20 addr v1:172.21.15.181:6789/0 2021-05-25T12:08:15.389 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.358+0000 7efd74825700 10 monclient(hunting): picked mon.noname-c con 0x7efd700954b0 addr v1:172.21.15.102:6790/0 2021-05-25T12:08:15.390 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.358+0000 7efd74825700 10 monclient(hunting): picked mon.noname-a con 0x7efd70095960 addr v1:172.21.15.102:6789/0 2021-05-25T12:08:15.390 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.358+0000 7efd74825700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd70122080 con 0x7efd70095960 2021-05-25T12:08:15.390 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.358+0000 7efd74825700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd70123e70 con 0x7efd70061b20 2021-05-25T12:08:15.390 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.358+0000 7efd74825700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd70124110 con 0x7efd700954b0 2021-05-25T12:08:15.391 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.358+0000 7efd74825700 10 monclient(hunting): _renew_subs 2021-05-25T12:08:15.391 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.358+0000 7efd74825700 10 monclient(hunting): authenticate will time out at 1295.955235s 2021-05-25T12:08:15.391 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.358+0000 7efd6f59e700 1 -- >> v1:172.21.15.102:6789/0 conn(0x7efd70095960 legacy=0x7efd70142fa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:08:15.391 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.358+0000 7efd6e59c700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7efd700954b0 0x7efd701255c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:57348/0 (socket says 172.21.15.102:57348) 2021-05-25T12:08:15.391 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.358+0000 7efd6e59c700 1 -- 172.21.15.102:0/2538876742 learned_addr learned my addr 172.21.15.102:0/2538876742 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:08:15.392 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.358+0000 7efd56ffd700 10 client.?.objecter ms_handle_connect 0x7efd700954b0 2021-05-25T12:08:15.392 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.358+0000 7efd56ffd700 10 client.?.objecter resend_mon_ops 2021-05-25T12:08:15.392 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.358+0000 7efd56ffd700 10 client.?.objecter ms_handle_connect 0x7efd70061b20 2021-05-25T12:08:15.392 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.358+0000 7efd56ffd700 10 client.?.objecter resend_mon_ops 2021-05-25T12:08:15.393 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.358+0000 7efd56ffd700 1 -- 172.21.15.102:0/2538876742 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3745252693 0 0) 0x7efd70124110 con 0x7efd700954b0 2021-05-25T12:08:15.393 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.358+0000 7efd56ffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:15.393 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.358+0000 7efd56ffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:15.393 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.358+0000 7efd56ffd700 10 monclient(hunting): my global_id is 244163 2021-05-25T12:08:15.393 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.358+0000 7efd56ffd700 1 -- 172.21.15.102:0/2538876742 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd50004c10 con 0x7efd700954b0 2021-05-25T12:08:15.394 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.358+0000 7efd56ffd700 1 -- 172.21.15.102:0/2538876742 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3483570467 0 0) 0x7efd70123e70 con 0x7efd70061b20 2021-05-25T12:08:15.394 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.358+0000 7efd56ffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:15.394 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.358+0000 7efd56ffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:15.394 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.358+0000 7efd56ffd700 10 monclient(hunting): my global_id is 224419 2021-05-25T12:08:15.395 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.358+0000 7efd56ffd700 1 -- 172.21.15.102:0/2538876742 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd70124110 con 0x7efd70061b20 2021-05-25T12:08:15.395 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.358+0000 7efd56ffd700 1 -- 172.21.15.102:0/2538876742 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 109857290 0 0) 0x7efd50004c10 con 0x7efd700954b0 2021-05-25T12:08:15.395 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.358+0000 7efd56ffd700 1 -- 172.21.15.102:0/2538876742 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efd70123e70 con 0x7efd700954b0 2021-05-25T12:08:15.395 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.358+0000 7efd56ffd700 1 -- 172.21.15.102:0/2538876742 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7efd58001c40 con 0x7efd700954b0 2021-05-25T12:08:15.396 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.358+0000 7efd56ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:15.396 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.358+0000 7efd56ffd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:08:15.396 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.358+0000 7efd56ffd700 10 monclient(hunting): dump: 2021-05-25T12:08:15.396 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:15.397 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:15.397 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:15.397 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:15.397 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:15.397 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:15.398 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:15.398 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:15.398 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:15.398 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:15.398 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.358+0000 7efd56ffd700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:08:15.399 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.358+0000 7efd56ffd700 1 -- 172.21.15.102:0/2538876742 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 244503542 0 0) 0x7efd70123e70 con 0x7efd700954b0 2021-05-25T12:08:15.399 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.358+0000 7efd56ffd700 1 -- 172.21.15.102:0/2538876742 >> v1:172.21.15.181:6789/0 conn(0x7efd70061b20 legacy=0x7efd7013ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:15.399 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.358+0000 7efd56ffd700 1 -- 172.21.15.102:0/2538876742 >> v1:172.21.15.102:6789/0 conn(0x7efd70095960 legacy=0x7efd70142fa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-05-25T12:08:15.399 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.358+0000 7efd56ffd700 10 monclient: _finish_hunting 0 2021-05-25T12:08:15.400 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.358+0000 7efd56ffd700 1 monclient: found mon.c 2021-05-25T12:08:15.400 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.358+0000 7efd56ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:08:15.400 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.358+0000 7efd56ffd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:08:15.400 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.358+0000 7efd56ffd700 1 -- 172.21.15.102:0/2538876742 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd70099e50 con 0x7efd700954b0 2021-05-25T12:08:15.400 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.358+0000 7efd56ffd700 10 monclient: _finish_auth 0 2021-05-25T12:08:15.401 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.358+0000 7efd56ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:08:15.401 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.358+0000 7efd74825700 5 monclient: authenticate success, global_id 244163 2021-05-25T12:08:15.401 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.358+0000 7efd74825700 10 monclient: _renew_subs 2021-05-25T12:08:15.401 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.358+0000 7efd74825700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:08:15.401 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.358+0000 7efd74825700 1 -- 172.21.15.102:0/2538876742 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd70125cd0 con 0x7efd700954b0 2021-05-25T12:08:15.402 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.358+0000 7efd74825700 10 client.244163.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:08:15.402 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.358+0000 7efd74825700 10 monclient: _renew_subs 2021-05-25T12:08:15.402 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.358+0000 7efd74825700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:08:15.402 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.358+0000 7efd74825700 1 -- 172.21.15.102:0/2538876742 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efd7013b320 con 0x7efd700954b0 2021-05-25T12:08:15.403 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.358+0000 7efd74825700 1 librados: init done 2021-05-25T12:08:15.403 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.358+0000 7efd56ffd700 1 -- 172.21.15.102:0/2538876742 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efd58004a70 con 0x7efd700954b0 2021-05-25T12:08:15.403 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.358+0000 7efd56ffd700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:08:15.403 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.358+0000 7efd56ffd700 1 -- 172.21.15.102:0/2538876742 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7efd58004e00 con 0x7efd700954b0 2021-05-25T12:08:15.403 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.358+0000 7efd56ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:15.404 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.359+0000 7efd56ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:08:15.404 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.358+0000 7efd6d59a700 4 set_mon_vals no callback set 2021-05-25T12:08:15.404 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.359+0000 7efd56ffd700 10 monclient: dump: 2021-05-25T12:08:15.404 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:15.404 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:15.405 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:15.405 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:15.405 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:15.405 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:15.405 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:15.406 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:15.406 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:15.406 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:15.406 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.359+0000 7efd56ffd700 1 -- 172.21.15.102:0/2538876742 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7efd58004fe0 con 0x7efd700954b0 2021-05-25T12:08:15.407 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.359+0000 7efd56ffd700 1 -- 172.21.15.102:0/2538876742 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(271..271 src has 1..271) v4 ==== 5542+0+0 (unknown 782829896 0 0) 0x7efd5803d920 con 0x7efd700954b0 2021-05-25T12:08:15.407 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.359+0000 7efd56ffd700 10 client.244163.objecter ms_dispatch 0x7efd70001550 osd_map(271..271 src has 1..271) v4 2021-05-25T12:08:15.407 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.359+0000 7efd56ffd700 3 client.244163.objecter handle_osd_map got epochs [271,271] > 0 2021-05-25T12:08:15.407 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.359+0000 7efd56ffd700 3 client.244163.objecter handle_osd_map decoding full epoch 271 2021-05-25T12:08:15.408 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.359+0000 7efd56ffd700 20 client.244163.objecter dump_active .. 0 homeless 2021-05-25T12:08:15.408 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.359+0000 7efd6ed9d700 10 monclient: get_auth_request con 0x7efd50033150 auth_method 0 2021-05-25T12:08:15.408 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.359+0000 7efd74825700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:08:15.408 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.359+0000 7efd74825700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:08:15.408 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.359+0000 7efd74825700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:08:15.409 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.359+0000 7efd74825700 1 -- 172.21.15.102:0/2538876742 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd48004db0 con 0x7efd700954b0 2021-05-25T12:08:15.409 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.360+0000 7efd56ffd700 10 client.244163.objecter ms_handle_connect 0x7efd50033150 2021-05-25T12:08:15.409 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.363+0000 7efd56ffd700 1 -- 172.21.15.102:0/2538876742 <== mon.2 v1:172.21.15.102:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7efd58012ff0 con 0x7efd700954b0 2021-05-25T12:08:15.409 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.363+0000 7efd56ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:08:15.410 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.363+0000 7efd56ffd700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:08:15.576 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.575+0000 7efd74825700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2021-05-25T12:08:15.577 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.575+0000 7efd74825700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2021-05-25T12:08:15.577 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.575+0000 7efd74825700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:08:15.577 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.575+0000 7efd74825700 1 -- 172.21.15.102:0/2538876742 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7efd480055e0 con 0x7efd700954b0 2021-05-25T12:08:15.577 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.575+0000 7efd56ffd700 1 -- 172.21.15.102:0/2538876742 <== mon.2 v1:172.21.15.102:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1342 (unknown 254746657 0 2161719127) 0x7efd58012e10 con 0x7efd700954b0 2021-05-25T12:08:15.578 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.575+0000 7efd56ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2021-05-25T12:08:15.578 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.575+0000 7efd56ffd700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:08:15.579 INFO:teuthology.orchestra.run.smithi102.stdout:{"election_epoch":68,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"b","quorum_age":3,"features":{"quorum_con":"4540138303579357183","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"20ee91b6-8935-4b98-b868-85c03789ccc4","modified":"2021-05-25T11:59:11.011522Z","created":"2021-05-25T11:59:11.011522Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6790","nonce":0}]},"addr":"172.21.15.102:6790/0","public_addr":"172.21.15.102:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2021-05-25T12:08:15.580 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.576+0000 7efd74825700 10 librados: watch_flush enter 2021-05-25T12:08:15.581 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.576+0000 7efd74825700 10 librados: watch_flush exit 2021-05-25T12:08:15.581 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.576+0000 7efd74825700 20 client.244163.objecter shutdown clearing up homeless session... 2021-05-25T12:08:15.581 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.577+0000 7efd74825700 10 client.244163.objecter successfully canceled tick 2021-05-25T12:08:15.581 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.577+0000 7efd74825700 1 -- 172.21.15.102:0/2538876742 >> v1:172.21.15.181:6812/29582 conn(0x7efd50033150 legacy=0x7efd50035610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:15.582 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.577+0000 7efd74825700 10 monclient: shutdown 2021-05-25T12:08:15.582 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.577+0000 7efd74825700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:08:15.582 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.577+0000 7efd74825700 1 -- 172.21.15.102:0/2538876742 >> v1:172.21.15.102:6790/0 conn(0x7efd700954b0 legacy=0x7efd701255c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:15.582 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.577+0000 7efd74825700 1 -- 172.21.15.102:0/2538876742 shutdown_connections 2021-05-25T12:08:15.582 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.577+0000 7efd74825700 1 -- 172.21.15.102:0/2538876742 >> 172.21.15.102:0/2538876742 conn(0x7efd70098300 msgr2=0x7efd70097100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:08:15.583 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.577+0000 7efd74825700 1 -- 172.21.15.102:0/2538876742 shutdown_connections 2021-05-25T12:08:15.583 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.577+0000 7efd74825700 1 -- 172.21.15.102:0/2538876742 wait complete. 2021-05-25T12:08:15.583 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:15.577+0000 7efd74825700 1 librados: shutdown 2021-05-25T12:08:18.590 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2021-05-25T12:08:18.714 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.712+0000 7f845ebe0700 10 monclient: get_monmap_and_config 2021-05-25T12:08:18.714 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.712+0000 7f845ebe0700 10 monclient: build_initial_monmap 2021-05-25T12:08:18.715 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.713+0000 7f845ebe0700 10 monclient: monmap: 2021-05-25T12:08:18.715 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:08:18.715 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:18.715 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:08:18.713895+0000 2021-05-25T12:08:18.715 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:08:18.713895+0000 2021-05-25T12:08:18.716 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:08:18.716 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:18.716 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:08:18.716 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:08:18.716 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:08:18.717 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:18.726 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.713+0000 7f845ebe0700 1 Processor -- start 2021-05-25T12:08:18.727 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.713+0000 7f845ebe0700 1 -- start start 2021-05-25T12:08:18.727 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.713+0000 7f845ebe0700 10 monclient: init 2021-05-25T12:08:18.727 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.714+0000 7f845ebe0700 10 monclient: _reopen_session rank -1 2021-05-25T12:08:18.728 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.714+0000 7f845ebe0700 10 monclient: _add_conns ranks=[0,1,2] 2021-05-25T12:08:18.728 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.714+0000 7f845ebe0700 10 monclient(hunting): picked mon.noname-a con 0x7f8458068d50 addr v1:172.21.15.102:6789/0 2021-05-25T12:08:18.728 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.714+0000 7f845ebe0700 10 monclient(hunting): picked mon.noname-c con 0x7f8458069130 addr v1:172.21.15.181:6789/0 2021-05-25T12:08:18.728 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.714+0000 7f845ebe0700 10 monclient(hunting): picked mon.noname-b con 0x7f84580695e0 addr v1:172.21.15.102:6790/0 2021-05-25T12:08:18.729 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.714+0000 7f845ebe0700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8458094380 con 0x7f8458068d50 2021-05-25T12:08:18.729 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.714+0000 7f845ebe0700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8458094620 con 0x7f8458069130 2021-05-25T12:08:18.729 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.714+0000 7f845ebe0700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84580948c0 con 0x7f84580695e0 2021-05-25T12:08:18.730 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.714+0000 7f845ebe0700 10 monclient(hunting): _renew_subs 2021-05-25T12:08:18.730 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.714+0000 7f845ebe0700 10 monclient(hunting): authenticate will time out at 1299.311374s 2021-05-25T12:08:18.730 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.714+0000 7f845c97c700 1 -- >> v1:172.21.15.102:6789/0 conn(0x7f8458068d50 legacy=0x7f8458129c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:08:18.730 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.714+0000 7f845d17d700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f84580695e0 0x7f8458132160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:57366/0 (socket says 172.21.15.102:57366) 2021-05-25T12:08:18.730 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.714+0000 7f845d17d700 1 -- 172.21.15.102:0/790488450 learned_addr learned my addr 172.21.15.102:0/790488450 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:08:18.731 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.714+0000 7f84577fe700 1 -- 172.21.15.102:0/790488450 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3406778176 0 0) 0x7f8458094620 con 0x7f8458069130 2021-05-25T12:08:18.731 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.714+0000 7f84577fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:18.731 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.714+0000 7f84577fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:18.731 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.714+0000 7f84577fe700 10 monclient(hunting): my global_id is 224422 2021-05-25T12:08:18.731 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.715+0000 7f84577fe700 1 -- 172.21.15.102:0/790488450 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8444003270 con 0x7f8458069130 2021-05-25T12:08:18.732 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.715+0000 7f84577fe700 1 -- 172.21.15.102:0/790488450 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 538006981 0 0) 0x7f8444003270 con 0x7f8458069130 2021-05-25T12:08:18.732 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.715+0000 7f84577fe700 1 -- 172.21.15.102:0/790488450 >> v1:172.21.15.102:6790/0 conn(0x7f84580695e0 legacy=0x7f8458132160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:18.732 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.715+0000 7f84577fe700 1 -- 172.21.15.102:0/790488450 >> v1:172.21.15.102:6789/0 conn(0x7f8458068d50 legacy=0x7f8458129c90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-05-25T12:08:18.733 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.715+0000 7f84577fe700 10 monclient: _finish_hunting 0 2021-05-25T12:08:18.733 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.715+0000 7f84577fe700 1 monclient: found mon.noname-c 2021-05-25T12:08:18.733 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.715+0000 7f84577fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:08:18.733 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.715+0000 7f84577fe700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.181:6789/0 2021-05-25T12:08:18.733 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.715+0000 7f84577fe700 1 -- 172.21.15.102:0/790488450 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8458094c80 con 0x7f8458069130 2021-05-25T12:08:18.734 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.715+0000 7f84577fe700 10 monclient: _finish_auth 0 2021-05-25T12:08:18.734 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.715+0000 7f84577fe700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:08:18.734 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.715+0000 7f84577fe700 1 -- 172.21.15.102:0/790488450 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f8440001610 con 0x7f8458069130 2021-05-25T12:08:18.734 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.715+0000 7f845ebe0700 5 monclient: authenticate success, global_id 224422 2021-05-25T12:08:18.735 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.715+0000 7f845ebe0700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:08:18.735 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.715+0000 7f84577fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:18.735 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.715+0000 7f84577fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:08:18.735 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.715+0000 7f84577fe700 10 monclient: dump: 2021-05-25T12:08:18.735 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:18.736 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:18.736 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:18.736 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:18.736 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:18.736 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:18.737 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:18.737 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:18.737 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:18.737 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:18.738 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.715+0000 7f845ebe0700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:08:18.738 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.715+0000 7f84577fe700 1 -- 172.21.15.102:0/790488450 <== mon.1 v1:172.21.15.181:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8440001930 con 0x7f8458069130 2021-05-25T12:08:18.738 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.715+0000 7f84577fe700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:08:18.738 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.715+0000 7f845ebe0700 10 monclient: get_monmap_and_config success 2021-05-25T12:08:18.738 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.715+0000 7f845ebe0700 4 set_mon_vals no callback set 2021-05-25T12:08:18.739 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.715+0000 7f84577fe700 1 -- 172.21.15.102:0/790488450 <== mon.1 v1:172.21.15.181:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f8440001d00 con 0x7f8458069130 2021-05-25T12:08:18.740 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.715+0000 7f84577fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:18.740 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.715+0000 7f84577fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T12:08:18.740 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.715+0000 7f84577fe700 10 monclient: dump: 2021-05-25T12:08:18.740 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:18.740 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:18.741 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:18.741 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:18.741 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:18.741 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:18.742 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:18.742 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:18.742 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:18.742 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:18.742 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.715+0000 7f845ebe0700 10 monclient: shutdown 2021-05-25T12:08:18.743 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.715+0000 7f845ebe0700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:08:18.743 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.715+0000 7f845ebe0700 1 -- 172.21.15.102:0/790488450 >> v1:172.21.15.181:6789/0 conn(0x7f8458069130 legacy=0x7f845812ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:18.743 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.715+0000 7f845ebe0700 1 -- 172.21.15.102:0/790488450 shutdown_connections 2021-05-25T12:08:18.743 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.715+0000 7f845ebe0700 1 -- 172.21.15.102:0/790488450 >> 172.21.15.102:0/790488450 conn(0x7f845809b370 msgr2=0x7f8458097430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:08:18.743 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.715+0000 7f845ebe0700 1 -- 172.21.15.102:0/790488450 shutdown_connections 2021-05-25T12:08:18.744 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.715+0000 7f845ebe0700 1 -- 172.21.15.102:0/790488450 wait complete. 2021-05-25T12:08:18.744 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.716+0000 7f845ebe0700 10 monclient: build_initial_monmap 2021-05-25T12:08:18.744 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.716+0000 7f845ebe0700 10 monclient: monmap: 2021-05-25T12:08:18.744 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:08:18.745 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:08:18.745 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:08:18.745 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:08:18.745 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:08:18.745 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:08:18.746 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:08:18.746 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:08:18.746 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:08:18.746 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:18.746 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.716+0000 7f845ebe0700 1 librados: starting msgr at 2021-05-25T12:08:18.747 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.716+0000 7f845ebe0700 1 librados: starting objecter 2021-05-25T12:08:18.747 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.716+0000 7f845ebe0700 1 Processor -- start 2021-05-25T12:08:18.747 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.717+0000 7f845ebe0700 1 -- start start 2021-05-25T12:08:18.747 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.717+0000 7f845ebe0700 1 librados: setting wanted keys 2021-05-25T12:08:18.747 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.717+0000 7f845ebe0700 1 librados: calling monclient init 2021-05-25T12:08:18.748 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.717+0000 7f845ebe0700 10 monclient: init 2021-05-25T12:08:18.748 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.717+0000 7f845ebe0700 10 monclient: _reopen_session rank -1 2021-05-25T12:08:18.748 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.717+0000 7f845ebe0700 10 monclient: _add_conns ranks=[0,1,2] 2021-05-25T12:08:18.748 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.717+0000 7f845ebe0700 10 monclient(hunting): picked mon.noname-a con 0x7f8458068d50 addr v1:172.21.15.102:6789/0 2021-05-25T12:08:18.749 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.717+0000 7f845ebe0700 10 monclient(hunting): picked mon.noname-b con 0x7f8458069130 addr v1:172.21.15.181:6789/0 2021-05-25T12:08:18.749 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.717+0000 7f845ebe0700 10 monclient(hunting): picked mon.noname-c con 0x7f84580695e0 addr v1:172.21.15.102:6790/0 2021-05-25T12:08:18.749 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.717+0000 7f845ebe0700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8458120ff0 con 0x7f8458068d50 2021-05-25T12:08:18.750 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.717+0000 7f845ebe0700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8458091240 con 0x7f8458069130 2021-05-25T12:08:18.750 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.717+0000 7f845ebe0700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8458122de0 con 0x7f84580695e0 2021-05-25T12:08:18.750 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.717+0000 7f845c97c700 1 -- >> v1:172.21.15.102:6789/0 conn(0x7f8458068d50 legacy=0x7f8458138ed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:08:18.750 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.717+0000 7f845ebe0700 10 monclient(hunting): _renew_subs 2021-05-25T12:08:18.751 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.717+0000 7f845ebe0700 10 monclient(hunting): authenticate will time out at 1299.314524s 2021-05-25T12:08:18.751 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.717+0000 7f845d17d700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f84580695e0 0x7f8458141080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:57374/0 (socket says 172.21.15.102:57374) 2021-05-25T12:08:18.751 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.717+0000 7f845d17d700 1 -- 172.21.15.102:0/1426928066 learned_addr learned my addr 172.21.15.102:0/1426928066 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:08:18.751 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.717+0000 7f8454ff9700 10 client.?.objecter ms_handle_connect 0x7f84580695e0 2021-05-25T12:08:18.752 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.717+0000 7f8454ff9700 10 client.?.objecter resend_mon_ops 2021-05-25T12:08:18.752 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.717+0000 7f8454ff9700 10 client.?.objecter ms_handle_connect 0x7f8458069130 2021-05-25T12:08:18.752 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.717+0000 7f8454ff9700 10 client.?.objecter resend_mon_ops 2021-05-25T12:08:18.752 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.718+0000 7f8454ff9700 1 -- 172.21.15.102:0/1426928066 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1875355390 0 0) 0x7f8458091240 con 0x7f8458069130 2021-05-25T12:08:18.753 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.718+0000 7f8454ff9700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:18.753 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.718+0000 7f8454ff9700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:18.753 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.718+0000 7f8454ff9700 10 monclient(hunting): my global_id is 224425 2021-05-25T12:08:18.753 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.718+0000 7f8454ff9700 1 -- 172.21.15.102:0/1426928066 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8438004c10 con 0x7f8458069130 2021-05-25T12:08:18.753 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.718+0000 7f8454ff9700 1 -- 172.21.15.102:0/1426928066 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1545533401 0 0) 0x7f8438004c10 con 0x7f8458069130 2021-05-25T12:08:18.754 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.718+0000 7f8454ff9700 1 -- 172.21.15.102:0/1426928066 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8458091240 con 0x7f8458069130 2021-05-25T12:08:18.754 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.718+0000 7f8454ff9700 1 -- 172.21.15.102:0/1426928066 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f8440001d00 con 0x7f8458069130 2021-05-25T12:08:18.754 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.718+0000 7f8454ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:18.754 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.718+0000 7f8454ff9700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:08:18.754 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.718+0000 7f8454ff9700 10 monclient(hunting): dump: 2021-05-25T12:08:18.755 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:18.755 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:18.755 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:18.755 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:18.756 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:18.756 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:18.756 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:18.756 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:18.756 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:18.757 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:18.757 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.718+0000 7f8454ff9700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:08:18.757 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.718+0000 7f8454ff9700 1 -- 172.21.15.102:0/1426928066 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 290038656 0 0) 0x7f8458091240 con 0x7f8458069130 2021-05-25T12:08:18.757 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.718+0000 7f8454ff9700 1 -- 172.21.15.102:0/1426928066 >> v1:172.21.15.102:6790/0 conn(0x7f84580695e0 legacy=0x7f8458141080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:18.757 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.718+0000 7f8454ff9700 1 -- 172.21.15.102:0/1426928066 >> v1:172.21.15.102:6789/0 conn(0x7f8458068d50 legacy=0x7f8458138ed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-05-25T12:08:18.758 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.718+0000 7f8454ff9700 10 monclient: _finish_hunting 0 2021-05-25T12:08:18.758 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.718+0000 7f8454ff9700 1 monclient: found mon.b 2021-05-25T12:08:18.758 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.718+0000 7f8454ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:08:18.758 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.718+0000 7f8454ff9700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T12:08:18.759 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.718+0000 7f8454ff9700 1 -- 172.21.15.102:0/1426928066 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8458123080 con 0x7f8458069130 2021-05-25T12:08:18.759 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.718+0000 7f8454ff9700 10 monclient: _finish_auth 0 2021-05-25T12:08:18.759 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.718+0000 7f8454ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:08:18.759 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.718+0000 7f845ebe0700 5 monclient: authenticate success, global_id 224425 2021-05-25T12:08:18.759 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.718+0000 7f845ebe0700 10 monclient: _renew_subs 2021-05-25T12:08:18.760 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.718+0000 7f845ebe0700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T12:08:18.760 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.718+0000 7f845ebe0700 1 -- 172.21.15.102:0/1426928066 --> v1:172.21.15.181:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84581395e0 con 0x7f8458069130 2021-05-25T12:08:18.760 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.718+0000 7f845ebe0700 10 client.224425.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:08:18.760 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.718+0000 7f845ebe0700 10 monclient: _renew_subs 2021-05-25T12:08:18.760 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.718+0000 7f845ebe0700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T12:08:18.761 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.718+0000 7f845ebe0700 1 -- 172.21.15.102:0/1426928066 --> v1:172.21.15.181:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8458124e80 con 0x7f8458069130 2021-05-25T12:08:18.761 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.718+0000 7f845ebe0700 1 librados: init done 2021-05-25T12:08:18.761 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.718+0000 7f8454ff9700 1 -- 172.21.15.102:0/1426928066 <== mon.1 v1:172.21.15.181:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8440001230 con 0x7f8458069130 2021-05-25T12:08:18.761 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.718+0000 7f8454ff9700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:08:18.761 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.718+0000 7f8454ff9700 1 -- 172.21.15.102:0/1426928066 <== mon.1 v1:172.21.15.181:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f8440002640 con 0x7f8458069130 2021-05-25T12:08:18.762 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.718+0000 7f8454ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:18.762 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.718+0000 7f8454ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T12:08:18.762 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.718+0000 7f8456ffd700 4 set_mon_vals no callback set 2021-05-25T12:08:18.762 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.718+0000 7f8454ff9700 10 monclient: dump: 2021-05-25T12:08:18.763 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:18.763 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:18.763 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:18.763 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:18.763 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:18.764 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:18.764 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:18.764 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:18.764 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:18.764 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:18.765 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.719+0000 7f8454ff9700 1 -- 172.21.15.102:0/1426928066 <== mon.1 v1:172.21.15.181:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f8440002820 con 0x7f8458069130 2021-05-25T12:08:18.765 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.719+0000 7f845ebe0700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:08:18.765 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.719+0000 7f845ebe0700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:08:18.765 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.719+0000 7f845ebe0700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T12:08:18.765 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.719+0000 7f845ebe0700 1 -- 172.21.15.102:0/1426928066 --> v1:172.21.15.181:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8430004db0 con 0x7f8458069130 2021-05-25T12:08:18.766 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.719+0000 7f8454ff9700 1 -- 172.21.15.102:0/1426928066 <== mon.1 v1:172.21.15.181:6789/0 8 ==== osd_map(271..271 src has 1..271) v4 ==== 5542+0+0 (unknown 782829896 0 0) 0x7f844003b340 con 0x7f8458069130 2021-05-25T12:08:18.766 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.719+0000 7f8454ff9700 10 client.224425.objecter ms_dispatch 0x7f8458001550 osd_map(271..271 src has 1..271) v4 2021-05-25T12:08:18.766 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.719+0000 7f8454ff9700 3 client.224425.objecter handle_osd_map got epochs [271,271] > 0 2021-05-25T12:08:18.766 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.719+0000 7f8454ff9700 3 client.224425.objecter handle_osd_map decoding full epoch 271 2021-05-25T12:08:18.767 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.719+0000 7f8454ff9700 20 client.224425.objecter dump_active .. 0 homeless 2021-05-25T12:08:18.767 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.719+0000 7f845c97c700 10 monclient: get_auth_request con 0x7f8438033010 auth_method 0 2021-05-25T12:08:18.767 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.719+0000 7f8454ff9700 10 client.224425.objecter ms_handle_connect 0x7f8438033010 2021-05-25T12:08:18.767 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.724+0000 7f8454ff9700 1 -- 172.21.15.102:0/1426928066 <== mon.1 v1:172.21.15.181:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7f8440003e60 con 0x7f8458069130 2021-05-25T12:08:18.767 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.724+0000 7f8454ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:08:18.768 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.724+0000 7f8454ff9700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:08:18.934 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.932+0000 7f845ebe0700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2021-05-25T12:08:18.935 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.932+0000 7f845ebe0700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2021-05-25T12:08:18.935 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.932+0000 7f845ebe0700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T12:08:18.935 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.932+0000 7f845ebe0700 1 -- 172.21.15.102:0/1426928066 --> v1:172.21.15.181:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f84300055e0 con 0x7f8458069130 2021-05-25T12:08:18.936 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.933+0000 7f8454ff9700 1 -- 172.21.15.102:0/1426928066 <== mon.1 v1:172.21.15.181:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1342 (unknown 254746657 0 3960880742) 0x7f8440010780 con 0x7f8458069130 2021-05-25T12:08:18.936 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.933+0000 7f8454ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2021-05-25T12:08:18.936 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.933+0000 7f8454ff9700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:08:18.936 INFO:teuthology.orchestra.run.smithi102.stdout:{"election_epoch":68,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"b","quorum_age":7,"features":{"quorum_con":"4540138303579357183","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"20ee91b6-8935-4b98-b868-85c03789ccc4","modified":"2021-05-25T11:59:11.011522Z","created":"2021-05-25T11:59:11.011522Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6790","nonce":0}]},"addr":"172.21.15.102:6790/0","public_addr":"172.21.15.102:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2021-05-25T12:08:18.938 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.934+0000 7f845ebe0700 10 librados: watch_flush enter 2021-05-25T12:08:18.939 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.934+0000 7f845ebe0700 10 librados: watch_flush exit 2021-05-25T12:08:18.939 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.934+0000 7f845ebe0700 20 client.224425.objecter shutdown clearing up homeless session... 2021-05-25T12:08:18.939 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.934+0000 7f845ebe0700 10 client.224425.objecter successfully canceled tick 2021-05-25T12:08:18.939 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.934+0000 7f845ebe0700 1 -- 172.21.15.102:0/1426928066 >> v1:172.21.15.181:6812/29582 conn(0x7f8438033010 legacy=0x7f84380354d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:18.940 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.934+0000 7f845ebe0700 10 monclient: shutdown 2021-05-25T12:08:18.940 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.934+0000 7f845ebe0700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:08:18.941 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.934+0000 7f845ebe0700 1 -- 172.21.15.102:0/1426928066 >> v1:172.21.15.181:6789/0 conn(0x7f8458069130 legacy=0x7f8458124530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:18.941 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.934+0000 7f845ebe0700 1 -- 172.21.15.102:0/1426928066 shutdown_connections 2021-05-25T12:08:18.941 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.934+0000 7f845ebe0700 1 -- 172.21.15.102:0/1426928066 >> 172.21.15.102:0/1426928066 conn(0x7f845809b370 msgr2=0x7f84581254d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:08:18.941 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.934+0000 7f845ebe0700 1 -- 172.21.15.102:0/1426928066 shutdown_connections 2021-05-25T12:08:18.942 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.934+0000 7f845ebe0700 1 -- 172.21.15.102:0/1426928066 wait complete. 2021-05-25T12:08:18.942 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:18.935+0000 7f845ebe0700 1 librados: shutdown 2021-05-25T12:08:21.948 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2021-05-25T12:08:22.065 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:22.064+0000 7fbbb3979700 10 monclient: get_monmap_and_config 2021-05-25T12:08:22.066 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:22.064+0000 7fbbb3979700 10 monclient: build_initial_monmap 2021-05-25T12:08:22.066 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:22.064+0000 7fbbb3979700 10 monclient: monmap: 2021-05-25T12:08:22.066 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:08:22.066 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:22.067 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:08:22.064966+0000 2021-05-25T12:08:22.067 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:08:22.064966+0000 2021-05-25T12:08:22.067 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:08:22.067 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:22.068 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:08:22.068 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:08:22.068 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:08:22.068 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:22.068 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:22.064+0000 7fbbb3979700 1 Processor -- start 2021-05-25T12:08:22.069 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:22.064+0000 7fbbb3979700 1 -- start start 2021-05-25T12:08:22.069 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:22.064+0000 7fbbb3979700 10 monclient: init 2021-05-25T12:08:22.070 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:22.065+0000 7fbbb3979700 10 monclient: _reopen_session rank -1 2021-05-25T12:08:22.071 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:22.065+0000 7fbbb3979700 10 monclient: _add_conns ranks=[2,0,1] 2021-05-25T12:08:22.071 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:22.065+0000 7fbbb3979700 10 monclient(hunting): picked mon.noname-b con 0x7fbbac090990 addr v1:172.21.15.102:6790/0 2021-05-25T12:08:22.071 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:22.065+0000 7fbbb3979700 10 monclient(hunting): picked mon.noname-a con 0x7fbbac090d70 addr v1:172.21.15.102:6789/0 2021-05-25T12:08:22.071 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:22.065+0000 7fbbb3979700 10 monclient(hunting): picked mon.noname-c con 0x7fbbac12e330 addr v1:172.21.15.181:6789/0 2021-05-25T12:08:22.072 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:22.065+0000 7fbbb3979700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbac096fa0 con 0x7fbbac090d70 2021-05-25T12:08:22.072 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:22.065+0000 7fbbb3979700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbac0926a0 con 0x7fbbac12e330 2021-05-25T12:08:22.072 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:22.065+0000 7fbbb3979700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbac092940 con 0x7fbbac090990 2021-05-25T12:08:22.072 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:22.065+0000 7fbbb3979700 10 monclient(hunting): _renew_subs 2021-05-25T12:08:22.072 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:22.065+0000 7fbbb3979700 10 monclient(hunting): authenticate will time out at 1302.662208s 2021-05-25T12:08:22.073 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:22.065+0000 7fbbb0f14700 1 -- >> v1:172.21.15.102:6789/0 conn(0x7fbbac090d70 legacy=0x7fbbac091200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:08:22.073 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:22.065+0000 7fbbb1715700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fbbac090990 0x7fbbac093240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:57380/0 (socket says 172.21.15.102:57380) 2021-05-25T12:08:22.073 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:22.065+0000 7fbbb1715700 1 -- 172.21.15.102:0/159955745 learned_addr learned my addr 172.21.15.102:0/159955745 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:08:22.267 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:22.266+0000 7fbbb0f14700 1 -- 172.21.15.102:0/159955745 >> v1:172.21.15.102:6789/0 conn(0x7fbbac090d70 legacy=0x7fbbac091200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:08:22.668 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:22.667+0000 7fbbb0f14700 1 -- 172.21.15.102:0/159955745 >> v1:172.21.15.102:6789/0 conn(0x7fbbac090d70 legacy=0x7fbbac091200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:08:23.469 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:23.468+0000 7fbbb0f14700 1 -- 172.21.15.102:0/159955745 >> v1:172.21.15.102:6789/0 conn(0x7fbbac090d70 legacy=0x7fbbac091200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:08:25.067 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:25.065+0000 7fbb9affd700 10 monclient(hunting): tick 2021-05-25T12:08:25.067 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:25.065+0000 7fbb9affd700 1 monclient(hunting): continuing hunt 2021-05-25T12:08:25.068 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:25.065+0000 7fbb9affd700 10 monclient(hunting): _reopen_session rank -1 2021-05-25T12:08:25.068 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:25.065+0000 7fbb9affd700 1 -- 172.21.15.102:0/159955745 >> v1:172.21.15.102:6790/0 conn(0x7fbbac090990 legacy=0x7fbbac093240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:25.068 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:25.065+0000 7fbb9affd700 1 -- 172.21.15.102:0/159955745 >> v1:172.21.15.181:6789/0 conn(0x7fbbac12e330 legacy=0x7fbbac130720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:25.068 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:25.065+0000 7fbb9affd700 1 -- 172.21.15.102:0/159955745 >> v1:172.21.15.102:6789/0 conn(0x7fbbac090d70 legacy=0x7fbbac091200 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-05-25T12:08:25.068 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:25.065+0000 7fbb9affd700 10 monclient: _add_conns ranks=[2,1,0] 2021-05-25T12:08:25.069 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:25.065+0000 7fbb9affd700 10 monclient(hunting): picked mon.noname-b con 0x7fbbac090990 addr v1:172.21.15.102:6790/0 2021-05-25T12:08:25.069 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:25.065+0000 7fbb9affd700 10 monclient(hunting): picked mon.noname-c con 0x7fbbac12e330 addr v1:172.21.15.181:6789/0 2021-05-25T12:08:25.069 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:25.065+0000 7fbbb1f16700 1 -- 172.21.15.102:0/159955745 >> v1:172.21.15.102:6789/0 conn(0x7fbbac090d70 legacy=0x7fbb8c00d4a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:08:25.069 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:25.065+0000 7fbb9affd700 10 monclient(hunting): picked mon.noname-a con 0x7fbbac090d70 addr v1:172.21.15.102:6789/0 2021-05-25T12:08:25.070 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:25.065+0000 7fbb9affd700 1 -- 172.21.15.102:0/159955745 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbac092be0 con 0x7fbbac090d70 2021-05-25T12:08:25.070 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:25.065+0000 7fbb9affd700 1 -- 172.21.15.102:0/159955745 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbac096fa0 con 0x7fbbac12e330 2021-05-25T12:08:25.070 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:25.065+0000 7fbb9affd700 1 -- 172.21.15.102:0/159955745 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb8c010ca0 con 0x7fbbac090990 2021-05-25T12:08:25.070 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:25.065+0000 7fbb9affd700 10 monclient(hunting): _renew_subs 2021-05-25T12:08:25.267 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:25.266+0000 7fbbb1f16700 1 -- 172.21.15.102:0/159955745 >> v1:172.21.15.102:6789/0 conn(0x7fbbac090d70 legacy=0x7fbb8c00d4a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:08:25.668 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:25.667+0000 7fbbb1f16700 1 -- 172.21.15.102:0/159955745 >> v1:172.21.15.102:6789/0 conn(0x7fbbac090d70 legacy=0x7fbb8c00d4a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:08:26.469 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:26.468+0000 7fbbb1f16700 1 -- 172.21.15.102:0/159955745 >> v1:172.21.15.102:6789/0 conn(0x7fbbac090d70 legacy=0x7fbb8c00d4a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:08:28.069 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:28.066+0000 7fbb9affd700 10 monclient(hunting): tick 2021-05-25T12:08:28.069 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:28.066+0000 7fbb9affd700 1 monclient(hunting): continuing hunt 2021-05-25T12:08:28.070 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:28.066+0000 7fbb9affd700 10 monclient(hunting): _reopen_session rank -1 2021-05-25T12:08:28.070 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:28.066+0000 7fbb9affd700 1 -- 172.21.15.102:0/159955745 >> v1:172.21.15.102:6790/0 conn(0x7fbbac090990 legacy=0x7fbb8c006520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:28.070 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:28.066+0000 7fbb9affd700 1 -- 172.21.15.102:0/159955745 >> v1:172.21.15.181:6789/0 conn(0x7fbbac12e330 legacy=0x7fbb8c009cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:28.071 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:28.066+0000 7fbb9affd700 1 -- 172.21.15.102:0/159955745 >> v1:172.21.15.102:6789/0 conn(0x7fbbac090d70 legacy=0x7fbb8c00d4a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-05-25T12:08:28.071 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:28.066+0000 7fbb9affd700 10 monclient: _add_conns ranks=[2,0,1] 2021-05-25T12:08:28.072 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:28.066+0000 7fbb9affd700 10 monclient(hunting): picked mon.noname-b con 0x7fbb8c00fd50 addr v1:172.21.15.102:6790/0 2021-05-25T12:08:28.072 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:28.066+0000 7fbb9affd700 10 monclient(hunting): picked mon.noname-a con 0x7fbb8c006c50 addr v1:172.21.15.102:6789/0 2021-05-25T12:08:28.072 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:28.066+0000 7fbb9affd700 10 monclient(hunting): picked mon.noname-c con 0x7fbbac12e330 addr v1:172.21.15.181:6789/0 2021-05-25T12:08:28.073 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:28.066+0000 7fbb9affd700 1 -- 172.21.15.102:0/159955745 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb8c011f10 con 0x7fbb8c006c50 2021-05-25T12:08:28.073 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:28.066+0000 7fbb9affd700 1 -- 172.21.15.102:0/159955745 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbac092be0 con 0x7fbbac12e330 2021-05-25T12:08:28.073 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:28.066+0000 7fbb9affd700 1 -- 172.21.15.102:0/159955745 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb8c007030 con 0x7fbb8c00fd50 2021-05-25T12:08:28.074 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:28.066+0000 7fbbb0f14700 1 -- 172.21.15.102:0/159955745 >> v1:172.21.15.102:6789/0 conn(0x7fbb8c006c50 legacy=0x7fbb8c006520 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:08:28.074 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:28.066+0000 7fbb9affd700 10 monclient(hunting): _renew_subs 2021-05-25T12:08:28.268 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:28.267+0000 7fbbb0f14700 1 -- 172.21.15.102:0/159955745 >> v1:172.21.15.102:6789/0 conn(0x7fbb8c006c50 legacy=0x7fbb8c006520 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:08:28.669 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:28.668+0000 7fbbb0f14700 1 -- 172.21.15.102:0/159955745 >> v1:172.21.15.102:6789/0 conn(0x7fbb8c006c50 legacy=0x7fbb8c006520 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:08:29.470 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.469+0000 7fbbb0f14700 1 -- 172.21.15.102:0/159955745 >> v1:172.21.15.102:6789/0 conn(0x7fbb8c006c50 legacy=0x7fbb8c006520 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:08:29.760 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.759+0000 7fbb9bfff700 1 -- 172.21.15.102:0/159955745 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1917788650 0 0) 0x7fbb8c007030 con 0x7fbb8c00fd50 2021-05-25T12:08:29.761 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.759+0000 7fbb9bfff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:29.761 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.759+0000 7fbb9bfff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:29.761 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.759+0000 7fbb9bfff700 10 monclient(hunting): my global_id is 244166 2021-05-25T12:08:29.761 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.759+0000 7fbb9bfff700 1 -- 172.21.15.102:0/159955745 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb94003270 con 0x7fbb8c00fd50 2021-05-25T12:08:29.764 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.759+0000 7fbb9bfff700 1 -- 172.21.15.102:0/159955745 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 446953057 0 0) 0x7fbb94003270 con 0x7fbb8c00fd50 2021-05-25T12:08:29.764 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.759+0000 7fbb9bfff700 1 -- 172.21.15.102:0/159955745 >> v1:172.21.15.181:6789/0 conn(0x7fbbac12e330 legacy=0x7fbb8c009cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:29.764 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.759+0000 7fbb9bfff700 1 -- 172.21.15.102:0/159955745 >> v1:172.21.15.102:6789/0 conn(0x7fbb8c006c50 legacy=0x7fbb8c006520 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-05-25T12:08:29.764 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.760+0000 7fbb9bfff700 10 monclient: _finish_hunting 0 2021-05-25T12:08:29.765 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.760+0000 7fbb9bfff700 1 monclient: found mon.noname-b 2021-05-25T12:08:29.765 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.760+0000 7fbb9bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:08:29.765 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.760+0000 7fbb9bfff700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T12:08:29.765 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.760+0000 7fbb9bfff700 1 -- 172.21.15.102:0/159955745 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb8c0071c0 con 0x7fbb8c00fd50 2021-05-25T12:08:29.765 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.760+0000 7fbb9bfff700 10 monclient: _finish_auth 0 2021-05-25T12:08:29.766 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.760+0000 7fbb9bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:08:29.766 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.760+0000 7fbbb3979700 5 monclient: authenticate success, global_id 244166 2021-05-25T12:08:29.766 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.760+0000 7fbbb3979700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:08:29.766 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.760+0000 7fbb9bfff700 1 -- 172.21.15.102:0/159955745 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fbb9c001590 con 0x7fbb8c00fd50 2021-05-25T12:08:29.767 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.760+0000 7fbb9bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:29.767 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.760+0000 7fbb9bfff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:08:29.767 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.760+0000 7fbb9bfff700 10 monclient: dump: 2021-05-25T12:08:29.767 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:29.767 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:29.768 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:29.768 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:29.768 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:29.768 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:29.769 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:29.769 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:29.769 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:29.769 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:29.769 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.760+0000 7fbbb3979700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:08:29.770 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.760+0000 7fbb9bfff700 1 -- 172.21.15.102:0/159955745 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbb9c001830 con 0x7fbb8c00fd50 2021-05-25T12:08:29.770 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.760+0000 7fbb9bfff700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:08:29.770 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.760+0000 7fbb9bfff700 1 -- 172.21.15.102:0/159955745 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fbb9c001bc0 con 0x7fbb8c00fd50 2021-05-25T12:08:29.770 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.760+0000 7fbbb3979700 10 monclient: get_monmap_and_config success 2021-05-25T12:08:29.771 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.760+0000 7fbbb3979700 4 set_mon_vals no callback set 2021-05-25T12:08:29.771 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.760+0000 7fbb9bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:29.771 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.760+0000 7fbb9bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:08:29.771 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.760+0000 7fbb9bfff700 10 monclient: dump: 2021-05-25T12:08:29.772 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:29.772 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:29.772 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:29.772 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:29.772 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:29.773 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:29.773 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:29.773 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:29.773 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:29.773 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:29.774 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.761+0000 7fbbb3979700 10 monclient: shutdown 2021-05-25T12:08:29.774 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.761+0000 7fbbb3979700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:08:29.774 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.761+0000 7fbbb3979700 1 -- 172.21.15.102:0/159955745 >> v1:172.21.15.102:6790/0 conn(0x7fbb8c00fd50 legacy=0x7fbb8c010190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:29.774 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.761+0000 7fbbb3979700 1 -- 172.21.15.102:0/159955745 shutdown_connections 2021-05-25T12:08:29.774 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.761+0000 7fbbb3979700 1 -- 172.21.15.102:0/159955745 >> 172.21.15.102:0/159955745 conn(0x7fbbac09b370 msgr2=0x7fbbac097430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:08:29.775 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.761+0000 7fbbb3979700 1 -- 172.21.15.102:0/159955745 shutdown_connections 2021-05-25T12:08:29.775 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.761+0000 7fbbb3979700 1 -- 172.21.15.102:0/159955745 wait complete. 2021-05-25T12:08:29.775 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.762+0000 7fbbb3979700 10 monclient: build_initial_monmap 2021-05-25T12:08:29.775 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.762+0000 7fbbb3979700 10 monclient: monmap: 2021-05-25T12:08:29.776 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:08:29.776 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:08:29.776 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:08:29.776 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:08:29.776 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:08:29.777 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:08:29.777 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:08:29.777 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:08:29.777 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:08:29.778 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:29.783 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.762+0000 7fbbb3979700 1 librados: starting msgr at 2021-05-25T12:08:29.784 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.762+0000 7fbbb3979700 1 librados: starting objecter 2021-05-25T12:08:29.784 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.762+0000 7fbbb3979700 1 Processor -- start 2021-05-25T12:08:29.784 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.762+0000 7fbbb3979700 1 -- start start 2021-05-25T12:08:29.784 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.762+0000 7fbbb3979700 1 librados: setting wanted keys 2021-05-25T12:08:29.785 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.762+0000 7fbbb3979700 1 librados: calling monclient init 2021-05-25T12:08:29.785 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.762+0000 7fbbb3979700 10 monclient: init 2021-05-25T12:08:29.785 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.763+0000 7fbbb3979700 10 monclient: _reopen_session rank -1 2021-05-25T12:08:29.785 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.763+0000 7fbbb3979700 10 monclient: _add_conns ranks=[2,1,0] 2021-05-25T12:08:29.786 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.763+0000 7fbbb3979700 10 monclient(hunting): picked mon.noname-c con 0x7fbb8c006c50 addr v1:172.21.15.102:6790/0 2021-05-25T12:08:29.786 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.763+0000 7fbbb3979700 10 monclient(hunting): picked mon.noname-b con 0x7fbb8c00fd50 addr v1:172.21.15.181:6789/0 2021-05-25T12:08:29.786 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.763+0000 7fbbb3979700 10 monclient(hunting): picked mon.noname-a con 0x7fbbac12e330 addr v1:172.21.15.102:6789/0 2021-05-25T12:08:29.786 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.763+0000 7fbbb3979700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbac12a460 con 0x7fbbac12e330 2021-05-25T12:08:29.787 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.763+0000 7fbbb3979700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbac126d10 con 0x7fbb8c00fd50 2021-05-25T12:08:29.787 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.763+0000 7fbbb3979700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbac068ba0 con 0x7fbb8c006c50 2021-05-25T12:08:29.787 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.763+0000 7fbbb3979700 10 monclient(hunting): _renew_subs 2021-05-25T12:08:29.787 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.763+0000 7fbbb3979700 10 monclient(hunting): authenticate will time out at 1310.360308s 2021-05-25T12:08:29.787 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.763+0000 7fbbb1f16700 1 -- >> v1:172.21.15.102:6789/0 conn(0x7fbbac12e330 legacy=0x7fbbac13f010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:08:29.788 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.763+0000 7fbbb1715700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fbb8c006c50 0x7fbbac065610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:57420/0 (socket says 172.21.15.102:57420) 2021-05-25T12:08:29.788 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.763+0000 7fbbb1715700 1 -- 172.21.15.102:0/4108381209 learned_addr learned my addr 172.21.15.102:0/4108381209 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:08:29.788 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.763+0000 7fbb997fa700 10 client.?.objecter ms_handle_connect 0x7fbb8c006c50 2021-05-25T12:08:29.788 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.763+0000 7fbb997fa700 10 client.?.objecter resend_mon_ops 2021-05-25T12:08:29.789 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.763+0000 7fbb997fa700 10 client.?.objecter ms_handle_connect 0x7fbb8c00fd50 2021-05-25T12:08:29.789 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.763+0000 7fbb997fa700 10 client.?.objecter resend_mon_ops 2021-05-25T12:08:29.789 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.763+0000 7fbb997fa700 1 -- 172.21.15.102:0/4108381209 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3810219916 0 0) 0x7fbbac126d10 con 0x7fbb8c00fd50 2021-05-25T12:08:29.789 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.763+0000 7fbb997fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:29.789 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.763+0000 7fbb997fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:29.790 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.763+0000 7fbb997fa700 10 monclient(hunting): my global_id is 224428 2021-05-25T12:08:29.790 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.763+0000 7fbb997fa700 1 -- 172.21.15.102:0/4108381209 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb8c007030 con 0x7fbb8c00fd50 2021-05-25T12:08:29.790 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.765+0000 7fbb997fa700 1 -- 172.21.15.102:0/4108381209 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4226104265 0 0) 0x7fbb8c007030 con 0x7fbb8c00fd50 2021-05-25T12:08:29.790 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.765+0000 7fbb997fa700 1 -- 172.21.15.102:0/4108381209 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbbac126d10 con 0x7fbb8c00fd50 2021-05-25T12:08:29.791 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.765+0000 7fbb997fa700 1 -- 172.21.15.102:0/4108381209 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fbba0004040 con 0x7fbb8c00fd50 2021-05-25T12:08:29.791 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.765+0000 7fbb997fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:29.791 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.765+0000 7fbb997fa700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:08:29.791 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.765+0000 7fbb997fa700 10 monclient(hunting): dump: 2021-05-25T12:08:29.792 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:29.792 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:29.792 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:29.792 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:29.792 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:29.793 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:29.793 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:29.793 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:29.793 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:29.794 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:29.794 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.765+0000 7fbb997fa700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:08:29.794 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.765+0000 7fbb997fa700 1 -- 172.21.15.102:0/4108381209 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2949479507 0 0) 0x7fbbac068ba0 con 0x7fbb8c006c50 2021-05-25T12:08:29.794 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.765+0000 7fbb997fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:29.795 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.765+0000 7fbb997fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:29.795 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.765+0000 7fbb997fa700 10 monclient(hunting): my global_id is 244169 2021-05-25T12:08:29.795 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.765+0000 7fbb997fa700 1 -- 172.21.15.102:0/4108381209 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb8c007030 con 0x7fbb8c006c50 2021-05-25T12:08:29.795 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.766+0000 7fbb997fa700 1 -- 172.21.15.102:0/4108381209 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3071125908 0 0) 0x7fbbac126d10 con 0x7fbb8c00fd50 2021-05-25T12:08:29.796 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.766+0000 7fbb997fa700 1 -- 172.21.15.102:0/4108381209 >> v1:172.21.15.102:6790/0 conn(0x7fbb8c006c50 legacy=0x7fbbac065610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:29.796 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.766+0000 7fbb997fa700 1 -- 172.21.15.102:0/4108381209 >> v1:172.21.15.102:6789/0 conn(0x7fbbac12e330 legacy=0x7fbbac13f010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-05-25T12:08:29.796 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.766+0000 7fbb997fa700 10 monclient: _finish_hunting 0 2021-05-25T12:08:29.796 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.766+0000 7fbb997fa700 1 monclient: found mon.b 2021-05-25T12:08:29.797 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.766+0000 7fbb997fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:08:29.797 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.766+0000 7fbb997fa700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T12:08:29.797 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.766+0000 7fbb997fa700 1 -- 172.21.15.102:0/4108381209 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbac068380 con 0x7fbb8c00fd50 2021-05-25T12:08:29.797 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.766+0000 7fbb997fa700 10 monclient: _finish_auth 0 2021-05-25T12:08:29.798 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.766+0000 7fbb997fa700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:08:29.798 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.766+0000 7fbbb3979700 5 monclient: authenticate success, global_id 224428 2021-05-25T12:08:29.798 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.766+0000 7fbbb3979700 10 monclient: _renew_subs 2021-05-25T12:08:29.798 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.766+0000 7fbbb3979700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T12:08:29.798 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.766+0000 7fbbb3979700 1 -- 172.21.15.102:0/4108381209 --> v1:172.21.15.181:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbac0679d0 con 0x7fbb8c00fd50 2021-05-25T12:08:29.799 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.766+0000 7fbbb3979700 10 client.224428.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:08:29.799 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.766+0000 7fbbb3979700 10 monclient: _renew_subs 2021-05-25T12:08:29.799 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.766+0000 7fbbb3979700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T12:08:29.799 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.766+0000 7fbbb3979700 1 -- 172.21.15.102:0/4108381209 --> v1:172.21.15.181:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbac066270 con 0x7fbb8c00fd50 2021-05-25T12:08:29.800 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.766+0000 7fbbb3979700 1 librados: init done 2021-05-25T12:08:29.800 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.766+0000 7fbb997fa700 1 -- 172.21.15.102:0/4108381209 <== mon.1 v1:172.21.15.181:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbba0004260 con 0x7fbb8c00fd50 2021-05-25T12:08:29.800 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.766+0000 7fbb997fa700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:08:29.800 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.766+0000 7fbb997fa700 1 -- 172.21.15.102:0/4108381209 <== mon.1 v1:172.21.15.181:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fbba00045f0 con 0x7fbb8c00fd50 2021-05-25T12:08:29.801 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.766+0000 7fbb997fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:29.801 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.766+0000 7fbb997fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T12:08:29.801 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.766+0000 7fbb9b7fe700 4 set_mon_vals no callback set 2021-05-25T12:08:29.801 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.766+0000 7fbb997fa700 10 monclient: dump: 2021-05-25T12:08:29.801 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:29.802 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:29.802 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:29.802 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:29.802 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:29.802 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:29.803 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:29.803 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:29.803 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:29.803 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:29.803 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.767+0000 7fbb997fa700 1 -- 172.21.15.102:0/4108381209 <== mon.1 v1:172.21.15.181:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7fbba00047d0 con 0x7fbb8c00fd50 2021-05-25T12:08:29.804 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.767+0000 7fbbb3979700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:08:29.804 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.767+0000 7fbbb3979700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:08:29.804 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.767+0000 7fbbb3979700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T12:08:29.804 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.767+0000 7fbbb3979700 1 -- 172.21.15.102:0/4108381209 --> v1:172.21.15.181:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb88004db0 con 0x7fbb8c00fd50 2021-05-25T12:08:29.804 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.767+0000 7fbb997fa700 1 -- 172.21.15.102:0/4108381209 <== mon.1 v1:172.21.15.181:6789/0 8 ==== osd_map(272..272 src has 1..272) v4 ==== 5433+0+0 (unknown 1668784668 0 0) 0x7fbba003d640 con 0x7fbb8c00fd50 2021-05-25T12:08:29.805 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.767+0000 7fbb997fa700 10 client.224428.objecter ms_dispatch 0x7fbbac001550 osd_map(272..272 src has 1..272) v4 2021-05-25T12:08:29.805 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.767+0000 7fbb997fa700 3 client.224428.objecter handle_osd_map got epochs [272,272] > 0 2021-05-25T12:08:29.805 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.767+0000 7fbb997fa700 3 client.224428.objecter handle_osd_map decoding full epoch 272 2021-05-25T12:08:29.805 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.767+0000 7fbb997fa700 20 client.224428.objecter dump_active .. 0 homeless 2021-05-25T12:08:29.805 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.767+0000 7fbbb1715700 10 monclient: get_auth_request con 0x7fbb8c036470 auth_method 0 2021-05-25T12:08:29.806 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.767+0000 7fbb997fa700 10 client.224428.objecter ms_handle_connect 0x7fbb8c036470 2021-05-25T12:08:29.806 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.771+0000 7fbb997fa700 1 -- 172.21.15.102:0/4108381209 <== mon.1 v1:172.21.15.181:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7fbba0012ba0 con 0x7fbb8c00fd50 2021-05-25T12:08:29.806 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.771+0000 7fbb997fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:08:29.806 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.771+0000 7fbb997fa700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:08:29.988 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.987+0000 7fbbb3979700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2021-05-25T12:08:29.988 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.987+0000 7fbbb3979700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2021-05-25T12:08:29.989 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.987+0000 7fbbb3979700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T12:08:29.989 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.987+0000 7fbbb3979700 1 -- 172.21.15.102:0/4108381209 --> v1:172.21.15.181:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fbb880055e0 con 0x7fbb8c00fd50 2021-05-25T12:08:29.990 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.987+0000 7fbb997fa700 1 -- 172.21.15.102:0/4108381209 <== mon.1 v1:172.21.15.181:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1336 (unknown 254746657 0 1185405105) 0x7fbba00129c0 con 0x7fbb8c00fd50 2021-05-25T12:08:29.990 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.987+0000 7fbb997fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2021-05-25T12:08:29.990 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.987+0000 7fbb997fa700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:08:29.991 INFO:teuthology.orchestra.run.smithi102.stdout:{"election_epoch":72,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138303579357183","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"20ee91b6-8935-4b98-b868-85c03789ccc4","modified":"2021-05-25T11:59:11.011522Z","created":"2021-05-25T11:59:11.011522Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6790","nonce":0}]},"addr":"172.21.15.102:6790/0","public_addr":"172.21.15.102:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2021-05-25T12:08:29.992 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.989+0000 7fbbb3979700 10 librados: watch_flush enter 2021-05-25T12:08:29.992 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.989+0000 7fbbb3979700 10 librados: watch_flush exit 2021-05-25T12:08:29.993 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.989+0000 7fbbb3979700 20 client.224428.objecter shutdown clearing up homeless session... 2021-05-25T12:08:29.993 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.989+0000 7fbbb3979700 10 client.224428.objecter successfully canceled tick 2021-05-25T12:08:29.993 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.989+0000 7fbbb3979700 1 -- 172.21.15.102:0/4108381209 >> v1:172.21.15.181:6812/29582 conn(0x7fbb8c036470 legacy=0x7fbb8c038930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:29.993 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.989+0000 7fbbb3979700 10 monclient: shutdown 2021-05-25T12:08:29.994 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.989+0000 7fbbb3979700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:08:29.994 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.989+0000 7fbbb3979700 1 -- 172.21.15.102:0/4108381209 >> v1:172.21.15.181:6789/0 conn(0x7fbb8c00fd50 legacy=0x7fbbac126600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:29.994 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.989+0000 7fbbb3979700 1 -- 172.21.15.102:0/4108381209 shutdown_connections 2021-05-25T12:08:29.994 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.989+0000 7fbbb3979700 1 -- 172.21.15.102:0/4108381209 >> 172.21.15.102:0/4108381209 conn(0x7fbbac09b370 msgr2=0x7fbbac1293d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:08:29.994 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.989+0000 7fbbb3979700 1 -- 172.21.15.102:0/4108381209 shutdown_connections 2021-05-25T12:08:29.995 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.989+0000 7fbbb3979700 1 -- 172.21.15.102:0/4108381209 wait complete. 2021-05-25T12:08:29.995 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:29.989+0000 7fbbb3979700 1 librados: shutdown 2021-05-25T12:08:30.002 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2021-05-25T12:08:30.003 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2021-05-25T12:08:30.112 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.110+0000 7f3d5ccbd700 10 monclient: get_monmap_and_config 2021-05-25T12:08:30.112 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.110+0000 7f3d5ccbd700 10 monclient: build_initial_monmap 2021-05-25T12:08:30.113 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.110+0000 7f3d5ccbd700 10 monclient: monmap: 2021-05-25T12:08:30.113 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:08:30.113 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:30.114 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:08:30.111642+0000 2021-05-25T12:08:30.114 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:08:30.111642+0000 2021-05-25T12:08:30.114 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:08:30.115 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:30.115 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:08:30.115 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:08:30.115 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:08:30.116 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:30.116 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.111+0000 7f3d5ccbd700 1 Processor -- start 2021-05-25T12:08:30.116 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.111+0000 7f3d5ccbd700 1 -- start start 2021-05-25T12:08:30.116 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.111+0000 7f3d5ccbd700 10 monclient: init 2021-05-25T12:08:30.133 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.111+0000 7f3d5ccbd700 10 monclient: _reopen_session rank -1 2021-05-25T12:08:30.133 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.111+0000 7f3d5ccbd700 10 monclient: _add_conns ranks=[0,2,1] 2021-05-25T12:08:30.134 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.111+0000 7f3d5ccbd700 10 monclient(hunting): picked mon.noname-a con 0x7f3d58068d50 addr v1:172.21.15.102:6789/0 2021-05-25T12:08:30.134 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.111+0000 7f3d5ccbd700 10 monclient(hunting): picked mon.noname-b con 0x7f3d58069130 addr v1:172.21.15.102:6790/0 2021-05-25T12:08:30.134 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.111+0000 7f3d5ccbd700 10 monclient(hunting): picked mon.noname-c con 0x7f3d580695e0 addr v1:172.21.15.181:6789/0 2021-05-25T12:08:30.135 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.111+0000 7f3d5ccbd700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d58094380 con 0x7f3d58068d50 2021-05-25T12:08:30.135 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.111+0000 7f3d5ccbd700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d58094620 con 0x7f3d580695e0 2021-05-25T12:08:30.136 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.111+0000 7f3d5ccbd700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d580948c0 con 0x7f3d58069130 2021-05-25T12:08:30.136 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.111+0000 7f3d5ccbd700 10 monclient(hunting): _renew_subs 2021-05-25T12:08:30.136 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.111+0000 7f3d5ccbd700 10 monclient(hunting): authenticate will time out at 1310.708868s 2021-05-25T12:08:30.137 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.111+0000 7f3d5659c700 1 -- >> v1:172.21.15.102:6789/0 conn(0x7f3d58068d50 legacy=0x7f3d58129c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:08:30.137 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.112+0000 7f3d55d9b700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f3d58069130 0x7f3d5812ea30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:57428/0 (socket says 172.21.15.102:57428) 2021-05-25T12:08:30.138 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.112+0000 7f3d55d9b700 1 -- 172.21.15.102:0/1884924071 learned_addr learned my addr 172.21.15.102:0/1884924071 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:08:30.138 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.112+0000 7f3d5559a700 1 -- 172.21.15.102:0/1884924071 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1555662604 0 0) 0x7f3d580948c0 con 0x7f3d58069130 2021-05-25T12:08:30.138 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.112+0000 7f3d5559a700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:30.139 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.112+0000 7f3d5559a700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:30.139 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.112+0000 7f3d5559a700 10 monclient(hunting): my global_id is 244178 2021-05-25T12:08:30.140 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.112+0000 7f3d5559a700 1 -- 172.21.15.102:0/1884924071 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d3c003270 con 0x7f3d58069130 2021-05-25T12:08:30.140 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.112+0000 7f3d5559a700 1 -- 172.21.15.102:0/1884924071 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2104599616 0 0) 0x7f3d58094620 con 0x7f3d580695e0 2021-05-25T12:08:30.140 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.112+0000 7f3d5559a700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:30.141 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.112+0000 7f3d5559a700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:30.141 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.112+0000 7f3d5559a700 10 monclient(hunting): my global_id is 224437 2021-05-25T12:08:30.142 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.112+0000 7f3d5559a700 1 -- 172.21.15.102:0/1884924071 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d580948c0 con 0x7f3d580695e0 2021-05-25T12:08:30.142 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.112+0000 7f3d5559a700 1 -- 172.21.15.102:0/1884924071 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2347048323 0 0) 0x7f3d3c003270 con 0x7f3d58069130 2021-05-25T12:08:30.142 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.112+0000 7f3d5559a700 1 -- 172.21.15.102:0/1884924071 >> v1:172.21.15.181:6789/0 conn(0x7f3d580695e0 legacy=0x7f3d58132160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:30.143 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.112+0000 7f3d5559a700 1 -- 172.21.15.102:0/1884924071 >> v1:172.21.15.102:6789/0 conn(0x7f3d58068d50 legacy=0x7f3d58129c90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-05-25T12:08:30.143 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.112+0000 7f3d5559a700 10 monclient: _finish_hunting 0 2021-05-25T12:08:30.144 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.112+0000 7f3d5559a700 1 monclient: found mon.noname-b 2021-05-25T12:08:30.144 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.112+0000 7f3d5559a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:08:30.144 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.112+0000 7f3d5559a700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T12:08:30.145 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.112+0000 7f3d5559a700 1 -- 172.21.15.102:0/1884924071 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d58094c80 con 0x7f3d58069130 2021-05-25T12:08:30.145 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.112+0000 7f3d5559a700 10 monclient: _finish_auth 0 2021-05-25T12:08:30.145 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.112+0000 7f3d5559a700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:08:30.146 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.112+0000 7f3d5ccbd700 5 monclient: authenticate success, global_id 244178 2021-05-25T12:08:30.146 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.112+0000 7f3d5559a700 1 -- 172.21.15.102:0/1884924071 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f3d40001610 con 0x7f3d58069130 2021-05-25T12:08:30.147 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.112+0000 7f3d5ccbd700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:08:30.147 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.112+0000 7f3d5559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:30.147 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.112+0000 7f3d5559a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:08:30.148 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.112+0000 7f3d5559a700 10 monclient: dump: 2021-05-25T12:08:30.148 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:30.149 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:30.149 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:30.149 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:30.150 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:30.150 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:30.151 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:30.151 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:30.151 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:30.152 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:30.152 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.112+0000 7f3d5ccbd700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:08:30.152 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.112+0000 7f3d5559a700 1 -- 172.21.15.102:0/1884924071 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3d400018f0 con 0x7f3d58069130 2021-05-25T12:08:30.152 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.112+0000 7f3d5559a700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:08:30.153 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.112+0000 7f3d5559a700 1 -- 172.21.15.102:0/1884924071 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f3d40001c80 con 0x7f3d58069130 2021-05-25T12:08:30.153 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.112+0000 7f3d5ccbd700 10 monclient: get_monmap_and_config success 2021-05-25T12:08:30.153 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.112+0000 7f3d5ccbd700 4 set_mon_vals no callback set 2021-05-25T12:08:30.153 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.112+0000 7f3d5559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:30.153 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.112+0000 7f3d5559a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:08:30.154 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.112+0000 7f3d5559a700 10 monclient: dump: 2021-05-25T12:08:30.154 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:30.154 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:30.154 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:30.155 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:30.155 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:30.155 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:30.155 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:30.156 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:30.156 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:30.156 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:30.156 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.113+0000 7f3d5ccbd700 10 monclient: shutdown 2021-05-25T12:08:30.156 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.113+0000 7f3d5ccbd700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:08:30.157 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.113+0000 7f3d5ccbd700 1 -- 172.21.15.102:0/1884924071 >> v1:172.21.15.102:6790/0 conn(0x7f3d58069130 legacy=0x7f3d5812ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:30.157 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.113+0000 7f3d5ccbd700 1 -- 172.21.15.102:0/1884924071 shutdown_connections 2021-05-25T12:08:30.157 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.113+0000 7f3d5ccbd700 1 -- 172.21.15.102:0/1884924071 >> 172.21.15.102:0/1884924071 conn(0x7f3d5809b370 msgr2=0x7f3d58097430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:08:30.157 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.113+0000 7f3d5ccbd700 1 -- 172.21.15.102:0/1884924071 shutdown_connections 2021-05-25T12:08:30.157 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.113+0000 7f3d5ccbd700 1 -- 172.21.15.102:0/1884924071 wait complete. 2021-05-25T12:08:30.158 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.113+0000 7f3d5ccbd700 10 monclient: build_initial_monmap 2021-05-25T12:08:30.158 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.113+0000 7f3d5ccbd700 10 monclient: monmap: 2021-05-25T12:08:30.158 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:08:30.158 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:08:30.159 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:08:30.159 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:08:30.159 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:08:30.159 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:08:30.159 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:08:30.160 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:08:30.160 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:08:30.160 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:30.160 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.113+0000 7f3d5ccbd700 1 librados: starting msgr at 2021-05-25T12:08:30.160 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.113+0000 7f3d5ccbd700 1 librados: starting objecter 2021-05-25T12:08:30.161 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.113+0000 7f3d5ccbd700 1 Processor -- start 2021-05-25T12:08:30.161 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.113+0000 7f3d5ccbd700 1 -- start start 2021-05-25T12:08:30.161 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.113+0000 7f3d5ccbd700 1 librados: setting wanted keys 2021-05-25T12:08:30.162 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.113+0000 7f3d5ccbd700 1 librados: calling monclient init 2021-05-25T12:08:30.162 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.113+0000 7f3d5ccbd700 10 monclient: init 2021-05-25T12:08:30.162 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.114+0000 7f3d5ccbd700 10 monclient: _reopen_session rank -1 2021-05-25T12:08:30.163 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.114+0000 7f3d5ccbd700 10 monclient: _add_conns ranks=[0,2,1] 2021-05-25T12:08:30.163 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.114+0000 7f3d5ccbd700 10 monclient(hunting): picked mon.noname-a con 0x7f3d58068d50 addr v1:172.21.15.102:6789/0 2021-05-25T12:08:30.164 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.114+0000 7f3d5ccbd700 10 monclient(hunting): picked mon.noname-c con 0x7f3d58069130 addr v1:172.21.15.102:6790/0 2021-05-25T12:08:30.164 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.114+0000 7f3d5ccbd700 10 monclient(hunting): picked mon.noname-b con 0x7f3d580695e0 addr v1:172.21.15.181:6789/0 2021-05-25T12:08:30.165 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.114+0000 7f3d5ccbd700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d58125080 con 0x7f3d58068d50 2021-05-25T12:08:30.165 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.114+0000 7f3d5ccbd700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d58093840 con 0x7f3d580695e0 2021-05-25T12:08:30.165 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.114+0000 7f3d5ccbd700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d58091240 con 0x7f3d58069130 2021-05-25T12:08:30.166 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.114+0000 7f3d5ccbd700 10 monclient(hunting): _renew_subs 2021-05-25T12:08:30.166 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.114+0000 7f3d5ccbd700 10 monclient(hunting): authenticate will time out at 1310.711418s 2021-05-25T12:08:30.167 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.114+0000 7f3d5659c700 1 -- >> v1:172.21.15.102:6789/0 conn(0x7f3d58068d50 legacy=0x7f3d5813b370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:08:30.167 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.114+0000 7f3d55d9b700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f3d58069130 0x7f3d58132b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:57434/0 (socket says 172.21.15.102:57434) 2021-05-25T12:08:30.167 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.114+0000 7f3d55d9b700 1 -- 172.21.15.102:0/3537010489 learned_addr learned my addr 172.21.15.102:0/3537010489 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:08:30.168 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.114+0000 7f3d467fc700 10 client.?.objecter ms_handle_connect 0x7f3d58069130 2021-05-25T12:08:30.168 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.114+0000 7f3d467fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:08:30.168 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.114+0000 7f3d467fc700 1 -- 172.21.15.102:0/3537010489 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2184754338 0 0) 0x7f3d58091240 con 0x7f3d58069130 2021-05-25T12:08:30.169 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.114+0000 7f3d467fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:30.169 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.114+0000 7f3d467fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:30.170 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.114+0000 7f3d467fc700 10 monclient(hunting): my global_id is 244181 2021-05-25T12:08:30.170 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.114+0000 7f3d467fc700 1 -- 172.21.15.102:0/3537010489 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d34004c10 con 0x7f3d58069130 2021-05-25T12:08:30.170 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.114+0000 7f3d467fc700 10 client.?.objecter ms_handle_connect 0x7f3d580695e0 2021-05-25T12:08:30.171 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.114+0000 7f3d467fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:08:30.171 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.114+0000 7f3d467fc700 1 -- 172.21.15.102:0/3537010489 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 966594724 0 0) 0x7f3d34004c10 con 0x7f3d58069130 2021-05-25T12:08:30.172 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.114+0000 7f3d467fc700 1 -- 172.21.15.102:0/3537010489 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3d58091240 con 0x7f3d58069130 2021-05-25T12:08:30.172 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.114+0000 7f3d467fc700 1 -- 172.21.15.102:0/3537010489 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f3d400015d0 con 0x7f3d58069130 2021-05-25T12:08:30.172 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.114+0000 7f3d467fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:30.173 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.114+0000 7f3d467fc700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:08:30.173 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.114+0000 7f3d467fc700 10 monclient(hunting): dump: 2021-05-25T12:08:30.173 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:30.174 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:30.174 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:30.175 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:30.175 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:30.175 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:30.176 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:30.176 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:30.176 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:30.177 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:30.177 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.114+0000 7f3d467fc700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:08:30.178 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.114+0000 7f3d467fc700 1 -- 172.21.15.102:0/3537010489 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2393756515 0 0) 0x7f3d58093840 con 0x7f3d580695e0 2021-05-25T12:08:30.178 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.114+0000 7f3d467fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:30.178 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.114+0000 7f3d467fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:30.179 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.114+0000 7f3d467fc700 10 monclient(hunting): my global_id is 224440 2021-05-25T12:08:30.179 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.114+0000 7f3d467fc700 1 -- 172.21.15.102:0/3537010489 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d34004c10 con 0x7f3d580695e0 2021-05-25T12:08:30.180 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.115+0000 7f3d467fc700 1 -- 172.21.15.102:0/3537010489 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2102231932 0 0) 0x7f3d58091240 con 0x7f3d58069130 2021-05-25T12:08:30.180 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.115+0000 7f3d467fc700 1 -- 172.21.15.102:0/3537010489 >> v1:172.21.15.181:6789/0 conn(0x7f3d580695e0 legacy=0x7f3d581432d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:30.180 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.115+0000 7f3d467fc700 1 -- 172.21.15.102:0/3537010489 >> v1:172.21.15.102:6789/0 conn(0x7f3d58068d50 legacy=0x7f3d5813b370 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-05-25T12:08:30.180 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.115+0000 7f3d467fc700 10 monclient: _finish_hunting 0 2021-05-25T12:08:30.181 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.115+0000 7f3d467fc700 1 monclient: found mon.c 2021-05-25T12:08:30.181 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.115+0000 7f3d467fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:08:30.181 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.115+0000 7f3d467fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:08:30.181 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.115+0000 7f3d467fc700 1 -- 172.21.15.102:0/3537010489 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d580932c0 con 0x7f3d58069130 2021-05-25T12:08:30.181 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.115+0000 7f3d467fc700 10 monclient: _finish_auth 0 2021-05-25T12:08:30.182 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.115+0000 7f3d467fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:08:30.182 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.115+0000 7f3d5ccbd700 5 monclient: authenticate success, global_id 244181 2021-05-25T12:08:30.182 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.115+0000 7f3d5ccbd700 10 monclient: _renew_subs 2021-05-25T12:08:30.182 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.115+0000 7f3d5ccbd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:08:30.183 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.115+0000 7f3d5ccbd700 1 -- 172.21.15.102:0/3537010489 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d58133210 con 0x7f3d58069130 2021-05-25T12:08:30.183 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.115+0000 7f3d5ccbd700 10 client.244181.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:08:30.183 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.115+0000 7f3d5ccbd700 10 monclient: _renew_subs 2021-05-25T12:08:30.183 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.115+0000 7f3d5ccbd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:08:30.184 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.115+0000 7f3d5ccbd700 1 -- 172.21.15.102:0/3537010489 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d58068340 con 0x7f3d58069130 2021-05-25T12:08:30.184 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.115+0000 7f3d5ccbd700 1 librados: init done 2021-05-25T12:08:30.184 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.115+0000 7f3d467fc700 1 -- 172.21.15.102:0/3537010489 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3d40004a40 con 0x7f3d58069130 2021-05-25T12:08:30.184 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.115+0000 7f3d467fc700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:08:30.184 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.115+0000 7f3d467fc700 1 -- 172.21.15.102:0/3537010489 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f3d40004dd0 con 0x7f3d58069130 2021-05-25T12:08:30.185 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.115+0000 7f3d467fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:30.185 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.115+0000 7f3d467fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:08:30.185 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.115+0000 7f3d54d99700 4 set_mon_vals no callback set 2021-05-25T12:08:30.185 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.115+0000 7f3d467fc700 10 monclient: dump: 2021-05-25T12:08:30.186 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:30.186 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:30.186 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:30.186 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:30.187 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:30.187 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:30.187 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:30.187 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:30.187 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:30.188 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:30.188 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.115+0000 7f3d5ccbd700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:08:30.188 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.115+0000 7f3d5ccbd700 1 -- 172.21.15.102:0/3537010489 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f3d380065a0 con 0x7f3d38002840 2021-05-25T12:08:30.188 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.115+0000 7f3d467fc700 1 -- 172.21.15.102:0/3537010489 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f3d40004fb0 con 0x7f3d58069130 2021-05-25T12:08:30.188 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.116+0000 7f3d467fc700 10 client.244181.objecter ms_handle_connect 0x7f3d38002840 2021-05-25T12:08:30.189 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.116+0000 7f3d467fc700 10 client.244181.objecter resend_mon_ops 2021-05-25T12:08:30.189 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.116+0000 7f3d467fc700 1 -- 172.21.15.102:0/3537010489 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(272..272 src has 1..272) v4 ==== 5433+0+0 (unknown 1668784668 0 0) 0x7f3d4003d910 con 0x7f3d58069130 2021-05-25T12:08:30.189 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.116+0000 7f3d467fc700 10 client.244181.objecter ms_dispatch 0x7f3d58001550 osd_map(272..272 src has 1..272) v4 2021-05-25T12:08:30.189 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.116+0000 7f3d467fc700 3 client.244181.objecter handle_osd_map got epochs [272,272] > 0 2021-05-25T12:08:30.190 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.116+0000 7f3d467fc700 3 client.244181.objecter handle_osd_map decoding full epoch 272 2021-05-25T12:08:30.190 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.116+0000 7f3d467fc700 20 client.244181.objecter dump_active .. 0 homeless 2021-05-25T12:08:30.190 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.116+0000 7f3d467fc700 1 -- 172.21.15.102:0/3537010489 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2406442263 0 0) 0x7f3d380065a0 con 0x7f3d38002840 2021-05-25T12:08:30.190 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.116+0000 7f3d467fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:30.191 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.116+0000 7f3d467fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:30.191 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.116+0000 7f3d467fc700 10 monclient(hunting): my global_id is 244184 2021-05-25T12:08:30.191 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.116+0000 7f3d467fc700 1 -- 172.21.15.102:0/3537010489 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d58091240 con 0x7f3d38002840 2021-05-25T12:08:30.191 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.116+0000 7f3d56d9d700 10 monclient: get_auth_request con 0x7f3d34033150 auth_method 0 2021-05-25T12:08:30.192 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.116+0000 7f3d467fc700 1 -- 172.21.15.102:0/3537010489 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3604238302 0 0) 0x7f3d58091240 con 0x7f3d38002840 2021-05-25T12:08:30.192 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.116+0000 7f3d467fc700 1 -- 172.21.15.102:0/3537010489 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3d380065a0 con 0x7f3d38002840 2021-05-25T12:08:30.192 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.116+0000 7f3d467fc700 1 -- 172.21.15.102:0/3537010489 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f3d4c003c80 con 0x7f3d38002840 2021-05-25T12:08:30.192 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.116+0000 7f3d467fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:30.193 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.116+0000 7f3d467fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:08:30.193 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.116+0000 7f3d467fc700 10 monclient: dump: 2021-05-25T12:08:30.193 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:30.193 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:30.194 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:30.194 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:30.194 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:30.194 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:30.194 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:30.195 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:30.195 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:30.195 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:30.195 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.116+0000 7f3d467fc700 10 client.244181.objecter ms_handle_connect 0x7f3d34033150 2021-05-25T12:08:30.196 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.116+0000 7f3d467fc700 1 -- 172.21.15.102:0/3537010489 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 94067299 0 0) 0x7f3d380065a0 con 0x7f3d38002840 2021-05-25T12:08:30.196 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.116+0000 7f3d467fc700 1 -- 172.21.15.102:0/3537010489 --> v1:172.21.15.102:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3d38009e40 con 0x7f3d38002840 2021-05-25T12:08:30.196 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.116+0000 7f3d467fc700 1 -- 172.21.15.102:0/3537010489 <== mon.2 v1:172.21.15.102:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7f3d4c0060b0 con 0x7f3d38002840 2021-05-25T12:08:30.196 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.116+0000 7f3d467fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:08:30.196 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.116+0000 7f3d467fc700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:08:30.197 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.116+0000 7f3d467fc700 1 -- 172.21.15.102:0/3537010489 >> v1:172.21.15.102:6790/0 conn(0x7f3d38002840 legacy=0x7f3d38004ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:30.197 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.116+0000 7f3d467fc700 1 -- 172.21.15.102:0/3537010489 >> v1:172.21.15.102:6790/0 conn(0x7f3d38002840 legacy=0x7f3d38004ce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:08:30.197 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.123+0000 7f3d5ccbd700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:08:30.197 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.123+0000 7f3d5ccbd700 1 -- 172.21.15.102:0/3537010489 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f3d38009ad0 con 0x7f3d38008d90 2021-05-25T12:08:30.198 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.123+0000 7f3d467fc700 10 client.244181.objecter ms_handle_connect 0x7f3d38008d90 2021-05-25T12:08:30.198 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.123+0000 7f3d467fc700 10 client.244181.objecter resend_mon_ops 2021-05-25T12:08:30.198 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.123+0000 7f3d467fc700 1 -- 172.21.15.102:0/3537010489 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 98674150 0 0) 0x7f3d38009ad0 con 0x7f3d38008d90 2021-05-25T12:08:30.198 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.123+0000 7f3d467fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:30.199 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.123+0000 7f3d467fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:30.199 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.123+0000 7f3d467fc700 10 monclient(hunting): my global_id is 244187 2021-05-25T12:08:30.199 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.123+0000 7f3d467fc700 1 -- 172.21.15.102:0/3537010489 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d380065a0 con 0x7f3d38008d90 2021-05-25T12:08:30.199 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.124+0000 7f3d467fc700 1 -- 172.21.15.102:0/3537010489 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3929536829 0 0) 0x7f3d380065a0 con 0x7f3d38008d90 2021-05-25T12:08:30.199 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.124+0000 7f3d467fc700 1 -- 172.21.15.102:0/3537010489 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3d38009ad0 con 0x7f3d38008d90 2021-05-25T12:08:30.200 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.124+0000 7f3d467fc700 1 -- 172.21.15.102:0/3537010489 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f3d4c004530 con 0x7f3d38008d90 2021-05-25T12:08:30.200 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.124+0000 7f3d467fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:30.200 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.124+0000 7f3d467fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:08:30.201 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.124+0000 7f3d467fc700 10 monclient: dump: 2021-05-25T12:08:30.201 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:30.201 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:30.202 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:30.202 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:30.202 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:30.202 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:30.203 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:30.203 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:30.203 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:30.203 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:30.203 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.124+0000 7f3d467fc700 1 -- 172.21.15.102:0/3537010489 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3740865981 0 0) 0x7f3d38009ad0 con 0x7f3d38008d90 2021-05-25T12:08:30.204 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.124+0000 7f3d467fc700 1 -- 172.21.15.102:0/3537010489 --> v1:172.21.15.102:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f3d38008740 con 0x7f3d38008d90 2021-05-25T12:08:30.204 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.124+0000 7f3d467fc700 1 -- 172.21.15.102:0/3537010489 <== mon.2 v1:172.21.15.102:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3929 (unknown 0 0 3596927394) 0x7f3d4c0059b0 con 0x7f3d38008d90 2021-05-25T12:08:30.204 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.124+0000 7f3d467fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:08:30.204 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.124+0000 7f3d467fc700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:08:30.205 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.124+0000 7f3d467fc700 1 -- 172.21.15.102:0/3537010489 >> v1:172.21.15.102:6790/0 conn(0x7f3d38008d90 legacy=0x7f3d38009220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:30.205 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.124+0000 7f3d467fc700 1 -- 172.21.15.102:0/3537010489 >> v1:172.21.15.102:6790/0 conn(0x7f3d38008d90 legacy=0x7f3d38009220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:08:30.206 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:08:30.206 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:08:30.206 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:08:30.206 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "peon", 2021-05-25T12:08:30.207 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 72, 2021-05-25T12:08:30.207 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:08:30.207 INFO:teuthology.orchestra.run.smithi102.stdout: 1, 2021-05-25T12:08:30.207 INFO:teuthology.orchestra.run.smithi102.stdout: 2 2021-05-25T12:08:30.207 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:08:30.208 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 0, 2021-05-25T12:08:30.208 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:08:30.208 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:08:30.208 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:08:30.209 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:08:30.209 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:08:30.209 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:08:30.209 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:08:30.210 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:08:30.210 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:08:30.210 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:08:30.210 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:08:30.210 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:08:30.211 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:08:30.211 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:08:30.211 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:08:30.211 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:08:30.212 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:08:30.212 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:08:30.212 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:08:30.212 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:08:30.213 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:08:30.213 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:08:30.213 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:08:30.213 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:08:30.213 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:30.214 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:30.214 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:08:30.214 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:08:30.214 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:08:30.215 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:08:30.215 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:08:30.215 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:08:30.215 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:08:30.215 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:08:30.216 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:08:30.216 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:08:30.216 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:08:30.216 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:08:30.216 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:08:30.217 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:08:30.217 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:08:30.217 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:08:30.217 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:08:30.218 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:08:30.218 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:08:30.218 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:08:30.218 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:08:30.218 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:08:30.219 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:08:30.219 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:08:30.219 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:08:30.220 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:08:30.220 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:30.220 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:08:30.220 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:30.220 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:08:30.221 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:08:30.221 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:08:30.221 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:08:30.221 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:30.221 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:08:30.222 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:08:30.222 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:08:30.222 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:30.222 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:30.223 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:30.223 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:08:30.223 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:08:30.223 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:08:30.224 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:08:30.224 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:08:30.224 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:30.224 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:30.224 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:08:30.225 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:08:30.225 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:08:30.225 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:08:30.225 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:30.225 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:08:30.226 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:08:30.226 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:08:30.226 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:30.226 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:30.227 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:30.227 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:08:30.227 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:08:30.227 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:08:30.227 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:08:30.228 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:08:30.228 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:30.228 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:30.228 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:08:30.228 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:08:30.229 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:08:30.229 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:08:30.229 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:30.229 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:08:30.229 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:08:30.230 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:08:30.230 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:30.230 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:30.230 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:30.231 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:08:30.231 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:08:30.231 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:08:30.231 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:08:30.231 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:08:30.232 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:30.232 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:30.232 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:30.232 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:08:30.233 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:08:30.233 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:30.233 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:08:30.233 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:08:30.233 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:08:30.234 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:30.234 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:08:30.234 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": [ 2021-05-25T12:08:30.234 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:30.234 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:08:30.235 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:08:30.235 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 2 2021-05-25T12:08:30.235 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:30.235 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:08:30.236 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:08:30.236 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:30.236 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:08:30.236 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:08:30.236 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 2 2021-05-25T12:08:30.237 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:30.237 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:30.237 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:30.237 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:08:30.237 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:08:30.240 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.124+0000 7f3d5ccbd700 10 librados: watch_flush enter 2021-05-25T12:08:30.240 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.124+0000 7f3d5ccbd700 10 librados: watch_flush exit 2021-05-25T12:08:30.240 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.124+0000 7f3d5ccbd700 20 client.244181.objecter shutdown clearing up homeless session... 2021-05-25T12:08:30.241 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.124+0000 7f3d5ccbd700 10 client.244181.objecter successfully canceled tick 2021-05-25T12:08:30.241 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.124+0000 7f3d5ccbd700 1 -- 172.21.15.102:0/3537010489 >> v1:172.21.15.181:6812/29582 conn(0x7f3d34033150 legacy=0x7f3d34035610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:30.241 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.124+0000 7f3d5ccbd700 10 monclient: shutdown 2021-05-25T12:08:30.241 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.124+0000 7f3d5ccbd700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:08:30.242 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.124+0000 7f3d5ccbd700 1 -- 172.21.15.102:0/3537010489 >> v1:172.21.15.102:6790/0 conn(0x7f3d58069130 legacy=0x7f3d58132b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:30.242 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.124+0000 7f3d56d9d700 1 -- 172.21.15.102:0/3537010489 reap_dead start 2021-05-25T12:08:30.242 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.124+0000 7f3d56d9d700 1 -- 172.21.15.102:0/3537010489 reap_dead start 2021-05-25T12:08:30.242 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.124+0000 7f3d5ccbd700 1 -- 172.21.15.102:0/3537010489 shutdown_connections 2021-05-25T12:08:30.243 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.124+0000 7f3d5ccbd700 1 -- 172.21.15.102:0/3537010489 >> 172.21.15.102:0/3537010489 conn(0x7f3d5809b370 msgr2=0x7f3d581254d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:08:30.243 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.124+0000 7f3d5ccbd700 1 -- 172.21.15.102:0/3537010489 shutdown_connections 2021-05-25T12:08:30.243 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.124+0000 7f3d5ccbd700 1 -- 172.21.15.102:0/3537010489 wait complete. 2021-05-25T12:08:30.243 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.124+0000 7f3d5ccbd700 1 librados: shutdown 2021-05-25T12:08:30.244 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2021-05-25T12:08:30.369 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.352+0000 7f596b59e700 10 monclient: get_monmap_and_config 2021-05-25T12:08:30.369 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.352+0000 7f596b59e700 10 monclient: build_initial_monmap 2021-05-25T12:08:30.370 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.352+0000 7f596b59e700 10 monclient: monmap: 2021-05-25T12:08:30.370 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:08:30.370 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:30.370 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:08:30.353243+0000 2021-05-25T12:08:30.370 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:08:30.353243+0000 2021-05-25T12:08:30.371 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:08:30.371 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:30.371 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:08:30.371 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:08:30.372 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:08:30.372 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:30.372 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.352+0000 7f596b59e700 1 Processor -- start 2021-05-25T12:08:30.372 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.352+0000 7f596b59e700 1 -- start start 2021-05-25T12:08:30.372 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.352+0000 7f596b59e700 10 monclient: init 2021-05-25T12:08:30.373 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.353+0000 7f596b59e700 10 monclient: _reopen_session rank -1 2021-05-25T12:08:30.373 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.353+0000 7f596b59e700 10 monclient: _add_conns ranks=[1,0,2] 2021-05-25T12:08:30.374 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.353+0000 7f596b59e700 10 monclient(hunting): picked mon.noname-c con 0x7f596c096d50 addr v1:172.21.15.181:6789/0 2021-05-25T12:08:30.374 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.353+0000 7f596b59e700 10 monclient(hunting): picked mon.noname-a con 0x7f596c12b180 addr v1:172.21.15.102:6789/0 2021-05-25T12:08:30.374 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.353+0000 7f596b59e700 10 monclient(hunting): picked mon.noname-b con 0x7f596c12ec90 addr v1:172.21.15.102:6790/0 2021-05-25T12:08:30.375 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.353+0000 7f596b59e700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f596c093230 con 0x7f596c12b180 2021-05-25T12:08:30.375 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.353+0000 7f596b59e700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f596c0934d0 con 0x7f596c096d50 2021-05-25T12:08:30.375 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.353+0000 7f596b59e700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f596c093770 con 0x7f596c12ec90 2021-05-25T12:08:30.375 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.353+0000 7f596b59e700 10 monclient(hunting): _renew_subs 2021-05-25T12:08:30.376 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.353+0000 7f596b59e700 10 monclient(hunting): authenticate will time out at 1310.950502s 2021-05-25T12:08:30.376 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.353+0000 7f5969d9b700 1 -- >> v1:172.21.15.102:6789/0 conn(0x7f596c12b180 legacy=0x7f596c12d570 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:08:30.376 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.353+0000 7f596ad9d700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f596c12ec90 0x7f596c131080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:57446/0 (socket says 172.21.15.102:57446) 2021-05-25T12:08:30.376 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.353+0000 7f596ad9d700 1 -- 172.21.15.102:0/795977579 learned_addr learned my addr 172.21.15.102:0/795977579 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:08:30.376 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.353+0000 7f596959a700 1 -- 172.21.15.102:0/795977579 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2308342660 0 0) 0x7f596c093770 con 0x7f596c12ec90 2021-05-25T12:08:30.377 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.353+0000 7f596959a700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:30.377 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.353+0000 7f596959a700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:30.377 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.353+0000 7f596959a700 10 monclient(hunting): my global_id is 244190 2021-05-25T12:08:30.377 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.353+0000 7f596959a700 1 -- 172.21.15.102:0/795977579 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5950003270 con 0x7f596c12ec90 2021-05-25T12:08:30.377 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.354+0000 7f596959a700 1 -- 172.21.15.102:0/795977579 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 317825473 0 0) 0x7f5950003270 con 0x7f596c12ec90 2021-05-25T12:08:30.378 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.354+0000 7f596959a700 1 -- 172.21.15.102:0/795977579 >> v1:172.21.15.181:6789/0 conn(0x7f596c096d50 legacy=0x7f596c0924d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:30.378 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.354+0000 7f596959a700 1 -- 172.21.15.102:0/795977579 >> v1:172.21.15.102:6789/0 conn(0x7f596c12b180 legacy=0x7f596c12d570 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-05-25T12:08:30.378 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.354+0000 7f596959a700 10 monclient: _finish_hunting 0 2021-05-25T12:08:30.378 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.354+0000 7f596959a700 1 monclient: found mon.noname-b 2021-05-25T12:08:30.379 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.354+0000 7f596959a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:08:30.379 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.354+0000 7f596959a700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T12:08:30.379 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.354+0000 7f596959a700 1 -- 172.21.15.102:0/795977579 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f596c092cd0 con 0x7f596c12ec90 2021-05-25T12:08:30.379 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.354+0000 7f596959a700 10 monclient: _finish_auth 0 2021-05-25T12:08:30.379 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.354+0000 7f596959a700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:08:30.380 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.354+0000 7f596b59e700 5 monclient: authenticate success, global_id 244190 2021-05-25T12:08:30.380 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.354+0000 7f596b59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:08:30.380 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.354+0000 7f596959a700 1 -- 172.21.15.102:0/795977579 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f595c0019e0 con 0x7f596c12ec90 2021-05-25T12:08:30.380 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.354+0000 7f596959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:30.381 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.354+0000 7f596959a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:08:30.381 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.354+0000 7f596959a700 10 monclient: dump: 2021-05-25T12:08:30.381 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:30.381 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:30.382 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:30.382 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:30.382 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:30.382 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:30.382 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:30.383 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:30.383 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:30.383 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:30.384 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.354+0000 7f596b59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:08:30.384 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.354+0000 7f596959a700 1 -- 172.21.15.102:0/795977579 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f595c001d00 con 0x7f596c12ec90 2021-05-25T12:08:30.384 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.354+0000 7f596959a700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:08:30.385 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.354+0000 7f596b59e700 10 monclient: get_monmap_and_config success 2021-05-25T12:08:30.385 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.354+0000 7f596959a700 1 -- 172.21.15.102:0/795977579 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f595c002090 con 0x7f596c12ec90 2021-05-25T12:08:30.385 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.354+0000 7f596b59e700 4 set_mon_vals no callback set 2021-05-25T12:08:30.385 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.354+0000 7f596959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:30.386 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.354+0000 7f596959a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:08:30.386 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.354+0000 7f596959a700 10 monclient: dump: 2021-05-25T12:08:30.386 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:30.386 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:30.386 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:30.387 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:30.387 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:30.387 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:30.387 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:30.387 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:30.388 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:30.388 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:30.388 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.354+0000 7f596b59e700 10 monclient: shutdown 2021-05-25T12:08:30.388 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.354+0000 7f596b59e700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:08:30.389 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.354+0000 7f596b59e700 1 -- 172.21.15.102:0/795977579 >> v1:172.21.15.102:6790/0 conn(0x7f596c12ec90 legacy=0x7f596c131080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:30.389 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.354+0000 7f596b59e700 1 -- 172.21.15.102:0/795977579 shutdown_connections 2021-05-25T12:08:30.389 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.354+0000 7f596b59e700 1 -- 172.21.15.102:0/795977579 >> 172.21.15.102:0/795977579 conn(0x7f596c0974b0 msgr2=0x7f596c0978c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:08:30.389 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.354+0000 7f596b59e700 1 -- 172.21.15.102:0/795977579 shutdown_connections 2021-05-25T12:08:30.389 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.354+0000 7f596b59e700 1 -- 172.21.15.102:0/795977579 wait complete. 2021-05-25T12:08:30.390 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.355+0000 7f596b59e700 10 monclient: build_initial_monmap 2021-05-25T12:08:30.390 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.355+0000 7f596b59e700 10 monclient: monmap: 2021-05-25T12:08:30.390 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:08:30.390 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:08:30.390 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:08:30.391 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:08:30.391 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:08:30.391 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:08:30.391 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:08:30.392 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:08:30.392 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:08:30.392 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:30.392 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.355+0000 7f596b59e700 1 librados: starting msgr at 2021-05-25T12:08:30.393 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.355+0000 7f596b59e700 1 librados: starting objecter 2021-05-25T12:08:30.393 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.355+0000 7f596b59e700 1 Processor -- start 2021-05-25T12:08:30.393 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.355+0000 7f596b59e700 1 -- start start 2021-05-25T12:08:30.393 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.355+0000 7f596b59e700 1 librados: setting wanted keys 2021-05-25T12:08:30.393 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.355+0000 7f596b59e700 1 librados: calling monclient init 2021-05-25T12:08:30.394 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.355+0000 7f596b59e700 10 monclient: init 2021-05-25T12:08:30.394 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.356+0000 7f596b59e700 10 monclient: _reopen_session rank -1 2021-05-25T12:08:30.395 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.356+0000 7f596b59e700 10 monclient: _add_conns ranks=[1,0,2] 2021-05-25T12:08:30.395 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.356+0000 7f596b59e700 10 monclient(hunting): picked mon.noname-b con 0x7f596c096d50 addr v1:172.21.15.181:6789/0 2021-05-25T12:08:30.395 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.356+0000 7f596b59e700 10 monclient(hunting): picked mon.noname-a con 0x7f596c12b180 addr v1:172.21.15.102:6789/0 2021-05-25T12:08:30.395 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.356+0000 7f596b59e700 10 monclient(hunting): picked mon.noname-c con 0x7f596c12ec90 addr v1:172.21.15.102:6790/0 2021-05-25T12:08:30.396 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.356+0000 7f596b59e700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f596c093e20 con 0x7f596c12b180 2021-05-25T12:08:30.396 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.356+0000 7f596b59e700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f596c096a60 con 0x7f596c096d50 2021-05-25T12:08:30.396 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.356+0000 7f596b59e700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f596c112c20 con 0x7f596c12ec90 2021-05-25T12:08:30.396 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.356+0000 7f596b59e700 10 monclient(hunting): _renew_subs 2021-05-25T12:08:30.397 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.356+0000 7f596b59e700 10 monclient(hunting): authenticate will time out at 1310.953147s 2021-05-25T12:08:30.397 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.356+0000 7f5969d9b700 1 -- >> v1:172.21.15.102:6789/0 conn(0x7f596c12b180 legacy=0x7f596c096350 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:08:30.397 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.356+0000 7f596ad9d700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f596c12ec90 0x7f596c1410d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:57454/0 (socket says 172.21.15.102:57454) 2021-05-25T12:08:30.397 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.356+0000 7f596ad9d700 1 -- 172.21.15.102:0/3283732007 learned_addr learned my addr 172.21.15.102:0/3283732007 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:08:30.398 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.356+0000 7f595a7fc700 10 client.?.objecter ms_handle_connect 0x7f596c12ec90 2021-05-25T12:08:30.398 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.356+0000 7f595a7fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:08:30.398 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.356+0000 7f595a7fc700 10 client.?.objecter ms_handle_connect 0x7f596c096d50 2021-05-25T12:08:30.398 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.356+0000 7f595a7fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:08:30.398 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.356+0000 7f595a7fc700 1 -- 172.21.15.102:0/3283732007 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3747421472 0 0) 0x7f596c112c20 con 0x7f596c12ec90 2021-05-25T12:08:30.399 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.356+0000 7f595a7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:30.399 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.356+0000 7f595a7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:30.399 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.356+0000 7f595a7fc700 10 monclient(hunting): my global_id is 244193 2021-05-25T12:08:30.399 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.356+0000 7f595a7fc700 1 -- 172.21.15.102:0/3283732007 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5948004c10 con 0x7f596c12ec90 2021-05-25T12:08:30.400 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.356+0000 7f595a7fc700 1 -- 172.21.15.102:0/3283732007 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 431509134 0 0) 0x7f596c096a60 con 0x7f596c096d50 2021-05-25T12:08:30.400 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.356+0000 7f595a7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:30.400 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.356+0000 7f595a7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:30.400 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.356+0000 7f595a7fc700 10 monclient(hunting): my global_id is 224446 2021-05-25T12:08:30.401 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.356+0000 7f595a7fc700 1 -- 172.21.15.102:0/3283732007 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f596c112c20 con 0x7f596c096d50 2021-05-25T12:08:30.401 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.356+0000 7f595a7fc700 1 -- 172.21.15.102:0/3283732007 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1894245039 0 0) 0x7f5948004c10 con 0x7f596c12ec90 2021-05-25T12:08:30.401 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.356+0000 7f595a7fc700 1 -- 172.21.15.102:0/3283732007 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f596c096a60 con 0x7f596c12ec90 2021-05-25T12:08:30.401 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.356+0000 7f595a7fc700 1 -- 172.21.15.102:0/3283732007 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f595c0019a0 con 0x7f596c12ec90 2021-05-25T12:08:30.401 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.356+0000 7f595a7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:30.402 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.356+0000 7f595a7fc700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:08:30.402 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.356+0000 7f595a7fc700 10 monclient(hunting): dump: 2021-05-25T12:08:30.402 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:30.402 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:30.402 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:30.403 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:30.403 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:30.403 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:30.403 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:30.404 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:30.404 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:30.404 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:30.404 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.356+0000 7f595a7fc700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:08:30.404 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.356+0000 7f595a7fc700 1 -- 172.21.15.102:0/3283732007 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1400503563 0 0) 0x7f596c112c20 con 0x7f596c096d50 2021-05-25T12:08:30.405 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.356+0000 7f595a7fc700 1 -- 172.21.15.102:0/3283732007 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5948004c10 con 0x7f596c096d50 2021-05-25T12:08:30.405 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.356+0000 7f595a7fc700 1 -- 172.21.15.102:0/3283732007 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f5960001840 con 0x7f596c096d50 2021-05-25T12:08:30.406 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.356+0000 7f595a7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:30.406 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.356+0000 7f595a7fc700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2021-05-25T12:08:30.406 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.356+0000 7f595a7fc700 10 monclient(hunting): dump: 2021-05-25T12:08:30.406 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:30.406 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:30.407 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:30.407 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:30.407 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:30.407 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:30.407 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:30.408 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:30.408 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:30.408 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:30.408 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.356+0000 7f595a7fc700 1 -- 172.21.15.102:0/3283732007 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2636441229 0 0) 0x7f596c096a60 con 0x7f596c12ec90 2021-05-25T12:08:30.408 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.356+0000 7f595a7fc700 1 -- 172.21.15.102:0/3283732007 >> v1:172.21.15.181:6789/0 conn(0x7f596c096d50 legacy=0x7f596c095c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:30.409 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.356+0000 7f595a7fc700 1 -- 172.21.15.102:0/3283732007 >> v1:172.21.15.102:6789/0 conn(0x7f596c12b180 legacy=0x7f596c096350 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-05-25T12:08:30.409 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.356+0000 7f595a7fc700 10 monclient: _finish_hunting 0 2021-05-25T12:08:30.409 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.356+0000 7f595a7fc700 1 monclient: found mon.c 2021-05-25T12:08:30.409 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.356+0000 7f595a7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:08:30.410 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.356+0000 7f595a7fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:08:30.410 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.356+0000 7f595a7fc700 1 -- 172.21.15.102:0/3283732007 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f596c112ec0 con 0x7f596c12ec90 2021-05-25T12:08:30.410 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.356+0000 7f595a7fc700 10 monclient: _finish_auth 0 2021-05-25T12:08:30.410 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.356+0000 7f595a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:08:30.410 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.356+0000 7f596b59e700 5 monclient: authenticate success, global_id 244193 2021-05-25T12:08:30.411 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.356+0000 7f596b59e700 10 monclient: _renew_subs 2021-05-25T12:08:30.411 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.356+0000 7f596b59e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:08:30.411 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.356+0000 7f596b59e700 1 -- 172.21.15.102:0/3283732007 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f596c1119f0 con 0x7f596c12ec90 2021-05-25T12:08:30.412 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.356+0000 7f596b59e700 10 client.244193.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:08:30.412 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.356+0000 7f596b59e700 10 monclient: _renew_subs 2021-05-25T12:08:30.412 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.356+0000 7f596b59e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:08:30.412 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.356+0000 7f596b59e700 1 -- 172.21.15.102:0/3283732007 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f596c12a940 con 0x7f596c12ec90 2021-05-25T12:08:30.412 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.356+0000 7f596b59e700 1 librados: init done 2021-05-25T12:08:30.413 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.356+0000 7f595a7fc700 1 -- 172.21.15.102:0/3283732007 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f595c004dd0 con 0x7f596c12ec90 2021-05-25T12:08:30.413 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.356+0000 7f595a7fc700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:08:30.413 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.356+0000 7f595a7fc700 1 -- 172.21.15.102:0/3283732007 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f595c005160 con 0x7f596c12ec90 2021-05-25T12:08:30.413 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.356+0000 7f595a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:30.413 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.356+0000 7f595a7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:08:30.414 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.356+0000 7f5968d99700 4 set_mon_vals no callback set 2021-05-25T12:08:30.414 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.356+0000 7f595a7fc700 10 monclient: dump: 2021-05-25T12:08:30.414 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:30.414 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:30.415 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:30.415 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:30.416 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:30.416 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:30.416 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:30.416 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:30.416 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:30.417 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:30.417 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.357+0000 7f596b59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:08:30.417 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.357+0000 7f596b59e700 1 -- 172.21.15.102:0/3283732007 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f594c0065a0 con 0x7f594c002840 2021-05-25T12:08:30.417 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.357+0000 7f595a7fc700 1 -- 172.21.15.102:0/3283732007 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f595c0052c0 con 0x7f596c12ec90 2021-05-25T12:08:30.418 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.357+0000 7f595a7fc700 1 -- 172.21.15.102:0/3283732007 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(272..272 src has 1..272) v4 ==== 5433+0+0 (unknown 1668784668 0 0) 0x7f595c03d9f0 con 0x7f596c12ec90 2021-05-25T12:08:30.418 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.357+0000 7f595a7fc700 10 client.244193.objecter ms_dispatch 0x7f596c001550 osd_map(272..272 src has 1..272) v4 2021-05-25T12:08:30.418 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.357+0000 7f595a7fc700 3 client.244193.objecter handle_osd_map got epochs [272,272] > 0 2021-05-25T12:08:30.418 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.357+0000 7f595a7fc700 3 client.244193.objecter handle_osd_map decoding full epoch 272 2021-05-25T12:08:30.418 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.357+0000 7f595a7fc700 20 client.244193.objecter dump_active .. 0 homeless 2021-05-25T12:08:30.419 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.357+0000 7f595a7fc700 10 client.244193.objecter ms_handle_connect 0x7f594c002840 2021-05-25T12:08:30.419 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.357+0000 7f595a7fc700 10 client.244193.objecter resend_mon_ops 2021-05-25T12:08:30.419 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.357+0000 7f5969d9b700 10 monclient: get_auth_request con 0x7f59480332a0 auth_method 0 2021-05-25T12:08:30.419 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.357+0000 7f595a7fc700 1 -- 172.21.15.102:0/3283732007 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1030328085 0 0) 0x7f594c0065a0 con 0x7f594c002840 2021-05-25T12:08:30.419 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.357+0000 7f595a7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:30.420 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.357+0000 7f595a7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:30.420 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.357+0000 7f595a7fc700 10 monclient(hunting): my global_id is 224449 2021-05-25T12:08:30.420 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.357+0000 7f595a7fc700 1 -- 172.21.15.102:0/3283732007 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f596c096a60 con 0x7f594c002840 2021-05-25T12:08:30.420 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.358+0000 7f595a7fc700 10 client.244193.objecter ms_handle_connect 0x7f59480332a0 2021-05-25T12:08:30.420 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.358+0000 7f595a7fc700 1 -- 172.21.15.102:0/3283732007 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 209932421 0 0) 0x7f596c096a60 con 0x7f594c002840 2021-05-25T12:08:30.421 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.358+0000 7f595a7fc700 1 -- 172.21.15.102:0/3283732007 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f594c0065a0 con 0x7f594c002840 2021-05-25T12:08:30.421 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.358+0000 7f595a7fc700 1 -- 172.21.15.102:0/3283732007 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f5960001f80 con 0x7f594c002840 2021-05-25T12:08:30.421 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.358+0000 7f595a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:30.421 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.358+0000 7f595a7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T12:08:30.421 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.358+0000 7f595a7fc700 10 monclient: dump: 2021-05-25T12:08:30.422 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:30.422 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:30.422 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:30.423 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:30.423 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:30.423 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:30.423 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:30.423 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:30.424 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:30.424 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:30.424 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.358+0000 7f595a7fc700 1 -- 172.21.15.102:0/3283732007 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2156851388 0 0) 0x7f594c0065a0 con 0x7f594c002840 2021-05-25T12:08:30.424 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.358+0000 7f595a7fc700 1 -- 172.21.15.102:0/3283732007 --> v1:172.21.15.181:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f594c009e40 con 0x7f594c002840 2021-05-25T12:08:30.424 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.358+0000 7f595a7fc700 1 -- 172.21.15.102:0/3283732007 <== mon.1 v1:172.21.15.181:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7f5960004310 con 0x7f594c002840 2021-05-25T12:08:30.425 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.358+0000 7f595a7fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:08:30.425 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.358+0000 7f595a7fc700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:08:30.425 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.358+0000 7f595a7fc700 1 -- 172.21.15.102:0/3283732007 >> v1:172.21.15.181:6789/0 conn(0x7f594c002840 legacy=0x7f594c004ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:30.425 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.358+0000 7f595a7fc700 1 -- 172.21.15.102:0/3283732007 >> v1:172.21.15.181:6789/0 conn(0x7f594c002840 legacy=0x7f594c004ce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:08:30.426 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.365+0000 7f596b59e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:08:30.426 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.365+0000 7f596b59e700 1 -- 172.21.15.102:0/3283732007 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f594c007510 con 0x7f594c0068b0 2021-05-25T12:08:30.427 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.365+0000 7f595a7fc700 10 client.244193.objecter ms_handle_connect 0x7f594c0068b0 2021-05-25T12:08:30.427 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.365+0000 7f595a7fc700 10 client.244193.objecter resend_mon_ops 2021-05-25T12:08:30.427 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.365+0000 7f595a7fc700 1 -- 172.21.15.102:0/3283732007 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4144563738 0 0) 0x7f594c007510 con 0x7f594c0068b0 2021-05-25T12:08:30.427 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.365+0000 7f595a7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:30.428 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.365+0000 7f595a7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:30.428 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.365+0000 7f595a7fc700 10 monclient(hunting): my global_id is 224452 2021-05-25T12:08:30.428 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.365+0000 7f595a7fc700 1 -- 172.21.15.102:0/3283732007 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f594c0065a0 con 0x7f594c0068b0 2021-05-25T12:08:30.428 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.366+0000 7f595a7fc700 1 -- 172.21.15.102:0/3283732007 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 44208963 0 0) 0x7f594c0065a0 con 0x7f594c0068b0 2021-05-25T12:08:30.429 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.366+0000 7f595a7fc700 1 -- 172.21.15.102:0/3283732007 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f594c007510 con 0x7f594c0068b0 2021-05-25T12:08:30.429 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.366+0000 7f595a7fc700 1 -- 172.21.15.102:0/3283732007 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f5960002770 con 0x7f594c0068b0 2021-05-25T12:08:30.429 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.366+0000 7f595a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:30.429 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.366+0000 7f595a7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T12:08:30.430 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.366+0000 7f595a7fc700 10 monclient: dump: 2021-05-25T12:08:30.430 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:30.430 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:30.430 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:30.431 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:30.431 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:30.431 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:30.431 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:30.432 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:30.432 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:30.432 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:30.432 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.366+0000 7f595a7fc700 1 -- 172.21.15.102:0/3283732007 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 958792548 0 0) 0x7f594c007510 con 0x7f594c0068b0 2021-05-25T12:08:30.432 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.366+0000 7f595a7fc700 1 -- 172.21.15.102:0/3283732007 --> v1:172.21.15.181:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f594c008770 con 0x7f594c0068b0 2021-05-25T12:08:30.433 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.366+0000 7f595a7fc700 1 -- 172.21.15.102:0/3283732007 <== mon.1 v1:172.21.15.181:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4101 (unknown 0 0 3703200846) 0x7f5960003bb0 con 0x7f594c0068b0 2021-05-25T12:08:30.433 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.366+0000 7f595a7fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:08:30.433 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.366+0000 7f595a7fc700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:08:30.433 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.366+0000 7f595a7fc700 1 -- 172.21.15.102:0/3283732007 >> v1:172.21.15.181:6789/0 conn(0x7f594c0068b0 legacy=0x7f594c006d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:30.434 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.366+0000 7f595a7fc700 1 -- 172.21.15.102:0/3283732007 >> v1:172.21.15.181:6789/0 conn(0x7f594c0068b0 legacy=0x7f594c006d40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:08:30.434 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:08:30.435 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:08:30.435 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:08:30.435 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "leader", 2021-05-25T12:08:30.435 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 72, 2021-05-25T12:08:30.435 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:08:30.436 INFO:teuthology.orchestra.run.smithi102.stdout: 1, 2021-05-25T12:08:30.436 INFO:teuthology.orchestra.run.smithi102.stdout: 2 2021-05-25T12:08:30.436 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:08:30.436 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 0, 2021-05-25T12:08:30.436 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:08:30.437 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:08:30.437 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:08:30.437 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:08:30.437 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:08:30.438 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:08:30.438 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:08:30.438 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:08:30.438 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:08:30.438 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:08:30.439 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:08:30.439 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:08:30.439 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:08:30.440 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:08:30.440 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:08:30.440 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:08:30.440 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:08:30.440 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:08:30.441 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:08:30.441 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:08:30.441 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:08:30.441 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:08:30.442 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:08:30.442 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:08:30.442 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:30.443 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:30.443 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:08:30.443 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:08:30.443 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:08:30.444 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:08:30.444 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:08:30.444 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:08:30.444 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:08:30.444 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:08:30.445 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:08:30.445 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:08:30.445 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:08:30.445 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:08:30.445 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:08:30.446 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:08:30.446 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:08:30.446 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:08:30.446 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:08:30.447 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:08:30.447 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:08:30.447 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:08:30.447 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:08:30.447 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:08:30.448 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:08:30.448 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:08:30.448 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:08:30.448 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:08:30.449 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:30.449 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:08:30.449 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:30.449 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:08:30.450 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:08:30.450 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:08:30.450 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:08:30.450 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:30.450 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:08:30.451 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:08:30.451 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:08:30.451 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:30.451 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:30.451 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:30.452 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:08:30.452 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:08:30.452 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:08:30.452 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:08:30.453 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:08:30.453 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:30.453 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:30.453 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:08:30.453 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:08:30.454 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:08:30.454 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:08:30.454 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:30.454 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:08:30.455 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:08:30.455 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:08:30.455 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:30.455 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:30.456 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:30.456 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:08:30.456 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:08:30.456 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:08:30.456 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:08:30.457 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:08:30.457 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:30.457 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:30.457 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:08:30.457 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:08:30.458 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:08:30.458 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:08:30.458 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:30.458 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:08:30.458 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:08:30.459 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:08:30.459 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:30.459 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:30.459 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:30.460 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:08:30.460 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:08:30.460 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:08:30.460 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:08:30.460 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:08:30.461 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:30.461 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:30.461 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:30.461 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:08:30.461 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:08:30.462 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:30.462 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:08:30.462 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:08:30.462 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:08:30.463 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:30.463 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:08:30.463 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": [ 2021-05-25T12:08:30.463 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:30.464 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:08:30.464 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:08:30.464 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 4 2021-05-25T12:08:30.464 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:30.464 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:08:30.465 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:08:30.465 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:30.465 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:08:30.465 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:08:30.465 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 6 2021-05-25T12:08:30.466 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:30.466 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:08:30.466 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": [ 2021-05-25T12:08:30.466 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:30.467 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:08:30.467 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:08:30.467 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:08:30.467 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:30.467 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:30.468 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:30.468 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:08:30.468 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:08:30.470 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.366+0000 7f596b59e700 10 librados: watch_flush enter 2021-05-25T12:08:30.471 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.366+0000 7f596b59e700 10 librados: watch_flush exit 2021-05-25T12:08:30.471 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.366+0000 7f596b59e700 20 client.244193.objecter shutdown clearing up homeless session... 2021-05-25T12:08:30.471 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.366+0000 7f596b59e700 10 client.244193.objecter successfully canceled tick 2021-05-25T12:08:30.472 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.366+0000 7f596b59e700 1 -- 172.21.15.102:0/3283732007 >> v1:172.21.15.181:6812/29582 conn(0x7f59480332a0 legacy=0x7f5948035760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:30.472 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.366+0000 7f596b59e700 10 monclient: shutdown 2021-05-25T12:08:30.472 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.366+0000 7f596b59e700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:08:30.472 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.366+0000 7f596b59e700 1 -- 172.21.15.102:0/3283732007 >> v1:172.21.15.102:6790/0 conn(0x7f596c12ec90 legacy=0x7f596c1410d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:30.473 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.366+0000 7f596ad9d700 1 -- 172.21.15.102:0/3283732007 reap_dead start 2021-05-25T12:08:30.473 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.366+0000 7f596ad9d700 1 -- 172.21.15.102:0/3283732007 reap_dead start 2021-05-25T12:08:30.473 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.366+0000 7f596b59e700 1 -- 172.21.15.102:0/3283732007 shutdown_connections 2021-05-25T12:08:30.473 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.366+0000 7f596b59e700 1 -- 172.21.15.102:0/3283732007 >> 172.21.15.102:0/3283732007 conn(0x7f596c0974b0 msgr2=0x7f596c0987c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:08:30.474 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.367+0000 7f596b59e700 1 -- 172.21.15.102:0/3283732007 shutdown_connections 2021-05-25T12:08:30.474 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.367+0000 7f596b59e700 1 -- 172.21.15.102:0/3283732007 wait complete. 2021-05-25T12:08:30.474 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:30.367+0000 7f596b59e700 1 librados: shutdown 2021-05-25T12:08:30.475 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2021-05-25T12:08:32.775 INFO:tasks.workunit.client.0.smithi181.stdout:::163:head 2021-05-25T12:08:32.775 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:5d165639:::164:head 2021-05-25T12:08:32.775 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:f43765fc:::165:head 2021-05-25T12:08:32.776 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:b4c720e9:::166:head 2021-05-25T12:08:32.776 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:e694b040:::167:head 2021-05-25T12:08:32.776 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:afa38db2:::168:head 2021-05-25T12:08:32.776 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:77ba9f53:::169:head 2021-05-25T12:08:32.777 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:87495034:::170:head 2021-05-25T12:08:32.777 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:7c96bf0e:::171:head 2021-05-25T12:08:32.777 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:dbe346cc:::172:head 2021-05-25T12:08:32.777 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:e943ec24:::173:head 2021-05-25T12:08:32.777 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:f97a9c0c:::174:head 2021-05-25T12:08:32.778 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:6f26e74d:::175:head 2021-05-25T12:08:32.778 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:4f95e106:::176:head 2021-05-25T12:08:32.778 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:0e6f2f8f:::177:head 2021-05-25T12:08:32.778 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:05db05f1:::178:head 2021-05-25T12:08:32.778 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:38a78d66:::179:head 2021-05-25T12:08:32.779 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:d095610b:::180:head 2021-05-25T12:08:32.779 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:a1a9d709:::181:head 2021-05-25T12:08:32.779 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:1e5d39db:::182:head 2021-05-25T12:08:32.779 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:f7df4fb9:::183:head 2021-05-25T12:08:32.780 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:03a7f161:::184:head 2021-05-25T12:08:32.780 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:ba70721e:::185:head 2021-05-25T12:08:32.780 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:28e5662d:::186:head 2021-05-25T12:08:32.780 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:973d52de:::187:head 2021-05-25T12:08:32.780 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:4303eb1c:::188:head 2021-05-25T12:08:32.781 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:b990b48e:::189:head 2021-05-25T12:08:32.781 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:29b8165b:::190:head 2021-05-25T12:08:32.781 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:3547f197:::191:head 2021-05-25T12:08:32.781 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:7e260936:::192:head 2021-05-25T12:08:32.781 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:1abec7b1:::193:head 2021-05-25T12:08:32.782 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:10fdda93:::194:head 2021-05-25T12:08:32.782 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:15817eea:::195:head 2021-05-25T12:08:32.782 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:770bab57:::196:head 2021-05-25T12:08:32.782 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:ed9e13e7:::197:head 2021-05-25T12:08:32.783 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:71471a8f:::198:head 2021-05-25T12:08:32.783 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: checking for 178:10fb1d02:::199:head 2021-05-25T12:08:32.783 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetWrite (7738 ms) 2021-05-25T12:08:32.783 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetTrim 2021-05-25T12:08:32.783 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: got ls 1621944451,0 2021-05-25T12:08:32.784 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: first is 1621944451 2021-05-25T12:08:32.784 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: got ls 1621944451,0 2021-05-25T12:08:32.784 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: got ls 1621944451,0 2021-05-25T12:08:32.784 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: got ls 1621944451,1621944453,1621944454,0 2021-05-25T12:08:32.784 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: got ls 1621944451,1621944453,1621944454,0 2021-05-25T12:08:32.785 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: got ls 1621944451,1621944453,1621944454,0 2021-05-25T12:08:32.785 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: got ls 1621944451,1621944453,1621944454,1621944456,1621944457,0 2021-05-25T12:08:32.785 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: got ls 1621944451,1621944453,1621944454,1621944456,1621944457,0 2021-05-25T12:08:32.785 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: got ls 1621944451,1621944453,1621944454,1621944456,1621944457,0 2021-05-25T12:08:32.785 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: got ls 1621944451,1621944453,1621944454,1621944456,1621944457,1621944459,1621944460,0 2021-05-25T12:08:32.786 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: got ls 1621944451,1621944453,1621944454,1621944456,1621944457,1621944459,1621944460,0 2021-05-25T12:08:32.786 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: got ls 1621944451,1621944453,1621944454,1621944456,1621944457,1621944459,1621944460,0 2021-05-25T12:08:32.786 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: got ls 1621944454,1621944456,1621944457,1621944459,1621944460,1621944462,1621944463,0 2021-05-25T12:08:32.786 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: first now 1621944454, trimmed 2021-05-25T12:08:32.787 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetTrim (28221 ms) 2021-05-25T12:08:32.787 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteOn2ndRead 2021-05-25T12:08:32.787 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: foo0 2021-05-25T12:08:32.787 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: verifying foo0 is eventually promoted 2021-05-25T12:08:32.787 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteOn2ndRead (13776 ms) 2021-05-25T12:08:32.788 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ProxyRead 2021-05-25T12:08:50.476 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2021-05-25T12:08:50.477 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2021-05-25T12:08:50.477 INFO:tasks.ceph.mon.a:Restarting daemon 2021-05-25T12:08:50.477 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2021-05-25T12:08:50.480 INFO:tasks.ceph.mon.a:Started 2021-05-25T12:08:50.481 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2021-05-25T12:08:50.481 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2021-05-25T12:08:50.608 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.605+0000 7fa1c23fc700 10 monclient: get_monmap_and_config 2021-05-25T12:08:50.608 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.605+0000 7fa1c23fc700 10 monclient: build_initial_monmap 2021-05-25T12:08:50.609 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.605+0000 7fa1c23fc700 10 monclient: monmap: 2021-05-25T12:08:50.609 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:08:50.609 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:50.609 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:08:50.607608+0000 2021-05-25T12:08:50.610 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:08:50.607608+0000 2021-05-25T12:08:50.610 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:08:50.610 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:50.610 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:08:50.610 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:08:50.611 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:08:50.611 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:50.619 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.606+0000 7fa1c23fc700 1 Processor -- start 2021-05-25T12:08:50.619 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.607+0000 7fa1c23fc700 1 -- start start 2021-05-25T12:08:50.620 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.607+0000 7fa1c23fc700 10 monclient: init 2021-05-25T12:08:50.620 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.607+0000 7fa1c23fc700 10 monclient: _reopen_session rank -1 2021-05-25T12:08:50.620 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.607+0000 7fa1c23fc700 10 monclient: _add_conns ranks=[0,2,1] 2021-05-25T12:08:50.620 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.607+0000 7fa1c23fc700 10 monclient(hunting): picked mon.noname-a con 0x7fa1bc095c40 addr v1:172.21.15.102:6789/0 2021-05-25T12:08:50.620 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.607+0000 7fa1c23fc700 10 monclient(hunting): picked mon.noname-b con 0x7fa1bc096020 addr v1:172.21.15.102:6790/0 2021-05-25T12:08:50.621 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.607+0000 7fa1c23fc700 10 monclient(hunting): picked mon.noname-c con 0x7fa1bc12e5c0 addr v1:172.21.15.181:6789/0 2021-05-25T12:08:50.621 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.608+0000 7fa1c23fc700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1bc092990 con 0x7fa1bc095c40 2021-05-25T12:08:50.621 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.608+0000 7fa1c23fc700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1bc092c30 con 0x7fa1bc12e5c0 2021-05-25T12:08:50.621 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.608+0000 7fa1c23fc700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1bc092ed0 con 0x7fa1bc096020 2021-05-25T12:08:50.621 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.608+0000 7fa1c23fc700 10 monclient(hunting): _renew_subs 2021-05-25T12:08:50.622 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.608+0000 7fa1c23fc700 10 monclient(hunting): authenticate will time out at 1331.206033s 2021-05-25T12:08:50.622 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.608+0000 7fa1c0bf9700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fa1bc096020 0x7fa1bc0964d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:57476/0 (socket says 172.21.15.102:57476) 2021-05-25T12:08:50.622 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.608+0000 7fa1c0bf9700 1 -- 172.21.15.102:0/2439300819 learned_addr learned my addr 172.21.15.102:0/2439300819 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:08:50.622 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.608+0000 7fa1c13fa700 1 --1- 172.21.15.102:0/2439300819 >> v1:172.21.15.102:6789/0 conn(0x7fa1bc095c40 0x7fa1bc093290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:34368/0 (socket says 172.21.15.102:34368) 2021-05-25T12:08:50.623 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.609+0000 7fa1abfff700 1 -- 172.21.15.102:0/2439300819 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2002822128 0 0) 0x7fa1bc092ed0 con 0x7fa1bc096020 2021-05-25T12:08:50.623 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.609+0000 7fa1abfff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:50.623 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.609+0000 7fa1abfff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:50.623 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.609+0000 7fa1abfff700 10 monclient(hunting): my global_id is 244211 2021-05-25T12:08:50.624 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.609+0000 7fa1abfff700 1 -- 172.21.15.102:0/2439300819 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1a4003270 con 0x7fa1bc096020 2021-05-25T12:08:50.624 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.609+0000 7fa1abfff700 1 -- 172.21.15.102:0/2439300819 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2298025239 0 0) 0x7fa1a4003270 con 0x7fa1bc096020 2021-05-25T12:08:50.624 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.609+0000 7fa1abfff700 1 -- 172.21.15.102:0/2439300819 >> v1:172.21.15.181:6789/0 conn(0x7fa1bc12e5c0 legacy=0x7fa1bc1309b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:50.624 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.609+0000 7fa1abfff700 1 -- 172.21.15.102:0/2439300819 >> v1:172.21.15.102:6789/0 conn(0x7fa1bc095c40 legacy=0x7fa1bc093290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:50.624 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.609+0000 7fa1abfff700 10 monclient: _finish_hunting 0 2021-05-25T12:08:50.625 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.609+0000 7fa1abfff700 1 monclient: found mon.noname-b 2021-05-25T12:08:50.625 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.609+0000 7fa1abfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:08:50.625 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.609+0000 7fa1abfff700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T12:08:50.625 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.609+0000 7fa1abfff700 1 -- 172.21.15.102:0/2439300819 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1bc092430 con 0x7fa1bc096020 2021-05-25T12:08:50.625 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.609+0000 7fa1abfff700 10 monclient: _finish_auth 0 2021-05-25T12:08:50.626 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.609+0000 7fa1abfff700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:08:50.626 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.609+0000 7fa1c23fc700 5 monclient: authenticate success, global_id 244211 2021-05-25T12:08:50.626 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.609+0000 7fa1abfff700 1 -- 172.21.15.102:0/2439300819 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fa1b0001610 con 0x7fa1bc096020 2021-05-25T12:08:50.626 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.609+0000 7fa1c23fc700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:08:50.626 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.609+0000 7fa1abfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:50.627 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.609+0000 7fa1abfff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:08:50.627 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.609+0000 7fa1abfff700 10 monclient: dump: 2021-05-25T12:08:50.627 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:50.627 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:50.627 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:50.628 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:50.628 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:50.628 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:50.628 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:50.628 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:50.629 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:50.629 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:50.629 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.609+0000 7fa1c23fc700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:08:50.629 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.610+0000 7fa1abfff700 1 -- 172.21.15.102:0/2439300819 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa1b00018c0 con 0x7fa1bc096020 2021-05-25T12:08:50.629 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.610+0000 7fa1abfff700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:08:50.630 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.610+0000 7fa1c23fc700 10 monclient: get_monmap_and_config success 2021-05-25T12:08:50.630 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.610+0000 7fa1abfff700 1 -- 172.21.15.102:0/2439300819 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fa1b0001c10 con 0x7fa1bc096020 2021-05-25T12:08:50.630 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.610+0000 7fa1abfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:50.630 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.610+0000 7fa1c23fc700 4 set_mon_vals no callback set 2021-05-25T12:08:50.630 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.610+0000 7fa1abfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:08:50.631 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.610+0000 7fa1abfff700 10 monclient: dump: 2021-05-25T12:08:50.631 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:50.631 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:50.631 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:50.632 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:50.632 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:50.632 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:50.632 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:50.632 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:50.633 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:50.633 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:50.633 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.610+0000 7fa1c23fc700 10 monclient: shutdown 2021-05-25T12:08:50.633 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.610+0000 7fa1c23fc700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:08:50.633 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.610+0000 7fa1c23fc700 1 -- 172.21.15.102:0/2439300819 >> v1:172.21.15.102:6790/0 conn(0x7fa1bc096020 legacy=0x7fa1bc0964d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:50.634 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.610+0000 7fa1c23fc700 1 -- 172.21.15.102:0/2439300819 shutdown_connections 2021-05-25T12:08:50.634 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.610+0000 7fa1c23fc700 1 -- 172.21.15.102:0/2439300819 >> 172.21.15.102:0/2439300819 conn(0x7fa1bc0974b0 msgr2=0x7fa1bc0978c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:08:50.634 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.610+0000 7fa1c23fc700 1 -- 172.21.15.102:0/2439300819 shutdown_connections 2021-05-25T12:08:50.634 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.610+0000 7fa1c23fc700 1 -- 172.21.15.102:0/2439300819 wait complete. 2021-05-25T12:08:50.634 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.611+0000 7fa1c23fc700 10 monclient: build_initial_monmap 2021-05-25T12:08:50.635 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.611+0000 7fa1c23fc700 10 monclient: monmap: 2021-05-25T12:08:50.635 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:08:50.635 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:08:50.635 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:08:50.635 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:08:50.636 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:08:50.636 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:08:50.636 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:08:50.636 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:08:50.636 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:08:50.637 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:50.637 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.611+0000 7fa1c23fc700 1 librados: starting msgr at 2021-05-25T12:08:50.637 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.611+0000 7fa1c23fc700 1 librados: starting objecter 2021-05-25T12:08:50.637 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.611+0000 7fa1c23fc700 1 Processor -- start 2021-05-25T12:08:50.637 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.611+0000 7fa1c23fc700 1 -- start start 2021-05-25T12:08:50.638 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.611+0000 7fa1c23fc700 1 librados: setting wanted keys 2021-05-25T12:08:50.638 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.611+0000 7fa1c23fc700 1 librados: calling monclient init 2021-05-25T12:08:50.638 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.611+0000 7fa1c23fc700 10 monclient: init 2021-05-25T12:08:50.638 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.612+0000 7fa1c23fc700 10 monclient: _reopen_session rank -1 2021-05-25T12:08:50.639 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.612+0000 7fa1c23fc700 10 monclient: _add_conns ranks=[2,0,1] 2021-05-25T12:08:50.639 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.612+0000 7fa1c23fc700 10 monclient(hunting): picked mon.noname-c con 0x7fa1bc095c40 addr v1:172.21.15.102:6790/0 2021-05-25T12:08:50.639 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.612+0000 7fa1c23fc700 10 monclient(hunting): picked mon.noname-a con 0x7fa1bc096020 addr v1:172.21.15.102:6789/0 2021-05-25T12:08:50.639 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.612+0000 7fa1c23fc700 10 monclient(hunting): picked mon.noname-b con 0x7fa1bc12e5c0 addr v1:172.21.15.181:6789/0 2021-05-25T12:08:50.639 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.612+0000 7fa1c23fc700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1bc099400 con 0x7fa1bc096020 2021-05-25T12:08:50.640 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.612+0000 7fa1c23fc700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1bc113a80 con 0x7fa1bc12e5c0 2021-05-25T12:08:50.640 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.612+0000 7fa1c23fc700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1bc113d20 con 0x7fa1bc095c40 2021-05-25T12:08:50.640 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.612+0000 7fa1c23fc700 10 monclient(hunting): _renew_subs 2021-05-25T12:08:50.640 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.612+0000 7fa1c23fc700 10 monclient(hunting): authenticate will time out at 1331.210487s 2021-05-25T12:08:50.640 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.612+0000 7fa1c13fa700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fa1bc095c40 0x7fa1bc1320d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:57482/0 (socket says 172.21.15.102:57482) 2021-05-25T12:08:50.641 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.612+0000 7fa1c13fa700 1 -- 172.21.15.102:0/2060392207 learned_addr learned my addr 172.21.15.102:0/2060392207 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:08:50.641 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.612+0000 7fa1a97fa700 10 client.?.objecter ms_handle_connect 0x7fa1bc095c40 2021-05-25T12:08:50.641 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.612+0000 7fa1a97fa700 10 client.?.objecter resend_mon_ops 2021-05-25T12:08:50.641 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.612+0000 7fa1a97fa700 10 client.?.objecter ms_handle_connect 0x7fa1bc096020 2021-05-25T12:08:50.642 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.612+0000 7fa1a97fa700 10 client.?.objecter resend_mon_ops 2021-05-25T12:08:50.642 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.612+0000 7fa1a97fa700 10 client.?.objecter ms_handle_connect 0x7fa1bc12e5c0 2021-05-25T12:08:50.642 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.612+0000 7fa1a97fa700 10 client.?.objecter resend_mon_ops 2021-05-25T12:08:50.642 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.612+0000 7fa1a97fa700 1 -- 172.21.15.102:0/2060392207 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2712663205 0 0) 0x7fa1bc113a80 con 0x7fa1bc12e5c0 2021-05-25T12:08:50.643 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.612+0000 7fa1a97fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:50.643 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.612+0000 7fa1a97fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:50.643 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.612+0000 7fa1a97fa700 10 monclient(hunting): my global_id is 224473 2021-05-25T12:08:50.643 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.612+0000 7fa1a97fa700 1 -- 172.21.15.102:0/2060392207 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa19c004c10 con 0x7fa1bc12e5c0 2021-05-25T12:08:50.643 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.613+0000 7fa1a97fa700 1 -- 172.21.15.102:0/2060392207 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1362109934 0 0) 0x7fa19c004c10 con 0x7fa1bc12e5c0 2021-05-25T12:08:50.644 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.613+0000 7fa1a97fa700 1 -- 172.21.15.102:0/2060392207 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa1bc113a80 con 0x7fa1bc12e5c0 2021-05-25T12:08:50.644 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.613+0000 7fa1a97fa700 1 -- 172.21.15.102:0/2060392207 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fa1b8001bd0 con 0x7fa1bc12e5c0 2021-05-25T12:08:50.644 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.613+0000 7fa1a97fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:50.644 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.613+0000 7fa1a97fa700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:08:50.644 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.613+0000 7fa1a97fa700 10 monclient(hunting): dump: 2021-05-25T12:08:50.645 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:50.645 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:50.645 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:50.645 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:50.645 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:50.646 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:50.646 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:50.646 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:50.646 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:50.646 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:50.647 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.613+0000 7fa1a97fa700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:08:50.647 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.613+0000 7fa1a97fa700 1 -- 172.21.15.102:0/2060392207 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2803771139 0 0) 0x7fa1bc113a80 con 0x7fa1bc12e5c0 2021-05-25T12:08:50.647 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.613+0000 7fa1a97fa700 1 -- 172.21.15.102:0/2060392207 >> v1:172.21.15.102:6790/0 conn(0x7fa1bc095c40 legacy=0x7fa1bc1320d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:50.647 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.613+0000 7fa1a97fa700 1 -- 172.21.15.102:0/2060392207 >> v1:172.21.15.102:6789/0 conn(0x7fa1bc096020 legacy=0x7fa1bc128300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:50.648 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.613+0000 7fa1a97fa700 10 monclient: _finish_hunting 0 2021-05-25T12:08:50.648 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.613+0000 7fa1a97fa700 1 monclient: found mon.b 2021-05-25T12:08:50.648 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.613+0000 7fa1a97fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:08:50.648 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.613+0000 7fa1a97fa700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T12:08:50.648 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.613+0000 7fa1a97fa700 1 -- 172.21.15.102:0/2060392207 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1bc129360 con 0x7fa1bc12e5c0 2021-05-25T12:08:50.649 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.613+0000 7fa1a97fa700 10 monclient: _finish_auth 0 2021-05-25T12:08:50.649 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.613+0000 7fa1a97fa700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:08:50.649 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.613+0000 7fa1c23fc700 5 monclient: authenticate success, global_id 224473 2021-05-25T12:08:50.649 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.613+0000 7fa1c23fc700 10 monclient: _renew_subs 2021-05-25T12:08:50.649 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.613+0000 7fa1c23fc700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T12:08:50.650 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.613+0000 7fa1c23fc700 1 -- 172.21.15.102:0/2060392207 --> v1:172.21.15.181:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1bc13bf90 con 0x7fa1bc12e5c0 2021-05-25T12:08:50.650 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.613+0000 7fa1c23fc700 10 client.224473.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:08:50.650 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.613+0000 7fa1c23fc700 10 monclient: _renew_subs 2021-05-25T12:08:50.650 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.613+0000 7fa1c23fc700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T12:08:50.650 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.613+0000 7fa1c23fc700 1 -- 172.21.15.102:0/2060392207 --> v1:172.21.15.181:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1bc0683f0 con 0x7fa1bc12e5c0 2021-05-25T12:08:50.651 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.613+0000 7fa1c23fc700 1 librados: init done 2021-05-25T12:08:50.651 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.613+0000 7fa1a97fa700 1 -- 172.21.15.102:0/2060392207 <== mon.1 v1:172.21.15.181:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa1b8001f70 con 0x7fa1bc12e5c0 2021-05-25T12:08:50.651 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.613+0000 7fa1a97fa700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:08:50.651 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.613+0000 7fa1a97fa700 1 -- 172.21.15.102:0/2060392207 <== mon.1 v1:172.21.15.181:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fa1b8002280 con 0x7fa1bc12e5c0 2021-05-25T12:08:50.651 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.613+0000 7fa1a97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:50.652 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.613+0000 7fa1ab7fe700 4 set_mon_vals no callback set 2021-05-25T12:08:50.652 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.613+0000 7fa1a97fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T12:08:50.652 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.613+0000 7fa1a97fa700 10 monclient: dump: 2021-05-25T12:08:50.652 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:50.652 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:50.653 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:50.653 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:50.653 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:50.653 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:50.653 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:50.654 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:50.654 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:50.654 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:50.654 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.614+0000 7fa1a97fa700 1 -- 172.21.15.102:0/2060392207 <== mon.1 v1:172.21.15.181:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7fa1b8002460 con 0x7fa1bc12e5c0 2021-05-25T12:08:50.655 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.614+0000 7fa1c23fc700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:08:50.655 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.614+0000 7fa1c23fc700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:08:50.655 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.614+0000 7fa1c23fc700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T12:08:50.655 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.614+0000 7fa1c23fc700 1 -- 172.21.15.102:0/2060392207 --> v1:172.21.15.181:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa194004db0 con 0x7fa1bc12e5c0 2021-05-25T12:08:50.655 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.614+0000 7fa1a97fa700 1 -- 172.21.15.102:0/2060392207 <== mon.1 v1:172.21.15.181:6789/0 8 ==== osd_map(292..292 src has 250..292) v4 ==== 6275+0+0 (unknown 2449608657 0 0) 0x7fa1b803b5c0 con 0x7fa1bc12e5c0 2021-05-25T12:08:50.656 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.614+0000 7fa1a97fa700 10 client.224473.objecter ms_dispatch 0x7fa1bc001550 osd_map(292..292 src has 250..292) v4 2021-05-25T12:08:50.656 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.614+0000 7fa1a97fa700 3 client.224473.objecter handle_osd_map got epochs [292,292] > 0 2021-05-25T12:08:50.656 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.614+0000 7fa1a97fa700 3 client.224473.objecter handle_osd_map decoding full epoch 292 2021-05-25T12:08:50.656 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.614+0000 7fa1a97fa700 20 client.224473.objecter dump_active .. 0 homeless 2021-05-25T12:08:50.656 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.614+0000 7fa1c13fa700 10 monclient: get_auth_request con 0x7fa19c033130 auth_method 0 2021-05-25T12:08:50.657 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.614+0000 7fa1a97fa700 10 client.224473.objecter ms_handle_connect 0x7fa19c033130 2021-05-25T12:08:50.657 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.619+0000 7fa1a97fa700 1 -- 172.21.15.102:0/2060392207 <== mon.1 v1:172.21.15.181:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7fa1b8010ad0 con 0x7fa1bc12e5c0 2021-05-25T12:08:50.658 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.619+0000 7fa1a97fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:08:50.658 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.619+0000 7fa1a97fa700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:08:50.840 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.837+0000 7fa1c23fc700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2021-05-25T12:08:50.840 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.837+0000 7fa1c23fc700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2021-05-25T12:08:50.840 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.837+0000 7fa1c23fc700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T12:08:50.840 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.837+0000 7fa1c23fc700 1 -- 172.21.15.102:0/2060392207 --> v1:172.21.15.181:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fa194005060 con 0x7fa1bc12e5c0 2021-05-25T12:08:50.841 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.838+0000 7fa1a97fa700 1 -- 172.21.15.102:0/2060392207 <== mon.1 v1:172.21.15.181:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1342 (unknown 254746657 0 4193038650) 0x7fa1b80106c0 con 0x7fa1bc12e5c0 2021-05-25T12:08:50.841 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.838+0000 7fa1a97fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2021-05-25T12:08:50.841 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.838+0000 7fa1a97fa700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:08:50.842 INFO:teuthology.orchestra.run.smithi102.stdout:{"election_epoch":74,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138303579357183","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"20ee91b6-8935-4b98-b868-85c03789ccc4","modified":"2021-05-25T11:59:11.011522Z","created":"2021-05-25T11:59:11.011522Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6790","nonce":0}]},"addr":"172.21.15.102:6790/0","public_addr":"172.21.15.102:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2021-05-25T12:08:50.843 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.839+0000 7fa1c23fc700 10 librados: watch_flush enter 2021-05-25T12:08:50.844 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.839+0000 7fa1c23fc700 10 librados: watch_flush exit 2021-05-25T12:08:50.844 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.839+0000 7fa1c23fc700 20 client.224473.objecter shutdown clearing up homeless session... 2021-05-25T12:08:50.845 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.839+0000 7fa1c23fc700 10 client.224473.objecter successfully canceled tick 2021-05-25T12:08:50.845 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.839+0000 7fa1c23fc700 1 -- 172.21.15.102:0/2060392207 >> v1:172.21.15.181:6812/29582 conn(0x7fa19c033130 legacy=0x7fa19c0355f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:50.845 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.839+0000 7fa1c23fc700 10 monclient: shutdown 2021-05-25T12:08:50.846 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.839+0000 7fa1c23fc700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:08:50.846 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.839+0000 7fa1c23fc700 1 -- 172.21.15.102:0/2060392207 >> v1:172.21.15.181:6789/0 conn(0x7fa1bc12e5c0 legacy=0x7fa1bc143460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:50.846 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.839+0000 7fa1c23fc700 1 -- 172.21.15.102:0/2060392207 shutdown_connections 2021-05-25T12:08:50.847 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.839+0000 7fa1c23fc700 1 -- 172.21.15.102:0/2060392207 >> 172.21.15.102:0/2060392207 conn(0x7fa1bc0974b0 msgr2=0x7fa1bc0943f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:08:50.847 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.840+0000 7fa1c23fc700 1 -- 172.21.15.102:0/2060392207 shutdown_connections 2021-05-25T12:08:50.847 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.840+0000 7fa1c23fc700 1 -- 172.21.15.102:0/2060392207 wait complete. 2021-05-25T12:08:50.847 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.840+0000 7fa1c23fc700 1 librados: shutdown 2021-05-25T12:08:50.853 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2021-05-25T12:08:50.853 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2021-05-25T12:08:50.967 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.964+0000 7fb8bede0700 10 monclient: get_monmap_and_config 2021-05-25T12:08:50.967 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.964+0000 7fb8bede0700 10 monclient: build_initial_monmap 2021-05-25T12:08:50.967 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.964+0000 7fb8bede0700 10 monclient: monmap: 2021-05-25T12:08:50.967 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:08:50.968 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:50.968 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:08:50.966699+0000 2021-05-25T12:08:50.968 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:08:50.966699+0000 2021-05-25T12:08:50.968 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:08:50.969 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:50.969 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:08:50.969 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:08:50.969 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:08:50.969 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:50.982 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.965+0000 7fb8bede0700 1 Processor -- start 2021-05-25T12:08:50.983 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.965+0000 7fb8bede0700 1 -- start start 2021-05-25T12:08:50.983 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.965+0000 7fb8bede0700 10 monclient: init 2021-05-25T12:08:50.983 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.965+0000 7fb8bede0700 10 monclient: _reopen_session rank -1 2021-05-25T12:08:50.983 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.965+0000 7fb8bede0700 10 monclient: _add_conns ranks=[0,1,2] 2021-05-25T12:08:50.984 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.965+0000 7fb8bede0700 10 monclient(hunting): picked mon.noname-a con 0x7fb8b8090990 addr v1:172.21.15.102:6789/0 2021-05-25T12:08:50.984 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.965+0000 7fb8bede0700 10 monclient(hunting): picked mon.noname-c con 0x7fb8b8090d70 addr v1:172.21.15.181:6789/0 2021-05-25T12:08:50.984 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.965+0000 7fb8bede0700 10 monclient(hunting): picked mon.noname-b con 0x7fb8b812e330 addr v1:172.21.15.102:6790/0 2021-05-25T12:08:50.984 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.965+0000 7fb8bede0700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8b8096fa0 con 0x7fb8b8090990 2021-05-25T12:08:50.985 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.965+0000 7fb8bede0700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8b80926a0 con 0x7fb8b8090d70 2021-05-25T12:08:50.985 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.965+0000 7fb8bede0700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8b8092940 con 0x7fb8b812e330 2021-05-25T12:08:50.985 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.965+0000 7fb8bede0700 10 monclient(hunting): _renew_subs 2021-05-25T12:08:50.985 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.965+0000 7fb8bede0700 10 monclient(hunting): authenticate will time out at 1331.563946s 2021-05-25T12:08:50.985 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.966+0000 7fb8bd37d700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fb8b812e330 0x7fb8b8130720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:57494/0 (socket says 172.21.15.102:57494) 2021-05-25T12:08:50.986 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.966+0000 7fb8bd37d700 1 -- 172.21.15.102:0/3864364911 learned_addr learned my addr 172.21.15.102:0/3864364911 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:08:50.986 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.966+0000 7fb8bcb7c700 1 --1- 172.21.15.102:0/3864364911 >> v1:172.21.15.102:6789/0 conn(0x7fb8b8090990 0x7fb8b8093240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:34380/0 (socket says 172.21.15.102:34380) 2021-05-25T12:08:50.986 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.966+0000 7fb8af7fe700 1 -- 172.21.15.102:0/3864364911 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3188602418 0 0) 0x7fb8b8096fa0 con 0x7fb8b8090990 2021-05-25T12:08:50.986 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.966+0000 7fb8af7fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:50.987 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.966+0000 7fb8af7fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:50.987 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.966+0000 7fb8af7fe700 10 monclient(hunting): my global_id is 264099 2021-05-25T12:08:50.987 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.966+0000 7fb8af7fe700 1 -- 172.21.15.102:0/3864364911 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8a4004380 con 0x7fb8b8090990 2021-05-25T12:08:50.987 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.966+0000 7fb8af7fe700 1 -- 172.21.15.102:0/3864364911 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4050124888 0 0) 0x7fb8b8092940 con 0x7fb8b812e330 2021-05-25T12:08:50.987 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.966+0000 7fb8af7fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:50.988 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.966+0000 7fb8af7fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:50.988 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.966+0000 7fb8af7fe700 10 monclient(hunting): my global_id is 244214 2021-05-25T12:08:50.988 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.966+0000 7fb8af7fe700 1 -- 172.21.15.102:0/3864364911 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8b8096fa0 con 0x7fb8b812e330 2021-05-25T12:08:50.988 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.966+0000 7fb8af7fe700 1 -- 172.21.15.102:0/3864364911 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4179799075 0 0) 0x7fb8b8096fa0 con 0x7fb8b812e330 2021-05-25T12:08:50.988 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.966+0000 7fb8af7fe700 1 -- 172.21.15.102:0/3864364911 >> v1:172.21.15.181:6789/0 conn(0x7fb8b8090d70 legacy=0x7fb8b8091200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:50.989 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.966+0000 7fb8af7fe700 1 -- 172.21.15.102:0/3864364911 >> v1:172.21.15.102:6789/0 conn(0x7fb8b8090990 legacy=0x7fb8b8093240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:50.989 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.966+0000 7fb8af7fe700 10 monclient: _finish_hunting 0 2021-05-25T12:08:50.989 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.966+0000 7fb8af7fe700 1 monclient: found mon.noname-b 2021-05-25T12:08:50.989 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.966+0000 7fb8af7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:08:50.989 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.966+0000 7fb8af7fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T12:08:50.990 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.966+0000 7fb8af7fe700 1 -- 172.21.15.102:0/3864364911 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8b8092be0 con 0x7fb8b812e330 2021-05-25T12:08:50.990 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.966+0000 7fb8af7fe700 10 monclient: _finish_auth 0 2021-05-25T12:08:50.990 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.966+0000 7fb8af7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:08:50.990 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.966+0000 7fb8af7fe700 1 -- 172.21.15.102:0/3864364911 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fb8b4001990 con 0x7fb8b812e330 2021-05-25T12:08:50.991 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.966+0000 7fb8bede0700 5 monclient: authenticate success, global_id 244214 2021-05-25T12:08:50.991 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.966+0000 7fb8bede0700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:08:50.991 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.966+0000 7fb8af7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:50.991 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.966+0000 7fb8af7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:08:50.991 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.966+0000 7fb8af7fe700 10 monclient: dump: 2021-05-25T12:08:50.992 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:50.992 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:50.992 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:50.992 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:50.992 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:50.993 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:50.993 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:50.993 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:50.993 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:50.993 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:50.994 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.966+0000 7fb8bede0700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:08:50.994 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.966+0000 7fb8af7fe700 1 -- 172.21.15.102:0/3864364911 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb8b4001b40 con 0x7fb8b812e330 2021-05-25T12:08:50.994 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.966+0000 7fb8af7fe700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:08:50.994 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.966+0000 7fb8bede0700 10 monclient: get_monmap_and_config success 2021-05-25T12:08:50.995 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.966+0000 7fb8af7fe700 1 -- 172.21.15.102:0/3864364911 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fb8b4001ed0 con 0x7fb8b812e330 2021-05-25T12:08:50.995 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.966+0000 7fb8bede0700 4 set_mon_vals no callback set 2021-05-25T12:08:50.995 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.966+0000 7fb8af7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:50.995 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.966+0000 7fb8af7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:08:50.995 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.966+0000 7fb8af7fe700 10 monclient: dump: 2021-05-25T12:08:50.996 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:50.996 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:50.996 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:50.996 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:50.997 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:50.997 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:50.997 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:50.997 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:50.997 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:50.998 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:50.998 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.967+0000 7fb8bede0700 10 monclient: shutdown 2021-05-25T12:08:50.998 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.967+0000 7fb8bede0700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:08:50.998 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.967+0000 7fb8bede0700 1 -- 172.21.15.102:0/3864364911 >> v1:172.21.15.102:6790/0 conn(0x7fb8b812e330 legacy=0x7fb8b8130720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:50.998 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.967+0000 7fb8bede0700 1 -- 172.21.15.102:0/3864364911 shutdown_connections 2021-05-25T12:08:50.999 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.967+0000 7fb8bede0700 1 -- 172.21.15.102:0/3864364911 >> 172.21.15.102:0/3864364911 conn(0x7fb8b809b370 msgr2=0x7fb8b8097430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:08:50.999 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.967+0000 7fb8bede0700 1 -- 172.21.15.102:0/3864364911 shutdown_connections 2021-05-25T12:08:50.999 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.967+0000 7fb8bede0700 1 -- 172.21.15.102:0/3864364911 wait complete. 2021-05-25T12:08:50.999 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.967+0000 7fb8bede0700 10 monclient: build_initial_monmap 2021-05-25T12:08:50.999 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.967+0000 7fb8bede0700 10 monclient: monmap: 2021-05-25T12:08:51.000 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:08:51.000 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:08:51.000 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:08:51.000 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:08:51.001 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:08:51.001 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:08:51.001 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:08:51.001 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:08:51.001 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:08:51.002 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:51.002 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.967+0000 7fb8bede0700 1 librados: starting msgr at 2021-05-25T12:08:51.002 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.967+0000 7fb8bede0700 1 librados: starting objecter 2021-05-25T12:08:51.002 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.967+0000 7fb8bede0700 1 Processor -- start 2021-05-25T12:08:51.002 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.968+0000 7fb8bede0700 1 -- start start 2021-05-25T12:08:51.003 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.968+0000 7fb8bede0700 1 librados: setting wanted keys 2021-05-25T12:08:51.003 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.968+0000 7fb8bede0700 1 librados: calling monclient init 2021-05-25T12:08:51.003 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.968+0000 7fb8bede0700 10 monclient: init 2021-05-25T12:08:51.003 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.968+0000 7fb8bede0700 10 monclient: _reopen_session rank -1 2021-05-25T12:08:51.004 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.968+0000 7fb8bede0700 10 monclient: _add_conns ranks=[0,2,1] 2021-05-25T12:08:51.004 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.968+0000 7fb8bede0700 10 monclient(hunting): picked mon.noname-a con 0x7fb8b8090990 addr v1:172.21.15.102:6789/0 2021-05-25T12:08:51.004 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.968+0000 7fb8bede0700 10 monclient(hunting): picked mon.noname-c con 0x7fb8b8090d70 addr v1:172.21.15.102:6790/0 2021-05-25T12:08:51.004 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.968+0000 7fb8bede0700 10 monclient(hunting): picked mon.noname-b con 0x7fb8b812e330 addr v1:172.21.15.181:6789/0 2021-05-25T12:08:51.004 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.968+0000 7fb8bede0700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8b8111d90 con 0x7fb8b8090990 2021-05-25T12:08:51.005 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.968+0000 7fb8bede0700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8b8113fb0 con 0x7fb8b812e330 2021-05-25T12:08:51.005 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.968+0000 7fb8bede0700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8b8114250 con 0x7fb8b8090d70 2021-05-25T12:08:51.005 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.968+0000 7fb8bede0700 10 monclient(hunting): _renew_subs 2021-05-25T12:08:51.005 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.968+0000 7fb8bede0700 10 monclient(hunting): authenticate will time out at 1331.566677s 2021-05-25T12:08:51.005 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.968+0000 7fb8bcb7c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fb8b8090990 0x7fb8b8110f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:34386/0 (socket says 172.21.15.102:34386) 2021-05-25T12:08:51.006 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.968+0000 7fb8bcb7c700 1 -- 172.21.15.102:0/1272998958 learned_addr learned my addr 172.21.15.102:0/1272998958 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:08:51.006 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.968+0000 7fb8affff700 1 --1- 172.21.15.102:0/1272998958 >> v1:172.21.15.102:6790/0 conn(0x7fb8b8090d70 0x7fb8b8111680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:57498/0 (socket says 172.21.15.102:57498) 2021-05-25T12:08:51.006 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.968+0000 7fb8acff9700 10 client.?.objecter ms_handle_connect 0x7fb8b8090990 2021-05-25T12:08:51.006 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.968+0000 7fb8acff9700 10 client.?.objecter resend_mon_ops 2021-05-25T12:08:51.007 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.968+0000 7fb8acff9700 10 client.?.objecter ms_handle_connect 0x7fb8b8090d70 2021-05-25T12:08:51.007 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.968+0000 7fb8acff9700 10 client.?.objecter resend_mon_ops 2021-05-25T12:08:51.007 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.968+0000 7fb8acff9700 1 -- 172.21.15.102:0/1272998958 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2425489775 0 0) 0x7fb8b8111d90 con 0x7fb8b8090990 2021-05-25T12:08:51.007 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.968+0000 7fb8acff9700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:51.007 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.968+0000 7fb8acff9700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:51.008 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.968+0000 7fb8acff9700 10 monclient(hunting): my global_id is 264102 2021-05-25T12:08:51.008 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.968+0000 7fb8acff9700 1 -- 172.21.15.102:0/1272998958 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb898004c10 con 0x7fb8b8090990 2021-05-25T12:08:51.008 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.968+0000 7fb8acff9700 10 client.?.objecter ms_handle_connect 0x7fb8b812e330 2021-05-25T12:08:51.008 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.968+0000 7fb8acff9700 10 client.?.objecter resend_mon_ops 2021-05-25T12:08:51.008 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.968+0000 7fb8acff9700 1 -- 172.21.15.102:0/1272998958 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 824986417 0 0) 0x7fb8b8114250 con 0x7fb8b8090d70 2021-05-25T12:08:51.009 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.968+0000 7fb8acff9700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:51.009 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.968+0000 7fb8acff9700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:51.009 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.968+0000 7fb8acff9700 10 monclient(hunting): my global_id is 244217 2021-05-25T12:08:51.009 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.968+0000 7fb8acff9700 1 -- 172.21.15.102:0/1272998958 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8b8111d90 con 0x7fb8b8090d70 2021-05-25T12:08:51.009 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.969+0000 7fb8acff9700 1 -- 172.21.15.102:0/1272998958 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1853374024 0 0) 0x7fb8b8113fb0 con 0x7fb8b812e330 2021-05-25T12:08:51.010 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.969+0000 7fb8acff9700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:51.010 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.969+0000 7fb8acff9700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:51.010 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.969+0000 7fb8acff9700 10 monclient(hunting): my global_id is 224479 2021-05-25T12:08:51.010 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.969+0000 7fb8acff9700 1 -- 172.21.15.102:0/1272998958 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8b8114250 con 0x7fb8b812e330 2021-05-25T12:08:51.010 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.969+0000 7fb8acff9700 1 -- 172.21.15.102:0/1272998958 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 564955533 0 0) 0x7fb898004c10 con 0x7fb8b8090990 2021-05-25T12:08:51.011 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.969+0000 7fb8acff9700 1 -- 172.21.15.102:0/1272998958 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb8b8113fb0 con 0x7fb8b8090990 2021-05-25T12:08:51.011 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.969+0000 7fb8acff9700 1 -- 172.21.15.102:0/1272998958 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fb8a8001a70 con 0x7fb8b8090990 2021-05-25T12:08:51.011 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.969+0000 7fb8acff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:51.011 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.969+0000 7fb8acff9700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:08:51.011 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.969+0000 7fb8acff9700 10 monclient(hunting): dump: 2021-05-25T12:08:51.012 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:51.012 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:51.012 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:51.012 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:51.012 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:51.013 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:51.013 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:51.013 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:51.013 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:51.013 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:51.014 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.969+0000 7fb8acff9700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:08:51.014 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.969+0000 7fb8acff9700 1 -- 172.21.15.102:0/1272998958 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2366651249 0 0) 0x7fb8b8111d90 con 0x7fb8b8090d70 2021-05-25T12:08:51.014 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.969+0000 7fb8acff9700 1 -- 172.21.15.102:0/1272998958 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb898004c10 con 0x7fb8b8090d70 2021-05-25T12:08:51.014 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.969+0000 7fb8acff9700 1 -- 172.21.15.102:0/1272998958 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fb8a0001840 con 0x7fb8b8090d70 2021-05-25T12:08:51.015 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.969+0000 7fb8acff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:51.015 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.969+0000 7fb8acff9700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2021-05-25T12:08:51.015 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.969+0000 7fb8acff9700 10 monclient(hunting): dump: 2021-05-25T12:08:51.015 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:51.015 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:51.016 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:51.016 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:51.016 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:51.016 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:51.017 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:51.017 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:51.017 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:51.017 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:51.017 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.969+0000 7fb8acff9700 1 -- 172.21.15.102:0/1272998958 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3419186367 0 0) 0x7fb8b8114250 con 0x7fb8b812e330 2021-05-25T12:08:51.018 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.969+0000 7fb8acff9700 1 -- 172.21.15.102:0/1272998958 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb8b8111d90 con 0x7fb8b812e330 2021-05-25T12:08:51.018 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.969+0000 7fb8acff9700 1 -- 172.21.15.102:0/1272998958 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fb8b4001950 con 0x7fb8b812e330 2021-05-25T12:08:51.018 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.969+0000 7fb8acff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:51.018 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.969+0000 7fb8acff9700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2021-05-25T12:08:51.019 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.969+0000 7fb8acff9700 10 monclient(hunting): dump: 2021-05-25T12:08:51.019 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:51.019 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:51.019 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:51.019 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:51.020 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:51.020 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:51.020 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:51.020 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:51.020 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:51.020 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:51.021 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.969+0000 7fb8acff9700 1 -- 172.21.15.102:0/1272998958 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1745964681 0 0) 0x7fb8b8113fb0 con 0x7fb8b8090990 2021-05-25T12:08:51.021 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.969+0000 7fb8acff9700 1 -- 172.21.15.102:0/1272998958 >> v1:172.21.15.102:6790/0 conn(0x7fb8b8090d70 legacy=0x7fb8b8111680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:51.021 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.969+0000 7fb8acff9700 1 -- 172.21.15.102:0/1272998958 >> v1:172.21.15.181:6789/0 conn(0x7fb8b812e330 legacy=0x7fb8b81410f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:51.021 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.969+0000 7fb8acff9700 10 monclient: _finish_hunting 0 2021-05-25T12:08:51.021 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.969+0000 7fb8acff9700 1 monclient: found mon.a 2021-05-25T12:08:51.022 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.969+0000 7fb8acff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:08:51.022 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.969+0000 7fb8acff9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:08:51.022 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.969+0000 7fb8acff9700 1 -- 172.21.15.102:0/1272998958 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8b81144f0 con 0x7fb8b8090990 2021-05-25T12:08:51.022 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.969+0000 7fb8acff9700 10 monclient: _finish_auth 0 2021-05-25T12:08:51.022 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.969+0000 7fb8acff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:08:51.023 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.969+0000 7fb8bede0700 5 monclient: authenticate success, global_id 264102 2021-05-25T12:08:51.023 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.969+0000 7fb8bede0700 10 monclient: _renew_subs 2021-05-25T12:08:51.023 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.969+0000 7fb8bede0700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:08:51.023 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.969+0000 7fb8bede0700 1 -- 172.21.15.102:0/1272998958 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8b8112620 con 0x7fb8b8090990 2021-05-25T12:08:51.024 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.969+0000 7fb8bede0700 10 client.264102.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:08:51.024 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.969+0000 7fb8bede0700 10 monclient: _renew_subs 2021-05-25T12:08:51.024 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.969+0000 7fb8bede0700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:08:51.024 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.969+0000 7fb8bede0700 1 -- 172.21.15.102:0/1272998958 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8b8113480 con 0x7fb8b8090990 2021-05-25T12:08:51.024 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.969+0000 7fb8acff9700 1 -- 172.21.15.102:0/1272998958 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb8a8001d30 con 0x7fb8b8090990 2021-05-25T12:08:51.025 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.969+0000 7fb8bede0700 1 librados: init done 2021-05-25T12:08:51.025 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.969+0000 7fb8acff9700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:08:51.025 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.969+0000 7fb8acff9700 1 -- 172.21.15.102:0/1272998958 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fb8a8004980 con 0x7fb8b8090990 2021-05-25T12:08:51.025 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.969+0000 7fb8acff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:51.025 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.969+0000 7fb8acff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:08:51.026 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.969+0000 7fb8acff9700 10 monclient: dump: 2021-05-25T12:08:51.026 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:51.026 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:51.026 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:51.026 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:51.027 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:51.027 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:51.027 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:51.027 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:51.027 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:51.028 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:51.028 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.969+0000 7fb8aeffd700 4 set_mon_vals no callback set 2021-05-25T12:08:51.028 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.970+0000 7fb8acff9700 1 -- 172.21.15.102:0/1272998958 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7fb8a800dca0 con 0x7fb8b8090990 2021-05-25T12:08:51.028 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.970+0000 7fb8bede0700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:08:51.029 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.970+0000 7fb8bede0700 1 -- 172.21.15.102:0/1272998958 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fb8b8094ba0 con 0x7fb8b8114c10 2021-05-25T12:08:51.029 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.970+0000 7fb8acff9700 1 -- 172.21.15.102:0/1272998958 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(293..293 src has 250..293) v4 ==== 6275+0+0 (unknown 4031507246 0 0) 0x7fb8a803d8e0 con 0x7fb8b8090990 2021-05-25T12:08:51.029 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.970+0000 7fb8acff9700 10 client.264102.objecter ms_dispatch 0x7fb8b8001550 osd_map(293..293 src has 250..293) v4 2021-05-25T12:08:51.029 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.970+0000 7fb8acff9700 3 client.264102.objecter handle_osd_map got epochs [293,293] > 0 2021-05-25T12:08:51.029 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.970+0000 7fb8acff9700 3 client.264102.objecter handle_osd_map decoding full epoch 293 2021-05-25T12:08:51.030 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.970+0000 7fb8acff9700 20 client.264102.objecter dump_active .. 0 homeless 2021-05-25T12:08:51.030 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.970+0000 7fb8acff9700 10 client.264102.objecter ms_handle_connect 0x7fb8b8114c10 2021-05-25T12:08:51.030 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.970+0000 7fb8acff9700 10 client.264102.objecter resend_mon_ops 2021-05-25T12:08:51.030 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.970+0000 7fb8acff9700 1 -- 172.21.15.102:0/1272998958 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1097898124 0 0) 0x7fb8b8094ba0 con 0x7fb8b8114c10 2021-05-25T12:08:51.031 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.970+0000 7fb8acff9700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:51.031 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.970+0000 7fb8acff9700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:51.031 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.970+0000 7fb8acff9700 10 monclient(hunting): my global_id is 264105 2021-05-25T12:08:51.031 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.970+0000 7fb8acff9700 1 -- 172.21.15.102:0/1272998958 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8b8113fb0 con 0x7fb8b8114c10 2021-05-25T12:08:51.031 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.970+0000 7fb8bd37d700 10 monclient: get_auth_request con 0x7fb8980334a0 auth_method 0 2021-05-25T12:08:51.032 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.970+0000 7fb8acff9700 1 -- 172.21.15.102:0/1272998958 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 512310231 0 0) 0x7fb8b8113fb0 con 0x7fb8b8114c10 2021-05-25T12:08:51.032 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.970+0000 7fb8acff9700 1 -- 172.21.15.102:0/1272998958 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb8b8094ba0 con 0x7fb8b8114c10 2021-05-25T12:08:51.032 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.970+0000 7fb8acff9700 1 -- 172.21.15.102:0/1272998958 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fb8a00021c0 con 0x7fb8b8114c10 2021-05-25T12:08:51.032 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.970+0000 7fb8acff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:51.032 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.970+0000 7fb8acff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:08:51.033 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.970+0000 7fb8acff9700 10 monclient: dump: 2021-05-25T12:08:51.033 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:51.033 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:51.033 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:51.033 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:51.034 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:51.034 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:51.034 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:51.034 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:51.035 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:51.035 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:51.035 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.970+0000 7fb8acff9700 1 -- 172.21.15.102:0/1272998958 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 946484978 0 0) 0x7fb8b8094ba0 con 0x7fb8b8114c10 2021-05-25T12:08:51.035 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.970+0000 7fb8acff9700 1 -- 172.21.15.102:0/1272998958 --> v1:172.21.15.102:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb8b8059730 con 0x7fb8b8114c10 2021-05-25T12:08:51.035 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.970+0000 7fb8acff9700 10 client.264102.objecter ms_handle_connect 0x7fb8980334a0 2021-05-25T12:08:51.036 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.971+0000 7fb8acff9700 1 -- 172.21.15.102:0/1272998958 <== mon.0 v1:172.21.15.102:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7fb8a0002570 con 0x7fb8b8114c10 2021-05-25T12:08:51.036 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.971+0000 7fb8acff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:08:51.036 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.971+0000 7fb8acff9700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:08:51.036 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.971+0000 7fb8acff9700 1 -- 172.21.15.102:0/1272998958 >> v1:172.21.15.102:6789/0 conn(0x7fb8b8114c10 legacy=0x7fb8b8058d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:51.037 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.971+0000 7fb8acff9700 1 -- 172.21.15.102:0/1272998958 >> v1:172.21.15.102:6789/0 conn(0x7fb8b8114c10 legacy=0x7fb8b8058d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:08:51.037 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.977+0000 7fb8bede0700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:08:51.037 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.977+0000 7fb8bede0700 1 -- 172.21.15.102:0/1272998958 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fb8b8096500 con 0x7fb8b8064d10 2021-05-25T12:08:51.037 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.977+0000 7fb8acff9700 10 client.264102.objecter ms_handle_connect 0x7fb8b8064d10 2021-05-25T12:08:51.037 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.977+0000 7fb8acff9700 10 client.264102.objecter resend_mon_ops 2021-05-25T12:08:51.038 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.977+0000 7fb8acff9700 1 -- 172.21.15.102:0/1272998958 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2898013854 0 0) 0x7fb8b8096500 con 0x7fb8b8064d10 2021-05-25T12:08:51.038 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.977+0000 7fb8acff9700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:51.038 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.977+0000 7fb8acff9700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:51.038 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.977+0000 7fb8acff9700 10 monclient(hunting): my global_id is 264108 2021-05-25T12:08:51.038 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.977+0000 7fb8acff9700 1 -- 172.21.15.102:0/1272998958 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8b8094ba0 con 0x7fb8b8064d10 2021-05-25T12:08:51.039 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.978+0000 7fb8acff9700 1 -- 172.21.15.102:0/1272998958 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4238213740 0 0) 0x7fb8b8094ba0 con 0x7fb8b8064d10 2021-05-25T12:08:51.039 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.978+0000 7fb8acff9700 1 -- 172.21.15.102:0/1272998958 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb8b8096500 con 0x7fb8b8064d10 2021-05-25T12:08:51.039 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.978+0000 7fb8acff9700 1 -- 172.21.15.102:0/1272998958 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fb8a0002b90 con 0x7fb8b8064d10 2021-05-25T12:08:51.039 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.978+0000 7fb8acff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:51.040 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.978+0000 7fb8acff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:08:51.040 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.978+0000 7fb8acff9700 10 monclient: dump: 2021-05-25T12:08:51.040 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:51.040 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:51.040 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:51.041 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:51.041 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:51.041 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:51.041 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:51.041 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:51.042 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:51.042 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:51.042 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.978+0000 7fb8acff9700 1 -- 172.21.15.102:0/1272998958 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3123604928 0 0) 0x7fb8b8096500 con 0x7fb8b8064d10 2021-05-25T12:08:51.042 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.978+0000 7fb8acff9700 1 -- 172.21.15.102:0/1272998958 --> v1:172.21.15.102:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fb8b805d870 con 0x7fb8b8064d10 2021-05-25T12:08:51.043 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.978+0000 7fb8acff9700 1 -- 172.21.15.102:0/1272998958 <== mon.0 v1:172.21.15.102:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3770 (unknown 0 0 1166450520) 0x7fb8a0006ed0 con 0x7fb8b8064d10 2021-05-25T12:08:51.043 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.978+0000 7fb8acff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:08:51.043 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.978+0000 7fb8acff9700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:08:51.043 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.978+0000 7fb8acff9700 1 -- 172.21.15.102:0/1272998958 >> v1:172.21.15.102:6789/0 conn(0x7fb8b8064d10 legacy=0x7fb8b8146830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:51.043 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.978+0000 7fb8acff9700 1 -- 172.21.15.102:0/1272998958 >> v1:172.21.15.102:6789/0 conn(0x7fb8b8064d10 legacy=0x7fb8b8146830 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:08:51.044 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:08:51.044 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:08:51.044 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:08:51.045 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "peon", 2021-05-25T12:08:51.045 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 74, 2021-05-25T12:08:51.045 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:08:51.045 INFO:teuthology.orchestra.run.smithi102.stdout: 0, 2021-05-25T12:08:51.046 INFO:teuthology.orchestra.run.smithi102.stdout: 1, 2021-05-25T12:08:51.046 INFO:teuthology.orchestra.run.smithi102.stdout: 2 2021-05-25T12:08:51.046 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:08:51.046 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 0, 2021-05-25T12:08:51.046 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:08:51.047 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:08:51.047 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:08:51.047 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:08:51.047 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:08:51.048 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:08:51.048 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:08:51.048 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:08:51.048 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:08:51.048 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:08:51.049 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:08:51.049 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:08:51.049 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:08:51.049 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:08:51.050 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:08:51.050 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:08:51.050 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:08:51.050 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:08:51.050 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:08:51.051 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:08:51.051 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:08:51.051 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:08:51.051 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:08:51.051 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:08:51.052 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:51.052 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:51.052 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:08:51.052 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:08:51.053 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:08:51.053 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:08:51.053 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:08:51.053 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:08:51.053 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:08:51.054 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:08:51.054 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:08:51.054 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:08:51.054 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:08:51.055 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:08:51.055 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:08:51.055 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:08:51.055 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:08:51.055 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:08:51.056 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:08:51.056 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:08:51.056 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:08:51.056 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:08:51.056 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:08:51.057 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:08:51.057 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:08:51.057 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:08:51.057 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:08:51.057 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:08:51.058 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:51.058 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:08:51.058 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:51.058 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:08:51.059 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:08:51.059 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:08:51.059 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:08:51.059 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:51.059 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:08:51.060 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:08:51.060 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:08:51.060 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:51.060 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:51.060 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:51.061 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:08:51.061 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:08:51.061 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:08:51.061 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:08:51.062 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:08:51.062 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:51.062 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:51.062 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:08:51.062 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:08:51.063 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:08:51.063 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:08:51.063 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:51.063 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:08:51.064 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:08:51.064 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:08:51.064 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:51.064 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:51.064 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:51.065 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:08:51.065 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:08:51.065 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:08:51.065 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:08:51.065 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:08:51.066 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:51.066 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:51.066 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:08:51.066 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:08:51.066 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:08:51.067 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:08:51.067 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:51.067 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:08:51.067 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:08:51.067 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:08:51.068 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:51.068 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:51.068 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:51.068 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:08:51.069 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:08:51.069 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:08:51.069 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:08:51.069 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:08:51.069 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:51.070 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:51.070 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:51.070 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:08:51.070 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:08:51.070 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:51.071 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:08:51.071 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:08:51.071 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:08:51.071 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:51.071 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:08:51.072 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:08:51.072 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:51.072 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:08:51.072 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:08:51.073 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 2 2021-05-25T12:08:51.073 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:51.073 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:51.073 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:51.073 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:08:51.074 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:08:51.076 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.978+0000 7fb8bede0700 10 librados: watch_flush enter 2021-05-25T12:08:51.076 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.978+0000 7fb8bede0700 10 librados: watch_flush exit 2021-05-25T12:08:51.076 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.978+0000 7fb8bede0700 20 client.264102.objecter shutdown clearing up homeless session... 2021-05-25T12:08:51.077 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.978+0000 7fb8bede0700 10 client.264102.objecter successfully canceled tick 2021-05-25T12:08:51.077 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.978+0000 7fb8bede0700 1 -- 172.21.15.102:0/1272998958 >> v1:172.21.15.181:6812/29582 conn(0x7fb8980334a0 legacy=0x7fb898035960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:51.077 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.978+0000 7fb8bede0700 10 monclient: shutdown 2021-05-25T12:08:51.077 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.978+0000 7fb8bede0700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:08:51.078 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.978+0000 7fb8bede0700 1 -- 172.21.15.102:0/1272998958 >> v1:172.21.15.102:6789/0 conn(0x7fb8b8090990 legacy=0x7fb8b8110f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:51.078 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.978+0000 7fb8bd37d700 1 -- 172.21.15.102:0/1272998958 reap_dead start 2021-05-25T12:08:51.078 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.978+0000 7fb8bd37d700 1 -- 172.21.15.102:0/1272998958 reap_dead start 2021-05-25T12:08:51.078 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.978+0000 7fb8bede0700 1 -- 172.21.15.102:0/1272998958 shutdown_connections 2021-05-25T12:08:51.078 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.978+0000 7fb8bede0700 1 -- 172.21.15.102:0/1272998958 >> 172.21.15.102:0/1272998958 conn(0x7fb8b809b370 msgr2=0x7fb8b8130720 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:08:51.079 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.978+0000 7fb8bede0700 1 -- 172.21.15.102:0/1272998958 shutdown_connections 2021-05-25T12:08:51.079 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.978+0000 7fb8bede0700 1 -- 172.21.15.102:0/1272998958 wait complete. 2021-05-25T12:08:51.079 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:50.979+0000 7fb8bede0700 1 librados: shutdown 2021-05-25T12:08:51.080 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2021-05-25T12:08:51.189 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.187+0000 7f35c5ac3700 10 monclient: get_monmap_and_config 2021-05-25T12:08:51.190 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.187+0000 7f35c5ac3700 10 monclient: build_initial_monmap 2021-05-25T12:08:51.190 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.187+0000 7f35c5ac3700 10 monclient: monmap: 2021-05-25T12:08:51.190 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:08:51.191 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:51.191 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:08:51.189597+0000 2021-05-25T12:08:51.191 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:08:51.189597+0000 2021-05-25T12:08:51.191 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:08:51.191 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:51.192 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:08:51.192 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:08:51.192 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:08:51.192 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:51.205 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.188+0000 7f35c5ac3700 1 Processor -- start 2021-05-25T12:08:51.206 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.188+0000 7f35c5ac3700 1 -- start start 2021-05-25T12:08:51.206 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.188+0000 7f35c5ac3700 10 monclient: init 2021-05-25T12:08:51.206 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.188+0000 7f35c5ac3700 10 monclient: _reopen_session rank -1 2021-05-25T12:08:51.206 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.188+0000 7f35c5ac3700 10 monclient: _add_conns ranks=[2,1,0] 2021-05-25T12:08:51.207 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.188+0000 7f35c5ac3700 10 monclient(hunting): picked mon.noname-b con 0x7f35c0069190 addr v1:172.21.15.102:6790/0 2021-05-25T12:08:51.207 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.188+0000 7f35c5ac3700 10 monclient(hunting): picked mon.noname-c con 0x7f35c0069570 addr v1:172.21.15.181:6789/0 2021-05-25T12:08:51.207 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.188+0000 7f35c5ac3700 10 monclient(hunting): picked mon.noname-a con 0x7f35c0131eb0 addr v1:172.21.15.102:6789/0 2021-05-25T12:08:51.207 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.188+0000 7f35c5ac3700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35c0096ed0 con 0x7f35c0131eb0 2021-05-25T12:08:51.208 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.188+0000 7f35c5ac3700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35c0095be0 con 0x7f35c0069570 2021-05-25T12:08:51.208 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.188+0000 7f35c5ac3700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35c0095e80 con 0x7f35c0069190 2021-05-25T12:08:51.208 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.188+0000 7f35c5ac3700 10 monclient(hunting): _renew_subs 2021-05-25T12:08:51.208 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.188+0000 7f35c5ac3700 10 monclient(hunting): authenticate will time out at 1331.786896s 2021-05-25T12:08:51.208 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.189+0000 7f35bf7fe700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f35c0069190 0x7f35c00967c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:57508/0 (socket says 172.21.15.102:57508) 2021-05-25T12:08:51.209 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.189+0000 7f35bf7fe700 1 -- 172.21.15.102:0/2063078063 learned_addr learned my addr 172.21.15.102:0/2063078063 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:08:51.209 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.189+0000 7f35bffff700 1 --1- 172.21.15.102:0/2063078063 >> v1:172.21.15.102:6789/0 conn(0x7f35c0131eb0 0x7f35c01342a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:34402/0 (socket says 172.21.15.102:34402) 2021-05-25T12:08:51.209 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.189+0000 7f35be7fc700 1 -- 172.21.15.102:0/2063078063 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1456801399 0 0) 0x7f35c0096ed0 con 0x7f35c0131eb0 2021-05-25T12:08:51.209 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.189+0000 7f35be7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:51.210 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.189+0000 7f35be7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:51.210 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.189+0000 7f35be7fc700 10 monclient(hunting): my global_id is 264111 2021-05-25T12:08:51.210 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.189+0000 7f35be7fc700 1 -- 172.21.15.102:0/2063078063 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35a4003270 con 0x7f35c0131eb0 2021-05-25T12:08:51.210 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.189+0000 7f35be7fc700 1 -- 172.21.15.102:0/2063078063 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1525870480 0 0) 0x7f35c0095e80 con 0x7f35c0069190 2021-05-25T12:08:51.210 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.189+0000 7f35be7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:51.211 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.189+0000 7f35be7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:51.211 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.189+0000 7f35be7fc700 10 monclient(hunting): my global_id is 244220 2021-05-25T12:08:51.211 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.189+0000 7f35be7fc700 1 -- 172.21.15.102:0/2063078063 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35c0096ed0 con 0x7f35c0069190 2021-05-25T12:08:51.211 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.189+0000 7f35be7fc700 1 -- 172.21.15.102:0/2063078063 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3733737699 0 0) 0x7f35a4003270 con 0x7f35c0131eb0 2021-05-25T12:08:51.212 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.189+0000 7f35be7fc700 1 -- 172.21.15.102:0/2063078063 >> v1:172.21.15.102:6790/0 conn(0x7f35c0069190 legacy=0x7f35c00967c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:51.212 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.189+0000 7f35be7fc700 1 -- 172.21.15.102:0/2063078063 >> v1:172.21.15.181:6789/0 conn(0x7f35c0069570 legacy=0x7f35c00699f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:51.212 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.189+0000 7f35be7fc700 10 monclient: _finish_hunting 0 2021-05-25T12:08:51.212 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.189+0000 7f35be7fc700 1 monclient: found mon.noname-a 2021-05-25T12:08:51.212 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.189+0000 7f35be7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:08:51.213 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.189+0000 7f35be7fc700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T12:08:51.213 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.189+0000 7f35be7fc700 1 -- 172.21.15.102:0/2063078063 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35c0096120 con 0x7f35c0131eb0 2021-05-25T12:08:51.213 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.189+0000 7f35be7fc700 10 monclient: _finish_auth 0 2021-05-25T12:08:51.213 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.189+0000 7f35be7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:08:51.214 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.189+0000 7f35be7fc700 1 -- 172.21.15.102:0/2063078063 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f35b4001990 con 0x7f35c0131eb0 2021-05-25T12:08:51.214 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.189+0000 7f35c5ac3700 5 monclient: authenticate success, global_id 264111 2021-05-25T12:08:51.214 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.189+0000 7f35be7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:51.214 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.189+0000 7f35be7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:08:51.214 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.189+0000 7f35be7fc700 10 monclient: dump: 2021-05-25T12:08:51.215 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:51.215 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:51.215 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:51.215 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:51.216 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:51.216 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:51.216 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:51.216 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:51.216 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:51.217 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:51.217 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.189+0000 7f35c5ac3700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:08:51.217 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.189+0000 7f35be7fc700 1 -- 172.21.15.102:0/2063078063 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f35b4001b80 con 0x7f35c0131eb0 2021-05-25T12:08:51.217 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.189+0000 7f35be7fc700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:08:51.218 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.189+0000 7f35c5ac3700 10 monclient: get_monmap_and_config success 2021-05-25T12:08:51.218 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.189+0000 7f35c5ac3700 4 set_mon_vals no callback set 2021-05-25T12:08:51.218 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.189+0000 7f35be7fc700 1 -- 172.21.15.102:0/2063078063 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f35b4001f10 con 0x7f35c0131eb0 2021-05-25T12:08:51.218 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.189+0000 7f35be7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:51.219 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.189+0000 7f35be7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:08:51.219 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.189+0000 7f35be7fc700 10 monclient: dump: 2021-05-25T12:08:51.219 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:51.219 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:51.219 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:51.220 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:51.220 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:51.220 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:51.220 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:51.221 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:51.221 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:51.221 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:51.221 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.190+0000 7f35c5ac3700 10 monclient: shutdown 2021-05-25T12:08:51.221 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.190+0000 7f35c5ac3700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:08:51.222 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.190+0000 7f35c5ac3700 1 -- 172.21.15.102:0/2063078063 >> v1:172.21.15.102:6789/0 conn(0x7f35c0131eb0 legacy=0x7f35c01342a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:51.222 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.190+0000 7f35c5ac3700 1 -- 172.21.15.102:0/2063078063 shutdown_connections 2021-05-25T12:08:51.222 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.190+0000 7f35c5ac3700 1 -- 172.21.15.102:0/2063078063 >> 172.21.15.102:0/2063078063 conn(0x7f35c009af90 msgr2=0x7f35c009b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:08:51.222 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.190+0000 7f35c5ac3700 1 -- 172.21.15.102:0/2063078063 shutdown_connections 2021-05-25T12:08:51.223 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.190+0000 7f35c5ac3700 1 -- 172.21.15.102:0/2063078063 wait complete. 2021-05-25T12:08:51.223 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.190+0000 7f35c5ac3700 10 monclient: build_initial_monmap 2021-05-25T12:08:51.223 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.190+0000 7f35c5ac3700 10 monclient: monmap: 2021-05-25T12:08:51.223 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:08:51.223 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:08:51.224 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:08:51.224 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:08:51.224 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:08:51.224 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:08:51.225 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:08:51.225 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:08:51.225 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:08:51.225 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:51.225 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.190+0000 7f35c5ac3700 1 librados: starting msgr at 2021-05-25T12:08:51.226 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.190+0000 7f35c5ac3700 1 librados: starting objecter 2021-05-25T12:08:51.226 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.191+0000 7f35c5ac3700 1 Processor -- start 2021-05-25T12:08:51.226 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.191+0000 7f35c5ac3700 1 -- start start 2021-05-25T12:08:51.226 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.191+0000 7f35c5ac3700 1 librados: setting wanted keys 2021-05-25T12:08:51.227 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.191+0000 7f35c5ac3700 1 librados: calling monclient init 2021-05-25T12:08:51.227 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.191+0000 7f35c5ac3700 10 monclient: init 2021-05-25T12:08:51.227 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.191+0000 7f35c5ac3700 10 monclient: _reopen_session rank -1 2021-05-25T12:08:51.227 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.191+0000 7f35c5ac3700 10 monclient: _add_conns ranks=[2,0,1] 2021-05-25T12:08:51.227 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.191+0000 7f35c5ac3700 10 monclient(hunting): picked mon.noname-c con 0x7f35c0069190 addr v1:172.21.15.102:6790/0 2021-05-25T12:08:51.228 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.191+0000 7f35c5ac3700 10 monclient(hunting): picked mon.noname-a con 0x7f35c0069570 addr v1:172.21.15.102:6789/0 2021-05-25T12:08:51.228 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.191+0000 7f35c5ac3700 10 monclient(hunting): picked mon.noname-b con 0x7f35c0131eb0 addr v1:172.21.15.181:6789/0 2021-05-25T12:08:51.228 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.191+0000 7f35c5ac3700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35c00952d0 con 0x7f35c0069570 2021-05-25T12:08:51.228 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.191+0000 7f35c5ac3700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35c01314e0 con 0x7f35c0131eb0 2021-05-25T12:08:51.229 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.191+0000 7f35c5ac3700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35c00970b0 con 0x7f35c0069190 2021-05-25T12:08:51.229 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.191+0000 7f35c5ac3700 10 monclient(hunting): _renew_subs 2021-05-25T12:08:51.229 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.191+0000 7f35c5ac3700 10 monclient(hunting): authenticate will time out at 1331.789543s 2021-05-25T12:08:51.229 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.191+0000 7f35bf7fe700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f35c0069190 0x7f35c01150f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:57514/0 (socket says 172.21.15.102:57514) 2021-05-25T12:08:51.229 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.191+0000 7f35bf7fe700 1 -- 172.21.15.102:0/4188810391 learned_addr learned my addr 172.21.15.102:0/4188810391 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:08:51.230 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.191+0000 7f35ab7fe700 10 client.?.objecter ms_handle_connect 0x7f35c0069190 2021-05-25T12:08:51.230 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.191+0000 7f35ab7fe700 10 client.?.objecter resend_mon_ops 2021-05-25T12:08:51.230 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.191+0000 7f35ab7fe700 10 client.?.objecter ms_handle_connect 0x7f35c0069570 2021-05-25T12:08:51.230 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.191+0000 7f35ab7fe700 10 client.?.objecter resend_mon_ops 2021-05-25T12:08:51.231 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.191+0000 7f35ab7fe700 1 -- 172.21.15.102:0/4188810391 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3714978089 0 0) 0x7f35c00970b0 con 0x7f35c0069190 2021-05-25T12:08:51.231 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.191+0000 7f35ab7fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:51.231 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.191+0000 7f35ab7fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:51.231 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.191+0000 7f35ab7fe700 10 monclient(hunting): my global_id is 244223 2021-05-25T12:08:51.231 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.191+0000 7f35ab7fe700 1 -- 172.21.15.102:0/4188810391 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35a0004c10 con 0x7f35c0069190 2021-05-25T12:08:51.232 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.191+0000 7f35ab7fe700 1 -- 172.21.15.102:0/4188810391 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1071628351 0 0) 0x7f35c00952d0 con 0x7f35c0069570 2021-05-25T12:08:51.232 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.191+0000 7f35ab7fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:51.232 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.191+0000 7f35ab7fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:51.232 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.191+0000 7f35ab7fe700 10 monclient(hunting): my global_id is 264114 2021-05-25T12:08:51.232 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.191+0000 7f35ab7fe700 1 -- 172.21.15.102:0/4188810391 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35c00970b0 con 0x7f35c0069570 2021-05-25T12:08:51.233 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.191+0000 7f35ab7fe700 10 client.?.objecter ms_handle_connect 0x7f35c0131eb0 2021-05-25T12:08:51.233 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.191+0000 7f35ab7fe700 10 client.?.objecter resend_mon_ops 2021-05-25T12:08:51.233 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.192+0000 7f35ab7fe700 1 -- 172.21.15.102:0/4188810391 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 707760470 0 0) 0x7f35a0004c10 con 0x7f35c0069190 2021-05-25T12:08:51.233 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.192+0000 7f35ab7fe700 1 -- 172.21.15.102:0/4188810391 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f35c00952d0 con 0x7f35c0069190 2021-05-25T12:08:51.234 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.192+0000 7f35ab7fe700 1 -- 172.21.15.102:0/4188810391 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f35ac001a30 con 0x7f35c0069190 2021-05-25T12:08:51.234 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.192+0000 7f35ab7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:51.234 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.192+0000 7f35ab7fe700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:08:51.234 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.192+0000 7f35ab7fe700 10 monclient(hunting): dump: 2021-05-25T12:08:51.234 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:51.235 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:51.235 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:51.235 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:51.235 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:51.236 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:51.236 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:51.236 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:51.240 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:51.241 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:51.241 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.192+0000 7f35ab7fe700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:08:51.241 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.192+0000 7f35ab7fe700 1 -- 172.21.15.102:0/4188810391 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3730881974 0 0) 0x7f35c00970b0 con 0x7f35c0069570 2021-05-25T12:08:51.241 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.192+0000 7f35ab7fe700 1 -- 172.21.15.102:0/4188810391 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f35a0004c10 con 0x7f35c0069570 2021-05-25T12:08:51.242 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.192+0000 7f35ab7fe700 1 -- 172.21.15.102:0/4188810391 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1222130808 0 0) 0x7f35c01314e0 con 0x7f35c0131eb0 2021-05-25T12:08:51.242 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.192+0000 7f35ab7fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:51.242 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.192+0000 7f35ab7fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:51.242 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.192+0000 7f35ab7fe700 10 monclient(hunting): my global_id is 224485 2021-05-25T12:08:51.242 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.192+0000 7f35ab7fe700 1 -- 172.21.15.102:0/4188810391 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35c00970b0 con 0x7f35c0131eb0 2021-05-25T12:08:51.243 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.192+0000 7f35ab7fe700 1 -- 172.21.15.102:0/4188810391 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f35b0001770 con 0x7f35c0069570 2021-05-25T12:08:51.243 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.192+0000 7f35ab7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:51.243 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.192+0000 7f35ab7fe700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2021-05-25T12:08:51.243 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.192+0000 7f35ab7fe700 10 monclient(hunting): dump: 2021-05-25T12:08:51.244 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:51.244 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:51.244 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:51.244 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:51.244 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:51.245 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:51.245 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:51.245 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:51.245 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:51.246 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:51.246 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.192+0000 7f35ab7fe700 1 -- 172.21.15.102:0/4188810391 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3428536318 0 0) 0x7f35c00952d0 con 0x7f35c0069190 2021-05-25T12:08:51.246 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.192+0000 7f35ab7fe700 1 -- 172.21.15.102:0/4188810391 >> v1:172.21.15.181:6789/0 conn(0x7f35c0131eb0 legacy=0x7f35c01432f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:51.246 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.192+0000 7f35ab7fe700 1 -- 172.21.15.102:0/4188810391 >> v1:172.21.15.102:6789/0 conn(0x7f35c0069570 legacy=0x7f35c0130dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:51.246 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.192+0000 7f35ab7fe700 10 monclient: _finish_hunting 0 2021-05-25T12:08:51.247 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.192+0000 7f35ab7fe700 1 monclient: found mon.c 2021-05-25T12:08:51.247 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.192+0000 7f35ab7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:08:51.247 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.192+0000 7f35ab7fe700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:08:51.247 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.192+0000 7f35ab7fe700 1 -- 172.21.15.102:0/4188810391 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35c01187b0 con 0x7f35c0069190 2021-05-25T12:08:51.248 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.192+0000 7f35ab7fe700 10 monclient: _finish_auth 0 2021-05-25T12:08:51.248 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.192+0000 7f35ab7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:08:51.248 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.192+0000 7f35c5ac3700 5 monclient: authenticate success, global_id 244223 2021-05-25T12:08:51.248 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.192+0000 7f35c5ac3700 10 monclient: _renew_subs 2021-05-25T12:08:51.248 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.192+0000 7f35c5ac3700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:08:51.249 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.192+0000 7f35c5ac3700 1 -- 172.21.15.102:0/4188810391 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35c0117270 con 0x7f35c0069190 2021-05-25T12:08:51.249 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.192+0000 7f35c5ac3700 10 client.244223.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:08:51.249 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.192+0000 7f35c5ac3700 10 monclient: _renew_subs 2021-05-25T12:08:51.249 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.192+0000 7f35c5ac3700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:08:51.249 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.192+0000 7f35c5ac3700 1 -- 172.21.15.102:0/4188810391 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f35c0115e00 con 0x7f35c0069190 2021-05-25T12:08:51.250 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.192+0000 7f35c5ac3700 1 librados: init done 2021-05-25T12:08:51.250 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.192+0000 7f35ab7fe700 1 -- 172.21.15.102:0/4188810391 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f35ac001c60 con 0x7f35c0069190 2021-05-25T12:08:51.250 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.192+0000 7f35ab7fe700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:08:51.250 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.192+0000 7f35ab7fe700 1 -- 172.21.15.102:0/4188810391 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f35ac004980 con 0x7f35c0069190 2021-05-25T12:08:51.251 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.192+0000 7f35ab7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:51.251 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.192+0000 7f35bdffb700 4 set_mon_vals no callback set 2021-05-25T12:08:51.251 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.192+0000 7f35ab7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:08:51.251 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.192+0000 7f35ab7fe700 10 monclient: dump: 2021-05-25T12:08:51.251 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:51.252 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:51.252 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:51.252 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:51.252 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:51.252 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:51.253 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:51.253 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:51.253 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:51.253 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:51.253 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.193+0000 7f35ab7fe700 1 -- 172.21.15.102:0/4188810391 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f35ac00dca0 con 0x7f35c0069190 2021-05-25T12:08:51.254 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.193+0000 7f35c5ac3700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:08:51.254 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.193+0000 7f35c5ac3700 1 -- 172.21.15.102:0/4188810391 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f35980065a0 con 0x7f3598002840 2021-05-25T12:08:51.254 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.193+0000 7f35ab7fe700 1 -- 172.21.15.102:0/4188810391 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(293..293 src has 250..293) v4 ==== 6275+0+0 (unknown 4031507246 0 0) 0x7f35ac03d890 con 0x7f35c0069190 2021-05-25T12:08:51.254 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.193+0000 7f35ab7fe700 10 client.244223.objecter ms_dispatch 0x7f35c0001550 osd_map(293..293 src has 250..293) v4 2021-05-25T12:08:51.255 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.193+0000 7f35ab7fe700 3 client.244223.objecter handle_osd_map got epochs [293,293] > 0 2021-05-25T12:08:51.255 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.193+0000 7f35ab7fe700 3 client.244223.objecter handle_osd_map decoding full epoch 293 2021-05-25T12:08:51.255 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.193+0000 7f35ab7fe700 20 client.244223.objecter dump_active .. 0 homeless 2021-05-25T12:08:51.255 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.193+0000 7f35ab7fe700 10 client.244223.objecter ms_handle_connect 0x7f3598002840 2021-05-25T12:08:51.255 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.193+0000 7f35ab7fe700 10 client.244223.objecter resend_mon_ops 2021-05-25T12:08:51.256 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.193+0000 7f35ab7fe700 1 -- 172.21.15.102:0/4188810391 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2644577464 0 0) 0x7f35980065a0 con 0x7f3598002840 2021-05-25T12:08:51.256 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.193+0000 7f35ab7fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:51.256 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.193+0000 7f35ab7fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:51.256 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.193+0000 7f35ab7fe700 10 monclient(hunting): my global_id is 244226 2021-05-25T12:08:51.257 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.193+0000 7f35ab7fe700 1 -- 172.21.15.102:0/4188810391 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35c00952d0 con 0x7f3598002840 2021-05-25T12:08:51.257 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.193+0000 7f35bffff700 10 monclient: get_auth_request con 0x7f35a0033400 auth_method 0 2021-05-25T12:08:51.257 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.193+0000 7f35ab7fe700 1 -- 172.21.15.102:0/4188810391 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3435598131 0 0) 0x7f35c00952d0 con 0x7f3598002840 2021-05-25T12:08:51.257 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.193+0000 7f35ab7fe700 1 -- 172.21.15.102:0/4188810391 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f35980065a0 con 0x7f3598002840 2021-05-25T12:08:51.257 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.193+0000 7f35ab7fe700 10 client.244223.objecter ms_handle_connect 0x7f35a0033400 2021-05-25T12:08:51.258 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.193+0000 7f35ab7fe700 1 -- 172.21.15.102:0/4188810391 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f35b00021b0 con 0x7f3598002840 2021-05-25T12:08:51.258 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.193+0000 7f35ab7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:51.258 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.193+0000 7f35ab7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:08:51.258 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.193+0000 7f35ab7fe700 10 monclient: dump: 2021-05-25T12:08:51.259 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:51.259 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:51.259 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:51.259 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:51.259 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:51.260 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:51.260 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:51.260 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:51.260 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:51.260 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:51.261 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.193+0000 7f35ab7fe700 1 -- 172.21.15.102:0/4188810391 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3295103125 0 0) 0x7f35980065a0 con 0x7f3598002840 2021-05-25T12:08:51.261 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.193+0000 7f35ab7fe700 1 -- 172.21.15.102:0/4188810391 --> v1:172.21.15.102:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3598009e40 con 0x7f3598002840 2021-05-25T12:08:51.261 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.194+0000 7f35ab7fe700 1 -- 172.21.15.102:0/4188810391 <== mon.2 v1:172.21.15.102:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7f35b0004500 con 0x7f3598002840 2021-05-25T12:08:51.261 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.194+0000 7f35ab7fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:08:51.261 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.194+0000 7f35ab7fe700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:08:51.262 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.194+0000 7f35ab7fe700 1 -- 172.21.15.102:0/4188810391 >> v1:172.21.15.102:6790/0 conn(0x7f3598002840 legacy=0x7f3598004ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:51.262 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.194+0000 7f35ab7fe700 1 -- 172.21.15.102:0/4188810391 >> v1:172.21.15.102:6790/0 conn(0x7f3598002840 legacy=0x7f3598004ce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:08:51.262 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.200+0000 7f35c5ac3700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:08:51.262 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.200+0000 7f35c5ac3700 1 -- 172.21.15.102:0/4188810391 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f3598009ae0 con 0x7f3598008da0 2021-05-25T12:08:51.263 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.200+0000 7f35ab7fe700 10 client.244223.objecter ms_handle_connect 0x7f3598008da0 2021-05-25T12:08:51.263 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.200+0000 7f35ab7fe700 10 client.244223.objecter resend_mon_ops 2021-05-25T12:08:51.263 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.201+0000 7f35ab7fe700 1 -- 172.21.15.102:0/4188810391 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2987545739 0 0) 0x7f3598009ae0 con 0x7f3598008da0 2021-05-25T12:08:51.263 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.201+0000 7f35ab7fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:51.263 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.201+0000 7f35ab7fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:51.264 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.201+0000 7f35ab7fe700 10 monclient(hunting): my global_id is 244229 2021-05-25T12:08:51.264 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.201+0000 7f35ab7fe700 1 -- 172.21.15.102:0/4188810391 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35980065a0 con 0x7f3598008da0 2021-05-25T12:08:51.264 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.201+0000 7f35ab7fe700 1 -- 172.21.15.102:0/4188810391 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4045322949 0 0) 0x7f35980065a0 con 0x7f3598008da0 2021-05-25T12:08:51.264 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.201+0000 7f35ab7fe700 1 -- 172.21.15.102:0/4188810391 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3598009ae0 con 0x7f3598008da0 2021-05-25T12:08:51.264 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.201+0000 7f35ab7fe700 1 -- 172.21.15.102:0/4188810391 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f35b00029e0 con 0x7f3598008da0 2021-05-25T12:08:51.265 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.201+0000 7f35ab7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:51.265 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.201+0000 7f35ab7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:08:51.265 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.201+0000 7f35ab7fe700 10 monclient: dump: 2021-05-25T12:08:51.265 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:51.266 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:51.266 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:51.266 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:51.266 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:51.267 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:51.267 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:51.267 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:51.267 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:51.267 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:51.268 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.201+0000 7f35ab7fe700 1 -- 172.21.15.102:0/4188810391 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 116767049 0 0) 0x7f3598009ae0 con 0x7f3598008da0 2021-05-25T12:08:51.268 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.201+0000 7f35ab7fe700 1 -- 172.21.15.102:0/4188810391 --> v1:172.21.15.102:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f3598008740 con 0x7f3598008da0 2021-05-25T12:08:51.268 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.201+0000 7f35ab7fe700 1 -- 172.21.15.102:0/4188810391 <== mon.2 v1:172.21.15.102:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3940 (unknown 0 0 4085073503) 0x7f35b0003e40 con 0x7f3598008da0 2021-05-25T12:08:51.268 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.201+0000 7f35ab7fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:08:51.268 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.201+0000 7f35ab7fe700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:08:51.269 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.201+0000 7f35ab7fe700 1 -- 172.21.15.102:0/4188810391 >> v1:172.21.15.102:6790/0 conn(0x7f3598008da0 legacy=0x7f3598009230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:51.269 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.201+0000 7f35ab7fe700 1 -- 172.21.15.102:0/4188810391 >> v1:172.21.15.102:6790/0 conn(0x7f3598008da0 legacy=0x7f3598009230 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:08:51.269 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:08:51.270 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:08:51.270 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:08:51.270 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "peon", 2021-05-25T12:08:51.270 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 74, 2021-05-25T12:08:51.271 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:08:51.271 INFO:teuthology.orchestra.run.smithi102.stdout: 0, 2021-05-25T12:08:51.271 INFO:teuthology.orchestra.run.smithi102.stdout: 1, 2021-05-25T12:08:51.271 INFO:teuthology.orchestra.run.smithi102.stdout: 2 2021-05-25T12:08:51.272 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:08:51.272 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 0, 2021-05-25T12:08:51.272 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:08:51.272 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:08:51.272 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:08:51.273 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:08:51.273 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:08:51.273 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:08:51.273 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:08:51.273 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:08:51.274 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:08:51.274 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:08:51.274 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:08:51.274 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:08:51.275 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:08:51.275 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:08:51.275 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:08:51.275 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:08:51.275 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:08:51.276 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:08:51.276 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:08:51.276 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:08:51.276 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:08:51.277 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:08:51.277 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:08:51.277 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:08:51.277 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:51.277 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:51.278 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:08:51.278 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:08:51.278 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:08:51.278 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:08:51.278 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:08:51.279 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:08:51.279 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:08:51.279 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:08:51.279 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:08:51.280 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:08:51.280 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:08:51.280 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:08:51.280 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:08:51.280 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:08:51.281 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:08:51.281 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:08:51.281 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:08:51.281 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:08:51.281 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:08:51.282 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:08:51.282 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:08:51.282 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:08:51.282 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:08:51.282 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:08:51.283 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:08:51.283 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:08:51.283 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:51.283 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:08:51.283 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:51.284 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:08:51.284 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:08:51.284 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:08:51.284 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:08:51.285 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:51.285 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:08:51.285 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:08:51.285 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:08:51.285 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:51.286 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:51.286 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:51.286 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:08:51.286 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:08:51.286 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:08:51.287 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:08:51.287 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:08:51.287 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:51.287 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:51.287 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:08:51.288 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:08:51.288 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:08:51.288 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:08:51.288 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:51.288 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:08:51.289 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:08:51.289 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:08:51.289 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:51.289 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:51.290 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:51.290 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:08:51.290 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:08:51.290 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:08:51.290 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:08:51.291 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:08:51.291 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:51.291 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:51.291 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:08:51.292 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:08:51.292 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:08:51.292 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:08:51.292 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:51.292 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:08:51.293 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:08:51.293 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:08:51.293 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:51.293 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:51.293 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:51.294 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:08:51.294 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:08:51.294 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:08:51.294 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:08:51.295 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:08:51.295 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:51.295 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:51.295 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:51.295 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:08:51.296 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:08:51.296 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:51.296 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:08:51.296 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:08:51.296 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:08:51.297 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:51.297 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:08:51.297 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": [ 2021-05-25T12:08:51.297 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:51.297 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:08:51.298 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:08:51.298 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 2 2021-05-25T12:08:51.298 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:51.298 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:08:51.299 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:08:51.299 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:51.299 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:08:51.299 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:08:51.299 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 2 2021-05-25T12:08:51.300 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:51.300 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:51.300 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:51.300 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:08:51.300 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:08:51.303 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.201+0000 7f35c5ac3700 10 librados: watch_flush enter 2021-05-25T12:08:51.303 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.201+0000 7f35c5ac3700 10 librados: watch_flush exit 2021-05-25T12:08:51.303 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.201+0000 7f35c5ac3700 20 client.244223.objecter shutdown clearing up homeless session... 2021-05-25T12:08:51.304 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.201+0000 7f35c5ac3700 10 client.244223.objecter successfully canceled tick 2021-05-25T12:08:51.304 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.201+0000 7f35c5ac3700 1 -- 172.21.15.102:0/4188810391 >> v1:172.21.15.181:6812/29582 conn(0x7f35a0033400 legacy=0x7f35a00358c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:51.304 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.201+0000 7f35c5ac3700 10 monclient: shutdown 2021-05-25T12:08:51.304 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.201+0000 7f35c5ac3700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:08:51.305 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.201+0000 7f35c5ac3700 1 -- 172.21.15.102:0/4188810391 >> v1:172.21.15.102:6790/0 conn(0x7f35c0069190 legacy=0x7f35c01150f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:51.305 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.201+0000 7f35bffff700 1 -- 172.21.15.102:0/4188810391 reap_dead start 2021-05-25T12:08:51.305 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.202+0000 7f35bffff700 1 -- 172.21.15.102:0/4188810391 reap_dead start 2021-05-25T12:08:51.305 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.202+0000 7f35c5ac3700 1 -- 172.21.15.102:0/4188810391 shutdown_connections 2021-05-25T12:08:51.305 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.202+0000 7f35c5ac3700 1 -- 172.21.15.102:0/4188810391 >> 172.21.15.102:0/4188810391 conn(0x7f35c009af90 msgr2=0x7f35c00946a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:08:51.306 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.202+0000 7f35c5ac3700 1 -- 172.21.15.102:0/4188810391 shutdown_connections 2021-05-25T12:08:51.306 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.202+0000 7f35c5ac3700 1 -- 172.21.15.102:0/4188810391 wait complete. 2021-05-25T12:08:51.306 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.202+0000 7f35c5ac3700 1 librados: shutdown 2021-05-25T12:08:51.307 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2021-05-25T12:08:51.418 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.415+0000 7fdee4e26700 10 monclient: get_monmap_and_config 2021-05-25T12:08:51.418 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.415+0000 7fdee4e26700 10 monclient: build_initial_monmap 2021-05-25T12:08:51.418 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.415+0000 7fdee4e26700 10 monclient: monmap: 2021-05-25T12:08:51.418 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:08:51.419 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:51.419 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:08:51.417628+0000 2021-05-25T12:08:51.419 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:08:51.417628+0000 2021-05-25T12:08:51.419 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:08:51.420 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:51.420 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:08:51.420 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:08:51.420 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:08:51.420 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:51.434 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.416+0000 7fdee4e26700 1 Processor -- start 2021-05-25T12:08:51.434 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.416+0000 7fdee4e26700 1 -- start start 2021-05-25T12:08:51.435 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.416+0000 7fdee4e26700 10 monclient: init 2021-05-25T12:08:51.435 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.416+0000 7fdee4e26700 10 monclient: _reopen_session rank -1 2021-05-25T12:08:51.435 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.416+0000 7fdee4e26700 10 monclient: _add_conns ranks=[2,1,0] 2021-05-25T12:08:51.435 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.416+0000 7fdee4e26700 10 monclient(hunting): picked mon.noname-b con 0x7fdee0096240 addr v1:172.21.15.102:6790/0 2021-05-25T12:08:51.436 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.416+0000 7fdee4e26700 10 monclient(hunting): picked mon.noname-c con 0x7fdee0094330 addr v1:172.21.15.181:6789/0 2021-05-25T12:08:51.436 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.416+0000 7fdee4e26700 10 monclient(hunting): picked mon.noname-a con 0x7fdee00947e0 addr v1:172.21.15.102:6789/0 2021-05-25T12:08:51.436 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.416+0000 7fdee4e26700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdee00970d0 con 0x7fdee00947e0 2021-05-25T12:08:51.436 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.416+0000 7fdee4e26700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdee0090bc0 con 0x7fdee0094330 2021-05-25T12:08:51.436 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.416+0000 7fdee4e26700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdee0090e60 con 0x7fdee0096240 2021-05-25T12:08:51.437 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.416+0000 7fdee4e26700 10 monclient(hunting): _renew_subs 2021-05-25T12:08:51.437 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.416+0000 7fdee4e26700 10 monclient(hunting): authenticate will time out at 1332.014993s 2021-05-25T12:08:51.437 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.417+0000 7fdedf7fe700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fdee0096240 0x7fdee0128a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:57526/0 (socket says 172.21.15.102:57526) 2021-05-25T12:08:51.437 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.417+0000 7fdedf7fe700 1 -- 172.21.15.102:0/2005391783 learned_addr learned my addr 172.21.15.102:0/2005391783 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:08:51.437 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.417+0000 7fdedffff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fdee00947e0 0x7fdee01344d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:34416/0 (socket says 172.21.15.102:34416) 2021-05-25T12:08:51.438 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.417+0000 7fdede7fc700 1 -- 172.21.15.102:0/2005391783 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2608009055 0 0) 0x7fdee00970d0 con 0x7fdee00947e0 2021-05-25T12:08:51.438 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.417+0000 7fdede7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:51.438 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.417+0000 7fdede7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:51.438 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.417+0000 7fdede7fc700 10 monclient(hunting): my global_id is 264117 2021-05-25T12:08:51.439 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.417+0000 7fdede7fc700 1 -- 172.21.15.102:0/2005391783 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdecc003270 con 0x7fdee00947e0 2021-05-25T12:08:51.439 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.417+0000 7fdede7fc700 1 -- 172.21.15.102:0/2005391783 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 159074555 0 0) 0x7fdee0090e60 con 0x7fdee0096240 2021-05-25T12:08:51.439 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.417+0000 7fdede7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:51.439 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.417+0000 7fdede7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:51.440 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.417+0000 7fdede7fc700 10 monclient(hunting): my global_id is 244232 2021-05-25T12:08:51.440 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.417+0000 7fdede7fc700 1 -- 172.21.15.102:0/2005391783 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdee00970d0 con 0x7fdee0096240 2021-05-25T12:08:51.440 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.417+0000 7fdede7fc700 1 -- 172.21.15.102:0/2005391783 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 674344443 0 0) 0x7fdee0090bc0 con 0x7fdee0094330 2021-05-25T12:08:51.440 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.417+0000 7fdede7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:51.440 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.417+0000 7fdede7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:51.441 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.417+0000 7fdede7fc700 10 monclient(hunting): my global_id is 224488 2021-05-25T12:08:51.441 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.417+0000 7fdede7fc700 1 -- 172.21.15.102:0/2005391783 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdee0090e60 con 0x7fdee0094330 2021-05-25T12:08:51.441 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.417+0000 7fdede7fc700 1 -- 172.21.15.102:0/2005391783 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2481460613 0 0) 0x7fdecc003270 con 0x7fdee00947e0 2021-05-25T12:08:51.441 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.417+0000 7fdede7fc700 1 -- 172.21.15.102:0/2005391783 >> v1:172.21.15.102:6790/0 conn(0x7fdee0096240 legacy=0x7fdee0128a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:51.441 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.417+0000 7fdede7fc700 1 -- 172.21.15.102:0/2005391783 >> v1:172.21.15.181:6789/0 conn(0x7fdee0094330 legacy=0x7fdee0130da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:51.442 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.417+0000 7fdede7fc700 10 monclient: _finish_hunting 0 2021-05-25T12:08:51.442 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.417+0000 7fdede7fc700 1 monclient: found mon.noname-a 2021-05-25T12:08:51.442 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.417+0000 7fdede7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:08:51.442 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.417+0000 7fdede7fc700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T12:08:51.443 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.417+0000 7fdede7fc700 1 -- 172.21.15.102:0/2005391783 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdee0069710 con 0x7fdee00947e0 2021-05-25T12:08:51.443 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.417+0000 7fdede7fc700 10 monclient: _finish_auth 0 2021-05-25T12:08:51.443 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.417+0000 7fdede7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:08:51.443 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.417+0000 7fdede7fc700 1 -- 172.21.15.102:0/2005391783 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fded40019e0 con 0x7fdee00947e0 2021-05-25T12:08:51.443 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.417+0000 7fdee4e26700 5 monclient: authenticate success, global_id 264117 2021-05-25T12:08:51.444 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.417+0000 7fdee4e26700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:08:51.444 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.417+0000 7fdede7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:51.444 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.417+0000 7fdede7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:08:51.444 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.417+0000 7fdede7fc700 10 monclient: dump: 2021-05-25T12:08:51.444 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:51.445 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:51.445 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:51.445 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:51.445 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:51.446 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:51.446 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:51.446 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:51.446 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:51.446 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:51.447 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.417+0000 7fdee4e26700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:08:51.447 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.417+0000 7fdede7fc700 1 -- 172.21.15.102:0/2005391783 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fded4001c40 con 0x7fdee00947e0 2021-05-25T12:08:51.447 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.417+0000 7fdede7fc700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:08:51.447 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.417+0000 7fdede7fc700 1 -- 172.21.15.102:0/2005391783 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fded4001fd0 con 0x7fdee00947e0 2021-05-25T12:08:51.447 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.417+0000 7fdee4e26700 10 monclient: get_monmap_and_config success 2021-05-25T12:08:51.448 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.417+0000 7fdee4e26700 4 set_mon_vals no callback set 2021-05-25T12:08:51.448 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.417+0000 7fdede7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:51.448 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.417+0000 7fdede7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:08:51.448 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.417+0000 7fdede7fc700 10 monclient: dump: 2021-05-25T12:08:51.449 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:51.449 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:51.449 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:51.449 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:51.449 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:51.450 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:51.450 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:51.450 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:51.450 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:51.450 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:51.451 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.418+0000 7fdee4e26700 10 monclient: shutdown 2021-05-25T12:08:51.451 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.418+0000 7fdee4e26700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:08:51.451 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.418+0000 7fdee4e26700 1 -- 172.21.15.102:0/2005391783 >> v1:172.21.15.102:6789/0 conn(0x7fdee00947e0 legacy=0x7fdee01344d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:51.451 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.418+0000 7fdee4e26700 1 -- 172.21.15.102:0/2005391783 shutdown_connections 2021-05-25T12:08:51.452 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.418+0000 7fdee4e26700 1 -- 172.21.15.102:0/2005391783 >> 172.21.15.102:0/2005391783 conn(0x7fdee00974b0 msgr2=0x7fdee00978c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:08:51.452 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.418+0000 7fdee4e26700 1 -- 172.21.15.102:0/2005391783 shutdown_connections 2021-05-25T12:08:51.452 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.418+0000 7fdee4e26700 1 -- 172.21.15.102:0/2005391783 wait complete. 2021-05-25T12:08:51.452 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.418+0000 7fdee4e26700 10 monclient: build_initial_monmap 2021-05-25T12:08:51.452 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.418+0000 7fdee4e26700 10 monclient: monmap: 2021-05-25T12:08:51.453 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:08:51.453 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:08:51.453 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:08:51.453 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:08:51.454 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:08:51.454 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:08:51.454 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:08:51.454 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:08:51.454 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:08:51.455 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:51.455 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.418+0000 7fdee4e26700 1 librados: starting msgr at 2021-05-25T12:08:51.455 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.418+0000 7fdee4e26700 1 librados: starting objecter 2021-05-25T12:08:51.455 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.419+0000 7fdee4e26700 1 Processor -- start 2021-05-25T12:08:51.456 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.419+0000 7fdee4e26700 1 -- start start 2021-05-25T12:08:51.456 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.419+0000 7fdee4e26700 1 librados: setting wanted keys 2021-05-25T12:08:51.456 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.419+0000 7fdee4e26700 1 librados: calling monclient init 2021-05-25T12:08:51.456 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.419+0000 7fdee4e26700 10 monclient: init 2021-05-25T12:08:51.456 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.419+0000 7fdee4e26700 10 monclient: _reopen_session rank -1 2021-05-25T12:08:51.457 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.419+0000 7fdee4e26700 10 monclient: _add_conns ranks=[2,1,0] 2021-05-25T12:08:51.457 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.419+0000 7fdee4e26700 10 monclient(hunting): picked mon.noname-c con 0x7fdee0094330 addr v1:172.21.15.102:6790/0 2021-05-25T12:08:51.457 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.419+0000 7fdee4e26700 10 monclient(hunting): picked mon.noname-b con 0x7fdee00947e0 addr v1:172.21.15.181:6789/0 2021-05-25T12:08:51.457 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.419+0000 7fdee4e26700 10 monclient(hunting): picked mon.noname-a con 0x7fdee0096240 addr v1:172.21.15.102:6789/0 2021-05-25T12:08:51.457 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.419+0000 7fdee4e26700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdee012e030 con 0x7fdee0096240 2021-05-25T12:08:51.458 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.419+0000 7fdee4e26700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdee0130960 con 0x7fdee00947e0 2021-05-25T12:08:51.458 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.419+0000 7fdee4e26700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdee0130c00 con 0x7fdee0094330 2021-05-25T12:08:51.458 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.419+0000 7fdee4e26700 10 monclient(hunting): _renew_subs 2021-05-25T12:08:51.458 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.419+0000 7fdee4e26700 10 monclient(hunting): authenticate will time out at 1332.017733s 2021-05-25T12:08:51.459 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.419+0000 7fdedf7fe700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fdee0094330 0x7fdee012d210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:57530/0 (socket says 172.21.15.102:57530) 2021-05-25T12:08:51.459 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.419+0000 7fdedf7fe700 1 -- 172.21.15.102:0/2167516697 learned_addr learned my addr 172.21.15.102:0/2167516697 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:08:51.459 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.419+0000 7fdec77fe700 10 client.?.objecter ms_handle_connect 0x7fdee0094330 2021-05-25T12:08:51.459 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.419+0000 7fdec77fe700 10 client.?.objecter resend_mon_ops 2021-05-25T12:08:51.459 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.419+0000 7fdec77fe700 10 client.?.objecter ms_handle_connect 0x7fdee0096240 2021-05-25T12:08:51.460 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.419+0000 7fdec77fe700 10 client.?.objecter resend_mon_ops 2021-05-25T12:08:51.460 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.419+0000 7fdec77fe700 1 -- 172.21.15.102:0/2167516697 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1350167509 0 0) 0x7fdee0130c00 con 0x7fdee0094330 2021-05-25T12:08:51.460 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.420+0000 7fdec77fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:51.460 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.420+0000 7fdec77fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:51.461 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.420+0000 7fdec77fe700 10 monclient(hunting): my global_id is 244235 2021-05-25T12:08:51.461 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.420+0000 7fdec77fe700 1 -- 172.21.15.102:0/2167516697 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdec0004c10 con 0x7fdee0094330 2021-05-25T12:08:51.461 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.420+0000 7fdec77fe700 10 client.?.objecter ms_handle_connect 0x7fdee00947e0 2021-05-25T12:08:51.461 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.420+0000 7fdec77fe700 10 client.?.objecter resend_mon_ops 2021-05-25T12:08:51.461 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.420+0000 7fdec77fe700 1 -- 172.21.15.102:0/2167516697 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1550642088 0 0) 0x7fdee012e030 con 0x7fdee0096240 2021-05-25T12:08:51.462 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.420+0000 7fdec77fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:51.462 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.420+0000 7fdec77fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:51.462 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.420+0000 7fdec77fe700 10 monclient(hunting): my global_id is 264120 2021-05-25T12:08:51.462 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.420+0000 7fdec77fe700 1 -- 172.21.15.102:0/2167516697 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdee0130c00 con 0x7fdee0096240 2021-05-25T12:08:51.462 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.420+0000 7fdec77fe700 1 -- 172.21.15.102:0/2167516697 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1399418570 0 0) 0x7fdee0130960 con 0x7fdee00947e0 2021-05-25T12:08:51.463 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.420+0000 7fdec77fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:51.463 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.420+0000 7fdec77fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:51.463 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.420+0000 7fdec77fe700 10 monclient(hunting): my global_id is 224491 2021-05-25T12:08:51.463 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.420+0000 7fdec77fe700 1 -- 172.21.15.102:0/2167516697 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdee012e030 con 0x7fdee00947e0 2021-05-25T12:08:51.464 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.420+0000 7fdec77fe700 1 -- 172.21.15.102:0/2167516697 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2072433585 0 0) 0x7fdee0130c00 con 0x7fdee0096240 2021-05-25T12:08:51.464 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.420+0000 7fdec77fe700 1 -- 172.21.15.102:0/2167516697 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdee0130960 con 0x7fdee0096240 2021-05-25T12:08:51.464 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.420+0000 7fdec77fe700 1 -- 172.21.15.102:0/2167516697 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fded40018e0 con 0x7fdee0096240 2021-05-25T12:08:51.464 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.420+0000 7fdec77fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:51.464 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.420+0000 7fdec77fe700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:08:51.465 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.420+0000 7fdec77fe700 10 monclient(hunting): dump: 2021-05-25T12:08:51.465 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:51.465 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:51.465 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:51.465 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:51.466 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:51.466 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:51.466 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:51.466 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:51.467 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:51.467 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:51.467 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.420+0000 7fdec77fe700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:08:51.467 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.420+0000 7fdec77fe700 1 -- 172.21.15.102:0/2167516697 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 62273441 0 0) 0x7fdec0004c10 con 0x7fdee0094330 2021-05-25T12:08:51.467 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.420+0000 7fdec77fe700 1 -- 172.21.15.102:0/2167516697 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdee0130c00 con 0x7fdee0094330 2021-05-25T12:08:51.468 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.420+0000 7fdec77fe700 1 -- 172.21.15.102:0/2167516697 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fded00019d0 con 0x7fdee0094330 2021-05-25T12:08:51.468 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.420+0000 7fdec77fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:51.468 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.420+0000 7fdec77fe700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2021-05-25T12:08:51.468 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.420+0000 7fdec77fe700 10 monclient(hunting): dump: 2021-05-25T12:08:51.468 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:51.469 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:51.469 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:51.469 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:51.469 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:51.470 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:51.470 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:51.470 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:51.470 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:51.470 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:51.471 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.420+0000 7fdec77fe700 1 -- 172.21.15.102:0/2167516697 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3145470835 0 0) 0x7fdee0130960 con 0x7fdee0096240 2021-05-25T12:08:51.471 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.420+0000 7fdec77fe700 1 -- 172.21.15.102:0/2167516697 >> v1:172.21.15.102:6790/0 conn(0x7fdee0094330 legacy=0x7fdee012d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:51.471 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.420+0000 7fdec77fe700 1 -- 172.21.15.102:0/2167516697 >> v1:172.21.15.181:6789/0 conn(0x7fdee00947e0 legacy=0x7fdee012d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:51.471 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.420+0000 7fdec77fe700 10 monclient: _finish_hunting 0 2021-05-25T12:08:51.472 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.420+0000 7fdec77fe700 1 monclient: found mon.a 2021-05-25T12:08:51.472 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.420+0000 7fdec77fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:08:51.472 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.420+0000 7fdec77fe700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:08:51.472 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.420+0000 7fdec77fe700 1 -- 172.21.15.102:0/2167516697 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdee0130ea0 con 0x7fdee0096240 2021-05-25T12:08:51.472 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.420+0000 7fdec77fe700 10 monclient: _finish_auth 0 2021-05-25T12:08:51.473 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.420+0000 7fdec77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:08:51.473 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.420+0000 7fdee4e26700 5 monclient: authenticate success, global_id 264120 2021-05-25T12:08:51.473 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.420+0000 7fdee4e26700 10 monclient: _renew_subs 2021-05-25T12:08:51.473 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.420+0000 7fdee4e26700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:08:51.474 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.420+0000 7fdee4e26700 1 -- 172.21.15.102:0/2167516697 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdee012ed30 con 0x7fdee0096240 2021-05-25T12:08:51.474 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.420+0000 7fdee4e26700 10 client.264120.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:08:51.474 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.420+0000 7fdee4e26700 10 monclient: _renew_subs 2021-05-25T12:08:51.474 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.420+0000 7fdee4e26700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:08:51.474 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.420+0000 7fdee4e26700 1 -- 172.21.15.102:0/2167516697 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdee0134390 con 0x7fdee0096240 2021-05-25T12:08:51.475 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.420+0000 7fdee4e26700 1 librados: init done 2021-05-25T12:08:51.475 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.420+0000 7fdec77fe700 1 -- 172.21.15.102:0/2167516697 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fded4004bb0 con 0x7fdee0096240 2021-05-25T12:08:51.475 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.420+0000 7fdec77fe700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:08:51.475 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.420+0000 7fdec77fe700 1 -- 172.21.15.102:0/2167516697 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fded40015b0 con 0x7fdee0096240 2021-05-25T12:08:51.475 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.420+0000 7fdec77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:51.476 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.420+0000 7fdec77fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:08:51.476 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.420+0000 7fdec77fe700 10 monclient: dump: 2021-05-25T12:08:51.476 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:51.476 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:51.477 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:51.477 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:51.477 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:51.477 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:51.477 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:51.478 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:51.478 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:51.478 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:51.478 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.420+0000 7fdeddffb700 4 set_mon_vals no callback set 2021-05-25T12:08:51.478 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.421+0000 7fdec77fe700 1 -- 172.21.15.102:0/2167516697 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7fded400e080 con 0x7fdee0096240 2021-05-25T12:08:51.479 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.421+0000 7fdee4e26700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:08:51.479 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.421+0000 7fdee4e26700 1 -- 172.21.15.102:0/2167516697 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fdeb80065a0 con 0x7fdeb8002840 2021-05-25T12:08:51.479 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.421+0000 7fdec77fe700 1 -- 172.21.15.102:0/2167516697 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(293..293 src has 250..293) v4 ==== 6275+0+0 (unknown 4031507246 0 0) 0x7fded403dbd0 con 0x7fdee0096240 2021-05-25T12:08:51.479 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.421+0000 7fdec77fe700 10 client.264120.objecter ms_dispatch 0x7fdee0001550 osd_map(293..293 src has 250..293) v4 2021-05-25T12:08:51.480 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.421+0000 7fdec77fe700 3 client.264120.objecter handle_osd_map got epochs [293,293] > 0 2021-05-25T12:08:51.480 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.421+0000 7fdec77fe700 3 client.264120.objecter handle_osd_map decoding full epoch 293 2021-05-25T12:08:51.480 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.421+0000 7fdec77fe700 20 client.264120.objecter dump_active .. 0 homeless 2021-05-25T12:08:51.480 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.421+0000 7fdec77fe700 10 client.264120.objecter ms_handle_connect 0x7fdeb8002840 2021-05-25T12:08:51.480 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.421+0000 7fdec77fe700 10 client.264120.objecter resend_mon_ops 2021-05-25T12:08:51.481 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.421+0000 7fdedeffd700 10 monclient: get_auth_request con 0x7fdec0033430 auth_method 0 2021-05-25T12:08:51.481 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.421+0000 7fdec77fe700 1 -- 172.21.15.102:0/2167516697 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2204102410 0 0) 0x7fdeb80065a0 con 0x7fdeb8002840 2021-05-25T12:08:51.481 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.421+0000 7fdec77fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:51.481 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.421+0000 7fdec77fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:51.482 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.421+0000 7fdec77fe700 10 monclient(hunting): my global_id is 224494 2021-05-25T12:08:51.482 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.421+0000 7fdec77fe700 1 -- 172.21.15.102:0/2167516697 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdee0130960 con 0x7fdeb8002840 2021-05-25T12:08:51.482 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.421+0000 7fdec77fe700 10 client.264120.objecter ms_handle_connect 0x7fdec0033430 2021-05-25T12:08:51.482 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.421+0000 7fdec77fe700 1 -- 172.21.15.102:0/2167516697 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1117553021 0 0) 0x7fdee0130960 con 0x7fdeb8002840 2021-05-25T12:08:51.482 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.421+0000 7fdec77fe700 1 -- 172.21.15.102:0/2167516697 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdeb80065a0 con 0x7fdeb8002840 2021-05-25T12:08:51.483 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.421+0000 7fdec77fe700 1 -- 172.21.15.102:0/2167516697 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fded00046d0 con 0x7fdeb8002840 2021-05-25T12:08:51.483 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.421+0000 7fdec77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:51.483 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.421+0000 7fdec77fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T12:08:51.483 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.421+0000 7fdec77fe700 10 monclient: dump: 2021-05-25T12:08:51.484 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:51.484 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:51.484 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:51.484 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:51.484 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:51.485 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:51.485 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:51.485 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:51.485 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:51.485 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:51.486 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.422+0000 7fdec77fe700 1 -- 172.21.15.102:0/2167516697 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2933670668 0 0) 0x7fdeb80065a0 con 0x7fdeb8002840 2021-05-25T12:08:51.486 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.422+0000 7fdec77fe700 1 -- 172.21.15.102:0/2167516697 --> v1:172.21.15.181:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdeb8009e40 con 0x7fdeb8002840 2021-05-25T12:08:51.486 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.422+0000 7fdec77fe700 1 -- 172.21.15.102:0/2167516697 <== mon.1 v1:172.21.15.181:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7fded00069c0 con 0x7fdeb8002840 2021-05-25T12:08:51.486 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.422+0000 7fdec77fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:08:51.487 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.422+0000 7fdec77fe700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:08:51.487 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.422+0000 7fdec77fe700 1 -- 172.21.15.102:0/2167516697 >> v1:172.21.15.181:6789/0 conn(0x7fdeb8002840 legacy=0x7fdeb8004ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:51.487 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.422+0000 7fdec77fe700 1 -- 172.21.15.102:0/2167516697 >> v1:172.21.15.181:6789/0 conn(0x7fdeb8002840 legacy=0x7fdeb8004ce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:08:51.487 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.428+0000 7fdee4e26700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:08:51.487 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.428+0000 7fdee4e26700 1 -- 172.21.15.102:0/2167516697 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fdeb8007540 con 0x7fdeb80068e0 2021-05-25T12:08:51.488 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.429+0000 7fdec77fe700 10 client.264120.objecter ms_handle_connect 0x7fdeb80068e0 2021-05-25T12:08:51.488 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.429+0000 7fdec77fe700 10 client.264120.objecter resend_mon_ops 2021-05-25T12:08:51.488 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.429+0000 7fdec77fe700 1 -- 172.21.15.102:0/2167516697 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2934763697 0 0) 0x7fdeb8007540 con 0x7fdeb80068e0 2021-05-25T12:08:51.488 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.429+0000 7fdec77fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:51.489 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.429+0000 7fdec77fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:51.489 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.429+0000 7fdec77fe700 10 monclient(hunting): my global_id is 224497 2021-05-25T12:08:51.489 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.429+0000 7fdec77fe700 1 -- 172.21.15.102:0/2167516697 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdeb80065a0 con 0x7fdeb80068e0 2021-05-25T12:08:51.489 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.429+0000 7fdec77fe700 1 -- 172.21.15.102:0/2167516697 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 21418149 0 0) 0x7fdeb80065a0 con 0x7fdeb80068e0 2021-05-25T12:08:51.489 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.429+0000 7fdec77fe700 1 -- 172.21.15.102:0/2167516697 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdeb8007540 con 0x7fdeb80068e0 2021-05-25T12:08:51.490 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.429+0000 7fdec77fe700 1 -- 172.21.15.102:0/2167516697 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fded0004e80 con 0x7fdeb80068e0 2021-05-25T12:08:51.490 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.429+0000 7fdec77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:51.490 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.429+0000 7fdec77fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T12:08:51.490 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.429+0000 7fdec77fe700 10 monclient: dump: 2021-05-25T12:08:51.490 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:51.491 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:51.491 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:51.491 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:51.491 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:51.492 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:51.492 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:51.492 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:51.492 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:51.492 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:51.493 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.429+0000 7fdec77fe700 1 -- 172.21.15.102:0/2167516697 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2771538155 0 0) 0x7fdeb8007540 con 0x7fdeb80068e0 2021-05-25T12:08:51.493 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.429+0000 7fdec77fe700 1 -- 172.21.15.102:0/2167516697 --> v1:172.21.15.181:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fdeb8008770 con 0x7fdeb80068e0 2021-05-25T12:08:51.493 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.430+0000 7fdec77fe700 1 -- 172.21.15.102:0/2167516697 <== mon.1 v1:172.21.15.181:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4112 (unknown 0 0 3502235923) 0x7fded0006bd0 con 0x7fdeb80068e0 2021-05-25T12:08:51.493 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.430+0000 7fdec77fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:08:51.493 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.430+0000 7fdec77fe700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:08:51.494 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.430+0000 7fdec77fe700 1 -- 172.21.15.102:0/2167516697 >> v1:172.21.15.181:6789/0 conn(0x7fdeb80068e0 legacy=0x7fdeb8006d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:51.494 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.430+0000 7fdec77fe700 1 -- 172.21.15.102:0/2167516697 >> v1:172.21.15.181:6789/0 conn(0x7fdeb80068e0 legacy=0x7fdeb8006d70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:08:51.495 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:08:51.495 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:08:51.495 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:08:51.495 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "leader", 2021-05-25T12:08:51.495 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 74, 2021-05-25T12:08:51.496 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:08:51.496 INFO:teuthology.orchestra.run.smithi102.stdout: 0, 2021-05-25T12:08:51.496 INFO:teuthology.orchestra.run.smithi102.stdout: 1, 2021-05-25T12:08:51.496 INFO:teuthology.orchestra.run.smithi102.stdout: 2 2021-05-25T12:08:51.497 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:08:51.497 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 0, 2021-05-25T12:08:51.497 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:08:51.497 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:08:51.497 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:08:51.498 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:08:51.498 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:08:51.498 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:08:51.498 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:08:51.498 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:08:51.499 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:08:51.499 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:08:51.499 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:08:51.499 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:08:51.500 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:08:51.500 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:08:51.500 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:08:51.500 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:08:51.500 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:08:51.501 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:08:51.501 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:08:51.501 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:08:51.501 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:08:51.501 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:08:51.502 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:08:51.502 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:08:51.502 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:51.502 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:51.503 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:08:51.503 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:08:51.503 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:08:51.503 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:08:51.503 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:08:51.504 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:08:51.504 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:08:51.504 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:08:51.504 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:08:51.504 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:08:51.505 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:08:51.505 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:08:51.505 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:08:51.505 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:08:51.506 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:08:51.506 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:08:51.506 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:08:51.506 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:08:51.506 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:08:51.507 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:08:51.507 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:08:51.507 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:08:51.507 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:08:51.508 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:08:51.508 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:08:51.508 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:08:51.508 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:51.508 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:08:51.509 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:51.509 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:08:51.509 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:08:51.509 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:08:51.510 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:08:51.510 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:51.510 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:08:51.510 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:08:51.510 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:08:51.511 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:51.511 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:51.511 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:51.511 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:08:51.511 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:08:51.512 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:08:51.512 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:08:51.512 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:08:51.512 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:51.513 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:51.513 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:08:51.513 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:08:51.513 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:08:51.513 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:08:51.514 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:51.514 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:08:51.514 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:08:51.514 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:08:51.515 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:51.515 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:51.515 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:51.515 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:08:51.515 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:08:51.516 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:08:51.516 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:08:51.516 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:08:51.516 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:51.516 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:51.517 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:08:51.517 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:08:51.517 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:08:51.517 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:08:51.517 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:51.518 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:08:51.518 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:08:51.518 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:08:51.518 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:51.519 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:51.519 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:51.519 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:08:51.519 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:08:51.519 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:08:51.520 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:08:51.520 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:08:51.520 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:51.520 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:51.521 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:51.521 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:08:51.521 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:08:51.521 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:51.521 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:08:51.522 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:08:51.522 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:08:51.522 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:51.522 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:08:51.522 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": [ 2021-05-25T12:08:51.523 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:51.523 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:08:51.523 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:08:51.523 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 4 2021-05-25T12:08:51.524 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:51.524 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:08:51.524 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:08:51.524 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:51.524 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:08:51.525 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:08:51.525 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 6 2021-05-25T12:08:51.525 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:51.525 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:08:51.525 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": [ 2021-05-25T12:08:51.526 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:51.526 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:08:51.526 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:08:51.526 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:08:51.526 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:51.527 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:51.527 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:51.527 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:08:51.527 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:08:51.529 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.430+0000 7fdee4e26700 10 librados: watch_flush enter 2021-05-25T12:08:51.530 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.430+0000 7fdee4e26700 10 librados: watch_flush exit 2021-05-25T12:08:51.530 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.430+0000 7fdee4e26700 20 client.264120.objecter shutdown clearing up homeless session... 2021-05-25T12:08:51.530 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.430+0000 7fdee4e26700 10 client.264120.objecter successfully canceled tick 2021-05-25T12:08:51.530 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.430+0000 7fdee4e26700 1 -- 172.21.15.102:0/2167516697 >> v1:172.21.15.181:6812/29582 conn(0x7fdec0033430 legacy=0x7fdec00358f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:51.531 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.430+0000 7fdee4e26700 10 monclient: shutdown 2021-05-25T12:08:51.531 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.430+0000 7fdee4e26700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:08:51.534 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.430+0000 7fdee4e26700 1 -- 172.21.15.102:0/2167516697 >> v1:172.21.15.102:6789/0 conn(0x7fdee0096240 legacy=0x7fdee0143db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:51.535 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.430+0000 7fdedffff700 1 -- 172.21.15.102:0/2167516697 reap_dead start 2021-05-25T12:08:51.535 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.430+0000 7fdedffff700 1 -- 172.21.15.102:0/2167516697 reap_dead start 2021-05-25T12:08:51.535 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.430+0000 7fdee4e26700 1 -- 172.21.15.102:0/2167516697 shutdown_connections 2021-05-25T12:08:51.535 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.430+0000 7fdee4e26700 1 -- 172.21.15.102:0/2167516697 >> 172.21.15.102:0/2167516697 conn(0x7fdee00974b0 msgr2=0x7fdee0127d90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:08:51.535 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.430+0000 7fdee4e26700 1 -- 172.21.15.102:0/2167516697 shutdown_connections 2021-05-25T12:08:51.536 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.430+0000 7fdee4e26700 1 -- 172.21.15.102:0/2167516697 wait complete. 2021-05-25T12:08:51.536 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.430+0000 7fdee4e26700 1 librados: shutdown 2021-05-25T12:08:51.536 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2021-05-25T12:08:51.537 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2021-05-25T12:08:51.660 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.646+0000 7f586925a700 10 monclient: get_monmap_and_config 2021-05-25T12:08:51.660 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.646+0000 7f586925a700 10 monclient: build_initial_monmap 2021-05-25T12:08:51.660 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.646+0000 7f586925a700 10 monclient: monmap: 2021-05-25T12:08:51.661 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:08:51.661 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:51.661 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:08:51.648816+0000 2021-05-25T12:08:51.661 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:08:51.648816+0000 2021-05-25T12:08:51.661 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:08:51.662 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:51.662 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:08:51.662 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:08:51.662 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:08:51.663 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:51.663 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.647+0000 7f586925a700 1 Processor -- start 2021-05-25T12:08:51.663 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.647+0000 7f586925a700 1 -- start start 2021-05-25T12:08:51.663 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.647+0000 7f586925a700 10 monclient: init 2021-05-25T12:08:51.663 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.647+0000 7f586925a700 10 monclient: _reopen_session rank -1 2021-05-25T12:08:51.664 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.647+0000 7f586925a700 10 monclient: _add_conns ranks=[1,2,0] 2021-05-25T12:08:51.664 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.647+0000 7f586925a700 10 monclient(hunting): picked mon.noname-c con 0x7f586412bab0 addr v1:172.21.15.181:6789/0 2021-05-25T12:08:51.664 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.647+0000 7f586925a700 10 monclient(hunting): picked mon.noname-b con 0x7f586412c400 addr v1:172.21.15.102:6790/0 2021-05-25T12:08:51.664 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.647+0000 7f586925a700 10 monclient(hunting): picked mon.noname-a con 0x7f5864132520 addr v1:172.21.15.102:6789/0 2021-05-25T12:08:51.665 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.647+0000 7f586925a700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f586409a670 con 0x7f5864132520 2021-05-25T12:08:51.665 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.647+0000 7f586925a700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f586409b210 con 0x7f586412bab0 2021-05-25T12:08:51.665 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.647+0000 7f586925a700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f586409b4b0 con 0x7f586412c400 2021-05-25T12:08:51.665 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.647+0000 7f586925a700 10 monclient(hunting): _renew_subs 2021-05-25T12:08:51.665 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.647+0000 7f586925a700 10 monclient(hunting): authenticate will time out at 1332.245921s 2021-05-25T12:08:51.666 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.648+0000 7f586259c700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f586412c400 0x7f5864130e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:57542/0 (socket says 172.21.15.102:57542) 2021-05-25T12:08:51.666 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.648+0000 7f586259c700 1 -- 172.21.15.102:0/2499239456 learned_addr learned my addr 172.21.15.102:0/2499239456 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:08:51.666 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.648+0000 7f586359e700 1 --1- 172.21.15.102:0/2499239456 >> v1:172.21.15.102:6789/0 conn(0x7f5864132520 0x7f5864134910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:34436/0 (socket says 172.21.15.102:34436) 2021-05-25T12:08:51.666 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.648+0000 7f5861d9b700 1 -- 172.21.15.102:0/2499239456 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1655808407 0 0) 0x7f586409b4b0 con 0x7f586412c400 2021-05-25T12:08:51.666 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.648+0000 7f5861d9b700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:51.667 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.648+0000 7f5861d9b700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:51.667 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.648+0000 7f5861d9b700 10 monclient(hunting): my global_id is 244238 2021-05-25T12:08:51.667 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.648+0000 7f5861d9b700 1 -- 172.21.15.102:0/2499239456 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5850003270 con 0x7f586412c400 2021-05-25T12:08:51.667 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.648+0000 7f5861d9b700 1 -- 172.21.15.102:0/2499239456 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3598211674 0 0) 0x7f586409a670 con 0x7f5864132520 2021-05-25T12:08:51.668 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.648+0000 7f5861d9b700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:51.668 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.648+0000 7f5861d9b700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:51.668 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.648+0000 7f5861d9b700 10 monclient(hunting): my global_id is 264123 2021-05-25T12:08:51.668 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.648+0000 7f5861d9b700 1 -- 172.21.15.102:0/2499239456 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f586409b4b0 con 0x7f5864132520 2021-05-25T12:08:51.668 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.648+0000 7f5861d9b700 1 -- 172.21.15.102:0/2499239456 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2466779841 0 0) 0x7f586409b210 con 0x7f586412bab0 2021-05-25T12:08:51.669 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.648+0000 7f5861d9b700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:51.669 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.648+0000 7f5861d9b700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:51.669 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.648+0000 7f5861d9b700 10 monclient(hunting): my global_id is 224500 2021-05-25T12:08:51.669 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.648+0000 7f5861d9b700 1 -- 172.21.15.102:0/2499239456 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f586409a670 con 0x7f586412bab0 2021-05-25T12:08:51.669 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.648+0000 7f5861d9b700 1 -- 172.21.15.102:0/2499239456 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1878770647 0 0) 0x7f5850003270 con 0x7f586412c400 2021-05-25T12:08:51.670 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.648+0000 7f5861d9b700 1 -- 172.21.15.102:0/2499239456 >> v1:172.21.15.181:6789/0 conn(0x7f586412bab0 legacy=0x7f5864098960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:51.670 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.648+0000 7f5861d9b700 1 -- 172.21.15.102:0/2499239456 >> v1:172.21.15.102:6789/0 conn(0x7f5864132520 legacy=0x7f5864134910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:51.670 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.648+0000 7f5861d9b700 10 monclient: _finish_hunting 0 2021-05-25T12:08:51.670 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.648+0000 7f5861d9b700 1 monclient: found mon.noname-b 2021-05-25T12:08:51.671 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.648+0000 7f5861d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:08:51.671 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.648+0000 7f5861d9b700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T12:08:51.671 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.648+0000 7f5861d9b700 1 -- 172.21.15.102:0/2499239456 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58640964a0 con 0x7f586412c400 2021-05-25T12:08:51.671 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.648+0000 7f5861d9b700 10 monclient: _finish_auth 0 2021-05-25T12:08:51.671 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.648+0000 7f5861d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:08:51.672 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.648+0000 7f5861d9b700 1 -- 172.21.15.102:0/2499239456 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f584c001610 con 0x7f586412c400 2021-05-25T12:08:51.672 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.648+0000 7f5861d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:51.672 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.648+0000 7f5861d9b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:08:51.672 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.648+0000 7f5861d9b700 10 monclient: dump: 2021-05-25T12:08:51.672 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:51.673 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:51.673 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:51.673 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:51.673 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:51.673 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:51.674 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:51.674 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:51.674 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:51.674 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:51.675 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.648+0000 7f586925a700 5 monclient: authenticate success, global_id 244238 2021-05-25T12:08:51.675 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.648+0000 7f586925a700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:08:51.675 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.648+0000 7f5861d9b700 1 -- 172.21.15.102:0/2499239456 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f584c001880 con 0x7f586412c400 2021-05-25T12:08:51.675 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.648+0000 7f5861d9b700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:08:51.675 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.648+0000 7f5861d9b700 1 -- 172.21.15.102:0/2499239456 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f584c001c10 con 0x7f586412c400 2021-05-25T12:08:51.676 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.648+0000 7f586925a700 10 monclient: get_monmap_and_config success 2021-05-25T12:08:51.676 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.648+0000 7f586925a700 4 set_mon_vals no callback set 2021-05-25T12:08:51.676 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.648+0000 7f5861d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:51.676 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.648+0000 7f5861d9b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:08:51.676 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.648+0000 7f5861d9b700 10 monclient: dump: 2021-05-25T12:08:51.677 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:51.677 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:51.677 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:51.677 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:51.678 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:51.678 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:51.678 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:51.678 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:51.678 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:51.679 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:51.679 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.649+0000 7f586925a700 10 monclient: shutdown 2021-05-25T12:08:51.679 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.649+0000 7f586925a700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:08:51.679 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.649+0000 7f586925a700 1 -- 172.21.15.102:0/2499239456 >> v1:172.21.15.102:6790/0 conn(0x7f586412c400 legacy=0x7f5864130e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:51.679 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.649+0000 7f586925a700 1 -- 172.21.15.102:0/2499239456 shutdown_connections 2021-05-25T12:08:51.680 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.649+0000 7f586925a700 1 -- 172.21.15.102:0/2499239456 >> 172.21.15.102:0/2499239456 conn(0x7f586409b850 msgr2=0x7f586409bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:08:51.680 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.649+0000 7f586925a700 1 -- 172.21.15.102:0/2499239456 shutdown_connections 2021-05-25T12:08:51.680 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.649+0000 7f586925a700 1 -- 172.21.15.102:0/2499239456 wait complete. 2021-05-25T12:08:51.680 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.649+0000 7f586925a700 10 monclient: build_initial_monmap 2021-05-25T12:08:51.680 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.649+0000 7f586925a700 10 monclient: monmap: 2021-05-25T12:08:51.681 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:08:51.681 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:08:51.681 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:08:51.681 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:08:51.682 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:08:51.682 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:08:51.682 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:08:51.682 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:08:51.682 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:08:51.683 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:51.683 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.649+0000 7f586925a700 1 librados: starting msgr at 2021-05-25T12:08:51.683 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.649+0000 7f586925a700 1 librados: starting objecter 2021-05-25T12:08:51.683 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.650+0000 7f586925a700 1 Processor -- start 2021-05-25T12:08:51.683 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.650+0000 7f586925a700 1 -- start start 2021-05-25T12:08:51.684 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.650+0000 7f586925a700 1 librados: setting wanted keys 2021-05-25T12:08:51.684 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.650+0000 7f586925a700 1 librados: calling monclient init 2021-05-25T12:08:51.684 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.650+0000 7f586925a700 10 monclient: init 2021-05-25T12:08:51.684 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.650+0000 7f586925a700 10 monclient: _reopen_session rank -1 2021-05-25T12:08:51.684 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.650+0000 7f586925a700 10 monclient: _add_conns ranks=[0,1,2] 2021-05-25T12:08:51.685 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.650+0000 7f586925a700 10 monclient(hunting): picked mon.noname-a con 0x7f586412bab0 addr v1:172.21.15.102:6789/0 2021-05-25T12:08:51.685 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.650+0000 7f586925a700 10 monclient(hunting): picked mon.noname-b con 0x7f586412c400 addr v1:172.21.15.181:6789/0 2021-05-25T12:08:51.685 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.650+0000 7f586925a700 10 monclient(hunting): picked mon.noname-c con 0x7f5864132520 addr v1:172.21.15.102:6790/0 2021-05-25T12:08:51.685 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.650+0000 7f586925a700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f586412e470 con 0x7f586412bab0 2021-05-25T12:08:51.685 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.650+0000 7f586925a700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f586409a930 con 0x7f586412c400 2021-05-25T12:08:51.686 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.650+0000 7f586925a700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f586412e6a0 con 0x7f5864132520 2021-05-25T12:08:51.686 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.650+0000 7f586925a700 10 monclient(hunting): _renew_subs 2021-05-25T12:08:51.686 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.650+0000 7f586925a700 10 monclient(hunting): authenticate will time out at 1332.248518s 2021-05-25T12:08:51.686 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.650+0000 7f5862d9d700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f586412bab0 0x7f5864130e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:34438/0 (socket says 172.21.15.102:34438) 2021-05-25T12:08:51.687 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.650+0000 7f586359e700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f5864132520 0x7f5864142e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:57552/0 (socket says 172.21.15.102:57552) 2021-05-25T12:08:51.687 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.650+0000 7f5862d9d700 1 -- 172.21.15.102:0/796712746 learned_addr learned my addr 172.21.15.102:0/796712746 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:08:51.687 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.650+0000 7f584affd700 10 client.?.objecter ms_handle_connect 0x7f5864132520 2021-05-25T12:08:51.687 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.650+0000 7f584affd700 10 client.?.objecter resend_mon_ops 2021-05-25T12:08:51.687 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.650+0000 7f584affd700 10 client.?.objecter ms_handle_connect 0x7f586412bab0 2021-05-25T12:08:51.688 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.650+0000 7f584affd700 10 client.?.objecter resend_mon_ops 2021-05-25T12:08:51.688 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.650+0000 7f584affd700 1 -- 172.21.15.102:0/796712746 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3774485859 0 0) 0x7f586412e6a0 con 0x7f5864132520 2021-05-25T12:08:51.688 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.650+0000 7f584affd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:51.688 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.650+0000 7f584affd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:51.688 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.650+0000 7f584affd700 10 monclient(hunting): my global_id is 244241 2021-05-25T12:08:51.689 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.650+0000 7f584affd700 1 -- 172.21.15.102:0/796712746 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5844004c10 con 0x7f5864132520 2021-05-25T12:08:51.689 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.650+0000 7f584affd700 1 -- 172.21.15.102:0/796712746 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3870598593 0 0) 0x7f586412e470 con 0x7f586412bab0 2021-05-25T12:08:51.689 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.650+0000 7f584affd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:51.689 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.650+0000 7f584affd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:51.689 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.650+0000 7f584affd700 10 monclient(hunting): my global_id is 264126 2021-05-25T12:08:51.690 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.650+0000 7f584affd700 1 -- 172.21.15.102:0/796712746 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f586412e6a0 con 0x7f586412bab0 2021-05-25T12:08:51.690 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.650+0000 7f584affd700 10 client.?.objecter ms_handle_connect 0x7f586412c400 2021-05-25T12:08:51.690 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.650+0000 7f584affd700 10 client.?.objecter resend_mon_ops 2021-05-25T12:08:51.690 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.651+0000 7f584affd700 1 -- 172.21.15.102:0/796712746 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1729971089 0 0) 0x7f586409a930 con 0x7f586412c400 2021-05-25T12:08:51.691 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.651+0000 7f584affd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:51.691 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.651+0000 7f584affd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:51.691 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.651+0000 7f584affd700 10 monclient(hunting): my global_id is 224503 2021-05-25T12:08:51.691 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.651+0000 7f584affd700 1 -- 172.21.15.102:0/796712746 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f586412e470 con 0x7f586412c400 2021-05-25T12:08:51.691 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.651+0000 7f584affd700 1 -- 172.21.15.102:0/796712746 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2577378610 0 0) 0x7f5844004c10 con 0x7f5864132520 2021-05-25T12:08:51.692 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.651+0000 7f584affd700 1 -- 172.21.15.102:0/796712746 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f586409a930 con 0x7f5864132520 2021-05-25T12:08:51.692 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.651+0000 7f584affd700 1 -- 172.21.15.102:0/796712746 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f5854001d50 con 0x7f5864132520 2021-05-25T12:08:51.692 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.651+0000 7f584affd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:51.692 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.651+0000 7f584affd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:08:51.692 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.651+0000 7f584affd700 10 monclient(hunting): dump: 2021-05-25T12:08:51.693 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:51.693 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:51.693 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:51.693 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:51.694 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:51.694 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:51.694 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:51.694 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:51.694 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:51.695 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:51.695 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.651+0000 7f584affd700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:08:51.695 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.651+0000 7f584affd700 1 -- 172.21.15.102:0/796712746 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1593650640 0 0) 0x7f586412e6a0 con 0x7f586412bab0 2021-05-25T12:08:51.695 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.651+0000 7f584affd700 1 -- 172.21.15.102:0/796712746 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5844004c10 con 0x7f586412bab0 2021-05-25T12:08:51.695 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.651+0000 7f584affd700 1 -- 172.21.15.102:0/796712746 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f5858001900 con 0x7f586412bab0 2021-05-25T12:08:51.696 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.651+0000 7f584affd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:51.696 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.651+0000 7f584affd700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2021-05-25T12:08:51.696 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.651+0000 7f584affd700 10 monclient(hunting): dump: 2021-05-25T12:08:51.696 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:51.697 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:51.697 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:51.697 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:51.697 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:51.697 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:51.698 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:51.698 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:51.698 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:51.698 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:51.699 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.651+0000 7f584affd700 1 -- 172.21.15.102:0/796712746 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 402664880 0 0) 0x7f586409a930 con 0x7f5864132520 2021-05-25T12:08:51.699 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.651+0000 7f584affd700 1 -- 172.21.15.102:0/796712746 >> v1:172.21.15.181:6789/0 conn(0x7f586412c400 legacy=0x7f586413ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:51.699 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.651+0000 7f584affd700 1 -- 172.21.15.102:0/796712746 >> v1:172.21.15.102:6789/0 conn(0x7f586412bab0 legacy=0x7f5864130e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:51.699 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.651+0000 7f584affd700 10 monclient: _finish_hunting 0 2021-05-25T12:08:51.699 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.651+0000 7f584affd700 1 monclient: found mon.c 2021-05-25T12:08:51.700 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.651+0000 7f584affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:08:51.700 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.651+0000 7f584affd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:08:51.700 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.651+0000 7f584affd700 1 -- 172.21.15.102:0/796712746 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f586412e940 con 0x7f5864132520 2021-05-25T12:08:51.700 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.651+0000 7f584affd700 10 monclient: _finish_auth 0 2021-05-25T12:08:51.700 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.651+0000 7f584affd700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:08:51.701 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.651+0000 7f586925a700 5 monclient: authenticate success, global_id 244241 2021-05-25T12:08:51.701 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.651+0000 7f586925a700 10 monclient: _renew_subs 2021-05-25T12:08:51.701 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.651+0000 7f586925a700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:08:51.701 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.651+0000 7f586925a700 1 -- 172.21.15.102:0/796712746 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5864131510 con 0x7f5864132520 2021-05-25T12:08:51.702 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.651+0000 7f586925a700 10 client.244241.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:08:51.702 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.651+0000 7f586925a700 10 monclient: _renew_subs 2021-05-25T12:08:51.702 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.651+0000 7f586925a700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:08:51.702 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.651+0000 7f586925a700 1 -- 172.21.15.102:0/796712746 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f586413b480 con 0x7f5864132520 2021-05-25T12:08:51.702 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.651+0000 7f586925a700 1 librados: init done 2021-05-25T12:08:51.703 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.651+0000 7f584affd700 1 -- 172.21.15.102:0/796712746 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f58540048b0 con 0x7f5864132520 2021-05-25T12:08:51.703 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.651+0000 7f584affd700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:08:51.703 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.651+0000 7f584affd700 1 -- 172.21.15.102:0/796712746 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f5854004bc0 con 0x7f5864132520 2021-05-25T12:08:51.703 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.651+0000 7f584affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:51.704 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.651+0000 7f584affd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:08:51.704 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.651+0000 7f586159a700 4 set_mon_vals no callback set 2021-05-25T12:08:51.704 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.651+0000 7f584affd700 10 monclient: dump: 2021-05-25T12:08:51.704 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:51.704 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:51.705 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:51.705 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:51.705 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:51.705 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:51.705 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:51.706 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:51.706 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:51.706 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:51.706 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.651+0000 7f584affd700 1 -- 172.21.15.102:0/796712746 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f585400dee0 con 0x7f5864132520 2021-05-25T12:08:51.707 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.652+0000 7f584affd700 1 -- 172.21.15.102:0/796712746 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(294..294 src has 250..294) v4 ==== 5858+0+0 (unknown 3138115535 0 0) 0x7f585403d9f0 con 0x7f5864132520 2021-05-25T12:08:51.707 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.652+0000 7f584affd700 10 client.244241.objecter ms_dispatch 0x7f5864001550 osd_map(294..294 src has 250..294) v4 2021-05-25T12:08:51.707 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.652+0000 7f584affd700 3 client.244241.objecter handle_osd_map got epochs [294,294] > 0 2021-05-25T12:08:51.707 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.652+0000 7f584affd700 3 client.244241.objecter handle_osd_map decoding full epoch 294 2021-05-25T12:08:51.707 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.652+0000 7f584affd700 20 client.244241.objecter dump_active .. 0 homeless 2021-05-25T12:08:51.708 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.652+0000 7f586925a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:08:51.708 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.652+0000 7f586925a700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:08:51.708 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.652+0000 7f586925a700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:08:51.708 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.652+0000 7f586925a700 1 -- 172.21.15.102:0/796712746 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f583c004db0 con 0x7f5864132520 2021-05-25T12:08:51.708 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.652+0000 7f5862d9d700 10 monclient: get_auth_request con 0x7f58440334b0 auth_method 0 2021-05-25T12:08:51.709 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.652+0000 7f584affd700 10 client.244241.objecter ms_handle_connect 0x7f58440334b0 2021-05-25T12:08:51.709 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.656+0000 7f584affd700 1 -- 172.21.15.102:0/796712746 <== mon.2 v1:172.21.15.102:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7f5854012f80 con 0x7f5864132520 2021-05-25T12:08:51.709 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.656+0000 7f584affd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:08:51.709 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.656+0000 7f584affd700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:08:51.868 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.865+0000 7f586925a700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2021-05-25T12:08:51.868 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.865+0000 7f586925a700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2021-05-25T12:08:51.868 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.865+0000 7f586925a700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:08:51.868 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.865+0000 7f586925a700 1 -- 172.21.15.102:0/796712746 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f583c005870 con 0x7f5864132520 2021-05-25T12:08:51.870 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.867+0000 7f584affd700 1 -- 172.21.15.102:0/796712746 <== mon.2 v1:172.21.15.102:6790/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7f5854012b70 con 0x7f5864132520 2021-05-25T12:08:51.870 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.867+0000 7f584affd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2021-05-25T12:08:51.870 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.867+0000 7f584affd700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:08:51.870 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:51.867+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:08:51.871 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:51.867+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=21,config=2,health=12,kv=5,logm=60} crc {auth=343112724,config=3140514009,health=1715047168,kv=825553405,logm=2220612507}) 2021-05-25T12:08:51.871 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:51.867+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=25,config=2,health=10,kv=5,logm=58} crc {auth=2913247552,config=3140514009,health=506663823,kv=825553405,logm=4042088442}) 2021-05-25T12:08:51.871 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:51.867+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:08:51.871 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:51.867+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=21,config=2,health=12,kv=5,logm=60} crc {auth=343112724,config=3140514009,health=1715047168,kv=825553405,logm=2220612507}) 2021-05-25T12:08:51.872 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:51.867+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=26,config=2,health=12,kv=6,logm=54} crc {auth=2985817901,config=3140514009,health=1715047168,kv=1184736763,logm=2548979498}) 2021-05-25T12:08:51.874 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.869+0000 7f586925a700 10 librados: watch_flush enter 2021-05-25T12:08:51.875 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.869+0000 7f586925a700 10 librados: watch_flush exit 2021-05-25T12:08:51.875 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.869+0000 7f586925a700 20 client.244241.objecter shutdown clearing up homeless session... 2021-05-25T12:08:51.875 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.869+0000 7f586925a700 10 client.244241.objecter successfully canceled tick 2021-05-25T12:08:51.875 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.869+0000 7f586925a700 1 -- 172.21.15.102:0/796712746 >> v1:172.21.15.181:6812/29582 conn(0x7f58440334b0 legacy=0x7f5844035970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:51.876 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.869+0000 7f586925a700 10 monclient: shutdown 2021-05-25T12:08:51.876 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.869+0000 7f586925a700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:08:51.876 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.869+0000 7f586925a700 1 -- 172.21.15.102:0/796712746 >> v1:172.21.15.102:6790/0 conn(0x7f5864132520 legacy=0x7f5864142e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:51.876 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.869+0000 7f586925a700 1 -- 172.21.15.102:0/796712746 shutdown_connections 2021-05-25T12:08:51.876 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.869+0000 7f586925a700 1 -- 172.21.15.102:0/796712746 >> 172.21.15.102:0/796712746 conn(0x7f586409b850 msgr2=0x7f586412a840 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:08:51.877 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.870+0000 7f586925a700 1 -- 172.21.15.102:0/796712746 shutdown_connections 2021-05-25T12:08:51.877 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.870+0000 7f586925a700 1 -- 172.21.15.102:0/796712746 wait complete. 2021-05-25T12:08:51.877 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:51.870+0000 7f586925a700 1 librados: shutdown 2021-05-25T12:08:51.877 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:51.868+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:08:51.878 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:51.868+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3071141480}) 2021-05-25T12:08:51.878 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:51.868+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1331163991}) 2021-05-25T12:08:51.878 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:51.868+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:08:51.878 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:51.868+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3071141480}) 2021-05-25T12:08:51.879 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:51.868+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1925947693}) 2021-05-25T12:08:51.879 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:51.869+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:08:51.879 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:51.869+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3241418967}) 2021-05-25T12:08:51.879 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:51.870+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2645117518}) 2021-05-25T12:08:51.879 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:51.870+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:08:51.880 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:51.870+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3241418967}) 2021-05-25T12:08:51.880 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:51.870+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2019850994}) 2021-05-25T12:08:51.880 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:51.872+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:08:51.880 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:51.872+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=220012969}) 2021-05-25T12:08:51.881 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:51.872+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3713290457}) 2021-05-25T12:08:51.881 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:51.872+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:08:51.881 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:51.872+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=220012969}) 2021-05-25T12:08:51.881 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:51.872+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2656726295}) 2021-05-25T12:08:51.883 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:51.876+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:08:51.883 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:51.876+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=531242740}) 2021-05-25T12:08:51.884 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:51.876+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=449532463}) 2021-05-25T12:08:51.884 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:51.876+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:08:51.884 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:51.876+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=531242740}) 2021-05-25T12:08:51.884 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:51.876+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=621140521}) 2021-05-25T12:08:51.884 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:51.879+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:08:51.885 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:51.879+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=80,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=8} crc {logm=3206157780,mdsmap=1531151339,mgr=2996579194,mgr_command_descs=3824783157,mgr_metadata=1267308364,mgrstat=2339209057}) 2021-05-25T12:08:51.885 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:51.879+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=77,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=4} crc {logm=4005316043,mdsmap=1531151339,mgr=3587935698,mgr_command_descs=3824783157,mgr_metadata=1267308364,mgrstat=2339209057,mon_config_key=4118394475}) 2021-05-25T12:08:51.885 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:51.879+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:08:51.885 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:51.879+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=80,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=8} crc {logm=3206157780,mdsmap=1531151339,mgr=2996579194,mgr_command_descs=3824783157,mgr_metadata=1267308364,mgrstat=2339209057}) 2021-05-25T12:08:51.886 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:51.879+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=81,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=7} crc {logm=2946200949,mdsmap=1531151339,mgr=2996579194,mgr_command_descs=3824783157,mgr_metadata=1267308364,mgrstat=1539620877}) 2021-05-25T12:08:51.886 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:51.881+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:08:51.886 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:51.881+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mon_config_key=4,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=28,osdmap=59} crc {mon_config_key=4118394475,monmap=3809614708,osd_metadata=252357741,osd_pg_creating=2091462384,osd_snap=3836255949,osdmap=142915363}) 2021-05-25T12:08:51.886 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:51.881+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=4,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=29,osdmap=58} crc {mon_config_key=4118394475,monmap=3809614708,osd_metadata=927457006,osd_pg_creating=2091462384,osd_snap=1641920236,osdmap=676078226}) 2021-05-25T12:08:51.887 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:51.881+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:08:51.887 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:51.881+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mon_config_key=4,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=28,osdmap=59} crc {mon_config_key=4118394475,monmap=3809614708,osd_metadata=252357741,osd_pg_creating=2091462384,osd_snap=3836255949,osdmap=142915363}) 2021-05-25T12:08:51.887 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:51.881+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mon_config_key=3,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=28,osdmap=60} crc {mon_config_key=2996166646,monmap=3809614708,osd_metadata=4154602222,osd_pg_creating=2091462384,osd_snap=3613131787,osdmap=2586115087}) 2021-05-25T12:08:51.890 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2021-05-25T12:08:51.891 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:51.889+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:08:51.891 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:51.889+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=27} crc {osdmap=576892903}) 2021-05-25T12:08:51.892 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:51.889+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=26} crc {osdmap=1476104961}) 2021-05-25T12:08:51.892 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:51.889+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2021-05-25T12:08:51.892 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:51.889+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=27} crc {osdmap=576892903}) 2021-05-25T12:08:51.892 INFO:tasks.ceph.mon.b.smithi181.stderr:2021-05-25T12:08:51.889+0000 7f2a1529d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=26} crc {osdmap=1476104961}) 2021-05-25T12:08:52.892 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2021-05-25T12:08:52.892 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2021-05-25T12:08:53.009 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.008+0000 7f2906ff0700 10 monclient: get_monmap_and_config 2021-05-25T12:08:53.010 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.008+0000 7f2906ff0700 10 monclient: build_initial_monmap 2021-05-25T12:08:53.010 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.008+0000 7f2906ff0700 10 monclient: monmap: 2021-05-25T12:08:53.010 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:08:53.010 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:53.011 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:08:53.009282+0000 2021-05-25T12:08:53.011 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:08:53.009282+0000 2021-05-25T12:08:53.011 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:08:53.011 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:53.011 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:08:53.012 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:08:53.012 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:08:53.012 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:53.023 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.008+0000 7f2906ff0700 1 Processor -- start 2021-05-25T12:08:53.023 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.008+0000 7f2906ff0700 1 -- start start 2021-05-25T12:08:53.023 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.008+0000 7f2906ff0700 10 monclient: init 2021-05-25T12:08:53.024 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.008+0000 7f2906ff0700 10 monclient: _reopen_session rank -1 2021-05-25T12:08:53.024 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.008+0000 7f2906ff0700 10 monclient: _add_conns ranks=[0,1,2] 2021-05-25T12:08:53.024 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.008+0000 7f2906ff0700 10 monclient(hunting): picked mon.noname-a con 0x7f2900090990 addr v1:172.21.15.102:6789/0 2021-05-25T12:08:53.024 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.008+0000 7f2906ff0700 10 monclient(hunting): picked mon.noname-c con 0x7f2900090d70 addr v1:172.21.15.181:6789/0 2021-05-25T12:08:53.024 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.008+0000 7f2906ff0700 10 monclient(hunting): picked mon.noname-b con 0x7f290012e330 addr v1:172.21.15.102:6790/0 2021-05-25T12:08:53.025 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.008+0000 7f2906ff0700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2900096fa0 con 0x7f2900090990 2021-05-25T12:08:53.025 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.008+0000 7f2906ff0700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29000926a0 con 0x7f2900090d70 2021-05-25T12:08:53.025 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.008+0000 7f2906ff0700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2900092940 con 0x7f290012e330 2021-05-25T12:08:53.025 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.008+0000 7f2906ff0700 10 monclient(hunting): _renew_subs 2021-05-25T12:08:53.026 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.008+0000 7f2906ff0700 10 monclient(hunting): authenticate will time out at 1333.606546s 2021-05-25T12:08:53.026 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.008+0000 7f290558d700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f290012e330 0x7f2900130720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:57558/0 (socket says 172.21.15.102:57558) 2021-05-25T12:08:53.026 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.008+0000 7f2904d8c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f2900090990 0x7f2900093240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:34448/0 (socket says 172.21.15.102:34448) 2021-05-25T12:08:53.026 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.008+0000 7f290558d700 1 -- 172.21.15.102:0/192315266 learned_addr learned my addr 172.21.15.102:0/192315266 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:08:53.026 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.008+0000 7f28f77fe700 1 -- 172.21.15.102:0/192315266 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 496061430 0 0) 0x7f2900092940 con 0x7f290012e330 2021-05-25T12:08:53.027 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.008+0000 7f28f77fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:53.027 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.008+0000 7f28f77fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:53.027 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.008+0000 7f28f77fe700 10 monclient(hunting): my global_id is 244244 2021-05-25T12:08:53.027 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.008+0000 7f28f77fe700 1 -- 172.21.15.102:0/192315266 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28ec004380 con 0x7f290012e330 2021-05-25T12:08:53.027 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.008+0000 7f28f77fe700 1 -- 172.21.15.102:0/192315266 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4219240892 0 0) 0x7f2900096fa0 con 0x7f2900090990 2021-05-25T12:08:53.028 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.008+0000 7f28f77fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:53.028 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.008+0000 7f28f77fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:53.028 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.009+0000 7f28f77fe700 10 monclient(hunting): my global_id is 264129 2021-05-25T12:08:53.028 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.009+0000 7f28f77fe700 1 -- 172.21.15.102:0/192315266 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2900092940 con 0x7f2900090990 2021-05-25T12:08:53.028 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.009+0000 7f28f77fe700 1 -- 172.21.15.102:0/192315266 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3705677854 0 0) 0x7f29000926a0 con 0x7f2900090d70 2021-05-25T12:08:53.029 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.009+0000 7f28f77fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:53.029 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.009+0000 7f28f77fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:53.029 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.009+0000 7f28f77fe700 10 monclient(hunting): my global_id is 224506 2021-05-25T12:08:53.029 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.009+0000 7f28f77fe700 1 -- 172.21.15.102:0/192315266 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2900096fa0 con 0x7f2900090d70 2021-05-25T12:08:53.030 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.009+0000 7f28f77fe700 1 -- 172.21.15.102:0/192315266 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 359779375 0 0) 0x7f28ec004380 con 0x7f290012e330 2021-05-25T12:08:53.030 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.009+0000 7f28f77fe700 1 -- 172.21.15.102:0/192315266 >> v1:172.21.15.181:6789/0 conn(0x7f2900090d70 legacy=0x7f2900091200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:53.030 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.009+0000 7f28f77fe700 1 -- 172.21.15.102:0/192315266 >> v1:172.21.15.102:6789/0 conn(0x7f2900090990 legacy=0x7f2900093240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:53.030 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.009+0000 7f28f77fe700 10 monclient: _finish_hunting 0 2021-05-25T12:08:53.030 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.009+0000 7f28f77fe700 1 monclient: found mon.noname-b 2021-05-25T12:08:53.031 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.009+0000 7f28f77fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:08:53.031 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.009+0000 7f28f77fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T12:08:53.031 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.009+0000 7f28f77fe700 1 -- 172.21.15.102:0/192315266 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2900092be0 con 0x7f290012e330 2021-05-25T12:08:53.031 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.009+0000 7f28f77fe700 10 monclient: _finish_auth 0 2021-05-25T12:08:53.031 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.009+0000 7f28f77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:08:53.032 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.009+0000 7f28f77fe700 1 -- 172.21.15.102:0/192315266 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f28fc001990 con 0x7f290012e330 2021-05-25T12:08:53.032 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.009+0000 7f2906ff0700 5 monclient: authenticate success, global_id 244244 2021-05-25T12:08:53.032 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.009+0000 7f2906ff0700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:08:53.032 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.009+0000 7f28f77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:53.032 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.009+0000 7f28f77fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:08:53.033 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.009+0000 7f28f77fe700 10 monclient: dump: 2021-05-25T12:08:53.033 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:53.033 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:53.033 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:53.033 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:53.034 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:53.034 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:53.034 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:53.034 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:53.034 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:53.035 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:53.035 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.009+0000 7f2906ff0700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:08:53.035 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.009+0000 7f28f77fe700 1 -- 172.21.15.102:0/192315266 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f28fc001c00 con 0x7f290012e330 2021-05-25T12:08:53.035 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.009+0000 7f28f77fe700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:08:53.035 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.009+0000 7f28f77fe700 1 -- 172.21.15.102:0/192315266 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f28fc001f90 con 0x7f290012e330 2021-05-25T12:08:53.036 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.009+0000 7f2906ff0700 10 monclient: get_monmap_and_config success 2021-05-25T12:08:53.036 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.009+0000 7f2906ff0700 4 set_mon_vals no callback set 2021-05-25T12:08:53.036 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.009+0000 7f28f77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:53.036 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.009+0000 7f28f77fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:08:53.036 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.009+0000 7f28f77fe700 10 monclient: dump: 2021-05-25T12:08:53.037 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:53.037 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:53.037 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:53.037 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:53.037 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:53.038 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:53.038 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:53.038 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:53.038 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:53.039 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:53.039 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.009+0000 7f2906ff0700 10 monclient: shutdown 2021-05-25T12:08:53.039 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.009+0000 7f2906ff0700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:08:53.039 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.009+0000 7f2906ff0700 1 -- 172.21.15.102:0/192315266 >> v1:172.21.15.102:6790/0 conn(0x7f290012e330 legacy=0x7f2900130720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:53.039 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.009+0000 7f2906ff0700 1 -- 172.21.15.102:0/192315266 shutdown_connections 2021-05-25T12:08:53.040 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.009+0000 7f2906ff0700 1 -- 172.21.15.102:0/192315266 >> 172.21.15.102:0/192315266 conn(0x7f290009b370 msgr2=0x7f2900097430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:08:53.040 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.009+0000 7f2906ff0700 1 -- 172.21.15.102:0/192315266 shutdown_connections 2021-05-25T12:08:53.040 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.009+0000 7f2906ff0700 1 -- 172.21.15.102:0/192315266 wait complete. 2021-05-25T12:08:53.040 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.010+0000 7f2906ff0700 10 monclient: build_initial_monmap 2021-05-25T12:08:53.040 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.010+0000 7f2906ff0700 10 monclient: monmap: 2021-05-25T12:08:53.041 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:08:53.041 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:08:53.041 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:08:53.041 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:08:53.041 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:08:53.042 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:08:53.042 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:08:53.042 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:08:53.042 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:08:53.042 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:53.043 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.010+0000 7f2906ff0700 1 librados: starting msgr at 2021-05-25T12:08:53.043 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.010+0000 7f2906ff0700 1 librados: starting objecter 2021-05-25T12:08:53.043 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.010+0000 7f2906ff0700 1 Processor -- start 2021-05-25T12:08:53.043 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.010+0000 7f2906ff0700 1 -- start start 2021-05-25T12:08:53.044 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.010+0000 7f2906ff0700 1 librados: setting wanted keys 2021-05-25T12:08:53.044 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.010+0000 7f2906ff0700 1 librados: calling monclient init 2021-05-25T12:08:53.044 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.010+0000 7f2906ff0700 10 monclient: init 2021-05-25T12:08:53.044 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.011+0000 7f2906ff0700 10 monclient: _reopen_session rank -1 2021-05-25T12:08:53.044 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.011+0000 7f2906ff0700 10 monclient: _add_conns ranks=[1,0,2] 2021-05-25T12:08:53.045 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.011+0000 7f2906ff0700 10 monclient(hunting): picked mon.noname-b con 0x7f2900090990 addr v1:172.21.15.181:6789/0 2021-05-25T12:08:53.045 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.011+0000 7f2906ff0700 10 monclient(hunting): picked mon.noname-a con 0x7f2900090d70 addr v1:172.21.15.102:6789/0 2021-05-25T12:08:53.045 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.011+0000 7f2906ff0700 10 monclient(hunting): picked mon.noname-c con 0x7f290012e330 addr v1:172.21.15.102:6790/0 2021-05-25T12:08:53.045 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.011+0000 7f2906ff0700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f290013ba60 con 0x7f2900090d70 2021-05-25T12:08:53.045 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.011+0000 7f2906ff0700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2900068620 con 0x7f2900090990 2021-05-25T12:08:53.046 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.011+0000 7f2906ff0700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29000688c0 con 0x7f290012e330 2021-05-25T12:08:53.046 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.011+0000 7f2906ff0700 10 monclient(hunting): _renew_subs 2021-05-25T12:08:53.046 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.011+0000 7f2906ff0700 10 monclient(hunting): authenticate will time out at 1333.609357s 2021-05-25T12:08:53.046 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.011+0000 7f290558d700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f290012e330 0x7f2900142ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:57566/0 (socket says 172.21.15.102:57566) 2021-05-25T12:08:53.046 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.011+0000 7f290558d700 1 -- 172.21.15.102:0/2041138138 learned_addr learned my addr 172.21.15.102:0/2041138138 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:08:53.047 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.011+0000 7f28f7fff700 1 --1- 172.21.15.102:0/2041138138 >> v1:172.21.15.102:6789/0 conn(0x7f2900090d70 0x7f2900126490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:34454/0 (socket says 172.21.15.102:34454) 2021-05-25T12:08:53.047 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.011+0000 7f28f4ff9700 10 client.?.objecter ms_handle_connect 0x7f290012e330 2021-05-25T12:08:53.047 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.011+0000 7f28f4ff9700 10 client.?.objecter resend_mon_ops 2021-05-25T12:08:53.047 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.011+0000 7f28f4ff9700 10 client.?.objecter ms_handle_connect 0x7f2900090d70 2021-05-25T12:08:53.047 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.011+0000 7f28f4ff9700 10 client.?.objecter resend_mon_ops 2021-05-25T12:08:53.048 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.011+0000 7f28f4ff9700 10 client.?.objecter ms_handle_connect 0x7f2900090990 2021-05-25T12:08:53.048 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.011+0000 7f28f4ff9700 10 client.?.objecter resend_mon_ops 2021-05-25T12:08:53.048 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.011+0000 7f28f4ff9700 1 -- 172.21.15.102:0/2041138138 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2662038095 0 0) 0x7f29000688c0 con 0x7f290012e330 2021-05-25T12:08:53.048 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.011+0000 7f28f4ff9700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:53.048 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.011+0000 7f28f4ff9700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:53.049 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.011+0000 7f28f4ff9700 10 monclient(hunting): my global_id is 244247 2021-05-25T12:08:53.049 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.011+0000 7f28f4ff9700 1 -- 172.21.15.102:0/2041138138 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28e0004c10 con 0x7f290012e330 2021-05-25T12:08:53.049 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.011+0000 7f28f4ff9700 1 -- 172.21.15.102:0/2041138138 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2097526754 0 0) 0x7f290013ba60 con 0x7f2900090d70 2021-05-25T12:08:53.049 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.011+0000 7f28f4ff9700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:53.050 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.011+0000 7f28f4ff9700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:53.050 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.011+0000 7f28f4ff9700 10 monclient(hunting): my global_id is 264132 2021-05-25T12:08:53.050 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.011+0000 7f28f4ff9700 1 -- 172.21.15.102:0/2041138138 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29000688c0 con 0x7f2900090d70 2021-05-25T12:08:53.050 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.011+0000 7f28f4ff9700 1 -- 172.21.15.102:0/2041138138 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 795811665 0 0) 0x7f2900068620 con 0x7f2900090990 2021-05-25T12:08:53.050 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.011+0000 7f28f4ff9700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:53.051 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.011+0000 7f28f4ff9700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:53.051 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.011+0000 7f28f4ff9700 10 monclient(hunting): my global_id is 224509 2021-05-25T12:08:53.051 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.011+0000 7f28f4ff9700 1 -- 172.21.15.102:0/2041138138 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f290013ba60 con 0x7f2900090990 2021-05-25T12:08:53.051 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.011+0000 7f28f4ff9700 1 -- 172.21.15.102:0/2041138138 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3109315977 0 0) 0x7f28e0004c10 con 0x7f290012e330 2021-05-25T12:08:53.051 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.011+0000 7f28f4ff9700 1 -- 172.21.15.102:0/2041138138 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2900068620 con 0x7f290012e330 2021-05-25T12:08:53.052 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.011+0000 7f28f4ff9700 1 -- 172.21.15.102:0/2041138138 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f28fc001950 con 0x7f290012e330 2021-05-25T12:08:53.052 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.011+0000 7f28f4ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:53.052 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.011+0000 7f28f4ff9700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:08:53.052 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.011+0000 7f28f4ff9700 10 monclient(hunting): dump: 2021-05-25T12:08:53.052 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:53.053 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:53.053 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:53.053 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:53.053 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:53.053 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:53.054 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:53.054 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:53.054 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:53.054 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:53.054 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.011+0000 7f28f4ff9700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:08:53.055 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.011+0000 7f28f4ff9700 1 -- 172.21.15.102:0/2041138138 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4102155529 0 0) 0x7f29000688c0 con 0x7f2900090d70 2021-05-25T12:08:53.055 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.012+0000 7f28f4ff9700 1 -- 172.21.15.102:0/2041138138 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f28e0004c10 con 0x7f2900090d70 2021-05-25T12:08:53.055 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.012+0000 7f28f4ff9700 1 -- 172.21.15.102:0/2041138138 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 354776029 0 0) 0x7f290013ba60 con 0x7f2900090990 2021-05-25T12:08:53.055 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.012+0000 7f28f4ff9700 1 -- 172.21.15.102:0/2041138138 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f29000688c0 con 0x7f2900090990 2021-05-25T12:08:53.055 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.012+0000 7f28f4ff9700 1 -- 172.21.15.102:0/2041138138 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f28e8001840 con 0x7f2900090d70 2021-05-25T12:08:53.056 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.012+0000 7f28f4ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:53.056 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.012+0000 7f28f4ff9700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2021-05-25T12:08:53.056 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.012+0000 7f28f4ff9700 10 monclient(hunting): dump: 2021-05-25T12:08:53.056 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:53.056 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:53.057 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:53.057 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:53.057 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:53.057 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:53.057 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:53.058 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:53.058 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:53.058 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:53.058 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.012+0000 7f28f4ff9700 1 -- 172.21.15.102:0/2041138138 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f28f00019d0 con 0x7f2900090990 2021-05-25T12:08:53.059 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.012+0000 7f28f4ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:53.059 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.012+0000 7f28f4ff9700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2021-05-25T12:08:53.059 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.012+0000 7f28f4ff9700 10 monclient(hunting): dump: 2021-05-25T12:08:53.059 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:53.059 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:53.059 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:53.060 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:53.060 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:53.060 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:53.060 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:53.061 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:53.061 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:53.061 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:53.061 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.012+0000 7f28f4ff9700 1 -- 172.21.15.102:0/2041138138 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2644660509 0 0) 0x7f28e0004c10 con 0x7f2900090d70 2021-05-25T12:08:53.061 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.012+0000 7f28f4ff9700 1 -- 172.21.15.102:0/2041138138 >> v1:172.21.15.102:6790/0 conn(0x7f290012e330 legacy=0x7f2900142ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:53.062 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.012+0000 7f28f4ff9700 1 -- 172.21.15.102:0/2041138138 >> v1:172.21.15.181:6789/0 conn(0x7f2900090990 legacy=0x7f290013ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:53.062 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.012+0000 7f28f4ff9700 10 monclient: _finish_hunting 0 2021-05-25T12:08:53.062 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.012+0000 7f28f4ff9700 1 monclient: found mon.a 2021-05-25T12:08:53.062 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.012+0000 7f28f4ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:08:53.062 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.012+0000 7f28f4ff9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:08:53.063 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.012+0000 7f28f4ff9700 1 -- 172.21.15.102:0/2041138138 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2900066750 con 0x7f2900090d70 2021-05-25T12:08:53.063 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.012+0000 7f28f4ff9700 10 monclient: _finish_auth 0 2021-05-25T12:08:53.063 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.012+0000 7f28f4ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:08:53.063 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.012+0000 7f2906ff0700 5 monclient: authenticate success, global_id 264132 2021-05-25T12:08:53.063 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.012+0000 7f2906ff0700 10 monclient: _renew_subs 2021-05-25T12:08:53.064 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.012+0000 7f2906ff0700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:08:53.064 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.012+0000 7f2906ff0700 1 -- 172.21.15.102:0/2041138138 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f290013b2a0 con 0x7f2900090d70 2021-05-25T12:08:53.064 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.012+0000 7f2906ff0700 10 client.264132.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:08:53.064 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.012+0000 7f2906ff0700 10 monclient: _renew_subs 2021-05-25T12:08:53.065 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.012+0000 7f2906ff0700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:08:53.065 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.012+0000 7f2906ff0700 1 -- 172.21.15.102:0/2041138138 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2900126d60 con 0x7f2900090d70 2021-05-25T12:08:53.065 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.012+0000 7f2906ff0700 1 librados: init done 2021-05-25T12:08:53.065 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.012+0000 7f28f4ff9700 1 -- 172.21.15.102:0/2041138138 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f28e8001a60 con 0x7f2900090d70 2021-05-25T12:08:53.065 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.012+0000 7f28f4ff9700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:08:53.066 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.012+0000 7f28f4ff9700 1 -- 172.21.15.102:0/2041138138 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f28e8001d70 con 0x7f2900090d70 2021-05-25T12:08:53.066 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.012+0000 7f28f4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:53.066 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.012+0000 7f28f4ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:08:53.066 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.012+0000 7f28f6ffd700 4 set_mon_vals no callback set 2021-05-25T12:08:53.066 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.012+0000 7f28f4ff9700 10 monclient: dump: 2021-05-25T12:08:53.067 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:53.067 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:53.067 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:53.067 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:53.067 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:53.068 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:53.068 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:53.068 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:53.068 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:53.068 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:53.069 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.012+0000 7f28f4ff9700 1 -- 172.21.15.102:0/2041138138 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f28e8001f90 con 0x7f2900090d70 2021-05-25T12:08:53.069 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.013+0000 7f2906ff0700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:08:53.069 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.013+0000 7f2906ff0700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:08:53.069 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.013+0000 7f2906ff0700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:08:53.070 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.013+0000 7f2906ff0700 1 -- 172.21.15.102:0/2041138138 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28d8004db0 con 0x7f2900090d70 2021-05-25T12:08:53.070 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.013+0000 7f28f4ff9700 1 -- 172.21.15.102:0/2041138138 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(295..295 src has 250..295) v4 ==== 5568+0+0 (unknown 2864237858 0 0) 0x7f28e803d470 con 0x7f2900090d70 2021-05-25T12:08:53.070 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.013+0000 7f28f4ff9700 10 client.264132.objecter ms_dispatch 0x7f2900001550 osd_map(295..295 src has 250..295) v4 2021-05-25T12:08:53.070 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.013+0000 7f28f4ff9700 3 client.264132.objecter handle_osd_map got epochs [295,295] > 0 2021-05-25T12:08:53.070 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.013+0000 7f28f4ff9700 3 client.264132.objecter handle_osd_map decoding full epoch 295 2021-05-25T12:08:53.071 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.013+0000 7f28f4ff9700 20 client.264132.objecter dump_active .. 0 homeless 2021-05-25T12:08:53.071 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.013+0000 7f2904d8c700 10 monclient: get_auth_request con 0x7f28e00334b0 auth_method 0 2021-05-25T12:08:53.071 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.013+0000 7f28f4ff9700 10 client.264132.objecter ms_handle_connect 0x7f28e00334b0 2021-05-25T12:08:53.071 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.018+0000 7f28f4ff9700 1 -- 172.21.15.102:0/2041138138 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7f28e8012c30 con 0x7f2900090d70 2021-05-25T12:08:53.071 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.018+0000 7f28f4ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:08:53.072 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.018+0000 7f28f4ff9700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:08:53.230 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.227+0000 7f2906ff0700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2021-05-25T12:08:53.230 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.227+0000 7f2906ff0700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2021-05-25T12:08:53.230 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.227+0000 7f2906ff0700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:08:53.231 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.227+0000 7f2906ff0700 1 -- 172.21.15.102:0/2041138138 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f28d80055e0 con 0x7f2900090d70 2021-05-25T12:08:53.231 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.227+0000 7f28f4ff9700 1 -- 172.21.15.102:0/2041138138 <== mon.0 v1:172.21.15.102:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1342 (unknown 254746657 0 1300041434) 0x7f28e8012820 con 0x7f2900090d70 2021-05-25T12:08:53.231 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.227+0000 7f28f4ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2021-05-25T12:08:53.232 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.227+0000 7f28f4ff9700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:08:53.232 INFO:teuthology.orchestra.run.smithi102.stdout:{"election_epoch":74,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"b","quorum_age":2,"features":{"quorum_con":"4540138303579357183","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"20ee91b6-8935-4b98-b868-85c03789ccc4","modified":"2021-05-25T11:59:11.011522Z","created":"2021-05-25T11:59:11.011522Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6790","nonce":0}]},"addr":"172.21.15.102:6790/0","public_addr":"172.21.15.102:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2021-05-25T12:08:53.234 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.229+0000 7f2906ff0700 10 librados: watch_flush enter 2021-05-25T12:08:53.234 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.229+0000 7f2906ff0700 10 librados: watch_flush exit 2021-05-25T12:08:53.234 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.229+0000 7f2906ff0700 20 client.264132.objecter shutdown clearing up homeless session... 2021-05-25T12:08:53.235 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.229+0000 7f2906ff0700 10 client.264132.objecter successfully canceled tick 2021-05-25T12:08:53.235 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.229+0000 7f2906ff0700 1 -- 172.21.15.102:0/2041138138 >> v1:172.21.15.181:6812/29582 conn(0x7f28e00334b0 legacy=0x7f28e0035970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:53.235 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.229+0000 7f2906ff0700 10 monclient: shutdown 2021-05-25T12:08:53.235 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.229+0000 7f2906ff0700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:08:53.236 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.229+0000 7f2906ff0700 1 -- 172.21.15.102:0/2041138138 >> v1:172.21.15.102:6789/0 conn(0x7f2900090d70 legacy=0x7f2900126490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:53.236 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.229+0000 7f2906ff0700 1 -- 172.21.15.102:0/2041138138 shutdown_connections 2021-05-25T12:08:53.236 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.229+0000 7f2906ff0700 1 -- 172.21.15.102:0/2041138138 >> 172.21.15.102:0/2041138138 conn(0x7f290009b370 msgr2=0x7f29001292f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:08:53.236 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.229+0000 7f2906ff0700 1 -- 172.21.15.102:0/2041138138 shutdown_connections 2021-05-25T12:08:53.236 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.229+0000 7f2906ff0700 1 -- 172.21.15.102:0/2041138138 wait complete. 2021-05-25T12:08:53.237 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.229+0000 7f2906ff0700 1 librados: shutdown 2021-05-25T12:08:53.243 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2021-05-25T12:08:53.243 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2021-05-25T12:08:53.243 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2021-05-25T12:08:53.353 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.350+0000 7fd75b4e9700 10 monclient: get_monmap_and_config 2021-05-25T12:08:53.353 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.350+0000 7fd75b4e9700 10 monclient: build_initial_monmap 2021-05-25T12:08:53.353 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.350+0000 7fd75b4e9700 10 monclient: monmap: 2021-05-25T12:08:53.354 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:08:53.354 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:53.354 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:08:53.352621+0000 2021-05-25T12:08:53.355 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:08:53.352621+0000 2021-05-25T12:08:53.355 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:08:53.355 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:53.355 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:08:53.355 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:08:53.356 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:08:53.356 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:53.372 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.351+0000 7fd75b4e9700 1 Processor -- start 2021-05-25T12:08:53.373 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.351+0000 7fd75b4e9700 1 -- start start 2021-05-25T12:08:53.373 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.351+0000 7fd75b4e9700 10 monclient: init 2021-05-25T12:08:53.373 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.351+0000 7fd75b4e9700 10 monclient: _reopen_session rank -1 2021-05-25T12:08:53.373 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.351+0000 7fd75b4e9700 10 monclient: _add_conns ranks=[1,0,2] 2021-05-25T12:08:53.374 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.351+0000 7fd75b4e9700 10 monclient(hunting): picked mon.noname-c con 0x7fd754069190 addr v1:172.21.15.181:6789/0 2021-05-25T12:08:53.374 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.351+0000 7fd75b4e9700 10 monclient(hunting): picked mon.noname-a con 0x7fd754069c80 addr v1:172.21.15.102:6789/0 2021-05-25T12:08:53.374 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.351+0000 7fd75b4e9700 10 monclient(hunting): picked mon.noname-b con 0x7fd754096ab0 addr v1:172.21.15.102:6790/0 2021-05-25T12:08:53.374 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.351+0000 7fd75b4e9700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7540959b0 con 0x7fd754069c80 2021-05-25T12:08:53.375 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.351+0000 7fd75b4e9700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7540914e0 con 0x7fd754069190 2021-05-25T12:08:53.375 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.351+0000 7fd75b4e9700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd754095c10 con 0x7fd754096ab0 2021-05-25T12:08:53.375 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.351+0000 7fd75b4e9700 10 monclient(hunting): _renew_subs 2021-05-25T12:08:53.375 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.351+0000 7fd75b4e9700 10 monclient(hunting): authenticate will time out at 1333.949989s 2021-05-25T12:08:53.375 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.353+0000 7fd759a86700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fd754096ab0 0x7fd754134560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:57574/0 (socket says 172.21.15.102:57574) 2021-05-25T12:08:53.376 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.353+0000 7fd758a84700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fd754069c80 0x7fd754130e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:34462/0 (socket says 172.21.15.102:34462) 2021-05-25T12:08:53.376 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.353+0000 7fd759a86700 1 -- 172.21.15.102:0/3583346092 learned_addr learned my addr 172.21.15.102:0/3583346092 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:08:53.376 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.353+0000 7fd743fff700 1 -- 172.21.15.102:0/3583346092 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2371738121 0 0) 0x7fd7540959b0 con 0x7fd754069c80 2021-05-25T12:08:53.376 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.353+0000 7fd743fff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:53.377 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.353+0000 7fd743fff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:53.377 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.353+0000 7fd743fff700 10 monclient(hunting): my global_id is 264135 2021-05-25T12:08:53.377 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.353+0000 7fd743fff700 1 -- 172.21.15.102:0/3583346092 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd73c003270 con 0x7fd754069c80 2021-05-25T12:08:53.377 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.353+0000 7fd743fff700 1 -- 172.21.15.102:0/3583346092 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2671899023 0 0) 0x7fd754095c10 con 0x7fd754096ab0 2021-05-25T12:08:53.378 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.353+0000 7fd743fff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:53.378 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.353+0000 7fd743fff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:53.378 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.353+0000 7fd743fff700 10 monclient(hunting): my global_id is 244250 2021-05-25T12:08:53.378 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.353+0000 7fd743fff700 1 -- 172.21.15.102:0/3583346092 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7540959b0 con 0x7fd754096ab0 2021-05-25T12:08:53.379 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.353+0000 7fd743fff700 1 -- 172.21.15.102:0/3583346092 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1207530730 0 0) 0x7fd7540914e0 con 0x7fd754069190 2021-05-25T12:08:53.379 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.353+0000 7fd743fff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:53.379 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.353+0000 7fd743fff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:53.379 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.353+0000 7fd743fff700 10 monclient(hunting): my global_id is 224512 2021-05-25T12:08:53.379 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.353+0000 7fd743fff700 1 -- 172.21.15.102:0/3583346092 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd754095c10 con 0x7fd754069190 2021-05-25T12:08:53.380 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.353+0000 7fd743fff700 1 -- 172.21.15.102:0/3583346092 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1007143813 0 0) 0x7fd73c003270 con 0x7fd754069c80 2021-05-25T12:08:53.380 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.353+0000 7fd743fff700 1 -- 172.21.15.102:0/3583346092 >> v1:172.21.15.102:6790/0 conn(0x7fd754096ab0 legacy=0x7fd754134560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:53.380 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.353+0000 7fd743fff700 1 -- 172.21.15.102:0/3583346092 >> v1:172.21.15.181:6789/0 conn(0x7fd754069190 legacy=0x7fd754069570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:53.380 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.353+0000 7fd743fff700 10 monclient: _finish_hunting 0 2021-05-25T12:08:53.381 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.353+0000 7fd743fff700 1 monclient: found mon.noname-a 2021-05-25T12:08:53.381 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.353+0000 7fd743fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:08:53.381 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.353+0000 7fd743fff700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2021-05-25T12:08:53.381 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.353+0000 7fd743fff700 1 -- 172.21.15.102:0/3583346092 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd754095eb0 con 0x7fd754069c80 2021-05-25T12:08:53.382 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.353+0000 7fd743fff700 10 monclient: _finish_auth 0 2021-05-25T12:08:53.382 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.353+0000 7fd743fff700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:08:53.382 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.353+0000 7fd75b4e9700 5 monclient: authenticate success, global_id 264135 2021-05-25T12:08:53.382 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.353+0000 7fd743fff700 1 -- 172.21.15.102:0/3583346092 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fd748001610 con 0x7fd754069c80 2021-05-25T12:08:53.383 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.353+0000 7fd743fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:53.383 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.353+0000 7fd743fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:08:53.383 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.353+0000 7fd743fff700 10 monclient: dump: 2021-05-25T12:08:53.383 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:53.383 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:53.384 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:53.384 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:53.384 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:53.384 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:53.385 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:53.385 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:53.385 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:53.385 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:53.386 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.353+0000 7fd75b4e9700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:08:53.386 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.353+0000 7fd743fff700 1 -- 172.21.15.102:0/3583346092 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd748001870 con 0x7fd754069c80 2021-05-25T12:08:53.386 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.353+0000 7fd743fff700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:08:53.386 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.353+0000 7fd743fff700 1 -- 172.21.15.102:0/3583346092 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fd748001c00 con 0x7fd754069c80 2021-05-25T12:08:53.387 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.353+0000 7fd75b4e9700 10 monclient: get_monmap_and_config success 2021-05-25T12:08:53.387 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.353+0000 7fd75b4e9700 4 set_mon_vals no callback set 2021-05-25T12:08:53.387 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.353+0000 7fd743fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:53.387 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.353+0000 7fd743fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:08:53.388 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.353+0000 7fd743fff700 10 monclient: dump: 2021-05-25T12:08:53.388 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:53.388 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:53.388 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:53.389 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:53.389 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:53.389 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:53.389 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:53.389 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:53.390 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:53.390 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:53.390 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.354+0000 7fd75b4e9700 10 monclient: shutdown 2021-05-25T12:08:53.390 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.354+0000 7fd75b4e9700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:08:53.391 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.354+0000 7fd75b4e9700 1 -- 172.21.15.102:0/3583346092 >> v1:172.21.15.102:6789/0 conn(0x7fd754069c80 legacy=0x7fd754130e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:53.391 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.354+0000 7fd75b4e9700 1 -- 172.21.15.102:0/3583346092 shutdown_connections 2021-05-25T12:08:53.391 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.354+0000 7fd75b4e9700 1 -- 172.21.15.102:0/3583346092 >> 172.21.15.102:0/3583346092 conn(0x7fd75409af90 msgr2=0x7fd75409b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:08:53.391 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.354+0000 7fd75b4e9700 1 -- 172.21.15.102:0/3583346092 shutdown_connections 2021-05-25T12:08:53.392 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.354+0000 7fd75b4e9700 1 -- 172.21.15.102:0/3583346092 wait complete. 2021-05-25T12:08:53.392 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.354+0000 7fd75b4e9700 10 monclient: build_initial_monmap 2021-05-25T12:08:53.392 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.354+0000 7fd75b4e9700 10 monclient: monmap: 2021-05-25T12:08:53.392 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:08:53.393 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:08:53.393 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:08:53.393 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:08:53.393 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:08:53.393 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:08:53.394 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:08:53.394 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:08:53.394 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:08:53.394 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:53.395 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.355+0000 7fd75b4e9700 1 librados: starting msgr at 2021-05-25T12:08:53.395 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.355+0000 7fd75b4e9700 1 librados: starting objecter 2021-05-25T12:08:53.395 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.355+0000 7fd75b4e9700 1 Processor -- start 2021-05-25T12:08:53.395 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.355+0000 7fd75b4e9700 1 -- start start 2021-05-25T12:08:53.396 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.355+0000 7fd75b4e9700 1 librados: setting wanted keys 2021-05-25T12:08:53.396 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.355+0000 7fd75b4e9700 1 librados: calling monclient init 2021-05-25T12:08:53.396 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.355+0000 7fd75b4e9700 10 monclient: init 2021-05-25T12:08:53.396 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.355+0000 7fd75b4e9700 10 monclient: _reopen_session rank -1 2021-05-25T12:08:53.396 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.355+0000 7fd75b4e9700 10 monclient: _add_conns ranks=[2,1,0] 2021-05-25T12:08:53.397 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.355+0000 7fd75b4e9700 10 monclient(hunting): picked mon.noname-c con 0x7fd754069190 addr v1:172.21.15.102:6790/0 2021-05-25T12:08:53.397 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.355+0000 7fd75b4e9700 10 monclient(hunting): picked mon.noname-b con 0x7fd754069c80 addr v1:172.21.15.181:6789/0 2021-05-25T12:08:53.397 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.355+0000 7fd75b4e9700 10 monclient(hunting): picked mon.noname-a con 0x7fd754096ab0 addr v1:172.21.15.102:6789/0 2021-05-25T12:08:53.397 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.355+0000 7fd75b4e9700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd754068680 con 0x7fd754096ab0 2021-05-25T12:08:53.398 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.355+0000 7fd75b4e9700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd754066ac0 con 0x7fd754069c80 2021-05-25T12:08:53.398 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.355+0000 7fd75b4e9700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7540688b0 con 0x7fd754069190 2021-05-25T12:08:53.398 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.355+0000 7fd75b4e9700 10 monclient(hunting): _renew_subs 2021-05-25T12:08:53.398 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.355+0000 7fd75b4e9700 10 monclient(hunting): authenticate will time out at 1333.952830s 2021-05-25T12:08:53.399 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.355+0000 7fd759a86700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fd754096ab0 0x7fd754147350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:34470/0 (socket says 172.21.15.102:34470) 2021-05-25T12:08:53.399 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.355+0000 7fd759285700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fd754069190 0x7fd75412dd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:57576/0 (socket says 172.21.15.102:57576) 2021-05-25T12:08:53.399 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.355+0000 7fd759a86700 1 -- 172.21.15.102:0/247298238 learned_addr learned my addr 172.21.15.102:0/247298238 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:08:53.399 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.356+0000 7fd7417fa700 10 client.?.objecter ms_handle_connect 0x7fd754096ab0 2021-05-25T12:08:53.400 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.356+0000 7fd7417fa700 10 client.?.objecter resend_mon_ops 2021-05-25T12:08:53.400 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.356+0000 7fd7417fa700 10 client.?.objecter ms_handle_connect 0x7fd754069190 2021-05-25T12:08:53.400 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.356+0000 7fd7417fa700 10 client.?.objecter resend_mon_ops 2021-05-25T12:08:53.400 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.356+0000 7fd7417fa700 1 -- 172.21.15.102:0/247298238 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2371901563 0 0) 0x7fd754068680 con 0x7fd754096ab0 2021-05-25T12:08:53.400 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.356+0000 7fd7417fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:53.401 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.356+0000 7fd7417fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:53.401 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.356+0000 7fd7417fa700 10 monclient(hunting): my global_id is 264138 2021-05-25T12:08:53.401 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.356+0000 7fd7417fa700 1 -- 172.21.15.102:0/247298238 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd734004c10 con 0x7fd754096ab0 2021-05-25T12:08:53.401 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.356+0000 7fd7417fa700 1 -- 172.21.15.102:0/247298238 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2232351695 0 0) 0x7fd7540688b0 con 0x7fd754069190 2021-05-25T12:08:53.402 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.356+0000 7fd7417fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:53.402 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.356+0000 7fd7417fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:53.402 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.356+0000 7fd7417fa700 10 monclient(hunting): my global_id is 244253 2021-05-25T12:08:53.402 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.356+0000 7fd7417fa700 1 -- 172.21.15.102:0/247298238 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd754068680 con 0x7fd754069190 2021-05-25T12:08:53.403 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.356+0000 7fd7417fa700 10 client.?.objecter ms_handle_connect 0x7fd754069c80 2021-05-25T12:08:53.403 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.356+0000 7fd7417fa700 10 client.?.objecter resend_mon_ops 2021-05-25T12:08:53.403 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.356+0000 7fd7417fa700 1 -- 172.21.15.102:0/247298238 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 232954753 0 0) 0x7fd734004c10 con 0x7fd754096ab0 2021-05-25T12:08:53.403 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.356+0000 7fd7417fa700 1 -- 172.21.15.102:0/247298238 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd7540688b0 con 0x7fd754096ab0 2021-05-25T12:08:53.403 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.356+0000 7fd7417fa700 1 -- 172.21.15.102:0/247298238 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fd750001cd0 con 0x7fd754096ab0 2021-05-25T12:08:53.404 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.356+0000 7fd7417fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:53.404 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.356+0000 7fd7417fa700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:08:53.404 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.356+0000 7fd7417fa700 10 monclient(hunting): dump: 2021-05-25T12:08:53.404 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:53.405 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:53.405 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:53.405 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:53.405 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:53.406 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:53.406 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:53.406 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:53.406 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:53.406 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:53.407 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.356+0000 7fd7417fa700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:08:53.407 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.356+0000 7fd7417fa700 1 -- 172.21.15.102:0/247298238 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3062460211 0 0) 0x7fd754068680 con 0x7fd754069190 2021-05-25T12:08:53.407 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.356+0000 7fd7417fa700 1 -- 172.21.15.102:0/247298238 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd734004c10 con 0x7fd754069190 2021-05-25T12:08:53.407 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.356+0000 7fd7417fa700 1 -- 172.21.15.102:0/247298238 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 887914098 0 0) 0x7fd754066ac0 con 0x7fd754069c80 2021-05-25T12:08:53.408 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.356+0000 7fd7417fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:53.408 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.356+0000 7fd7417fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:53.408 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.356+0000 7fd7417fa700 10 monclient(hunting): my global_id is 224515 2021-05-25T12:08:53.408 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.356+0000 7fd7417fa700 1 -- 172.21.15.102:0/247298238 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd754068680 con 0x7fd754069c80 2021-05-25T12:08:53.409 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.356+0000 7fd7417fa700 1 -- 172.21.15.102:0/247298238 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fd744002970 con 0x7fd754069190 2021-05-25T12:08:53.409 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.356+0000 7fd7417fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:53.409 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.356+0000 7fd7417fa700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2021-05-25T12:08:53.409 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.356+0000 7fd7417fa700 10 monclient(hunting): dump: 2021-05-25T12:08:53.409 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:53.410 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:53.410 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:53.410 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:53.410 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:53.411 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:53.411 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:53.411 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:53.411 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:53.412 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:53.412 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.356+0000 7fd7417fa700 1 -- 172.21.15.102:0/247298238 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1343726550 0 0) 0x7fd7540688b0 con 0x7fd754096ab0 2021-05-25T12:08:53.412 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.356+0000 7fd7417fa700 1 -- 172.21.15.102:0/247298238 >> v1:172.21.15.102:6790/0 conn(0x7fd754069190 legacy=0x7fd75412dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:53.412 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.356+0000 7fd7417fa700 1 -- 172.21.15.102:0/247298238 >> v1:172.21.15.181:6789/0 conn(0x7fd754069c80 legacy=0x7fd75413f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:53.412 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.356+0000 7fd7417fa700 10 monclient: _finish_hunting 0 2021-05-25T12:08:53.413 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.356+0000 7fd7417fa700 1 monclient: found mon.a 2021-05-25T12:08:53.413 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.356+0000 7fd7417fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:08:53.413 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.356+0000 7fd7417fa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:08:53.413 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.356+0000 7fd7417fa700 1 -- 172.21.15.102:0/247298238 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd754068b50 con 0x7fd754096ab0 2021-05-25T12:08:53.414 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.356+0000 7fd7417fa700 10 monclient: _finish_auth 0 2021-05-25T12:08:53.414 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.356+0000 7fd7417fa700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:08:53.414 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.356+0000 7fd75b4e9700 5 monclient: authenticate success, global_id 264138 2021-05-25T12:08:53.414 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.356+0000 7fd75b4e9700 10 monclient: _renew_subs 2021-05-25T12:08:53.415 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.356+0000 7fd75b4e9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:08:53.415 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.356+0000 7fd75b4e9700 1 -- 172.21.15.102:0/247298238 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd754097140 con 0x7fd754096ab0 2021-05-25T12:08:53.415 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.356+0000 7fd75b4e9700 10 client.264138.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:08:53.415 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.356+0000 7fd75b4e9700 10 monclient: _renew_subs 2021-05-25T12:08:53.416 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.356+0000 7fd75b4e9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:08:53.416 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.356+0000 7fd75b4e9700 1 -- 172.21.15.102:0/247298238 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd75413f710 con 0x7fd754096ab0 2021-05-25T12:08:53.416 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.356+0000 7fd75b4e9700 1 librados: init done 2021-05-25T12:08:53.416 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.356+0000 7fd7417fa700 1 -- 172.21.15.102:0/247298238 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd7500048b0 con 0x7fd754096ab0 2021-05-25T12:08:53.417 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.356+0000 7fd7417fa700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:08:53.417 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.356+0000 7fd7417fa700 1 -- 172.21.15.102:0/247298238 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fd750004bc0 con 0x7fd754096ab0 2021-05-25T12:08:53.417 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.356+0000 7fd7417fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:53.417 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.356+0000 7fd7417fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:08:53.417 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.356+0000 7fd7437fe700 4 set_mon_vals no callback set 2021-05-25T12:08:53.418 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.356+0000 7fd7417fa700 10 monclient: dump: 2021-05-25T12:08:53.418 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:53.418 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:53.418 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:53.419 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:53.419 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:53.419 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:53.419 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:53.419 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:53.420 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:53.420 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:53.420 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.357+0000 7fd7417fa700 1 -- 172.21.15.102:0/247298238 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7fd75000df60 con 0x7fd754096ab0 2021-05-25T12:08:53.420 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.357+0000 7fd75b4e9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:08:53.421 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.357+0000 7fd75b4e9700 1 -- 172.21.15.102:0/247298238 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fd72c0065a0 con 0x7fd72c002840 2021-05-25T12:08:53.421 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.357+0000 7fd7417fa700 1 -- 172.21.15.102:0/247298238 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(295..295 src has 250..295) v4 ==== 5568+0+0 (unknown 2864237858 0 0) 0x7fd75003da00 con 0x7fd754096ab0 2021-05-25T12:08:53.421 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.357+0000 7fd7417fa700 10 client.264138.objecter ms_dispatch 0x7fd754001550 osd_map(295..295 src has 250..295) v4 2021-05-25T12:08:53.421 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.357+0000 7fd7417fa700 3 client.264138.objecter handle_osd_map got epochs [295,295] > 0 2021-05-25T12:08:53.421 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.357+0000 7fd7417fa700 3 client.264138.objecter handle_osd_map decoding full epoch 295 2021-05-25T12:08:53.422 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.357+0000 7fd7417fa700 20 client.264138.objecter dump_active .. 0 homeless 2021-05-25T12:08:53.422 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.357+0000 7fd7417fa700 10 client.264138.objecter ms_handle_connect 0x7fd72c002840 2021-05-25T12:08:53.422 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.357+0000 7fd7417fa700 10 client.264138.objecter resend_mon_ops 2021-05-25T12:08:53.422 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.357+0000 7fd7417fa700 1 -- 172.21.15.102:0/247298238 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 876539034 0 0) 0x7fd72c0065a0 con 0x7fd72c002840 2021-05-25T12:08:53.423 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.357+0000 7fd7417fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:53.423 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.357+0000 7fd7417fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:53.423 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.357+0000 7fd7417fa700 10 monclient(hunting): my global_id is 264141 2021-05-25T12:08:53.423 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.357+0000 7fd7417fa700 1 -- 172.21.15.102:0/247298238 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7540688b0 con 0x7fd72c002840 2021-05-25T12:08:53.424 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.357+0000 7fd758a84700 10 monclient: get_auth_request con 0x7fd734033460 auth_method 0 2021-05-25T12:08:53.424 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.357+0000 7fd7417fa700 1 -- 172.21.15.102:0/247298238 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4270537934 0 0) 0x7fd7540688b0 con 0x7fd72c002840 2021-05-25T12:08:53.424 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.357+0000 7fd7417fa700 1 -- 172.21.15.102:0/247298238 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd72c0065a0 con 0x7fd72c002840 2021-05-25T12:08:53.424 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.357+0000 7fd7417fa700 1 -- 172.21.15.102:0/247298238 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fd744001490 con 0x7fd72c002840 2021-05-25T12:08:53.425 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.357+0000 7fd7417fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:53.425 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.357+0000 7fd7417fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:08:53.425 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.357+0000 7fd7417fa700 10 monclient: dump: 2021-05-25T12:08:53.425 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:53.425 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:53.426 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:53.426 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:53.426 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:53.426 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:53.427 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:53.427 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:53.427 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:53.427 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:53.428 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.357+0000 7fd7417fa700 10 client.264138.objecter ms_handle_connect 0x7fd734033460 2021-05-25T12:08:53.428 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.358+0000 7fd7417fa700 1 -- 172.21.15.102:0/247298238 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1973063292 0 0) 0x7fd72c0065a0 con 0x7fd72c002840 2021-05-25T12:08:53.428 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.358+0000 7fd7417fa700 1 -- 172.21.15.102:0/247298238 --> v1:172.21.15.102:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd72c009e40 con 0x7fd72c002840 2021-05-25T12:08:53.428 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.358+0000 7fd7417fa700 1 -- 172.21.15.102:0/247298238 <== mon.0 v1:172.21.15.102:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7fd744001890 con 0x7fd72c002840 2021-05-25T12:08:53.428 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.358+0000 7fd7417fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:08:53.429 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.358+0000 7fd7417fa700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:08:53.429 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.358+0000 7fd7417fa700 1 -- 172.21.15.102:0/247298238 >> v1:172.21.15.102:6789/0 conn(0x7fd72c002840 legacy=0x7fd72c004ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:53.429 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.358+0000 7fd7417fa700 1 -- 172.21.15.102:0/247298238 >> v1:172.21.15.102:6789/0 conn(0x7fd72c002840 legacy=0x7fd72c004ce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:08:53.429 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.364+0000 7fd75b4e9700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:08:53.430 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.364+0000 7fd75b4e9700 1 -- 172.21.15.102:0/247298238 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fd72c009ad0 con 0x7fd72c008d90 2021-05-25T12:08:53.430 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.365+0000 7fd7417fa700 10 client.264138.objecter ms_handle_connect 0x7fd72c008d90 2021-05-25T12:08:53.430 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.365+0000 7fd7417fa700 10 client.264138.objecter resend_mon_ops 2021-05-25T12:08:53.430 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.365+0000 7fd7417fa700 1 -- 172.21.15.102:0/247298238 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2658279890 0 0) 0x7fd72c009ad0 con 0x7fd72c008d90 2021-05-25T12:08:53.431 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.365+0000 7fd7417fa700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:53.431 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.365+0000 7fd7417fa700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:53.431 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.365+0000 7fd7417fa700 10 monclient(hunting): my global_id is 264144 2021-05-25T12:08:53.431 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.365+0000 7fd7417fa700 1 -- 172.21.15.102:0/247298238 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd72c0065a0 con 0x7fd72c008d90 2021-05-25T12:08:53.431 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.365+0000 7fd7417fa700 1 -- 172.21.15.102:0/247298238 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4187053942 0 0) 0x7fd72c0065a0 con 0x7fd72c008d90 2021-05-25T12:08:53.432 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.365+0000 7fd7417fa700 1 -- 172.21.15.102:0/247298238 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd72c009ad0 con 0x7fd72c008d90 2021-05-25T12:08:53.432 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.365+0000 7fd7417fa700 1 -- 172.21.15.102:0/247298238 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fd744001f50 con 0x7fd72c008d90 2021-05-25T12:08:53.432 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.365+0000 7fd7417fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:53.432 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.365+0000 7fd7417fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:08:53.433 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.365+0000 7fd7417fa700 10 monclient: dump: 2021-05-25T12:08:53.433 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:53.433 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:53.433 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:53.433 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:53.434 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:53.434 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:53.434 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:53.434 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:53.435 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:53.435 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:53.435 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.365+0000 7fd7417fa700 1 -- 172.21.15.102:0/247298238 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2610616102 0 0) 0x7fd72c009ad0 con 0x7fd72c008d90 2021-05-25T12:08:53.435 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.365+0000 7fd7417fa700 1 -- 172.21.15.102:0/247298238 --> v1:172.21.15.102:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fd72c008740 con 0x7fd72c008d90 2021-05-25T12:08:53.436 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.365+0000 7fd7417fa700 1 -- 172.21.15.102:0/247298238 <== mon.0 v1:172.21.15.102:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3770 (unknown 0 0 3702660101) 0x7fd744001db0 con 0x7fd72c008d90 2021-05-25T12:08:53.436 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.365+0000 7fd7417fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:08:53.436 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.365+0000 7fd7417fa700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:08:53.436 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.365+0000 7fd7417fa700 1 -- 172.21.15.102:0/247298238 >> v1:172.21.15.102:6789/0 conn(0x7fd72c008d90 legacy=0x7fd72c009220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:53.436 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.365+0000 7fd7417fa700 1 -- 172.21.15.102:0/247298238 >> v1:172.21.15.102:6789/0 conn(0x7fd72c008d90 legacy=0x7fd72c009220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:08:53.437 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:08:53.437 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:08:53.438 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:08:53.438 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "peon", 2021-05-25T12:08:53.438 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 74, 2021-05-25T12:08:53.438 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:08:53.439 INFO:teuthology.orchestra.run.smithi102.stdout: 0, 2021-05-25T12:08:53.439 INFO:teuthology.orchestra.run.smithi102.stdout: 1, 2021-05-25T12:08:53.439 INFO:teuthology.orchestra.run.smithi102.stdout: 2 2021-05-25T12:08:53.439 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:08:53.440 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 2, 2021-05-25T12:08:53.440 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:08:53.440 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:08:53.440 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:08:53.441 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:08:53.441 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:08:53.441 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:08:53.441 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:08:53.441 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:08:53.442 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:08:53.442 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:08:53.442 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:08:53.443 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:08:53.443 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:08:53.443 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:08:53.443 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:08:53.443 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:08:53.444 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:08:53.444 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:08:53.444 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:08:53.444 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:08:53.445 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:08:53.445 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:08:53.445 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:08:53.445 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:08:53.446 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:53.446 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:53.446 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:08:53.446 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:08:53.447 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:08:53.447 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:08:53.447 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:08:53.447 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:08:53.448 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:08:53.448 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:08:53.448 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:08:53.448 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:08:53.449 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:08:53.449 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:08:53.449 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:08:53.449 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:08:53.449 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:08:53.450 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:08:53.450 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:08:53.450 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:08:53.450 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:08:53.451 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:08:53.451 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:08:53.451 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:08:53.451 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:08:53.452 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:08:53.452 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:08:53.452 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:08:53.452 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:53.453 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:08:53.453 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:53.453 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:08:53.453 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:08:53.454 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:08:53.454 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:08:53.454 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:53.454 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:08:53.454 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:08:53.455 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:08:53.455 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:53.455 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:53.455 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:53.456 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:08:53.456 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:08:53.456 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:08:53.456 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:08:53.456 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:08:53.457 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:53.457 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:53.457 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:08:53.457 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:08:53.458 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:08:53.458 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:08:53.458 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:53.458 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:08:53.459 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:08:53.585 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:08:53.585 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:53.586 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:53.586 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:53.586 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:08:53.586 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:08:53.587 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:08:53.587 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:08:53.587 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:08:53.587 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:53.588 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:53.588 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:08:53.588 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:08:53.588 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:08:53.588 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:08:53.589 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:53.589 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:08:53.589 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:08:53.589 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:08:53.589 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:53.590 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:53.590 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:53.590 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:08:53.590 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:08:53.591 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:08:53.591 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:08:53.591 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:08:53.591 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:53.591 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:53.592 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:53.592 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:08:53.592 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:08:53.592 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:53.592 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:08:53.593 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:08:53.593 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:08:53.593 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:53.593 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:08:53.594 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:08:53.594 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:53.594 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:08:53.594 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:08:53.594 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 2 2021-05-25T12:08:53.595 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:53.595 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:53.595 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:53.595 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:08:53.596 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:08:53.598 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.365+0000 7fd75b4e9700 10 librados: watch_flush enter 2021-05-25T12:08:53.598 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.365+0000 7fd75b4e9700 10 librados: watch_flush exit 2021-05-25T12:08:53.598 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.365+0000 7fd75b4e9700 20 client.264138.objecter shutdown clearing up homeless session... 2021-05-25T12:08:53.599 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.365+0000 7fd75b4e9700 10 client.264138.objecter successfully canceled tick 2021-05-25T12:08:53.599 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.366+0000 7fd75b4e9700 1 -- 172.21.15.102:0/247298238 >> v1:172.21.15.181:6812/29582 conn(0x7fd734033460 legacy=0x7fd734035920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:53.599 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.366+0000 7fd75b4e9700 10 monclient: shutdown 2021-05-25T12:08:53.599 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.366+0000 7fd75b4e9700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:08:53.600 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.366+0000 7fd75b4e9700 1 -- 172.21.15.102:0/247298238 >> v1:172.21.15.102:6789/0 conn(0x7fd754096ab0 legacy=0x7fd754147350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:53.600 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.366+0000 7fd759a86700 1 -- 172.21.15.102:0/247298238 reap_dead start 2021-05-25T12:08:53.600 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.366+0000 7fd759a86700 1 -- 172.21.15.102:0/247298238 reap_dead start 2021-05-25T12:08:53.600 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.366+0000 7fd75b4e9700 1 -- 172.21.15.102:0/247298238 shutdown_connections 2021-05-25T12:08:53.600 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.366+0000 7fd75b4e9700 1 -- 172.21.15.102:0/247298238 >> 172.21.15.102:0/247298238 conn(0x7fd75409af90 msgr2=0x7fd754093900 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:08:53.601 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.366+0000 7fd75b4e9700 1 -- 172.21.15.102:0/247298238 shutdown_connections 2021-05-25T12:08:53.601 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.366+0000 7fd75b4e9700 1 -- 172.21.15.102:0/247298238 wait complete. 2021-05-25T12:08:53.601 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.366+0000 7fd75b4e9700 1 librados: shutdown 2021-05-25T12:08:53.602 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2021-05-25T12:08:53.715 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.714+0000 7f5dcf19e700 10 monclient: get_monmap_and_config 2021-05-25T12:08:53.716 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.714+0000 7f5dcf19e700 10 monclient: build_initial_monmap 2021-05-25T12:08:53.716 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.714+0000 7f5dcf19e700 10 monclient: monmap: 2021-05-25T12:08:53.716 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:08:53.717 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:53.717 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:08:53.715483+0000 2021-05-25T12:08:53.717 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:08:53.715483+0000 2021-05-25T12:08:53.717 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:08:53.717 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:53.718 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:08:53.718 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:08:53.718 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:08:53.718 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:53.732 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.715+0000 7f5dcf19e700 1 Processor -- start 2021-05-25T12:08:53.732 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.715+0000 7f5dcf19e700 1 -- start start 2021-05-25T12:08:53.732 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.715+0000 7f5dcf19e700 10 monclient: init 2021-05-25T12:08:53.732 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.715+0000 7f5dcf19e700 10 monclient: _reopen_session rank -1 2021-05-25T12:08:53.733 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.715+0000 7f5dcf19e700 10 monclient: _add_conns ranks=[1,0,2] 2021-05-25T12:08:53.733 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.715+0000 7f5dcf19e700 10 monclient(hunting): picked mon.noname-c con 0x7f5dc8090990 addr v1:172.21.15.181:6789/0 2021-05-25T12:08:53.733 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.715+0000 7f5dcf19e700 10 monclient(hunting): picked mon.noname-a con 0x7f5dc8090d70 addr v1:172.21.15.102:6789/0 2021-05-25T12:08:53.733 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.715+0000 7f5dcf19e700 10 monclient(hunting): picked mon.noname-b con 0x7f5dc812e330 addr v1:172.21.15.102:6790/0 2021-05-25T12:08:53.734 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.715+0000 7f5dcf19e700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dc8096fa0 con 0x7f5dc8090d70 2021-05-25T12:08:53.734 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.715+0000 7f5dcf19e700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dc80926a0 con 0x7f5dc8090990 2021-05-25T12:08:53.735 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.715+0000 7f5dcf19e700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dc8092940 con 0x7f5dc812e330 2021-05-25T12:08:53.735 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.715+0000 7f5dcf19e700 10 monclient(hunting): _renew_subs 2021-05-25T12:08:53.735 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.715+0000 7f5dcf19e700 10 monclient(hunting): authenticate will time out at 1334.312847s 2021-05-25T12:08:53.736 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.715+0000 7f5dbffff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f5dc8090d70 0x7f5dc8091200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:34478/0 (socket says 172.21.15.102:34478) 2021-05-25T12:08:53.736 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.715+0000 7f5dcd73b700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f5dc812e330 0x7f5dc8130720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:57592/0 (socket says 172.21.15.102:57592) 2021-05-25T12:08:53.736 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.715+0000 7f5dbffff700 1 -- 172.21.15.102:0/4277931875 learned_addr learned my addr 172.21.15.102:0/4277931875 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:08:53.736 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.716+0000 7f5dbf7fe700 1 -- 172.21.15.102:0/4277931875 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1006423562 0 0) 0x7f5dc8092940 con 0x7f5dc812e330 2021-05-25T12:08:53.737 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.716+0000 7f5dbf7fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:53.737 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.716+0000 7f5dbf7fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:53.737 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.716+0000 7f5dbf7fe700 10 monclient(hunting): my global_id is 244262 2021-05-25T12:08:53.738 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.716+0000 7f5dbf7fe700 1 -- 172.21.15.102:0/4277931875 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5db4003270 con 0x7f5dc812e330 2021-05-25T12:08:53.738 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.716+0000 7f5dbf7fe700 1 -- 172.21.15.102:0/4277931875 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 603114630 0 0) 0x7f5dc8096fa0 con 0x7f5dc8090d70 2021-05-25T12:08:53.738 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.716+0000 7f5dbf7fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:53.738 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.716+0000 7f5dbf7fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:53.739 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.716+0000 7f5dbf7fe700 10 monclient(hunting): my global_id is 264153 2021-05-25T12:08:53.739 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.716+0000 7f5dbf7fe700 1 -- 172.21.15.102:0/4277931875 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5dc8092940 con 0x7f5dc8090d70 2021-05-25T12:08:53.739 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.716+0000 7f5dbf7fe700 1 -- 172.21.15.102:0/4277931875 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 6319956 0 0) 0x7f5db4003270 con 0x7f5dc812e330 2021-05-25T12:08:53.739 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.716+0000 7f5dbf7fe700 1 -- 172.21.15.102:0/4277931875 >> v1:172.21.15.181:6789/0 conn(0x7f5dc8090990 legacy=0x7f5dc8093240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:53.740 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.716+0000 7f5dbf7fe700 1 -- 172.21.15.102:0/4277931875 >> v1:172.21.15.102:6789/0 conn(0x7f5dc8090d70 legacy=0x7f5dc8091200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:53.740 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.716+0000 7f5dbf7fe700 10 monclient: _finish_hunting 0 2021-05-25T12:08:53.740 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.716+0000 7f5dbf7fe700 1 monclient: found mon.noname-b 2021-05-25T12:08:53.740 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.716+0000 7f5dbf7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:08:53.741 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.716+0000 7f5dbf7fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T12:08:53.741 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.716+0000 7f5dbf7fe700 1 -- 172.21.15.102:0/4277931875 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dc8092be0 con 0x7f5dc812e330 2021-05-25T12:08:53.741 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.716+0000 7f5dbf7fe700 10 monclient: _finish_auth 0 2021-05-25T12:08:53.741 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.716+0000 7f5dbf7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:08:53.741 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.716+0000 7f5dbf7fe700 1 -- 172.21.15.102:0/4277931875 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f5dc4001990 con 0x7f5dc812e330 2021-05-25T12:08:53.742 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.716+0000 7f5dcf19e700 5 monclient: authenticate success, global_id 244262 2021-05-25T12:08:53.742 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.716+0000 7f5dbf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:53.742 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.716+0000 7f5dbf7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:08:53.742 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.716+0000 7f5dbf7fe700 10 monclient: dump: 2021-05-25T12:08:53.743 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:53.743 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:53.743 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:53.744 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:53.744 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:53.744 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:53.744 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:53.744 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:53.745 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:53.745 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:53.745 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.716+0000 7f5dcf19e700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:08:53.745 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.716+0000 7f5dbf7fe700 1 -- 172.21.15.102:0/4277931875 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5dc4001b40 con 0x7f5dc812e330 2021-05-25T12:08:53.745 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.716+0000 7f5dbf7fe700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:08:53.746 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.716+0000 7f5dbf7fe700 1 -- 172.21.15.102:0/4277931875 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f5dc4001ed0 con 0x7f5dc812e330 2021-05-25T12:08:53.746 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.716+0000 7f5dcf19e700 10 monclient: get_monmap_and_config success 2021-05-25T12:08:53.746 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.716+0000 7f5dcf19e700 4 set_mon_vals no callback set 2021-05-25T12:08:53.746 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.716+0000 7f5dbf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:53.747 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.716+0000 7f5dbf7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:08:53.747 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.716+0000 7f5dbf7fe700 10 monclient: dump: 2021-05-25T12:08:53.747 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:53.747 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:53.748 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:53.748 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:53.748 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:53.748 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:53.748 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:53.749 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:53.749 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:53.749 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:53.749 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.716+0000 7f5dcf19e700 10 monclient: shutdown 2021-05-25T12:08:53.749 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.716+0000 7f5dcf19e700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:08:53.750 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.716+0000 7f5dcf19e700 1 -- 172.21.15.102:0/4277931875 >> v1:172.21.15.102:6790/0 conn(0x7f5dc812e330 legacy=0x7f5dc8130720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:53.750 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.717+0000 7f5dcf19e700 1 -- 172.21.15.102:0/4277931875 shutdown_connections 2021-05-25T12:08:53.750 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.717+0000 7f5dcf19e700 1 -- 172.21.15.102:0/4277931875 >> 172.21.15.102:0/4277931875 conn(0x7f5dc809b370 msgr2=0x7f5dc8097430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:08:53.750 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.717+0000 7f5dcf19e700 1 -- 172.21.15.102:0/4277931875 shutdown_connections 2021-05-25T12:08:53.751 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.717+0000 7f5dcf19e700 1 -- 172.21.15.102:0/4277931875 wait complete. 2021-05-25T12:08:53.751 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.717+0000 7f5dcf19e700 10 monclient: build_initial_monmap 2021-05-25T12:08:53.751 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.717+0000 7f5dcf19e700 10 monclient: monmap: 2021-05-25T12:08:53.751 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:08:53.752 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:08:53.752 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:08:53.752 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:08:53.752 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:08:53.752 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:08:53.753 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:08:53.753 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:08:53.753 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:08:53.753 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:53.754 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.717+0000 7f5dcf19e700 1 librados: starting msgr at 2021-05-25T12:08:53.754 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.717+0000 7f5dcf19e700 1 librados: starting objecter 2021-05-25T12:08:53.754 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.717+0000 7f5dcf19e700 1 Processor -- start 2021-05-25T12:08:53.754 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.718+0000 7f5dcf19e700 1 -- start start 2021-05-25T12:08:53.754 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.718+0000 7f5dcf19e700 1 librados: setting wanted keys 2021-05-25T12:08:53.755 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.718+0000 7f5dcf19e700 1 librados: calling monclient init 2021-05-25T12:08:53.755 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.718+0000 7f5dcf19e700 10 monclient: init 2021-05-25T12:08:53.755 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.718+0000 7f5dcf19e700 10 monclient: _reopen_session rank -1 2021-05-25T12:08:53.755 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.718+0000 7f5dcf19e700 10 monclient: _add_conns ranks=[1,2,0] 2021-05-25T12:08:53.756 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.718+0000 7f5dcf19e700 10 monclient(hunting): picked mon.noname-b con 0x7f5dc8090990 addr v1:172.21.15.181:6789/0 2021-05-25T12:08:53.756 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.718+0000 7f5dcf19e700 10 monclient(hunting): picked mon.noname-c con 0x7f5dc8090d70 addr v1:172.21.15.102:6790/0 2021-05-25T12:08:53.756 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.718+0000 7f5dcf19e700 10 monclient(hunting): picked mon.noname-a con 0x7f5dc812e330 addr v1:172.21.15.102:6789/0 2021-05-25T12:08:53.756 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.718+0000 7f5dcf19e700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dc8116d30 con 0x7f5dc812e330 2021-05-25T12:08:53.757 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.718+0000 7f5dcf19e700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dc8115890 con 0x7f5dc8090990 2021-05-25T12:08:53.757 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.718+0000 7f5dcf19e700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dc8118150 con 0x7f5dc8090d70 2021-05-25T12:08:53.757 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.718+0000 7f5dcf19e700 10 monclient(hunting): _renew_subs 2021-05-25T12:08:53.757 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.718+0000 7f5dcf19e700 10 monclient(hunting): authenticate will time out at 1334.315461s 2021-05-25T12:08:53.758 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.718+0000 7f5dbffff700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f5dc8090d70 0x7f5dc81400a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:57596/0 (socket says 172.21.15.102:57596) 2021-05-25T12:08:53.758 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.718+0000 7f5dbffff700 1 -- 172.21.15.102:0/3587400960 learned_addr learned my addr 172.21.15.102:0/3587400960 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:08:53.758 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.718+0000 7f5dbcff9700 10 client.?.objecter ms_handle_connect 0x7f5dc8090d70 2021-05-25T12:08:53.758 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.718+0000 7f5dbcff9700 10 client.?.objecter resend_mon_ops 2021-05-25T12:08:53.759 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.718+0000 7f5dbcff9700 10 client.?.objecter ms_handle_connect 0x7f5dc812e330 2021-05-25T12:08:53.759 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.718+0000 7f5dbcff9700 10 client.?.objecter resend_mon_ops 2021-05-25T12:08:53.759 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.718+0000 7f5dbcff9700 1 -- 172.21.15.102:0/3587400960 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2502111479 0 0) 0x7f5dc8118150 con 0x7f5dc8090d70 2021-05-25T12:08:53.759 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.718+0000 7f5dbcff9700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:53.760 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.718+0000 7f5dbcff9700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:53.760 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.718+0000 7f5dbcff9700 10 monclient(hunting): my global_id is 244265 2021-05-25T12:08:53.760 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.718+0000 7f5dbcff9700 1 -- 172.21.15.102:0/3587400960 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5da8004c10 con 0x7f5dc8090d70 2021-05-25T12:08:53.760 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.718+0000 7f5dbcff9700 10 client.?.objecter ms_handle_connect 0x7f5dc8090990 2021-05-25T12:08:53.760 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.718+0000 7f5dbcff9700 10 client.?.objecter resend_mon_ops 2021-05-25T12:08:53.761 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.718+0000 7f5dbcff9700 1 -- 172.21.15.102:0/3587400960 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 422179202 0 0) 0x7f5dc8116d30 con 0x7f5dc812e330 2021-05-25T12:08:53.761 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.718+0000 7f5dbcff9700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:53.761 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.718+0000 7f5dbcff9700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:53.761 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.718+0000 7f5dbcff9700 10 monclient(hunting): my global_id is 264156 2021-05-25T12:08:53.762 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.718+0000 7f5dbcff9700 1 -- 172.21.15.102:0/3587400960 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5dc8118150 con 0x7f5dc812e330 2021-05-25T12:08:53.762 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.718+0000 7f5dbcff9700 1 -- 172.21.15.102:0/3587400960 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 81464256 0 0) 0x7f5dc8115890 con 0x7f5dc8090990 2021-05-25T12:08:53.762 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.718+0000 7f5dbcff9700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:53.762 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.718+0000 7f5dbcff9700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:53.763 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.718+0000 7f5dbcff9700 10 monclient(hunting): my global_id is 224527 2021-05-25T12:08:53.763 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.718+0000 7f5dbcff9700 1 -- 172.21.15.102:0/3587400960 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5dc8116d30 con 0x7f5dc8090990 2021-05-25T12:08:53.763 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.719+0000 7f5dbcff9700 1 -- 172.21.15.102:0/3587400960 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1461047771 0 0) 0x7f5da8004c10 con 0x7f5dc8090d70 2021-05-25T12:08:53.763 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.719+0000 7f5dbcff9700 1 -- 172.21.15.102:0/3587400960 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5dc8115890 con 0x7f5dc8090d70 2021-05-25T12:08:53.763 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.719+0000 7f5dbcff9700 1 -- 172.21.15.102:0/3587400960 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f5db00019e0 con 0x7f5dc8090d70 2021-05-25T12:08:53.764 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.719+0000 7f5dbcff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:53.764 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.719+0000 7f5dbcff9700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:08:53.764 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.719+0000 7f5dbcff9700 10 monclient(hunting): dump: 2021-05-25T12:08:53.764 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:53.764 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:53.765 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:53.765 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:53.765 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:53.765 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:53.766 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:53.766 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:53.766 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:53.767 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:53.767 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.719+0000 7f5dbcff9700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:08:53.767 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.719+0000 7f5dbcff9700 1 -- 172.21.15.102:0/3587400960 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1399434803 0 0) 0x7f5dc8118150 con 0x7f5dc812e330 2021-05-25T12:08:53.767 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.719+0000 7f5dbcff9700 1 -- 172.21.15.102:0/3587400960 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5da8004c10 con 0x7f5dc812e330 2021-05-25T12:08:53.768 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.719+0000 7f5dbcff9700 1 -- 172.21.15.102:0/3587400960 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f5dc4001e50 con 0x7f5dc812e330 2021-05-25T12:08:53.768 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.719+0000 7f5dbcff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:53.768 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.719+0000 7f5dbcff9700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2021-05-25T12:08:53.768 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.719+0000 7f5dbcff9700 10 monclient(hunting): dump: 2021-05-25T12:08:53.768 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:53.769 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:53.769 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:53.769 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:53.769 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:53.770 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:53.770 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:53.770 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:53.770 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:53.771 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:53.771 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.719+0000 7f5dbcff9700 1 -- 172.21.15.102:0/3587400960 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1522715803 0 0) 0x7f5dc8116d30 con 0x7f5dc8090990 2021-05-25T12:08:53.771 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.719+0000 7f5dbcff9700 1 -- 172.21.15.102:0/3587400960 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5dc8118150 con 0x7f5dc8090990 2021-05-25T12:08:53.771 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.719+0000 7f5dbcff9700 1 -- 172.21.15.102:0/3587400960 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f5db80018c0 con 0x7f5dc8090990 2021-05-25T12:08:53.771 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.719+0000 7f5dbcff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:53.772 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.719+0000 7f5dbcff9700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2021-05-25T12:08:53.772 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.719+0000 7f5dbcff9700 10 monclient(hunting): dump: 2021-05-25T12:08:53.772 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:53.772 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:53.772 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:53.773 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:53.773 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:53.773 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:53.773 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:53.774 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:53.774 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:53.774 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:53.774 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.719+0000 7f5dbcff9700 1 -- 172.21.15.102:0/3587400960 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 37150397 0 0) 0x7f5dc8115890 con 0x7f5dc8090d70 2021-05-25T12:08:53.774 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.719+0000 7f5dbcff9700 1 -- 172.21.15.102:0/3587400960 >> v1:172.21.15.181:6789/0 conn(0x7f5dc8090990 legacy=0x7f5dc8129700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:53.775 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.719+0000 7f5dbcff9700 1 -- 172.21.15.102:0/3587400960 >> v1:172.21.15.102:6789/0 conn(0x7f5dc812e330 legacy=0x7f5dc81437d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:53.775 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.719+0000 7f5dbcff9700 10 monclient: _finish_hunting 0 2021-05-25T12:08:53.775 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.719+0000 7f5dbcff9700 1 monclient: found mon.c 2021-05-25T12:08:53.776 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.719+0000 7f5dbcff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:08:53.776 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.719+0000 7f5dbcff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:08:53.776 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.719+0000 7f5dbcff9700 1 -- 172.21.15.102:0/3587400960 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dc8091fe0 con 0x7f5dc8090d70 2021-05-25T12:08:53.776 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.719+0000 7f5dbcff9700 10 monclient: _finish_auth 0 2021-05-25T12:08:53.777 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.719+0000 7f5dbcff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:08:53.777 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.719+0000 7f5dcf19e700 5 monclient: authenticate success, global_id 244265 2021-05-25T12:08:53.777 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.719+0000 7f5dcf19e700 10 monclient: _renew_subs 2021-05-25T12:08:53.777 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.719+0000 7f5dcf19e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:08:53.777 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.719+0000 7f5dcf19e700 1 -- 172.21.15.102:0/3587400960 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dc812a230 con 0x7f5dc8090d70 2021-05-25T12:08:53.778 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.719+0000 7f5dcf19e700 10 client.244265.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:08:53.778 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.719+0000 7f5dcf19e700 10 monclient: _renew_subs 2021-05-25T12:08:53.778 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.719+0000 7f5dcf19e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:08:53.778 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.719+0000 7f5dcf19e700 1 -- 172.21.15.102:0/3587400960 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dc8129e90 con 0x7f5dc8090d70 2021-05-25T12:08:53.779 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.719+0000 7f5dcf19e700 1 librados: init done 2021-05-25T12:08:53.779 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.719+0000 7f5dbcff9700 1 -- 172.21.15.102:0/3587400960 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5db0001ce0 con 0x7f5dc8090d70 2021-05-25T12:08:53.779 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.719+0000 7f5dbcff9700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:08:53.779 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.719+0000 7f5dbcff9700 1 -- 172.21.15.102:0/3587400960 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f5db0004980 con 0x7f5dc8090d70 2021-05-25T12:08:53.779 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.719+0000 7f5dbcff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:53.780 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.719+0000 7f5dbeffd700 4 set_mon_vals no callback set 2021-05-25T12:08:53.780 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.719+0000 7f5dbcff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:08:53.780 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.719+0000 7f5dbcff9700 10 monclient: dump: 2021-05-25T12:08:53.780 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:53.781 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:53.781 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:53.781 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:53.781 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:53.781 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:53.782 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:53.782 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:53.782 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:53.783 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:53.783 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.719+0000 7f5dcf19e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:08:53.783 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.719+0000 7f5dbcff9700 1 -- 172.21.15.102:0/3587400960 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f5db000dca0 con 0x7f5dc8090d70 2021-05-25T12:08:53.783 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.719+0000 7f5dcf19e700 1 -- 172.21.15.102:0/3587400960 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f5da00065a0 con 0x7f5da0002840 2021-05-25T12:08:53.784 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.720+0000 7f5dbcff9700 1 -- 172.21.15.102:0/3587400960 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(296..296 src has 250..296) v4 ==== 5459+0+0 (unknown 1521303069 0 0) 0x7f5db003d570 con 0x7f5dc8090d70 2021-05-25T12:08:53.784 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.720+0000 7f5dbcff9700 10 client.244265.objecter ms_dispatch 0x7f5dc8001550 osd_map(296..296 src has 250..296) v4 2021-05-25T12:08:53.784 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.720+0000 7f5dbcff9700 3 client.244265.objecter handle_osd_map got epochs [296,296] > 0 2021-05-25T12:08:53.785 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.720+0000 7f5dbcff9700 3 client.244265.objecter handle_osd_map decoding full epoch 296 2021-05-25T12:08:53.785 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.720+0000 7f5dbcff9700 20 client.244265.objecter dump_active .. 0 homeless 2021-05-25T12:08:53.785 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.720+0000 7f5dbcff9700 10 client.244265.objecter ms_handle_connect 0x7f5da0002840 2021-05-25T12:08:53.785 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.720+0000 7f5dbcff9700 10 client.244265.objecter resend_mon_ops 2021-05-25T12:08:53.786 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.720+0000 7f5dbcff9700 1 -- 172.21.15.102:0/3587400960 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3459978755 0 0) 0x7f5da00065a0 con 0x7f5da0002840 2021-05-25T12:08:53.786 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.720+0000 7f5dbcff9700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:53.786 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.720+0000 7f5dbcff9700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:53.786 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.720+0000 7f5dbcff9700 10 monclient(hunting): my global_id is 244268 2021-05-25T12:08:53.787 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.720+0000 7f5dbcff9700 1 -- 172.21.15.102:0/3587400960 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5dc8115890 con 0x7f5da0002840 2021-05-25T12:08:53.787 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.720+0000 7f5dcd73b700 10 monclient: get_auth_request con 0x7f5da8033500 auth_method 0 2021-05-25T12:08:53.787 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.720+0000 7f5dbcff9700 1 -- 172.21.15.102:0/3587400960 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 781318150 0 0) 0x7f5dc8115890 con 0x7f5da0002840 2021-05-25T12:08:53.787 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.720+0000 7f5dbcff9700 1 -- 172.21.15.102:0/3587400960 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5da00065a0 con 0x7f5da0002840 2021-05-25T12:08:53.788 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.720+0000 7f5dbcff9700 1 -- 172.21.15.102:0/3587400960 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f5db8001f00 con 0x7f5da0002840 2021-05-25T12:08:53.788 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.720+0000 7f5dbcff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:53.788 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.720+0000 7f5dbcff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:08:53.788 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.720+0000 7f5dbcff9700 10 monclient: dump: 2021-05-25T12:08:53.788 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:53.789 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:53.789 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:53.789 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:53.789 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:53.789 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:53.790 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:53.790 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:53.790 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:53.790 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:53.791 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.720+0000 7f5dbcff9700 10 client.244265.objecter ms_handle_connect 0x7f5da8033500 2021-05-25T12:08:53.791 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.720+0000 7f5dbcff9700 1 -- 172.21.15.102:0/3587400960 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2435869251 0 0) 0x7f5da00065a0 con 0x7f5da0002840 2021-05-25T12:08:53.791 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.720+0000 7f5dbcff9700 1 -- 172.21.15.102:0/3587400960 --> v1:172.21.15.102:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5da0009e40 con 0x7f5da0002840 2021-05-25T12:08:53.791 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.720+0000 7f5dbcff9700 1 -- 172.21.15.102:0/3587400960 <== mon.2 v1:172.21.15.102:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7f5db80042b0 con 0x7f5da0002840 2021-05-25T12:08:53.791 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.720+0000 7f5dbcff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:08:53.792 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.720+0000 7f5dbcff9700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:08:53.792 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.720+0000 7f5dbcff9700 1 -- 172.21.15.102:0/3587400960 >> v1:172.21.15.102:6790/0 conn(0x7f5da0002840 legacy=0x7f5da0004ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:53.792 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.720+0000 7f5dbcff9700 1 -- 172.21.15.102:0/3587400960 >> v1:172.21.15.102:6790/0 conn(0x7f5da0002840 legacy=0x7f5da0004ce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:08:53.792 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.727+0000 7f5dcf19e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:08:53.792 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.727+0000 7f5dcf19e700 1 -- 172.21.15.102:0/3587400960 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f5da0009ad0 con 0x7f5da0008d90 2021-05-25T12:08:53.793 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.727+0000 7f5dbcff9700 10 client.244265.objecter ms_handle_connect 0x7f5da0008d90 2021-05-25T12:08:53.793 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.727+0000 7f5dbcff9700 10 client.244265.objecter resend_mon_ops 2021-05-25T12:08:53.793 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.727+0000 7f5dbcff9700 1 -- 172.21.15.102:0/3587400960 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1496720466 0 0) 0x7f5da0009ad0 con 0x7f5da0008d90 2021-05-25T12:08:53.793 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.727+0000 7f5dbcff9700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:53.794 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.727+0000 7f5dbcff9700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:53.794 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.727+0000 7f5dbcff9700 10 monclient(hunting): my global_id is 244271 2021-05-25T12:08:53.794 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.727+0000 7f5dbcff9700 1 -- 172.21.15.102:0/3587400960 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5da00065a0 con 0x7f5da0008d90 2021-05-25T12:08:53.794 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.727+0000 7f5dbcff9700 1 -- 172.21.15.102:0/3587400960 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 718267883 0 0) 0x7f5da00065a0 con 0x7f5da0008d90 2021-05-25T12:08:53.795 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.727+0000 7f5dbcff9700 1 -- 172.21.15.102:0/3587400960 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5da0009ad0 con 0x7f5da0008d90 2021-05-25T12:08:53.795 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.727+0000 7f5dbcff9700 1 -- 172.21.15.102:0/3587400960 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f5db80026b0 con 0x7f5da0008d90 2021-05-25T12:08:53.795 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.727+0000 7f5dbcff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:53.795 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.727+0000 7f5dbcff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:08:53.796 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.727+0000 7f5dbcff9700 10 monclient: dump: 2021-05-25T12:08:53.796 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:53.796 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:53.796 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:53.797 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:53.797 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:53.797 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:53.797 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:53.797 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:53.797 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:53.798 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:53.798 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.727+0000 7f5dbcff9700 1 -- 172.21.15.102:0/3587400960 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2293922387 0 0) 0x7f5da0009ad0 con 0x7f5da0008d90 2021-05-25T12:08:53.798 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.727+0000 7f5dbcff9700 1 -- 172.21.15.102:0/3587400960 --> v1:172.21.15.102:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f5da0008740 con 0x7f5da0008d90 2021-05-25T12:08:53.798 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.728+0000 7f5dbcff9700 1 -- 172.21.15.102:0/3587400960 <== mon.2 v1:172.21.15.102:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3940 (unknown 0 0 1822413503) 0x7f5db8003a00 con 0x7f5da0008d90 2021-05-25T12:08:53.799 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.728+0000 7f5dbcff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:08:53.799 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.728+0000 7f5dbcff9700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:08:53.799 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.728+0000 7f5dbcff9700 1 -- 172.21.15.102:0/3587400960 >> v1:172.21.15.102:6790/0 conn(0x7f5da0008d90 legacy=0x7f5da0009220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:53.799 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.728+0000 7f5dbcff9700 1 -- 172.21.15.102:0/3587400960 >> v1:172.21.15.102:6790/0 conn(0x7f5da0008d90 legacy=0x7f5da0009220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:08:53.800 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:08:53.800 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:08:53.800 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:08:53.801 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "peon", 2021-05-25T12:08:53.801 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 74, 2021-05-25T12:08:53.801 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:08:53.801 INFO:teuthology.orchestra.run.smithi102.stdout: 0, 2021-05-25T12:08:53.802 INFO:teuthology.orchestra.run.smithi102.stdout: 1, 2021-05-25T12:08:53.802 INFO:teuthology.orchestra.run.smithi102.stdout: 2 2021-05-25T12:08:53.802 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:08:53.802 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 3, 2021-05-25T12:08:53.802 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:08:53.803 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:08:53.803 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:08:53.804 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:08:53.804 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:08:53.804 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:08:53.805 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:08:53.805 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:08:53.805 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:08:53.805 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:08:53.805 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:08:53.806 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:08:53.806 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:08:53.806 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:08:53.807 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:08:53.807 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:08:53.807 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:08:53.807 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:08:53.807 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:08:53.808 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:08:53.808 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:08:53.808 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:08:53.808 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:08:53.809 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:08:53.809 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:53.809 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:53.810 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:08:53.810 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:08:53.810 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:08:53.811 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:08:53.811 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:08:53.811 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:08:53.811 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:08:53.812 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:08:53.812 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:08:53.812 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:08:53.812 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:08:53.812 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:08:53.813 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:08:53.813 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:08:53.813 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:08:53.813 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:08:53.813 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:08:53.814 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:08:53.814 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:08:53.815 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:08:53.815 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:08:53.815 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:08:53.815 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:08:53.816 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:08:53.816 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:08:53.817 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:08:53.817 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:53.817 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:08:53.817 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:53.817 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:08:53.818 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:08:53.818 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:08:53.818 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:08:53.819 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:53.819 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:08:53.819 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:08:53.819 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:08:53.819 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:53.820 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:53.820 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:53.820 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:08:53.820 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:08:53.820 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:08:53.821 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:08:53.821 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:08:53.821 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:53.821 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:53.821 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:08:53.822 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:08:53.822 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:08:53.822 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:08:53.822 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:53.822 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:08:53.823 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:08:53.823 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:08:53.823 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:53.823 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:53.824 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:53.824 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:08:53.824 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:08:53.824 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:08:53.824 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:08:53.825 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:08:53.825 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:53.825 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:53.825 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:08:53.826 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:08:53.826 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:08:53.826 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:08:53.826 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:53.827 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:08:53.827 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:08:53.827 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:08:53.827 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:53.827 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:53.828 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:53.828 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:08:53.828 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:08:53.828 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:08:53.828 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:08:53.829 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:08:53.829 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:53.829 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:53.829 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:53.829 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:08:53.830 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:08:53.830 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:53.830 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:08:53.830 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:08:53.831 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:08:53.831 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:53.831 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:08:53.831 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": [ 2021-05-25T12:08:53.832 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:53.832 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:08:53.832 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:08:53.832 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 2 2021-05-25T12:08:53.832 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:53.833 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:08:53.833 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:08:53.833 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:53.833 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:08:53.833 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:08:53.834 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 2 2021-05-25T12:08:53.834 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:53.834 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:53.834 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:53.835 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:08:53.835 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:08:53.838 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.728+0000 7f5dcf19e700 10 librados: watch_flush enter 2021-05-25T12:08:53.838 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.728+0000 7f5dcf19e700 10 librados: watch_flush exit 2021-05-25T12:08:53.838 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.728+0000 7f5dcf19e700 20 client.244265.objecter shutdown clearing up homeless session... 2021-05-25T12:08:53.838 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.728+0000 7f5dcf19e700 10 client.244265.objecter successfully canceled tick 2021-05-25T12:08:53.839 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.728+0000 7f5dcf19e700 1 -- 172.21.15.102:0/3587400960 >> v1:172.21.15.181:6812/29582 conn(0x7f5da8033500 legacy=0x7f5da80359c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:53.839 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.728+0000 7f5dcf19e700 10 monclient: shutdown 2021-05-25T12:08:53.839 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.728+0000 7f5dcf19e700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:08:53.839 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.728+0000 7f5dcf19e700 1 -- 172.21.15.102:0/3587400960 >> v1:172.21.15.102:6790/0 conn(0x7f5dc8090d70 legacy=0x7f5dc81400a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:53.840 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.728+0000 7f5dcd73b700 1 -- 172.21.15.102:0/3587400960 reap_dead start 2021-05-25T12:08:53.840 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.728+0000 7f5dcd73b700 1 -- 172.21.15.102:0/3587400960 reap_dead start 2021-05-25T12:08:53.840 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.728+0000 7f5dcf19e700 1 -- 172.21.15.102:0/3587400960 shutdown_connections 2021-05-25T12:08:53.840 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.728+0000 7f5dcf19e700 1 -- 172.21.15.102:0/3587400960 >> 172.21.15.102:0/3587400960 conn(0x7f5dc809b370 msgr2=0x7f5dc812d8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:08:53.841 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.728+0000 7f5dcf19e700 1 -- 172.21.15.102:0/3587400960 shutdown_connections 2021-05-25T12:08:53.841 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.728+0000 7f5dcf19e700 1 -- 172.21.15.102:0/3587400960 wait complete. 2021-05-25T12:08:53.841 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.728+0000 7f5dcf19e700 1 librados: shutdown 2021-05-25T12:08:53.841 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2021-05-25T12:08:53.976 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.951+0000 7f059ab52700 10 monclient: get_monmap_and_config 2021-05-25T12:08:53.977 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.951+0000 7f059ab52700 10 monclient: build_initial_monmap 2021-05-25T12:08:53.977 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.951+0000 7f059ab52700 10 monclient: monmap: 2021-05-25T12:08:53.977 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:08:53.977 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:53.978 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:08:53.952051+0000 2021-05-25T12:08:53.978 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:08:53.952051+0000 2021-05-25T12:08:53.978 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:08:53.978 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:53.979 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:08:53.979 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:08:53.979 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:08:53.979 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:53.980 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.951+0000 7f059ab52700 1 Processor -- start 2021-05-25T12:08:53.980 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.951+0000 7f059ab52700 1 -- start start 2021-05-25T12:08:53.980 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.951+0000 7f059ab52700 10 monclient: init 2021-05-25T12:08:53.980 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.952+0000 7f059ab52700 10 monclient: _reopen_session rank -1 2021-05-25T12:08:53.980 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.952+0000 7f059ab52700 10 monclient: _add_conns ranks=[1,2,0] 2021-05-25T12:08:53.981 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.952+0000 7f059ab52700 10 monclient(hunting): picked mon.noname-c con 0x7f0594063840 addr v1:172.21.15.181:6789/0 2021-05-25T12:08:53.981 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.952+0000 7f059ab52700 10 monclient(hunting): picked mon.noname-b con 0x7f0594094da0 addr v1:172.21.15.102:6790/0 2021-05-25T12:08:53.981 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.952+0000 7f059ab52700 10 monclient(hunting): picked mon.noname-a con 0x7f05941323c0 addr v1:172.21.15.102:6789/0 2021-05-25T12:08:53.981 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.952+0000 7f059ab52700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0594063fb0 con 0x7f05941323c0 2021-05-25T12:08:53.982 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.952+0000 7f059ab52700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05940982f0 con 0x7f0594063840 2021-05-25T12:08:53.982 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.952+0000 7f059ab52700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0594064510 con 0x7f0594094da0 2021-05-25T12:08:53.982 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.952+0000 7f059ab52700 10 monclient(hunting): _renew_subs 2021-05-25T12:08:53.982 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.952+0000 7f059ab52700 10 monclient(hunting): authenticate will time out at 1334.549291s 2021-05-25T12:08:53.983 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.952+0000 7f05990ef700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f05941323c0 0x7f05941347b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:34498/0 (socket says 172.21.15.102:34498) 2021-05-25T12:08:53.983 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.952+0000 7f058bfff700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f0594094da0 0x7f0594097be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:57606/0 (socket says 172.21.15.102:57606) 2021-05-25T12:08:53.983 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.952+0000 7f05990ef700 1 -- 172.21.15.102:0/2354520017 learned_addr learned my addr 172.21.15.102:0/2354520017 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:08:53.983 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.952+0000 7f058b7fe700 1 -- 172.21.15.102:0/2354520017 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3338362975 0 0) 0x7f0594064510 con 0x7f0594094da0 2021-05-25T12:08:53.983 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.952+0000 7f058b7fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:53.984 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.952+0000 7f058b7fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:53.984 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.952+0000 7f058b7fe700 10 monclient(hunting): my global_id is 244274 2021-05-25T12:08:53.984 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.952+0000 7f058b7fe700 1 -- 172.21.15.102:0/2354520017 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0580003270 con 0x7f0594094da0 2021-05-25T12:08:53.984 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.952+0000 7f058b7fe700 1 -- 172.21.15.102:0/2354520017 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3889164059 0 0) 0x7f0594063fb0 con 0x7f05941323c0 2021-05-25T12:08:53.985 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.952+0000 7f058b7fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:53.985 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.952+0000 7f058b7fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:53.985 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.952+0000 7f058b7fe700 10 monclient(hunting): my global_id is 264159 2021-05-25T12:08:53.985 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.952+0000 7f058b7fe700 1 -- 172.21.15.102:0/2354520017 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0594064510 con 0x7f05941323c0 2021-05-25T12:08:53.986 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.952+0000 7f058b7fe700 1 -- 172.21.15.102:0/2354520017 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3896241043 0 0) 0x7f0580003270 con 0x7f0594094da0 2021-05-25T12:08:53.986 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.952+0000 7f058b7fe700 1 -- 172.21.15.102:0/2354520017 >> v1:172.21.15.181:6789/0 conn(0x7f0594063840 legacy=0x7f05940974d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:53.986 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.952+0000 7f058b7fe700 1 -- 172.21.15.102:0/2354520017 >> v1:172.21.15.102:6789/0 conn(0x7f05941323c0 legacy=0x7f05941347b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:53.986 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.952+0000 7f058b7fe700 10 monclient: _finish_hunting 0 2021-05-25T12:08:53.986 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.952+0000 7f058b7fe700 1 monclient: found mon.noname-b 2021-05-25T12:08:53.987 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.952+0000 7f058b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:08:53.987 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.952+0000 7f058b7fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T12:08:53.987 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.952+0000 7f058b7fe700 1 -- 172.21.15.102:0/2354520017 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05940647b0 con 0x7f0594094da0 2021-05-25T12:08:53.987 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.952+0000 7f058b7fe700 10 monclient: _finish_auth 0 2021-05-25T12:08:53.988 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.952+0000 7f058b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:08:53.988 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.952+0000 7f058b7fe700 1 -- 172.21.15.102:0/2354520017 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f057c001610 con 0x7f0594094da0 2021-05-25T12:08:53.988 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.952+0000 7f059ab52700 5 monclient: authenticate success, global_id 244274 2021-05-25T12:08:53.988 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.952+0000 7f058b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:53.989 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.952+0000 7f058b7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:08:53.989 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.952+0000 7f058b7fe700 10 monclient: dump: 2021-05-25T12:08:53.989 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:53.989 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:53.989 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:53.990 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:53.990 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:53.990 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:53.990 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:53.991 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:53.991 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:53.991 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:53.991 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.952+0000 7f059ab52700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:08:53.991 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.953+0000 7f058b7fe700 1 -- 172.21.15.102:0/2354520017 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f057c0017c0 con 0x7f0594094da0 2021-05-25T12:08:53.992 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.953+0000 7f058b7fe700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:08:53.992 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.953+0000 7f058b7fe700 1 -- 172.21.15.102:0/2354520017 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f057c001b50 con 0x7f0594094da0 2021-05-25T12:08:53.992 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.953+0000 7f059ab52700 10 monclient: get_monmap_and_config success 2021-05-25T12:08:53.992 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.953+0000 7f059ab52700 4 set_mon_vals no callback set 2021-05-25T12:08:53.993 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.953+0000 7f058b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:53.993 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.953+0000 7f058b7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:08:53.993 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.953+0000 7f058b7fe700 10 monclient: dump: 2021-05-25T12:08:53.993 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:53.993 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:53.994 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:53.994 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:53.994 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:53.994 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:53.995 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:53.995 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:53.995 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:53.995 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:53.995 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.953+0000 7f059ab52700 10 monclient: shutdown 2021-05-25T12:08:53.996 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.953+0000 7f059ab52700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:08:53.996 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.953+0000 7f059ab52700 1 -- 172.21.15.102:0/2354520017 >> v1:172.21.15.102:6790/0 conn(0x7f0594094da0 legacy=0x7f0594097be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:53.996 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.953+0000 7f059ab52700 1 -- 172.21.15.102:0/2354520017 shutdown_connections 2021-05-25T12:08:53.996 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.953+0000 7f059ab52700 1 -- 172.21.15.102:0/2354520017 >> 172.21.15.102:0/2354520017 conn(0x7f059409cb30 msgr2=0x7f059409cf40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:08:53.997 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.953+0000 7f059ab52700 1 -- 172.21.15.102:0/2354520017 shutdown_connections 2021-05-25T12:08:53.997 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.953+0000 7f059ab52700 1 -- 172.21.15.102:0/2354520017 wait complete. 2021-05-25T12:08:53.997 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.953+0000 7f059ab52700 10 monclient: build_initial_monmap 2021-05-25T12:08:53.997 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.953+0000 7f059ab52700 10 monclient: monmap: 2021-05-25T12:08:53.998 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:08:53.998 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:08:53.998 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:08:53.998 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:08:53.998 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:08:53.999 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:08:53.999 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:08:53.999 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:08:53.999 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:08:54.000 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:54.000 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.954+0000 7f059ab52700 1 librados: starting msgr at 2021-05-25T12:08:54.000 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.954+0000 7f059ab52700 1 librados: starting objecter 2021-05-25T12:08:54.000 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.954+0000 7f059ab52700 1 Processor -- start 2021-05-25T12:08:54.000 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.954+0000 7f059ab52700 1 -- start start 2021-05-25T12:08:54.001 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.954+0000 7f059ab52700 1 librados: setting wanted keys 2021-05-25T12:08:54.001 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.954+0000 7f059ab52700 1 librados: calling monclient init 2021-05-25T12:08:54.001 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.954+0000 7f059ab52700 10 monclient: init 2021-05-25T12:08:54.001 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.954+0000 7f059ab52700 10 monclient: _reopen_session rank -1 2021-05-25T12:08:54.002 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.954+0000 7f059ab52700 10 monclient: _add_conns ranks=[2,1,0] 2021-05-25T12:08:54.002 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.954+0000 7f059ab52700 10 monclient(hunting): picked mon.noname-c con 0x7f0594063840 addr v1:172.21.15.102:6790/0 2021-05-25T12:08:54.002 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.954+0000 7f059ab52700 10 monclient(hunting): picked mon.noname-b con 0x7f0594094da0 addr v1:172.21.15.181:6789/0 2021-05-25T12:08:54.002 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.954+0000 7f059ab52700 10 monclient(hunting): picked mon.noname-a con 0x7f05941323c0 addr v1:172.21.15.102:6789/0 2021-05-25T12:08:54.002 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.954+0000 7f059ab52700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f059409b5f0 con 0x7f05941323c0 2021-05-25T12:08:54.003 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.954+0000 7f059ab52700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f059412d620 con 0x7f0594094da0 2021-05-25T12:08:54.003 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.954+0000 7f059ab52700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f059412ee90 con 0x7f0594063840 2021-05-25T12:08:54.003 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.954+0000 7f059ab52700 10 monclient(hunting): _renew_subs 2021-05-25T12:08:54.003 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.954+0000 7f059ab52700 10 monclient(hunting): authenticate will time out at 1334.551941s 2021-05-25T12:08:54.004 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.955+0000 7f05990ef700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f05941323c0 0x7f0594143770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:34506/0 (socket says 172.21.15.102:34506) 2021-05-25T12:08:54.004 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.955+0000 7f05988ee700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f0594063840 0x7f059412c800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:57612/0 (socket says 172.21.15.102:57612) 2021-05-25T12:08:54.004 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.955+0000 7f05990ef700 1 -- 172.21.15.102:0/4001666364 learned_addr learned my addr 172.21.15.102:0/4001666364 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:08:54.004 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.955+0000 7f0588ff9700 10 client.?.objecter ms_handle_connect 0x7f05941323c0 2021-05-25T12:08:54.005 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.955+0000 7f0588ff9700 10 client.?.objecter resend_mon_ops 2021-05-25T12:08:54.005 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.955+0000 7f0588ff9700 10 client.?.objecter ms_handle_connect 0x7f0594063840 2021-05-25T12:08:54.005 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.955+0000 7f0588ff9700 10 client.?.objecter resend_mon_ops 2021-05-25T12:08:54.005 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.955+0000 7f0588ff9700 10 client.?.objecter ms_handle_connect 0x7f0594094da0 2021-05-25T12:08:54.005 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.955+0000 7f0588ff9700 10 client.?.objecter resend_mon_ops 2021-05-25T12:08:54.006 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.955+0000 7f0588ff9700 1 -- 172.21.15.102:0/4001666364 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2111221443 0 0) 0x7f059409b5f0 con 0x7f05941323c0 2021-05-25T12:08:54.006 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.955+0000 7f0588ff9700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:54.006 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.955+0000 7f0588ff9700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:54.006 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.955+0000 7f0588ff9700 10 monclient(hunting): my global_id is 264162 2021-05-25T12:08:54.007 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.955+0000 7f0588ff9700 1 -- 172.21.15.102:0/4001666364 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0574004c10 con 0x7f05941323c0 2021-05-25T12:08:54.007 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.955+0000 7f0588ff9700 1 -- 172.21.15.102:0/4001666364 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1410125317 0 0) 0x7f059412ee90 con 0x7f0594063840 2021-05-25T12:08:54.007 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.955+0000 7f0588ff9700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:54.007 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.955+0000 7f0588ff9700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:54.007 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.955+0000 7f0588ff9700 10 monclient(hunting): my global_id is 244277 2021-05-25T12:08:54.008 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.955+0000 7f0588ff9700 1 -- 172.21.15.102:0/4001666364 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f059409b5f0 con 0x7f0594063840 2021-05-25T12:08:54.008 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.955+0000 7f0588ff9700 1 -- 172.21.15.102:0/4001666364 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3582854038 0 0) 0x7f059412d620 con 0x7f0594094da0 2021-05-25T12:08:54.008 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.955+0000 7f0588ff9700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:54.008 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.955+0000 7f0588ff9700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:54.009 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.955+0000 7f0588ff9700 10 monclient(hunting): my global_id is 224533 2021-05-25T12:08:54.009 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.955+0000 7f0588ff9700 1 -- 172.21.15.102:0/4001666364 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f059412ee90 con 0x7f0594094da0 2021-05-25T12:08:54.009 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.955+0000 7f0588ff9700 1 -- 172.21.15.102:0/4001666364 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4214170345 0 0) 0x7f0574004c10 con 0x7f05941323c0 2021-05-25T12:08:54.009 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.955+0000 7f0588ff9700 1 -- 172.21.15.102:0/4001666364 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f059412d620 con 0x7f05941323c0 2021-05-25T12:08:54.009 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.955+0000 7f0588ff9700 1 -- 172.21.15.102:0/4001666364 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f0590001d70 con 0x7f05941323c0 2021-05-25T12:08:54.010 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.955+0000 7f0588ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:54.010 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.955+0000 7f0588ff9700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2021-05-25T12:08:54.010 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.955+0000 7f0588ff9700 10 monclient(hunting): dump: 2021-05-25T12:08:54.010 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:54.011 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:54.011 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:54.011 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:54.011 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:54.011 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:54.012 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:54.012 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:54.012 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:54.012 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:54.013 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.955+0000 7f0588ff9700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:08:54.013 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.955+0000 7f0588ff9700 1 -- 172.21.15.102:0/4001666364 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1816799654 0 0) 0x7f059409b5f0 con 0x7f0594063840 2021-05-25T12:08:54.013 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.955+0000 7f0588ff9700 1 -- 172.21.15.102:0/4001666364 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0574004c10 con 0x7f0594063840 2021-05-25T12:08:54.013 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.955+0000 7f0588ff9700 1 -- 172.21.15.102:0/4001666364 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f0584001800 con 0x7f0594063840 2021-05-25T12:08:54.013 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.955+0000 7f0588ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:54.014 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.955+0000 7f0588ff9700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2021-05-25T12:08:54.014 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.955+0000 7f0588ff9700 10 monclient(hunting): dump: 2021-05-25T12:08:54.014 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:54.014 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:54.015 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:54.015 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:54.015 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:54.015 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:54.015 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:54.016 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:54.016 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:54.016 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:54.016 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.955+0000 7f0588ff9700 1 -- 172.21.15.102:0/4001666364 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 712104203 0 0) 0x7f059412ee90 con 0x7f0594094da0 2021-05-25T12:08:54.017 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.955+0000 7f0588ff9700 1 -- 172.21.15.102:0/4001666364 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f059409b5f0 con 0x7f0594094da0 2021-05-25T12:08:54.017 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.955+0000 7f0588ff9700 1 -- 172.21.15.102:0/4001666364 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 297151542 0 0) 0x7f059412d620 con 0x7f05941323c0 2021-05-25T12:08:54.017 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.955+0000 7f0588ff9700 1 -- 172.21.15.102:0/4001666364 >> v1:172.21.15.102:6790/0 conn(0x7f0594063840 legacy=0x7f059412c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:54.017 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.955+0000 7f0588ff9700 1 -- 172.21.15.102:0/4001666364 >> v1:172.21.15.181:6789/0 conn(0x7f0594094da0 legacy=0x7f059412cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:54.018 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.955+0000 7f0588ff9700 10 monclient: _finish_hunting 0 2021-05-25T12:08:54.018 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.955+0000 7f0588ff9700 1 monclient: found mon.a 2021-05-25T12:08:54.018 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.955+0000 7f0588ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:08:54.018 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.955+0000 7f0588ff9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:08:54.018 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.955+0000 7f0588ff9700 1 -- 172.21.15.102:0/4001666364 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0594096750 con 0x7f05941323c0 2021-05-25T12:08:54.019 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.955+0000 7f0588ff9700 10 monclient: _finish_auth 0 2021-05-25T12:08:54.019 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.955+0000 7f0588ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:08:54.019 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.955+0000 7f059ab52700 5 monclient: authenticate success, global_id 264162 2021-05-25T12:08:54.019 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.955+0000 7f059ab52700 10 monclient: _renew_subs 2021-05-25T12:08:54.020 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.955+0000 7f059ab52700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:08:54.020 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.955+0000 7f059ab52700 1 -- 172.21.15.102:0/4001666364 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0594068980 con 0x7f05941323c0 2021-05-25T12:08:54.020 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.955+0000 7f059ab52700 10 client.264162.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:08:54.020 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.955+0000 7f059ab52700 10 monclient: _renew_subs 2021-05-25T12:08:54.021 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.955+0000 7f059ab52700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2021-05-25T12:08:54.021 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.955+0000 7f059ab52700 1 -- 172.21.15.102:0/4001666364 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f059412fa70 con 0x7f05941323c0 2021-05-25T12:08:54.021 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.955+0000 7f059ab52700 1 librados: init done 2021-05-25T12:08:54.022 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.955+0000 7f0588ff9700 1 -- 172.21.15.102:0/4001666364 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f05900020b0 con 0x7f05941323c0 2021-05-25T12:08:54.022 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.955+0000 7f0588ff9700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:08:54.022 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.955+0000 7f0588ff9700 1 -- 172.21.15.102:0/4001666364 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f0590004d80 con 0x7f05941323c0 2021-05-25T12:08:54.022 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.955+0000 7f0588ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:54.023 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.955+0000 7f0588ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-05-25T12:08:54.023 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.955+0000 7f058affd700 4 set_mon_vals no callback set 2021-05-25T12:08:54.023 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.955+0000 7f0588ff9700 10 monclient: dump: 2021-05-25T12:08:54.023 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:54.023 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:54.024 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:54.024 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:54.024 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:54.024 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:54.025 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:54.025 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:54.025 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:54.025 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:54.025 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.956+0000 7f0588ff9700 1 -- 172.21.15.102:0/4001666364 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f059000e120 con 0x7f05941323c0 2021-05-25T12:08:54.026 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.956+0000 7f059ab52700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:08:54.026 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.956+0000 7f0588ff9700 1 -- 172.21.15.102:0/4001666364 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(296..296 src has 250..296) v4 ==== 5459+0+0 (unknown 1521303069 0 0) 0x7f059003da20 con 0x7f05941323c0 2021-05-25T12:08:54.026 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.956+0000 7f0588ff9700 10 client.264162.objecter ms_dispatch 0x7f0594001550 osd_map(296..296 src has 250..296) v4 2021-05-25T12:08:54.026 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.956+0000 7f0588ff9700 3 client.264162.objecter handle_osd_map got epochs [296,296] > 0 2021-05-25T12:08:54.027 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.956+0000 7f0588ff9700 3 client.264162.objecter handle_osd_map decoding full epoch 296 2021-05-25T12:08:54.027 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.956+0000 7f059ab52700 1 -- 172.21.15.102:0/4001666364 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f056c0065a0 con 0x7f056c002840 2021-05-25T12:08:54.027 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.956+0000 7f0588ff9700 20 client.264162.objecter dump_active .. 0 homeless 2021-05-25T12:08:54.027 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.956+0000 7f05988ee700 10 monclient: get_auth_request con 0x7f05740335c0 auth_method 0 2021-05-25T12:08:54.028 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.956+0000 7f0588ff9700 10 client.264162.objecter ms_handle_connect 0x7f056c002840 2021-05-25T12:08:54.028 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.956+0000 7f0588ff9700 10 client.264162.objecter resend_mon_ops 2021-05-25T12:08:54.028 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.957+0000 7f0588ff9700 10 client.264162.objecter ms_handle_connect 0x7f05740335c0 2021-05-25T12:08:54.028 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.957+0000 7f0588ff9700 1 -- 172.21.15.102:0/4001666364 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2873556710 0 0) 0x7f056c0065a0 con 0x7f056c002840 2021-05-25T12:08:54.028 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.957+0000 7f0588ff9700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:54.029 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.957+0000 7f0588ff9700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:54.029 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.957+0000 7f0588ff9700 10 monclient(hunting): my global_id is 224536 2021-05-25T12:08:54.029 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.957+0000 7f0588ff9700 1 -- 172.21.15.102:0/4001666364 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f059412d620 con 0x7f056c002840 2021-05-25T12:08:54.029 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.957+0000 7f0588ff9700 1 -- 172.21.15.102:0/4001666364 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 803509002 0 0) 0x7f059412d620 con 0x7f056c002840 2021-05-25T12:08:54.030 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.957+0000 7f0588ff9700 1 -- 172.21.15.102:0/4001666364 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f056c0065a0 con 0x7f056c002840 2021-05-25T12:08:54.030 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.957+0000 7f0588ff9700 1 -- 172.21.15.102:0/4001666364 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f057c0046d0 con 0x7f056c002840 2021-05-25T12:08:54.030 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.957+0000 7f0588ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:54.030 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.957+0000 7f0588ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T12:08:54.030 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.957+0000 7f0588ff9700 10 monclient: dump: 2021-05-25T12:08:54.031 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:54.031 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:54.031 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:54.031 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:54.032 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:54.032 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:54.032 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:54.032 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:54.033 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:54.033 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:54.033 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.957+0000 7f0588ff9700 1 -- 172.21.15.102:0/4001666364 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2740934819 0 0) 0x7f056c0065a0 con 0x7f056c002840 2021-05-25T12:08:54.033 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.957+0000 7f0588ff9700 1 -- 172.21.15.102:0/4001666364 --> v1:172.21.15.181:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f056c009e40 con 0x7f056c002840 2021-05-25T12:08:54.033 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.957+0000 7f0588ff9700 1 -- 172.21.15.102:0/4001666364 <== mon.1 v1:172.21.15.181:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7f057c006890 con 0x7f056c002840 2021-05-25T12:08:54.034 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.957+0000 7f0588ff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:08:54.034 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.957+0000 7f0588ff9700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:08:54.034 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.957+0000 7f0588ff9700 1 -- 172.21.15.102:0/4001666364 >> v1:172.21.15.181:6789/0 conn(0x7f056c002840 legacy=0x7f056c004ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:54.034 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.957+0000 7f0588ff9700 1 -- 172.21.15.102:0/4001666364 >> v1:172.21.15.181:6789/0 conn(0x7f056c002840 legacy=0x7f056c004ce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:08:54.035 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.964+0000 7f059ab52700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:08:54.035 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.964+0000 7f059ab52700 1 -- 172.21.15.102:0/4001666364 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f056c007510 con 0x7f056c0068b0 2021-05-25T12:08:54.035 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.964+0000 7f0588ff9700 10 client.264162.objecter ms_handle_connect 0x7f056c0068b0 2021-05-25T12:08:54.035 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.964+0000 7f0588ff9700 10 client.264162.objecter resend_mon_ops 2021-05-25T12:08:54.036 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.965+0000 7f0588ff9700 1 -- 172.21.15.102:0/4001666364 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 212851822 0 0) 0x7f056c007510 con 0x7f056c0068b0 2021-05-25T12:08:54.036 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.965+0000 7f0588ff9700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:54.036 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.965+0000 7f0588ff9700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:54.036 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.965+0000 7f0588ff9700 10 monclient(hunting): my global_id is 224539 2021-05-25T12:08:54.036 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.965+0000 7f0588ff9700 1 -- 172.21.15.102:0/4001666364 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f056c0065a0 con 0x7f056c0068b0 2021-05-25T12:08:54.037 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.965+0000 7f0588ff9700 1 -- 172.21.15.102:0/4001666364 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3119536061 0 0) 0x7f056c0065a0 con 0x7f056c0068b0 2021-05-25T12:08:54.037 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.965+0000 7f0588ff9700 1 -- 172.21.15.102:0/4001666364 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f056c007510 con 0x7f056c0068b0 2021-05-25T12:08:54.037 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.965+0000 7f0588ff9700 1 -- 172.21.15.102:0/4001666364 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f057c001be0 con 0x7f056c0068b0 2021-05-25T12:08:54.037 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.965+0000 7f0588ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:54.038 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.965+0000 7f0588ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T12:08:54.038 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.965+0000 7f0588ff9700 10 monclient: dump: 2021-05-25T12:08:54.038 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:54.038 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:54.038 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:54.039 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:54.039 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:54.039 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:54.039 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:54.040 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:54.040 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:54.040 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:54.040 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.965+0000 7f0588ff9700 1 -- 172.21.15.102:0/4001666364 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1772937281 0 0) 0x7f056c007510 con 0x7f056c0068b0 2021-05-25T12:08:54.041 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.965+0000 7f0588ff9700 1 -- 172.21.15.102:0/4001666364 --> v1:172.21.15.181:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f056c008770 con 0x7f056c0068b0 2021-05-25T12:08:54.041 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.965+0000 7f0588ff9700 1 -- 172.21.15.102:0/4001666364 <== mon.1 v1:172.21.15.181:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4112 (unknown 0 0 2252772152) 0x7f057c006bb0 con 0x7f056c0068b0 2021-05-25T12:08:54.041 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.965+0000 7f0588ff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:08:54.041 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.965+0000 7f0588ff9700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:08:54.041 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.965+0000 7f0588ff9700 1 -- 172.21.15.102:0/4001666364 >> v1:172.21.15.181:6789/0 conn(0x7f056c0068b0 legacy=0x7f056c006d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:54.042 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.965+0000 7f0588ff9700 1 -- 172.21.15.102:0/4001666364 >> v1:172.21.15.181:6789/0 conn(0x7f056c0068b0 legacy=0x7f056c006d40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:08:54.042 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:08:54.043 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:08:54.043 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:08:54.043 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "leader", 2021-05-25T12:08:54.043 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 74, 2021-05-25T12:08:54.044 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:08:54.044 INFO:teuthology.orchestra.run.smithi102.stdout: 0, 2021-05-25T12:08:54.044 INFO:teuthology.orchestra.run.smithi102.stdout: 1, 2021-05-25T12:08:54.044 INFO:teuthology.orchestra.run.smithi102.stdout: 2 2021-05-25T12:08:54.044 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:08:54.045 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 3, 2021-05-25T12:08:54.045 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:08:54.045 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:08:54.045 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:08:54.046 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:08:54.046 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:08:54.046 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:08:54.046 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:08:54.047 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:08:54.047 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:08:54.047 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:08:54.047 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:08:54.048 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:08:54.048 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:08:54.048 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:08:54.048 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:08:54.049 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:08:54.049 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:08:54.049 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:08:54.049 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:08:54.049 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:08:54.050 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:08:54.050 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:08:54.050 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:08:54.050 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:08:54.050 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:54.051 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:54.051 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:08:54.051 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:08:54.051 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:08:54.052 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:08:54.052 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:08:54.052 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:08:54.052 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:08:54.052 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:08:54.053 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:08:54.053 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:08:54.053 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:08:54.054 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:08:54.054 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:08:54.054 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:08:54.054 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:08:54.055 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:08:54.055 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:08:54.055 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:08:54.055 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:08:54.056 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:08:54.056 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:08:54.056 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:08:54.056 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:08:54.056 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:08:54.057 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:08:54.057 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:08:54.057 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:54.057 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:08:54.057 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:54.058 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:08:54.058 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:08:54.058 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:08:54.059 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:08:54.059 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:54.059 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:08:54.059 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:08:54.059 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:08:54.060 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:54.060 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:54.060 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:54.060 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:08:54.061 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:08:54.061 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:08:54.061 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:08:54.062 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:08:54.062 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:54.062 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:54.063 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:08:54.063 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:08:54.063 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:08:54.064 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:08:54.064 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:54.065 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:08:54.065 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:08:54.065 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:08:54.066 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:54.066 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:54.066 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:54.067 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:08:54.067 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:08:54.067 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:08:54.068 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:08:54.068 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:08:54.069 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:54.069 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:54.069 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:08:54.070 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:08:54.070 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:08:54.071 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:08:54.071 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:54.071 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:08:54.072 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:08:54.072 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:08:54.072 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:54.073 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:54.073 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:54.074 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:08:54.074 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:08:54.074 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:08:54.075 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:08:54.075 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:08:54.075 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:54.076 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:54.076 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:54.077 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:08:54.077 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:08:54.077 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:54.078 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:08:54.078 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:08:54.079 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:08:54.079 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:54.079 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:08:54.080 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": [ 2021-05-25T12:08:54.080 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:54.080 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:08:54.081 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:08:54.081 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 4 2021-05-25T12:08:54.081 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:54.082 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:08:54.082 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:08:54.083 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:54.083 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:08:54.083 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:08:54.084 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 5 2021-05-25T12:08:54.084 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:54.084 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:08:54.085 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": [ 2021-05-25T12:08:54.085 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:08:54.085 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:08:54.086 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:08:54.086 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:08:54.087 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:08:54.087 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:08:54.087 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:08:54.088 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:08:54.088 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:08:54.092 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.966+0000 7f059ab52700 10 librados: watch_flush enter 2021-05-25T12:08:54.092 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.966+0000 7f059ab52700 10 librados: watch_flush exit 2021-05-25T12:08:54.093 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.966+0000 7f059ab52700 20 client.264162.objecter shutdown clearing up homeless session... 2021-05-25T12:08:54.093 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.966+0000 7f059ab52700 10 client.264162.objecter successfully canceled tick 2021-05-25T12:08:54.093 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.966+0000 7f059ab52700 1 -- 172.21.15.102:0/4001666364 >> v1:172.21.15.181:6812/29582 conn(0x7f05740335c0 legacy=0x7f0574035a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:54.094 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.966+0000 7f059ab52700 10 monclient: shutdown 2021-05-25T12:08:54.094 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.966+0000 7f059ab52700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:08:54.095 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.966+0000 7f059ab52700 1 -- 172.21.15.102:0/4001666364 >> v1:172.21.15.102:6789/0 conn(0x7f05941323c0 legacy=0x7f0594143770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:54.095 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.966+0000 7f05990ef700 1 -- 172.21.15.102:0/4001666364 reap_dead start 2021-05-25T12:08:54.095 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.966+0000 7f05990ef700 1 -- 172.21.15.102:0/4001666364 reap_dead start 2021-05-25T12:08:54.096 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.966+0000 7f059ab52700 1 -- 172.21.15.102:0/4001666364 shutdown_connections 2021-05-25T12:08:54.096 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.966+0000 7f059ab52700 1 -- 172.21.15.102:0/4001666364 >> 172.21.15.102:0/4001666364 conn(0x7f059409cb30 msgr2=0x7f0594095220 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:08:54.097 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.966+0000 7f059ab52700 1 -- 172.21.15.102:0/4001666364 shutdown_connections 2021-05-25T12:08:54.097 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.966+0000 7f059ab52700 1 -- 172.21.15.102:0/4001666364 wait complete. 2021-05-25T12:08:54.097 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:53.966+0000 7f059ab52700 1 librados: shutdown 2021-05-25T12:08:54.098 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2021-05-25T12:08:54.099 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2021-05-25T12:08:54.099 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2021-05-25T12:08:54.100 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2021-05-25T12:08:54.100 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2021-05-25T12:08:54.101 INFO:teuthology.orchestra.run:waiting for 300 2021-05-25T12:08:54.110 INFO:tasks.ceph.mon.a:Stopped 2021-05-25T12:08:54.110 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2021-05-25T12:08:54.111 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2021-05-25T12:08:54.259 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.257+0000 7fabb5766700 10 monclient: get_monmap_and_config 2021-05-25T12:08:54.259 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.257+0000 7fabb5766700 10 monclient: build_initial_monmap 2021-05-25T12:08:54.260 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.257+0000 7fabb5766700 10 monclient: monmap: 2021-05-25T12:08:54.260 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:08:54.260 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:54.261 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:08:54.258668+0000 2021-05-25T12:08:54.261 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:08:54.258668+0000 2021-05-25T12:08:54.262 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:08:54.262 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:54.262 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:08:54.263 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:08:54.263 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:08:54.264 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:54.282 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.258+0000 7fabb5766700 1 Processor -- start 2021-05-25T12:08:54.282 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.258+0000 7fabb5766700 1 -- start start 2021-05-25T12:08:54.282 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.258+0000 7fabb5766700 10 monclient: init 2021-05-25T12:08:54.283 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.258+0000 7fabb5766700 10 monclient: _reopen_session rank -1 2021-05-25T12:08:54.283 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.258+0000 7fabb5766700 10 monclient: _add_conns ranks=[2,1,0] 2021-05-25T12:08:54.284 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.258+0000 7fabb5766700 10 monclient(hunting): picked mon.noname-b con 0x7fabb0061b20 addr v1:172.21.15.102:6790/0 2021-05-25T12:08:54.284 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.258+0000 7fabb5766700 10 monclient(hunting): picked mon.noname-c con 0x7fabb00954b0 addr v1:172.21.15.181:6789/0 2021-05-25T12:08:54.284 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.258+0000 7fabb5766700 10 monclient(hunting): picked mon.noname-a con 0x7fabb0095960 addr v1:172.21.15.102:6789/0 2021-05-25T12:08:54.285 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.258+0000 7fabb5766700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabb009ad80 con 0x7fabb0095960 2021-05-25T12:08:54.285 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.258+0000 7fabb5766700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabb009b020 con 0x7fabb00954b0 2021-05-25T12:08:54.285 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.258+0000 7fabb5766700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabb009b3e0 con 0x7fabb0061b20 2021-05-25T12:08:54.286 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.258+0000 7fabb5766700 10 monclient(hunting): _renew_subs 2021-05-25T12:08:54.286 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.258+0000 7fabb5766700 10 monclient(hunting): authenticate will time out at 1334.855975s 2021-05-25T12:08:54.287 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.259+0000 7fabb4f65700 1 -- >> v1:172.21.15.102:6789/0 conn(0x7fabb0095960 legacy=0x7fabb0134130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:08:54.287 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.259+0000 7fabaffff700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fabb0061b20 0x7fabb0094da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:57624/0 (socket says 172.21.15.102:57624) 2021-05-25T12:08:54.287 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.259+0000 7fabaffff700 1 -- 172.21.15.102:0/1396854980 learned_addr learned my addr 172.21.15.102:0/1396854980 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:08:54.288 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.259+0000 7fabaeffd700 1 -- 172.21.15.102:0/1396854980 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3192206203 0 0) 0x7fabb009b3e0 con 0x7fabb0061b20 2021-05-25T12:08:54.288 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.259+0000 7fabaeffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:54.289 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.259+0000 7fabaeffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:54.289 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.259+0000 7fabaeffd700 10 monclient(hunting): my global_id is 244280 2021-05-25T12:08:54.289 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.259+0000 7fabaeffd700 1 -- 172.21.15.102:0/1396854980 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab98003270 con 0x7fabb0061b20 2021-05-25T12:08:54.290 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.259+0000 7fabaeffd700 1 -- 172.21.15.102:0/1396854980 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3454359869 0 0) 0x7fab98003270 con 0x7fabb0061b20 2021-05-25T12:08:54.290 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.259+0000 7fabaeffd700 1 -- 172.21.15.102:0/1396854980 >> v1:172.21.15.181:6789/0 conn(0x7fabb00954b0 legacy=0x7fabb0130a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:54.290 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.259+0000 7fabaeffd700 1 -- 172.21.15.102:0/1396854980 >> v1:172.21.15.102:6789/0 conn(0x7fabb0095960 legacy=0x7fabb0134130 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-05-25T12:08:54.291 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.259+0000 7fabaeffd700 10 monclient: _finish_hunting 0 2021-05-25T12:08:54.291 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.259+0000 7fabaeffd700 1 monclient: found mon.noname-b 2021-05-25T12:08:54.292 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.259+0000 7fabaeffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:08:54.292 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.259+0000 7fabaeffd700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T12:08:54.292 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.259+0000 7fabaeffd700 1 -- 172.21.15.102:0/1396854980 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabb00695e0 con 0x7fabb0061b20 2021-05-25T12:08:54.293 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.259+0000 7fabaeffd700 10 monclient: _finish_auth 0 2021-05-25T12:08:54.293 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.259+0000 7fabaeffd700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:08:54.293 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.259+0000 7fabaeffd700 1 -- 172.21.15.102:0/1396854980 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fab94001610 con 0x7fabb0061b20 2021-05-25T12:08:54.294 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.259+0000 7fabb5766700 5 monclient: authenticate success, global_id 244280 2021-05-25T12:08:54.294 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.259+0000 7fabb5766700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:08:54.295 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.259+0000 7fabaeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:54.295 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.259+0000 7fabaeffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:08:54.295 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.259+0000 7fabaeffd700 10 monclient: dump: 2021-05-25T12:08:54.296 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:54.296 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:54.296 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:54.297 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:54.297 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:54.298 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:54.298 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:54.298 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:54.299 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:54.299 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:54.299 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.259+0000 7fabb5766700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:08:54.300 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.259+0000 7fabaeffd700 1 -- 172.21.15.102:0/1396854980 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fab94001930 con 0x7fabb0061b20 2021-05-25T12:08:54.300 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.259+0000 7fabaeffd700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:08:54.300 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.259+0000 7fabb5766700 10 monclient: get_monmap_and_config success 2021-05-25T12:08:54.301 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.259+0000 7fabaeffd700 1 -- 172.21.15.102:0/1396854980 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fab94001d00 con 0x7fabb0061b20 2021-05-25T12:08:54.301 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.259+0000 7fabaeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:54.302 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.259+0000 7fabb5766700 4 set_mon_vals no callback set 2021-05-25T12:08:54.302 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.259+0000 7fabaeffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:08:54.302 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.259+0000 7fabaeffd700 10 monclient: dump: 2021-05-25T12:08:54.303 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:54.303 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:54.303 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:54.304 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:54.304 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:54.304 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:54.305 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:54.305 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:54.306 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:54.306 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:54.306 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.260+0000 7fabb5766700 10 monclient: shutdown 2021-05-25T12:08:54.307 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.260+0000 7fabb5766700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:08:54.307 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.260+0000 7fabb5766700 1 -- 172.21.15.102:0/1396854980 >> v1:172.21.15.102:6790/0 conn(0x7fabb0061b20 legacy=0x7fabb0094da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:54.307 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.260+0000 7fabb5766700 1 -- 172.21.15.102:0/1396854980 shutdown_connections 2021-05-25T12:08:54.308 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.260+0000 7fabb5766700 1 -- 172.21.15.102:0/1396854980 >> 172.21.15.102:0/1396854980 conn(0x7fabb0098300 msgr2=0x7fabb0098710 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:08:54.308 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.260+0000 7fabb5766700 1 -- 172.21.15.102:0/1396854980 shutdown_connections 2021-05-25T12:08:54.309 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.260+0000 7fabb5766700 1 -- 172.21.15.102:0/1396854980 wait complete. 2021-05-25T12:08:54.309 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.260+0000 7fabb5766700 10 monclient: build_initial_monmap 2021-05-25T12:08:54.309 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.260+0000 7fabb5766700 10 monclient: monmap: 2021-05-25T12:08:54.310 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:08:54.310 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:08:54.310 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:08:54.311 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:08:54.311 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:08:54.311 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:08:54.312 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:08:54.312 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:08:54.313 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:08:54.313 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:54.313 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.260+0000 7fabb5766700 1 librados: starting msgr at 2021-05-25T12:08:54.314 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.260+0000 7fabb5766700 1 librados: starting objecter 2021-05-25T12:08:54.314 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.260+0000 7fabb5766700 1 Processor -- start 2021-05-25T12:08:54.314 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.261+0000 7fabb5766700 1 -- start start 2021-05-25T12:08:54.315 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.261+0000 7fabb5766700 1 librados: setting wanted keys 2021-05-25T12:08:54.315 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.261+0000 7fabb5766700 1 librados: calling monclient init 2021-05-25T12:08:54.316 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.261+0000 7fabb5766700 10 monclient: init 2021-05-25T12:08:54.316 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.261+0000 7fabb5766700 10 monclient: _reopen_session rank -1 2021-05-25T12:08:54.316 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.261+0000 7fabb5766700 10 monclient: _add_conns ranks=[0,2,1] 2021-05-25T12:08:54.317 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.261+0000 7fabb5766700 10 monclient(hunting): picked mon.noname-a con 0x7fabb0061b20 addr v1:172.21.15.102:6789/0 2021-05-25T12:08:54.317 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.261+0000 7fabb5766700 10 monclient(hunting): picked mon.noname-c con 0x7fabb00954b0 addr v1:172.21.15.102:6790/0 2021-05-25T12:08:54.317 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.261+0000 7fabb5766700 10 monclient(hunting): picked mon.noname-b con 0x7fabb0095960 addr v1:172.21.15.181:6789/0 2021-05-25T12:08:54.318 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.261+0000 7fabb5766700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabb0128310 con 0x7fabb0061b20 2021-05-25T12:08:54.318 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.261+0000 7fabb5766700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabb0120e90 con 0x7fabb0095960 2021-05-25T12:08:54.319 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.261+0000 7fabb5766700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabb0121130 con 0x7fabb00954b0 2021-05-25T12:08:54.319 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.261+0000 7fabb5766700 10 monclient(hunting): _renew_subs 2021-05-25T12:08:54.319 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.261+0000 7fabb5766700 10 monclient(hunting): authenticate will time out at 1334.858540s 2021-05-25T12:08:54.320 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.261+0000 7fabaffff700 1 -- >> v1:172.21.15.102:6789/0 conn(0x7fabb0061b20 legacy=0x7fabb013ace0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:08:54.320 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.261+0000 7fabaf7fe700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fabb00954b0 0x7fabb0125610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:57632/0 (socket says 172.21.15.102:57632) 2021-05-25T12:08:54.320 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.261+0000 7fabaf7fe700 1 -- 172.21.15.102:0/1527775818 learned_addr learned my addr 172.21.15.102:0/1527775818 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:08:54.321 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.261+0000 7faba3fff700 10 client.?.objecter ms_handle_connect 0x7fabb00954b0 2021-05-25T12:08:54.321 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.261+0000 7faba3fff700 10 client.?.objecter resend_mon_ops 2021-05-25T12:08:54.322 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.261+0000 7faba3fff700 10 client.?.objecter ms_handle_connect 0x7fabb0095960 2021-05-25T12:08:54.322 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.261+0000 7faba3fff700 10 client.?.objecter resend_mon_ops 2021-05-25T12:08:54.322 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.261+0000 7faba3fff700 1 -- 172.21.15.102:0/1527775818 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1039822932 0 0) 0x7fabb0121130 con 0x7fabb00954b0 2021-05-25T12:08:54.323 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.261+0000 7faba3fff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:54.323 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.261+0000 7faba3fff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:54.323 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.261+0000 7faba3fff700 10 monclient(hunting): my global_id is 244283 2021-05-25T12:08:54.324 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.261+0000 7faba3fff700 1 -- 172.21.15.102:0/1527775818 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab8c004c10 con 0x7fabb00954b0 2021-05-25T12:08:54.324 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.261+0000 7faba3fff700 1 -- 172.21.15.102:0/1527775818 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 699442672 0 0) 0x7fabb0120e90 con 0x7fabb0095960 2021-05-25T12:08:54.325 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.261+0000 7faba3fff700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:54.325 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.261+0000 7faba3fff700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:54.325 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.261+0000 7faba3fff700 10 monclient(hunting): my global_id is 224545 2021-05-25T12:08:54.326 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.261+0000 7faba3fff700 1 -- 172.21.15.102:0/1527775818 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabb0121130 con 0x7fabb0095960 2021-05-25T12:08:54.326 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.261+0000 7faba3fff700 1 -- 172.21.15.102:0/1527775818 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 620923927 0 0) 0x7fab8c004c10 con 0x7fabb00954b0 2021-05-25T12:08:54.326 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.262+0000 7faba3fff700 1 -- 172.21.15.102:0/1527775818 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fabb0120e90 con 0x7fabb00954b0 2021-05-25T12:08:54.327 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.262+0000 7faba3fff700 1 -- 172.21.15.102:0/1527775818 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fab9c001980 con 0x7fabb00954b0 2021-05-25T12:08:54.327 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.262+0000 7faba3fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:54.327 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.262+0000 7faba3fff700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:08:54.328 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.262+0000 7faba3fff700 10 monclient(hunting): dump: 2021-05-25T12:08:54.328 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:54.328 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:54.328 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:54.329 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:54.329 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:54.329 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:54.329 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:54.329 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:54.330 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:54.330 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:54.330 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.262+0000 7faba3fff700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:08:54.330 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.262+0000 7faba3fff700 1 -- 172.21.15.102:0/1527775818 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 742876821 0 0) 0x7fabb0120e90 con 0x7fabb00954b0 2021-05-25T12:08:54.331 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.262+0000 7faba3fff700 1 -- 172.21.15.102:0/1527775818 >> v1:172.21.15.181:6789/0 conn(0x7fabb0095960 legacy=0x7fabb0143030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:54.331 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.262+0000 7faba3fff700 1 -- 172.21.15.102:0/1527775818 >> v1:172.21.15.102:6789/0 conn(0x7fabb0061b20 legacy=0x7fabb013ace0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-05-25T12:08:54.331 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.262+0000 7faba3fff700 10 monclient: _finish_hunting 0 2021-05-25T12:08:54.331 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.262+0000 7faba3fff700 1 monclient: found mon.c 2021-05-25T12:08:54.331 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.262+0000 7faba3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:08:54.332 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.262+0000 7faba3fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:08:54.332 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.262+0000 7faba3fff700 1 -- 172.21.15.102:0/1527775818 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabb0067920 con 0x7fabb00954b0 2021-05-25T12:08:54.332 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.262+0000 7faba3fff700 10 monclient: _finish_auth 0 2021-05-25T12:08:54.332 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.262+0000 7faba3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:08:54.332 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.262+0000 7fabb5766700 5 monclient: authenticate success, global_id 244283 2021-05-25T12:08:54.333 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.262+0000 7fabb5766700 10 monclient: _renew_subs 2021-05-25T12:08:54.333 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.262+0000 7fabb5766700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:08:54.333 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.262+0000 7fabb5766700 1 -- 172.21.15.102:0/1527775818 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabb013b3f0 con 0x7fabb00954b0 2021-05-25T12:08:54.333 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.262+0000 7fabb5766700 10 client.244283.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:08:54.333 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.262+0000 7fabb5766700 10 monclient: _renew_subs 2021-05-25T12:08:54.334 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.262+0000 7fabb5766700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:08:54.334 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.262+0000 7fabb5766700 1 -- 172.21.15.102:0/1527775818 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fabb013b6a0 con 0x7fabb00954b0 2021-05-25T12:08:54.334 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.262+0000 7fabb5766700 1 librados: init done 2021-05-25T12:08:54.334 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.262+0000 7faba3fff700 1 -- 172.21.15.102:0/1527775818 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fab9c001cc0 con 0x7fabb00954b0 2021-05-25T12:08:54.335 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.262+0000 7faba3fff700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:08:54.335 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.262+0000 7faba3fff700 1 -- 172.21.15.102:0/1527775818 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fab9c002410 con 0x7fabb00954b0 2021-05-25T12:08:54.335 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.262+0000 7faba3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:54.335 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.262+0000 7faba3fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:08:54.335 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.262+0000 7fabae7fc700 4 set_mon_vals no callback set 2021-05-25T12:08:54.336 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.262+0000 7faba3fff700 10 monclient: dump: 2021-05-25T12:08:54.336 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:54.336 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:54.336 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:54.336 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:54.337 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:54.337 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:54.337 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:54.337 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:54.337 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:54.338 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:54.338 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.262+0000 7faba3fff700 1 -- 172.21.15.102:0/1527775818 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7fab9c0025f0 con 0x7fabb00954b0 2021-05-25T12:08:54.338 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.263+0000 7fabb5766700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:08:54.338 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.263+0000 7fabb5766700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:08:54.338 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.263+0000 7fabb5766700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:08:54.339 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.263+0000 7faba3fff700 1 -- 172.21.15.102:0/1527775818 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(296..296 src has 250..296) v4 ==== 5459+0+0 (unknown 1521303069 0 0) 0x7fab9c03d5f0 con 0x7fabb00954b0 2021-05-25T12:08:54.339 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.263+0000 7fabb5766700 1 -- 172.21.15.102:0/1527775818 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab90004db0 con 0x7fabb00954b0 2021-05-25T12:08:54.339 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.263+0000 7faba3fff700 10 client.244283.objecter ms_dispatch 0x7fabb0001550 osd_map(296..296 src has 250..296) v4 2021-05-25T12:08:54.339 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.263+0000 7faba3fff700 3 client.244283.objecter handle_osd_map got epochs [296,296] > 0 2021-05-25T12:08:54.340 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.263+0000 7faba3fff700 3 client.244283.objecter handle_osd_map decoding full epoch 296 2021-05-25T12:08:54.340 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.263+0000 7faba3fff700 20 client.244283.objecter dump_active .. 0 homeless 2021-05-25T12:08:54.340 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.263+0000 7fabaffff700 10 monclient: get_auth_request con 0x7fab8c033150 auth_method 0 2021-05-25T12:08:54.340 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.263+0000 7faba3fff700 10 client.244283.objecter ms_handle_connect 0x7fab8c033150 2021-05-25T12:08:54.340 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.267+0000 7faba3fff700 1 -- 172.21.15.102:0/1527775818 <== mon.2 v1:172.21.15.102:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7fab9c012bf0 con 0x7fabb00954b0 2021-05-25T12:08:54.341 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.267+0000 7faba3fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:08:54.341 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.267+0000 7faba3fff700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:08:54.477 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.475+0000 7fabb5766700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2021-05-25T12:08:54.477 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.475+0000 7fabb5766700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2021-05-25T12:08:54.477 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.475+0000 7fabb5766700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:08:54.477 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.475+0000 7fabb5766700 1 -- 172.21.15.102:0/1527775818 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fab900055e0 con 0x7fabb00954b0 2021-05-25T12:08:54.478 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.475+0000 7faba3fff700 1 -- 172.21.15.102:0/1527775818 <== mon.2 v1:172.21.15.102:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1342 (unknown 254746657 0 389402922) 0x7fab9c012a10 con 0x7fabb00954b0 2021-05-25T12:08:54.478 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.475+0000 7faba3fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2021-05-25T12:08:54.478 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.475+0000 7faba3fff700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:08:54.479 INFO:teuthology.orchestra.run.smithi102.stdout:{"election_epoch":74,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"b","quorum_age":3,"features":{"quorum_con":"4540138303579357183","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"20ee91b6-8935-4b98-b868-85c03789ccc4","modified":"2021-05-25T11:59:11.011522Z","created":"2021-05-25T11:59:11.011522Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6790","nonce":0}]},"addr":"172.21.15.102:6790/0","public_addr":"172.21.15.102:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2021-05-25T12:08:54.480 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.477+0000 7fabb5766700 10 librados: watch_flush enter 2021-05-25T12:08:54.480 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.477+0000 7fabb5766700 10 librados: watch_flush exit 2021-05-25T12:08:54.481 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.477+0000 7fabb5766700 20 client.244283.objecter shutdown clearing up homeless session... 2021-05-25T12:08:54.481 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.477+0000 7fabb5766700 10 client.244283.objecter successfully canceled tick 2021-05-25T12:08:54.481 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.477+0000 7fabb5766700 1 -- 172.21.15.102:0/1527775818 >> v1:172.21.15.181:6812/29582 conn(0x7fab8c033150 legacy=0x7fab8c035610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:54.481 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.477+0000 7fabb5766700 10 monclient: shutdown 2021-05-25T12:08:54.481 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.477+0000 7fabb5766700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:08:54.482 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.477+0000 7fabb5766700 1 -- 172.21.15.102:0/1527775818 >> v1:172.21.15.102:6790/0 conn(0x7fabb00954b0 legacy=0x7fabb0125610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:54.482 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.477+0000 7fabb5766700 1 -- 172.21.15.102:0/1527775818 shutdown_connections 2021-05-25T12:08:54.482 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.477+0000 7fabb5766700 1 -- 172.21.15.102:0/1527775818 >> 172.21.15.102:0/1527775818 conn(0x7fabb0098300 msgr2=0x7fabb0097100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:08:54.482 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.477+0000 7fabb5766700 1 -- 172.21.15.102:0/1527775818 shutdown_connections 2021-05-25T12:08:54.483 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.477+0000 7fabb5766700 1 -- 172.21.15.102:0/1527775818 wait complete. 2021-05-25T12:08:54.483 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:54.477+0000 7fabb5766700 1 librados: shutdown 2021-05-25T12:08:57.490 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2021-05-25T12:08:57.614 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.613+0000 7f2b55d65700 10 monclient: get_monmap_and_config 2021-05-25T12:08:57.615 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.613+0000 7f2b55d65700 10 monclient: build_initial_monmap 2021-05-25T12:08:57.615 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.613+0000 7f2b55d65700 10 monclient: monmap: 2021-05-25T12:08:57.615 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:08:57.616 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:57.616 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:08:57.614522+0000 2021-05-25T12:08:57.616 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:08:57.614522+0000 2021-05-25T12:08:57.616 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:08:57.616 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:57.617 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:08:57.617 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:08:57.617 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:08:57.617 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:57.628 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.614+0000 7f2b55d65700 1 Processor -- start 2021-05-25T12:08:57.628 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.614+0000 7f2b55d65700 1 -- start start 2021-05-25T12:08:57.628 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.614+0000 7f2b55d65700 10 monclient: init 2021-05-25T12:08:57.628 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.614+0000 7f2b55d65700 10 monclient: _reopen_session rank -1 2021-05-25T12:08:57.629 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.614+0000 7f2b55d65700 10 monclient: _add_conns ranks=[0,2,1] 2021-05-25T12:08:57.629 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.614+0000 7f2b55d65700 10 monclient(hunting): picked mon.noname-a con 0x7f2b50090990 addr v1:172.21.15.102:6789/0 2021-05-25T12:08:57.629 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.614+0000 7f2b55d65700 10 monclient(hunting): picked mon.noname-b con 0x7f2b50090d70 addr v1:172.21.15.102:6790/0 2021-05-25T12:08:57.629 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.614+0000 7f2b55d65700 10 monclient(hunting): picked mon.noname-c con 0x7f2b5012e330 addr v1:172.21.15.181:6789/0 2021-05-25T12:08:57.630 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.615+0000 7f2b55d65700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b50096fa0 con 0x7f2b50090990 2021-05-25T12:08:57.630 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.615+0000 7f2b55d65700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b500926a0 con 0x7f2b5012e330 2021-05-25T12:08:57.630 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.615+0000 7f2b55d65700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b50092940 con 0x7f2b50090d70 2021-05-25T12:08:57.630 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.615+0000 7f2b55d65700 10 monclient(hunting): _renew_subs 2021-05-25T12:08:57.631 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.615+0000 7f2b55d65700 10 monclient(hunting): authenticate will time out at 1338.212041s 2021-05-25T12:08:57.631 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.615+0000 7f2b4f7fe700 1 -- >> v1:172.21.15.102:6789/0 conn(0x7f2b50090990 legacy=0x7f2b50093240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:08:57.631 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.615+0000 7f2b4effd700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f2b50090d70 0x7f2b50091200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:57644/0 (socket says 172.21.15.102:57644) 2021-05-25T12:08:57.631 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.615+0000 7f2b4effd700 1 -- 172.21.15.102:0/827224240 learned_addr learned my addr 172.21.15.102:0/827224240 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:08:57.631 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.615+0000 7f2b4e7fc700 1 -- 172.21.15.102:0/827224240 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2593955195 0 0) 0x7f2b500926a0 con 0x7f2b5012e330 2021-05-25T12:08:57.632 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.615+0000 7f2b4e7fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:57.632 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.615+0000 7f2b4e7fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:57.632 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.615+0000 7f2b4e7fc700 10 monclient(hunting): my global_id is 224548 2021-05-25T12:08:57.632 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.615+0000 7f2b4e7fc700 1 -- 172.21.15.102:0/827224240 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b34003270 con 0x7f2b5012e330 2021-05-25T12:08:57.632 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.615+0000 7f2b4e7fc700 1 -- 172.21.15.102:0/827224240 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3338587174 0 0) 0x7f2b34003270 con 0x7f2b5012e330 2021-05-25T12:08:57.633 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.615+0000 7f2b4e7fc700 1 -- 172.21.15.102:0/827224240 >> v1:172.21.15.102:6790/0 conn(0x7f2b50090d70 legacy=0x7f2b50091200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:57.633 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.615+0000 7f2b4e7fc700 1 -- 172.21.15.102:0/827224240 >> v1:172.21.15.102:6789/0 conn(0x7f2b50090990 legacy=0x7f2b50093240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-05-25T12:08:57.633 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.615+0000 7f2b4e7fc700 10 monclient: _finish_hunting 0 2021-05-25T12:08:57.633 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.615+0000 7f2b4e7fc700 1 monclient: found mon.noname-c 2021-05-25T12:08:57.633 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.615+0000 7f2b4e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:08:57.634 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.615+0000 7f2b4e7fc700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.181:6789/0 2021-05-25T12:08:57.634 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.615+0000 7f2b4e7fc700 1 -- 172.21.15.102:0/827224240 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b50092be0 con 0x7f2b5012e330 2021-05-25T12:08:57.634 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.615+0000 7f2b4e7fc700 10 monclient: _finish_auth 0 2021-05-25T12:08:57.634 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.615+0000 7f2b4e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:08:57.634 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.615+0000 7f2b4e7fc700 1 -- 172.21.15.102:0/827224240 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f2b44001990 con 0x7f2b5012e330 2021-05-25T12:08:57.635 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.615+0000 7f2b55d65700 5 monclient: authenticate success, global_id 224548 2021-05-25T12:08:57.635 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.615+0000 7f2b55d65700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:08:57.635 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.615+0000 7f2b4e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:57.635 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.615+0000 7f2b4e7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:08:57.636 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.615+0000 7f2b4e7fc700 10 monclient: dump: 2021-05-25T12:08:57.636 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:57.636 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:57.636 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:57.636 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:57.637 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:57.637 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:57.637 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:57.637 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:57.638 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:57.638 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:57.638 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.615+0000 7f2b55d65700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:08:57.638 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.615+0000 7f2b4e7fc700 1 -- 172.21.15.102:0/827224240 <== mon.1 v1:172.21.15.181:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2b44001c40 con 0x7f2b5012e330 2021-05-25T12:08:57.638 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.615+0000 7f2b4e7fc700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:08:57.639 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.615+0000 7f2b55d65700 10 monclient: get_monmap_and_config success 2021-05-25T12:08:57.639 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.615+0000 7f2b4e7fc700 1 -- 172.21.15.102:0/827224240 <== mon.1 v1:172.21.15.181:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f2b44001fd0 con 0x7f2b5012e330 2021-05-25T12:08:57.639 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.615+0000 7f2b4e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:57.639 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.615+0000 7f2b55d65700 4 set_mon_vals no callback set 2021-05-25T12:08:57.640 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.615+0000 7f2b4e7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T12:08:57.640 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.615+0000 7f2b4e7fc700 10 monclient: dump: 2021-05-25T12:08:57.640 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:57.640 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:57.640 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:57.641 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:57.641 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:57.641 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:57.641 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:57.641 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:57.642 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:57.642 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:57.642 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.616+0000 7f2b55d65700 10 monclient: shutdown 2021-05-25T12:08:57.642 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.616+0000 7f2b55d65700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:08:57.643 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.616+0000 7f2b55d65700 1 -- 172.21.15.102:0/827224240 >> v1:172.21.15.181:6789/0 conn(0x7f2b5012e330 legacy=0x7f2b50130720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:57.643 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.616+0000 7f2b55d65700 1 -- 172.21.15.102:0/827224240 shutdown_connections 2021-05-25T12:08:57.643 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.616+0000 7f2b55d65700 1 -- 172.21.15.102:0/827224240 >> 172.21.15.102:0/827224240 conn(0x7f2b5009b370 msgr2=0x7f2b50097430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:08:57.643 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.616+0000 7f2b55d65700 1 -- 172.21.15.102:0/827224240 shutdown_connections 2021-05-25T12:08:57.643 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.616+0000 7f2b55d65700 1 -- 172.21.15.102:0/827224240 wait complete. 2021-05-25T12:08:57.644 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.617+0000 7f2b55d65700 10 monclient: build_initial_monmap 2021-05-25T12:08:57.644 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.617+0000 7f2b55d65700 10 monclient: monmap: 2021-05-25T12:08:57.644 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:08:57.644 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:08:57.644 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:08:57.645 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:08:57.645 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:08:57.645 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:08:57.645 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:08:57.645 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:08:57.646 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:08:57.646 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:57.646 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.617+0000 7f2b55d65700 1 librados: starting msgr at 2021-05-25T12:08:57.646 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.617+0000 7f2b55d65700 1 librados: starting objecter 2021-05-25T12:08:57.647 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.617+0000 7f2b55d65700 1 Processor -- start 2021-05-25T12:08:57.647 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.617+0000 7f2b55d65700 1 -- start start 2021-05-25T12:08:57.647 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.617+0000 7f2b55d65700 1 librados: setting wanted keys 2021-05-25T12:08:57.647 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.617+0000 7f2b55d65700 1 librados: calling monclient init 2021-05-25T12:08:57.647 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.617+0000 7f2b55d65700 10 monclient: init 2021-05-25T12:08:57.648 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.618+0000 7f2b55d65700 10 monclient: _reopen_session rank -1 2021-05-25T12:08:57.648 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.618+0000 7f2b55d65700 10 monclient: _add_conns ranks=[0,2,1] 2021-05-25T12:08:57.648 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.618+0000 7f2b55d65700 10 monclient(hunting): picked mon.noname-a con 0x7f2b50090990 addr v1:172.21.15.102:6789/0 2021-05-25T12:08:57.648 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.618+0000 7f2b55d65700 10 monclient(hunting): picked mon.noname-c con 0x7f2b50090d70 addr v1:172.21.15.102:6790/0 2021-05-25T12:08:57.649 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.618+0000 7f2b55d65700 10 monclient(hunting): picked mon.noname-b con 0x7f2b5012e330 addr v1:172.21.15.181:6789/0 2021-05-25T12:08:57.649 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.618+0000 7f2b55d65700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b50091fe0 con 0x7f2b50090990 2021-05-25T12:08:57.649 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.618+0000 7f2b55d65700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b5013ba80 con 0x7f2b5012e330 2021-05-25T12:08:57.649 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.618+0000 7f2b55d65700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b50068680 con 0x7f2b50090d70 2021-05-25T12:08:57.649 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.618+0000 7f2b55d65700 10 monclient(hunting): _renew_subs 2021-05-25T12:08:57.650 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.618+0000 7f2b55d65700 10 monclient(hunting): authenticate will time out at 1338.215263s 2021-05-25T12:08:57.650 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.618+0000 7f2b4f7fe700 1 -- >> v1:172.21.15.102:6789/0 conn(0x7f2b50090990 legacy=0x7f2b5013ab90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:08:57.650 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.618+0000 7f2b4effd700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f2b50090d70 0x7f2b50126490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:57648/0 (socket says 172.21.15.102:57648) 2021-05-25T12:08:57.650 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.618+0000 7f2b4effd700 1 -- 172.21.15.102:0/1368239511 learned_addr learned my addr 172.21.15.102:0/1368239511 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:08:57.651 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.618+0000 7f2b3b7fe700 10 client.?.objecter ms_handle_connect 0x7f2b50090d70 2021-05-25T12:08:57.651 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.618+0000 7f2b3b7fe700 10 client.?.objecter resend_mon_ops 2021-05-25T12:08:57.651 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.618+0000 7f2b3b7fe700 10 client.?.objecter ms_handle_connect 0x7f2b5012e330 2021-05-25T12:08:57.651 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.618+0000 7f2b3b7fe700 10 client.?.objecter resend_mon_ops 2021-05-25T12:08:57.651 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.618+0000 7f2b3b7fe700 1 -- 172.21.15.102:0/1368239511 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4000616470 0 0) 0x7f2b5013ba80 con 0x7f2b5012e330 2021-05-25T12:08:57.652 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.618+0000 7f2b3b7fe700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:08:57.652 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.618+0000 7f2b3b7fe700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:08:57.652 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.618+0000 7f2b3b7fe700 10 monclient(hunting): my global_id is 224551 2021-05-25T12:08:57.653 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.618+0000 7f2b3b7fe700 1 -- 172.21.15.102:0/1368239511 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b30004c10 con 0x7f2b5012e330 2021-05-25T12:08:57.653 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.619+0000 7f2b3b7fe700 1 -- 172.21.15.102:0/1368239511 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 516618767 0 0) 0x7f2b30004c10 con 0x7f2b5012e330 2021-05-25T12:08:57.653 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.619+0000 7f2b3b7fe700 1 -- 172.21.15.102:0/1368239511 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2b5013ba80 con 0x7f2b5012e330 2021-05-25T12:08:57.653 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.619+0000 7f2b3b7fe700 1 -- 172.21.15.102:0/1368239511 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f2b44001950 con 0x7f2b5012e330 2021-05-25T12:08:57.654 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.619+0000 7f2b3b7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:57.654 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.619+0000 7f2b3b7fe700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:08:57.654 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.619+0000 7f2b3b7fe700 10 monclient(hunting): dump: 2021-05-25T12:08:57.654 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:57.654 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:57.655 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:57.655 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:57.655 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:57.655 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:57.655 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:57.656 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:57.656 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:57.656 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:57.656 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.619+0000 7f2b3b7fe700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:08:57.657 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.619+0000 7f2b3b7fe700 1 -- 172.21.15.102:0/1368239511 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 464025964 0 0) 0x7f2b5013ba80 con 0x7f2b5012e330 2021-05-25T12:08:57.657 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.619+0000 7f2b3b7fe700 1 -- 172.21.15.102:0/1368239511 >> v1:172.21.15.102:6790/0 conn(0x7f2b50090d70 legacy=0x7f2b50126490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:57.657 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.619+0000 7f2b3b7fe700 1 -- 172.21.15.102:0/1368239511 >> v1:172.21.15.102:6789/0 conn(0x7f2b50090990 legacy=0x7f2b5013ab90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-05-25T12:08:57.657 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.619+0000 7f2b3b7fe700 10 monclient: _finish_hunting 0 2021-05-25T12:08:57.657 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.619+0000 7f2b3b7fe700 1 monclient: found mon.b 2021-05-25T12:08:57.658 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.619+0000 7f2b3b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:08:57.658 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.619+0000 7f2b3b7fe700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T12:08:57.658 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.619+0000 7f2b3b7fe700 1 -- 172.21.15.102:0/1368239511 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b50068920 con 0x7f2b5012e330 2021-05-25T12:08:57.658 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.619+0000 7f2b3b7fe700 10 monclient: _finish_auth 0 2021-05-25T12:08:57.659 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.619+0000 7f2b3b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:08:57.659 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.619+0000 7f2b55d65700 5 monclient: authenticate success, global_id 224551 2021-05-25T12:08:57.659 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.619+0000 7f2b55d65700 10 monclient: _renew_subs 2021-05-25T12:08:57.659 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.619+0000 7f2b55d65700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T12:08:57.659 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.619+0000 7f2b55d65700 1 -- 172.21.15.102:0/1368239511 --> v1:172.21.15.181:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b50066810 con 0x7f2b5012e330 2021-05-25T12:08:57.660 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.619+0000 7f2b55d65700 10 client.224551.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:08:57.660 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.619+0000 7f2b55d65700 10 monclient: _renew_subs 2021-05-25T12:08:57.660 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.619+0000 7f2b55d65700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T12:08:57.660 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.619+0000 7f2b55d65700 1 -- 172.21.15.102:0/1368239511 --> v1:172.21.15.181:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b50126ba0 con 0x7f2b5012e330 2021-05-25T12:08:57.660 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.619+0000 7f2b55d65700 1 librados: init done 2021-05-25T12:08:57.661 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.619+0000 7f2b3b7fe700 1 -- 172.21.15.102:0/1368239511 <== mon.1 v1:172.21.15.181:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2b44002690 con 0x7f2b5012e330 2021-05-25T12:08:57.661 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.619+0000 7f2b3b7fe700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:08:57.661 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.619+0000 7f2b4dffb700 4 set_mon_vals no callback set 2021-05-25T12:08:57.661 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.619+0000 7f2b3b7fe700 1 -- 172.21.15.102:0/1368239511 <== mon.1 v1:172.21.15.181:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f2b440029a0 con 0x7f2b5012e330 2021-05-25T12:08:57.661 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.619+0000 7f2b3b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:08:57.662 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.619+0000 7f2b3b7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T12:08:57.662 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.619+0000 7f2b3b7fe700 10 monclient: dump: 2021-05-25T12:08:57.662 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:08:57.662 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:08:57.663 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:57.663 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:08:57.663 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:08:57.663 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:08:57.663 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:08:57.664 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:08:57.664 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:08:57.664 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:08:57.664 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.620+0000 7f2b55d65700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:08:57.664 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.620+0000 7f2b3b7fe700 1 -- 172.21.15.102:0/1368239511 <== mon.1 v1:172.21.15.181:6789/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f2b44002b80 con 0x7f2b5012e330 2021-05-25T12:08:57.665 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.620+0000 7f2b55d65700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:08:57.665 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.620+0000 7f2b55d65700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T12:08:57.665 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.620+0000 7f2b55d65700 1 -- 172.21.15.102:0/1368239511 --> v1:172.21.15.181:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b28004db0 con 0x7f2b5012e330 2021-05-25T12:08:57.665 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.620+0000 7f2b3b7fe700 1 -- 172.21.15.102:0/1368239511 <== mon.1 v1:172.21.15.181:6789/0 8 ==== osd_map(296..296 src has 250..296) v4 ==== 5459+0+0 (unknown 1521303069 0 0) 0x7f2b4403b610 con 0x7f2b5012e330 2021-05-25T12:08:57.665 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.620+0000 7f2b3b7fe700 10 client.224551.objecter ms_dispatch 0x7f2b50001550 osd_map(296..296 src has 250..296) v4 2021-05-25T12:08:57.666 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.620+0000 7f2b3b7fe700 3 client.224551.objecter handle_osd_map got epochs [296,296] > 0 2021-05-25T12:08:57.666 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.620+0000 7f2b3b7fe700 3 client.224551.objecter handle_osd_map decoding full epoch 296 2021-05-25T12:08:57.666 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.620+0000 7f2b3b7fe700 20 client.224551.objecter dump_active .. 0 homeless 2021-05-25T12:08:57.666 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.620+0000 7f2b4f7fe700 10 monclient: get_auth_request con 0x7f2b300330b0 auth_method 0 2021-05-25T12:08:57.667 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.621+0000 7f2b3b7fe700 10 client.224551.objecter ms_handle_connect 0x7f2b300330b0 2021-05-25T12:08:57.667 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.625+0000 7f2b3b7fe700 1 -- 172.21.15.102:0/1368239511 <== mon.1 v1:172.21.15.181:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7f2b44010e00 con 0x7f2b5012e330 2021-05-25T12:08:57.667 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.625+0000 7f2b3b7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:08:57.667 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.625+0000 7f2b3b7fe700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:08:57.833 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.831+0000 7f2b55d65700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2021-05-25T12:08:57.833 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.831+0000 7f2b55d65700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2021-05-25T12:08:57.834 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.831+0000 7f2b55d65700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.181:6789/0 2021-05-25T12:08:57.834 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.831+0000 7f2b55d65700 1 -- 172.21.15.102:0/1368239511 --> v1:172.21.15.181:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f2b280055e0 con 0x7f2b5012e330 2021-05-25T12:08:57.835 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.832+0000 7f2b3b7fe700 1 -- 172.21.15.102:0/1368239511 <== mon.1 v1:172.21.15.181:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1342 (unknown 254746657 0 2080036891) 0x7f2b440109f0 con 0x7f2b5012e330 2021-05-25T12:08:57.835 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.832+0000 7f2b3b7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2021-05-25T12:08:57.835 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.832+0000 7f2b3b7fe700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:08:57.836 INFO:teuthology.orchestra.run.smithi102.stdout:{"election_epoch":74,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"b","quorum_age":7,"features":{"quorum_con":"4540138303579357183","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"20ee91b6-8935-4b98-b868-85c03789ccc4","modified":"2021-05-25T11:59:11.011522Z","created":"2021-05-25T11:59:11.011522Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6790","nonce":0}]},"addr":"172.21.15.102:6790/0","public_addr":"172.21.15.102:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2021-05-25T12:08:57.837 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.833+0000 7f2b55d65700 10 librados: watch_flush enter 2021-05-25T12:08:57.837 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.833+0000 7f2b55d65700 10 librados: watch_flush exit 2021-05-25T12:08:57.837 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.833+0000 7f2b55d65700 20 client.224551.objecter shutdown clearing up homeless session... 2021-05-25T12:08:57.838 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.833+0000 7f2b55d65700 10 client.224551.objecter successfully canceled tick 2021-05-25T12:08:57.838 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.833+0000 7f2b55d65700 1 -- 172.21.15.102:0/1368239511 >> v1:172.21.15.181:6812/29582 conn(0x7f2b300330b0 legacy=0x7f2b30035570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:57.838 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.833+0000 7f2b55d65700 10 monclient: shutdown 2021-05-25T12:08:57.838 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.833+0000 7f2b55d65700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:08:57.839 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.833+0000 7f2b55d65700 1 -- 172.21.15.102:0/1368239511 >> v1:172.21.15.181:6789/0 conn(0x7f2b5012e330 legacy=0x7f2b50142f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:08:57.839 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.833+0000 7f2b55d65700 1 -- 172.21.15.102:0/1368239511 shutdown_connections 2021-05-25T12:08:57.839 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.833+0000 7f2b55d65700 1 -- 172.21.15.102:0/1368239511 >> 172.21.15.102:0/1368239511 conn(0x7f2b5009b370 msgr2=0x7f2b50129390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:08:57.839 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.833+0000 7f2b55d65700 1 -- 172.21.15.102:0/1368239511 shutdown_connections 2021-05-25T12:08:57.839 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.833+0000 7f2b55d65700 1 -- 172.21.15.102:0/1368239511 wait complete. 2021-05-25T12:08:57.840 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:08:57.833+0000 7f2b55d65700 1 librados: shutdown 2021-05-25T12:09:00.847 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2021-05-25T12:09:00.965 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:00.963+0000 7f98b0ad4700 10 monclient: get_monmap_and_config 2021-05-25T12:09:00.965 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:00.963+0000 7f98b0ad4700 10 monclient: build_initial_monmap 2021-05-25T12:09:00.965 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:00.963+0000 7f98b0ad4700 10 monclient: monmap: 2021-05-25T12:09:00.966 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:09:00.966 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:09:00.966 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:09:00.964456+0000 2021-05-25T12:09:00.966 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:09:00.964456+0000 2021-05-25T12:09:00.967 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:09:00.967 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:09:00.967 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:09:00.967 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:09:00.967 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:09:00.968 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:09:00.968 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:00.964+0000 7f98b0ad4700 1 Processor -- start 2021-05-25T12:09:00.968 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:00.964+0000 7f98b0ad4700 1 -- start start 2021-05-25T12:09:00.968 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:00.964+0000 7f98b0ad4700 10 monclient: init 2021-05-25T12:09:00.970 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:00.964+0000 7f98b0ad4700 10 monclient: _reopen_session rank -1 2021-05-25T12:09:00.970 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:00.964+0000 7f98b0ad4700 10 monclient: _add_conns ranks=[2,1,0] 2021-05-25T12:09:00.970 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:00.964+0000 7f98b0ad4700 10 monclient(hunting): picked mon.noname-b con 0x7f98ac069190 addr v1:172.21.15.102:6790/0 2021-05-25T12:09:00.970 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:00.964+0000 7f98b0ad4700 10 monclient(hunting): picked mon.noname-c con 0x7f98ac069c80 addr v1:172.21.15.181:6789/0 2021-05-25T12:09:00.971 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:00.964+0000 7f98b0ad4700 10 monclient(hunting): picked mon.noname-a con 0x7f98ac096ab0 addr v1:172.21.15.102:6789/0 2021-05-25T12:09:00.971 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:00.964+0000 7f98b0ad4700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98ac0959b0 con 0x7f98ac096ab0 2021-05-25T12:09:00.971 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:00.964+0000 7f98b0ad4700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98ac0914e0 con 0x7f98ac069c80 2021-05-25T12:09:00.971 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:00.964+0000 7f98b0ad4700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98ac095c10 con 0x7f98ac069190 2021-05-25T12:09:00.972 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:00.964+0000 7f98b0ad4700 10 monclient(hunting): _renew_subs 2021-05-25T12:09:00.972 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:00.964+0000 7f98b0ad4700 10 monclient(hunting): authenticate will time out at 1341.561823s 2021-05-25T12:09:00.972 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:00.964+0000 7f98aad9d700 1 -- >> v1:172.21.15.102:6789/0 conn(0x7f98ac096ab0 legacy=0x7f98ac134560 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:09:00.972 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:00.964+0000 7f98aa59c700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f98ac069190 0x7f98ac069570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:57654/0 (socket says 172.21.15.102:57654) 2021-05-25T12:09:00.973 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:00.964+0000 7f98aa59c700 1 -- 172.21.15.102:0/2063570865 learned_addr learned my addr 172.21.15.102:0/2063570865 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:09:01.166 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:01.165+0000 7f98aad9d700 1 -- 172.21.15.102:0/2063570865 >> v1:172.21.15.102:6789/0 conn(0x7f98ac096ab0 legacy=0x7f98ac134560 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:09:01.567 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:01.566+0000 7f98aad9d700 1 -- 172.21.15.102:0/2063570865 >> v1:172.21.15.102:6789/0 conn(0x7f98ac096ab0 legacy=0x7f98ac134560 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:09:02.368 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:02.367+0000 7f98aad9d700 1 -- 172.21.15.102:0/2063570865 >> v1:172.21.15.102:6789/0 conn(0x7f98ac096ab0 legacy=0x7f98ac134560 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:09:03.966 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:03.964+0000 7f9897fff700 10 monclient(hunting): tick 2021-05-25T12:09:03.967 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:03.964+0000 7f9897fff700 1 monclient(hunting): continuing hunt 2021-05-25T12:09:03.967 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:03.964+0000 7f9897fff700 10 monclient(hunting): _reopen_session rank -1 2021-05-25T12:09:03.967 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:03.964+0000 7f9897fff700 1 -- 172.21.15.102:0/2063570865 >> v1:172.21.15.102:6790/0 conn(0x7f98ac069190 legacy=0x7f98ac069570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:09:03.967 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:03.964+0000 7f9897fff700 1 -- 172.21.15.102:0/2063570865 >> v1:172.21.15.181:6789/0 conn(0x7f98ac069c80 legacy=0x7f98ac130e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:09:03.968 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:03.964+0000 7f9897fff700 1 -- 172.21.15.102:0/2063570865 >> v1:172.21.15.102:6789/0 conn(0x7f98ac096ab0 legacy=0x7f98ac134560 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-05-25T12:09:03.968 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:03.964+0000 7f9897fff700 10 monclient: _add_conns ranks=[1,2,0] 2021-05-25T12:09:03.968 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:03.964+0000 7f9897fff700 10 monclient(hunting): picked mon.noname-c con 0x7f98880033e0 addr v1:172.21.15.181:6789/0 2021-05-25T12:09:03.968 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:03.964+0000 7f9897fff700 10 monclient(hunting): picked mon.noname-b con 0x7f9888007070 addr v1:172.21.15.102:6790/0 2021-05-25T12:09:03.968 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:03.964+0000 7f9897fff700 10 monclient(hunting): picked mon.noname-a con 0x7f98ac096ab0 addr v1:172.21.15.102:6789/0 2021-05-25T12:09:03.969 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:03.964+0000 7f9897fff700 1 -- 172.21.15.102:0/2063570865 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98ac095eb0 con 0x7f98ac096ab0 2021-05-25T12:09:03.969 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:03.964+0000 7f9897fff700 1 -- 172.21.15.102:0/2063570865 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98ac0959b0 con 0x7f98880033e0 2021-05-25T12:09:03.969 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:03.965+0000 7f9897fff700 1 -- 172.21.15.102:0/2063570865 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98880105c0 con 0x7f9888007070 2021-05-25T12:09:03.969 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:03.965+0000 7f9897fff700 10 monclient(hunting): _renew_subs 2021-05-25T12:09:03.969 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:03.965+0000 7f98aad9d700 1 -- 172.21.15.102:0/2063570865 >> v1:172.21.15.102:6789/0 conn(0x7f98ac096ab0 legacy=0x7f988800cdf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:09:04.167 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:04.166+0000 7f98aad9d700 1 -- 172.21.15.102:0/2063570865 >> v1:172.21.15.102:6789/0 conn(0x7f98ac096ab0 legacy=0x7f988800cdf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:09:04.569 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:04.567+0000 7f98aad9d700 1 -- 172.21.15.102:0/2063570865 >> v1:172.21.15.102:6789/0 conn(0x7f98ac096ab0 legacy=0x7f988800cdf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:09:05.369 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:05.368+0000 7f98aad9d700 1 -- 172.21.15.102:0/2063570865 >> v1:172.21.15.102:6789/0 conn(0x7f98ac096ab0 legacy=0x7f988800cdf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:09:06.967 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:06.965+0000 7f9897fff700 10 monclient(hunting): tick 2021-05-25T12:09:06.967 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:06.965+0000 7f9897fff700 1 monclient(hunting): continuing hunt 2021-05-25T12:09:06.967 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:06.965+0000 7f9897fff700 10 monclient(hunting): _reopen_session rank -1 2021-05-25T12:09:06.968 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:06.965+0000 7f9897fff700 1 -- 172.21.15.102:0/2063570865 >> v1:172.21.15.102:6790/0 conn(0x7f9888007070 legacy=0x7f98880094e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:09:06.968 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:06.965+0000 7f9897fff700 1 -- 172.21.15.102:0/2063570865 >> v1:172.21.15.181:6789/0 conn(0x7f98880033e0 legacy=0x7f9888005830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:09:06.968 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:06.965+0000 7f9897fff700 1 -- 172.21.15.102:0/2063570865 >> v1:172.21.15.102:6789/0 conn(0x7f98ac096ab0 legacy=0x7f988800cdf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-05-25T12:09:06.968 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:06.965+0000 7f9897fff700 10 monclient: _add_conns ranks=[0,2,1] 2021-05-25T12:09:06.968 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:06.965+0000 7f9897fff700 10 monclient(hunting): picked mon.noname-a con 0x7f98ac096ab0 addr v1:172.21.15.102:6789/0 2021-05-25T12:09:06.969 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:06.965+0000 7f9897fff700 10 monclient(hunting): picked mon.noname-b con 0x7f9888007070 addr v1:172.21.15.102:6790/0 2021-05-25T12:09:06.969 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:06.965+0000 7f9897fff700 10 monclient(hunting): picked mon.noname-c con 0x7f98880033e0 addr v1:172.21.15.181:6789/0 2021-05-25T12:09:06.969 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:06.965+0000 7f9897fff700 1 -- 172.21.15.102:0/2063570865 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98880117d0 con 0x7f98ac096ab0 2021-05-25T12:09:06.969 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:06.965+0000 7f9897fff700 1 -- 172.21.15.102:0/2063570865 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98ac095eb0 con 0x7f98880033e0 2021-05-25T12:09:06.969 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:06.965+0000 7f9897fff700 1 -- 172.21.15.102:0/2063570865 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f988800ce70 con 0x7f9888007070 2021-05-25T12:09:06.970 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:06.965+0000 7f9897fff700 10 monclient(hunting): _renew_subs 2021-05-25T12:09:06.970 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:06.965+0000 7f98aa59c700 1 -- 172.21.15.102:0/2063570865 >> v1:172.21.15.102:6789/0 conn(0x7f98ac096ab0 legacy=0x7f988800f670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:09:07.167 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:07.166+0000 7f98aa59c700 1 -- 172.21.15.102:0/2063570865 >> v1:172.21.15.102:6789/0 conn(0x7f98ac096ab0 legacy=0x7f988800f670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:09:07.568 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:07.567+0000 7f98aa59c700 1 -- 172.21.15.102:0/2063570865 >> v1:172.21.15.102:6789/0 conn(0x7f98ac096ab0 legacy=0x7f988800f670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:09:08.369 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.368+0000 7f98aa59c700 1 -- 172.21.15.102:0/2063570865 >> v1:172.21.15.102:6789/0 conn(0x7f98ac096ab0 legacy=0x7f988800f670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:09:08.671 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.670+0000 7f98a959a700 1 -- 172.21.15.102:0/2063570865 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1586365491 0 0) 0x7f988800ce70 con 0x7f9888007070 2021-05-25T12:09:08.671 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.670+0000 7f98a959a700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:09:08.671 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.670+0000 7f98a959a700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:09:08.672 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.670+0000 7f98a959a700 10 monclient(hunting): my global_id is 244286 2021-05-25T12:09:08.672 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.670+0000 7f98a959a700 1 -- 172.21.15.102:0/2063570865 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9890003270 con 0x7f9888007070 2021-05-25T12:09:08.676 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.670+0000 7f98a959a700 1 -- 172.21.15.102:0/2063570865 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2815262005 0 0) 0x7f9890003270 con 0x7f9888007070 2021-05-25T12:09:08.676 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.670+0000 7f98a959a700 1 -- 172.21.15.102:0/2063570865 >> v1:172.21.15.181:6789/0 conn(0x7f98880033e0 legacy=0x7f9888005830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:09:08.677 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.670+0000 7f98a959a700 1 -- 172.21.15.102:0/2063570865 >> v1:172.21.15.102:6789/0 conn(0x7f98ac096ab0 legacy=0x7f988800f670 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-05-25T12:09:08.677 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.670+0000 7f98a959a700 10 monclient: _finish_hunting 0 2021-05-25T12:09:08.677 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.670+0000 7f98a959a700 1 monclient: found mon.noname-b 2021-05-25T12:09:08.677 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.670+0000 7f98a959a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:09:08.678 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.670+0000 7f98a959a700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T12:09:08.678 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.670+0000 7f98a959a700 1 -- 172.21.15.102:0/2063570865 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f988800d070 con 0x7f9888007070 2021-05-25T12:09:08.678 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.670+0000 7f98a959a700 10 monclient: _finish_auth 0 2021-05-25T12:09:08.678 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.670+0000 7f98a959a700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:09:08.678 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.670+0000 7f98a959a700 1 -- 172.21.15.102:0/2063570865 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f9898001590 con 0x7f9888007070 2021-05-25T12:09:08.679 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.670+0000 7f98b0ad4700 5 monclient: authenticate success, global_id 244286 2021-05-25T12:09:08.679 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.670+0000 7f98b0ad4700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:09:08.679 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.670+0000 7f98a959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:09:08.679 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.670+0000 7f98a959a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:09:08.680 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.670+0000 7f98a959a700 10 monclient: dump: 2021-05-25T12:09:08.680 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:09:08.680 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:09:08.680 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:09:08.680 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:09:08.681 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:09:08.681 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:09:08.681 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:09:08.681 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:09:08.682 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:09:08.682 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:09:08.682 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.670+0000 7f98b0ad4700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:09:08.682 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.671+0000 7f98a959a700 1 -- 172.21.15.102:0/2063570865 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f98980017e0 con 0x7f9888007070 2021-05-25T12:09:08.682 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.671+0000 7f98a959a700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:09:08.683 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.671+0000 7f98b0ad4700 10 monclient: get_monmap_and_config success 2021-05-25T12:09:08.683 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.671+0000 7f98b0ad4700 4 set_mon_vals no callback set 2021-05-25T12:09:08.683 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.672+0000 7f98a959a700 1 -- 172.21.15.102:0/2063570865 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f9898001bb0 con 0x7f9888007070 2021-05-25T12:09:08.683 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.672+0000 7f98a959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:09:08.683 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.672+0000 7f98a959a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:09:08.684 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.672+0000 7f98a959a700 10 monclient: dump: 2021-05-25T12:09:08.684 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:09:08.684 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:09:08.684 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:09:08.685 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:09:08.685 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:09:08.685 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:09:08.685 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:09:08.685 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:09:08.686 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:09:08.686 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:09:08.686 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.672+0000 7f98b0ad4700 10 monclient: shutdown 2021-05-25T12:09:08.686 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.672+0000 7f98b0ad4700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:09:08.686 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.672+0000 7f98b0ad4700 1 -- 172.21.15.102:0/2063570865 >> v1:172.21.15.102:6790/0 conn(0x7f9888007070 legacy=0x7f98880094e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:09:08.687 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.672+0000 7f98b0ad4700 1 -- 172.21.15.102:0/2063570865 shutdown_connections 2021-05-25T12:09:08.687 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.672+0000 7f98b0ad4700 1 -- 172.21.15.102:0/2063570865 >> 172.21.15.102:0/2063570865 conn(0x7f98ac09af90 msgr2=0x7f98ac09b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:09:08.687 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.672+0000 7f98b0ad4700 1 -- 172.21.15.102:0/2063570865 shutdown_connections 2021-05-25T12:09:08.687 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.672+0000 7f98b0ad4700 1 -- 172.21.15.102:0/2063570865 wait complete. 2021-05-25T12:09:08.687 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.673+0000 7f98b0ad4700 10 monclient: build_initial_monmap 2021-05-25T12:09:08.688 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.673+0000 7f98b0ad4700 10 monclient: monmap: 2021-05-25T12:09:08.688 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:09:08.688 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:09:08.688 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:09:08.688 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:09:08.689 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:09:08.689 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:09:08.689 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:09:08.689 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:09:08.690 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:09:08.690 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:09:08.690 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.673+0000 7f98b0ad4700 1 librados: starting msgr at 2021-05-25T12:09:08.690 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.673+0000 7f98b0ad4700 1 librados: starting objecter 2021-05-25T12:09:08.690 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.673+0000 7f98b0ad4700 1 Processor -- start 2021-05-25T12:09:08.691 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.673+0000 7f98b0ad4700 1 -- start start 2021-05-25T12:09:08.691 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.673+0000 7f98b0ad4700 1 librados: setting wanted keys 2021-05-25T12:09:08.691 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.673+0000 7f98b0ad4700 1 librados: calling monclient init 2021-05-25T12:09:08.691 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.673+0000 7f98b0ad4700 10 monclient: init 2021-05-25T12:09:08.691 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.674+0000 7f98b0ad4700 10 monclient: _reopen_session rank -1 2021-05-25T12:09:08.692 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.674+0000 7f98b0ad4700 10 monclient: _add_conns ranks=[1,0,2] 2021-05-25T12:09:08.692 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.674+0000 7f98b0ad4700 10 monclient(hunting): picked mon.noname-b con 0x7f98880033e0 addr v1:172.21.15.181:6789/0 2021-05-25T12:09:08.692 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.674+0000 7f98b0ad4700 10 monclient(hunting): picked mon.noname-a con 0x7f9888007070 addr v1:172.21.15.102:6789/0 2021-05-25T12:09:08.692 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.674+0000 7f98b0ad4700 10 monclient(hunting): picked mon.noname-c con 0x7f98ac096ab0 addr v1:172.21.15.102:6790/0 2021-05-25T12:09:08.692 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.674+0000 7f98b0ad4700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98ac13bc40 con 0x7f9888007070 2021-05-25T12:09:08.693 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.674+0000 7f98b0ad4700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98ac118c90 con 0x7f98880033e0 2021-05-25T12:09:08.693 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.674+0000 7f98b0ad4700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98ac118f30 con 0x7f98ac096ab0 2021-05-25T12:09:08.693 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.674+0000 7f98b0ad4700 10 monclient(hunting): _renew_subs 2021-05-25T12:09:08.693 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.674+0000 7f98b0ad4700 10 monclient(hunting): authenticate will time out at 1349.271270s 2021-05-25T12:09:08.693 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.674+0000 7f98a9d9b700 1 -- >> v1:172.21.15.102:6789/0 conn(0x7f9888007070 legacy=0x7f98ac140120 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:09:08.694 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.674+0000 7f98aad9d700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f98ac096ab0 0x7f98ac143850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:57694/0 (socket says 172.21.15.102:57694) 2021-05-25T12:09:08.694 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.674+0000 7f98aad9d700 1 -- 172.21.15.102:0/871915377 learned_addr learned my addr 172.21.15.102:0/871915377 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:09:08.694 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.674+0000 7f98967fc700 10 client.?.objecter ms_handle_connect 0x7f98ac096ab0 2021-05-25T12:09:08.694 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.674+0000 7f98967fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:09:08.694 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.674+0000 7f98967fc700 10 client.?.objecter ms_handle_connect 0x7f98880033e0 2021-05-25T12:09:08.695 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.674+0000 7f98967fc700 10 client.?.objecter resend_mon_ops 2021-05-25T12:09:08.699 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.676+0000 7f98967fc700 1 -- 172.21.15.102:0/871915377 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1566573013 0 0) 0x7f98ac118c90 con 0x7f98880033e0 2021-05-25T12:09:08.699 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.676+0000 7f98967fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:09:08.699 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.676+0000 7f98967fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:09:08.700 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.676+0000 7f98967fc700 10 monclient(hunting): my global_id is 224554 2021-05-25T12:09:08.700 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.676+0000 7f98967fc700 1 -- 172.21.15.102:0/871915377 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98880117d0 con 0x7f98880033e0 2021-05-25T12:09:08.700 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.676+0000 7f98967fc700 1 -- 172.21.15.102:0/871915377 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2527385882 0 0) 0x7f98ac118f30 con 0x7f98ac096ab0 2021-05-25T12:09:08.700 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.676+0000 7f98967fc700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:09:08.700 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.676+0000 7f98967fc700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:09:08.701 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.676+0000 7f98967fc700 10 monclient(hunting): my global_id is 244289 2021-05-25T12:09:08.701 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.676+0000 7f98967fc700 1 -- 172.21.15.102:0/871915377 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98ac118c90 con 0x7f98ac096ab0 2021-05-25T12:09:08.701 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.676+0000 7f98967fc700 1 -- 172.21.15.102:0/871915377 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2570037023 0 0) 0x7f98ac118c90 con 0x7f98ac096ab0 2021-05-25T12:09:08.701 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.676+0000 7f98967fc700 1 -- 172.21.15.102:0/871915377 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f98ac118f30 con 0x7f98ac096ab0 2021-05-25T12:09:08.701 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.676+0000 7f98967fc700 1 -- 172.21.15.102:0/871915377 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f98a0004060 con 0x7f98ac096ab0 2021-05-25T12:09:08.702 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.676+0000 7f98967fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:09:08.702 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.676+0000 7f98967fc700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:09:08.702 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.676+0000 7f98967fc700 10 monclient(hunting): dump: 2021-05-25T12:09:08.702 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:09:08.703 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:09:08.703 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:09:08.703 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:09:08.703 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:09:08.703 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:09:08.704 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:09:08.704 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:09:08.704 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:09:08.704 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:09:08.704 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.676+0000 7f98967fc700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:09:08.705 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.676+0000 7f98967fc700 1 -- 172.21.15.102:0/871915377 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2482537283 0 0) 0x7f98880117d0 con 0x7f98880033e0 2021-05-25T12:09:08.705 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.676+0000 7f98967fc700 1 -- 172.21.15.102:0/871915377 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f98ac118c90 con 0x7f98880033e0 2021-05-25T12:09:08.705 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.676+0000 7f98967fc700 1 -- 172.21.15.102:0/871915377 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f989c0019f0 con 0x7f98880033e0 2021-05-25T12:09:08.705 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.676+0000 7f98967fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:09:08.706 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.676+0000 7f98967fc700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2021-05-25T12:09:08.706 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.676+0000 7f98967fc700 10 monclient(hunting): dump: 2021-05-25T12:09:08.706 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:09:08.706 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:09:08.706 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:09:08.707 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:09:08.707 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:09:08.707 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:09:08.707 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:09:08.707 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:09:08.708 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:09:08.708 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:09:08.708 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.677+0000 7f98967fc700 1 -- 172.21.15.102:0/871915377 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1623672547 0 0) 0x7f98ac118f30 con 0x7f98ac096ab0 2021-05-25T12:09:08.708 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.677+0000 7f98967fc700 1 -- 172.21.15.102:0/871915377 >> v1:172.21.15.181:6789/0 conn(0x7f98880033e0 legacy=0x7f98ac12b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:09:08.709 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.677+0000 7f98967fc700 1 -- 172.21.15.102:0/871915377 >> v1:172.21.15.102:6789/0 conn(0x7f9888007070 legacy=0x7f98ac140120 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-05-25T12:09:08.709 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.677+0000 7f98967fc700 10 monclient: _finish_hunting 0 2021-05-25T12:09:08.709 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.677+0000 7f98967fc700 1 monclient: found mon.c 2021-05-25T12:09:08.709 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.677+0000 7f98967fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:09:08.709 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.677+0000 7f98967fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:09:08.710 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.677+0000 7f98967fc700 1 -- 172.21.15.102:0/871915377 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98ac117060 con 0x7f98ac096ab0 2021-05-25T12:09:08.710 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.677+0000 7f98967fc700 10 monclient: _finish_auth 0 2021-05-25T12:09:08.710 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.677+0000 7f98967fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:09:08.710 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.677+0000 7f98b0ad4700 5 monclient: authenticate success, global_id 244289 2021-05-25T12:09:08.711 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.677+0000 7f98b0ad4700 10 monclient: _renew_subs 2021-05-25T12:09:08.711 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.677+0000 7f98b0ad4700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:09:08.711 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.677+0000 7f98b0ad4700 1 -- 172.21.15.102:0/871915377 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98ac12bea0 con 0x7f98ac096ab0 2021-05-25T12:09:08.711 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.677+0000 7f98b0ad4700 10 client.244289.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:09:08.711 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.677+0000 7f98b0ad4700 10 monclient: _renew_subs 2021-05-25T12:09:08.712 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.677+0000 7f98b0ad4700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:09:08.712 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.677+0000 7f98b0ad4700 1 -- 172.21.15.102:0/871915377 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f98ac1164a0 con 0x7f98ac096ab0 2021-05-25T12:09:08.712 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.677+0000 7f98b0ad4700 1 librados: init done 2021-05-25T12:09:08.712 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.677+0000 7f98967fc700 1 -- 172.21.15.102:0/871915377 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f98a00045f0 con 0x7f98ac096ab0 2021-05-25T12:09:08.712 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.677+0000 7f98967fc700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:09:08.713 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.677+0000 7f98967fc700 1 -- 172.21.15.102:0/871915377 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f98a0004980 con 0x7f98ac096ab0 2021-05-25T12:09:08.713 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.677+0000 7f98967fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:09:08.713 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.677+0000 7f98967fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:09:08.713 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.677+0000 7f98a8d99700 4 set_mon_vals no callback set 2021-05-25T12:09:08.713 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.677+0000 7f98967fc700 10 monclient: dump: 2021-05-25T12:09:08.714 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:09:08.714 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:09:08.714 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:09:08.714 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:09:08.715 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:09:08.715 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:09:08.715 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:09:08.715 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:09:08.715 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:09:08.716 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:09:08.716 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.677+0000 7f98967fc700 1 -- 172.21.15.102:0/871915377 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f98a0004b60 con 0x7f98ac096ab0 2021-05-25T12:09:08.716 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.678+0000 7f98967fc700 1 -- 172.21.15.102:0/871915377 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(297..297 src has 250..297) v4 ==== 5744+0+0 (unknown 282307743 0 0) 0x7f98a003d850 con 0x7f98ac096ab0 2021-05-25T12:09:08.716 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.678+0000 7f98b0ad4700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:09:08.716 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.678+0000 7f98967fc700 10 client.244289.objecter ms_dispatch 0x7f98ac001550 osd_map(297..297 src has 250..297) v4 2021-05-25T12:09:08.717 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.678+0000 7f98967fc700 3 client.244289.objecter handle_osd_map got epochs [297,297] > 0 2021-05-25T12:09:08.717 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.678+0000 7f98967fc700 3 client.244289.objecter handle_osd_map decoding full epoch 297 2021-05-25T12:09:08.717 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.678+0000 7f98b0ad4700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:09:08.717 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.678+0000 7f98b0ad4700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:09:08.717 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.678+0000 7f98b0ad4700 1 -- 172.21.15.102:0/871915377 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f988c004db0 con 0x7f98ac096ab0 2021-05-25T12:09:08.718 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.678+0000 7f98967fc700 20 client.244289.objecter dump_active .. 0 homeless 2021-05-25T12:09:08.718 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.678+0000 7f98aa59c700 10 monclient: get_auth_request con 0x7f98880341b0 auth_method 0 2021-05-25T12:09:08.718 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.678+0000 7f98967fc700 10 client.244289.objecter ms_handle_connect 0x7f98880341b0 2021-05-25T12:09:08.718 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.682+0000 7f98967fc700 1 -- 172.21.15.102:0/871915377 <== mon.2 v1:172.21.15.102:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152788 (unknown 1092875540 0 4120683270) 0x7f98a0012eb0 con 0x7f98ac096ab0 2021-05-25T12:09:08.718 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.682+0000 7f98967fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:09:08.719 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.682+0000 7f98967fc700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:09:08.898 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.897+0000 7f98b0ad4700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2021-05-25T12:09:08.899 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.897+0000 7f98b0ad4700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2021-05-25T12:09:08.899 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.897+0000 7f98b0ad4700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:09:08.899 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.897+0000 7f98b0ad4700 1 -- 172.21.15.102:0/871915377 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f988c005060 con 0x7f98ac096ab0 2021-05-25T12:09:08.900 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.898+0000 7f98967fc700 1 -- 172.21.15.102:0/871915377 <== mon.2 v1:172.21.15.102:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1336 (unknown 254746657 0 4143514067) 0x7f98a0012aa0 con 0x7f98ac096ab0 2021-05-25T12:09:08.900 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.898+0000 7f98967fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2021-05-25T12:09:08.901 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.898+0000 7f98967fc700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:09:08.901 INFO:teuthology.orchestra.run.smithi102.stdout:{"election_epoch":78,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138303579357183","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"20ee91b6-8935-4b98-b868-85c03789ccc4","modified":"2021-05-25T11:59:11.011522Z","created":"2021-05-25T11:59:11.011522Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.102:6790","nonce":0}]},"addr":"172.21.15.102:6790/0","public_addr":"172.21.15.102:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2021-05-25T12:09:08.903 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.899+0000 7f98b0ad4700 10 librados: watch_flush enter 2021-05-25T12:09:08.903 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.899+0000 7f98b0ad4700 10 librados: watch_flush exit 2021-05-25T12:09:08.903 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.899+0000 7f98b0ad4700 20 client.244289.objecter shutdown clearing up homeless session... 2021-05-25T12:09:08.903 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.899+0000 7f98b0ad4700 10 client.244289.objecter successfully canceled tick 2021-05-25T12:09:08.903 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.899+0000 7f98b0ad4700 1 -- 172.21.15.102:0/871915377 >> v1:172.21.15.181:6812/29582 conn(0x7f98880341b0 legacy=0x7f9888036670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:09:08.904 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.899+0000 7f98b0ad4700 10 monclient: shutdown 2021-05-25T12:09:08.904 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.899+0000 7f98b0ad4700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:09:08.904 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.899+0000 7f98b0ad4700 1 -- 172.21.15.102:0/871915377 >> v1:172.21.15.102:6790/0 conn(0x7f98ac096ab0 legacy=0x7f98ac143850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:09:08.904 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.899+0000 7f98b0ad4700 1 -- 172.21.15.102:0/871915377 shutdown_connections 2021-05-25T12:09:08.904 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.899+0000 7f98b0ad4700 1 -- 172.21.15.102:0/871915377 >> 172.21.15.102:0/871915377 conn(0x7f98ac09af90 msgr2=0x7f98ac094a90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:09:08.905 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.899+0000 7f98b0ad4700 1 -- 172.21.15.102:0/871915377 shutdown_connections 2021-05-25T12:09:08.905 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.899+0000 7f98b0ad4700 1 -- 172.21.15.102:0/871915377 wait complete. 2021-05-25T12:09:08.905 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:08.899+0000 7f98b0ad4700 1 librados: shutdown 2021-05-25T12:09:08.912 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2021-05-25T12:09:08.912 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2021-05-25T12:09:09.022 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.021+0000 7fc8d95ac700 10 monclient: get_monmap_and_config 2021-05-25T12:09:09.022 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.021+0000 7fc8d95ac700 10 monclient: build_initial_monmap 2021-05-25T12:09:09.022 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.021+0000 7fc8d95ac700 10 monclient: monmap: 2021-05-25T12:09:09.023 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:09:09.023 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:09:09.023 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:09:09.022006+0000 2021-05-25T12:09:09.023 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:09:09.022006+0000 2021-05-25T12:09:09.024 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:09:09.024 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:09:09.024 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:09:09.024 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:09:09.025 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:09:09.025 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:09:09.036 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.021+0000 7fc8d95ac700 1 Processor -- start 2021-05-25T12:09:09.036 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.021+0000 7fc8d95ac700 1 -- start start 2021-05-25T12:09:09.037 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.021+0000 7fc8d95ac700 10 monclient: init 2021-05-25T12:09:09.037 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.022+0000 7fc8d95ac700 10 monclient: _reopen_session rank -1 2021-05-25T12:09:09.037 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.022+0000 7fc8d95ac700 10 monclient: _add_conns ranks=[0,1,2] 2021-05-25T12:09:09.037 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.022+0000 7fc8d95ac700 10 monclient(hunting): picked mon.noname-a con 0x7fc8d4069190 addr v1:172.21.15.102:6789/0 2021-05-25T12:09:09.038 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.022+0000 7fc8d95ac700 10 monclient(hunting): picked mon.noname-c con 0x7fc8d4069c80 addr v1:172.21.15.181:6789/0 2021-05-25T12:09:09.038 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.022+0000 7fc8d95ac700 10 monclient(hunting): picked mon.noname-b con 0x7fc8d4096ab0 addr v1:172.21.15.102:6790/0 2021-05-25T12:09:09.038 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.022+0000 7fc8d95ac700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8d40959b0 con 0x7fc8d4069190 2021-05-25T12:09:09.038 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.022+0000 7fc8d95ac700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8d40914e0 con 0x7fc8d4069c80 2021-05-25T12:09:09.038 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.022+0000 7fc8d95ac700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8d4095c10 con 0x7fc8d4096ab0 2021-05-25T12:09:09.039 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.022+0000 7fc8d95ac700 10 monclient(hunting): _renew_subs 2021-05-25T12:09:09.039 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.022+0000 7fc8d95ac700 10 monclient(hunting): authenticate will time out at 1349.619151s 2021-05-25T12:09:09.039 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.022+0000 7fc8d2ffd700 1 -- >> v1:172.21.15.102:6789/0 conn(0x7fc8d4069190 legacy=0x7fc8d4069570 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:09:09.039 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.022+0000 7fc8d37fe700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fc8d4096ab0 0x7fc8d4134560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:57702/0 (socket says 172.21.15.102:57702) 2021-05-25T12:09:09.039 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.022+0000 7fc8d37fe700 1 -- 172.21.15.102:0/3506434252 learned_addr learned my addr 172.21.15.102:0/3506434252 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:09:09.040 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.022+0000 7fc8d1ffb700 1 -- 172.21.15.102:0/3506434252 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2806194281 0 0) 0x7fc8d4095c10 con 0x7fc8d4096ab0 2021-05-25T12:09:09.040 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.022+0000 7fc8d1ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:09:09.040 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.022+0000 7fc8d1ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:09:09.040 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.022+0000 7fc8d1ffb700 10 monclient(hunting): my global_id is 244292 2021-05-25T12:09:09.040 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.022+0000 7fc8d1ffb700 1 -- 172.21.15.102:0/3506434252 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8c0003270 con 0x7fc8d4096ab0 2021-05-25T12:09:09.041 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.022+0000 7fc8d1ffb700 1 -- 172.21.15.102:0/3506434252 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3044770520 0 0) 0x7fc8c0003270 con 0x7fc8d4096ab0 2021-05-25T12:09:09.041 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.022+0000 7fc8d1ffb700 1 -- 172.21.15.102:0/3506434252 >> v1:172.21.15.181:6789/0 conn(0x7fc8d4069c80 legacy=0x7fc8d4130e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:09:09.041 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.022+0000 7fc8d1ffb700 1 -- 172.21.15.102:0/3506434252 >> v1:172.21.15.102:6789/0 conn(0x7fc8d4069190 legacy=0x7fc8d4069570 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-05-25T12:09:09.041 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.022+0000 7fc8d1ffb700 10 monclient: _finish_hunting 0 2021-05-25T12:09:09.041 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.022+0000 7fc8d1ffb700 1 monclient: found mon.noname-b 2021-05-25T12:09:09.042 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.022+0000 7fc8d1ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:09:09.042 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.022+0000 7fc8d1ffb700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T12:09:09.042 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.022+0000 7fc8d1ffb700 1 -- 172.21.15.102:0/3506434252 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8d4095eb0 con 0x7fc8d4096ab0 2021-05-25T12:09:09.042 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.022+0000 7fc8d1ffb700 10 monclient: _finish_auth 0 2021-05-25T12:09:09.043 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.022+0000 7fc8d1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:09:09.043 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.022+0000 7fc8d95ac700 5 monclient: authenticate success, global_id 244292 2021-05-25T12:09:09.043 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.022+0000 7fc8d1ffb700 1 -- 172.21.15.102:0/3506434252 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fc8c4001990 con 0x7fc8d4096ab0 2021-05-25T12:09:09.043 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.022+0000 7fc8d95ac700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:09:09.043 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.022+0000 7fc8d1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:09:09.044 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.022+0000 7fc8d1ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:09:09.044 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.022+0000 7fc8d1ffb700 10 monclient: dump: 2021-05-25T12:09:09.044 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:09:09.044 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:09:09.044 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:09:09.045 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:09:09.045 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:09:09.045 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:09:09.045 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:09:09.045 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:09:09.046 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:09:09.046 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:09:09.046 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.022+0000 7fc8d95ac700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:09:09.046 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.022+0000 7fc8d1ffb700 1 -- 172.21.15.102:0/3506434252 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc8c4001cb0 con 0x7fc8d4096ab0 2021-05-25T12:09:09.047 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.022+0000 7fc8d1ffb700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:09:09.047 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.022+0000 7fc8d1ffb700 1 -- 172.21.15.102:0/3506434252 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fc8c4002040 con 0x7fc8d4096ab0 2021-05-25T12:09:09.047 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.022+0000 7fc8d95ac700 10 monclient: get_monmap_and_config success 2021-05-25T12:09:09.047 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.022+0000 7fc8d95ac700 4 set_mon_vals no callback set 2021-05-25T12:09:09.047 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.022+0000 7fc8d1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:09:09.048 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.022+0000 7fc8d1ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:09:09.048 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.022+0000 7fc8d1ffb700 10 monclient: dump: 2021-05-25T12:09:09.048 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:09:09.048 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:09:09.048 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:09:09.049 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:09:09.049 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:09:09.049 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:09:09.049 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:09:09.049 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:09:09.049 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:09:09.050 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:09:09.050 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.023+0000 7fc8d95ac700 10 monclient: shutdown 2021-05-25T12:09:09.050 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.023+0000 7fc8d95ac700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:09:09.050 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.023+0000 7fc8d95ac700 1 -- 172.21.15.102:0/3506434252 >> v1:172.21.15.102:6790/0 conn(0x7fc8d4096ab0 legacy=0x7fc8d4134560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:09:09.050 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.023+0000 7fc8d95ac700 1 -- 172.21.15.102:0/3506434252 shutdown_connections 2021-05-25T12:09:09.051 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.023+0000 7fc8d95ac700 1 -- 172.21.15.102:0/3506434252 >> 172.21.15.102:0/3506434252 conn(0x7fc8d409af90 msgr2=0x7fc8d409b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:09:09.051 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.023+0000 7fc8d95ac700 1 -- 172.21.15.102:0/3506434252 shutdown_connections 2021-05-25T12:09:09.051 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.023+0000 7fc8d95ac700 1 -- 172.21.15.102:0/3506434252 wait complete. 2021-05-25T12:09:09.051 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.023+0000 7fc8d95ac700 10 monclient: build_initial_monmap 2021-05-25T12:09:09.051 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.023+0000 7fc8d95ac700 10 monclient: monmap: 2021-05-25T12:09:09.052 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:09:09.052 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:09:09.052 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:09:09.052 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:09:09.052 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:09:09.052 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:09:09.053 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:09:09.053 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:09:09.053 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:09:09.053 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:09:09.053 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.024+0000 7fc8d95ac700 1 librados: starting msgr at 2021-05-25T12:09:09.054 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.024+0000 7fc8d95ac700 1 librados: starting objecter 2021-05-25T12:09:09.054 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.024+0000 7fc8d95ac700 1 Processor -- start 2021-05-25T12:09:09.054 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.024+0000 7fc8d95ac700 1 -- start start 2021-05-25T12:09:09.054 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.024+0000 7fc8d95ac700 1 librados: setting wanted keys 2021-05-25T12:09:09.054 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.024+0000 7fc8d95ac700 1 librados: calling monclient init 2021-05-25T12:09:09.055 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.024+0000 7fc8d95ac700 10 monclient: init 2021-05-25T12:09:09.055 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.024+0000 7fc8d95ac700 10 monclient: _reopen_session rank -1 2021-05-25T12:09:09.055 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.024+0000 7fc8d95ac700 10 monclient: _add_conns ranks=[2,1,0] 2021-05-25T12:09:09.055 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.024+0000 7fc8d95ac700 10 monclient(hunting): picked mon.noname-c con 0x7fc8d4069190 addr v1:172.21.15.102:6790/0 2021-05-25T12:09:09.055 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.024+0000 7fc8d95ac700 10 monclient(hunting): picked mon.noname-b con 0x7fc8d4069c80 addr v1:172.21.15.181:6789/0 2021-05-25T12:09:09.056 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.024+0000 7fc8d95ac700 10 monclient(hunting): picked mon.noname-a con 0x7fc8d4096ab0 addr v1:172.21.15.102:6789/0 2021-05-25T12:09:09.056 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.024+0000 7fc8d95ac700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8d411a7d0 con 0x7fc8d4096ab0 2021-05-25T12:09:09.056 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.024+0000 7fc8d95ac700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8d411aa00 con 0x7fc8d4069c80 2021-05-25T12:09:09.056 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.024+0000 7fc8d95ac700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8d411aca0 con 0x7fc8d4069190 2021-05-25T12:09:09.057 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.024+0000 7fc8d95ac700 10 monclient(hunting): _renew_subs 2021-05-25T12:09:09.057 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.024+0000 7fc8d95ac700 10 monclient(hunting): authenticate will time out at 1349.621761s 2021-05-25T12:09:09.057 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.024+0000 7fc8d37fe700 1 -- >> v1:172.21.15.102:6789/0 conn(0x7fc8d4096ab0 legacy=0x7fc8d4143700 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:09:09.057 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.024+0000 7fc8d2ffd700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fc8d4069190 0x7fc8d413b2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:57706/0 (socket says 172.21.15.102:57706) 2021-05-25T12:09:09.057 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.024+0000 7fc8d2ffd700 1 -- 172.21.15.102:0/915905625 learned_addr learned my addr 172.21.15.102:0/915905625 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:09:09.058 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.024+0000 7fc8baffd700 10 client.?.objecter ms_handle_connect 0x7fc8d4069190 2021-05-25T12:09:09.058 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.024+0000 7fc8baffd700 10 client.?.objecter resend_mon_ops 2021-05-25T12:09:09.058 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.025+0000 7fc8baffd700 1 -- 172.21.15.102:0/915905625 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2708007938 0 0) 0x7fc8d411aca0 con 0x7fc8d4069190 2021-05-25T12:09:09.058 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.025+0000 7fc8baffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:09:09.058 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.025+0000 7fc8baffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:09:09.059 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.025+0000 7fc8baffd700 10 monclient(hunting): my global_id is 244295 2021-05-25T12:09:09.059 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.025+0000 7fc8baffd700 1 -- 172.21.15.102:0/915905625 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8b4004c10 con 0x7fc8d4069190 2021-05-25T12:09:09.059 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.025+0000 7fc8baffd700 10 client.?.objecter ms_handle_connect 0x7fc8d4069c80 2021-05-25T12:09:09.059 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.025+0000 7fc8baffd700 10 client.?.objecter resend_mon_ops 2021-05-25T12:09:09.059 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.025+0000 7fc8baffd700 1 -- 172.21.15.102:0/915905625 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1045028114 0 0) 0x7fc8d411aa00 con 0x7fc8d4069c80 2021-05-25T12:09:09.060 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.025+0000 7fc8baffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:09:09.060 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.025+0000 7fc8baffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:09:09.060 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.025+0000 7fc8baffd700 10 monclient(hunting): my global_id is 224560 2021-05-25T12:09:09.060 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.025+0000 7fc8baffd700 1 -- 172.21.15.102:0/915905625 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8d411aca0 con 0x7fc8d4069c80 2021-05-25T12:09:09.060 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.025+0000 7fc8baffd700 1 -- 172.21.15.102:0/915905625 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1485294505 0 0) 0x7fc8b4004c10 con 0x7fc8d4069190 2021-05-25T12:09:09.061 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.025+0000 7fc8baffd700 1 -- 172.21.15.102:0/915905625 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc8d411aa00 con 0x7fc8d4069190 2021-05-25T12:09:09.061 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.025+0000 7fc8baffd700 1 -- 172.21.15.102:0/915905625 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fc8bc003ef0 con 0x7fc8d4069190 2021-05-25T12:09:09.061 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.025+0000 7fc8baffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:09:09.061 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.025+0000 7fc8baffd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:09:09.061 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.025+0000 7fc8baffd700 10 monclient(hunting): dump: 2021-05-25T12:09:09.061 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:09:09.062 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:09:09.062 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:09:09.062 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:09:09.062 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:09:09.062 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:09:09.063 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:09:09.063 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:09:09.063 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:09:09.063 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:09:09.064 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.025+0000 7fc8baffd700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:09:09.064 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.025+0000 7fc8baffd700 1 -- 172.21.15.102:0/915905625 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1147840288 0 0) 0x7fc8d411aa00 con 0x7fc8d4069190 2021-05-25T12:09:09.064 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.025+0000 7fc8baffd700 1 -- 172.21.15.102:0/915905625 >> v1:172.21.15.181:6789/0 conn(0x7fc8d4069c80 legacy=0x7fc8d413ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:09:09.064 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.025+0000 7fc8baffd700 1 -- 172.21.15.102:0/915905625 >> v1:172.21.15.102:6789/0 conn(0x7fc8d4096ab0 legacy=0x7fc8d4143700 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-05-25T12:09:09.064 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.025+0000 7fc8baffd700 10 monclient: _finish_hunting 0 2021-05-25T12:09:09.065 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.025+0000 7fc8baffd700 1 monclient: found mon.c 2021-05-25T12:09:09.065 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.025+0000 7fc8baffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:09:09.065 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.025+0000 7fc8baffd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:09:09.065 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.025+0000 7fc8baffd700 1 -- 172.21.15.102:0/915905625 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8d411af40 con 0x7fc8d4069190 2021-05-25T12:09:09.065 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.025+0000 7fc8baffd700 10 monclient: _finish_auth 0 2021-05-25T12:09:09.066 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.025+0000 7fc8baffd700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:09:09.066 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.025+0000 7fc8d95ac700 5 monclient: authenticate success, global_id 244295 2021-05-25T12:09:09.066 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.025+0000 7fc8d95ac700 10 monclient: _renew_subs 2021-05-25T12:09:09.066 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.025+0000 7fc8d95ac700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:09:09.066 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.025+0000 7fc8d95ac700 1 -- 172.21.15.102:0/915905625 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8d4118e30 con 0x7fc8d4069190 2021-05-25T12:09:09.067 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.025+0000 7fc8d95ac700 10 client.244295.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:09:09.067 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.025+0000 7fc8d95ac700 10 monclient: _renew_subs 2021-05-25T12:09:09.067 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.025+0000 7fc8d95ac700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:09:09.067 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.025+0000 7fc8d95ac700 1 -- 172.21.15.102:0/915905625 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8d413ba30 con 0x7fc8d4069190 2021-05-25T12:09:09.067 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.025+0000 7fc8d95ac700 1 librados: init done 2021-05-25T12:09:09.068 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.025+0000 7fc8baffd700 1 -- 172.21.15.102:0/915905625 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc8bc004110 con 0x7fc8d4069190 2021-05-25T12:09:09.068 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.025+0000 7fc8baffd700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:09:09.068 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.025+0000 7fc8baffd700 1 -- 172.21.15.102:0/915905625 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fc8bc0044a0 con 0x7fc8d4069190 2021-05-25T12:09:09.068 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.025+0000 7fc8baffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:09:09.068 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.025+0000 7fc8baffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:09:09.069 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.025+0000 7fc8d17fa700 4 set_mon_vals no callback set 2021-05-25T12:09:09.069 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.025+0000 7fc8baffd700 10 monclient: dump: 2021-05-25T12:09:09.069 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:09:09.069 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:09:09.069 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:09:09.070 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:09:09.070 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:09:09.070 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:09:09.070 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:09:09.070 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:09:09.071 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:09:09.071 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:09:09.071 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.026+0000 7fc8baffd700 1 -- 172.21.15.102:0/915905625 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7fc8bc004680 con 0x7fc8d4069190 2021-05-25T12:09:09.071 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.026+0000 7fc8baffd700 1 -- 172.21.15.102:0/915905625 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(297..297 src has 250..297) v4 ==== 5744+0+0 (unknown 282307743 0 0) 0x7fc8bc03d720 con 0x7fc8d4069190 2021-05-25T12:09:09.071 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.026+0000 7fc8baffd700 10 client.244295.objecter ms_dispatch 0x7fc8d4001550 osd_map(297..297 src has 250..297) v4 2021-05-25T12:09:09.072 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.026+0000 7fc8baffd700 3 client.244295.objecter handle_osd_map got epochs [297,297] > 0 2021-05-25T12:09:09.072 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.026+0000 7fc8baffd700 3 client.244295.objecter handle_osd_map decoding full epoch 297 2021-05-25T12:09:09.072 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.026+0000 7fc8d95ac700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:09:09.072 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.026+0000 7fc8d95ac700 1 -- 172.21.15.102:0/915905625 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fc8ac0065a0 con 0x7fc8ac002840 2021-05-25T12:09:09.072 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.026+0000 7fc8baffd700 20 client.244295.objecter dump_active .. 0 homeless 2021-05-25T12:09:09.073 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.026+0000 7fc8baffd700 10 client.244295.objecter ms_handle_connect 0x7fc8ac002840 2021-05-25T12:09:09.073 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.026+0000 7fc8baffd700 10 client.244295.objecter resend_mon_ops 2021-05-25T12:09:09.073 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.026+0000 7fc8d27fc700 10 monclient: get_auth_request con 0x7fc8b4033180 auth_method 0 2021-05-25T12:09:09.073 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.026+0000 7fc8baffd700 1 -- 172.21.15.102:0/915905625 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 101472910 0 0) 0x7fc8ac0065a0 con 0x7fc8ac002840 2021-05-25T12:09:09.073 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.026+0000 7fc8baffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:09:09.074 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.026+0000 7fc8baffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:09:09.074 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.026+0000 7fc8baffd700 10 monclient(hunting): my global_id is 244298 2021-05-25T12:09:09.074 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.026+0000 7fc8baffd700 1 -- 172.21.15.102:0/915905625 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8d411aa00 con 0x7fc8ac002840 2021-05-25T12:09:09.074 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.026+0000 7fc8baffd700 1 -- 172.21.15.102:0/915905625 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3575506082 0 0) 0x7fc8d411aa00 con 0x7fc8ac002840 2021-05-25T12:09:09.074 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.026+0000 7fc8baffd700 1 -- 172.21.15.102:0/915905625 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc8ac0065a0 con 0x7fc8ac002840 2021-05-25T12:09:09.075 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.026+0000 7fc8baffd700 10 client.244295.objecter ms_handle_connect 0x7fc8b4033180 2021-05-25T12:09:09.075 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.026+0000 7fc8baffd700 1 -- 172.21.15.102:0/915905625 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fc8c4001e20 con 0x7fc8ac002840 2021-05-25T12:09:09.075 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.026+0000 7fc8baffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:09:09.075 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.026+0000 7fc8baffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:09:09.075 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.026+0000 7fc8baffd700 10 monclient: dump: 2021-05-25T12:09:09.076 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:09:09.076 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:09:09.076 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:09:09.076 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:09:09.076 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:09:09.076 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:09:09.077 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:09:09.077 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:09:09.077 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:09:09.077 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:09:09.077 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.026+0000 7fc8baffd700 1 -- 172.21.15.102:0/915905625 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 321679634 0 0) 0x7fc8ac0065a0 con 0x7fc8ac002840 2021-05-25T12:09:09.078 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.026+0000 7fc8baffd700 1 -- 172.21.15.102:0/915905625 --> v1:172.21.15.102:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc8ac009e40 con 0x7fc8ac002840 2021-05-25T12:09:09.078 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.027+0000 7fc8baffd700 1 -- 172.21.15.102:0/915905625 <== mon.2 v1:172.21.15.102:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7fc8c4002050 con 0x7fc8ac002840 2021-05-25T12:09:09.078 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.027+0000 7fc8baffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:09:09.078 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.027+0000 7fc8baffd700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:09:09.078 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.027+0000 7fc8baffd700 1 -- 172.21.15.102:0/915905625 >> v1:172.21.15.102:6790/0 conn(0x7fc8ac002840 legacy=0x7fc8ac004ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:09:09.079 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.027+0000 7fc8baffd700 1 -- 172.21.15.102:0/915905625 >> v1:172.21.15.102:6790/0 conn(0x7fc8ac002840 legacy=0x7fc8ac004ce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:09:09.079 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.033+0000 7fc8d95ac700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:09:09.079 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.033+0000 7fc8d95ac700 1 -- 172.21.15.102:0/915905625 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fc8ac007510 con 0x7fc8ac0068b0 2021-05-25T12:09:09.079 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.033+0000 7fc8baffd700 10 client.244295.objecter ms_handle_connect 0x7fc8ac0068b0 2021-05-25T12:09:09.079 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.033+0000 7fc8baffd700 10 client.244295.objecter resend_mon_ops 2021-05-25T12:09:09.080 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.034+0000 7fc8baffd700 1 -- 172.21.15.102:0/915905625 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2582179455 0 0) 0x7fc8ac007510 con 0x7fc8ac0068b0 2021-05-25T12:09:09.080 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.034+0000 7fc8baffd700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:09:09.080 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.034+0000 7fc8baffd700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:09:09.080 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.034+0000 7fc8baffd700 10 monclient(hunting): my global_id is 244301 2021-05-25T12:09:09.080 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.034+0000 7fc8baffd700 1 -- 172.21.15.102:0/915905625 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8ac0065a0 con 0x7fc8ac0068b0 2021-05-25T12:09:09.081 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.034+0000 7fc8baffd700 1 -- 172.21.15.102:0/915905625 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2997702279 0 0) 0x7fc8ac0065a0 con 0x7fc8ac0068b0 2021-05-25T12:09:09.081 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.034+0000 7fc8baffd700 1 -- 172.21.15.102:0/915905625 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc8ac007510 con 0x7fc8ac0068b0 2021-05-25T12:09:09.081 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.034+0000 7fc8baffd700 1 -- 172.21.15.102:0/915905625 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7fc8c4004f10 con 0x7fc8ac0068b0 2021-05-25T12:09:09.081 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.034+0000 7fc8baffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:09:09.081 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.034+0000 7fc8baffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:09:09.081 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.034+0000 7fc8baffd700 10 monclient: dump: 2021-05-25T12:09:09.082 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:09:09.082 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:09:09.082 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:09:09.082 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:09:09.082 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:09:09.083 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:09:09.083 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:09:09.083 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:09:09.083 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:09:09.083 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:09:09.084 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.034+0000 7fc8baffd700 1 -- 172.21.15.102:0/915905625 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1747636928 0 0) 0x7fc8ac007510 con 0x7fc8ac0068b0 2021-05-25T12:09:09.084 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.034+0000 7fc8baffd700 1 -- 172.21.15.102:0/915905625 --> v1:172.21.15.102:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fc8ac008770 con 0x7fc8ac0068b0 2021-05-25T12:09:09.084 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.034+0000 7fc8baffd700 1 -- 172.21.15.102:0/915905625 <== mon.2 v1:172.21.15.102:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3929 (unknown 0 0 3410642474) 0x7fc8c40051b0 con 0x7fc8ac0068b0 2021-05-25T12:09:09.084 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.034+0000 7fc8baffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:09:09.084 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.034+0000 7fc8baffd700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:09:09.085 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.034+0000 7fc8baffd700 1 -- 172.21.15.102:0/915905625 >> v1:172.21.15.102:6790/0 conn(0x7fc8ac0068b0 legacy=0x7fc8ac006d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:09:09.085 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.034+0000 7fc8baffd700 1 -- 172.21.15.102:0/915905625 >> v1:172.21.15.102:6790/0 conn(0x7fc8ac0068b0 legacy=0x7fc8ac006d40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:09:09.085 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:09:09.086 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:09:09.086 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:09:09.086 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "peon", 2021-05-25T12:09:09.086 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 78, 2021-05-25T12:09:09.086 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:09:09.087 INFO:teuthology.orchestra.run.smithi102.stdout: 1, 2021-05-25T12:09:09.087 INFO:teuthology.orchestra.run.smithi102.stdout: 2 2021-05-25T12:09:09.087 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:09:09.087 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 0, 2021-05-25T12:09:09.088 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:09:09.088 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:09:09.088 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:09:09.088 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:09:09.088 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:09:09.089 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:09:09.089 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:09:09.089 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:09:09.089 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:09:09.090 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:09:09.090 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:09:09.090 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:09:09.090 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:09:09.090 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:09:09.091 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:09:09.091 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:09:09.091 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:09:09.091 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:09:09.092 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:09:09.092 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:09:09.092 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:09:09.092 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:09:09.092 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:09:09.093 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:09:09.093 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:09:09.093 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:09:09.093 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:09:09.093 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:09:09.094 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:09:09.094 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:09:09.094 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:09:09.094 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:09:09.094 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:09:09.095 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:09:09.095 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:09:09.095 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:09:09.095 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:09:09.095 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:09:09.096 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:09:09.096 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:09:09.096 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:09:09.096 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:09:09.097 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:09:09.097 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:09:09.097 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:09:09.097 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:09:09.098 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:09:09.098 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:09:09.098 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:09:09.098 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:09:09.098 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:09:09.099 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:09:09.099 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:09:09.099 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:09:09.099 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:09:09.100 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:09:09.100 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:09:09.100 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:09:09.100 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:09:09.100 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:09:09.101 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:09:09.101 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:09:09.101 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:09:09.101 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:09:09.101 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:09:09.102 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:09:09.102 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:09:09.102 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:09:09.102 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:09:09.102 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:09:09.103 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:09:09.103 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:09:09.103 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:09:09.103 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:09:09.104 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:09:09.104 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:09:09.104 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:09:09.104 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:09:09.104 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:09:09.105 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:09:09.105 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:09:09.105 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:09:09.105 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:09:09.105 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:09:09.106 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:09:09.106 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:09:09.106 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:09:09.106 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:09:09.106 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:09:09.107 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:09:09.107 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:09:09.107 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:09:09.107 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:09:09.107 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:09:09.108 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:09:09.108 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:09:09.108 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:09:09.108 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:09:09.108 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:09:09.109 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:09:09.109 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:09:09.109 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:09:09.109 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:09:09.110 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:09:09.110 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:09:09.110 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:09:09.110 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:09:09.110 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:09:09.111 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:09:09.111 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:09:09.111 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:09:09.111 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:09:09.111 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:09:09.112 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:09:09.112 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:09:09.112 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:09:09.112 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:09:09.113 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:09:09.113 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": [ 2021-05-25T12:09:09.113 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:09:09.113 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:09:09.113 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:09:09.114 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 2 2021-05-25T12:09:09.114 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:09:09.114 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:09:09.114 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:09:09.114 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:09:09.115 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:09:09.115 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:09:09.115 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 2 2021-05-25T12:09:09.115 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:09:09.116 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:09:09.116 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:09:09.116 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:09:09.116 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:09:09.118 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.034+0000 7fc8d95ac700 10 librados: watch_flush enter 2021-05-25T12:09:09.119 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.034+0000 7fc8d95ac700 10 librados: watch_flush exit 2021-05-25T12:09:09.119 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.034+0000 7fc8d95ac700 20 client.244295.objecter shutdown clearing up homeless session... 2021-05-25T12:09:09.119 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.034+0000 7fc8d95ac700 10 client.244295.objecter successfully canceled tick 2021-05-25T12:09:09.119 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.034+0000 7fc8d95ac700 1 -- 172.21.15.102:0/915905625 >> v1:172.21.15.181:6812/29582 conn(0x7fc8b4033180 legacy=0x7fc8b4035640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:09:09.120 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.034+0000 7fc8d95ac700 10 monclient: shutdown 2021-05-25T12:09:09.120 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.034+0000 7fc8d95ac700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:09:09.120 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.034+0000 7fc8d95ac700 1 -- 172.21.15.102:0/915905625 >> v1:172.21.15.102:6790/0 conn(0x7fc8d4069190 legacy=0x7fc8d413b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:09:09.120 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.034+0000 7fc8d37fe700 1 -- 172.21.15.102:0/915905625 reap_dead start 2021-05-25T12:09:09.120 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.035+0000 7fc8d95ac700 1 -- 172.21.15.102:0/915905625 shutdown_connections 2021-05-25T12:09:09.121 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.035+0000 7fc8d95ac700 1 -- 172.21.15.102:0/915905625 >> 172.21.15.102:0/915905625 conn(0x7fc8d409af90 msgr2=0x7fc8d412eac0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:09:09.121 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.035+0000 7fc8d95ac700 1 -- 172.21.15.102:0/915905625 shutdown_connections 2021-05-25T12:09:09.121 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.035+0000 7fc8d95ac700 1 -- 172.21.15.102:0/915905625 wait complete. 2021-05-25T12:09:09.121 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.035+0000 7fc8d95ac700 1 librados: shutdown 2021-05-25T12:09:09.122 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2021-05-25T12:09:09.233 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.231+0000 7f72301c1700 10 monclient: get_monmap_and_config 2021-05-25T12:09:09.233 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.231+0000 7f72301c1700 10 monclient: build_initial_monmap 2021-05-25T12:09:09.233 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.231+0000 7f72301c1700 10 monclient: monmap: 2021-05-25T12:09:09.233 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:09:09.234 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:09:09.234 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T12:09:09.232792+0000 2021-05-25T12:09:09.234 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T12:09:09.232792+0000 2021-05-25T12:09:09.234 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:09:09.235 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:09:09.235 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:09:09.235 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c 2021-05-25T12:09:09.235 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2021-05-25T12:09:09.235 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:09:09.248 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.232+0000 7f72301c1700 1 Processor -- start 2021-05-25T12:09:09.248 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.232+0000 7f72301c1700 1 -- start start 2021-05-25T12:09:09.248 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.232+0000 7f72301c1700 10 monclient: init 2021-05-25T12:09:09.248 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.232+0000 7f72301c1700 10 monclient: _reopen_session rank -1 2021-05-25T12:09:09.248 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.233+0000 7f72301c1700 10 monclient: _add_conns ranks=[2,0,1] 2021-05-25T12:09:09.249 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.233+0000 7f72301c1700 10 monclient(hunting): picked mon.noname-b con 0x7f7228093d90 addr v1:172.21.15.102:6790/0 2021-05-25T12:09:09.249 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.233+0000 7f72301c1700 10 monclient(hunting): picked mon.noname-a con 0x7f7228132900 addr v1:172.21.15.102:6789/0 2021-05-25T12:09:09.249 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.233+0000 7f72301c1700 10 monclient(hunting): picked mon.noname-c con 0x7f7228136410 addr v1:172.21.15.181:6789/0 2021-05-25T12:09:09.249 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.233+0000 7f72301c1700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7228091ed0 con 0x7f7228132900 2021-05-25T12:09:09.250 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.233+0000 7f72301c1700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7228092170 con 0x7f7228136410 2021-05-25T12:09:09.250 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.233+0000 7f72301c1700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7228092410 con 0x7f7228093d90 2021-05-25T12:09:09.250 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.233+0000 7f72301c1700 10 monclient(hunting): _renew_subs 2021-05-25T12:09:09.250 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.233+0000 7f72301c1700 10 monclient(hunting): authenticate will time out at 1349.830086s 2021-05-25T12:09:09.251 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.233+0000 7f722d75c700 1 -- >> v1:172.21.15.102:6789/0 conn(0x7f7228132900 legacy=0x7f7228134cf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:09:09.251 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.233+0000 7f722df5d700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f7228093d90 0x7f72280691c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:57716/0 (socket says 172.21.15.102:57716) 2021-05-25T12:09:09.251 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.233+0000 7f722df5d700 1 -- 172.21.15.102:0/2816161056 learned_addr learned my addr 172.21.15.102:0/2816161056 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:09:09.251 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.233+0000 7f722cf5b700 1 -- 172.21.15.102:0/2816161056 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2134210141 0 0) 0x7f7228092410 con 0x7f7228093d90 2021-05-25T12:09:09.252 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.233+0000 7f722cf5b700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:09:09.252 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.233+0000 7f722cf5b700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:09:09.252 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.233+0000 7f722cf5b700 10 monclient(hunting): my global_id is 244304 2021-05-25T12:09:09.252 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.233+0000 7f722cf5b700 1 -- 172.21.15.102:0/2816161056 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7210003270 con 0x7f7228093d90 2021-05-25T12:09:09.252 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.233+0000 7f722cf5b700 1 -- 172.21.15.102:0/2816161056 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 445732083 0 0) 0x7f7228092170 con 0x7f7228136410 2021-05-25T12:09:09.253 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.233+0000 7f722cf5b700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:09:09.253 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.233+0000 7f722cf5b700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:09:09.253 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.233+0000 7f722cf5b700 10 monclient(hunting): my global_id is 224563 2021-05-25T12:09:09.253 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.233+0000 7f722cf5b700 1 -- 172.21.15.102:0/2816161056 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7228092410 con 0x7f7228136410 2021-05-25T12:09:09.253 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.233+0000 7f722cf5b700 1 -- 172.21.15.102:0/2816161056 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2484123125 0 0) 0x7f7210003270 con 0x7f7228093d90 2021-05-25T12:09:09.254 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.233+0000 7f722cf5b700 1 -- 172.21.15.102:0/2816161056 >> v1:172.21.15.181:6789/0 conn(0x7f7228136410 legacy=0x7f7228138800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:09:09.254 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.233+0000 7f722cf5b700 1 -- 172.21.15.102:0/2816161056 >> v1:172.21.15.102:6789/0 conn(0x7f7228132900 legacy=0x7f7228134cf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-05-25T12:09:09.254 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.233+0000 7f722cf5b700 10 monclient: _finish_hunting 0 2021-05-25T12:09:09.254 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.233+0000 7f722cf5b700 1 monclient: found mon.noname-b 2021-05-25T12:09:09.255 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.233+0000 7f722cf5b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:09:09.255 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.233+0000 7f722cf5b700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2021-05-25T12:09:09.255 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.233+0000 7f722cf5b700 1 -- 172.21.15.102:0/2816161056 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7228094170 con 0x7f7228093d90 2021-05-25T12:09:09.255 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.233+0000 7f722cf5b700 10 monclient: _finish_auth 0 2021-05-25T12:09:09.255 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.233+0000 7f722cf5b700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:09:09.256 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.233+0000 7f72301c1700 5 monclient: authenticate success, global_id 244304 2021-05-25T12:09:09.256 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.233+0000 7f722cf5b700 1 -- 172.21.15.102:0/2816161056 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f7218001610 con 0x7f7228093d90 2021-05-25T12:09:09.256 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.233+0000 7f72301c1700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:09:09.256 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.233+0000 7f722cf5b700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:09:09.257 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.233+0000 7f722cf5b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-05-25T12:09:09.257 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.233+0000 7f722cf5b700 10 monclient: dump: 2021-05-25T12:09:09.257 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:09:09.257 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:09:09.257 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:09:09.258 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:09:09.258 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:09:09.258 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:09:09.258 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:09:09.259 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:09:09.259 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:09:09.259 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:09:09.259 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.233+0000 7f72301c1700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-05-25T12:09:09.259 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.233+0000 7f722cf5b700 1 -- 172.21.15.102:0/2816161056 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f72180018f0 con 0x7f7228093d90 2021-05-25T12:09:09.260 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.233+0000 7f722cf5b700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:09:09.260 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.233+0000 7f72301c1700 10 monclient: get_monmap_and_config success 2021-05-25T12:09:09.260 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.233+0000 7f722cf5b700 1 -- 172.21.15.102:0/2816161056 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f7218001c80 con 0x7f7228093d90 2021-05-25T12:09:09.260 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.233+0000 7f722cf5b700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:09:09.261 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.233+0000 7f72301c1700 4 set_mon_vals no callback set 2021-05-25T12:09:09.261 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.233+0000 7f722cf5b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:09:09.261 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.233+0000 7f722cf5b700 10 monclient: dump: 2021-05-25T12:09:09.261 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:09:09.261 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:09:09.262 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:09:09.262 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:09:09.262 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:09:09.262 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:09:09.262 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:09:09.263 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:09:09.263 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:09:09.263 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:09:09.263 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.234+0000 7f72301c1700 10 monclient: shutdown 2021-05-25T12:09:09.264 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.234+0000 7f72301c1700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:09:09.264 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.234+0000 7f72301c1700 1 -- 172.21.15.102:0/2816161056 >> v1:172.21.15.102:6790/0 conn(0x7f7228093d90 legacy=0x7f72280691c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:09:09.264 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.234+0000 7f72301c1700 1 -- 172.21.15.102:0/2816161056 shutdown_connections 2021-05-25T12:09:09.264 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.234+0000 7f72301c1700 1 -- 172.21.15.102:0/2816161056 >> 172.21.15.102:0/2816161056 conn(0x7f722809b370 msgr2=0x7f7228097430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:09:09.264 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.234+0000 7f72301c1700 1 -- 172.21.15.102:0/2816161056 shutdown_connections 2021-05-25T12:09:09.265 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.234+0000 7f72301c1700 1 -- 172.21.15.102:0/2816161056 wait complete. 2021-05-25T12:09:09.265 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.234+0000 7f72301c1700 10 monclient: build_initial_monmap 2021-05-25T12:09:09.265 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.234+0000 7f72301c1700 10 monclient: monmap: 2021-05-25T12:09:09.265 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2021-05-25T12:09:09.265 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-05-25T12:09:09.266 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2021-05-25T12:09:09.266 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2021-05-25T12:09:09.266 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2021-05-25T12:09:09.266 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2021-05-25T12:09:09.267 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2021-05-25T12:09:09.267 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.noname-b 2021-05-25T12:09:09.267 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2021-05-25T12:09:09.267 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:09:09.267 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.234+0000 7f72301c1700 1 librados: starting msgr at 2021-05-25T12:09:09.268 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.234+0000 7f72301c1700 1 librados: starting objecter 2021-05-25T12:09:09.268 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.235+0000 7f72301c1700 1 Processor -- start 2021-05-25T12:09:09.268 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.235+0000 7f72301c1700 1 -- start start 2021-05-25T12:09:09.268 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.235+0000 7f72301c1700 1 librados: setting wanted keys 2021-05-25T12:09:09.268 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.235+0000 7f72301c1700 1 librados: calling monclient init 2021-05-25T12:09:09.269 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.235+0000 7f72301c1700 10 monclient: init 2021-05-25T12:09:09.269 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.235+0000 7f72301c1700 10 monclient: _reopen_session rank -1 2021-05-25T12:09:09.269 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.235+0000 7f72301c1700 10 monclient: _add_conns ranks=[2,0,1] 2021-05-25T12:09:09.269 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.235+0000 7f72301c1700 10 monclient(hunting): picked mon.noname-c con 0x7f7228093d90 addr v1:172.21.15.102:6790/0 2021-05-25T12:09:09.270 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.235+0000 7f72301c1700 10 monclient(hunting): picked mon.noname-a con 0x7f7228132900 addr v1:172.21.15.102:6789/0 2021-05-25T12:09:09.270 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.235+0000 7f72301c1700 10 monclient(hunting): picked mon.noname-b con 0x7f7228136410 addr v1:172.21.15.181:6789/0 2021-05-25T12:09:09.270 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.235+0000 7f72301c1700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7228068d50 con 0x7f7228132900 2021-05-25T12:09:09.270 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.235+0000 7f72301c1700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72281401d0 con 0x7f7228136410 2021-05-25T12:09:09.270 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.235+0000 7f72301c1700 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7228069580 con 0x7f7228093d90 2021-05-25T12:09:09.271 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.235+0000 7f72301c1700 10 monclient(hunting): _renew_subs 2021-05-25T12:09:09.271 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.235+0000 7f72301c1700 10 monclient(hunting): authenticate will time out at 1349.832574s 2021-05-25T12:09:09.271 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.235+0000 7f722d75c700 1 -- >> v1:172.21.15.102:6789/0 conn(0x7f7228132900 legacy=0x7f722813fac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.102:6789/0 2021-05-25T12:09:09.271 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.235+0000 7f722df5d700 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f7228093d90 0x7f722813f3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:57722/0 (socket says 172.21.15.102:57722) 2021-05-25T12:09:09.271 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.235+0000 7f722df5d700 1 -- 172.21.15.102:0/1204253791 learned_addr learned my addr 172.21.15.102:0/1204253791 (peer_addr_for_me v1:172.21.15.102:0/0) 2021-05-25T12:09:09.272 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.235+0000 7f7215ffb700 10 client.?.objecter ms_handle_connect 0x7f7228093d90 2021-05-25T12:09:09.272 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.235+0000 7f7215ffb700 10 client.?.objecter resend_mon_ops 2021-05-25T12:09:09.272 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.235+0000 7f7215ffb700 1 -- 172.21.15.102:0/1204253791 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3968506324 0 0) 0x7f7228069580 con 0x7f7228093d90 2021-05-25T12:09:09.272 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.235+0000 7f7215ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:09:09.273 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.235+0000 7f7215ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:09:09.273 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.235+0000 7f7215ffb700 10 monclient(hunting): my global_id is 244307 2021-05-25T12:09:09.273 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.235+0000 7f7215ffb700 1 -- 172.21.15.102:0/1204253791 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7208004c10 con 0x7f7228093d90 2021-05-25T12:09:09.273 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.235+0000 7f7215ffb700 10 client.?.objecter ms_handle_connect 0x7f7228136410 2021-05-25T12:09:09.273 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.235+0000 7f7215ffb700 10 client.?.objecter resend_mon_ops 2021-05-25T12:09:09.274 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.236+0000 7f7215ffb700 1 -- 172.21.15.102:0/1204253791 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2946025212 0 0) 0x7f7208004c10 con 0x7f7228093d90 2021-05-25T12:09:09.274 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.236+0000 7f7215ffb700 1 -- 172.21.15.102:0/1204253791 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7228069580 con 0x7f7228093d90 2021-05-25T12:09:09.274 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.236+0000 7f7215ffb700 1 -- 172.21.15.102:0/1204253791 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f72180015d0 con 0x7f7228093d90 2021-05-25T12:09:09.274 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.236+0000 7f7215ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-05-25T12:09:09.274 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.236+0000 7f7215ffb700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-05-25T12:09:09.275 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.236+0000 7f7215ffb700 10 monclient(hunting): dump: 2021-05-25T12:09:09.275 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:09:09.275 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:09:09.275 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:09:09.276 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:09:09.276 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:09:09.276 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:09:09.276 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:09:09.276 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:09:09.277 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:09:09.277 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:09:09.277 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.236+0000 7f7215ffb700 10 monclient(hunting): _finish_auth 0 2021-05-25T12:09:09.277 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.236+0000 7f7215ffb700 1 -- 172.21.15.102:0/1204253791 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 445762241 0 0) 0x7f72281401d0 con 0x7f7228136410 2021-05-25T12:09:09.277 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.236+0000 7f7215ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:09:09.278 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.236+0000 7f7215ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:09:09.278 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.236+0000 7f7215ffb700 10 monclient(hunting): my global_id is 224566 2021-05-25T12:09:09.278 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.236+0000 7f7215ffb700 1 -- 172.21.15.102:0/1204253791 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7208004c10 con 0x7f7228136410 2021-05-25T12:09:09.278 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.236+0000 7f7215ffb700 1 -- 172.21.15.102:0/1204253791 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3414875494 0 0) 0x7f7228069580 con 0x7f7228093d90 2021-05-25T12:09:09.279 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.236+0000 7f7215ffb700 1 -- 172.21.15.102:0/1204253791 >> v1:172.21.15.181:6789/0 conn(0x7f7228136410 legacy=0x7f7228147700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:09:09.279 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.236+0000 7f7215ffb700 1 -- 172.21.15.102:0/1204253791 >> v1:172.21.15.102:6789/0 conn(0x7f7228132900 legacy=0x7f722813fac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-05-25T12:09:09.279 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.236+0000 7f7215ffb700 10 monclient: _finish_hunting 0 2021-05-25T12:09:09.279 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.236+0000 7f7215ffb700 1 monclient: found mon.c 2021-05-25T12:09:09.279 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.236+0000 7f7215ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-05-25T12:09:09.280 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.236+0000 7f7215ffb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:09:09.280 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.236+0000 7f7215ffb700 1 -- 172.21.15.102:0/1204253791 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7228094420 con 0x7f7228093d90 2021-05-25T12:09:09.280 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.236+0000 7f7215ffb700 10 monclient: _finish_auth 0 2021-05-25T12:09:09.280 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.236+0000 7f7215ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2021-05-25T12:09:09.280 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.236+0000 7f72301c1700 5 monclient: authenticate success, global_id 244307 2021-05-25T12:09:09.281 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.236+0000 7f72301c1700 10 monclient: _renew_subs 2021-05-25T12:09:09.281 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.236+0000 7f72301c1700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:09:09.281 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.236+0000 7f72301c1700 1 -- 172.21.15.102:0/1204253791 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7228095f30 con 0x7f7228093d90 2021-05-25T12:09:09.281 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.236+0000 7f7215ffb700 1 -- 172.21.15.102:0/1204253791 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7218004a80 con 0x7f7228093d90 2021-05-25T12:09:09.282 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.236+0000 7f7215ffb700 10 monclient: handle_config config(0 keys) v1 2021-05-25T12:09:09.282 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.236+0000 7f7215ffb700 1 -- 172.21.15.102:0/1204253791 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f7218004d90 con 0x7f7228093d90 2021-05-25T12:09:09.282 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.236+0000 7f7215ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:09:09.282 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.236+0000 7f7215ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-05-25T12:09:09.282 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.236+0000 7f7215ffb700 10 monclient: dump: 2021-05-25T12:09:09.283 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:09:09.283 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:09:09.283 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:09:09.283 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:09:09.283 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:09:09.284 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:09:09.284 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:09:09.284 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:09:09.284 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:09:09.285 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:09:09.285 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.236+0000 7f72301c1700 10 client.244307.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-05-25T12:09:09.285 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.236+0000 7f7217fff700 4 set_mon_vals no callback set 2021-05-25T12:09:09.285 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.236+0000 7f72301c1700 10 monclient: _renew_subs 2021-05-25T12:09:09.286 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.236+0000 7f72301c1700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2021-05-25T12:09:09.286 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.236+0000 7f72301c1700 1 -- 172.21.15.102:0/1204253791 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7228131ee0 con 0x7f7228093d90 2021-05-25T12:09:09.286 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.236+0000 7f72301c1700 1 librados: init done 2021-05-25T12:09:09.286 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.237+0000 7f7215ffb700 1 -- 172.21.15.102:0/1204253791 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 37293+0+0 (unknown 3322602430 0 0) 0x7f7218004f70 con 0x7f7228093d90 2021-05-25T12:09:09.286 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.237+0000 7f72301c1700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-05-25T12:09:09.287 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.237+0000 7f7215ffb700 1 -- 172.21.15.102:0/1204253791 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(297..297 src has 250..297) v4 ==== 5744+0+0 (unknown 282307743 0 0) 0x7f721803d9a0 con 0x7f7228093d90 2021-05-25T12:09:09.287 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.237+0000 7f7215ffb700 10 client.244307.objecter ms_dispatch 0x7f7228001550 osd_map(297..297 src has 250..297) v4 2021-05-25T12:09:09.287 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.237+0000 7f7215ffb700 3 client.244307.objecter handle_osd_map got epochs [297,297] > 0 2021-05-25T12:09:09.287 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.237+0000 7f7215ffb700 3 client.244307.objecter handle_osd_map decoding full epoch 297 2021-05-25T12:09:09.288 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.237+0000 7f72301c1700 1 -- 172.21.15.102:0/1204253791 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f7228139500 con 0x7f7228058d10 2021-05-25T12:09:09.288 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.237+0000 7f7215ffb700 20 client.244307.objecter dump_active .. 0 homeless 2021-05-25T12:09:09.288 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.237+0000 7f722d75c700 10 monclient: get_auth_request con 0x7f7208033170 auth_method 0 2021-05-25T12:09:09.288 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.237+0000 7f7215ffb700 10 client.244307.objecter ms_handle_connect 0x7f7228058d10 2021-05-25T12:09:09.288 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.237+0000 7f7215ffb700 10 client.244307.objecter resend_mon_ops 2021-05-25T12:09:09.289 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.237+0000 7f7215ffb700 10 client.244307.objecter ms_handle_connect 0x7f7208033170 2021-05-25T12:09:09.289 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.237+0000 7f7215ffb700 1 -- 172.21.15.102:0/1204253791 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2135903365 0 0) 0x7f7228139500 con 0x7f7228058d10 2021-05-25T12:09:09.289 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.237+0000 7f7215ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:09:09.289 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.237+0000 7f7215ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:09:09.290 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.237+0000 7f7215ffb700 10 monclient(hunting): my global_id is 224569 2021-05-25T12:09:09.290 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.237+0000 7f7215ffb700 1 -- 172.21.15.102:0/1204253791 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7228069580 con 0x7f7228058d10 2021-05-25T12:09:09.290 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.237+0000 7f7215ffb700 1 -- 172.21.15.102:0/1204253791 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3013823667 0 0) 0x7f7228069580 con 0x7f7228058d10 2021-05-25T12:09:09.290 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.237+0000 7f7215ffb700 1 -- 172.21.15.102:0/1204253791 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7228139500 con 0x7f7228058d10 2021-05-25T12:09:09.290 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.237+0000 7f7215ffb700 1 -- 172.21.15.102:0/1204253791 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f7224001ac0 con 0x7f7228058d10 2021-05-25T12:09:09.291 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.237+0000 7f7215ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:09:09.291 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.237+0000 7f7215ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T12:09:09.291 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.237+0000 7f7215ffb700 10 monclient: dump: 2021-05-25T12:09:09.291 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:09:09.291 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:09:09.292 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:09:09.292 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:09:09.292 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:09:09.292 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:09:09.293 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:09:09.293 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:09:09.293 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:09:09.293 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:09:09.293 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.238+0000 7f7215ffb700 1 -- 172.21.15.102:0/1204253791 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 746034232 0 0) 0x7f7228139500 con 0x7f7228058d10 2021-05-25T12:09:09.294 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.238+0000 7f7215ffb700 1 -- 172.21.15.102:0/1204253791 --> v1:172.21.15.181:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f722806bb20 con 0x7f7228058d10 2021-05-25T12:09:09.294 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.238+0000 7f7215ffb700 1 -- 172.21.15.102:0/1204253791 <== mon.1 v1:172.21.15.181:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (unknown 0 0 2801740443) 0x7f7224001ec0 con 0x7f7228058d10 2021-05-25T12:09:09.294 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.238+0000 7f7215ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2021-05-25T12:09:09.294 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.238+0000 7f7215ffb700 10 monclient: _finish_command 1 = system:0 2021-05-25T12:09:09.295 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.238+0000 7f7215ffb700 1 -- 172.21.15.102:0/1204253791 >> v1:172.21.15.181:6789/0 conn(0x7f7228058d10 legacy=0x7f7228132ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:09:09.295 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.238+0000 7f7215ffb700 1 -- 172.21.15.102:0/1204253791 >> v1:172.21.15.181:6789/0 conn(0x7f7228058d10 legacy=0x7f7228132ce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:09:09.295 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.245+0000 7f72301c1700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2021-05-25T12:09:09.295 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.245+0000 7f72301c1700 1 -- 172.21.15.102:0/1204253791 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f7228059500 con 0x7f72280590f0 2021-05-25T12:09:09.295 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.245+0000 7f7215ffb700 10 client.244307.objecter ms_handle_connect 0x7f72280590f0 2021-05-25T12:09:09.296 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.245+0000 7f7215ffb700 10 client.244307.objecter resend_mon_ops 2021-05-25T12:09:09.296 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.245+0000 7f7215ffb700 1 -- 172.21.15.102:0/1204253791 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4046356320 0 0) 0x7f7228059500 con 0x7f72280590f0 2021-05-25T12:09:09.296 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.245+0000 7f7215ffb700 10 monclient(hunting): _init_auth method 2 2021-05-25T12:09:09.296 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.245+0000 7f7215ffb700 10 monclient(hunting): _init_auth creating new auth 2021-05-25T12:09:09.296 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.245+0000 7f7215ffb700 10 monclient(hunting): my global_id is 224572 2021-05-25T12:09:09.297 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.245+0000 7f7215ffb700 1 -- 172.21.15.102:0/1204253791 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7228139500 con 0x7f72280590f0 2021-05-25T12:09:09.297 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.245+0000 7f7215ffb700 1 -- 172.21.15.102:0/1204253791 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1014009380 0 0) 0x7f7228139500 con 0x7f72280590f0 2021-05-25T12:09:09.297 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.245+0000 7f7215ffb700 1 -- 172.21.15.102:0/1204253791 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7228059500 con 0x7f72280590f0 2021-05-25T12:09:09.297 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.245+0000 7f7215ffb700 1 -- 172.21.15.102:0/1204253791 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3592099882 0 0) 0x7f7224002530 con 0x7f72280590f0 2021-05-25T12:09:09.298 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.245+0000 7f7215ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-05-25T12:09:09.298 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.245+0000 7f7215ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-05-25T12:09:09.298 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.245+0000 7f7215ffb700 10 monclient: dump: 2021-05-25T12:09:09.298 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2021-05-25T12:09:09.298 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 20ee91b6-8935-4b98-b868-85c03789ccc4 2021-05-25T12:09:09.299 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2021-05-25T11:59:11.011522+0000 2021-05-25T12:09:09.299 INFO:teuthology.orchestra.run.smithi102.stderr:created 2021-05-25T11:59:11.011522+0000 2021-05-25T12:09:09.299 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 17 (quincy) 2021-05-25T12:09:09.299 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2021-05-25T12:09:09.300 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2021-05-25T12:09:09.300 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.181:6789/0 mon.b 2021-05-25T12:09:09.300 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2021-05-25T12:09:09.300 INFO:teuthology.orchestra.run.smithi102.stderr: 2021-05-25T12:09:09.300 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.246+0000 7f7215ffb700 1 -- 172.21.15.102:0/1204253791 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3680424677 0 0) 0x7f7228059500 con 0x7f72280590f0 2021-05-25T12:09:09.301 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.246+0000 7f7215ffb700 1 -- 172.21.15.102:0/1204253791 --> v1:172.21.15.181:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f7228059730 con 0x7f72280590f0 2021-05-25T12:09:09.301 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.246+0000 7f7215ffb700 1 -- 172.21.15.102:0/1204253791 <== mon.1 v1:172.21.15.181:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4101 (unknown 0 0 604743022) 0x7f7224003850 con 0x7f72280590f0 2021-05-25T12:09:09.301 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.246+0000 7f7215ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2021-05-25T12:09:09.301 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.246+0000 7f7215ffb700 10 monclient: _finish_command 2 = system:0 2021-05-25T12:09:09.302 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.246+0000 7f7215ffb700 1 -- 172.21.15.102:0/1204253791 >> v1:172.21.15.181:6789/0 conn(0x7f72280590f0 legacy=0x7f722814ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:09:09.302 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.246+0000 7f7215ffb700 1 -- 172.21.15.102:0/1204253791 >> v1:172.21.15.181:6789/0 conn(0x7f72280590f0 legacy=0x7f722814ce40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-05-25T12:09:09.302 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2021-05-25T12:09:09.303 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:09:09.303 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:09:09.303 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "leader", 2021-05-25T12:09:09.303 INFO:teuthology.orchestra.run.smithi102.stdout: "election_epoch": 78, 2021-05-25T12:09:09.304 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum": [ 2021-05-25T12:09:09.304 INFO:teuthology.orchestra.run.smithi102.stdout: 1, 2021-05-25T12:09:09.304 INFO:teuthology.orchestra.run.smithi102.stdout: 2 2021-05-25T12:09:09.304 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:09:09.304 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_age": 0, 2021-05-25T12:09:09.305 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:09:09.305 INFO:teuthology.orchestra.run.smithi102.stdout: "required_con": "2449958755906961412", 2021-05-25T12:09:09.305 INFO:teuthology.orchestra.run.smithi102.stdout: "required_mon": [ 2021-05-25T12:09:09.305 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:09:09.306 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:09:09.306 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:09:09.306 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:09:09.306 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:09:09.306 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:09:09.307 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:09:09.307 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:09:09.307 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:09:09.307 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:09:09.308 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_con": "4540138303579357183", 2021-05-25T12:09:09.308 INFO:teuthology.orchestra.run.smithi102.stdout: "quorum_mon": [ 2021-05-25T12:09:09.308 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:09:09.308 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:09:09.308 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:09:09.309 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:09:09.309 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:09:09.309 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:09:09.309 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:09:09.310 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:09:09.310 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:09:09.310 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:09:09.310 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:09:09.310 INFO:teuthology.orchestra.run.smithi102.stdout: "outside_quorum": [], 2021-05-25T12:09:09.311 INFO:teuthology.orchestra.run.smithi102.stdout: "extra_probe_peers": [], 2021-05-25T12:09:09.311 INFO:teuthology.orchestra.run.smithi102.stdout: "sync_provider": [], 2021-05-25T12:09:09.311 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": { 2021-05-25T12:09:09.311 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 1, 2021-05-25T12:09:09.312 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "20ee91b6-8935-4b98-b868-85c03789ccc4", 2021-05-25T12:09:09.312 INFO:teuthology.orchestra.run.smithi102.stdout: "modified": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:09:09.312 INFO:teuthology.orchestra.run.smithi102.stdout: "created": "2021-05-25T11:59:11.011522Z", 2021-05-25T12:09:09.312 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release": 17, 2021-05-25T12:09:09.312 INFO:teuthology.orchestra.run.smithi102.stdout: "min_mon_release_name": "quincy", 2021-05-25T12:09:09.313 INFO:teuthology.orchestra.run.smithi102.stdout: "election_strategy": 3, 2021-05-25T12:09:09.313 INFO:teuthology.orchestra.run.smithi102.stdout: "disallowed_leaders: ": "", 2021-05-25T12:09:09.313 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false, 2021-05-25T12:09:09.313 INFO:teuthology.orchestra.run.smithi102.stdout: "features": { 2021-05-25T12:09:09.313 INFO:teuthology.orchestra.run.smithi102.stdout: "persistent": [ 2021-05-25T12:09:09.314 INFO:teuthology.orchestra.run.smithi102.stdout: "kraken", 2021-05-25T12:09:09.314 INFO:teuthology.orchestra.run.smithi102.stdout: "luminous", 2021-05-25T12:09:09.314 INFO:teuthology.orchestra.run.smithi102.stdout: "mimic", 2021-05-25T12:09:09.314 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap-prune", 2021-05-25T12:09:09.315 INFO:teuthology.orchestra.run.smithi102.stdout: "nautilus", 2021-05-25T12:09:09.315 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus", 2021-05-25T12:09:09.315 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific", 2021-05-25T12:09:09.315 INFO:teuthology.orchestra.run.smithi102.stdout: "elector-pinging", 2021-05-25T12:09:09.315 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy" 2021-05-25T12:09:09.316 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:09:09.316 INFO:teuthology.orchestra.run.smithi102.stdout: "optional": [] 2021-05-25T12:09:09.316 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:09:09.316 INFO:teuthology.orchestra.run.smithi102.stdout: "mons": [ 2021-05-25T12:09:09.316 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:09:09.317 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 0, 2021-05-25T12:09:09.317 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "a", 2021-05-25T12:09:09.317 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:09:09.317 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:09:09.317 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:09:09.318 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:09:09.318 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789", 2021-05-25T12:09:09.318 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:09:09.318 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:09:09.319 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:09:09.319 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:09:09.319 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6789/0", 2021-05-25T12:09:09.319 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6789/0", 2021-05-25T12:09:09.319 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:09:09.320 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:09:09.320 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:09:09.320 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:09:09.320 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:09:09.320 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 1, 2021-05-25T12:09:09.321 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "b", 2021-05-25T12:09:09.321 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:09:09.321 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:09:09.321 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:09:09.321 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:09:09.322 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789", 2021-05-25T12:09:09.322 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:09:09.322 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:09:09.322 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:09:09.323 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:09:09.323 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.181:6789/0", 2021-05-25T12:09:09.323 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.181:6789/0", 2021-05-25T12:09:09.323 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:09:09.323 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:09:09.324 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:09:09.324 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:09:09.324 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:09:09.324 INFO:teuthology.orchestra.run.smithi102.stdout: "rank": 2, 2021-05-25T12:09:09.324 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "c", 2021-05-25T12:09:09.325 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrs": { 2021-05-25T12:09:09.325 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2021-05-25T12:09:09.325 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:09:09.325 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2021-05-25T12:09:09.326 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790", 2021-05-25T12:09:09.326 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 0 2021-05-25T12:09:09.326 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:09:09.326 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:09:09.326 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:09:09.327 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6790/0", 2021-05-25T12:09:09.327 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "172.21.15.102:6790/0", 2021-05-25T12:09:09.327 INFO:teuthology.orchestra.run.smithi102.stdout: "priority": 0, 2021-05-25T12:09:09.327 INFO:teuthology.orchestra.run.smithi102.stdout: "weight": 0, 2021-05-25T12:09:09.327 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "{}" 2021-05-25T12:09:09.328 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:09:09.328 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:09:09.328 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:09:09.328 INFO:teuthology.orchestra.run.smithi102.stdout: "feature_map": { 2021-05-25T12:09:09.328 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": [ 2021-05-25T12:09:09.329 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:09:09.329 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:09:09.329 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:09:09.329 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:09:09.330 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:09:09.330 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:09:09.330 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": [ 2021-05-25T12:09:09.330 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:09:09.330 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:09:09.331 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:09:09.331 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 4 2021-05-25T12:09:09.331 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:09:09.331 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:09:09.331 INFO:teuthology.orchestra.run.smithi102.stdout: "client": [ 2021-05-25T12:09:09.332 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:09:09.332 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:09:09.332 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:09:09.332 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 6 2021-05-25T12:09:09.333 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:09:09.333 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2021-05-25T12:09:09.333 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": [ 2021-05-25T12:09:09.333 INFO:teuthology.orchestra.run.smithi102.stdout: { 2021-05-25T12:09:09.333 INFO:teuthology.orchestra.run.smithi102.stdout: "features": "0x3f01cfbb7ffdffff", 2021-05-25T12:09:09.334 INFO:teuthology.orchestra.run.smithi102.stdout: "release": "luminous", 2021-05-25T12:09:09.334 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 1 2021-05-25T12:09:09.334 INFO:teuthology.orchestra.run.smithi102.stdout: } 2021-05-25T12:09:09.334 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2021-05-25T12:09:09.334 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2021-05-25T12:09:09.335 INFO:teuthology.orchestra.run.smithi102.stdout: "stretch_mode": false 2021-05-25T12:09:09.335 INFO:teuthology.orchestra.run.smithi102.stdout:} 2021-05-25T12:09:09.337 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.246+0000 7f72301c1700 10 librados: watch_flush enter 2021-05-25T12:09:09.337 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.246+0000 7f72301c1700 10 librados: watch_flush exit 2021-05-25T12:09:09.337 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.246+0000 7f72301c1700 20 client.244307.objecter shutdown clearing up homeless session... 2021-05-25T12:09:09.338 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.246+0000 7f72301c1700 10 client.244307.objecter successfully canceled tick 2021-05-25T12:09:09.338 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.246+0000 7f72301c1700 1 -- 172.21.15.102:0/1204253791 >> v1:172.21.15.181:6812/29582 conn(0x7f7208033170 legacy=0x7f7208035630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:09:09.338 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.246+0000 7f72301c1700 10 monclient: shutdown 2021-05-25T12:09:09.338 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.246+0000 7f72301c1700 20 monclient: shutdown discarding 0 pending message(s) 2021-05-25T12:09:09.339 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.246+0000 7f72301c1700 1 -- 172.21.15.102:0/1204253791 >> v1:172.21.15.102:6790/0 conn(0x7f7228093d90 legacy=0x7f722813f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-05-25T12:09:09.339 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.246+0000 7f722e75e700 1 -- 172.21.15.102:0/1204253791 reap_dead start 2021-05-25T12:09:09.339 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.246+0000 7f72301c1700 1 -- 172.21.15.102:0/1204253791 shutdown_connections 2021-05-25T12:09:09.339 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.246+0000 7f72301c1700 1 -- 172.21.15.102:0/1204253791 >> 172.21.15.102:0/1204253791 conn(0x7f722809b370 msgr2=0x7f7228138df0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-05-25T12:09:09.339 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.246+0000 7f72301c1700 1 -- 172.21.15.102:0/1204253791 shutdown_connections 2021-05-25T12:09:09.340 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.246+0000 7f72301c1700 1 -- 172.21.15.102:0/1204253791 wait complete. 2021-05-25T12:09:09.340 INFO:teuthology.orchestra.run.smithi102.stderr:2021-05-25T12:09:09.246+0000 7f72301c1700 1 librados: shutdown 2021-05-25T12:09:09.340 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2021-05-25T12:09:14.638 INFO:tasks.workunit.client.0.smithi181.stdout: api_tier_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP (28123 ms) 2021-05-25T12:09:14.639 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP2 2021-05-25T12:09:14.639 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP2 (2995 ms) 2021-05-25T12:09:14.639 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [----------] 28 tests from LibRadosAio (242352 ms total) 2021-05-25T12:09:14.639 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: 2021-05-25T12:09:14.640 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP 2021-05-25T12:09:14.640 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.ReadIntoBufferlist 2021-05-25T12:09:14.640 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ OK ] LibRadosAioPP.ReadIntoBufferlist (3024 ms) 2021-05-25T12:09:14.640 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.XattrsRoundTripPP 2021-05-25T12:09:14.640 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ OK ] LibRadosAioPP.XattrsRoundTripPP (8984 ms) 2021-05-25T12:09:14.641 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RmXattrPP 2021-05-25T12:09:14.641 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RmXattrPP (59808 ms) 2021-05-25T12:09:14.641 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RemoveTestPP 2021-05-25T12:09:14.642 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RemoveTestPP (2993 ms) 2021-05-25T12:09:14.642 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP (74809 ms total) 2021-05-25T12:09:14.642 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: 2021-05-25T12:09:14.642 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP 2021-05-25T12:09:14.643 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ RUN ] LibRadosIoPP.XattrListPP 2021-05-25T12:09:14.643 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ OK ] LibRadosIoPP.XattrListPP (2908 ms) 2021-05-25T12:09:14.643 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP (2908 ms total) 2021-05-25T12:09:14.643 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: 2021-05-25T12:09:14.643 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC 2021-05-25T12:09:14.644 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleWritePP 2021-05-25T12:09:14.644 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleWritePP (13983 ms) 2021-05-25T12:09:14.644 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.WaitForSafePP 2021-05-25T12:09:14.644 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ OK ] LibRadosAioEC.WaitForSafePP (21102 ms) 2021-05-25T12:09:14.645 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP 2021-05-25T12:09:14.645 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP (6905 ms) 2021-05-25T12:09:14.645 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP2 2021-05-25T12:09:14.645 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP2 (7044 ms) 2021-05-25T12:09:14.645 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP3 2021-05-25T12:09:14.646 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP3 (2981 ms) 2021-05-25T12:09:14.646 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripSparseReadPP 2021-05-25T12:09:14.646 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripSparseReadPP (21138 ms) 2021-05-25T12:09:14.646 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripAppendPP 2021-05-25T12:09:14.646 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripAppendPP (6713 ms) 2021-05-25T12:09:14.647 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsCompletePP 2021-05-25T12:09:14.647 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsCompletePP (7021 ms) 2021-05-25T12:09:14.647 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsSafePP 2021-05-25T12:09:14.647 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsSafePP (7056 ms) 2021-05-25T12:09:14.648 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ReturnValuePP 2021-05-25T12:09:14.648 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ReturnValuePP (21183 ms) 2021-05-25T12:09:14.648 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushPP 2021-05-25T12:09:14.648 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushPP (6754 ms) 2021-05-25T12:09:14.648 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushAsyncPP 2021-05-25T12:09:14.649 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushAsyncPP (6991 ms) 2021-05-25T12:09:14.649 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP 2021-05-25T12:09:14.649 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP (21144 ms) 2021-05-25T12:09:14.650 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP2 2021-05-25T12:09:14.650 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP2 (3008 ms) 2021-05-25T12:09:14.650 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPP 2021-05-25T12:09:14.650 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPP (7015 ms) 2021-05-25T12:09:14.650 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPPNS 2021-05-25T12:09:14.651 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPPNS (7039 ms) 2021-05-25T12:09:14.651 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.StatRemovePP 2021-05-25T12:09:14.651 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ OK ] LibRadosAioEC.StatRemovePP (6805 ms) 2021-05-25T12:09:14.652 INFO:tasks.workunit.client.0.smithi181.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ExecuteClassPP 2021-05-25T12:09:25.918 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2021-05-25T12:09:25.922 DEBUG:teuthology.task.console_log:Killing console logger for smithi102 2021-05-25T12:09:25.923 DEBUG:teuthology.task.console_log:Killing console logger for smithi181 2021-05-25T12:09:25.923 DEBUG:teuthology.exit:Finished running handlers